Compiling Node.js from source is easy enough so here’s how.

Installing from a tarball

Node.js needs a few things to compile so make sure they are installed.

1
2

sudo apt-get update
sudo apt-get install build-essential openssl libssl-dev pkg-config

Get the link for the latest tarball from the Source Code link on the Node.js homepage. Then download it and extract it.

1
2
3
4
5

cd /usr/local/src
sudo mkdir node
cd node
sudo wget http://nodejs.org/dist/v0.6.6/node-v0.6.6.tar.gz
sudo tar -xzf node-v0.6.6.tar.gz

Now you can compile and install the binary. By default Node.js will be installed to /usr/local/bin/node and npm will be installed to /usr/local/bin/npm.

1
2
3
4

cd node-v0.6.6/
sudo ./configure
sudo make
sudo make install

Upgrading via a tarball

To upgrade Node.js simply download the new tarball, extract and repeat the installation process.

Installing from Git

If you plan to upgrade each time a new release comes out you may find cloning the Git repository more convenient than downloading and extracting a tarball each time.

You will need Git and the dependencies to be installed first.

1
2

sudo apt-get update
sudo apt-get install git-core build-essential openssl libssl-dev pkg-config

Using Git means that instead of downloading extracting a tarball you can just clone the repository and checkout the latest version. If you need to check the versions that are available run git tag.

1
2
3
4

cd /usr/local/src
sudo git clone git://github.com/joyent/node.git
cd node
sudo git checkout v0.6.6

Now you can compile the binary as before

1
2
3

sudo ./configure
sudo make
sudo make install

Upgrading via Git

To upgrade to a new release of Node.js first pull the latest source.

1
2
3

cd /usr/local/src/node
sudo git checkout master
sudo git pull origin master

Then checkout the latest version where v.x.x.x is the version you want.

1

sudo git checkout vx.x.x

Then follow the installation process as before

1
2
3

sudo ./configure
sudo make
sudo make install

This will upgrade Node.js and overwrite the previous version.

rbenv is great

If you haven’t seen rbenv yet I highly recommend it. It is a small collection of shell scripts that lets you manage rubies on UNIX type machines. To date I’ve used rvm to install versions of ruby and manage gemsets. Don’t get me wrong - I’m hugely greatful to the work that has gone into the rvm project and it is still a great tool. But I prefer the UNIX philosophy of doing one thing well and rbenv embraces that. I also really like that with rbenv you just need to amend your PATH and you are done. It follows UNIX conventions making integration with other UNIX tools easier - for me at least.

rbenv basics

We have a dev machine at work which runs a variety of Sinatra and Rails projects (including a Rails 1 project). As such these projects need lots of different versions of Ruby. To date I’ve used rvm to manage this and it has worked well but I always came up against issues when trying to integrate tools like Monit, puppet or init.d scripts. So I bit the bullet and switched to rbenv.

Each app has a UNIX user so rbenv is installed locally for each user when logged in as the user account

1
2

cd
git clone git://github.com/sstephenson/rbenv.git .rbenv

You need to load rbenv into the shell and you are good to go

1
2
3

echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc
source ~/.bashrc

If you have installed ruby-build you can then install rubies with rbenv install.

1

rbenv install 1.9.3-rc1

You can set a global ruby for the user account with

1

rbenv global 1.9.3-rc1

And you can also use an .rbenv-version file to set the version. I choose to use this in projects and check it into git.

Deployment with capistrano

I like that rbenv doesn’t try and manage rubies for you - we have bundler for that. I deploy as the same user as the app accounts so I just need to add the following to my cap recipe to use rbenv since I’m already including require 'bundler/capistrano'.

1
2
3

set :default_environment, {
  'PATH' => "/home/youruser/.rbenv/shims:/home/youruser/.rbenv/bin:$PATH"
}

This loads rbenv into the shell that capistrano uses. That’s pretty much all you need to use your rbenv installed ruby.

Following a thread on github you can also apply a clever technique to allow you switch versions of ruby by pushing a new .rbenv-version file with capistrano. From version 1.1rc bundler allows you to specify a shebang for binstubs. To use this add the following to your capistrano recipe.

1

set :bundle_flags, "--deployment --quiet --binstubs --shebang ruby-local-exec"

This generates executables in the bin folder of your Rails app which reference ruby-local-exec as the shebang. This command will execute whichever ruby is specified in the .rbenv-version file. In my case I have an init.d script to manage unicorn that references bin/unicorn. If I wanted to upgrade the ruby used by the app I would install it on the remote machine and then just update my .rbenv-version file commit, cap deploy and I’m done.

Integrating with other tools

I like to use monit to monitor processes. Monit exexutes scripts with a very limited shell but since rbenv follows UNIX conventions we can easily create a small bash script to load rbenv into the path and then do what we want.

Here’s the script I’m using for starting a resque_worker

1
2
3
4
5
6
7
8
9

#!/usr/bin/env bash
USER=youruser
APP_PATH=/srv/yourapp.com/current
PATH=/home/$USER/.rbenv/bin:/home/$USER/.rbenv/shims:$PATH
RAILS_ENV=production

cd $APP_PATH
RAILS_ENV=$RAILS_ENV bin/rake environment resque:work& > $APP_PATH/log/resque_worker.log
echo $! > $APP_PATH/tmp/pids/resque_worker.pid

Then my monit task looks like this

1
2
3
4

check process yourapp_resque_worker
    with pidfile /srv/yourapp.com/current/tmp/pids/resque_worker.pid
    start program = "/bin/sh -c '/home/youruser/bin/start_resque_worker.sh'" as uid youruser and gid youruser
    stop program = "/bin/sh -c 'cd /srv/yourapp.com/current && kill -s quit `cat tmp/pids/resque_worker.pid` && rm -f tmp/pids/resque_worker.pid; exit 0;'"

Better separation

By using rbenv, bundler, capistrano and monit together we have great separation between what these tools do

• rbenv manages rubies
• bundler manages gems
• capistrano manages deployments
• monit monitors processes

This feels clean and manageable to me.

By using binstubs with the ruby-local-exec shebang we also separate our app from versions of rubies, init.d and monit scripts, making it super easy to upgrade.

Thanks rvm, onwards with rbenv

I’m really grateful to Wayne E Seguin and the work he has put into rvm and I hope the project continues to thrive. rbenv fits the way I work though and I’m really enjoying using it. It makes piecing tools like puppet, monit, init.d scripts, capistrano and bundler much easier, so a big thanks to Sam Stephenson for creating it.

Riding Unicorns

For many reasons I like using Unicorn as a web server for Ruby apps. Ryan Tomayko wrote a great article and the many Unix features that it has. It has the killer feature of supporting hot restarts so you can deploy new releases with zero downtime. This means that in most cases you can deploy as often as you want and end users won’t even notice.

More power == more responsibility

With all of the great stuff that comes with Unicorn there is more configuration to do. This is the point at which the less intrepid opt for Passenger, that to be fair is stil a good option. But if you want a web server that uses great features of Unix and enables hot restarts you should persevere.

For Unicorn you need to set up a reverse proxy with nginx. I’ve written before about how to do that if you need a steer. You’ll also need to use upstart or script in /etc/init.d/ to manage Unicorn. There are a few scripts around on the web. I’ve posted the one that I use up as a gist on Github. You can then use something like this in your capistrano recipes to do a hot restart.

1
2
3
4
5

namespace :deploy do
  task :restart do
    sudo "/etc/init.d/unicorn upgrade"
  end
end

This sends a USR2 signal to the Unicorn master process to upgrade itself. It will create a new master process and then switch over when it is ready. See - zero downtime!

Up the workers

The Unicorn master process looks after workers and will reap workers that die from broken apps. In my scenario I was working on a memory constrained box and I wanted to make sure that workers did not consume too much memory. I like to use Monit to monitor processes. You can also use god or bluepill but I’ve found these tools consume much more memory that Monit. Call me old fashioned but I like to use Unix tools if they are available over Ruby.

So I want to monitor the Unicorn workers that are a child processes of the master Unicorn process. Bluepill supports monitoring child processes but it turns out this is easy enough to do with Monit.

Unicorn’s config file has an after_fork method that allows you to write out the pid of a worker. Here’s how

1
2
3
4
5

after_fork do |server, worker|
  defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
  child_pid = server.config[:pid].sub('.pid', ".#{worker.nr}.pid")
  system("echo #{Process.pid} > #{child_pid}")
end

If you want to see my full unicorn.conf it is here. This will write out the pid of each worker /tmp/pids/unicorn.[pid_id].pid, so now we can use Monit to watch it.

Adding in Monit

Now that we have the pids of workers and an init.d script that can manage workers too (see kill_worker) we can use Monit to keep everything in check. The only downside is that you need to tell Monit about each worker process that you have. If you are spawning and reaping Unicorn workers dynamically with TTIN and TTOU this probably isn’t going to work for you. But if the numbers of workers are fixed you are good.

1
2
3
4
5
6
7

check process unicorn_worker_0
    with pidfile /path/to/your/app/shared/pids/unicorn.0.pid
    start program = "/bin/cat /dev/null"
    stop program = "/etc/init.d/unicorn kill_worker 0"
    if mem is greater than 175.0 MB for 1 cycles then restart
    if cpu is greater than 22% for 2 cycles then alert
    if cpu is greater than 25% for 1 cycles then restart

Monit will watch the worker for memory growth and kill it off if it consumes too much. The unicorn master will spawn a new worker when it is killed so it is a neat solution to keep things running.

Much of this article was derived from Andrew Grim’s article ‘Where Unicorns go to die: Watching unicorn workers with monit’, so thanks to Andrew for the pointers.

Slicehost was a game changer

I’ve been a long standing Slicehost customer, after moving from Media Temple. Slicehost was perfect for my needs - I knew my way around Linux so I just wanted SSH access and the ability to configure it myself. At $20 a month Slicehost was a ground-breaking service. Coupled with a blog that spoke to developers about how to do things developers were doing every day it provided a range of distributions and was a massive hit with the developer community. On October 22nd 2008 Slicehost announced that they had been acquired by Rackspace. This was a just reward for what Slicehost had achieved but immediately there were concerns as to the direction of the service.

From public accounts Rackspace has used Slicehost’s underlying technology to build their Rackspace Cloud. In May 2011 Rackspace contacted Slicehost customers to say they would be migrated to the Rackspace Cloud product.

PR Car Crash

The email to customers raised more questions that it gave answers causing many customers to leave Slicehost there and then. There were so many questions that a further post was added to the forum which in fairness addressed many of the questions that customers had. At that point it became clear that the service Rackspace were offering was very different from Slicehost:

• DNS moving to Rackspace’s service. Unknown if it would be free
• Lower bandwidth allowance with pricing moving to pay what you use
• Some servers moving datacenters and changing IP addresses
• Removal of some Slice sizes
• Slicehost API likely to close
• A requirement to migrate to Rackspace by May 2012

Particularly if you use more than 60GB of bandwidth a month the service Rackspace’s Cloud Service would be more expensive.

Free markets are good

Thankfully the hosting market is very competitive and even Slicehost has competition in the form of Linode, a service launched in 2003. Linode offer a very similar service to Slicehost with the ability to provision servers, get SSH access and build it yourself. In fact Linode’s offering to customers is more competitive than Slicehost with 512MB of RAM offered for the entry price of $20 a month. Linode also offer free DNS servers, a great API and has also been a strong hit with the developer coummunity.

After looking at the direction of Rackspace Cloud and what Linode offer for the same price the clear choice was to migrate to the better product: Linode.

The pain of migrating

I’ve been with Slicehost for a while so with that comes many configurations, DNS zones and a server setup exactly how I like it. Rebuilding servers is tedious and sadly I built mine before Puppet was around. I use Puppet at work and this has shown me there is huge value in automating server management, especially in the context of Cloud providers.

After the decision to migrate I was staring at a terminal with many hours of work ahead of me, until the power of open source came to the rescue.

Migrating DNS

Firstly for migrating DNS records Rob Schultz created a short Ruby script to migrate DNS records from Slicehost to Linode. Once you’ve installed the script and necessary gem dependencies migrating your records is as simple at

1

./slicedns2linode.rb domain1.com.

Within a few minutes I had migrated all of my DNS records.

Migrating the server

If you are not already using Chef or Puppet there is a project called Blueprint that lets your reverse engineer a server configuration to a bash script, Puppet manifests or Chef cookbook. There is a good tutorial on a basic migration. Once you’ve installed blueprint and the dependencies you can create a shell script that will build another server with

1

blueprint create -S tutorial

Once you’ve created your scripts, just run them on the target server and it will build the server in the image of the old one.

My experience of Blueprint

The migration via blueprint went very smoothly. I had installed lots of packages adding third party repositories and compiling from source. Other than gitosis packages were mostly copied over and iptables was successfully configured.

Finishing up with scp

Finally I had to copy over website, home folder files and a few config files. The perfect tool for this was scp. First I created a tar.bz2 archive for folders I wanted to copy.

1

tar -cjf vhosts.tar.bz2 /var/www/vhosts

Then I copied over the archive to my new Linode instance

1

scp vhosts.tar.bz2 george@123.456.78.90:/var/www

Finally I unarchived it on the Linode instance

1

tar -xjf vhosts.tar.bz2

The great thing about using this approach is that permissions are preserved.

The last part was to change the name servers for my domains over to Linode.

Thanks Slicehost

Slicehost was a great service and the articles in particular helped me to skill up on UNIX and respond to many different scenarios. I’m sad that Slicehost will no longer be around and in particular that the service offering is so different from Rackspace. Slicehost was cloud hosting before it became trendy and now cloud hosting is something entirely different. I hope Linode can make me as happy - so far so good.

Counting is fun!

This is a walkthrough on how to create a simple realtime counter of visitors to a page using node.js, socket.io and express. This example is very simple but could easily be extended to a range of applications.

Just show me the app

http://counter.nodester.com

Pulling the pieces together

Express is a great web framework for creating sites in node.js. I’ve written before about creating a basic site. This time we are going to use a couple more libraries to add realtime communication to the application.

socket.io is an amazing library that takes the pain out of websockets and provides a reliable fallback depending on browser capabilites. Here’s a simple example from the socket.io site.

1
2
3
4
5
6
7
8

var io = require('socket.io').listen(80);

io.sockets.on('connection', function (socket) {
  socket.emit('news', { hello: 'world' });
  socket.on('my other event', function (data) {
    console.log(data);
  });
});

This sets up the server side. Then you just need to add some client-side JavaScript and you are ready to go

1
2
3
4
5
6
7
8

<script src="/socket.io/socket.io.js"></script>
<script>
  var socket = io.connect('http://localhost');
  socket.on('news', function (data) {
    console.log(data);
    socket.emit('my other event', { my: 'data' });
  });
</script>

I’ve used this setup to push the number of connected clients out to the browser and a simple counter to increment and decrement the number of connected clients on the relevant events. I’ve had some issues with different browsers on hosting environments. For now the demo works with Safari and Chrome.

Smoothie Charts is a great library that will draw graphs in JavaScript. It is very customisable, and lightweight and is specifically designed for live, streaming data. It uses canvas to draw the graphs and the tutorial shows just how easy it is to create a realtime charts.

Bringing these libraries together gives us live, realtime graphs. There’s no persistence here but if we wanted to we could add something like Redis into the mix to persist the maximum number of connections.

You can browse the source and see how the libraries are used on Github. Fork it!

Potentials uses

We could use this kind of setup to show realtime stats on server resources of the kind provided by Munin, to create realtime statistics for a page, or to stream live data from a third party API to a page. I’m excited about what people are going to make with these libraries!

The problem

To begin working on a Rails app I need to do things like starting a server, starting a console, ensure background processes are running, start a terminal window so I can watch logs, start a text editor etc etc.

Doing all this is tedious. Wouldn’t it be nice if we could just do something like?

1

start_myapp

Well you can!

Tmux

Tmux is a ‘terminal multiplexer’. What the hell’s that? It basially allows you to create terminal sessions - similar to GNU Screen where you can detach from a session and come back later. If you are not familiar with tmux there are a few good resources available

• A quick tmux (terminal multiplexer) screencast (a better GNU screen)
• Love, hate, & tmux
• TMUX - The Terminal Multiplexer (Part 1)
• TMUX - The Terminal Multiplexer (Part 2)

You can install tmux with homebrew

1

brew install tmux

It takes a little reading and configuring but is well worth effort if you are a heavy terminal user.

Tmuxinator

Tmuxinator is an excellent gem that lets you define the layouts of your tmux sessions.

You can install tmuxinator from RubyGems

1

gem install tmuxinator

By defining a yaml file we can layout things exactly how we want them.

1
2
3
4
5
6
7
8
9
10

project_name: myapp
project_root: ~/Sites/myapp
tabs:
  - zsh:
  - vim: vim .
  - foreman: bundle exec foreman start
  - git: git pull
  - console: bundle exec rails console
  - server: bundle exec rails server
  - logs: tail -f log/development.log

Then you can start your session with start_myapp and your layout will be ready for you. Here’s a screenshot from the documentation:

There’s excellent documentation on using tmuxinator in the project README

Foreman

Enter the final piece of the jigsaw - foreman. Foreman is a gem that lets you manage background processes associated with your application through a Procfile. You just add any processes you want to be started into the Procfile and you are done. Here’s an example

1
2
3
4
5

worker: bundle exec rake resque:work QUEUE=sweep_orders
apn_sender: bundle exec rake apn:sender
resque_scheduler: bundle exec rake resque:scheduler
resque_web: bundle exec resque-web --foreground --server thin --port $PORT --no-launch
sphinx: bundle exec rake ts:run_in_foreground

You can then start these process by running

1

foreman start

The processes will run in the foreground and spit out any logs messages to standard output - perfect for development.

Using foreman you can add a tab to tmuxinator that will start your processes and have a nice log of what they are doing. We have a neat way of starting any background processes in our app within the context of our unified tmux development environment.

So with one command we can set things up exactly the way we want and get straight into developing.

Alternatives

This technique will also work well for screen and regular terminal users with the Screeninator and Terminitor gems so if you are not sold on tmux you can still use it.

The goal

On a regular basis at pebble.code where I work we have a requirement to provision servers for clients, mostly for Rails apps. In the past we have used a combination of Cloud PaaS providers like Heroku or Engine Yard, we have built servers manually (gasp) or used bash scripts.

Puppet is a tool for building and managing servers and looked like a great fit for our requirements.

Learning curve

Puppet has a learning curve but there are lots of excellent resources available on the web. I found Pulling Strings With Puppet to be an excellent resource and there is also some good documentation on the Puppet site. I’ve written before on how to set up Puppet on Ubuntu 10.04 and also how to connect clients to a Puppet master.

Once you learn about resources and modules you are pretty much ready to get going.

Setting up Puppet Dashboard

Puppet comes with a dashboard that can be run on the Puppet Master to give a graphic view into what your nodes are doing, providing reports of updates and letting you know if there are any problems. There’s some good documentation offered by PuppetLabs on this to get you started.

I had some problems getting this going getting the error

1

can't activate rack (~> 1.0.1), already activated rack-1.2.2

This thread on the puppet-users mailing list gives a fix

1
2
3
4

cd <path-to-dashboard> 
git clone git://github.com/puppetlabs/puppet-dashboard.git 
rm -r vendor/gems/rack-1.0.1 
sed -i -e 's,~> 1.0.1,~> 1.2.2,' vendor/rails/actionpack/lib/action_controller.rb 

After that I was able to see the nodes and updates.

Rails stack choices

Then came the hard part - creating Puppet modules and using third party ones to install software on the server. I ended up writing my own for some but used existing modules where possible.

The ones I found most useful in building a Rails stack were

• puppet-rvm
• puppet-nginx
• puppetlabs-mysql
• puppet-monit

One issue is where to put the responsibility for some items in the stack. The web server is a good example. In my case I chose to use Unicorn and manage the installation of this via bundler and the Gemfile of the application. I could have chosen to use Puppet for this with something like Passenger. For automation Passenger is difficult to upgrade as it needs to be recompiled against Apache or Nginx. By chosing Unicorn I can upgrade it easily using bundler. To manage Unicorn I use an init script and hook this up to Monit and capistrano recipes. I’m still not sure about this approach but it works at least.

Switching the Puppet Master to Unicorn

As the number of nodes grew, and particularly where a new node was added I found that the default WEBrick server that ships with Puppet was not good enough and I started to hit some errors. Thankfully there is an excellent wiki article on the Puppet Labs site walking through how to switch Puppet to using Unicorn. Unicorn is a great web server and handles load balancing itself - perfect for my requirements.

Workflow

I chose to deploy using Capistrano and a git based workflow. I have an init script for the Unicorn Master powering Puppet so I can take advantage of the rolling restarts offered by Unicorn.

Still to do…

I still need to create a good framework for testing. Currently I run things against a blank VM. There are a few patterns for testing which I would like to become a bit more familiar with and use. I’d like to extend a few existing modules and write a few new ones, particularly for user managemet.

The good bit

The good bit is that we can now create a secure, bootstrapped server that can host any rack application with any cloud provider within about 20 minutes. Moreover we can manage SSH keys with Puppet so it is easy to grant and revoke developer access.

There’s more to learn but in my opinion the investment of time was well worth it.

After reading this article Matt Tanase contacted me about a great project called Blueprint. If you are porting existing infrastructure to Puppet have a look.

At pebble.coe where I work we have a Ubuntu 10.04 LTS server for our Rails and Ruby projects. We use this mostly internally but sometimes expose sites outside of our network to allow clients to quickly review work. When we started the business we built the Rails stack using the standard tools - Apache and Passenger. As our client list grew and we took on a couple of legacy projects we found that this stack no longer matched the environments that we were deploying too and didn’t give us the flexibility we wanted. So here’s how we changed it.

Promoting nginx to port 80

Nginx is quite simply an awesome web server. It is simple to configure, the memory footprint is small and it can be used to proxy traffic through the web server of our choice and can serve static assets like html, images and css directly before it hits the Rails app.

From having the ability to serve Rails sites only from Apache and Passenger we have moved to this.

We’ve moved Apache to port 81 so if a site needs to be tested on that setup we can still do it. Otherwise we can proxy from nginx directly to Apache. The real flexibility comes from the server we use to serve the Rails app. We deploy by choice on Thin or Unicorn so we can replicate this stack easily. We are looking at Goliath at the moment so we could easily add this in beneath Nginx. We are playing with some other technologies like node.js too. With Nginx we can support this too.

Getting nginx

If you want to make the switch on Ubuntu there is a PPA repo you can add

1
2
3
4
5

sudo -s
nginx=stable # use nginx=development for latest development version
add-apt-repository ppa:nginx/$nginx
apt-get update
apt-get install nginx

If you are currently using Apache you’ll need to decide whether or not to keep it on port 80. We switched Apache to port 81 and promoted Nginx to port 80.

Example configs

The Ubuntu install uses sites-available and sites-enabled folders to allow you to create site configs. Here are some examples that you can use to get you started.

Logs are disabled in these examples. If you want logging just add a path to the log file.

Default proxy settings

To keep your Nginx configs nice and DRY you can add a file to hold common proxy settings under /etc/nginx/conf.d called in my case proxy.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

proxy_redirect     off;

proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;

client_max_body_size       10m;
client_body_buffer_size    128k;

proxy_connect_timeout      90;
proxy_send_timeout         90;
proxy_read_timeout         90;

proxy_buffer_size          4k;
proxy_buffers              4 32k;
proxy_busy_buffers_size    64k;
proxy_temp_file_write_size 64k;

Proxy to Apache

This expects that you have setup an Apache virtual host on port 81.

1
2
3
4
5
6
7
8
9
10
11

server {
    listen       80;
    server_name  site1.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://127.0.0.1:81/;
    }
}

Proxy to Thin

1
2
3
4
5
6
7
8
9
10
11
12
13
14

upstream thin {
    server unix:/var/www/vhosts/site2.com/httpdocs/current/tmp/sockets/thin.0.sock;
}
server {
    listen       80;
    server_name  site2.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://thin;
    }
}

Proxy to Unicorn

1
2
3
4
5
6
7
8
9
10
11
12
13

upstream unicorn {
    server unix:/var/www/vhosts/site3.com/httpdocs/current/tmp/sockets/unicorn.sock fail_timeout=0;
}
server {
    listen       80;
    server_name  site3.com;

    access_log  off;
    error_log off;

    location / {
        proxy_pass         http://unicorn;
    }

So which one is best?

Having played a fair bit recently with servers to serve up Rails apps there isn’t a clear answer I’m afraid. I may get round to publishing my rudimentary benchmarks but I’ve found that for low memory deployments Nginx and Thin is a great choice. The response times are excellent and memory usage is much lower than Passenger or Unicorn. Thin doesn’t come with the same in-built load balancing that you get with Unicorn so HAProxy comes in to play here. HAProxy is another piece in the stack though which may put some off.

If memory isn’t an issue then Unicorn behind Nginx is a great choice. The rolling deployments are a great feature, but you also get free load balancing with Unicorn thanks to the power of UNIX.

If you are not interested in memory or tweaking you are probably best with Passenger behind Nginx.

If you are not interested in doing any sys admin work use Heroku, EngineYard or Brightbox.

Conclusion

If you are rolling your own servers and using Apache switch to Nginx!

What we are going to do

This walkthrough will go over setting up a basic site using node.js and Express. The walkthrough is aimed at beginners exploring node.js as I’ve had many questions from friends and colleagues about creating and deploying node apps. If you are not a beginner the article probably won’t be of much use to you. We are going to use express, an excellent web framework for node created by TJ Holowaychuk who seems to be pumping out node.js libraries like he was ten men.

Here is the site we are going to create. You might also want to grab the source code.

Setup

First we need to setup our development environment. If you are on OSX I’ve covered how to setup node.js and npm on OSX in a previous article. If you haven’t got everything installed follow that article.

If you are on Linux there are plenty of articles on Google.

For Windows users there are also resources on Google but it is a bit more tricky.

Prerequisites

If everything has installed ok you should now have node.js and npm running on your machine. At the terminal type node -v and npm -v and you should see something like:

1
2
3
4

node -v
v0.4.5
npm -v
1.0.1rc7

You’ll see I’m using the RC of version 1 of npm. I encourage you to install this as this will soon become the default.

Create an Express site

Still with me? We’ve covered a lot already! Now let’s create an Express site.

First let’s create a folder for our project and install Express.

1
2
3

mkdir express_example
cd express_example
npm install express

This installs Express into the directory we created. Now we can create the skeleton site. We are going to use jade and stylus for templating and css. More on those later.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

./node_modules/express/bin/express -t jade -c stylus
destination is not empty, continue? y
   create : .
   create : ./app.js
   create : ./public/stylesheets
   create : ./public/stylesheets/style.styl
   create : ./public/images
   create : ./public/javascripts
   create : ./logs
   create : ./pids
   create : ./test
   create : ./test/app.test.js
   create : ./views
   create : ./views/layout.jade
   create : ./views/index.jade
   - make sure you have installed stylus: $ npm install stylus
   - make sure you have installed jade: $ npm install jade

You’ll see that we need to install a couple of dependencies for stylus and jade support so let’s do that.

1

npm install stylus jade

Boot the app

That’s all the setup you need. Phew. Now you can boot the app:

1

node app.js

You should see Express server listening on port 3000 and if you open http://0.0.0.0:3000 you’ll see the default Express page.

Using Git

Git is a version control system that is used heavily in the node.js ecosystem, particulary with Github. If you aren’t familiar with Git Scott Chacon is your go-to man. He’s written extensively and eloquently on Git for beginners and experts. Checkout Gitcasts for if you are a beginner and ProGit for more advanced stuff. We are going to use git to version our site and publish it so let’s set up our repo now. If your Express server is still running hit CTRL + C to stop it.

1
2
3

git init
git add .
git commit -m 'initial commit'

Developing node.js sites

Normally when you develop a node.js site you’ll need ot restart your application each time you make a change. Thankfully our home-grown British JavaScript genius Remy Sharp has solved this problem with nodemon. Nodemon will reload your application each time it changes so you don’t need to restart it. If you have used Shotgun for Ruby with Sinatra it is similar to that. To install run

1

npm install -g nodemon

Then you can start your app with

1

nodemon app.js

Using nodemon means you don’t have to restart your app each time you make a change. For more infomation on nodemon see the README

HTML in Express

Express is agnostic as to which templating language you use. Templating languages can be a hot topic of debate but for this article I’m going to use jade. If you’ve used haml it is similar to that. In the example we use jade to setup a layout template.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

!!! 5
html
  head
    title= title
    link(rel='stylesheet', href='/stylesheets/style.css')
    link(rel='stylesheet', href='/stylesheets/chunkfive-fontface.css')
  body
    header
      nav
        ul
          li
            a(href="/") Home
          li
            a(href="/about") About
          li
            a(href="/contact") Contact
    section#wrapper!= body
        footer
          section.css-table
            section.four-column
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet
              section.cell
                p Mauris porttitor <br />felis eu leo aliquet<br /> ac rutrum odio aliquet

This is a common template we can reuse. The line section#wrapper!= body pulls in content from the page it is used on. Express also supports variables that you pass through to the template. In this case we pass the title variable. If you are coming from Sinatra this will all be familiar to you. If you are not I recommend consulting the Express documentation.

CSS in Express

Again Express is agnostic to what you use to generate your CSS - you can use vanilla CSS but for this example I’m using Stylus. This is very similar to Sass and supports variables, mixins, functions and more. I really like it! Here’s an example from our stylesheet

1
2
3
4
5
6
7
8
9
10
11
12
13

body
  font 62.5%/1.5  Helvetica, Arial, "Lucida Grande", "Lucida Sans", Tahoma, Verdana, sans-serif
  text-align center
  background #000

#wrapper
  width 920px
  text-align left
  margin-left auto
  margin-right auto
  background #fff
  padding 20px
  border-bottom-radius(15px)

You’ll see that stylus is very terse - you don’t need brackets or commas.

Routing in Express

Routing is similar to Sinatra, allowing you to set up RESTful routes.

In this example we setup three routes in app.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

app.get('/', function(req, res){
  res.render('index', {
    title: 'Home'
  });
});

app.get('/about', function(req, res){
  res.render('about', {
    title: 'About'
  });
});

app.get('/contact', function(req, res){
  res.render('contact', {
    title: 'Contact'
  });
});

See the Express documentation for more.

Publishing your site

We’ve now developed a basic node.js site using express and we want to host it somewhere.

There are a number of hosting options for node. It isn’t so difficult to host it yourself but for me the easiest and best is Nodester. If you have used Heroku the deployment process is very similar. You’ll need to request an invite for Nodester (it won’t come through immediately though).

1

curl -X POST -d "email=your_address@gmail.com" http://nodester.com/coupon

Chris Matthieu has created a handy video going over how to deploy to Nodester so go and watch that if you don’t fancy reading the documentation.

Let’s create the site on nodester

1

nodester app create express_example app.js

Now we can get the information on the site

1
2
3
4
5

nodester app info express_example
nodester info Gathering information about: express_example
nodester warn express_example on port 9451 running: false (pid: unknown)
nodester info gitrepo: ec2-user@nodester.com:/node/hosted_apps/shapeshed/1298-ec0117a54b696d7a9781c79e5283692e.git
nodester info appfile: app.js

To publish the site we need to add the git repo as a remote our git repo

1

git remote add nodester ec2-user@nodester.com:/node/hosted_apps/shapeshed/1298-ec0117a54b696d7a9781c79e5283692e.git

And finally we can publish the site by pushing

1

git push nodester master

Nodester will let you know about the deploy - if the deploy was successful you can see your app at http://[your_app_name].nodester.com/. In this example you can see the site at http://express_example.nodester.com

Some other node.js hosting providers include nodejitsu, Joyent, Cloud Foundry and Duostack.

Conclusion

This article has showed how to create a very basic site using node.js and Express. It has introduced a number of things from the node.js ecosystem and showed you how to deploy your app to Nodester.

The strengths of node.js as a technology are not so much in building static websites like this. I encourage you to explore some of the node.js libraries to see what it can do. Particularly for real-time applications node.js is extremely exciting and I think we’ll see some great apps built on node.js. Try starting with socket.io for a taste of what to expect.

If you find any inaccuracies in the post send me an email and I’ll update the post.

Further reading

• node.js
• express - node web framework
• npm - node package manager
• jade - node.js templating language
• stylus - node.js css framework
• Setting up node.js and npm on Mac OSX
• Nodester - Open Source Hosting for node.js
• Source code for this article

Open Source PaaS

VMware this week announced Cloud Foundry, an open source PaaS. You can use VMware’s cloud (which is where they’ll monetise this investment) or you can host your own. I believe you can even have a hybrid of both. Cloud Foundry joins other players in the market like Heroku, Joyent, Engine Yard and Nodester.

Why PaaS?

Developers don’t want to be Sys Admins. Especially in small businesses developers take on a lot - some or all of business analysis / requirements gathering, server admin, development, testing, design, client management. The list goes on. So the emergence of services like Heroku for Ruby and Nodester for node.js is very attractive. Developers can remove one thing from the list of things that they need to be responsible for. It is an easy sell to the management layer too - it is cheaper in terms of man hours, with (almost) no maintenance.

The providers doing well in the market are the ones that realise that developers have the keys to the choices that get made. Heroku has built a kick-ass platform with a super slick API. Frankly it is a joy to use. Nodester emerged from nowhere as an open-source weekend project and for me is beating players like Joyent in their node.js offering.

One Paas to rule them all?

VMware have put considerable effort into building a PaaS to accommodate three technologies - Java, Ruby and node.js. They must be applauded for open-sourcing the project and making it available on GitHub. There are a considerable amount of man hours there and a significant investment from VMware. Thank you.

Taking it for a spin

So I took it for a spin firing up a VirtualBox Ubuntu 10.04.2 LTS VM. Installation was simple enough following the README. I created the SSH tunnel as advised and within about half an hour had built a private cloud capable of running Java, Ruby and node.js apps, with support for MySQL, Redis and MongoDB. Impressive.

The VMware team have thought hard about the API. If you are a RubyGems user you can check it out by installing the vmc rubygem and then running help

1
2

gem install vmc --no-ri --no-rdoc
vmc --help

Once you have setup a user on your private cloud deploying an app is super simple. Here’s the process of deploying an app

1

vmc push [yourappname]

Err.. that’s it. Amazing!

Testing deployment on my cloud

Then I set about deploying a node.js app. It was a simple express based node.js application with express, jade and stylus as dependencies. So let’s give it a go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

vmc push myapp
Would you like to deploy from the current directory? [Yn]: y
Application Deployed URL: 'myapp.vcap.me'?
Detected a Node.js Application, is this correct? [Yn]: y
Memory Reservation [Default:64M] (64M, 128M, 256M, 512M, 1G or 2G)
Creating Application: OK
Would you like to bind any services to 'myapp'? [yN]: n
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (613K): OK
Push Status: OK
Staging Application: OK
Starting Application: ..........................Error 306: Error retrieving file 'logs/startup.log'

It failed. I used the log commands but there wasn’t any useful output - I guess this is early days. But I did find this article which required changing a couple of things in the app file. After that it was as easy as

1
2
3
4
5
6
7
8
9
10

vmc update myapp
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (15K): OK
Push Status: OK
Stopping Application: OK
Staging Application: OK
Starting Application: OK

Lovely! The API follows exactly the same process for deploying a Sinatra / Rails which is really attractive.

At pebble.code where I work I’m responsible for managing servers and setting up development sites for the team. This can be a time-consuming chore and I’ve been working towards a Puppet setup to allow me to automate much of this. A private cloud where developers can spin up apps with three words is massively better.

What’s good about Cloud Foundry

I love the fact that VMware understand that PaaS is about being open and not being tied to one platform. Being open makes me more likely to use a platform than one that makes it hard for me to get my data in and out. I can easily move my apps in and out at any time. I can create my own cloud and move my apps onto that. I’m free and I like that.

I love the fact that Cloud Foundry is open source and created using Ruby. The Ruby community is full of talented and smart developers and it is a wise move by VMware to engage that community. I anticipate the product will be enhanced by open source contributions.

What could be improved

It is a real shame that VMware didn’t go with git based deployment. Both the Ruby/Rails and node.js communities use git heavily and are used to git based PaaS services. I’m not sure why this design decision was made - perhaps the old skool Java heads at VMware won out. It would be great to see git deployment added to the platform. Currently there is no versioning on the platform - it would be a quick feature win to leverage the power of git to add this.

That said with the project being open source the developer community can easily fork it and enhance the platform. The fact that the project is open source is massive.

Will I use it?

Maybe. Currently I’m happy with the PaaS providers I’m using - Heroku and Nodester. I feel both are doing a great job. Heroku is closed source, whilst Nodester is open-source. I have the option to host a Nodester private cloud somewhere but Cloud Foundry is the only solution to offer both Ruby and node.js. I like that. Currently I mostly use PaaS providers for quickly prototyping an idea and getting it out there. I could see a Cloud Foundry private cloud working well for this.

The API is a big thing for me and I think VMware has done a great job here. Where Heroku has led others have followed and that’s great news for developers.

But.. for many of our production apps they have more complex technical architectures involving Resque, Redis, Sphinx, cron jobs and background processes. In short we need to have full control over the server. Cloud Foundry goes a long way to solving this by offering MongoDB and Redis but currently unless I’m mistaken I’d be unable to host apps with complex requirements. This is an issue with PaaS in general. Heroku’s add-ons solve most of this issue but there are times when simply you need root access.

So overall thank you VMware! I also need to say thank you for your support of Redis!

What are you waiting for? Give it a go, or fork it.

Further Reading

• Cloud Foundry
• VMware
• Heroku
• Nodester
• Cloud Foundry Source Code