Background job nirvana with Resque, Redis and God
Recently a requirement came up to add a background job queue to a Rails application. Thanks to a couple of excellent open source libraries I was up and running in no time.
Resque is a Ruby library that lets you place jobs on queues and process them later. It was created by defunkt and is in production use on Github. The documentation is excellent. I'm using Resque on a Rails app but it can be used with any Rack application.
First you need to install Redis. On OSX this is trivial with homebrew
brew install redis
If you installed Redis with homebrew it spits out some helpful information about how to start Redis on boot or you can start the server with
The installation notes for Resque are excellent so I won't repeat them here - head over to the README.
Booting the GUI
Resque ships with an great GUI that lets you track jobs and workers. To launch it from the root of your site run
Then you'll probably want to spawn some workers
COUNT=5 QUEUE=* rake resque:workers
In the GUI you should now see 5 workers waiting to do your bidding
Now you can start sending jobs to Resque and watch your workers eat them up. The GUI also provides a great way to review failures. The stack trace is even included!
Settings for production
There are a few issues to consider in production. Firstly if your Redis server is used by other applications you might want to specify a namespace for the application to use. You can do this by adding the following to an initialiser after Redis is configured.
Resque.Redis.namespace = "resque:YourApp"
As Redis supports running multiple databases on a server it may be possible to specify which database to use but that wasn't a requirement for me. I couldn't find anything in the documentation about this. If anyone knows anything about this let me know and I'll update the post.
In production it will be important that Redis and the workers have high availability. Redis can easily be configured to start on reboot but God can also monitor a live system and restart Redis if necessary. God is another Ruby based library for monitoring services on a server. The configuration files are written with Ruby so it is easy to extend it to whatever you'd like. Using God we can easily make sure our workers are well-managed and that if Redis does go down it will be restarted. There is even an example script in Resque that will get you started with managing workers.
With God running on the server we can be sure that Redis and our workers will have high availability, or that we'll know about it quickly if not.
Adding a highly available and robust background queue to the application took less than a day. Thanks to defunkt and mojombo for their excellent open source libraries!