Background job nirvana with Resque, Redis and God
Recently a requirement came up to add a background job queue to a Rails application. Thanks to a couple of excellent open source libraries I was up and running in no time.
Resque ¶
Resque is a Ruby library that lets you place jobs on queues and process them later. It was created by defunkt and is in production use on Github. The documentation is excellent. I’m using Resque on a Rails app but it can be used with any Rack application.
First you need to install Redis. On OSX this is trivial with homebrew
brew install redis
If you installed Redis with homebrew it spits out some helpful information about how to start Redis on boot or you can start the server with
redis-server
The installation notes for Resque are excellent so I won’t repeat them here - head over to the README.
Booting the GUI ¶
Resque ships with an great GUI that lets you track jobs and workers. To launch it from the root of your site run
resque-web
Then you’ll probably want to spawn some workers
COUNT=5 QUEUE=* rake resque:workers
In the GUI you should now see 5 workers waiting to do your bidding
Now you can start sending jobs to Resque and watch your workers eat them up. The GUI also provides a great way to review failures. The stack trace is even included!
Settings for production ¶
There are a few issues to consider in production. Firstly if your Redis server is used by other applications you might want to specify a namespace for the application to use. You can do this by adding the following to an initialiser after Redis is configured.
Resque.Redis.namespace = "resque:YourApp"
As Redis supports running multiple databases on a server it may be possible to specify which database to use but that wasn’t a requirement for me. I couldn’t find anything in the documentation about this. If anyone knows anything about this let me know and I’ll update the post.
In production it will be important that Redis and the workers have high availability. Redis can easily be configured to start on reboot but God can also monitor a live system and restart Redis if necessary. God is another Ruby based library for monitoring services on a server. The configuration files are written with Ruby so it is easy to extend it to whatever you’d like. Using God we can easily make sure our workers are well-managed and that if Redis does go down it will be restarted. There is even an example script in Resque that will get you started with managing workers.
With God running on the server we can be sure that Redis and our workers will have high availability, or that we’ll know about it quickly if not.
General win ¶
Adding a highly available and robust background queue to the application took less than a day. Thanks to defunkt and mojombo for their excellent open source libraries!
Tags
Can you help make this article better? You can edit it here and send me a pull request.
See Also
-
Sass is a beautiful thing
Sass, or Syntactically Awesome Stylesheets, is a great tool to make managing CSS files more intelligent and easier to maintain. -
Developing subdomain Rails sites locally
Developing local Rails sites with subdomains can be a pain but thanks to a couple of public URLs it can be much simpler. -
Easing Rails development with Hirb
If you spend a lot of time in the console using Hirb is a great way to work faster and improve the output