Resque is a Ruby library that lets you place jobs on queues and process them later. It was created by defunkt and is in production use on Github. The documentation is excellent. I’m using Resque on a Rails app but it can be used with any Rack application.
First you need to install Redis. On OSX this is trivial with homebrew
brew install redis
If you installed Redis with homebrew it spits out some helpful information about how to start Redis on boot or you can start the server with
The installation notes for Resque are excellent so I won’t repeat them here - head over to the README.
Resque ships with an great GUI that lets you track jobs and workers. To launch it from the root of your site run
Then you’ll probably want to spawn some workers
COUNT=5 QUEUE=* rake resque:workers
In the GUI you should now see 5 workers waiting to do your bidding
Now you can start sending jobs to Resque and watch your workers eat them up. The GUI also provides a great way to review failures. The stack trace is even included!
There are a few issues to consider in production. Firstly if your Redis server is used by other applications you might want to specify a namespace for the application to use. You can do this by adding the following to an initialiser after Redis is configured.
Resque.Redis.namespace = "resque:YourApp"
As Redis supports running multiple databases on a server it may be possible to specify which database to use but that wasn’t a requirement for me. I couldn’t find anything in the documentation about this. If anyone knows anything about this let me know and I’ll update the post.
In production it will be important that Redis and the workers have high availability. Redis can easily be configured to start on reboot but God can also monitor a live system and restart Redis if necessary. God is another Ruby based library for monitoring services on a server. The configuration files are written with Ruby so it is easy to extend it to whatever you’d like. Using God we can easily make sure our workers are well-managed and that if Redis does go down it will be restarted. There is even an example script in Resque that will get you started with managing workers.
With God running on the server we can be sure that Redis and our workers will have high availability, or that we’ll know about it quickly if not.
Adding a highly available and robust background queue to the application took less than a day. Thanks to defunkt and mojombo for their excellent open source libraries!
Have an update or suggestion for this article? You can edit it here and send me a pull request.
Using HashiCorp Vault with LDAP
How to use HashiCorp Vault to setup an LDAP backed secret store with read-only access for users in groups and read-write access for specific users
Linux and Unix xargs command tutorial with examples
Tutorial on using xargs, a UNIX and Linux command for building and executing command lines from standard input. Examples of cutting by character, byte position, cutting based on delimiter and how to modify the output delimiter.
Copy a file in Go
How to copy a file in Go. The ioutil package does not offer a shorthand way of copying a file. Instead the os package should be used.