When we started using Resque two years ago we were impressed by two things: its power out-of-the-box and its opportunities for scalability. Over the past two years, we\u2019ve explored Resque internals and plugins. We\u2019d like to share what we\u2019ve learned from our practical experience using Resque during different phases of the web application life cycle.<\/p>\n\n\n\n
The first challenge we encountered was deciding how to organize development for non-Ruby developers and test environments. Our aims were to keep most of the team free from knowledge about Resque and Workers and to avoid stubs in tests. This idea resulted in the inline mode for Resque that solved the problem perfectly.<\/p>\n\n\n\n
Since there are many articles covering how to deploy Resque, we\u2019ll focus on issues that haven\u2019t been thoroughly described before.<\/p>\n\n\n\n
After two weeks in production it was clear to us that there was an issue with Resque and ActiveRecord. In some cases you may enqueue a Resque job while inside a database transaction, but Redis commands are independent from database transactions. Sometimes a worker starts processing a job before the transaction that creates the specific job commits. After a few ugly solutions that forced us to restructure the code, we discovered what we needed in the\u00a0ar_after_transaction<\/a>\u00a0gem. This\u00a0Resque FAQ<\/a>\u00a0details how to make a Resque job wait for an ActiveRecord transaction commit, so that it can see all the changes made by that transaction. Of course, if you ensure database transactions are committed prior to enqueuing jobs, you can structure your application in any manner you desire.<\/p>\n\n\n\n External HTTP calls are a common bottleneck for web requests and need to be moved to the background because of unpredictable response time and downtime for these APIs. You may find the\u00a0resque-retry<\/a>\u00a0plugin (and\u00a0resque-scheduler<\/a>\u00a0plugin as a dependency) useful, allowing you to retry exceptions in workers with a customizable delay.<\/p>\n\n\n\n Here are some common HTTP errors in the \u201cjust try again to fix\u201d category:<\/p>\n\n\n\n N.B. <\/strong>Errno codes are platform-specific, make sure you understand how portable your code needs to be.<\/p>\n\n\n\n If you are using an external SMTP server to send email, you will need to move the email delivery to the background \u2014 with Resque\u2019s help, of course. There are number of solutions available, such as\u00a0ar_mailer<\/a>. We decided to use\u00a0resque_mailer<\/a>. We encountered an initial problem with\u00a0 Use this class as the base class for all your mailers and retry configuration will be shared among them.<\/p>\n\n\n\n As the number of users and load grew, we decided it was a good idea to include other plugins like\u00a0resque-loner<\/a>\u00a0(to track job uniqueness) and\u00a0resque-cleaner<\/a>\u00a0(to cleanup failed jobs). This required fixing and improving these libraries:<\/p>\n\n\n\n The original Resque design does not allow you to receive something back after the worker completes. This may be beneficial for most use cases. However, for our use case (payment checkout through an outer Authorization gateway) it was important to know whether the worker was in progress or not, and if not \u2013 whether it was successful or not.<\/p>\n\n\n\n Many Resque plugins introduce a job identifier based on arguments passed to this job, but there is no standardization regarding how it should be done. Here are two examples:<\/p>\n\n\n\n resque-retry:<\/p>\n\n\n\n resque-loner:<\/p>\n\n\n\n In order to synchronize a job identifier across plugins, we implemented our own\u00a0interface for jobs with completion status<\/a>. This can be helpful for people that need something like this. However, don\u2019t confuse this with execution status in\u00a0resque-status<\/a>.<\/p>\n\n\n\n Last but not least, thank you to these people responsible for supporting our patches:<\/p>\n\n\n\nSecond step in production: Outer HTTP APIs with Resque<\/h3>\n\n\n\n
@retry_exceptions = [\n Timeout::Error,\n Errno::ECONNREFUSED,\n Errno::ECONNRESET,\n # errors from your favorite\n # Net::HTTP wrapping library goes here\n]<\/pre>\n<\/div>\n<\/div>\n\n\n\n
Third step in production: Email sending<\/h3>\n\n\n\n
Net::SMTPServerBusy<\/code>\u00a0and\u00a0Timeout::Error<\/code>\u00a0exceptions that appeared randomly while sending email. We found\u00a0resque-retry<\/a>\u00a0is also useful here. In the case of resque_mailer we wanted to have shared configuration for resque-retry for every Mailer class. We found that this was not easy because historically Resque is configured through instance variables in a class that are not inherited. We needed a base class that could share all instance variables across any child class:<\/p>\n\n\n\nclass AsyncApplicationMailer < ActionMailer::Base\n\n include Resque::Mailer\n\n extend Resque::Plugins::Retry\n\n # All Notifiers inherited from this class\n # require same resque-retry options.\n # Resque workers are classes but not instances of classes.\n # That is why resque retry require class variables that is not inherited\n # In order to setup same resque-retry class variables\n # for every inherited class we need this hack.\n def self.inherited(host)\n super(host)\n host.class_eval do\n @retry_exceptions = [Net::SMTPServerBusy, Timeout::Error, Resque::DirtyExit]\n @retry_limit = 3\n @retry_delay = 60 #seconds\n end\n end\n\nend<\/pre>\n<\/div>\n<\/div>\n\n\n\n
Play Minesweeper: Bug fixing along the way<\/h3>\n\n\n\n
ActionMailer::Base.perform_deliveries<\/code>\u00a0configuration option<\/a><\/li>Business requirements increase: Returning results from jobs<\/h3>\n\n\n\n
# @abstract You may override to implement a custom identifier,\n# you should consider doing this if your job arguments\n# are many\/long or may not cleanly cleanly to strings.\n#\n# Builds an identifier using the job arguments. This identifier\n# is used as part of the redis key.\ndef identifier(*args)<\/pre>\n<\/div>\n<\/div>\n\n\n\n
#\n# Payload is what Resque stored for this job along with the job's class name.\n# On a Resque with no plugins installed, this is a hash containing :class and :args\n#\ndef redis_key(payload)<\/pre>\n<\/div>\n<\/div>\n\n\n\n
Contribution to open source<\/h3>\n\n\n\n