The easiest and most straightforward solution was the creation of appropriate Rake tasks and scheduling their use using Cron. It is easy to implement, easy to test (just run a Rake task), and we used a perfect Whenever<\/a> gem for defining those Cron jobs with a clean Ruby syntax.<\/p>\n\n\n\n
As our projects progressed, the number of background tasks increased. Most of our Rake tasks had one great disadvantage: they were dependent on Rails itself, and so every time we launched those tasks new Rails instances were initialized. It was OK for us to have Rake jobs, each of which takes, let’s say, 5-7 seconds and 100mb of RAM to initialize Rails and then less than a few seconds for doing the job itself, because we had just 2-3 jobs per hour.
Once we got about 3-4 different tasks running, each 5-10 seconds for every job, Rails initialization time and its memory consumption became a problem. So we had to use find a thriftier solution.<\/p>\n\n\n\n
Custom Ruby daemons<\/h3>\n\n\n\n\n\n\n\n
Custom Ruby daemons (as were shown in Railscast<\/a> by Ryan Bates) seemed a good alternative to the Rake and Cron combination. We just moved scheduling from Cron to daemon classes and pulled task calls there from Rake tasks. So we ended up with a few daemons, each for a group of background tasks (it’s possible to put all the tasks into a single daemon, but this will complicate daemon logic dramatically). We also found it useful to use Cron for restarting deamons a few time per day as a way to fight Ruby memory bloats. But still we were a manly men: we struggled, we fought the bugs and found some workarounds, we were even ready to build statistics support and a job queue into our daemons.<\/p>\n\n\n\n Yet, soon our project required a different type of asynchronous tasks \u2014 nonscheduled ones. I mean nonrecurring tasks that have to be addressed ASAP. Usually this means slow controller actions that can be offloaded from Rails to speed up request handling, for example actions that require remote API calls for rendering a page part. Luckily, Github at about that time released their Resque tool inspired by Delayed Job (but more advanced) and based on Redis database for its job queue. Resque suited our needs very well. We picked it instead of a custom daemon and finally become happy!<\/p>\n\n\n\n There are two types of tasks settings. The first one is for non-reccuring tasks: workers are using Rails, and tasks are queued directly from the rails app.<\/p>\n\n\n\n The second type of tasks are recurring tasks that need to be scheduled. We use a special Rails controller for putting tasks outside of the main application. Restful HTTP client (launched as a Cron job) enqueues tasks on Resque. However, some people may consider this overengineering, since even a CURL-based client may be enough. There is an issue if you use PostgreSQL database and access it via ActiveRecord from Resque workers. It looks like Resque forks its workers and tries to reuse the same PostgreSQL connection, while PostgreSQL does not allow the use of the same connection from different processes. Of course, you can modify the Resque source code or tweak its workers but we found a much easier workaround: PgPool-II<\/a> for PostgreSQL connection pooling.<\/p>\n\n\n\n Guys at a Ratepoint have much higher loads and require more flexibility. So they implemented their own amqp-consuming daemons based on a Daemon-kit Gem, Nanite and RabbitMQ as a job queue. And I know that they are quite happy with this custom solution.<\/p>\n","protected":false},"excerpt":{"rendered":" Background Rake tasks When we start one of our projects there is always the need for some background tasks: updating online\/offline user counts, doing some remote API calls, fetching a fresh version of GeoIP database, etc.The easiest and most straightforward solution was the creation of appropriate Rake tasks and scheduling their use using Cron. It is easy to implement,…<\/p>\n","protected":false},"author":1,"featured_media":3764,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[3],"tags":[],"coauthors":["admin"],"class_list":["post-3","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"acf":[],"aioseo_notices":[],"categories_data":[{"name":"Engineering","link":"https:\/\/railsware.com\/blog?category=development"}],"post_thumbnails":"\/\/farm1.static.flickr.com\/51\/132869163_3a547ea876.jpg","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/3","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/comments?post=3"}],"version-history":[{"count":21,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/3\/revisions"}],"predecessor-version":[{"id":16643,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/3\/revisions\/16643"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media\/3764"}],"wp:attachment":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media?parent=3"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/categories?post=3"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/tags?post=3"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/coauthors?post=3"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
This solution was quite resource-friendly, but brought a lot of drawbacks:<\/p>\n\n\n\n\n
![\"Job](\"\/\/farm1.static.flickr.com\/51\/132869163_3a547ea876.jpg\")
<\/a><\/figure>\n\n\n\n
That was a turning point, because we decided to stop just being manly men and to start being a little smarter. We had to improve our daemons or pick a solution that didn’t require us to reinvent the wheel, for example, by taking existing background jobs solutions like Workling or Delayed Job.<\/p>\n\n\n\nResque rescued us from pain<\/h3>\n\n\n\n
So, why do we love resque so much?<\/h4>\n\n\n\n
\n
OK, I see. So, what’s bad about Resque?<\/h4>\n\n\n\n
\n
How do we use it?<\/h4>\n\n\n\n
Whenever gem is used for managing Cron schedules of reccuring tasks. [Not sure about the logic of this sentence. The use of Gem triggers Cron scheduling for recurring tasks? Can we clarify? Thanks.]<\/p>\n\n\n\nResque and PostgreSQL<\/h3>\n\n\n\n
What are the alternatives?<\/h3>\n\n\n\n