which is not threadsafe. It means you can’t use it while developing multithreaded Rails applications(especially for JRuby). But several monkeypatches can do the trick.<\/p>\n\n\n\n
Workaround<\/h2>\n\n\n\n
When we discovered this limitation almost a year ago, we had next options:<\/p>\n\n\n\n
- development without code reloading and restart server after every change<\/li>
- fix autoload threadsafe issue<\/li>
- create some workaround<\/li><\/ul>\n\n\n\n
As any RoR developer, I can’t imagine my life without automatic code
reloading, so server restart was not an option.
Fixing autoload is a nontrivial task. Corresponded bug was reported 4
years ago, and still wasn’t fixed.<\/p>\n\n\n\nSo, the only choice we had is to create some workaround. Solution
quickly came to mind – simply don’t use multithreading in development mode. We created fallback which made our application logic work in single-threaded mode for development, and in multithreaded mode for production.<\/p>\n\n\n\nFix of autoload<\/h2>\n\n\n\n
After 4 years of struggling, autoload was fixed in ruby-trunk<\/a>. It’s
scheduled to be delivered with Ruby 1.9.4 at the beginning of 2013. Luckily for us, patch was ported into JRuby 1.6.6, so we don’t have to wait till Christmas.<\/p>\n\n\n\nrvm install jruby-1.6.7\n<\/pre>\n\n\n\n
Then I disabled our fallback for development mode, and … found out that server crashed repeatedly across different places. Switching to production mode instantly fixed all exceptions. It looks like threadsafety of autoload was not the only issue.<\/p>\n\n\n\n
Digging into Rails guts<\/h2>\n\n\n\n
Every exception was different, but in most cases it was database
related issue. E.g. some objects in database were missing, some were not updated, deadlocking of SQL-queries and so on.<\/p>\n\n\n\nI started with grep-ing database query logs. Very soon quite weird behavior was observed: bunch of SQL-queries from one controller action were handled by different database connections. But they all should be issued by only one AR-connection!<\/p>\n\n\n\n
After this finding, the main suspect became infamous ActiveRecord::ConnectionAdapters::ConnectionPool. Again! In case you
don’t know, ConnectionPool is not threadsafe in Rails < 3.2.4, so we already had
monkeypatch<\/a> for it.
After adding some traces to ConnectionPool .checkin<\/em> and .checkout<\/em>
methods, we got next flow:<\/p>\n\n\n\n![\"\"](\"\/\/railsware.com\/blog\/wp-content\/uploads\/2012\/11\/code_reload_blog_pic1_v21.png\" "\\"code_reload_blog_pic1_v2\\"")
<\/figure>\n\n\n\nBut why was connection checked-in in the middle of job processing?
ConnectionPool<\/em> has 3 working strategies<\/a>, and
we used the simplest one. Connection is checked-out on demand, and when controller action is done, all connections used by
current thread are checked-in by clear_active_connections!<\/em> method.<\/p>\n\n\n\nBut in development mode something checks-in connections while they are actively used. Such behavior may not affect some applications, but with active usage of database transactions, which are bound to db-connections, disaster is guaranteed.<\/p>\n\n\n\n
There are just a few methods in ActiveRecord, which checkin\/checkout connections. Putting several additional traces I got next flow:
![\"\"](\"\/\/railsware.com\/blog\/wp-content\/uploads\/2012\/11\/code_reload_blog_pic2_v2.png\" "\\"code_reload_blog_pic2_v2\\"")
Connections were checked-in by clear_reloadable_connections! which is
called after processing of every request in development mode. Here is how it looks:<\/p>\n\n\n\ndef clear_reloadable_connections!\n #this part checkin all checked-out connections even used by other threads!\n @reserved_connections.each do |name, conn|\n checkin conn\n end\n @reserved_connections = {}\n #this part disconnects all conections which requires reloading.\n @connections.each do |conn|\n conn.disconnect! if conn.requires_reloading?\n end\n @connections.delete_if do |conn|\n conn.requires_reloading?\n end\nend\n<\/pre>\n\n\n\nIt can be split in two parts. First one is the root of our issue: it simply checks-in all active connections from all threads, no matter they are used or not. So why is it so lame? Git-blame shows, that this method was added 6 years ago<\/a>. Commit message says “Only reload connections in development mode that supports (and requires that) — in other words, only do it for SQLite”.
And here is how it looked 6 years ago:<\/p>\n\n\n\ndef clear_reloadable_connections!\n @@active_connections.each do |name, conn|\n conn.disconnect! if conn.supports_reloading?\n @@active_connections.delete(name)\n end\nend\n<\/pre>\n\n\n\n
A bit simplier, but it does two things:<\/p>\n\n\n\n
- disconnects SQLite connections(.support_reloading? method returns true only for SQLite connection)<\/li>
- checks-in ALL connections without any respect to other threads<\/li><\/ul>\n\n\n\n
Before this commit, instead of clear_reloadable_connections!<\/em> another
method was called: clear_active_connections!<\/em>, which was:<\/p>\n\n\n\ndef clear_active_connections!\n clear_cache!(@@active_connections) do |name, conn|\n conn.disconnect!\n end\nend\n\ndef clear_cache!(cache, thread_id = nil, &block)\n if cache\n if @@allow_concurrency\n thread_id ||= Thread.current.object_id \n #in multithreaded environment it will work only with current thread,\n #leaving others threads cache intact\n thread_cache, cache = cache, cache[thread_id]\n return unless cache\n end\n\n cache.each(&block) if block_given?\n cache.clear\n end\nensure\n if thread_cache && @@allow_concurrency\n thread_cache.delete(thread_id)\n end\nend\n<\/pre>\n\n\n\n
Unlike clear_reloadable_connections!<\/em>, this one respects other threads and clears connections used by current thread only. Looks like clear_reloadable_connections!<\/em> was de-threadsafed unintentionally and this should be fixed.<\/p>\n\n\n\n
Solution<\/h2>\n\n\n\n
If we don’t use SQLite why bother ourselves with fix for clear_reloadable_connections!<\/em>? Instead we can just disable it :). Here is a dumbest monkey-patch:<\/p>\n\n\n\n
class ActiveRecord::ConnectionAdapters::ConnectionPool\n def clear_reloadable_connections!;end \n # disable it, since it checks-in ALL connections from ALL threads after every request\n # in development mode\nend\n<\/pre>\n\n\n\n
We have like 5 minutes to celebrate the victory, but ActiveRecord::ConnectionTimeoutError, “could not obtain a database connection within 5 seconds” brings us back to earth. Looks like database connections leak from ConnectionPool.<\/p>\n\n\n\n
Further digging showed that WEBrick leaks connections when exception is raised (like RoutingError). Nobody noticed this because clear_reloadable_connections!<\/em> checks-out all connections after every request.<\/p>\n\n\n\n
ActiveRecord has special method – clear_stale_cached_connections!<\/a>.
It checks-in all leaked connections from dead threads, leaving active intact. Easiest place to call it is at the beginning of the checkout<\/em> method:<\/p>\n\n\n\nclass ActiveRecord::ConnectionAdapters::ConnectionPool\n def checkout\n clear_stale_cached_connections! if Rails.env.development? #hack for WEBrick\n ...\n end\nend\n<\/pre>\n\n\n\n
Voila! We have multithreaded application running in development mode with code reload!<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
To make code reloading work in multithreaded mode you need:<\/p>\n\n\n\n
- JRuby >= 1.6.6 or MRI >= 1.9.4, where autoload threadsafety is fixed<\/li>
- monkeypatch ConnectionPool to disable clear_reloadable_connections!<\/li>
- monkeypatch ConnectionPool to deal with connections leakage in WEBrick<\/li>
- update to Rails >= 3.2.4, where threadsafety issues<\/a> with ConnectionPool is fixed<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
Preface In Rails, automatic code reloading in development mode is based on autoload,which is not threadsafe. It means you can’t use it while developing multithreaded Rails applications(especially for JRuby). But several monkeypatches can do the trick. Workaround When we discovered this limitation almost a year ago, we had next options: development without code reloading and…<\/p>\n","protected":false},"author":43,"featured_media":9452,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[3],"tags":[],"coauthors":["Oleksandr Bondar"],"class_list":["post-3368","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t