Preface
In Rails, automatic code reloading in development mode is based on autoload, which is not threadsafe. It means you can’t use it while developing multithreaded Rails applications(especially for JRuby). But several monkeypatches can do the trick.
Workaround
When we discovered this limitation almost a year ago, we had next options:- development without code reloading and restart server after every change
- fix autoload threadsafe issue
- create some workaround
Fix of autoload
After 4 years of struggling, autoload was fixed in ruby-trunk. It’s scheduled to be delivered with Ruby 1.9.4 at the beginning of 2013. Luckily for us, patch was ported into JRuby 1.6.6, so we don’t have to wait till Christmas.rvm install jruby-1.6.7
Then I disabled our fallback for development mode, and … found out that server crashed repeatedly across different places. Switching to production mode instantly fixed all exceptions. It looks like threadsafety of autoload was not the only issue.Digging into Rails guts
Every exception was different, but in most cases it was database related issue. E.g. some objects in database were missing, some were not updated, deadlocking of SQL-queries and so on.I started with grep-ing database query logs. Very soon quite weird behavior was observed: bunch of SQL-queries from one controller action were handled by different database connections. But they all should be issued by only one AR-connection!After this finding, the main suspect became infamous ActiveRecord::ConnectionAdapters::ConnectionPool. Again! In case you don’t know, ConnectionPool is not threadsafe in Rails < 3.2.4, so we already had monkeypatch for it. After adding some traces to ConnectionPool .checkin and .checkout methods, we got next flow:
But why was connection checked-in in the middle of job processing? ConnectionPool has 3 working strategies, and we used the simplest one. Connection is checked-out on demand, and when controller action is done, all connections used by current thread are checked-in by clear_active_connections! method.But in development mode something checks-in connections while they are actively used. Such behavior may not affect some applications, but with active usage of database transactions, which are bound to db-connections, disaster is guaranteed.There are just a few methods in ActiveRecord, which checkin/checkout connections. Putting several additional traces I got next flow: 
Connections were checked-in by clear_reloadable_connections! which is called after processing of every request in development mode. Here is how it looks:def clear_reloadable_connections!
#this part checkin all checked-out connections even used by other threads!
@reserved_connections.each do |name, conn|
checkin conn
end
@reserved_connections = {}
#this part disconnects all conections which requires reloading.
@connections.each do |conn|
conn.disconnect! if conn.requires_reloading?
end
@connections.delete_if do |conn|
conn.requires_reloading?
end
end
It can be split in two parts. First one is the root of our issue: it simply checks-in all active connections from all threads, no matter they are used or not. So why is it so lame? Git-blame shows, that this method was added 6 years ago. Commit message says “Only reload connections in development mode that supports (and requires that) — in other words, only do it for SQLite”. And here is how it looked 6 years ago:def clear_reloadable_connections!
@@active_connections.each do |name, conn|
conn.disconnect! if conn.supports_reloading?
@@active_connections.delete(name)
end
end
A bit simplier, but it does two things:- disconnects SQLite connections(.support_reloading? method returns true only for SQLite connection)
- checks-in ALL connections without any respect to other threads
def clear_active_connections!
clear_cache!(@@active_connections) do |name, conn|
conn.disconnect!
end
end
def clear_cache!(cache, thread_id = nil, &block)
if cache
if @@allow_concurrency
thread_id ||= Thread.current.object_id
#in multithreaded environment it will work only with current thread,
#leaving others threads cache intact
thread_cache, cache = cache, cache[thread_id]
return unless cache
end
cache.each(&block) if block_given?
cache.clear
end
ensure
if thread_cache && @@allow_concurrency
thread_cache.delete(thread_id)
end
end
Unlike clear_reloadable_connections!, this one respects other threads and clears connections used by current thread only. Looks like clear_reloadable_connections! was de-threadsafed unintentionally and this should be fixed.Solution
If we don’t use SQLite why bother ourselves with fix for clear_reloadable_connections!? Instead we can just disable it :). Here is a dumbest monkey-patch:class ActiveRecord::ConnectionAdapters::ConnectionPool
def clear_reloadable_connections!;end
# disable it, since it checks-in ALL connections from ALL threads after every request
# in development mode
end
We have like 5 minutes to celebrate the victory, but ActiveRecord::ConnectionTimeoutError, “could not obtain a database connection within 5 seconds” brings us back to earth. Looks like database connections leak from ConnectionPool.Further digging showed that WEBrick leaks connections when exception is raised (like RoutingError). Nobody noticed this because clear_reloadable_connections! checks-out all connections after every request.ActiveRecord has special method – clear_stale_cached_connections!. It checks-in all leaked connections from dead threads, leaving active intact. Easiest place to call it is at the beginning of the checkout method:class ActiveRecord::ConnectionAdapters::ConnectionPool
def checkout
clear_stale_cached_connections! if Rails.env.development? #hack for WEBrick
...
end
end
Voila! We have multithreaded application running in development mode with code reload!Summary
To make code reloading work in multithreaded mode you need:- JRuby >= 1.6.6 or MRI >= 1.9.4, where autoload threadsafety is fixed
- monkeypatch ConnectionPool to disable clear_reloadable_connections!
- monkeypatch ConnectionPool to deal with connections leakage in WEBrick
- update to Rails >= 3.2.4, where threadsafety issues with ConnectionPool is fixed