TL; DR;
This article describes how to refactor and improve readability of complex data processing with syntax similar to Haskell do-notation and State monad.
Motivation
Latest project I’ve been involved in deals with a lot of data processing.
Typical case was a method that receives a data chunk and chains it with a couple of Enumerator methods like map,each_with_objects etc.
Some of the chain blocks were simple, some of them not.
But overall method readability degraded significantly even with a two of them chained together.
I’d love to have this refactored and decomposition looks like an obvious solution. Slice the big method into small ones and then chain them together. Seems straightforward.
Refactoring
Let’s say we have a public interface with a method like this:
module Work
def process(data, parameter)
data.group_by do |point|
compute_point_key(point)
end.each_with_object({}) do |(key, points), memo|
memo[key] = compute_value(points, parameter)
end
end
end
Don’t try to guess what is going on here – method is completely made up.
Let’s try to refactor it in several ways.
Iteration I: extra variable
module Work
def process(data, parameter)
grouped_data = group_by_key(data)
compute_values(grouped_data, parameter)
end
def group_by_key(points)
points.group_by do |point|
compute_point_key(point)
end
end
def compute_values(points, parameter)
points.each_with_object({}) do |(key, points), memo|
memo[key] = compute_value(points, parameter)
end
end
end
This looks fine, however notice the extra grouped_data variable.
The more blocks you chain, the more extra variables you’ll have to deal with.
Without extra variables it gets even more clumsy due to parameter argument and reversed order of function invocations (from right to left).
def process(data, parameter) compute_values (group_by_key data), parameter end
Iteration II: adding state
class Work < Struct(:data)
def process(parameter)
group_by_key
compute_values(parameter)
end
def group_by_key
data.group_by! do |point|
compute_point_key(point)
end
end
def compute_values(parameter)
data.each_with_object!({}) do |(key, points), memo|
memo[key] = compute_value(points, parameter)
end
end
end
Now the process looks much better (I would even say ideal). The trade-off is that group_by_key and compute_values are forced to use data state variable.
But I don’t want to convert all my modules into classes every time I refactor the code.
Especially when my module is shared between other multiple classes.
ChainFlow
Could we somehow preserve the syntax from Iteration II and not constraint ourselves
to keep the state?
Hang tight, meet ChainFlow:
require 'chain_flow'
module Work
include ChainFlow
def process(data, parameter)
flow(data) do
group_by_key
compute_values(parameter)
end
end
def group_by_key(points)
points.group_by do |point|
compute_point_key(point)
end
end
def compute_values(points, parameter)
points.each_with_object({}) do |(key, points), memo|
memo[key] = compute_value(points, parameter)
end
end
end
Now we have an emulation of Iteratioin II syntax beauty.
The order of the flow is not reversed, which is a great benefit for readability of our public interface.
Another variation provided by chain_flow is similar to Arel chains we’ve got used to:
def process(data, parameter)
chain { data }.group_by_key.compute_values(parameter).fetch
end
Notice how compute_values receives 2 arguments, but only the second (parameter) is passed.
First argument is considered to be the state and being passed silently.
Interested? Let’s see how it works.
It’s a kind of magic, magic, magic…
Actually it’s a plain meta-programming.
Notice both chain and flow methods provided by ChainFlow module are capturing the context using closures.
All the following processing functions calls (group_by_key and compute_values) are intercepted with method_missing behind the scenes.
And re-executed one-by-one in a captured context pipelining the initial data through them along with other params.
By omitting the first paramater in our new syntax we emphasize the fact that state (hidden under the hood) is unimportant.
We are concentrating not on the temporary variables to pass it through, but rather on processing calls which form the pipeline.
The chain_flow code itself is quite small.
Feel the Functional flavor
The one who is into Haskell might notice that syntax provided by flow resembles Haskell do-notation.
Haskell do-notation is a syntax sugar aimed to imporove the look of monadic functions composition.
The do-notation produces especially beautiful syntax in case State monad. See this code snippet manipulating the Stack:
stackManip :: State Stack Int stackManip = do push 3 pop pop
While the actual state is hidden, stackManip composes 3 state-full computations and
as a result produces a computation which (when executed on an initial stack) will push 3 to the stack and then pop 2 times from it.
The idea behind chain_flow was to build similar syntax in Ruby.
Performance
Of course nothing comes for free. And here the trade off is the speed. eval and other meta programming tricks are quite expensive (as well as lambdas).
That said, if you’re dealing with reasonable amount of data and using chain_flow only for processing method calls time/resources necessary for the chain_flow ‘magic’ is rather small in comparison with actual data processing.
P.S.
See funkify library which provides Haskell-style partial application and composition for Ruby methods. It relies heavily on Ruby lambdas though.
Good examples of monad implementations in Ruby is monadic. do-notation provides sort of a do-notation syntax with a couple of monad implementations as well.
See also the docile gem – the very first example for Array modification looks great!
After finishing this post, I came across awesome article by Pat Shaugnessy. It’s good to know other people moving in the same direction. Here’s an attempt to refactor Pat’s initial parse1 method from the article with chain_flow.