This article describes how to refactor and improve readability of complex data processing with syntax similar to Haskell do-notation and Latest project I’ve been involved in deals with a lot of data processing. I’d love to have this refactored and decomposition looks like an obvious solution. Slice the big method into small ones and then chain them together. Seems straightforward.<\/p>\n\n\n\n Let’s say we have a public interface with a method like this:<\/p>\n\n\n\n Don’t try to guess what is going on here – method is completely made up. This looks fine, however notice the extra Without extra variables it gets even more clumsy due to Now the Could we somehow preserve the syntax from Iteration II and not constraint ourselves Hang tight, meet ChainFlow:<\/p>\n\n\n\n Now we have an emulation of Iteratioin II syntax beauty. Another variation provided by chain_flow is similar to Arel chains we’ve got used to:<\/p>\n\n\n\n Notice how Interested? Let’s see how it works.<\/p>\n\n\n\n Actually it’s a plain meta-programming.<\/p>\n\n\n\n Notice both The chain_flow code<\/a> itself is quite small.<\/p>\n\n\n\n The one who is into Haskell might notice that syntax provided by While the actual state is hidden, Of course nothing comes for free. And here the trade off is the speed. See funkify<\/a> library which provides Haskell-style partial application and composition for Ruby methods<\/em>. It relies heavily on Ruby lambdas though. After finishing this post, I came across awesome article<\/a> by Pat Shaugnessy. It’s good to know other people moving in the same direction. Here’s an attempt<\/a> to refactor Pat’s initial TL; DR; This article describes how to refactor and improve readability of complex data processing with syntax similar to Haskell do-notation and State monad. Motivation Latest project I’ve been involved in deals with a lot of data processing.Typical case was a method that receives a data chunk and chains it with a couple of Enumerator…<\/p>\n","protected":false},"author":34,"featured_media":9436,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[3],"tags":[],"coauthors":["Innokenty Mihailov"],"class_list":["post-6899","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"acf":[],"aioseo_notices":[],"categories_data":[{"name":"Engineering","link":"https:\/\/railsware.com\/blog?category=development"}],"post_thumbnails":"https:\/\/railsware.com\/blog\/wp-content\/themes\/railsware\/vendors\/images\/article-thumbnail-default.jpg","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/6899","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/users\/34"}],"replies":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/comments?post=6899"}],"version-history":[{"count":24,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/6899\/revisions"}],"predecessor-version":[{"id":14125,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/posts\/6899\/revisions\/14125"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media\/9436"}],"wp:attachment":[{"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/media?parent=6899"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/categories?post=6899"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/tags?post=6899"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/railsware.com\/blog\/wp-json\/wp\/v2\/coauthors?post=6899"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}State<\/code> monad.<\/p>\n\n\n\nMotivation<\/h2>\n\n\n\n
Typical case was a method that receives a data chunk and chains it with a couple of Enumerator<\/code> methods like map<\/code>,each_with_objects<\/code> etc.
Some of the chain blocks were simple, some of them not.
But overall method readability degraded significantly even with a two of them chained together.<\/p>\n\n\n\nRefactoring<\/h2>\n\n\n\n
module Work\n def process(data, parameter)\n data.group_by do |point|\n compute_point_key(point)\n end.each_with_object({}) do |(key, points), memo|\n memo[key] = compute_value(points, parameter)\n end\n end\nend\n<\/pre>\n\n\n\n
Let’s try to refactor it in several ways.<\/p>\n\n\n\nIteration I: extra variable<\/h3>\n\n\n\n
module Work\n def process(data, parameter)\n grouped_data = group_by_key(data)\n compute_values(grouped_data, parameter)\n end\n\n def group_by_key(points)\n points.group_by do |point|\n compute_point_key(point)\n end\n end\n\n def compute_values(points, parameter)\n points.each_with_object({}) do |(key, points), memo|\n memo[key] = compute_value(points, parameter)\n end\n end\nend\n<\/pre>\n\n\n\ngrouped_data<\/code> variable.
The more blocks you chain, the more extra variables you’ll have to deal with.<\/p>\n\n\n\nparameter<\/code> argument and reversed order of function invocations (from right to left).<\/p>\n\n\n\ndef process(data, parameter)\n compute_values (group_by_key data), parameter\nend\n<\/pre>\n\n\n\n
Iteration II: adding state<\/h3>\n\n\n\n
class Work < Struct(:data)\n def process(parameter)\n group_by_key\n compute_values(parameter)\n end\n\n def group_by_key\n data.group_by! do |point|\n compute_point_key(point)\n end\n end\n\n def compute_values(parameter)\n data.each_with_object!({}) do |(key, points), memo|\n memo[key] = compute_value(points, parameter)\n end\n end\nend\n<\/pre>\n\n\n\nprocess<\/code> looks much better (I would even say ideal). The trade-off is that group_by_key<\/code> and compute_values<\/code> are forced to use data<\/code> state variable.
But I don’t want to convert all my modules into classes every time I refactor the code.
Especially when my module is shared between other multiple classes.<\/p>\n\n\n\nChainFlow<\/h2>\n\n\n\n
to keep the state?<\/p>\n\n\n\nrequire 'chain_flow'\n\nmodule Work\n include ChainFlow\n\n def process(data, parameter)\n flow(data) do\n group_by_key\n compute_values(parameter)\n end\n end\n\n def group_by_key(points)\n points.group_by do |point|\n compute_point_key(point)\n end\n end\n\n def compute_values(points, parameter)\n points.each_with_object({}) do |(key, points), memo|\n memo[key] = compute_value(points, parameter)\n end\n end\nend\n<\/pre>\n\n\n\n
The order of the flow is not reversed, which is a great benefit for readability of our public interface.<\/p>\n\n\n\ndef process(data, parameter)\n chain { data }.group_by_key.compute_values(parameter).fetch\nend\n<\/pre>\n\n\n\ncompute_values<\/code> receives 2 arguments, but only the second (parameter<\/code>) is passed.
First argument is considered to be the state and being passed silently.<\/p>\n\n\n\nIt’s a kind of magic, magic, magic…<\/h2>\n\n\n\n
chain<\/code> and flow<\/code> methods provided by ChainFlow module are capturing the context using closures.
All the following processing functions calls (group_by_key<\/code> and compute_values<\/code>) are intercepted with method_missing<\/code> behind the scenes.
And re-executed one-by-one in a captured context pipelining the initial data through them along with other params.
By omitting the first paramater in our new syntax we emphasize the fact that state (hidden under the hood) is unimportant.
We are concentrating not on the temporary variables to pass it through, but rather on processing calls which form the pipeline.<\/p>\n\n\n\nFeel the Functional flavor<\/h2>\n\n\n\n
flow<\/code> resembles Haskell do-notation.
Haskell do-notation is a syntax sugar aimed to imporove the look of monadic functions composition.
The do-notation produces especially beautiful syntax in case State monad<\/a>. See this code snippet manipulating the Stack:<\/p>\n\n\n\nstackManip :: State Stack Int\nstackManip = do\n push 3\n pop\n pop\n<\/pre>\n\n\n\n
stackManip<\/code> composes 3 state-full computations and
as a result produces a computation which (when executed on an initial stack) will push 3 to the stack and then pop 2 times from it.
The idea behind chain_flow was to build similar syntax in Ruby.<\/p>\n\n\n\nPerformance<\/h2>\n\n\n\n
eval<\/code> and other meta programming tricks are quite expensive (as well as lambdas).
That said, if you’re dealing with reasonable amount of data and using chain_flow only for processing method calls time\/resources necessary for the chain_flow ‘magic’ is rather small in comparison with actual data processing.<\/p>\n\n\n\nP.S.<\/h2>\n\n\n\n
Good examples of monad implementations in Ruby is monadic<\/a>. do-notation<\/a> provides sort of a do-notation syntax with a couple of monad implementations as well.
See also the docile gem<\/a> – the very first example for Array modification looks great!<\/p>\n\n\n\nparse1<\/code> method from the article with chain_flow.<\/p>\n","protected":false},"excerpt":{"rendered":"