Representable 2.4: How Functional Programming Speeds Up Rendering And Parsing.

The great thing about being unemployed is you finally get to work on Open-Source features you always wanted to do but never had the time to.

Representable 2.4 internally is completely restructured, it has lost tons of code in favor for a simpler, faster, functional approach, we got rid of internal state, and it now allows to hook into parsing and rendering with your own logic without being restricted to predefined execution paths.

And, do I really have to mention that this results in +200% speedup for both rendering and parsing?

To cut it short: This version of Representable, which backs many other gems like Roar or Reform, feels great and I’m happy to throw it at you.

Here are the outstanding changes followed by a discussion how we could achieve this using functional techniques.

Speed

Representable 2.4 is about 3.2x faster than older versions. This is for both, rendering and parsing.

I have no idea what else to say about this right now.

Defaults

Yes, you may now define defaults for your representer.

class SongRepresenter < Representable::Decorator
  defaults render_nil: true

  property :title # does have render_nil: true

The defaults feature, mostly written by Sadjow Leão, also allows crunching default options using a block.

class SongRepresenter < Representable::Decorator
  defaults do |name|
    { as: name.to_s.camelize }
  end

  property :email_address # does have as: EmailAddress

A pretty handy feature that’s been due a long time. It is fully documented on the new, beautiful website.

Unified Lambda Options

The positional arguments for option lambdas I found incredibly annoying.

Every time I used :instance or :setter I had to look up their API (my own API!) since every option had its own.

For example, :instance exposes the following API.

instance: ->(fragment, [i], args) { }

Whereas :setter comes with another signature.

setter: ->(value, args) { }

In 2.4, every dynamic option receives a hash containing all the stakeholders you might need.

setter: ->(options) { options[:fragment] }
setter: ->(options) { options[:binding] }

This works extremely well with keyword arguments in Ruby 2.1 and above.

instance: ->(fragment:, index:, **) { puts "#{fragment} @ #{index}" }

Since I’m a good person, I deprecated all options but :render_filter and :parse_filter. Running your code with 2.4 will work but print tons of deprecation warnings.

Once your code is updated, you may switch of deprecation mode and speed up the execution.

Representable.deprecations = false

Note that this will be default behavior in 2.5.

Inject Behavior

In case you had to juggle a lot with Representable’s options to achieve what you want, I have good news. You can now inject custom behavior and replace parts or the entire pipeline.

For instance, I could make Representable use my own parsing logic for a specific property. This is a bit similar to :reader but gives you full control.

class SongRepresenter (input, options) do
    options[:represented].title = input.upcase
  end

  property :title, parse_pipeline: ->(*) { Upcase }

:parse_pipeline expects a callable object. Usually, that is an instance of Representable::Pipeline with many functions lined up, but it can also be a simple proc.

Here’s what happens.

song = OpenStruct.new

SongRepresenter.new(song).from_hash("title"=>"Seventh Sign")
song.title #=> "SEVENTH SIGN"

Without any additional logic, you implemented a simple parser for the title property.

Skip Execution

You can also setup your own pipeline using Representable’s functions, plus the ability to stop the pipeline when emitting a special symbol.

property :title, parse_pipeline: ->(*) do
  Representable::Pipeline[
    Representable::ReadFragment,
    SkipOnNil,
    Upper,
    Representable::SetValue
  ]
end

The implementation of the two custom functions is here.

SkipOnNil = ->(input, **) { input.nil? Pipeline::Stop : input }
Upper     = ->(input, **) { input.upcase }

By emiting Stop, the execution of the pipeline stops and nothing more happens. If the input fragment is not nil, it will be uppercased and set on the represented object.

Pipeline Mechanics

Every low-level function in a pipeline receives two arguments.

SkipOnNil = ->(input, options) { "new input" }

In pipelines, the second options argument is immutable, whereas the return value of the last function becomes the input of the next function.

This really functional approach was highly inspired by my friend Piotr Solnica and his “FP-infected mind”.

The same works with :render_pipeline as well, but rendering is boring.

How We Got It Faster.

Where we had tons of procedural code, ifs and elses, many hash lookups and different implementationsf for collections and scalar properties, we now have simple pipelines.

Remember, in Representable you always define document fragments using property.

class SongRepresenter < Representable::Decorator
  property :title
end

Now, let’s say we were to parse a document using this representer.

SongRepresenter.new(Song.new).from_hash("title" => "Havasu")

In older versions, Representable will now grab the "title" value, and then traverse the following pseudo-code.

if ! fragment
  if binding[:default]
    return binding[:default]
  end
else
  if binding[:skip_parse]
    return
  else
    if binding[:typed]
      if binding[:class]
        return ..
      elsif binding[:instance]
        return ..
      end
    else
      return fragment
    end
  end

Without knowing any details here, you can see that the flow is a deeply nested, procedural mess. Basically, every step represents one of the options you might be using every day, such as :default or :class.

Not only was it incredibly hard to follow Representable’s logic, as this procedural flow is spread across many files, it was also slow!

For every property being rendered or parsed, there had to be around 20 hash lookups on the binding, often followed by evaluations of the option. For example, :class could be unset, a class constant, or a dynamic lambda.

Projecting this to realistic representers with about 50-100 properties this quickly becomes thousands of hash lookups for only one object, just to find out something that has been defined at compile time.

Static Flow

Another problem was that the flow was static, making it really hard to add custom behavior.

if ! fragment
  if fragment == nil # injected, new behavior!
    fragment = []    # change nils to empty arrays.
  end

  if binding[:default]
    return binding[:default]

There was no clean way to inject additional behavior without abusing dynamic options or overriding Binding classes, which was the opposite of intuitive.

It was also a physical impossibility to stop the workflow at a particular point, since you couldn’t simply inject returns into the existing code. For example, say your :class lambda already handled the entire deserialization, you still had to fight with the options that are called after :class.

What I found myself doing a lot was adding more and more code to “versatile” options like :instance since the flow couldn’t be modified at runtime.

Pipelines

Sometimes you need to take a step back and ask yourself: “What am I actually trying to do?”. You must actively cut out all those nasty little edge-cases and special requirements your code also handles to see the big picture.

Strictly speaking, when parsing a document, Representable goes through its defined schema properties and invokes parsing for every binding. Each binding, and that’s the new insight, has a pipelined workflow.

• Grab fragment.
• Not there? Abort.
• Nil? Use Default if present. Abort.
• Skip parsing? Abort.
• If typed and :class, instantiate.
• If typed and :instance, instantiate.
• If typed, deserialize.
• Return.

Instead of oddly programming that in a procedural way, each binding now uses its very own pipeline. For decorators, the pipeline is computed at compile-time. This means depending on the options used for this property, a custom pipeline is built.

  property :artist,
    skip_parse: ->(fragment:, **) { fragment == "n/a" }
    class: Artist,

The above property will be roughly translated to the following pipeline (simplified).

Pipeline[
  ReadFragment,
  SkipParse,    # because of :skip_parse
  StopOnNotFound,
  CreateObject, # because of :class.
  Decorate,     # because of :class.
  Deserialize,  # because of :class.
  SetValue
]

This pipeline is intuitively understandable. Each element is a function, a simple Ruby proc defined for serializing and deserializing.

Again, the pipeline is created once at compile-time. This means all checks like if binding[:default] are done once when building the pipeline, reducing hash lookups on the binding to a negligible handful.

The fewer options a property uses, the less functions will be in the pipeline, shortening the execution time at run-time.

A tremendous speed-up of minimum 200% is the result.

Benchmarks

In a, what we call realistic benchmark, we wrote a representer with 50 properties, where each property is a nested representer with another 50 properties.

We then rendered 100 objects using that representer. Here are the benchmarks.

4.660000   0.000000   4.660000 (  4.667668) # 2.3
1.400000   0.010000   1.410000 (  1.410015) # 2.4

As you can see, Representable is now 3.32x faster.

Looking at the top of the profiler stack, it becomes very obvious why.

%self      calls  name
 13.92  6630522   Representable::Definition#[]
  5.28   255001   Representable::Binding#initialize
  4.81  1790109   Representable::Binding#[]
  2.90   515102   Uber::Options::Value#call
  2.36   510002   Representable::Definition#typed?

This is for 2.3, where an insane amount of time is wasted on hash lookups at run-time. Imagine, for every property the “pipeline” is computed at runtime (of course, the concept of pipeline didn’t exist, yet).

For 2.4, this is slightly different.

 %self     calls  name
  4.03   255001   Representable::Hash::Binding#write
  3.00   260101   Representable::Binding#exec_context
  2.77   255000   Representable::Binding#skipable_empty_value?
  2.44   255001   Representable::Binding#render_pipeline
  0.16     5100   Representable::Function::Decorate#call
  0.16    10201   Representable::Binding#[]

The highest call count is “only” 255K, which is a method we do have to call for each property. Other than that, expensive hash lookups and option evaluations are minimized drastically, requiring less than 1% computing time.

Declarative

I also got around to finally extract all declarative logic into a gem named – surprise! – Declarative. If you now think “Oh no, not another gem!” you should have a look at it.

In former versions, we’d use Representable in other gems just to get the DSL for property and collection, etc., without using Representable’s render/parse logic, which is what makes Representable.

This is now completely decoupled and reusable without any JSON, Hash or XML dependencies.

It also implements the inheritance between modules, representers and decorators in a simpler, more understandable way.

Debugging

To learn more about how pipelines work, you should make use of the Representable::Debug feature.

SongRepresenter.new(song).extend(Representable::Debug).from_hash(..)

The output is highly interesting!

Advertisement