Toxic Elephant

Don't bury it in your back yard!

Benchmarking Dynamic Method Creation in Ruby

Posted by matijs 22/04/2011 at 09h30

Let’s look at dynamic method generation. I need it for GirFFI, and if you do any kind of metaprogramming, you probably need it too. It was already shown a long time ago that using string evaluation is preferable to using define_method with a block.

That is, if you care at all about speed.

How preferable? About a factor of 1.8 on Ruby 1.8.7:

                    user     system      total        real
regular:        0.270000   0.000000   0.270000 (  0.267164)
string eval:    0.270000   0.000000   0.270000 (  0.273170)
define_method:  0.490000   0.000000   0.490000 (  0.493258)

These numbers are relatively easy to explain. The string evaluation basically has the exact same result as regular method definition, because you actually do the same thing: You define a method using def. With define_method, you have a lot of overhead because a block is more than a bunch of code. It’s actually a closure, and Ruby has to set up the closure’s binding every time you call the method.

(Aside: There are of course other factors to consider. Using string evaluation gives you much more power to build exactly the method you want, while the fact that the blocks passed to define_method are closures allows you to do things now ordinary method can do.)

On Ruby 1.9.2, the results are quite similar, although the difference is now only a factor of 1.5:

                    user     system      total        real
regular:        0.140000   0.010000   0.150000 (  0.142998)
string eval:    0.140000   0.000000   0.140000 (  0.141439)
define_method:  0.210000   0.000000   0.210000 (  0.211936)

Too bad. It seems we’re stuck with string eval. Let’s look at JRuby:

                    user     system      total        real
regular:        0.324000   0.000000   0.324000 (  0.324000)
string eval:    0.208000   0.000000   0.208000 (  0.208000)
define_method:  0.690000   0.000000   0.690000 (  0.690000)

Wait, that can’t be right. Let’s run that again:

                    user     system      total        real
regular:        0.424000   0.000000   0.424000 (  0.424000)
string eval:    0.241000   0.000000   0.241000 (  0.241000)
define_method:  0.756000   0.000000   0.756000 (  0.756000)

Hm, it got a little slower, but the pattern is the same: The method defined using string eval is the fastest of the lot. What is going on here?

Quick, let’s try rubinius.

                    user     system      total        real
regular:        0.348022   0.000000   0.348022 (  0.183686)
string eval:    0.380024   0.004000   0.384024 (  0.196908)
define_method:  0.412026   0.008001   0.420027 (  0.215781)

Uh-huh. Again please.

                    user     system      total        real
regular:        0.376023   0.004000   0.380023 (  0.192683)
string eval:    0.356022   0.000000   0.356022 (  0.189258)
define_method:  0.164010   0.000000   0.164010 (  0.138058)

Huh? Like, maybe the benchmark is wrong?

Okay, let’s try bmbm instead of bm. For rubinius:

Rehearsal --------------------------------------------------
regular:         0.332020   0.004001   0.336021 (  0.172166)
string eval:     0.352022   0.004000   0.356022 (  0.185557)
define_method:   0.192012   0.000000   0.192012 (  0.152656)
----------------------------------------- total: 0.884055sec

                     user     system      total        real
regular:         0.052003   0.000000   0.052003 (  0.050464)
string eval:     0.052003   0.000000   0.052003 (  0.050471)
define_method:   0.076005   0.000000   0.076005 (  0.076912)

That looks more sane. Let’s try JRuby:

Rehearsal --------------------------------------------------
regular:         0.408000   0.000000   0.408000 (  0.408000)
string eval:     0.196000   0.000000   0.196000 (  0.196000)
define_method:   0.657000   0.000000   0.657000 (  0.657000)
----------------------------------------- total: 1.261000sec

                     user     system      total        real
regular:         0.095000   0.000000   0.095000 (  0.096000)
string eval:     0.109000   0.000000   0.109000 (  0.109000)
define_method:   0.416000   0.000000   0.416000 (  0.416000)

Much better. Notice that the difference between string eval and define_method is a stunning factor of four!

Now go back to rubinius. Did you notice how fast it was? That’s stunning. So what’s the difference there between define_method and string eval? Not so big. But the numbers are small so there may be some influence from the environment. Let’s look at how things scale: 10 million iterations:

Rehearsal --------------------------------------------------
regular:         1.240078   0.004000   1.244078 (  1.034290)
string eval:     1.076067   0.000000   1.076067 (  1.058813)
define_method:   0.848053   0.000000   0.848053 (  0.838424)
----------------------------------------- total: 3.168198sec

                     user     system      total        real
regular:         0.496031   0.000000   0.496031 (  0.496565)
string eval:     0.528033   0.000000   0.528033 (  0.500421)
define_method:   0.864054   0.000000   0.864054 (  0.863875)

That’s a factor of about 1.6, somewhere between MRI 1.8.7 and 1.9.2.

Conclusions

The basic conclusion holds: String evaluation leads to faster methods. How much of a difference it makes depends on which Ruby you’re using, and will probably depend on the particular method you’re creating.

Benchmarking is tricky: If you don’t watch out, you could draw the wrong conclusions.

Finally, Rubinius is fast. Really fast. Wow.

The Code

Benchmarks were generated with the following program:

<typo:code lang=”ruby”> require ‘benchmark’

class Foo def regular; end

eval “def stringeval; end”

define_method(:block) {} end

foo = Foo.new

n = 10000000

Benchmark.bmbm do |x| x.report(“regular: “) { n.times { foo.regular } } x.report(“string eval: “) { n.times { foo.stringeval } } x.report(“define_method:”) { n.times { foo.block } } end </typo:code>

Tags no comments no trackbacks

Comments

Comments are disabled