Benchmarking Ruby 3.2 with YJIT

Ruby 3.2.0 was released today! This is a perfect moment to check it out and see how YJIT behaves. I tested it out against Hanami test suite and then ran a bunch of benchmarks that we have in dry-rb projects and the results are absolutely fascinating!

Hanami test suite

This is a really representative test suite because it uses Hanami applications fully configured and tests a lot of the framework top-level features. During setup, it creates entire application directory structures with real files defining real classes. This means that a lot is going on while you’re watching the dots on the screen.

I checked it against Ruby 3.1 and then Ruby 3.2 with and without YJIT enabled. Here are the results.

Ruby 3.1

Finished in 18.81 seconds (files took 0.34758 seconds to load)
385 examples, 0 failures, 1 pending

Ruby 3.2 without YJIT

Finished in 17.71 seconds (files took 0.33394 seconds to load)
385 examples, 0 failures, 1 pending

Ruby 3.2 with YJIT

Finished in 21.23 seconds (files took 0.83591 seconds to load)
385 examples, 0 failures, 1 pending

As you can see Ruby 3.2 is slightly faster in no-YJIT mode vs Ruby 3.1. One thing that surprised me was that running this test suite on Ruby 3.1 is sometimes much slower and goes up to ~30s. I initially thought that there was a big difference between 3.1 and 3.2 but it’s clearly not the case.

Ruby 3.2 with YJIT enabled is a bit slower and also notice that loading files is actually way slower, which was an interesting finding.

It was pointed out to me that enabling YJIT in test suites is in general not recommended:

@solnic @joeldrapper By default, YJIT optimises a method on the 30th time you call it. You can change that with a command line param. But if your methods aren't run very many times, YJIT is often not a help. So we don't recommend it for unit tests, as a rule.

Noah Gibbs

Benchmarking dry-validation

Much more interesting results come from running dry-validation’s benchmarks. We compare validating a simple “model” with 2 attributes in two scenarios: valid and invalid.

Here’s the first one testing the valid case:

# frozen_string_literal: true
 
 require "benchmark/ips"
 require "active_model"
 
 require "i18n"
 require "dry-validation"
 
 require_relative "active_record_setup"
 
 module AM
   class User
     include ActiveModel::Validations
 
     attr_reader :email, :age
 
     validates :email, :age, presence: true
     validates :age, presence: true, numericality: {greater_than: 18}
 
     def initialize(attrs)
       @email, @age = attrs.values_at("email", "age")
     end
   end
 end
 
 contract = Dry::Validation::Contract.build {
   config.messages.backend = :i18n
 
   params do
     required(:email).filled(:string)
     required(:age).filled(:integer)
   end
 
   rule(:age) do
     failure("must be greater than 18") if values[:age] <= 18
   end
 }
 
 params = {"email" => "jane@doe.org", "age" => "19"}
 
 puts contract.(params).inspect
 puts AM::User.new(params).validate
 
 Benchmark.ips do |x|
   x.report("ActiveModel::Validations") do
     user = AM::User.new(params)
     user.validate
     user.errors.messages
   end
 
   x.report("ActiveRecord") do
     user = AR::User.new(params)
     user.validate
     user.errors.messages
   end
 
   x.report("dry-validation") do
     contract.(params).errors
   end
 
   x.compare!
 end

Ruby 3.1

Warming up --------------------------------------
ActiveModel::Validations
                         8.342k i/100ms
        ActiveRecord     2.262k i/100ms
      dry-validation     2.719k i/100ms
Calculating -------------------------------------
ActiveModel::Validations
                         83.106k (± 2.2%) i/s -    417.100k in   5.021417s
        ActiveRecord     21.414k (± 3.6%) i/s -    108.576k in   5.077040s
      dry-validation     27.500k (± 5.1%) i/s -    138.669k in   5.056987s

Comparison:
ActiveModel::Validations:    83106.0 i/s
      dry-validation:    27500.1 i/s - 3.02x  (± 0.00) slower
        ActiveRecord:    21414.3 i/s - 3.88x  (± 0.00) slower

Ruby 3.2 without YJIT

Warming up --------------------------------------
ActiveModel::Validations
                         7.324k i/100ms
        ActiveRecord     2.199k i/100ms
      dry-validation     2.946k i/100ms
Calculating -------------------------------------
ActiveModel::Validations
                         73.929k (± 5.4%) i/s -    373.524k in   5.068697s
        ActiveRecord     22.085k (± 3.9%) i/s -    112.149k in   5.086915s
      dry-validation     29.497k (± 2.5%) i/s -    150.246k in   5.097062s

Comparison:
ActiveModel::Validations:    73929.5 i/s
      dry-validation:    29497.2 i/s - 2.51x  (± 0.00) slower
        ActiveRecord:    22085.3 i/s - 3.35x  (± 0.00) slower

Ruby 3.2 with YJIT

Warming up --------------------------------------
ActiveModel::Validations
                        10.136k i/100ms
        ActiveRecord     5.064k i/100ms
      dry-validation     5.983k i/100ms
Calculating -------------------------------------
ActiveModel::Validations
                        100.336k (± 2.8%) i/s -    506.800k in   5.055425s
        ActiveRecord     50.015k (± 2.4%) i/s -    253.200k in   5.065686s
      dry-validation     60.162k (± 1.5%) i/s -    305.133k in   5.072970s

Comparison:
ActiveModel::Validations:   100335.8 i/s
      dry-validation:    60161.9 i/s - 1.67x  (± 0.00) slower
        ActiveRecord:    50015.0 i/s - 2.01x  (± 0.00) slower

There are three really interesting things here:

  1. Plain object initialization with instance variable assignment is faster under Ruby 3.2
  2. Enabling YJIT gives a significant performance boost as we go from 150.246k in 5.097062s to 305.133k in 5.072970s in case of dry-validation
  3. The difference between dry-validation and ActiveRecord vs ActiveModel is significantly smaller when YJIT is enabled

Benchmarking dry-struct

Now this one is just nuts. dry-struct is a library that I created with the intention to replace Virtus. It’s a simple DSL for defining typed structs, just like Virtus; however, it’s meant to be used for defining pure data structures, that’s why it does not provide attribute writers and there are no methods to mutate structs.

One of the goals of dry-struct was to also achieve great performance, because this is what Virtus struggled with. We have a basic benchmark that tests dry-struct against 3 other similar libraries: Virtus, attrio and fast_attributes.

Here’s how it’s implemented:

# frozen_string_literal: true
 
 require "dry/struct"
 require "virtus"
 require "fast_attributes"
 require "attrio"
 require "ostruct"
 
 require "benchmark/ips"
 
 class VirtusUser
   include Virtus.model
 
   attribute :name, String
   attribute :age, Integer
 end
 
 class FastUser
   extend FastAttributes
 
   define_attributes initialize: true, attributes: true do
     attribute :name, String
     attribute :age,  Integer
   end
 end
 
 class AttrioUser
   include Attrio
 
   define_attributes do
     attr :name, String
     attr :age, Integer
   end
 
   def initialize(attributes = {})
     self.attributes = attributes
   end
 
   def attributes=(attributes = {})
     attributes.each do |attr, value|
       send("#{attr}=", value) if respond_to?("#{attr}=")
     end
   end
 end
 
 class DryStructUser < Dry::Struct
   attributes(name: "strict.string", age: "params.integer")
 end
 
 puts DryStructUser.new(name: "Jane", age: "21").inspect
 
 Benchmark.ips do |x|
   x.report("virtus") { VirtusUser.new(name: "Jane", age: "21") }
   x.report("fast_attributes") { FastUser.new(name: "Jane", age: "21") }
   x.report("attrio") { AttrioUser.new(name: "Jane", age: "21") }
   x.report("dry-struct") { DryStructUser.new(name: "Jane", age: "21") }
 
   x.compare!
 end

Up until now, fast_attributes was…you guessed it! Fast. The fastest, to be precise.

This is no longer the case though, all you need to do is to enable YJIT! Check out the results:

Ruby 3.1

Warming up --------------------------------------
              virtus     5.920k i/100ms
     fast_attributes    76.795k i/100ms
              attrio    13.118k i/100ms
          dry-struct    42.729k i/100ms
Calculating -------------------------------------
              virtus     66.051k (± 2.6%) i/s -    331.520k in   5.022813s
     fast_attributes    829.331k (± 3.9%) i/s -      4.147M in   5.009166s
              attrio    141.829k (± 2.4%) i/s -    721.490k in   5.090232s
          dry-struct    439.638k (± 1.0%) i/s -      2.222M in   5.054435s

Comparison:
     fast_attributes:   829330.9 i/s
          dry-struct:   439637.7 i/s - 1.89x  (± 0.00) slower
              attrio:   141829.1 i/s - 5.85x  (± 0.00) slower
              virtus:    66050.6 i/s - 12.56x  (± 0.00) slower

Ruby 3.2 without YJIT

Warming up --------------------------------------
              virtus     7.001k i/100ms
     fast_attributes    85.866k i/100ms
              attrio    14.706k i/100ms
          dry-struct    45.756k i/100ms
Calculating -------------------------------------
              virtus     68.503k (± 2.9%) i/s -    343.049k in   5.012351s
     fast_attributes     845.828k (± 2.4%) i/s -      4.293M in   5.079085s
              attrio    141.372k (± 3.7%) i/s -    720.594k in   5.104778s
          dry-struct    447.745k (± 1.5%) i/s -      2.242M in   5.008645s

Comparison:
     fast_attributes:   845827.5 i/s
          dry-struct:   447745.3 i/s - 1.89x  (± 0.00) slower
              attrio:   141371.8 i/s - 5.98x  (± 0.00) slower
              virtus:    68502.8 i/s - 12.35x  (± 0.00) slower

Ruby 3.2 with YJIT 😱 ⚡

Warming up --------------------------------------
              virtus     8.977k i/100ms
     fast_attributes    96.806k i/100ms
              attrio    20.130k i/100ms
          dry-struct   111.189k i/100ms
Calculating -------------------------------------
              virtus     88.392k (± 1.7%) i/s -    448.850k in   5.079484s
     fast_attributes    973.297k (± 1.3%) i/s -      4.937M in   5.073376s
              attrio    197.840k (± 3.7%) i/s -      1.006M in   5.095283s
          dry-struct      1.103M (± 1.6%) i/s -      5.559M in   5.042373s

Comparison:
          dry-struct:  1102846.4 i/s
     fast_attributes:   973296.6 i/s - 1.13x  (± 0.00) slower
              attrio:   197840.1 i/s - 5.57x  (± 0.00) slower
              virtus:    88391.5 i/s - 12.48x  (± 0.00) slower

We went from 2.242M in 5.008645s to 5.559M in 5.042373s and fast_attributes is no longer faster than dry-struct. Amazing!

This really blew my mind. I reached out to Nikita asking how on earth is this possible, to which he replied:

I just optimized it

🤣 So now we know, he just optimized it 🙂

On a serious note, this was a series of many optimizations spanning many releases of dry-struct. It’s just that the full potential of these improvements can be observed now with YJIT enabled. I’d love to dive deeper into what happened there, maybe I’ll manage to write about this in detail in another article. Kudos to Nikita for his outstanding work 👏

Ruby 3.2 can be really fast!

Benchmarking is tricky but if you compare the same benchmark using different versions of the same programming language, then chances are the results are really meaningful.

I’ve very glad to see that our test suites are passing and benchmarks show such great results. This is a fantastic release, I’m very happy to see this. Now I can’t wait to upgrade my app to Ruby 3.2 with YJIT enabled and see how it works.

Huge thanks to everybody who worked on Ruby 3.2! ❤️

Enjoy the rest of the holidays!

Subscribe to solnic.dev

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe