Benchmarking Ruby 3.2 with YJIT
Ruby 3.2.0 was released today! This is a perfect moment to check it out and see how YJIT behaves. I tested it out against Hanami test suite and then ran a bunch of benchmarks that we have in dry-rb projects and the results are absolutely fascinating!
Hanami test suite
This is a really representative test suite because it uses Hanami applications fully configured and tests a lot of the framework top-level features. During setup, it creates entire application directory structures with real files defining real classes. This means that a lot is going on while you’re watching the dots on the screen.
I checked it against Ruby 3.1 and then Ruby 3.2 with and without YJIT enabled. Here are the results.
Ruby 3.1
Finished in 18.81 seconds (files took 0.34758 seconds to load)
385 examples, 0 failures, 1 pendingRuby 3.2 without YJIT
Finished in 17.71 seconds (files took 0.33394 seconds to load)
385 examples, 0 failures, 1 pendingRuby 3.2 with YJIT
Finished in 21.23 seconds (files took 0.83591 seconds to load)
385 examples, 0 failures, 1 pendingAs you can see Ruby 3.2 is slightly faster in no-YJIT mode vs Ruby 3.1. One thing that surprised me was that running this test suite on Ruby 3.1 is sometimes much slower and goes up to ~30s. I initially thought that there was a big difference between 3.1 and 3.2 but it’s clearly not the case.
Ruby 3.2 with YJIT enabled is a bit slower and also notice that loading files is actually way slower, which was an interesting finding.
It was pointed out to me that enabling YJIT in test suites is in general not recommended:
@solnic @joeldrapper By default, YJIT optimises a method on the 30th time you call it. You can change that with a command line param. But if your methods aren't run very many times, YJIT is often not a help. So we don't recommend it for unit tests, as a rule.
Noah Gibbs
Benchmarking dry-validation
Much more interesting results come from running dry-validation’s benchmarks. We compare validating a simple “model” with 2 attributes in two scenarios: valid and invalid.
Here’s the first one testing the valid case:
# frozen_string_literal: true
require "benchmark/ips"
require "active_model"
require "i18n"
require "dry-validation"
require_relative "active_record_setup"
module AM
class User
include ActiveModel::Validations
attr_reader :email, :age
validates :email, :age, presence: true
validates :age, presence: true, numericality: {greater_than: 18}
def initialize(attrs)
@email, @age = attrs.values_at("email", "age")
end
end
end
contract = Dry::Validation::Contract.build {
config.messages.backend = :i18n
params do
required(:email).filled(:string)
required(:age).filled(:integer)
end
rule(:age) do
failure("must be greater than 18") if values[:age] <= 18
end
}
params = {"email" => "jane@doe.org", "age" => "19"}
puts contract.(params).inspect
puts AM::User.new(params).validate
Benchmark.ips do |x|
x.report("ActiveModel::Validations") do
user = AM::User.new(params)
user.validate
user.errors.messages
end
x.report("ActiveRecord") do
user = AR::User.new(params)
user.validate
user.errors.messages
end
x.report("dry-validation") do
contract.(params).errors
end
x.compare!
end
Ruby 3.1
Warming up --------------------------------------
ActiveModel::Validations
8.342k i/100ms
ActiveRecord 2.262k i/100ms
dry-validation 2.719k i/100ms
Calculating -------------------------------------
ActiveModel::Validations
83.106k (± 2.2%) i/s - 417.100k in 5.021417s
ActiveRecord 21.414k (± 3.6%) i/s - 108.576k in 5.077040s
dry-validation 27.500k (± 5.1%) i/s - 138.669k in 5.056987s
Comparison:
ActiveModel::Validations: 83106.0 i/s
dry-validation: 27500.1 i/s - 3.02x (± 0.00) slower
ActiveRecord: 21414.3 i/s - 3.88x (± 0.00) slowerRuby 3.2 without YJIT
Warming up --------------------------------------
ActiveModel::Validations
7.324k i/100ms
ActiveRecord 2.199k i/100ms
dry-validation 2.946k i/100ms
Calculating -------------------------------------
ActiveModel::Validations
73.929k (± 5.4%) i/s - 373.524k in 5.068697s
ActiveRecord 22.085k (± 3.9%) i/s - 112.149k in 5.086915s
dry-validation 29.497k (± 2.5%) i/s - 150.246k in 5.097062s
Comparison:
ActiveModel::Validations: 73929.5 i/s
dry-validation: 29497.2 i/s - 2.51x (± 0.00) slower
ActiveRecord: 22085.3 i/s - 3.35x (± 0.00) slowerRuby 3.2 with YJIT
Warming up --------------------------------------
ActiveModel::Validations
10.136k i/100ms
ActiveRecord 5.064k i/100ms
dry-validation 5.983k i/100ms
Calculating -------------------------------------
ActiveModel::Validations
100.336k (± 2.8%) i/s - 506.800k in 5.055425s
ActiveRecord 50.015k (± 2.4%) i/s - 253.200k in 5.065686s
dry-validation 60.162k (± 1.5%) i/s - 305.133k in 5.072970s
Comparison:
ActiveModel::Validations: 100335.8 i/s
dry-validation: 60161.9 i/s - 1.67x (± 0.00) slower
ActiveRecord: 50015.0 i/s - 2.01x (± 0.00) slowerThere are three really interesting things here:
- Plain object initialization with instance variable assignment is faster under Ruby 3.2
- Enabling YJIT gives a significant performance boost as we go from 150.246k in 5.097062s to 305.133k in 5.072970s in case of dry-validation
- The difference between dry-validation and ActiveRecord vs ActiveModel is significantly smaller when YJIT is enabled
Benchmarking dry-struct
Now this one is just nuts. dry-struct is a library that I created with the intention to replace Virtus. It’s a simple DSL for defining typed structs, just like Virtus; however, it’s meant to be used for defining pure data structures, that’s why it does not provide attribute writers and there are no methods to mutate structs.
One of the goals of dry-struct was to also achieve great performance, because this is what Virtus struggled with. We have a basic benchmark that tests dry-struct against 3 other similar libraries: Virtus, attrio and fast_attributes.
Here’s how it’s implemented:
# frozen_string_literal: true
require "dry/struct"
require "virtus"
require "fast_attributes"
require "attrio"
require "ostruct"
require "benchmark/ips"
class VirtusUser
include Virtus.model
attribute :name, String
attribute :age, Integer
end
class FastUser
extend FastAttributes
define_attributes initialize: true, attributes: true do
attribute :name, String
attribute :age, Integer
end
end
class AttrioUser
include Attrio
define_attributes do
attr :name, String
attr :age, Integer
end
def initialize(attributes = {})
self.attributes = attributes
end
def attributes=(attributes = {})
attributes.each do |attr, value|
send("#{attr}=", value) if respond_to?("#{attr}=")
end
end
end
class DryStructUser < Dry::Struct
attributes(name: "strict.string", age: "params.integer")
end
puts DryStructUser.new(name: "Jane", age: "21").inspect
Benchmark.ips do |x|
x.report("virtus") { VirtusUser.new(name: "Jane", age: "21") }
x.report("fast_attributes") { FastUser.new(name: "Jane", age: "21") }
x.report("attrio") { AttrioUser.new(name: "Jane", age: "21") }
x.report("dry-struct") { DryStructUser.new(name: "Jane", age: "21") }
x.compare!
end
Up until now, fast_attributes was…you guessed it! Fast. The fastest, to be precise.
This is no longer the case though, all you need to do is to enable YJIT! Check out the results:
Ruby 3.1
Warming up --------------------------------------
virtus 5.920k i/100ms
fast_attributes 76.795k i/100ms
attrio 13.118k i/100ms
dry-struct 42.729k i/100ms
Calculating -------------------------------------
virtus 66.051k (± 2.6%) i/s - 331.520k in 5.022813s
fast_attributes 829.331k (± 3.9%) i/s - 4.147M in 5.009166s
attrio 141.829k (± 2.4%) i/s - 721.490k in 5.090232s
dry-struct 439.638k (± 1.0%) i/s - 2.222M in 5.054435s
Comparison:
fast_attributes: 829330.9 i/s
dry-struct: 439637.7 i/s - 1.89x (± 0.00) slower
attrio: 141829.1 i/s - 5.85x (± 0.00) slower
virtus: 66050.6 i/s - 12.56x (± 0.00) slowerRuby 3.2 without YJIT
Warming up --------------------------------------
virtus 7.001k i/100ms
fast_attributes 85.866k i/100ms
attrio 14.706k i/100ms
dry-struct 45.756k i/100ms
Calculating -------------------------------------
virtus 68.503k (± 2.9%) i/s - 343.049k in 5.012351s
fast_attributes 845.828k (± 2.4%) i/s - 4.293M in 5.079085s
attrio 141.372k (± 3.7%) i/s - 720.594k in 5.104778s
dry-struct 447.745k (± 1.5%) i/s - 2.242M in 5.008645s
Comparison:
fast_attributes: 845827.5 i/s
dry-struct: 447745.3 i/s - 1.89x (± 0.00) slower
attrio: 141371.8 i/s - 5.98x (± 0.00) slower
virtus: 68502.8 i/s - 12.35x (± 0.00) slowerRuby 3.2 with YJIT 😱 ⚡
Warming up --------------------------------------
virtus 8.977k i/100ms
fast_attributes 96.806k i/100ms
attrio 20.130k i/100ms
dry-struct 111.189k i/100ms
Calculating -------------------------------------
virtus 88.392k (± 1.7%) i/s - 448.850k in 5.079484s
fast_attributes 973.297k (± 1.3%) i/s - 4.937M in 5.073376s
attrio 197.840k (± 3.7%) i/s - 1.006M in 5.095283s
dry-struct 1.103M (± 1.6%) i/s - 5.559M in 5.042373s
Comparison:
dry-struct: 1102846.4 i/s
fast_attributes: 973296.6 i/s - 1.13x (± 0.00) slower
attrio: 197840.1 i/s - 5.57x (± 0.00) slower
virtus: 88391.5 i/s - 12.48x (± 0.00) slowerWe went from 2.242M in 5.008645s to 5.559M in 5.042373s and fast_attributes is no longer faster than dry-struct. Amazing!
This really blew my mind. I reached out to Nikita asking how on earth is this possible, to which he replied:
I just optimized it
🤣 So now we know, he just optimized it 🙂
On a serious note, this was a series of many optimizations spanning many releases of dry-struct. It’s just that the full potential of these improvements can be observed now with YJIT enabled. I’d love to dive deeper into what happened there, maybe I’ll manage to write about this in detail in another article. Kudos to Nikita for his outstanding work 👏
Ruby 3.2 can be really fast!
Benchmarking is tricky but if you compare the same benchmark using different versions of the same programming language, then chances are the results are really meaningful.
I’ve very glad to see that our test suites are passing and benchmarks show such great results. This is a fantastic release, I’m very happy to see this. Now I can’t wait to upgrade my app to Ruby 3.2 with YJIT enabled and see how it works.
Huge thanks to everybody who worked on Ruby 3.2! ❤️
Enjoy the rest of the holidays!