[ruby-core:115251] [Ruby master Bug#19989] Fix refinement refine performance
Issue #19989 has been reported by bkuhlmann (Brooke Kuhlmann). ---------------------------------------- Bug #19989: Fix refinement refine performance https://bugs.ruby-lang.org/issues/19989 * Author: bkuhlmann (Brooke Kuhlmann) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Why Hello. 👋 Would it be possible to improve the performance of refining an object as I'm seeing a significant performance cost? I know some of this is already known but, when searching through existing issues, I couldn't find any issue that specifically mentioned or called attention to `refine` performance. My C skills are not that great but it seems that `rb_mod_refine` ([source](https://github.com/ruby/ruby/blob/master/eval.c#L1412-L1459)) might be the primary source of complexity and performance degradation? By fixing this, we could improve refinement performance and encourage more adoption since refinements are a powerful way to make small adjustments to the language without resorting to more complicated solutions -- or worse -- using monkey patches. ## How Here's the benchmark I'm using: ``` #! /usr/bin/env ruby # frozen_string_literal: true require "bundler/inline" gemfile true do source "https://rubygems.org" gem "benchmark-ips" end Benchmark.ips do |benchmark| benchmark.config time: 5, warmup: 2 benchmark.report "With" do Module.new do refine String do def dud = true end end end benchmark.report "Without" do Module.new do def dud = true end end benchmark.compare! end ``` When running the above, I'm getting the following results (this is a random sample): **Ruby 2.6.6** (ruby 2.6.6p146 (2020-03-31 revision 67876) [-darwin22]) *ℹ️ Temporarily refactored the benchmark to not use endless methods for this version.* ``` Warming up -------------------------------------- With 745.000 i/100ms Without 67.593k i/100ms Calculating ------------------------------------- With 10.849k (±168.6%) i/s - 18.625k in 5.359230s Without 680.199k (± 5.8%) i/s - 3.447M in 5.085558s Comparison: Without: 680198.8 i/s With: 10849.1 i/s - 62.70x slower ``` **Ruby 3.2.2** (ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0]) ``` Warming up -------------------------------------- With 959.000 i/100ms Without 297.091k i/100ms Calculating ------------------------------------- With 21.856k (±198.5%) i/s - 30.688k in 5.509103s Without 3.281M (± 5.8%) i/s - 16.637M in 5.082246s Comparison: Without: 3281310.6 i/s With: 21856.1 i/s - 150.13x slower ``` **Ruby 3.3.3 Preview 2** (ruby 3.3.0preview2 (2023-09-14 master e50fcca9a7) +YJIT [arm64-darwin22.6.0]) ``` Warming up -------------------------------------- With 1.044k i/100ms Without 288.264k i/100ms Calculating ------------------------------------- With 24.044k (±192.0%) i/s - 29.232k in 5.305651s Without 3.096M (± 8.0%) i/s - 15.566M in 5.059470s Comparison: Without: 3095640.6 i/s With: 24043.8 i/s - 128.75x slower ``` ## Notes Here's the environment I'm using: - **macOS**: 13.6.1, Apple M1 -- https://bugs.ruby-lang.org/
Issue #19989 has been updated by ahorek (Pavel Rosický).
By fixing this, we could improve refinement performance and encourage more adoption since refinements are a powerful way to make small adjustments to the language without resorting to more complicated solutions -- or worse -- using monkey patches.
this shouldn't really affect the runtime performance of real programs unless you're dynamically creating refined modules like in your benchmark (which is also slow). Do you have an example where this could be a problem? ---------------------------------------- Bug #19989: Fix refinement refine performance https://bugs.ruby-lang.org/issues/19989#change-105163 * Author: bkuhlmann (Brooke Kuhlmann) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Why Hello. 👋 Would it be possible to improve the performance of refining an object as I'm seeing a significant performance cost? I know some of this is already known but, when searching through existing issues, I couldn't find any issue that specifically mentioned or called attention to `refine` performance. My C skills are not that great but it seems that `rb_mod_refine` ([source](https://github.com/ruby/ruby/blob/master/eval.c#L1412-L1459)) might be the primary source of complexity and performance degradation? By fixing this, we could improve refinement performance and encourage more adoption since refinements are a powerful way to make small adjustments to the language without resorting to more complicated solutions -- or worse -- using monkey patches. ## How Here's the benchmark I'm using: ``` #! /usr/bin/env ruby # frozen_string_literal: true require "bundler/inline" gemfile true do source "https://rubygems.org" gem "benchmark-ips" end Benchmark.ips do |benchmark| benchmark.config time: 5, warmup: 2 benchmark.report "With" do Module.new do refine String do def dud = true end end end benchmark.report "Without" do Module.new do def dud = true end end benchmark.compare! end ``` When running the above, I'm getting the following results (this is a random sample): **Ruby 2.6.6** (ruby 2.6.6p146 (2020-03-31 revision 67876) [-darwin22]) *ℹ️ Temporarily refactored the benchmark to not use endless methods for this version.* ``` Warming up -------------------------------------- With 745.000 i/100ms Without 67.593k i/100ms Calculating ------------------------------------- With 10.849k (±168.6%) i/s - 18.625k in 5.359230s Without 680.199k (± 5.8%) i/s - 3.447M in 5.085558s Comparison: Without: 680198.8 i/s With: 10849.1 i/s - 62.70x slower ``` **Ruby 3.2.2** (ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0]) ``` Warming up -------------------------------------- With 959.000 i/100ms Without 297.091k i/100ms Calculating ------------------------------------- With 21.856k (±198.5%) i/s - 30.688k in 5.509103s Without 3.281M (± 5.8%) i/s - 16.637M in 5.082246s Comparison: Without: 3281310.6 i/s With: 21856.1 i/s - 150.13x slower ``` **Ruby 3.3.3 Preview 2** (ruby 3.3.0preview2 (2023-09-14 master e50fcca9a7) +YJIT [arm64-darwin22.6.0]) ``` Warming up -------------------------------------- With 1.044k i/100ms Without 288.264k i/100ms Calculating ------------------------------------- With 24.044k (±192.0%) i/s - 29.232k in 5.305651s Without 3.096M (± 8.0%) i/s - 15.566M in 5.059470s Comparison: Without: 3095640.6 i/s With: 24043.8 i/s - 128.75x slower ``` ## Notes Here's the environment I'm using: - **macOS**: 13.6.1, Apple M1 -- https://bugs.ruby-lang.org/
Issue #19989 has been updated by bkuhlmann (Brooke Kuhlmann).
this shouldn't really affect the runtime performance of real programs
Thanks and fair point. I'm using the above benchmark to track performance across Ruby versions. I agree, dynamically creating refinements *isn't normal* but wanted to simulate the *defining* of a refinement. I know there is cost in defining the module each time but am trying to zero in on the use of `refine`, specifically. Maybe this is the wrong way to tackle the problem and I should be using a different benchmark?
Do you have an example where this could be a problem?
I'm afraid not. Well, not as easily encapsulated, that is. I do have the following: - [Benchmarks](https://alchemists.io/projects/benchmarks#_scriptsrefinementsmessage): This is an attempt to benchmark refinement performance from multiple angles (along with other non-refinement based benchmarks). - [Refinements](https://alchemists.io/projects/refinements): The gem -- which I maintain -- and have been using in multiple projects. Also the impetus for wanting to investigate the cost of defining refinements. ---------------------------------------- Bug #19989: Fix refinement refine performance https://bugs.ruby-lang.org/issues/19989#change-105164 * Author: bkuhlmann (Brooke Kuhlmann) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Why Hello. 👋 Would it be possible to improve the performance of refining an object as I'm seeing a significant performance cost? I know some of this is already known but, when searching through existing issues, I couldn't find any issue that specifically mentioned or called attention to `refine` performance. My C skills are not that great but it seems that `rb_mod_refine` ([source](https://github.com/ruby/ruby/blob/master/eval.c#L1412-L1459)) might be the primary source of complexity and performance degradation? By fixing this, we could improve refinement performance and encourage more adoption since refinements are a powerful way to make small adjustments to the language without resorting to more complicated solutions -- or worse -- using monkey patches. ## How Here's the benchmark I'm using: ``` #! /usr/bin/env ruby # frozen_string_literal: true require "bundler/inline" gemfile true do source "https://rubygems.org" gem "benchmark-ips" end Benchmark.ips do |benchmark| benchmark.config time: 5, warmup: 2 benchmark.report "With" do Module.new do refine String do def dud = true end end end benchmark.report "Without" do Module.new do def dud = true end end benchmark.compare! end ``` When running the above, I'm getting the following results (this is a random sample): **Ruby 2.6.6** (ruby 2.6.6p146 (2020-03-31 revision 67876) [-darwin22]) *ℹ️ Temporarily refactored the benchmark to not use endless methods for this version.* ``` Warming up -------------------------------------- With 745.000 i/100ms Without 67.593k i/100ms Calculating ------------------------------------- With 10.849k (±168.6%) i/s - 18.625k in 5.359230s Without 680.199k (± 5.8%) i/s - 3.447M in 5.085558s Comparison: Without: 680198.8 i/s With: 10849.1 i/s - 62.70x slower ``` **Ruby 3.2.2** (ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0]) ``` Warming up -------------------------------------- With 959.000 i/100ms Without 297.091k i/100ms Calculating ------------------------------------- With 21.856k (±198.5%) i/s - 30.688k in 5.509103s Without 3.281M (± 5.8%) i/s - 16.637M in 5.082246s Comparison: Without: 3281310.6 i/s With: 21856.1 i/s - 150.13x slower ``` **Ruby 3.3.3 Preview 2** (ruby 3.3.0preview2 (2023-09-14 master e50fcca9a7) +YJIT [arm64-darwin22.6.0]) ``` Warming up -------------------------------------- With 1.044k i/100ms Without 288.264k i/100ms Calculating ------------------------------------- With 24.044k (±192.0%) i/s - 29.232k in 5.305651s Without 3.096M (± 8.0%) i/s - 15.566M in 5.059470s Comparison: Without: 3095640.6 i/s With: 24043.8 i/s - 128.75x slower ``` ## Notes Here's the environment I'm using: - **macOS**: 13.6.1, Apple M1 -- https://bugs.ruby-lang.org/
Issue #19989 has been updated by Eregon (Benoit Daloze).
Maybe this is the wrong way to tackle the problem and I should be using a different benchmark?
Yes, the performance of `refine` itself should be irrelevant, it's paid once during startup (unless it would amount to taking a significant overhead to load a real app). Just like calling `Module.new` in the inner loop of an app is a mistake, those tools (module definition, defining refinements) are meant to be used during code loading and very little after. ---------------------------------------- Bug #19989: Fix refinement refine performance https://bugs.ruby-lang.org/issues/19989#change-105210 * Author: bkuhlmann (Brooke Kuhlmann) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Why Hello. 👋 Would it be possible to improve the performance of refining an object as I'm seeing a significant performance cost? I know some of this is already known but, when searching through existing issues, I couldn't find any issue that specifically mentioned or called attention to `refine` performance. My C skills are not that great but it seems that `rb_mod_refine` ([source](https://github.com/ruby/ruby/blob/master/eval.c#L1412-L1459)) might be the primary source of complexity and performance degradation? By fixing this, we could improve refinement performance and encourage more adoption since refinements are a powerful way to make small adjustments to the language without resorting to more complicated solutions -- or worse -- using monkey patches. ## How Here's the benchmark I'm using: ``` #! /usr/bin/env ruby # frozen_string_literal: true require "bundler/inline" gemfile true do source "https://rubygems.org" gem "benchmark-ips" end Benchmark.ips do |benchmark| benchmark.config time: 5, warmup: 2 benchmark.report "With" do Module.new do refine String do def dud = true end end end benchmark.report "Without" do Module.new do def dud = true end end benchmark.compare! end ``` When running the above, I'm getting the following results (this is a random sample): **Ruby 2.6.6** (ruby 2.6.6p146 (2020-03-31 revision 67876) [-darwin22]) *ℹ️ Temporarily refactored the benchmark to not use endless methods for this version.* ``` Warming up -------------------------------------- With 745.000 i/100ms Without 67.593k i/100ms Calculating ------------------------------------- With 10.849k (±168.6%) i/s - 18.625k in 5.359230s Without 680.199k (± 5.8%) i/s - 3.447M in 5.085558s Comparison: Without: 680198.8 i/s With: 10849.1 i/s - 62.70x slower ``` **Ruby 3.2.2** (ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0]) ``` Warming up -------------------------------------- With 959.000 i/100ms Without 297.091k i/100ms Calculating ------------------------------------- With 21.856k (±198.5%) i/s - 30.688k in 5.509103s Without 3.281M (± 5.8%) i/s - 16.637M in 5.082246s Comparison: Without: 3281310.6 i/s With: 21856.1 i/s - 150.13x slower ``` **Ruby 3.3.3 Preview 2** (ruby 3.3.0preview2 (2023-09-14 master e50fcca9a7) +YJIT [arm64-darwin22.6.0]) ``` Warming up -------------------------------------- With 1.044k i/100ms Without 288.264k i/100ms Calculating ------------------------------------- With 24.044k (±192.0%) i/s - 29.232k in 5.305651s Without 3.096M (± 8.0%) i/s - 15.566M in 5.059470s Comparison: Without: 3095640.6 i/s With: 24043.8 i/s - 128.75x slower ``` ## Notes Here's the environment I'm using: - **macOS**: 13.6.1, Apple M1 -- https://bugs.ruby-lang.org/
Issue #19989 has been updated by bkuhlmann (Brooke Kuhlmann). OK, thanks. Make sense. Feel free to close this. ---------------------------------------- Bug #19989: Fix refinement refine performance https://bugs.ruby-lang.org/issues/19989#change-105211 * Author: bkuhlmann (Brooke Kuhlmann) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Why Hello. 👋 Would it be possible to improve the performance of refining an object as I'm seeing a significant performance cost? I know some of this is already known but, when searching through existing issues, I couldn't find any issue that specifically mentioned or called attention to `refine` performance. My C skills are not that great but it seems that `rb_mod_refine` ([source](https://github.com/ruby/ruby/blob/master/eval.c#L1412-L1459)) might be the primary source of complexity and performance degradation? By fixing this, we could improve refinement performance and encourage more adoption since refinements are a powerful way to make small adjustments to the language without resorting to more complicated solutions -- or worse -- using monkey patches. ## How Here's the benchmark I'm using: ``` #! /usr/bin/env ruby # frozen_string_literal: true require "bundler/inline" gemfile true do source "https://rubygems.org" gem "benchmark-ips" end Benchmark.ips do |benchmark| benchmark.config time: 5, warmup: 2 benchmark.report "With" do Module.new do refine String do def dud = true end end end benchmark.report "Without" do Module.new do def dud = true end end benchmark.compare! end ``` When running the above, I'm getting the following results (this is a random sample): **Ruby 2.6.6** (ruby 2.6.6p146 (2020-03-31 revision 67876) [-darwin22]) *ℹ️ Temporarily refactored the benchmark to not use endless methods for this version.* ``` Warming up -------------------------------------- With 745.000 i/100ms Without 67.593k i/100ms Calculating ------------------------------------- With 10.849k (±168.6%) i/s - 18.625k in 5.359230s Without 680.199k (± 5.8%) i/s - 3.447M in 5.085558s Comparison: Without: 680198.8 i/s With: 10849.1 i/s - 62.70x slower ``` **Ruby 3.2.2** (ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22.4.0]) ``` Warming up -------------------------------------- With 959.000 i/100ms Without 297.091k i/100ms Calculating ------------------------------------- With 21.856k (±198.5%) i/s - 30.688k in 5.509103s Without 3.281M (± 5.8%) i/s - 16.637M in 5.082246s Comparison: Without: 3281310.6 i/s With: 21856.1 i/s - 150.13x slower ``` **Ruby 3.3.3 Preview 2** (ruby 3.3.0preview2 (2023-09-14 master e50fcca9a7) +YJIT [arm64-darwin22.6.0]) ``` Warming up -------------------------------------- With 1.044k i/100ms Without 288.264k i/100ms Calculating ------------------------------------- With 24.044k (±192.0%) i/s - 29.232k in 5.305651s Without 3.096M (± 8.0%) i/s - 15.566M in 5.059470s Comparison: Without: 3095640.6 i/s With: 24043.8 i/s - 128.75x slower ``` ## Notes Here's the environment I'm using: - **macOS**: 13.6.1, Apple M1 -- https://bugs.ruby-lang.org/
participants (3)
-
ahorek -
bkuhlmann (Brooke Kuhlmann)
-
Eregon (Benoit Daloze)