[ruby-core:123845] [Ruby Bug#21696] Performance degradation for long running processes in Ruby 4.0.0-preview2
Issue #21696 has been reported by easydwh (Ivo Herweijer). ---------------------------------------- Bug #21696: Performance degradation for long running processes in Ruby 4.0.0-preview2 https://bugs.ruby-lang.org/issues/21696 * Author: easydwh (Ivo Herweijer) * Status: Open * ruby -v: ruby 4.0.0preview2 (2025-11-17 master 4fa6e9938c) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When running my RubyMeasureResponsetime tool (https://github.com/easydatawarehousing/ruby_measure_responsetime) on Ruby 4.0.0-preview2, a slow but steady performance degradation is measurable. Both the Rails and the Roda based test applications show this. And both with and without yjit. The Rails application when using yjit also seems to have increasing memory usage over time. Running the same tests on Ruby 3.4.7 shows stable performance and memory usage. I have included some plots showing this behavior. ---Files-------------------------------- rails_devise_2_ruby-4.0.0.jpg (965 KB) rodauth_2_ruby-4.0.0 YJIT.jpg (973 KB) rodauth_2_ruby-4.0.0.jpg (1.02 MB) rails_devise_2_ruby-4.0.0 YJIT.jpg (917 KB) rails_devise_0_memory.png (31.2 KB) -- https://bugs.ruby-lang.org/
Issue #21696 has been updated by ufuk (Ufuk Kayserilioglu). Did 3.5-preview1 have the same behaviour? If not, could you bisect between the two releases to find the commit that changed the behaviour? ---------------------------------------- Bug #21696: Performance degradation for long running processes in Ruby 4.0.0-preview2 https://bugs.ruby-lang.org/issues/21696#change-115244 * Author: easydwh (Ivo Herweijer) * Status: Open * ruby -v: ruby 4.0.0preview2 (2025-11-17 master 4fa6e9938c) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When running my RubyMeasureResponsetime tool (https://github.com/easydatawarehousing/ruby_measure_responsetime) on Ruby 4.0.0-preview2, a slow but steady performance degradation is measurable. Both the Rails and the Roda based test applications show this. And both with and without yjit. The Rails application when using yjit also seems to have increasing memory usage over time. Running the same tests on Ruby 3.4.7 shows stable performance and memory usage. I have included some plots showing this behavior. ---Files-------------------------------- rails_devise_2_ruby-4.0.0.jpg (965 KB) rodauth_2_ruby-4.0.0 YJIT.jpg (973 KB) rodauth_2_ruby-4.0.0.jpg (1.02 MB) rails_devise_2_ruby-4.0.0 YJIT.jpg (917 KB) rails_devise_0_memory.png (31.2 KB) -- https://bugs.ruby-lang.org/
Issue #21696 has been updated by easydwh (Ivo Herweijer). ufuk (Ufuk Kayserilioglu) wrote in #note-1:
Did 3.5-preview1 have the same behaviour? If not, could you bisect between the two releases to find the commit that changed the behaviour?
Just ran the test, only for the Roda app. Results look very similar (almost identical) to 3.4.7. Logically the culprit should be a commit for 4.0-preview. I don't know anything about Ruby source code, so pinpointing the commit(s) involved is beyond my knowledge. ---------------------------------------- Bug #21696: Performance degradation for long running processes in Ruby 4.0.0-preview2 https://bugs.ruby-lang.org/issues/21696#change-115247 * Author: easydwh (Ivo Herweijer) * Status: Open * ruby -v: ruby 4.0.0preview2 (2025-11-17 master 4fa6e9938c) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When running my RubyMeasureResponsetime tool (https://github.com/easydatawarehousing/ruby_measure_responsetime) on Ruby 4.0.0-preview2, a slow but steady performance degradation is measurable. Both the Rails and the Roda based test applications show this. And both with and without yjit. The Rails application when using yjit also seems to have increasing memory usage over time. Running the same tests on Ruby 3.4.7 shows stable performance and memory usage. I have included some plots showing this behavior. ---Files-------------------------------- rails_devise_2_ruby-4.0.0.jpg (965 KB) rodauth_2_ruby-4.0.0 YJIT.jpg (973 KB) rodauth_2_ruby-4.0.0.jpg (1.02 MB) rails_devise_2_ruby-4.0.0 YJIT.jpg (917 KB) rails_devise_0_memory.png (31.2 KB) -- https://bugs.ruby-lang.org/
Issue #21696 has been updated by luke-gru (Luke Gruber). File rodauth_0_overview.png added File rodauth_0_memory.png added File rodauth_1_ruby-3.4.7.png added File rodauth_1_ruby-3.4.7 YJIT.png added File rodauth_1_ruby-4.0.0.png added File rodauth_1_ruby-4.0.0 YJIT.png added I ran the benchmarks, but unfortunately some of the gems are out of date for anyone looking into this. The Gemfile needs: ```ruby gem "sqlite3" # instead of "1.4.2" gem "nio4r", "2.7.5" # instead of "2.5.8" ``` I also had to create a chruby file in `lib/ruby_measure_responsetime/chruby.rb`: ```ruby # frozen_string_literal: true class Chruby def self.ruby_manager if self.ruby_manager_installed? self end end def self.installed_rubies versions = [["master-release", nil]] versions << ["master-release", "--yjit"] end def self.cmd_initialize_ruby_version_manager; end def self.cmd_switch_to_ruby(version) "chruby #{version} > /dev/null;" end private def self.ruby_manager_installed? true end end ``` The plot images are attached. What I saw was: * There is more memory usage with 4.0.0 compared to 3.4.7, but this is expected. We will mention this in the release notes, and we're working on reducing it. * Running 100k requests doesn't take that long (around 1 min on my machine) so it's hard to say anything about performance degradation or memory growth across the lifetime of the process. You would need more data across a longer timeline for this. * I don't see a reduction in performance according to the images generated from my run. * The memory and histogram graphs are very hard to make out or they didn't work. I attached them anyway. ---------------------------------------- Bug #21696: Performance degradation for long running processes in Ruby 4.0.0-preview2 https://bugs.ruby-lang.org/issues/21696#change-115264 * Author: easydwh (Ivo Herweijer) * Status: Open * ruby -v: ruby 4.0.0preview2 (2025-11-17 master 4fa6e9938c) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When running my RubyMeasureResponsetime tool (https://github.com/easydatawarehousing/ruby_measure_responsetime) on Ruby 4.0.0-preview2, a slow but steady performance degradation is measurable. Both the Rails and the Roda based test applications show this. And both with and without yjit. The Rails application when using yjit also seems to have increasing memory usage over time. Running the same tests on Ruby 3.4.7 shows stable performance and memory usage. I have included some plots showing this behavior. ---Files-------------------------------- rails_devise_2_ruby-4.0.0.jpg (965 KB) rodauth_2_ruby-4.0.0 YJIT.jpg (973 KB) rodauth_2_ruby-4.0.0.jpg (1.02 MB) rails_devise_2_ruby-4.0.0 YJIT.jpg (917 KB) rails_devise_0_memory.png (31.2 KB) rodauth_0_overview.png (43.8 KB) rodauth_0_memory.png (15.6 KB) rodauth_1_ruby-3.4.7.png (152 KB) rodauth_1_ruby-3.4.7 YJIT.png (121 KB) rodauth_1_ruby-4.0.0.png (119 KB) rodauth_1_ruby-4.0.0 YJIT.png (118 KB) -- https://bugs.ruby-lang.org/
Issue #21696 has been updated by easydwh (Ivo Herweijer). Thanks for running the tests! On my machine (x64 Linux/AMD Ryzen 7 5800X) Roda works fine with the Gemfile from the repository. Yes, most gems are (very) old, I have pinned these in order to have a stable set of gems so comparing results to previous test runs is possible. For the test itself it doesn't really matter if gems are up to date or not as long as they are same for every tested Ruby version. Ran the test again with 100k requests and the gem versions you mentioned. On my machine this takes 345 and 302 seconds for 4.0-preview2. Especially on the detail plot (like rodauth_2_ruby-4.0.0.png) the increasing response time is clearly visible. Same test on Ruby 3.4.7 shows neat horizontal lines. You machine is clearly a lot faster than mine, maybe this hides performance degradation? Another difference between your plots and mine is the number of major garbage collection runs. These are indicated by the short blue vertical lines. Your plots show fairly regular GC runs. Mine only 1 very early in the test. If higher memory usage is expected for 4.0 then no problems there. Memory usage seems to level off after ~10k requests anyway. Usually I run the tests 3 times, so more dots in the memory plot and better readable lines. I don't know why in your memory plot there are no measurements after ~15k. All values should be in file 'data/rodauth/memory.csv'. Anyway, a bug is not a bug when it isn't reproducible. If you don't see any performance degradation then this bug report should be closed. ---------------------------------------- Bug #21696: Performance degradation for long running processes in Ruby 4.0.0-preview2 https://bugs.ruby-lang.org/issues/21696#change-115280 * Author: easydwh (Ivo Herweijer) * Status: Open * ruby -v: ruby 4.0.0preview2 (2025-11-17 master 4fa6e9938c) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When running my RubyMeasureResponsetime tool (https://github.com/easydatawarehousing/ruby_measure_responsetime) on Ruby 4.0.0-preview2, a slow but steady performance degradation is measurable. Both the Rails and the Roda based test applications show this. And both with and without yjit. The Rails application when using yjit also seems to have increasing memory usage over time. Running the same tests on Ruby 3.4.7 shows stable performance and memory usage. I have included some plots showing this behavior. ---Files-------------------------------- rails_devise_2_ruby-4.0.0.jpg (965 KB) rodauth_2_ruby-4.0.0 YJIT.jpg (973 KB) rodauth_2_ruby-4.0.0.jpg (1.02 MB) rails_devise_2_ruby-4.0.0 YJIT.jpg (917 KB) rails_devise_0_memory.png (31.2 KB) rodauth_0_overview.png (43.8 KB) rodauth_0_memory.png (15.6 KB) rodauth_1_ruby-3.4.7.png (152 KB) rodauth_1_ruby-3.4.7 YJIT.png (121 KB) rodauth_1_ruby-4.0.0.png (119 KB) rodauth_1_ruby-4.0.0 YJIT.png (118 KB) -- https://bugs.ruby-lang.org/
Issue #21696 has been updated by easydwh (Ivo Herweijer). @luke-gru FWIW 4.0.0-preview3 performs normally. ---------------------------------------- Bug #21696: Performance degradation for long running processes in Ruby 4.0.0-preview2 https://bugs.ruby-lang.org/issues/21696#change-115825 * Author: easydwh (Ivo Herweijer) * Status: Closed * ruby -v: ruby 4.0.0preview2 (2025-11-17 master 4fa6e9938c) +PRISM [x86_64-linux] * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- When running my RubyMeasureResponsetime tool (https://github.com/easydatawarehousing/ruby_measure_responsetime) on Ruby 4.0.0-preview2, a slow but steady performance degradation is measurable. Both the Rails and the Roda based test applications show this. And both with and without yjit. The Rails application when using yjit also seems to have increasing memory usage over time. Running the same tests on Ruby 3.4.7 shows stable performance and memory usage. I have included some plots showing this behavior. ---Files-------------------------------- rails_devise_2_ruby-4.0.0.jpg (965 KB) rodauth_2_ruby-4.0.0 YJIT.jpg (973 KB) rodauth_2_ruby-4.0.0.jpg (1.02 MB) rails_devise_2_ruby-4.0.0 YJIT.jpg (917 KB) rails_devise_0_memory.png (31.2 KB) rodauth_0_overview.png (43.8 KB) rodauth_0_memory.png (15.6 KB) rodauth_1_ruby-3.4.7.png (152 KB) rodauth_1_ruby-3.4.7 YJIT.png (121 KB) rodauth_1_ruby-4.0.0.png (119 KB) rodauth_1_ruby-4.0.0 YJIT.png (118 KB) -- https://bugs.ruby-lang.org/
participants (3)
-
easydwh (Ivo Herweijer) -
luke-gru (Luke Gruber) -
ufuk (Ufuk Kayserilioglu)