[ruby-core:120637] [Ruby master Bug#21032] `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path

Issue #21032 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by Eregon (Benoit Daloze). For curiosity, what's adding a relative path in `$LOAD_PATH`? That sounds rather dubious/wrong. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-111468 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by byroot (Jean Boussier). @fxn tracked it down to https://github.com/emailage/Emailage_Ruby/blob/64b9762cda7608ac1eeced2a85ad5..., and yes that's an anti-pattern in my opinion too. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-111469 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by byroot (Jean Boussier). I dug more into this today, based on @nobu's review. `autoload?` isn't the only thing slowed down in such case. Perhaps we should try to emit a performance warning when a relative path or non-string object is appended to `$LOAD_PATH`. That would at least make it easier to notice this issue. The problem of course is that `$LOAD_PATH` is just a regular array, so there isn't a clean hook where to check for this. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-111484 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by Eregon (Benoit Daloze). byroot (Jean Boussier) wrote in #note-2:
@fxn tracked it down to https://github.com/emailage/Emailage_Ruby/blob/64b9762cda7608ac1eeced2a85ad5..., and yes that's an anti-pattern in my opinion too.
Right, that's just broken code. It would consider files under `lib` in $PWD for require which is wrong.
Perhaps we should try to emit a performance warning when a relative path or non-string object is appended to `$LOAD_PATH`. That would at least make it easier to notice this issue.
The problem of course is that `$LOAD_PATH` is just a regular array, so there isn't a clean hook where to check for this.
Maybe just warn on `require` or so, i.e. when noticing the path is relative, if it's too hard to notice when it's added? I think there is also a cache for the expanded load path, maybe that could be a reasonable place to check. It wouldn't immediately point at the culprit, but printing the $LOADED_FEATURES should help find it (maybe there should be a CLI flag/ENV var to print files as they are loaded, TruffleRuby has that). ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-111487 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by byroot (Jean Boussier). Status changed from Open to Closed I merged https://github.com/ruby/ruby/pull/12562 / d4a1a2780c39bc648496ac92fc6e6ce2eb38ab47 because I couldn't find any case where this would change behavior. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-111668 * Author: byroot (Jean Boussier) * Status: Closed * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by k0kubun (Takashi Kokubun). Backport changed from 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED to 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: DONE ruby_3_4 commit:ead3bbc2405ad1df2228c44133ee1c6574ef5973 merged revision(s) commit:d4a1a2780c39bc648496ac92fc6e6ce2eb38ab47. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-111931 * Author: byroot (Jean Boussier) * Status: Closed * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: DONE ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by nagachika (Tomoyuki Chikanaga). Backport changed from 3.1: WONTFIX, 3.2: REQUIRED, 3.3: REQUIRED, 3.4: DONE to 3.1: WONTFIX, 3.2: REQUIRED, 3.3: DONE, 3.4: DONE ruby_3_3 commit:54dd27d89d2e6814114f1aff18836a987d5a4ab1 merged revision(s) commit:d4a1a2780c39bc648496ac92fc6e6ce2eb38ab47. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-112223 * Author: byroot (Jean Boussier) * Status: Closed * Backport: 3.1: WONTFIX, 3.2: REQUIRED, 3.3: DONE, 3.4: DONE ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/

Issue #21032 has been updated by hsbt (Hiroshi SHIBATA). Backport changed from 3.1: WONTFIX, 3.2: REQUIRED, 3.3: DONE, 3.4: DONE to 3.1: WONTFIX, 3.2: DONE, 3.3: DONE, 3.4: DONE ruby_3_2 commit:fd15d0f4cbddc8cf69013959e2a60734d0995d56 merged revision(s) commit:d4a1a2780c39bc648496ac92fc6e6ce2eb38ab47. ---------------------------------------- Bug #21032: `Module#autoload?` is slow when `$LOAD_PATH` contains a relative path https://bugs.ruby-lang.org/issues/21032#change-112277 * Author: byroot (Jean Boussier) * Status: Closed * Backport: 3.1: WONTFIX, 3.2: DONE, 3.3: DONE, 3.4: DONE ---------------------------------------- Reproduction script: ```ruby require 'benchmark' $LOAD_PATH << 'relative-path' autoload :FOO, '/tmp/foo.rb' puts Benchmark.realtime { 500_000.times do Object.autoload?(:FOO) end } ``` The above takes 2.5 to 3 seconds on my machine, but just removing `$LOAD_PATH << 'relative-path'` make it complete in 50ms. It's such a stark difference that I think it is a bug, and it cause Zeitwerk, a very popular gem, to be way slower than it should when the load path contains relative paths. I have a patch for it, that passes all tests, but I'd appreciate some eyes on it: https://github.com/ruby/ruby/pull/12562 cc @fxn -- https://bugs.ruby-lang.org/
participants (5)
-
byroot (Jean Boussier)
-
Eregon (Benoit Daloze)
-
hsbt (Hiroshi SHIBATA)
-
k0kubun (Takashi Kokubun)
-
nagachika (Tomoyuki Chikanaga)