February 2025 - ruby-core - ml.ruby-lang.org

[ruby-core:120958] [Ruby master Feature#15663] Documenting autoload semantics
by Eregon (Benoit Daloze) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by Eregon (Benoit Daloze). Right, that's a good summary. It's natural for the constant lookup to use the constant scope as usual, and that part is fine. The part that's messy is when doing the `require` for an `autoload`, and it doesn't define the constant it "promised" to. In such a case I think an exception would be warranted, with of course a migration path of first warning such cases. We actually already have these warnings: ``` $ cat a.rb A = 42 $ ruby -w -I. -e 'autoload :A, "a"; p A' 42 $ ruby -w -I. -e 'autoload :B, "a"; p B' -e:1: warning: Expected a to define B but it didn't -e:1:in `<main>': uninitialized constant B (NameError) autoload :B, "a"; p B ^ $ ruby -w -I. -e 'module M; autoload :A, "a"; p A; end' -e:1: warning: Expected a to define M::A but it didn't 42 ``` So I guess I should make a ticket about turning those warnings into errors now :) At least it would be one complex aspect of autoload removed. ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111853 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120955] [Ruby master Feature#15663] Documenting autoload semantics
by fxn (Xavier Noria) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by fxn (Xavier Noria). I see what you mean, but let me rephrase it a bit. The autoload does not depend on the nesting (depending on your definition of "autoload"). It is triggered by a missing constant reference when the receiver is the cref, and the execution of the required file is not influence by the nesting in the caller (you of course know this, only describing). Success/failure of the `require` call does not depend on the nesting. Constant lookup does depend on the nesting, but constant lookup and autoload are independent of each other (except that one may trigger the other one). I agree with you in those rules, I would also prefer this to be simpler and more strict. Either you define the constant in the receiver or error. I'd go further for the sake of the exercise (but impossible to change). An autoload is a promise, but it may fail. In that sense, if this was day 0 and I was the author of the language, autoloads would not appear in the constants API. Only "actual" constants would. ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111850 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120954] [Ruby master Feature#15663] Documenting autoload semantics
by Eregon (Benoit Daloze) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by Eregon (Benoit Daloze). fxn (Xavier Noria) wrote in #note-9: > Autoload is just a trigger for Kernel#require, there is no expectation on its side-effects. There certainly is for any Rubyist though (the whole point of an autoload is it lazily defines some constant on first access), but yeah the current implementation doesn't match that so strictly unfortunately. ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111849 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120953] [Ruby master Feature#15663] Documenting autoload semantics
by Eregon (Benoit Daloze) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by Eregon (Benoit Daloze). Right, I recall those, I feel these should really give a NameError or LoadError when the autoload is triggered, because it's basically an incorrect usage of autoload. It can result in a pretty weird state for M::N::X / MyGem::Date, what are those now, are they still autoload-registered constants (but then it might trigger the autoload again)?, are they removed (seems a bit weird if removed but then the access to the constant is still OK)? are they "undefined"? are they actually set to the same value as M::X/::Date (even though the Ruby code doesn't do that)? IIRC it changed from undefined to removed in some Ruby version. The fact the constant scope (IOW `Module.nesting`) is used when resolving an autoload is IMO wrong, because then resolving an autoload in one or another place can give a different result. IOW, I'd like when an autoload is triggered to simplify the outcome to just two outcomes, when the require for the autoload is done: * the constant which used to be the autoload constant has been replaced by a value, all good, autoloading succeeded, return that * otherwise error like NameError or LoadError because the autoload was incorrectly registered or the loaded file didn't define the correct constant. Similar to `require 'file_which_does_not_exist'`. Right now we also have: * the file defined a different constant, and that might still work due to constant lookup, but it depends on which constant access triggered the autoload first, which is obviously brittle and makes things really complicated. ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111848 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120952] [Ruby master Feature#15663] Documenting autoload semantics
by fxn (Xavier Noria) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by fxn (Xavier Noria). BTW, I am _describing_ how it works, not implying I like it :). ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111847 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120951] [Ruby master Feature#15663] Documenting autoload semantics
by fxn (Xavier Noria) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by fxn (Xavier Noria). A classic way to exploit that behavior I've seen out there (in the past): ```ruby module MyGem autoload :Date, 'date' end ``` That meant, make `Date` available in case client code uses my gem in a way that hits an internal reference to `Date`. But do it with `autoload` so that it is not eager loaded if not needed. ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111846 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120950] [Ruby master Feature#15663] Documenting autoload semantics
by fxn (Xavier Noria) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by fxn (Xavier Noria). Hi @eregon! > From that POV any autoload which when triggered does not define the constant should be an error, specifically a LoadError Not quite! Autoload is just a trigger for `Kernel#require`, there is no expectation on its side-effects. Once triggered, constant lookup continues and it is constant lookup what drives possible errors. For example, consider ``` ruby # x.rb M::X = 1 # m.rb module M module N autoload :X, 'x' p X # prints 1! end end ``` That works, because `autoload` does not care about what happened. It triggered `Kernel#require`, which loaded the file, control back. `N::X` does not exist, up the nesting `M::X` exist, so the constant is resolved. (Zeitwerk does require the constant to be defined in the receiver, but that is a contract enforced by the library to match file name conventions and have more strict rules.) ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111845 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120948] [Ruby master Feature#15663] Documenting autoload semantics
by Eregon (Benoit Daloze) 12 Feb '25

12 Feb '25

Issue #15663 has been updated by Eregon (Benoit Daloze). @fxn Regarding the comment above, I think always treating autoload constants as if they already existed seems one way to be more consistent. From that POV any autoload which when triggered does not define the constant should be an error, specifically a LoadError or so, which the program shouldn't catch and so should be considered a "fatal" error of that program. Similar to e.g. a constant is defined but some bad code does `remove_const` of it later, those kind of broken code edge cases should be considered bugs of the program, and in fact anyone writing Ruby code relies on gems not doing such nasty things. ---------------------------------------- Feature #15663: Documenting autoload semantics https://bugs.ruby-lang.org/issues/15663#change-111844 * Author: Eregon (Benoit Daloze) * Status: Open ---------------------------------------- The semantics of autoload are extremely complicated. As far as I can see, they are unfortunately not documented. ruby/spec tries to test [many aspects](https://github.com/ruby/spec/blob/master/core/module/autoload_spec… of it, `test/ruby/test_autoload.rb` has a few tests, and e.g. zeitwerk tests [some other parts](https://github.com/fxn/zeitwerk/blob/master/test/lib/zeitwerk/test_r…. One could of course read the MRI source code, but I find it very hard to follow around `autoload`. For the context, I'm trying to implement `autoload` as correct as possible in TruffleRuby and finding it very difficult given the inconsistencies (see below) and lack of documentation. There is nowhere a document on how it should behave, and given the complexity of it I am not even sure MRI behaves as expected. Could we create this document? For instance, there is such a [document for refinements](https://github.com/ruby/ruby/blob/trunk/doc/syntax/refinements…. Here is an example how confusing autoload can be, and I would love to hear the rationale or have some written semantics on why it is that way. main.rb: ```ruby require "pp" $: << __dir__ Object.autoload(:Foo, "foo") CHECK = -> state { checks = -> { { defined: defined?(Foo), const_defined: Object.const_defined?(:Foo), autoload?: Object.autoload?(:Foo), in_constants: Object.constants.include?(:Foo), } } pp when: state, **checks.call, other_thread: Thread.new { checks.call }.value } CHECK.call(:before_require) if ARGV.first == "require" require "foo" else Foo # trigger the autoload end CHECK.call(:after) p Foo ``` foo.rb: ```ruby CHECK.call(:during_before_defining) module Foo end CHECK.call(:during_after_defining) ``` Here are the results for MRI 2.6.1: ```ruby $ ruby main.rb {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, the constant looks not defined during the autoload for the Thread loading it, but looks defined and as an autoload for other threads. Now we can discover other subtle semantics, by using `require` on the autoload file instead of accessing the constant: ```ruby $ ruby main.rb require {:when=>:before_require, :defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>"foo", :in_constants=>true}} {:when=>:during_before_defining, :defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>nil, :const_defined=>false, :autoload?=>nil, :in_constants=>true}} {:when=>:during_after_defining, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} {:when=>:after, :defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true, :other_thread=> {:defined=>"constant", :const_defined=>true, :autoload?=>nil, :in_constants=>true}} Foo ``` Looking at during_before_defining, now the other threads seem to see the constant not defined, although it is still in `Object.constants`. But of course, the constant cannot be removed, as otherwise that would not be thread-safe and other threads would raise NameError when accessing the constant. In fact, we can see other threads actually wait for the constant, by changing to `Thread.new { Foo; checks.call }`, and then we get a deadlock: ``` Traceback (most recent call last): 2: from main.rb:20:in `<main>' 1: from main.rb:17:in `block in <main>' main.rb:17:in `value': No live threads left. Deadlock? (fatal) 3 threads, 3 sleeps current:0x00007f0124004cb0 main thread:0x000055929cc2c470 * #<Thread:0x000055929cc5b348 sleep_forever> rb_thread_t:0x000055929cc2c470 native:0x00007f013381d700 int:0 main.rb:17:in `value' main.rb:17:in `block in <main>' main.rb:20:in `<main>' * #<Thread:0x000055929ce2b380@main.rb:17 sleep_forever> rb_thread_t:0x000055929ce026d0 native:0x00007f0129007700 int:0 depended by: tb_thread_id:0x000055929cc2c470 main.rb:17:in `value' main.rb:17:in `block in <main>' foo.rb:1:in `<top (required)>' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' /home/eregon/.rubies/ruby-2.6.1/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require' main.rb:17:in `block (2 levels) in <main>' * #<Thread:0x000055929ce29c38@main.rb:17 sleep_forever> rb_thread_t:0x00007f0124004cb0 native:0x00007f0128e05700 int:0 depended by: tb_thread_id:0x000055929ce026d0 main.rb:17:in `block (2 levels) in <main>' ``` This is quite weird. Is the second behavior a bug? Why should other threads suddenly see the constant as "not defined" while it is loading via `require` in the main thread? It's also inconsistent with the first case. I would have thought `require autoload_path` would basically do the same as triggering the autoload of the constant (such as `Foo`). But the results above show they differ. There are many more complex cases for autoload, such as [this spec](https://github.com/ruby/spec/blob/72bd058b5cf0a9d9de5a188052db2fba021…, or how is thread-safety is achieved when methods are defined incrementally in Ruby but the module is defined immediately. Who is knowledgeable about `autoload` and could answer these questions? Could we start a document specifying the semantics? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:120940] [Ruby master Bug#21128] dir.c defines O_CLOEXEC before it is defined by the system.
by collinfunk (Collin Funk) 12 Feb '25

12 Feb '25

Issue #21128 has been reported by collinfunk (Collin Funk). ---------------------------------------- Bug #21128: dir.c defines O_CLOEXEC before it is defined by the system. https://bugs.ruby-lang.org/issues/21128 * Author: collinfunk (Collin Funk) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Building from master commit hash bfcf36fe2243d34a01b5707754be9164849150c9 on GNU/Linux, I see: ``` compiling dir.c In file included from /usr/include/bits/fcntl.h:61, from /usr/include/fcntl.h:35, from dir.c:39: /usr/include/bits/fcntl-linux.h:144:10: warning: "O_CLOEXEC" redefined 144 | # define O_CLOEXEC __O_CLOEXEC /* Set close_on_exec. */ | ^~~~~~~~~ dir.c:26:11: note: this is the location of the previous definition 26 | # define O_CLOEXEC 0 | ^~~~~~~~~ ``` This is because the file includes unistd.h before falling back to 0 for O_CLOEXEC. But glibc requires inclusion of fcntl.h for O_CLOEXEC. Since that file is included after, it is redefined. The C standard leaves the behavior of redefining a variable undefined and it is a simple fix, so I will create a pull request. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:120939] [Ruby master Bug#18605] Fails to run on (newer) 32bit Windows with ucrt
by luke-gru (Luke Gruber) 11 Feb '25

11 Feb '25

Issue #18605 has been updated by luke-gru (Luke Gruber). I'm trying to get ARM64 build of Windows to work so I can debug an issue I was having with a branch I'm working on that's failing in CI with Visual Studio 2022. It would be great if ARM64 Windows was working. Any news on this? ---------------------------------------- Bug #18605: Fails to run on (newer) 32bit Windows with ucrt https://bugs.ruby-lang.org/issues/18605#change-111830 * Author: lazka (Christoph Reiter) * Status: Open * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- 32bit ruby using ucrt has started to fail on newer Windows with "unexpected ucrtbase.dll" -> https://github.com/ruby/ruby/blob/3fb7d2cadc18472ec107b14234933b017a33c14d/… The problem is that ruby depends on ucrt internals and those have apparently changed with newer versions. See https://github.com/msys2/MINGW-packages/pull/10878 and https://github.com/msys2/MINGW-packages/issues/10896 for some background and a potential fix. But ideally ruby wouldn't depend on Windows internals like this. thanks! -- https://bugs.ruby-lang.org/

1 0