[ruby-core:121142] [Ruby master Misc#21154] Document or change Module#autoload?
Issue #21154 has been reported by fxn (Xavier Noria). ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). I would like you to learn how to write a bug report.
the second line could evaluate to `nil`.
Please write a self-contained reproducible example that actually does that, not just say “could”. Then write down the results you expect to get with that code, and the actual results. Also, I want you to write the background (what you want to do finally), and why the current behavior is troubling for your purpose. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112097 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh).
Should that second line always return "foo"?
No, not necessarily. The document says "if name is registered as autoload". And the document of Kernel#autoload says "If const is defined but not as autoload, does nothing." Thus, there is no guarantee that your second line will always return `"foo"`. I assume you have https://bugs.ruby-lang.org/issues/21035#note-3 in mind. IMO, in your case, the situation could be considered equivalent to the “If const is defined but not as autoload, does nothing.” case. Therefore, I don't think it is surprising that `autoload?` returns `nil`. So should the documentation be updated? I personally think that is not necessary either. Documentation is documentation, not a spec. I think it is more useful to have a simple description of typical behavior and usage. Currently, I feel the document is good enough. I admit people may have different opinions here, though. I think what you should do is not arguing about consistency with the documentation, but rather briefly explaining your purpose and why the current behavior is troubling against it. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112098 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Hi @mame, thank for your reply.
No, not necessarily. The document says "if name is registered as autoload". And the document of Kernel#autoload says "If const is defined but not as autoload, does nothing." Thus, there is no guarantee that your second line will always return "foo".
That is right. However, if we have a file `foo.rb` ```ruby autoload :Foo, 'foo' p autoload?(:Foo) ``` That prints `nil` if you `ruby -I. -e 'require "foo"'`. So, the constant is not defined, the autoload is registered, and you get `nil`. This does not agree with the documentation, do you agree? You may say "circularity!". And I'll say, cool, but undocumented.
I think what you should do is not arguing about consistency with the documentation, but rather briefly explaining your purpose and why the current behavior is troubling against it.
I am not arguing, I believe my tickets cannot be more respectful, formulated as questions, and leaving the door open to an explanation of how things work that is not documented. You do not always come to ask for questions with a use case in mind. I do use `autoload?` and this edge case affects me, yes. But what I am trying to accomplish is know Ruby better. I understand you don't want to documente this (we disagree, but that is your call), but you also have to understand that as a Ruby programmer I strive to understand how Ruby works per se, that is my goal. And not being a committer, I cannot by myself know if the mismatches with the docs are intentional, an overlook, or if intentional, which is the mental model that fits that mismatch. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112099 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). You have also take into account that in that example `const_added` is called, `Module#constants` lists `:Foo`. So, the whole thing is kind of inconsistent right now. But as a Ruby programmer, I cannot make sense of it without your help. Maybe the response of the Ruby team is "oh, that is right, `autoload? should be a fast lookup and nothing else!", or "actually not, the way to make all this fit together is to revise such and such APIs". ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112100 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Finally, this ticket was intentionally not categorized as "Bug", but it is "Misc". Because I do not know if there is a bug. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112101 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). @mame Let me put it all together: Consider a script `foo.rb`: ```ruby autoload :Foo, 'foo' p autoload?(:Foo) ``` If we execute that with `ruby -I. -rfoo -e1`, we get `nil`. But this does not agree with the docs: 1. The constant `Foo` does not exist, therefore (according to the docs), the autoload is **registered**. 2. `Module#constants` lists `:Foo`, which is consistent. 3. `Module#const_added` is called with `:Foo`, which is consistent. Now, the docs of `Module #autoload?` say:
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
**since the autoload is registered, I should get a filename back, not `nil`**. My **question** to the Ruby team is: Is the behavior OK and the docs should be updated, or should the implementation be updated instead? My suggestion would be to keep the docs as they are and simplify `autoload?` to be a simple lookup in the constants table. (Hope that explains it better.) ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112102 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). I understood as follows: You want @matz to clarify the behavior around autoload under circular require patterns, but you never want to explain your motivation example. I said in #21035 that at the last dev meeting, the decision was made not to discuss anything by using code examples in the circular require pattern. You are requesting that we discuss the same agenda item again with little new material. Frankly, such a request would only waste the time of the dev meeting and is not productive. I don't understand why you are so concerned about circular require patterns. This is just my imagination, but you, the author of the zeitwerk, cannot avoid the presence of circular require patterns in the user code? If so, for example, does it help you by issuing a warning or error for autoloading that would result in a circular require pattern? ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112103 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). @mame I am showing a registered autoload whose `autoload?` does not return a file name. If proves the documentation is wrong. That is the point I am trying to make. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112104 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). It may also help to know that I am writing a book about Ruby constants, so it would be helpful to know if the documentation is wrong, or the implementation is wrong. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112105 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). @mame I don't want anyone to waste their time, if you believe discussing this is so, please feel free to close. I opened this ticket because the discussion went on in the previous one and @eregon, who is a Ruby committer, suggested to do so. Why I don't show source code? Because this is not a ticket motivated by a need I have. Zeitwerk has been living with this edge case for 6 years. I am fine. I _thought_ that pointing out that the API is not consistent would be of your interest _per se_. Resolutions could be several: * Oh, thank you, `autoload?` is OK, we could document it checks features. * Oh, thank you, `autoload` should be the one checking features, actually, the piece that is not right here is `autoload`. * Oh, thank you, `autolad` is fine, but `autoload?` should return the file name, let's fix that. But we do not seem to be aligned. It is OK, please, feel free to close. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112106 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh).
Why I don't show source code? Because this is not a ticket motivated by a need I have.
I see, you really do not have a use case. I finally understand. I had misunderstood that you were hiding your use cases and somewhat challenging us. (I am not sure if my English wording is appropriate here, sorry if it makes a wrong sense.) Sorry for the frustration. Let me explain why I am eager to hear the use cases. In principle, all proposed changes to Ruby are considered based on the use cases that would require the change. Even if it is just an inconsistency, we force ourselves to assume use cases that could be affected by the inconsistency and then consider how (and whether) to fix it. Therefore, ticket discussions at the dev meetings begin with an understanding of the use case. Without a background explanation in the ticket, a great deal of time is spent to try identifying use cases. Still, it is impossible to cover all the use cases in a limited amount of time, so we could reach incomplete conclusions. We sometimes end up finding a different use case than the one the proposer has in mind, leading to a conclusion that the users, including the proposer, do not want. If a reasonable use case cannot be found, we need to ask the proposer to clarify, which takes time. Or we may choose WONTFIX, or make a random choice based on matz's intuition, which is also often not the result desired by the proposer. You may think that Ruby committers has the "answer" to the question you have and you want to just get it out of us. In fact, we have no "answer". At least I do not have. I believe matz does not have it either for your question yet. We need to decide how (and whether) to fix it by using the use cases. If there is no particular use case that is troubling you, then we tend to choose "don't touch it" or just rely on matz's intuition. As for autoload, there is a long history of subtle and repeated fine-tuning based on problems encountered in real-workd use cases. Therefore, I feel strong resistance to changing something based on intuition alone. I am afraid that by making a bad change, Zeitwerk stops working, which will affect many people. This is the reason why I insisted on asking you, the author of Zeitwerk, for background on this issue. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112109 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Oohhh, now I understand! Thanks for explaining this. I am also not familiar with how dev meetings are run, so we were coming from different angles to this and I guess we were not quite understanding each other in the threads. Let me think if it is worth explaining how usage of `autoload?` in Zeitwerk is affected by this (the gem has a workaround to deal with this behavior). Let me think, I'll write back, thanks again for your clarifications @mame. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112110 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Hi @mame. My main motivation for starting this discussion was to 1) understand Ruby better, 2) bring some inconsistencies to the attention of the Ruby team and ask for clarification (here, I assumed the team knows autoload inside out and matz would instantly know the correct mental model/logic, your reply yesterday helped me understand this assumption is not entirely correct) 3) know what to write in my book. Now that I understand better how dev meetings work and how issues are evaluated, I am not sure the use case in Zeitwerk affected by this behavior is worth a debate. Lastly, let me address this sentence:
I had misunderstood that you were hiding your use cases and somewhat challenging us.
You can be 100% confident I'd never do that. My questions & discussions are in good faith and with an OSS collaboration spirit. I have a huge respect for everybody in the Ruby team. When I wrote Zeitwerk, I told Matz that if he wanted to keep the deprecation of autoload I'd cancel the project, and I meant it. Suggested to Matz in Sarajevo last year that if he wanted to remove autoload from Ruby and think of an alternative, I'd be OK and willing to help. We may agree or disagree in specific technical details, that is normal, but always with good intention! I think we can close, thanks @mame! ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112129 * Author: fxn (Xavier Noria) * Status: Open ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). Status changed from Closed to Assigned Assignee set to mame (Yusuke Endoh) This was discussed at the dev meeting. It is decided to add a sentence to documents of `Kernel#autoload` and `Module#autoload` that prohibits registering a file being loaded. I have created a PR. https://github.com/ruby/ruby/pull/12926 It was also discussed to determine this condition at runtime and raise a warning or exception. However, the overhead of resolving file path when autoload is called may be a problem for programs that set a large number of autoloads, so it is not decided. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112304 * Author: fxn (Xavier Noria) * Status: Assigned * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Thanks @mame! I believe this may not entirely work. That makes sense to be asked if you are thinking about explicit source code doing the autoload. However, in situations in which the autoloads are set dynamically, as it is the case with Zeitwerk, you don't have API to know if a given file is being loaded at the moment. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112310 * Author: fxn (Xavier Noria) * Status: Assigned * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). Status changed from Closed to Feedback @fxn I think it makes sense. Would it help you if such a new API is introduced to determine if a file is currently being loaded? ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112319 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). @mame thanks for considering. I am not sure it would be a good idea in practice for the same reason it was decided to not do that when `Module#autoload`, it would be too costly. Zeitwerk needs to keep track of these edge cases, I call them "inceptions" in the source code ([usage example](https://github.com/fxn/zeitwerk/blob/4cf0ddff474ee81398932b9391a8ee06d9bea00...)). Checking that would be prohibitive, because in a project managed by Zeitwerk everything has an `autoload` call set. Even when eager loading (for consistency, eager loading is a recursive `const_get`, not a recursive `require`). If I ever come with a concrete alternative, I may followup, but meanwhile perhaps we can leave it here. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112322 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Hi @mame. After thinking about this, I have an important use case that I'd like to share, because it conflicts with that documentation patch. I recorded [this video](https://www.dropbox.com/scl/fi/f0oemgyf0jmhzszx6hj0p/autoload_and_require.mp4?rlkey=i8jrt0tgyf09c47h7rl4swh0e&st=ah35tkxw&dl=0) to explain it to you and all the team. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112372 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Hi @mame, did you have a chance to watch the video or discuss it in a dev meeting? ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112632 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). @fxn Sorry for the delay. I finally watched your video. I'm not exactly sure what you meant by "const_get hook", but perhaps you meant "const_added"? I'll assume that's the case and continue. Here's how I understand the issue. Suppose we have a gem with lib/foo.rb and lib/foo/bar.rb, and an app.rb that requires it: ```ruby # lib/foo.rb Zeitwerk::Loader.for_gem.setup module Foo end ``` ```ruby # lib/foo/bar.rb module Foo class Bar end end ``` ```ruby # app.rb require "foo" Foo::Bar ``` In this setup, `Zeitwerk::Loader.for_gem.setup` in lib/foo.rb effectively does something like this: ```ruby # Since lib/foo.rb exists, it sets autoload for ::Foo autoload :Foo, "lib/foo" # Then, using the const_added hook, it sets autoload for Foo::Bar when Foo is defined: class Object def self.const_added(name) if name == :Foo Foo.autoload(:Bar, "lib/foo/bar") end end end ``` This means that lib/foo.rb sets `autoload :Foo, "lib/foo"`, which would violate the rule of my documentation patch. Is this understanding correct? --- If that's the case, I think lib/foo.rb is doing something quite tricky. Let me show a simpler example: ```ruby # test.rb p 1 autoload :Foo, "./test" p 2 class Foo p 3 end ``` Running this code produces the following output: ``` $ ruby test.rb 1 2 1 2 3 3 ``` This shows that `class Foo` triggers the autoload and ends up circularly loading test.rb itself. As a result, the file ends up being evaluated twice. Is this kind of circular loading what actually happens in `Zeitwerk::Loader.for_gem.setup`? If so, is this really the intended behavior? --- By the way, I appreciate your kindness in recording a video, but to be honest, I'm not very good at understanding spoken English. I'd appreciate it if you could explain things in text and using code examples. Also, this month's dev meeting will be held at RubyKaigi, a Japanese conference, and will be shorter than usual, so I don't think we'll have time to discuss this topic. I'm sorry, but I think we'll need to revisit it in the May dev meeting. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112678 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). Aha, when test.rb is required, currently `autoload` does (almost) nothing because test.rb is already being required, so test.rb is never executed twice. ``` $ ruby -r./test.rb -e '' 1 2 3 ``` Also, when I execute `action_cable.rb` directly, I got constant redefinition warnings. This means `action_cable.rb` was indeed evaluated twice :-) ``` $ ruby /home/mame/.rbenv/versions/ruby-dev/lib/ruby/gems/3.5.0+0/gems/actioncable-8.0.2/lib/action_cable.rb /home/mame/.rbenv/versions/ruby-dev/lib/ruby/gems/3.5.0+0/gems/actioncable-8.0.2/lib/action_cable.rb:58: warning: already initialized constant ActionCable::INTERNAL /home/mame/.rbenv/versions/ruby-dev/lib/ruby/gems/3.5.0+0/gems/actioncable-8.0.2/lib/action_cable.rb:58: warning: previous definition of INTERNAL was here ``` I think I understand the problem and can explain it at the May dev meeting. However, I am not sure what is the best solution for this issue. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112679 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Hi @mame!
I'm not exactly sure what you meant by "const_get hook", but perhaps you meant "const_added"? I'll assume that's the case and continue.
Right, I had a lapsus there.
This means that lib/foo.rb sets autoload :Foo, "lib/foo", which would violate the rule of my documentation patch.
Exactly, I could not comply with the new documentation without breaking [hundreds of gems](https://rubygems.org/api/v1/gems/zeitwerk/reverse_dependencies.json). And there are more existing edge cases, I only showed the most important one. Since I know compatibility is something the Ruby team has as a high priority, I believed I had to raise this concern. And, besides these examples, code issuing dynamic autoloads cannot comply with the docs either, because it has no way to check if a given file is being loaded at the moment. But this point is kind of more theoretical, while the point above is very real.
By the way, I appreciate your kindness in recording a video, but to be honest, I'm not very good at understanding spoken English. I'd appreciate it if you could explain things in text and using code examples.
My pleasure. I'll remember that for other tickets. Regarding your remarks about loading the entrypoint by passing it to the interpreter, or to `Kernel#load`, etc. these are [documented](https://github.com/fxn/zeitwerk?tab=readme-ov-file#for_gem) to not be supported: *"Due to technical reasons, the entry point of the gem has to be loaded with Kernel#require, which is the standard way to load a gem. Loading that file with Kernel#load or Kernel#require_relative won't generally work."*
I think I understand the problem and can explain it at the May dev meeting. However, I am not sure what is the best solution for this issue.
Excellent, thank you. The discussion here was related to the predicate `Module#autoload?`, I guess you guys ended up addressing this one by disallowing circularity altogether, but (as you mentioned in some comment), that turns to uncover stuff that perhaps was not foreseen during the discussion. May I suggest restoring the original documentation? Thanks again, hope you folks have a great time in RubyKaigi. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112712 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). I had a quick chat with @akr. `Zeitwerk::Loader.for_gem.setup` identifies its caller file (i.e., the entrypoint file of the gem) by using `caller_locations`. https://github.com/fxn/zeitwerk/blob/67b1396cab1bb517b9a63833417831d648794fc... Because the entrypoint file is already `require`ed, Zeitwerk doesn't have to set autoload on the file. So, it is possible to avoid the circular autoload, and follow the new limitation introduced by my documentation change, right? ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112720 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Perhaps that would work for this exact use case. But it would be ad-hoc and limited, it would not be addressing the conflict at its root cause. As documented, `for_gem` is only a convenience shortcut for a common use case. Gems can use the generic API, instantiate the loader in arbitrary places, and some do. And I have more use cases, like a gem that migrates to Zeitwerk and existing client code has `require` calls for files inside the gem. Nowadays, those calls work. All of this works today, [it even has test coverage](https://github.com/fxn/zeitwerk/blob/67b1396cab1bb517b9a63833417831d648794fc...) for other interpreters. If the change was: "If the autoload is circular, it is ignored and the return value says so", I _believe_ I could modify the gem to take that into account, provided that nowadays I actually have that logic by asking `autoload?` after it. But it was discarded in a previous comment as "too costly" (from memory). But, the key point is you cannot honor what the docs say in `master`. I suspect they have static usage in mind, but are not considering dynamic usage where, as far as I know, you do not have the tools to check if an arbitrary file is being loaded at the moment, so you cannot now if you can set an autoload, and strictly speaking then you cannot set ANY autoload. Let's do this experiment: 1. I'd like to set an autoload for this file dynamically. 2. As per the docs, you cannot do that if the file is being loaded. 3. How can I check that? 4. You cannot (unless I am missing some API). 5. Therefore, I cannot set the autoload. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112721 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by akr (Akira Tanaka). autoload is virtual require. (like virtual memory) In general, a user of require or autoload expects the specified library is usable after require or autoload. However, it is not hold when a user invokes require or autoload a library currently loading. It is simply impossible. It is nice to document it and applications does not depend the autoload behavior in the situation. I think it is possible to avoid that by checking caller_locations. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112722 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). akr (Akira Tanaka) wrote in #note-28:
I think it is possible to avoid that by checking caller_locations.
As for autoload with a full path, it would be easy (and that's the case for Zeitwerk, I think). In general, autoload with a relative path needs to resolve the path to absolute one. This resolution may be too costly to do for all autoload calls by default. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112723 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). mame (Yusuke Endoh) wrote in #note-29:
As for autoload with a full path, it would be easy (and that's the case for Zeitwerk, I think).
Correct, Zeitwerk only sets autoloads with absolute paths (that may contain symlinks).
In general, autoload with a relative path needs to resolve the path to absolute one. This resolution may be too costly to do for all autoload calls by default.
And that is what the predicate `autoload?` is doing, which is the origin of this ticket. I don't even know if that is intentional of it is implementation leaking (I'd expect `autoload?` to simply return the configured path or `nil` right away, and have the circularity check when the autoload is hit.) ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112724 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). fxn (Xavier Noria) wrote in #note-30:
Correct, Zeitwerk only sets autoloads with absolute paths (that may contain symlinks).
Thank you. So, Zeitwerk is able to avoid the limitation clarified in this ticket, right?
In general, autoload with a relative path needs to resolve the path to absolute one. This resolution may be too costly to do for all autoload calls by default.
And that is what the predicate `autoload?` is doing, which is the origin of this ticket.
Please note that the performance of `Kernel#autoload?` is not considered important. The purpose of `autoload` is to speed up boot time by deferring `require`. One typical use case involves configuring many `autoload`s but exiting the program without actually triggering most of them. Therefore, the performance of `Kernel#autoload` is important. On the other hand, metaprogramming features like `Kernel#autoload?` are not expected to require high performance.
I don't even know if that is intentional or it is implementation leaking
The current behavior of `autoload?` and `const_defined?` is intentional. These methods return truthy values when the constant is considered available in the current context. ```ruby # main.rb autoload :Foo, "./foo.rb" # In the context between the autoload setup and the point when it is triggered, # the constant Foo is considered (virtually) available. # So autoload? and const_defined? return truthy values. p autoload?(:Foo) #=> "./foo.rb" p Object.const_defined?(:Foo) #=> true # Trigger the autoload Foo ``` ```ruby # foo.rb # In the context between triggering the autoload and the actual definition of the constant, # the constant Foo is not considered available. # So autoload? and const_defined? return falsy values. p autoload?(:Foo) #=> nil p Object.const_defined?(:Foo) #=> false module Foo end ``` To achieve this behavior, `Kernel#autoload?` checks whether the autoload for the corresponding constant has already been triggered. In the current implementation, this check is effectively done by seeing whether the associated `require` has been called or not. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112756 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria).
So, Zeitwerk is able to avoid the limitation clarified in this ticket, right?
Maybe, but do you think that any Ruby program setting autoloads should manually check `caller_locations` and emulate what `autoload?` does? And that is not only about dynamic calls, you should do it for static calls too. Think about this. You ship a gem like this: ```ruby # my_gem.rb module MyGem autoload :M, ... end # my_gem/m.rb module MyGem module M autoload :X, ... # (1) end end # my_gem/m/x.rb module MyGem module M X = 1 end end ``` Now, Bundler loads `my_gem.rb` when the project boots, and the user does in their application: ```ruby require "my_gem/m/x" ``` When (1) is reached, the file is being required, but that is out of the control of the author of the gem. So, in (1) it should manually check `caller_locations` to comply with the documentation?
The current behavior of autoload? and const_defined? is intentional. These methods return truthy values when the constant is considered available in the current context.
Wouldn't you prefer to leave things as they were, and document this? Not having this documented was a problem for me, because I expected the predicate to return what the docs say. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112757 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Let me add that the example above is not made up. I have seen many Rails applications with `require` calls for autoloadable files. Gems may do refactors that switch to `autoload` internally, but that is internal and existing client code may have working `require` calls. We said that it would be too costly for `autoload` to perform the `autoload?` check and return a flag, but my conclusion is that the current documentation implies code registering has to do that manually anyway. Because, in order to comply with the new docs, you have to make sure by hand the autoload can be set. However, if we restore the original documentation, you do not need to do that check, autoload logic takes care of that either in `autoload?` or, if not invoked, when the autoload is triggered anyway. But would be nice to have additional docs in `autoload?` and `const_defined?`, at least. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-112761 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh).
Think about this. You ship a gem like this:
@fxn Thank you for the very interesting example. It became a topic of discussion for @akr, @naruse, and me for a total of 4 hours over two dev meetings. With a slight modification to your example, we can create a case that deadlocks. ```ruby # my_gem.rb module MyGem autoload :M, "./my_gem/m" end # my_gem/m.rb module MyGem module M autoload :X, "./my_gem/m/x.rb" sleep 1 X # (1) fires the autoload of my_gem/m/x.rb end end # my_gem/m/x.rb module MyGem sleep 1 module M # (2) fires the autoload of my_gem/m.rb X = 1 end end # main.rb require "./my_gem" Thread.new { MyGem::M } require "./my_gem/m/x" ``` ``` $ ruby main.rb /home/mame/work/ruby/my_gem/m/x.rb:4:in '<module:MyGem>': No live threads left. Deadlock? (fatal) 2 threads, 2 sleeps current:0x00005fda1937c900 main thread:0x00005fda19088ad0 *snip* ``` When `Kernel#require` attempts to require a file that is currently being required by another thread, it waits for the prior `require` to complete. In this example, `my_gem/m.rb` and `my_gem/m/x.rb` end up trying to require each other simultaneously, leading to a deadlock. --- At present, it's fair to say that Ruby doesn't well support to require/autoload mutually-dependent files from multiple entrypoints. In this specific case, to put it simply, users must not do `require "my_gem/m/x"`. We know this limitation is too strict. We explored ways to allow `require "my_gem/m/x"`: 1. Can we improve the Ruby interpreter to prevent deadlocks in the above case? 2. Can we document guidelines for what content is safe to write in files that are to be required (i.e., content that we can safely require from multiple entrypoints)? --- The simplest approach would be to introduce a single global lock for `require`, limiting it to at most one thread at a time. This would prevent deadlocks. However, if one thread requires a file that enters an infinite loop, `require` in other threads would be blocked indefinitely. We haven't come up with a better locking mechanism. --- It would be great if we could document clear guidelines like "if users write Ruby code this way, it's safe to require/autoload from multiple files," but we haven't found such clear guidelines. In the deadlock example above, the cause is accessing `X` within `my_gem/m.rb`, which triggers the autoload of `my_gem/m/x.rb`. Therefore, one might think that avoiding the immediate triggering of an autoload that one has set up would be sufficient. However, a deadlock still occurs if `my_gem/m.rb` directly requires `my_gem/m/x.rb`, like this: ```ruby # my_gem/m.rb sleep 1 require "./my_gem/m/x" # This directly requires the other file module MyGem module M end end ``` This `my_gem/m.rb` looks innocent to me, so what is wrong? It is a bit surprising that `module M` in `my_gem/m/x.rb` fires the autoload of `M`. One may think Ruby should stop doing that. However, it would not be rare to refer to `M` in `my_gem/m/x.rb`, such as, `M::Y`. What guideline can users follow to ensure that such deadlocks do not occur? Do you have any ideas? ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-113630 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by mame (Yusuke Endoh). I'd like to share a simplified example that demonstrates the autoload/require issue. ```ruby # my_gem.rb module MyGem autoload :M, "my_gem/m" # (2) end # my_gem/m.rb module MyGem module M autoload :X, "my_gem/m/x" # (5) SingletonX = X.new # (6) end end # my_gem/m/x.rb module MyGem module M # (4) class X end end end ``` Running this with the following code results in `uninitialized constant MyGem::M::X (NameError)`. ```ruby # main.rb require "my_gem" # (1) require "my_gem/m/x" # (3) ``` ``` $ ruby -I. main.rb /home/mame/work/ruby/tmp/my_gem/m.rb:5:in '<module:M>': uninitialized constant MyGem::M::X (NameError) SingletonX = X.new ^ from /home/mame/work/ruby/tmp/my_gem/m.rb:2:in '<module:MyGem>' from /home/mame/work/ruby/tmp/my_gem/m.rb:1:in '<top (required)>' from /home/mame/work/ruby/tmp/my_gem/m/x.rb:2:in 'Kernel#require' from /home/mame/work/ruby/tmp/my_gem/m/x.rb:2:in '<module:MyGem>' from /home/mame/work/ruby/tmp/my_gem/m/x.rb:1:in '<top (required)>' from base.rb:2:in 'Kernel#require' from base.rb:2:in '<main>' ``` Here is the sequence of events: * (1) in `main.rb`: `require "my_gem"`. * (2) in `my_gem.rb`: Sets autoload for `MyGem::M`. * (3) in `main.rb`: `require "my_gem/m/x"`. * (4) in `my_gem/m/x.rb`: The definition `module M` triggers the autoload for `MyGem::M`. * (5) in `my_gem/m.rb`: Sets autoload for `MyGem::M::X`. * (6) in `my_gem/m.rb`: `SingletonX = X.new` triggers the autoload for `MyGem::M::X`. However, since `my_gem/m/x.rb` is already in the middle of being required (from step 3), this is treated as a circular require, so Ruby does nothing. * As a result, `MyGem::M::X` remains undefined, raising `uninitialized constant MyGem::M::X`. What is wrong here? I believe the issue arises from performing a "cherry-picking" require (requiring internal files directly) on a library that relies on `autoload`. Therefore: * Users must not perform cherry-picking requires (e.g., `require "my_gem/m/x"`) unless the library author explicitly permits it. * If a library author intends to support cherry-picking requires, they must not use `autoload`. It is important to note that simply replacing `autoload :X, "my_gem/m/x"` with `require "my_gem/m/x"` is not sufficient to fix this. The issue persists unless the parent `autoload :M, "my_gem/m"` is also changed to `require "my_gem/m"`. It is a bit difficult to determine exactly which `autoload` definitions need to be replaced to avoid the issue. I believe the following is a good rule of thumb: Libraries that wish to support cherry-picking requires should not use `autoload` entirely. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-115454 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). Thanks for the example @mame. In my view, this is more a problem of circularity than a problem of autoload vs require. When you are loading a file, a require is executed and loads a file that needs a constant that was still not defined. You can do that with requires too. Same with cherry-picking in projects that do not use autoload. Users cannot cherry-pick arbitrary files without authors saying that is fine. Because requires are tricky to setup and normally the author loads things in a way that is consistent. Easy example is ```ruby # a/b/c.rb module A::B::C ``` In your project, you control how things are loaded and you know that when `a/b/c.rb` is loaded `A::B` is defined. End-users cannot go and happily load that file directly and expect that to work. So, my reflection is that cherry-picking without explicit docs allowing it, is in general not supposed to work. end ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-115476 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
Issue #21154 has been updated by fxn (Xavier Noria). BTW, I do not know if you know it, but while I started working on Zeitwerk to fix autoloading in Rails, I changed my mind in the mid of it and decided to do something more generic that supports multiple autoloaders in the same process managing different source trees, and usable by any Ruby project. And the reason was precisely that: Getting the requires right in a project of a decent size is hard. In particular, cherry-picking _in your own project_ is hard. With Zeitwerk, you and client code can refer to anything, anywhere. ---------------------------------------- Misc #21154: Document or change Module#autoload? https://bugs.ruby-lang.org/issues/21154#change-115480 * Author: fxn (Xavier Noria) * Status: Feedback * Assignee: mame (Yusuke Endoh) ---------------------------------------- The documentation of `Module#autoload?` says
Returns filename to be loaded if name is registered as autoload in the namespace of mod or one of its ancestors.
Cool, but in the following snippet ```ruby autoload :Foo, 'foo' autoload?(:Foo) ``` the second line could evaluate to `nil`, and this does not seem to agree. I just registered an autoload, therefore (according to the documentation) I should get "foo" back in line 2. I'd like to ask for clarification from the Ruby team: 1. Is the documentation complete? Should that second line always return "foo"? 2. If the answer is no, which is the logic missing in the docs? Thank you! -- https://bugs.ruby-lang.org/
participants (3)
-
akr (Akira Tanaka) -
fxn (Xavier Noria)
-
mame (Yusuke Endoh)