Issue #21795 has been updated by Eregon (Benoit Daloze). @kddnewton Thanks for taking a look. The warning and the behavior with parse.y are also concerns, but the more fundamental blocker is correctness: the implementation can return the wrong node. This is not merely an unstable or experimental API: the method can successfully return a valid-looking but incorrect node, and callers have no way to detect that. A comment or warning therefore does not address the correctness problem. This also matches the condition Matz stated in #note-19 : the parser that interpreted the program and the parser returning the syntax tree must be the same, and the methods should not be added until that identity is guaranteed. `node_id` stability across Prism versions is not guaranteed, and in practice [many things work against it](https://gist.github.com/eregon/469e40f447f6edac813a389e7f3f4832). Using an older Prism together with `#syntax_tree` would return the wrong node, like examples below. Future Prism versions are also likely to change `node_id` for some nodes, so merely requiring a recent Prism would not solve the problem. Ruby code compiled by Ruby 4.1 could later be reparsed using a newer Prism gem and return the wrong node. Requiring the exact same Prism version would solve the correctness problem, but would make the API fail whenever a `Gemfile` uses a different `prism` version, including through a transitive dependency. `Prism::Node#node_id` exists since Prism 1.2.0 which was shipped with Ruby 3.4.0. Since then, there have been multiple cases of `node_id` changing, which means returning the wrong node. The following `Prism.find` examples exercise the same cross-version `node_id` lookup on which the `#syntax_tree` prototype relies. These 2 examples demonstrate the issue in the latest Ruby X.Y versions: ### Ruby 3.4.10 / Prism 1.5.3 ```ruby if false p <<-A, %w[j\ i A j] end def a end require "prism" node = Prism.find(method(:a)) p node.class ``` Should return a `DefNode` but returns an `IfNode`, i.e. the wrong node. ``` $ cd prism $ chruby ruby-3.4.10 $ bundle exec rake compile $ ruby -Ilib example.rb Prism::IfNode ``` ### Ruby 4.0.6 / Prism 1.8.1 ```ruby def target = a rescue b rescue c require "prism" node = Prism.find(method(:target)) p [node.class, node.name, node.slice] ``` Should return a `DefNode` but returns a `CallNode`, i.e. the wrong node. ``` $ cd prism $ chruby ruby-4.0.6 $ bundle exec rake compile $ ruby -Ilib example.rb [Prism::CallNode, :c, "c"] ``` ### More cases See [here](https://gist.github.com/eregon/469e40f447f6edac813a389e7f3f4832?permalink_co...) for 5 more cases of returning the wrong node on Ruby 3.4. This is not exhaustive, there are likely more. One could analyze all Prism commits to find every time that `node_id` changed for some node (and that's still limited by the Ruby corpus given to it). Another example is a Prism change [swapped](https://gist.github.com/eregon/469e40f447f6edac813a389e7f3f4832?permalink_co...) the `node_id`s of `ImaginaryNode` and `IntegerNode` for `1i`. This doesn't affect `Prism.find`/`#syntax_tree` but illustrates it's common for `node_id` to change. ### Instability in future Prism changes Those cases are from past Prism versions but it seems rather clear that it will happen again in the future. There are many factors for that, detailed [here](https://gist.github.com/eregon/469e40f447f6edac813a389e7f3f4832?permalink_co...). I'll list them here to give an idea: * `node_id` is actually the allocation order in Prism, so allocating nodes in a different order changes `node_id` * Temporary nodes also consume IDs * An ordinary parser fix changed the number of nodes * `ItParametersNode` added an allocation * `ShareableConstantNode` added a wrapper * `ConstantPathNode` lost a child node * Nodes that are especially likely to change: `ImplicitNode`, `ImplicitRestNode`, `ShareableConstantNode`, `ItParametersNode`, `NumberedParametersNode`, etc. ---------------------------------------- Feature #21795: Methods for retrieving ASTs https://bugs.ruby-lang.org/issues/21795#change-118334 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- I would like to propose a handful of methods for retrieving ASTs from various objects that correspond to locations in code. This includes: * Proc#ast * Method#ast * UnboundMethod#ast * Thread::Backtrace::Location#ast * TracePoint#ast (on call/return events) The purpose of this is to make tooling easier to write and maintain. Specifically, this would be able to be used in irb, power_assert, error_highlight, and various other tools both in core and not that make use of source code. There have been many previous discussions of retrieving node_id, source_location, source, etc. All of these use cases are covered by returning the AST for some entity. In this case node_id becomes an implementation detail, invisible to the user. Source location can be derived from the information on the AST itself. Similarly, source can be derived from the AST. Internally, I do not think we have to store any more information than we already do (since we have node_id for the first four of these, it becomes rather trivial). For TracePoint we can have a larger discussion about it, but I think it should not be too much work. In terms of implementation, the only caveat I would put is that if the ISEQ were compiled through the old parser/compiler, this should return `nil`, as the node ids do not match up and we do not want to further propagate the RubyVM::AST API. The reason I am opening up this ticket with 5 different methods requested in it is to get approval first for the direction, then I can open individual tickets or just PRs for each method. I believe this feature would ease the maintenance burden of many core libraries, and unify otherwise disparate efforts to achieve the same thing. -- https://bugs.ruby-lang.org/