Issue #21795 has been updated by Earlopain (Earlopain _).
Concretely, this means either integrating the Prism repository into ruby/ruby
I don't think that would be a very good solution, prism is not only the parser as used by CRuby. It has bindings to other languages (rust, javascript, java). Also the various translators for previous ruby syntax parser gems. Then there's integration with other other runtimes like jruby and truffleruby that also live in prism. These are all equally tied to the prism version, same as CRuby. As an example the java integration with jruby/truffleruby has seen much activity recently. It is good that this is not happening in ruby/ruby since it's not relevant and also would make it more difficult for them.
To add a personal note, I find it structurally unnatural that the parser, the component that defines the language's syntax, is primarily developed outside ruby/ruby
The C library does not have much connection to ruby (if you ignore the one big point that it is a parser for the syntax) and can exist without it. It's one of the main reasons why it sees such big adoption. It's true that they are tightly integrated and don't make sense in isolation but it's not necessary to drive development exclusively in ruby/prism. For me it is a preference since I have no permissions on ruby/ruby but it is also more managable since it's a smaller project overall. Ruby also does not run all the tests, either because they rely on external gems like `parser` or because they just aren't synced (intentionally). But especially changes that tweak behaviour in some way tend to be done in ruby/ruby first and later synced back instead of the other way around (failing syntax tests, prism_compile.c changes, etc.) ----- Anyways, I don't think moving prism entirely into ruby/ruby would really change anything. The main problem is the mismatch between gem and standard version. You can move development into ruby/ruby but the code is already synced anyways and users can still make use of prism the gem, which gives you the same problem. As long as the prism version that ruby ships with is not used (or any other of the proposed solutions), you do not gain much from it. And solving it that way doesn't require such a drastic change. In the end you need to integrate _something_ but there's nothing stopping you from doing that today already. It would only really work if there is no prism gem that users can cause a mismatch with, so to me it sounds more like you are arguing for `Ruby::Node` instead.
Either way, I'm not sure when I would recommend using Ruby::Node, because it seems like it would always be an out-of-date version of Prism::Node.
It would fill the gap where `ripper` is currently used. It's always exactly what ruby uses and it's clear that there's use-cases for it, especially in connection with `node_id`. Not many would need it, prism the gem is plenty for majority of the cases but it would indeed be very helpful for usage and exposure in ruby itself. It goes back to https://bugs.ruby-lang.org/issues/21618 where I was happy with how it is handled when using the gem but with ruby internals it is not always good enough. --- More generally about the `node_id` mismatch from @eregon. I haven't checked all the cases but it looks like they are all the result of one bug or another in prism where it misparsed the input (some examples are also syntax invalid in earlier prism versions, so I don't think they should be part of the list). I'm not saying that `node_id` should be considered stable or anything. Just practically it rarely happens, even less so as prism continues to mature and never on code that people actually write. Of course, `node_id` cannot change for any other reason or risk breaking things. ---------------------------------------- Feature #21795: Methods for retrieving ASTs https://bugs.ruby-lang.org/issues/21795#change-117003 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- I would like to propose a handful of methods for retrieving ASTs from various objects that correspond to locations in code. This includes: * Proc#ast * Method#ast * UnboundMethod#ast * Thread::Backtrace::Location#ast * TracePoint#ast (on call/return events) The purpose of this is to make tooling easier to write and maintain. Specifically, this would be able to be used in irb, power_assert, error_highlight, and various other tools both in core and not that make use of source code. There have been many previous discussions of retrieving node_id, source_location, source, etc. All of these use cases are covered by returning the AST for some entity. In this case node_id becomes an implementation detail, invisible to the user. Source location can be derived from the information on the AST itself. Similarly, source can be derived from the AST. Internally, I do not think we have to store any more information than we already do (since we have node_id for the first four of these, it becomes rather trivial). For TracePoint we can have a larger discussion about it, but I think it should not be too much work. In terms of implementation, the only caveat I would put is that if the ISEQ were compiled through the old parser/compiler, this should return `nil`, as the node ids do not match up and we do not want to further propagate the RubyVM::AST API. The reason I am opening up this ticket with 5 different methods requested in it is to get approval first for the direction, then I can open individual tickets or just PRs for each method. I believe this feature would ease the maintenance burden of many core libraries, and unify otherwise disparate efforts to achieve the same thing. -- https://bugs.ruby-lang.org/