Issue #19772 has been updated by Eregon (Benoit Daloze). FWIW I don't think renaming YARP at this point makes sense, now that the `yarp` gem [is owned](https://rubygems.org/gems/yarp) by @kddnewton and after the [detailed blog post](https://railsatscale.com/2023-06-12-rewriting-the-ruby-parser/). Everyone calls it YARP, renaming now would feel silly. If/when YARP would become the default then it would just be `RubyVM::InstructionSequence.compile`. So I think option 2 is the best and makes it easy to experiment with a different default. The implementation should be part of `compile.c`, so IMO it also makes sense the API is under `RubyVM` and not `YARP`. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-103915 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by rubyFeedback (robert heiler). I believe YARP.compile() is a bit of a strange name since it is so specific to YARP. If possible I would perhaps recommend something like the other suggestion: RubyVM::InstructionSequence.compile(yarp: true) Or perhaps another name. Residing under the RubyVM namespace may make sense, so at the least I think this is better than YARP as primary name. You could even just pass in a symbol such as: RubyVM::InstructionSequence.compile(:yarp) (I do not know which alternatives may exist, so please keep in mind that I am solely commenting from an API point of view.) The reason why I come to a similar conclusion as Eregon is mostly because I think this may seem cleaner and may allow for alternative implementations/models too. See the older syck gem, and psych, where ideally the user could just do: require 'yaml' But change the engine if necessary. I used that for a while, before I transitioned into psych completely some years ago. So on the API idea I concur with Eregon; I also think it may be useful if other ruby implementations integrate YARP eventually (jruby, natalie and so forth) or the feature/functionality, however it may be applicable (I am not very familiar with the jruby/truffleruby implementation but they try to stay as compatible as possible with MRI, with a slight implenentation delay normally). ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-103921 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by jemmai (Jemma Issroff). nobu (Nobuyoshi Nakada) wrote in #note-5:

I don't think "YA-" names nice in general, unless for development code name.

Agreed with you that `"YA-"` prefix is overloaded. Is your preference then something like `YARP.parse.compile` (or a different name)?

As for `YARP.compile`, "a parser compiles code" sounds strange a little to me. It might be better that `YARP.parse` will return AST and the AST has `compile` method to return ISeq.

I have added this topic to the dev meeting agenda and hope we can resolve it there. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104144 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by ko1 (Koichi Sasada). If the motivation of this API is testing YARP compilation to ISeq, I agree to introduce `ISeq.compile_yarp` to Ruby 3.3 and remove it from Ruby 3.4. (Note that the name `YARP` is renaming one) Because if YARP is introduced as the MRI parser, we don't need it (compile and compile_yarp are same). ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104192 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by mame (Yusuke Endoh). We discussed How to merge YARP to Ruby 3.3 at the dev meeting. Here is what @matz approved: * MRI will use parse.y by default. Using an option (not concretely decided yet), it will use the built-in YARP. * We introduce `::Ruby` module. * This namespace is owned and predefined by the Ruby interpreter. * Any gem must not define any constants under the namespace. * Built-in YARP can use used from Ruby in `Ruby::Parser`. * `Ruby::Parser` is not loaded by default. `require 'ruby/parser'` will define `Ruby::Parser`. * `lib/ruby/parser.rb`, etc. will be part of the standard library (it will not be part of default/bundled gems). * `Ruby::Parser` should not be defined/modified by any gem (= gem cannot replace `Ruby::Parser`). * yarp/prism gem will use a different namespace than `Ruby::Parser`. * We haven't decided whether to bundle the yarp/prism gem. * The name of the gem can be either yarp or prism. * yarp's implementation for multi-version support is up to yarp maintainers. (Anyone who was at the meeting, please point out if I'm wrong) ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104270 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze). Why introduce `Ruby::Parser` as a different namespace if it does the same as `YARP`? It doesn't seem a good idea to have 2 APIs for the same thing (memory overhead, defining every node class twice, etc) and in fact it is problematic on JRuby & TruffleRuby where YARP uses FFI and so 2 APIs would mean 2 librubyparser.so loaded in the same process which would cause symbol conflicts. The solution to this is simple, single YARP version in a process by being a gem and single API: YARP. And also Ruby::Parser is extremely confusing with existing [RubyParser](https://github.com/seattlerb/ruby_parser). ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104274 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze).

In summary the main thing I would like to keep in mind is that the Ruby API will be used by tool developers, and we should make it as easy as possible for them to maintain the tools.

Exactly. I believe for this reason the best and simplest would be: * Only copy YARP C files to CRuby, to be used (optionally) as the parser for CRuby. Then it's clear, the part of YARP in CRuby core is just for parsing Ruby code to compile to bytecode, not a Ruby API. * Add yarp as a bundled gem, so it's already there when installing Ruby. It's just a gem and if tools need a newer version, it just works. That would avoid any confusion about where the Ruby API of YARP is and the various implications, as discussed extensively recently on CRuby Slack. It also solves the symbol conflicts issue I mentioned in my previous comment. Longer-term there is the question of using YARP to implement Ripper. I think that could just `require "yarp"` behind the scenes and so use the bundled gem. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104277 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by k0kubun (Takashi Kokubun).

Going forward, the only people that will be using the Ruby API of the parser will be tool developers. (For example, formatters like Syntax Tree, linters like Rubocop, static analysis tools like Steep.)

Template engines like Haml that parse Ruby code for optimization purposes, which are executed in production code and probably not considered a "tool". Do you think these gems should keep using Ripper forever?

At this point after the release, we'll begin developing on the 3.4 branch. If consumers of the gem want to parse code with 3.3's behavior, they will be able to use the old version of the gem. We will have the same maintenance cycle as CRuby does.

I think it's problematic for some use cases that you cannot use a parser for the running Ruby version when you have YARP for an old Ruby version in Gemfile. For example, * You're parsing Ruby 3.3 code with some tool (e.g. Rubocop) that is running on Ruby 3.4. When you parse Ruby 3.3 code, the parser should be a parser for Ruby 3.3. But when you debug the process with `binding.irb` or debug.gem, the parser used by them should be a parser for Ruby 3.4 because the code runs on Ruby 3.4. * Similarly, the Haml template engine should always use a Ruby 3.4 parser for parsing Ruby code for optimizing templates even if Gemfile specifies YARP for an old Ruby version for tooling purposes. Essentially, it means that you shouldn't use YARP for parsing Ruby code that is going to be `eval`-ed in the same process because the syntax that yarp.gem parses is not guaranteed to be the same as what `eval` parses. parser.gem's `Parser::CurrentRuby` supports this, `Ripper` supports this, Matz's `Ruby::Parser` idea supports this, but this yarp.gem design doesn't. If we ship the current yarp.gem as is and don't ship `Ruby::Parser`, I will keep using `Ripper` for those use cases because `Ripper` is the only Ruby API that is guaranteed to be compatible with the current Ruby version. --- I can think of the following options that can support the use cases of parsing `eval`-ed Ruby code. 1. [current] Keep maintaining `Ripper` as the only Ruby API that wraps the parser used by the interpreter. 2. [matz's idea] Provide `Ruby::Parser` that wraps the parser used by the interpreter with YARP-like API (but with different constant names). 3. yarp.gem is shipped with a parser for a single Ruby version, and `required_ruby_version` allows only that (minor) version. 4. yarp.gem is shipped with a parser for a single Ruby version that could be old, but also has another class that wraps YARP's C API the interpreter provides, which can be used like parser.gem's `Parser::CurrentRuby`. * C code and Ruby code could be provided by different YARP versions, so I'm not sure if it's easy to maintain. 5. yarp.gem is shipped with parsers for multiple Ruby versions, and has a class that selects a parser for the current Ruby version, which works like parser.gem's `Parser::CurrentRuby`. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104284 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze). @kddnewton

There are a couple of technical hurdles we will need to overcome if we introduce additional fields between versions, but I believe these will be easily overcome.

How? That seems near impossible to me, without having 2 versions of the C code (*.h and *.c and `api_node.c`) & Ruby code too. If there is a single `api_node.c` it would use the struct definitions in the gem for the result of `yp_parse` in interpreter, which would break as soon as there is any field change in any of the node structs. E.g. if there is a new field in the yarp gem then it would read out of bounds. And if we think about serialization I think it's more obvious it cannot work without duplicating all files related to serialization & deserialization and their dependencies (so basically every file of YARP), since it would be different YARP::VERSION and the check when deserializing would (correctly) fail. --- Here is how I see those 5 solutions: 1. not really a solution because the Ripper API is very inconvenient to use and longer-term Ripper might use YARP, and/or be deprecated. 2. this requires quite a lot of work and magic to adapt the namespace. It feels rather messy, but probably feasible. Obviously there is the overhead of defining every Ruby node class twice, etc. 3. this is the simplest solution. It might be the best because of that. 4. I think this cannot work. It would need to duplicate every file but also if you install an older YARP gem how would it find the {.c,.h,.rb} files for current Ruby? It seems impossible. 5. this is the most generic solution and would allow to fully replace the `parser` gem, and make adoption much easier for any tool using the `parser` gem currently. It means more `if` in the codebase but OTOH a single branch to maintain. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104335 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze). kddnewton (Kevin Newton) wrote in #note-19:

We already have two copies of the C code. We have the vendored one that is copied into CRuby/JRuby/TruffleRuby, and we have the one that is inside the gem. The symbols are purposefully hidden so that they can be matched up with the correct caller. The struct definitions would be fine, because they would each belong to their respective file sets.

Right, but then we need to fully use that copy and that also means we need a copy of `api_node.c` and `extension.c` (otherwise it's mixing the node structs). And on JRuby/TruffleRuby the C extension is not used, so then we need a copy of Ruby files too (the Ruby files contain the logic to deserialize). Even on CRuby we would need a copy of Ruby files because of different node names/fields/visitor methods/etc between versions. And that implies a different namespace to avoid redefinition warnings. So basically that's option 2 and 2 full independent copies of YARP files (one in the Ruby impl, one in the gem), with different namespaces (the name of the second namespace could be something else than `Ruby::Parser` of course).

When translating into Ruby or serializing, we could handle it any number of ways. We could introduce new node types whenever we add a field, which would deserialize into the previous node types but with nil values. We could add explicit markers saying "new field here, skip on previous versions". There are plenty of ways around this.

I don't think any of these would work well in practice. Mixing files of two YARP versions is a recipe for segfaults and complicated hard-to-debug errors. And the yarp gem being used might be newer or older, e.g., we cannot assume a node type number is free. The serialization and deserialization need to match exactly (the version check there is not optional, it crashes without), and the same for C structs and `api_node.c`. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104342 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by kddnewton (Kevin Newton). From this thread and others, this is my understanding of the requirements: * The `YARP` name is not good, since we don't want `YA`- prefixes in production code. * We could like to be able to improve the Ruby API without having to go through the Ruby release cycle. * We would like for users to be able to parse Ruby code as various versions of Ruby have previously parsed Ruby code. * We would like for users to be able to parse Ruby code exactly as the current version of Ruby is parsing Ruby code. Here are the kinds of changes we would like to be able to make outside of the Ruby release cycle: * We would like to be able to add fields to the various nodes. * We would like to be able to add flags to the various nodes. * We would like to be able to change the Ruby API to provide additional functionality/classes. Here are the steps I would like to take to address these requirements: 1. I would like to rename the `YARP` gem to `Prism`. 2. I would like to change the way we build Ruby objects from both the C API and the serialization API. Currently when we build Ruby objects from these APIs, we call `rb_class_new_instance` directly with the fields coming back from the parser. Instead, I will change this to call a Ruby method (either `XXXNode.new` or `YARP.build_xxx_node`) that will instantiate those objects. 3. When we add fields to nodes, we will patch the factory method to handle the various combinations coming back from the various APIs. Those factory methods will check the target Ruby version and handle the fields appropriately for that version of Ruby. 4. When we add flags to nodes or change the Ruby API to provide additional functionality, nothing should have to change. 5. I would like to add `YARP.parse_native` which will take advantage of the previous bullets here to specify that the target Ruby version should be exactly the version of Ruby that is currently running. With these changes in place: * We will have a better name. * We will be able to upgrade the Ruby API in non-breaking ways without going through a Ruby release cycle. * Users will be able to install one version of the gem and be able to parse as previous versions of Ruby did. * Users will be able to install one version of the gem and be able to parse as the current version of Ruby does. * We will not be able to make breaking changes to the C parser. (We don't want to do this outside the Ruby release cycle anyway.) Does this approach sound good to everyone, or are there any other requirements that I have missed? ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104398 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze). Also in some cases the Ruby version to parse might not be known early or be fixed. Imagine some gems documentation website. It needs to generate docs using rdoc or yard, and that may include snippets of Ruby code, which if there is any syntax incompatibility between Ruby versions should be parsed with a version the is gem compatible with. And that information might be in the gemspec or CI files or based on the published date, etc, it would be quite messy if one has to dynamically generate a `Gemfile` based on that version and run another process vs just using the latest release of YARP and being able to specify the version dynamically. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104400 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by mame (Yusuke Endoh). Hi @kddnewton kddnewton (Kevin Newton) wrote in #note-22:

From this thread and others, this is my understanding of the requirements:

* The `YARP` name is not good, since we don't want `YA`- prefixes in production code.

There may be a communication error. More precisely, * The `YARP` name is not good as a built-in name, e.g., built-in constant names (like `String`). * The `YARP` name is acceptable as a gem name. If the Ruby API of YARP is provided as a gem, there is no need to rename it.

* We could like to be able to improve the Ruby API without having to go through the Ruby release cycle.

Whose requirement is this? I wasn't sure who was asking for it. This requirement is new to Ruby and makes the discussion complicated. If YARP is a built-in feature, I recommend that you drop this requirement. If YARP is a gem, you are free of course.

* We would like for users to be able to parse Ruby code as various versions of Ruby have previously parsed Ruby code. * We would like for users to be able to parse Ruby code exactly as the current version of Ruby is parsing Ruby code.

As @eregon says, I think that two different non-current Ruby grammer versions should be available at the same time. The parser gem allows that. I am concerned that if there is something that the parser gem can do but YARP cannot, then the original goal of making YARP a universal parser might not be achieved.

Here are the kinds of changes we would like to be able to make outside of the Ruby release cycle:

* We would like to be able to add fields to the various nodes. * We would like to be able to add flags to the various nodes.

Sorry, what are "fields" and "flags" here?

* We would like to be able to change the Ruby API to provide additional functionality/classes.

Note that, if YARP is a built-in feature, all these changes will require Matz's approval. With that in mind, I would never recommend doing it outside of the Ruby release cycle. If YARP is a gem, you can do it freely.

Here are the steps I would like to take to address these requirements:

1. I would like to rename the `YARP` gem to `Prism`.

You may change it if you wish, but * As a built-in name, neither YARP nor Prism is allowed. The only name accepted for a built-in name so far is `Ruby::Parser`. * As a gem name, both YARP and Prism are okay. (However, Ruby::Parser is not a suitable name for a gem.) So I think it will solve nothing.

2. I would like to change the way we build Ruby objects from both the C API and the serialization API. Currently when we build Ruby objects from these APIs, we call `rb_class_new_instance` directly with the fields coming back from the parser. Instead, I will change this to call a Ruby method (either `XXXNode.new` or `YARP.build_xxx_node`) that will instantiate those objects. 3. When we add fields to nodes, we will patch the factory method to handle the various combinations coming back from the various APIs. Those factory methods will check the target Ruby version and handle the fields appropriately for that version of Ruby. 4. When we add flags to nodes or change the Ruby API to provide additional functionality, nothing should have to change. 5. I would like to add `YARP.parse_native` which will take advantage of the previous bullets here to specify that the target Ruby version should be exactly the version of Ruby that is currently running.

This seems interesting. Let me confirm. I understood it as follows: * We introduce, as a built-in feature, only one method to invoke the YARP parser linked into Ruby. Let's call it `RubyVM.invoke_yarp(src, handler)` as a tentative name. * `RubyVM.invoke_yarp` parses the `src` string and calls `handler.build_xxx_node(*child_nodes)` when creating a node. For example, `RubyVM.invoke_yarp("1 + x")` calls 1. `handler.built_integer_node("1")` (To be precise, it should pass a code location rather than a code fragment) 2. `handler.built_local_var_node("x")` 3. `handler.built_call_node(:+, (the result of 1), (the result of 2))` and finally returns the result of 3. Everything else (such as node class definitions, helper methods, etc.) is placed in the YARP gem. `YARP.parse_native` calls `RubyVM.invoke_yarp`. Is my understanding correct? It's similar to Ripper's SAX API. If this is correct, it would be good if you could focus your proposal on the `RubyVM.invoke_yarp` spec. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104536 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze). @mame Thanks for the clarifications, notably about the name. My impression is if YARP supports multiple Ruby grammar versions then basically everything discussed in this is solved (and no need to rename, and only C files of yarp are copied in CRuby + available as a bundled gem). Regarding that handler, where would it be defined? It cannot be vendored in CRuby, otherwise e.g. if IntegerNode has a new field (in a newer yarp gem) it would call the IntegerNode constructor with not enough arguments. If it is in the yarp gem, and IntegerNode has a new field (in newer yarp gem) then `build_integer_node` needs e.g. 2 vs 1 arguments. So positional arguments cannot work. It could be keyword arguments, maybe but then it becomes extremely expensive to call those methods from a C extension. That overhead seems big enough to hurt YARP adoption, so I think this is not a good way. I think multi-version support is the best solution here.

I am concerned that if there is something that the parser gem can do but YARP cannot, then the original goal of making YARP a universal parser might not be achieved.

Exactly. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104555 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/

Issue #19772 has been updated by Eregon (Benoit Daloze). mame (Yusuke Endoh) wrote in #note-29:

He came up with an idea to enable a user to use multi-version yarp gems at a time by Namespace on read (#19744) in the future. So he said that the yarp gem doesn't have to support multi-version.

That's an interesting way to solve it but this seems clearly not for the near future (I think that is several years): * This feature is not merged yet, it might be difficult to implement reliably (there are so many shenanigans with symbol loaders/linkers/etc and inconsistencies between platforms, if the approach works on one platform it might not on another platform) * It might be even more difficult to implement on some other Rubies like JRuby and TruffleRuby. Both of these use the FFI gem and not the C extension for yarp, and I'm really not sure it's possible to load 2+ librubyparser in the same process with FFI without everything conflicting or segfaulting (and even if it happens to work for 2 versions on some platform, how to be confident it will be the case for other versions/platforms?). Until that is figured out I think we should assume this approach to be non-viable for other Rubies than CRuby which I think should be a blocker. * This would require extensive changes in RubyGems and Bundler, which might take a while as well and might be difficult for compatibility reasons. I'm not sure many Rubyists want a Bundler that feels like `npm`/`node_modules` with every dependency duplicated N times (+ of course the longer bundle times, etc). We should probably ask the opinion of RubyGems & Bundler maintainers in #19744. Also, until all the above is done, this doesn't solve that e.g. the parser gem couldn't rely on yarp and add it as a gem dependency then, and the same for all other tools which want to use yarp but don't want to force a given version (and if they do they will then be incompatible with other tools). So I think this solution is unrealistic for the near future. OTOH there seems to be a lot of work on yarp to also worry about multi-versions support now. So maybe multi-versions support in yarp could be added later, e.g. after the CRuby 3.3 release. Until then YARP can only parse the "Ruby 3.3" dialect which seems an OK limitation for the time being.

In light of the above, here is a suggestion.

* How about having the ruby repository mirror only the C implementation of YARP, but not Ruby API? It should be used only when `ruby --enable-yarp` or something are given. * How about bundling the "prism" gem as a bundled gem to provide a Ruby API for YARP?

I think this is good and should be done anyway. It is also easier to support YARP like that in JRuby & TruffleRuby, as the YARP Ruby API is then "just a normal gem" and can be updated with `gem install` etc. In fact JRuby & TruffleRuby are both integrating YARP and at least so far they assumed that yarp would be a bundled gem, only C+Java files (so not the Ruby API) are imported in JRuby & TruffleRuby repositories. ---------------------------------------- Misc #19772: API naming for YARP compiler https://bugs.ruby-lang.org/issues/19772#change-104707 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- We are working on the YARP compiler, and have [the first PR ready](https://github.com/ruby/ruby/pull/8042) which introduces the YARP compile method. Our only outstanding question before merging it is about naming. How should we expose the public API for YARP's compile method? Potential suggestions: 1. YARP.compile 2. RubyVM::InstructionSequence.compile(yarp: true) 3. RubyVM::InstructionSequence.compile_yarp 4. Any of the above options, with a name other than yarp (please suggest an alternative) Regarding option 1, which would mirror `YARP.parse`, is the top level constant `YARP` acceptable? cc @matz @ko1 -- https://bugs.ruby-lang.org/