Issue #21618 has been updated by tenderlovemaking (Aaron Patterson). I think this is a good idea. I recall discussing this type of feature with @kddnewton at one point. He had some ideas about implementing it, but I don't recall the exact ideas. The use case seems very clear though. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114723 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Earlopain (Earlopain _). `RubyVM.parse` doesn't sound particularly nice to me. If you put it there then there have to be `defined?` checks and fallback implementations for non-CRuby which to me kinda defeats the point of using prism in the first place. Of course, I don't have any better suggestions at the moment. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114726 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Eregon (Benoit Daloze). Why is `Prism.parse_file(filepath, version: RUBY_VERSION)` not good enough? In which concrete scenarios does it not work? The `prism` default gem shipped with Ruby 3.3+ will always support `version: RUBY_VERSION`. Another `prism` version might be used when using `bundler` or installing a newer `prism` (e.g. `gem install prism`). If that prism version is newer, `version: RUBY_VERSION` will work. If that prism version is older, it's possible it doesn't recognize the RUBY_VERSION. That case though could be seen as "that old prism doesn't support a recent Ruby", i.e. the prism dependency should be updated to be less old. In that sense it's not different than many other gems with older versions which might not work on the newest Ruby version. Another POV here is: either `prism` is not specified in the Gemfile, and `version: RUBY_VERSION` will always work, or it's specified in the Gemfile and then it's the responsibility of the maintainers of that Gemfile to keep it recent enough, e.g. by not using a `Gemfile.lock` or by regularly updating the `Gemfile.lcok`, at least when supported a new Ruby version (which is the same workflow as for other gem dependencies). --- I'm guessing what you are asking is to get the Prism Ruby API for the prism C parser used by the Ruby interpreter to parse Ruby code? I think that cannot be provided easily, because it would conflict with the prism gem, of which another version might be loaded. I suppose one could try to nest that "current Ruby" Prism Ruby API under a different module than `::Prism`, but it feels suboptimal and complicated, I think there is not a strong enough need to justify such complexity. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114741 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Earlopain (Earlopain _).

I'm guessing what you are asking is to get the Prism Ruby API for the prism C parser used by the Ruby interpreter to parse Ruby code?

Ripper for all its faults was able to do just that. I understand that there are implementation concerns, I am just asking for something to make that possible again. I don't think parsing exactly (not just almost the same) as the ruby I am running as is an unreasonable request. You should always bump all gems on ruby version updates. But prism is not patch level aware (that is fine) so even if you do `RUBY_VERSION` it is not perfect. 3.2.0 I think had a bug with nested anonymous block parameters that had a pretty big impact. Now I am required to also bump prism with every patch version. If I run older patch versions, I would be have to run older prism versions from around the time my patch version was released.

From the description it seems a (the?) main concern is that Prism.parse(code) etc use the latest version by default

Yeah, I am not a fan of that behaviour as well. I guess this is intential, similar to how the unversioned parser translator class also uses the latest version. But changing that was not really what I was asking for. ---- To be clear, patch versions of ruby should never contain any big syntax changes. The impact of using any prism version that understands my major/minor version should be minimal in the majority of cases. But, I would prefer to just have something where I don't even have to worry about such things. `Prism.parse("foo", version: RUBY_VERSION)` works good enough but differently to the real intention to the point where it makes me uncomfortable. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114746 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Eregon (Benoit Daloze). Taking the concerns from the description: Earlopain (Earlopain _) wrote:

Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise.

If it's locked to an older prism in lockfile, it's expected to not work. Same as any other gem really. Need to update in lockfile.

Similarly, it may parse as an older patch version that has subtle behaviour differences.

If the differences are relevant those could be handled in Prism. They are probably just bug fixes and if so then the fixed behavior seems always best. IOW, I think they are theoretical concerns. Is there a concrete realistic problem with using `Prism.parse_file(filepath, version: RUBY_VERSION)` where the solution is not trivial? ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114753 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Eregon (Benoit Daloze). Earlopain (Earlopain _) wrote in #note-10:

I don't think changing the default would be much of an improvement. You can't tell what people really want.

I *think* we can establish it's only 2 use cases: * either they want the same as the running Ruby version, i.e they just want to parse code like the current Ruby parses code (and that's why I think that should be the default), * or they want some specific other version, in which case they really need to specify it (e.g. "3.4"). I don't know of a realistic use case to parse with "latest version supported by Prism" on an older Ruby (especially considering that changes based on the Prism version), that seems basically always an error to me (though it might accidentally work good enough for a while), but maybe I'm missing something? So with that I think it means the default of "latest" is basically of no use, and therefore changing that to "current" fixes it for the first use case (probably the most common), where one expects `Prism.parse(code)` to parse code with the same syntax version as RUBY_VERSION. For the second use case one need to be explicit anyway, so the default doesn't matter there.

For now, I will just update places to use `RUBY_VERSION` since that is clearly more correct that not passing anything at all.

I think this is the way to go. If there is practical issue with that let's reconsider what to do to address it. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114769 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Earlopain (Earlopain _). They are theoretical for now, at least to me. Perhaps I could find something if I really tried right now but I don't think that would be time well spent. Right now we are actually at option 5 since prism can parse multiple versions of ruby syntax. If it where option 3, that would be a rather big/majorly inconvenient problem for the described use-case of that comment you linked. It's also the use-case I openend this issue with. This is just about patch-level compatibility. `Parser::CurrentRuby` for example also only provides major/minor parsing. It's pretty ok the way it is right now already, as long as you don't do `Prism.parse` without a version and expect it to use the current ruby instead.

The other issue discussed in here is that prism defaults to the latest Ruby syntax version

I am fine with either using the latest or the current ruby as the default. I think such an API would actually be better served without any default though. But yeah, we don't have to discuss that here. If `Prism.parse(foo, version: "current_ruby")` or similar would be possible, that'd be great actually. Mostly I have complained about outdated gems in the lockfile etc., with that prism itself can raise a very descriptive error if that happens. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114806 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/