Issue #21618 has been updated by kddnewton (Kevin Newton). Yes, this was brought up by @k0kubun originally as one of the requirements. At the moment the official answer is `Prism.parse_file(filepath, version: RUBY_VERSION)`, because that will work for all Ruby versions in normal maintenance right now. That being said, I do think it would make sense to support a feature like this going forward, it just becomes difficult for other implementations, depending on how they ship Prism. It might make sense for it to be `RubyVM.parse` to make it clear it's just for CRuby. I'm not sure if @Eregon or @enebo would have thoughts on this. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114725 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by kddnewton (Kevin Newton). So the issue is that there is the C library and then there is the Ruby library. The C library can parse and provide an AST in C structures, but to reify it into Ruby objects requires a specific Ruby library. This is possible, but only if the runtime ships the same matching version for both. I don't think any other implementation does this, because they only need the C library + FFI (JRuby/TruffleRuby/Garnet) or the C library + native bindings (Natalie/Opal). Of course it is always possible to include methods that are just unimplemented for other runtimes, it just becomes less portable. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114727 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Eregon (Benoit Daloze). From the description it seems a (the?) main concern is that `Prism.parse(code)` etc use the latest version by default and not `version: RUBY_VERSION` by default. I share that concern, that feels error-prone. Is there a use case for the default being latest? I think not, usages would either want the current Ruby syntax or a specific one and then they would specify it explicitly (like in RuboCop). So how about changing the default parse/syntax version of Prism then to be RUBY_VERSION? ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114744 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Eregon (Benoit Daloze). Earlopain (Earlopain _) wrote in #note-7:

I don't think parsing exactly (not just almost the same) as the ruby I am running as is an unreasonable request.

I think they are exactly the same, and if it's not the case that's a bug we can fix in Prism. I think it is unreasonable to have a duplicate Ruby API just for this under another module (BTW it's quite a lot of classes and memory footprint, not great to have effectively 2 copies of Prism in the same process), + namespacing functions at the C level too (otherwise we get native symbol conflicts), that's a huge amount of complications for as far as I can see no useful difference with using `version: RUBY_VERSION`. There is no possible way to share Prism node classes, as e.g. newer/older versions might have extra node classes.

But prism is not patch level aware (that is fine) so even if you do `RUBY_VERSION` it is not perfect. 3.2.0 I think had a bug with nested anonymous block parameters that had a pretty big impact.

I think if there is a relevant difference between Ruby X.Y.Z and X.Y.(Z+1) then Prism could/should take that into account, since it already takes `RUBY_VERSION` as the input. I would guess though the only differences are bug fixes, and then is it really useful to have the buggy behavior on the buggy Ruby, isn't it better to just have the bug-fixed behavior by ensuring to use a new enough `prism` gem? BTW do you have a link for that bug? That's before Prism became a default gem though, so of course it had less testing and usage back then.

Now I am required to also bump prism with every patch version. If I run older patch versions, I would be have to run older prism versions from around the time my patch version was released.

Only if there is a such a bug and it matters for your usage and you don't use the prism default gem (since that would have the fix). That's a lot of `if`, IOW I think it's so rare in practice it doesn't matter. It's also a totally normal workflow to update a gem to get a bug fix. In such a scenario it's also clearly better to use a newer prism >=A.B.C as a gem/bundle dependency than to get the broken behavior on Ruby 3.2.0 and the correct behavior on 3.2.1.

...

From the description it seems a (the?) main concern is that Prism.parse(code) etc use the latest version by default

Yeah, I am not a fan of that behaviour as well. I guess this is intential, similar to how the unversioned parser translator class also uses the latest version. But changing that was not really what I was asking for.

I think it's historical at this point, and maybe comes from Prism initially only having one version, but @kddnewton would know better. My guess is changing the default here would fix more issues than cause incompatibilities.

----

To be clear, patch versions of ruby should never contain any big syntax changes. The impact of using any prism version that understands my major/minor version should be minimal in the majority of cases. But, I would prefer to just have something where I don't even have to worry about such things.

`Prism.parse("foo", version: RUBY_VERSION)` works good enough but differently to the real intention to the point where it makes me uncomfortable.

I think the solution there is, if there is a relevant difference between Ruby X.Y.Z and X.Y.(Z+1) then Prism should take that into account, and then it matches the intention exactly. Another thought is if you really want to ensure you use the exact same Prism version as the one used by the interpreter, you could check that as long as that default version version is exposed somehow (e.g. it is though gemspecs). I don't think that's great though then, because it prevents other gems to use a newer Prism version (which would work just fine for your use case too). ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114752 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/

Issue #21618 has been updated by Earlopain (Earlopain _).

I think they are exactly the same, and if it's not the case that's a bug we can fix in Prism.

It's already a non-trivial amount of effort to provide major/minor support. Taking patch into account is unreasonable in my eyes. Accepting X.Y.Z as an input is just for convenience.

BTW do you have a link for that bug?

It was 3.3.0: https://bugs.ruby-lang.org/issues/20090

My guess is changing the default here would fix more issues than cause incompatibilities.

I don't think changing the default would be much of an improvement. You can't tell what people really want. I would prefer no default at all but that ship has sailed.

IOW, I think they are theoretical concerns.

They are. With ripper they were none, which is mostly what I am saying. Prism is otherwise better in every way. For now, I will just update places to use `RUBY_VERSION` since that is clearly more correct that not passing anything at all. I don't think there would be much discussion if it were trivial to implement. It's ok to reject based on that. `RUBY_VERSION` gets you 99% there but it would be nice if it could be 100%. ---------------------------------------- Bug #21618: Allow to use the build-in prism version to parse code https://bugs.ruby-lang.org/issues/21618#change-114754 * Author: Earlopain (Earlopain _) * Status: Open * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- Prism is a multi-version parser, which is a great feature. If one calls `Prism.parse("foo")`, it will always use the latest version prism knows about. This may or may not be the version that is currently executing. This is problematic when one wants to introspect code the same way that the currently running ruby version would. Consider that some syntax will be disallowed in the next ruby version. I would have to specify the ruby version I want to parse as in order to not run into syntax errors: `Prism.parse("foo", version: RUBY_VERSION)`. But doing so is not feasable because `prism` is distributed out of sync with ruby itself. If the user already has prism in their lockfile, the user may run a prism version that doesn't yet know about lets say ruby 4.0 and thus raise. Similarly, it may parse as an older patch version that has subtle behaviour differences. `ripper` does not have this issue because it is always tied to the ruby version and it is not distributed as a gem, so what the user has will always be exactly what ruby shipped with. I wish for a similar API that utilizes the prism version bundled with ruby itself. Libraries like `rails` have moved from ripper to prism because of its superior developer experience but it risks breaking in unexpected ways with new `prism` versions that know about more recent syntax. `error_highlight` for example also suffers from the same defect. It seems like prism currently has 34 methods that optionally take a version (per the rbi file). Many of these are trivial helper methods like `Prism.parse_success?` (does the parse result have any errors?). I would be happy with supporting only the minimal functions like `Prism.parse` and `Prism.parse_file`. Currently I don't have a use-case for any of the other methods. Pretty much just functions to mirror `RubyVM::InstructionSequence.compile_file` and `RubyVM::InstructionSequence.compile`. It would be optimal if I would be able to transparently call `Prism.parse("foo", version: "current")` (or maybe even have an unspecified version mean the build-in prism version) but I am not sure how feasable that is since I'm pretty sure logic for this would have to live in ruby itself. @kddnewton does this feature request make sense? Do you have any other ideas/inputs? -- https://bugs.ruby-lang.org/