Issue #19371 has been updated by tombruijn (Tom de Bruijn).
hsbt (Hiroshi SHIBATA) wrote in #note-3:
@tombruijn I'll do work this issue at
https://github.com/rubygems/rubygems. Can you move this issue to rubygems repo? Thanks.
I have created an issue here:
https://github.com/rubygems/rubygems/issues/6611
Sorry for the late response on transferring the issue to this issue tracker. The
notification from the Ruby issue tracker was marked as spam for me because of some domain
verification issue.
<img
src="https://user-images.githubusercontent.com/282402/232135175-0fc446…
width="400px">
----------------------------------------
Bug #19371: Having Psych 5 installed raises an error during another gem's C-extension
installation when parsing YAML
https://bugs.ruby-lang.org/issues/19371#change-102815
* Author: tombruijn (Tom de Bruijn)
* Status: Third Party's Issue
* Priority: Normal
* Assignee: hsbt (Hiroshi SHIBATA)
* ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
## Summary
There's an issue on Ruby versions with Psych 4 installed by default (Ruby 2.6 through
3.1) after installing the Psych gem version 5. This problem occurs when a Ruby gem has a
C-extension installation script that parses a YAML string.
I'm reporting it here and not with on the Psych gem repo, because it looks more like
an issue with which Ruby C-extension is load during other gem's C-extension
installation.
## Background
I have a gem that parses a YAML string in the C-extension installation script, or it calls
`Gem.configuration[:http_proxy]`, which parses the `.gemrc` file as YAML. This triggers
the error mentioned below.
This YAML parsing is done in the gem's `ext/extconf.rb` file. An example gem can be
found in this repository:
https://github.com/tombruijn/yaml-dummy-gem, see the
[`ext/extconf.rb`
file](https://github.com/tombruijn/yaml-dummy-gem/blob/main/ext/extconf.rb#….
## The problem
On Ruby 3.1.3 Psych version 4 is installed by default. When it parses the YAML file, it
will use Psych 4.
When Psych 5 is also installed on Ruby 3.1.3, it is no longer be able to parse the YAML
file. The following error is raised:
```
$ bundle install
Fetching
https://github.com/tombruijn/yaml-dummy-gem.git
Resolving dependencies...
Using bundler 2.3.7
Using yaml-dummy-gem 1.0.0 from
https://github.com/tombruijn/yaml-dummy-gem.git (at
main@a48852d)
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d/ext
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r ./siteconf20230123-730-rmbnnl.rb
extconf.rb
/usr/local/lib/ruby/3.1.0/psych.rb:459:in `parse_stream': undefined method `parse'
for #<Psych::Parser:0x0000ffff8078c7f8
@handler=#<Psych::Handlers::DocumentStream:0x0000ffff8078c910 @stack=[], @last=nil,
@root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil,
@block=#<Proc:0x0000ffff8078c848 /usr/local/lib/ruby/3.1.0/psych.rb:399>>,
@external_encoding=0> (NoMethodError)
parser.parse yaml, filename
^^^^^^
from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse'
from extconf.rb:3:in `<main>'
extconf failed, exit code 1
Gem files will remain installed in
/usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d for inspection.
Results logged to
/usr/local/bundle/bundler/gems/extensions/aarch64-linux/3.1.0/yaml-dummy-gem-a48852dac33d/gem_make.out
/usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run'
/usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build'
/usr/local/lib/ruby/3.1.0/tempfile.rb:317:in `open'
/usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
/usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension'
/usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in
build_extensions'
/usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each'
/usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions'
/usr/local/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions'
/usr/local/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in
`build_extensions'
/usr/local/lib/ruby/3.1.0/bundler/source/path/installer.rb:34:in `post_install'
/usr/local/lib/ruby/3.1.0/bundler/source/path.rb:244:in `generate_bin'
/usr/local/lib/ruby/3.1.0/bundler/source/git.rb:194:in `install'
/usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install'
/usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in
`do_install'
/usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in
worker_pool'
/usr/local/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func'
/usr/local/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue'
/usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop'
/usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue'
/usr/local/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in
create_threads'
An error occurred while installing yaml-dummy-gem (1.0.0), and Bundler cannot continue.
In Gemfile:
yaml-dummy-gem
```
## Debugging results
The error is raised because the `Psych::Parser#parse` method cannot be found. In Psych
version 4, this method is [defined by the Psych
C-extension](https://github.com/ruby/psych/blob/2c3708e0a483c6d44ebddaff0b5….
In Psych version 5 the `parse` method is [defined in the gem's Ruby
code](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb52d….
This method calls a [C function registered as the private `_native_parse`
method](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb5…,
which is the renamed version of the `parse` C-function in Psych version 4.
From what I can tell, the Psych version 4 C-extension is no longer loaded when Psych
version 5 is installed in this scenario. There is a mix up in which Psych gem
version's C-extension is loaded during my dummy gem's C-extension installation. It
load the Psych 4 Ruby code, with the Psych 5 C-extension∂.
I confirmed this by modifying the standard installed Psych gem's code on Ruby 3.1
(Docker image `ruby:3.1`), with the following the change, which prints `true` on error.
This means the Psych 4 gem has the Psych 5 C-extension loaded where `_native_parse` is
defined.
```diff
diff --git lib/psych.rb lib/psych.rb
index 42d79ef..1a690d2 100644
--- lib/psych.rb
+++ lib/psych.rb
@@ -452,6 +452,9 @@ def self.parser
def self.parse_stream yaml, filename: nil, &block
if block_given?
parser = Psych::Parser.new(Handlers::DocumentStream.new(&block))
+ # This returns `true`, but it should be `false`. The `_native_parse`
+ # method is defined in the Psych 5 C-extension, not Psych 4.
+ puts parser.respond_to? :_native_parse, true # => true
parser.parse yaml, filename
```
This error only occurs during the gem's extension installation in `ext/extconf.rb`. If
the gem parses YAML when a Ruby app is running, it will not produce the same error with
Psych version 5 installed.
This issue does not occur on Ruby 3.2, where Psych version 5 is installed by default.
I have confirmed this error occurs on the latest patch releases of the following Ruby
versions: 3.1, 3.0, 2.7 and 2.6.
```
$ gem list psych
psych (5.0.2, default: 4.0.3)
```
## Code to reproduce
Here is a basic Ruby gem that only parses a YAML file during extension installation:
https://github.com/tombruijn/yaml-dummy-gem
Here is a small project that triggers the error:
https://github.com/tombruijn/yaml-dummy-ruby-app
A GitHub actions workflow shows the results for all affected Ruby versions:
https://github.com/tombruijn/yaml-dummy-ruby-app/actions/runs/3969088933
The [example app
repo](https://github.com/tombruijn/yaml-dummy-ruby-app) also has
instructions to run the example app locally. Please follow the instructions in the README
to see the error.
--
https://bugs.ruby-lang.org/