March 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113050] [Ruby master Feature#19560] IO#close_on_fork= and IO#close_on_fork?
by byroot (Jean Boussier) 19 Apr '23

19 Apr '23

Issue #19560 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19560: IO#close_on_fork= and IO#close_on_fork? https://bugs.ruby-lang.org/issues/19560 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context Forking setups are extremely common in the Ruby ecosystem, as they remain the primary way to get parallelism with MRI. Generally speaking it works very well, however there are two main issues library authors and application owners need to be careful of: - Restarting threads - Closing inherited connections and other file descriptors. I believe we could make the second one much easier. ### O_CLOFORK A couple years ago, [a new flag was added to the POSIX spec: `O_CLOFORK`](https://austingroupbugs.net/view.php?id=1318). Similar to `O_CLOEXEC`, this file descriptor flag make it so the file descriptor is automatically closed upon forking. Unfortunately its support is relatively limited for now. It's supported on macOS and some relatively exotic unixes, but not in Linux nor most BSDs. [The feature was discussed on Linux mailing list](https://lore.kernel.org/lkml/20200525081626.GA16796@amd/T/#m5b8b20ea6…, but it seem to have encountered some strong opposition, so it's unclear if we can hope for it to be added. That said, I don't think it would be too hard for Ruby to shim this feature by closing all IOs with `close_on_fork?` right after fork. ### Ruby shim This can be implemented as a Ruby shim starting in Ruby 3.1 using the `Process._fork` callback ```ruby class IO def close_on_fork=(enabled) if enabled ::CloseIOOnFork::IOS[self] = true end @close_on_fork = enabled end def close_on_fork? @close_on_fork end end module CloseIOOnFork IOS = ObjectSpace::WeakMap.new def _fork pid = super if pid == 0 # child ::CloseIOOnFork::IOS.each_key do |io| io.close if io.close_on_fork? end end pid end end Process.singleton_class.prepend(CloseIOOnFork) rd, rw = IO.pipe rw.close_on_fork = true pid = fork do p rw.closed? # => true end Process.wait(pid) ``` ### Usage With such feature, many network client would mostly just need to set this flag on their sockets, and just properly handle unexpectedly closed connections, which most already do. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:112183] [Ruby master Bug#19401] [Doc] Broken links in CSV documentation
by eightbitraptor (Matthew Valentine-House) 19 Apr '23

19 Apr '23

Issue #19401 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19401: [Doc] Broken links in CSV documentation https://bugs.ruby-lang.org/issues/19401 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The link "Recipes for CSV" on [this page](https://docs.ruby-lang.org/en/master/CSV.html) points to a broken link: [https://docs.ruby-lang.org/en/master/csv/recipes/recipes_rdoc.html](https:/…. It doesn't look like the intended target `doc/csv/recipes/recipes.rdoc` is being included in the rdoc output. There isn't an equivalent HTML file generated in `.ext/html` after I run `make doc` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:111990] [Ruby master Bug#19371] Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML
by tombruijn (Tom de Bruijn) 14 Apr '23

14 Apr '23

Issue #19371 has been reported by tombruijn (Tom de Bruijn). ---------------------------------------- Bug #19371: Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML https://bugs.ruby-lang.org/issues/19371 * Author: tombruijn (Tom de Bruijn) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Summary There's an issue on Ruby versions with Psych 4 installed by default (Ruby 2.6 through 3.1) after installing the Psych gem version 5. This problem occurs when a Ruby gem has a C-extension installation script that parses a YAML string. I'm reporting it here and not with on the Psych gem repo, because it looks more like an issue with which Ruby C-extension is load during other gem's C-extension installation. ## Background I have a gem that parses a YAML string in the C-extension installation script, or it calls `Gem.configuration[:http_proxy]`, which parses the `.gemrc` file as YAML. This triggers the error mentioned below. This YAML parsing is done in the gem's `ext/extconf.rb` file. An example gem can be found in this repository: https://github.com/tombruijn/yaml-dummy-gem, see the [`ext/extconf.rb` file](https://github.com/tombruijn/yaml-dummy-gem/blob/main/ext/extconf.rb#…. ## The problem On Ruby 3.1.3 Psych version 4 is installed by default. When it parses the YAML file, it will use Psych 4. When Psych 5 is also installed on Ruby 3.1.3, it is no longer be able to parse the YAML file. The following error is raised: ``` $ bundle install Fetching https://github.com/tombruijn/yaml-dummy-gem.git Resolving dependencies... Using bundler 2.3.7 Using yaml-dummy-gem 1.0.0 from https://github.com/tombruijn/yaml-dummy-gem.git (at main@a48852d) Gem::Ext::BuildError: ERROR: Failed to build gem native extension. current directory: /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d/ext /usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r ./siteconf20230123-730-rmbnnl.rb extconf.rb /usr/local/lib/ruby/3.1.0/psych.rb:459:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x0000ffff8078c7f8 @handler=#<Psych::Handlers::DocumentStream:0x0000ffff8078c910 @stack=[], @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil, @block=#<Proc:0x0000ffff8078c848 /usr/local/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError) parser.parse yaml, filename ^^^^^^ from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse' from extconf.rb:3:in `<main>' extconf failed, exit code 1 Gem files will remain installed in /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d for inspection. Results logged to /usr/local/bundle/bundler/gems/extensions/aarch64-linux/3.1.0/yaml-dummy-gem-a48852dac33d/gem_make.out /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run' /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build' /usr/local/lib/ruby/3.1.0/tempfile.rb:317:in `open' /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in build_extensions' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions' /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions' /usr/local/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions' /usr/local/lib/ruby/3.1.0/bundler/source/path/installer.rb:34:in `post_install' /usr/local/lib/ruby/3.1.0/bundler/source/path.rb:244:in `generate_bin' /usr/local/lib/ruby/3.1.0/bundler/source/git.rb:194:in `install' /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install' /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in `install_from_spec' /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in `do_install' /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in create_threads' An error occurred while installing yaml-dummy-gem (1.0.0), and Bundler cannot continue. In Gemfile: yaml-dummy-gem ``` ## Debugging results The error is raised because the `Psych::Parser#parse` method cannot be found. In Psych version 4, this method is [defined by the Psych C-extension](https://github.com/ruby/psych/blob/2c3708e0a483c6d44ebddaff0b5…. In Psych version 5 the `parse` method is [defined in the gem's Ruby code](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb52d…. This method calls a [C function registered as the private `_native_parse` method](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb5…, which is the renamed version of the `parse` C-function in Psych version 4. From what I can tell, the Psych version 4 C-extension is no longer loaded when Psych version 5 is installed in this scenario. There is a mix up in which Psych gem version's C-extension is loaded during my dummy gem's C-extension installation. It load the Psych 4 Ruby code, with the Psych 5 C-extension∂. I confirmed this by modifying the standard installed Psych gem's code on Ruby 3.1 (Docker image `ruby:3.1`), with the following the change, which prints `true` on error. This means the Psych 4 gem has the Psych 5 C-extension loaded where `_native_parse` is defined. ```diff diff --git lib/psych.rb lib/psych.rb index 42d79ef..1a690d2 100644 --- lib/psych.rb +++ lib/psych.rb @@ -452,6 +452,9 @@ def self.parser def self.parse_stream yaml, filename: nil, &block if block_given? parser = Psych::Parser.new(Handlers::DocumentStream.new(&block)) + # This returns `true`, but it should be `false`. The `_native_parse` + # method is defined in the Psych 5 C-extension, not Psych 4. + puts parser.respond_to? :_native_parse, true # => true parser.parse yaml, filename ``` This error only occurs during the gem's extension installation in `ext/extconf.rb`. If the gem parses YAML when a Ruby app is running, it will not produce the same error with Psych version 5 installed. This issue does not occur on Ruby 3.2, where Psych version 5 is installed by default. I have confirmed this error occurs on the latest patch releases of the following Ruby versions: 3.1, 3.0, 2.7 and 2.6. ``` $ gem list psych psych (5.0.2, default: 4.0.3) ``` ## Code to reproduce Here is a basic Ruby gem that only parses a YAML file during extension installation: https://github.com/tombruijn/yaml-dummy-gem Here is a small project that triggers the error: https://github.com/tombruijn/yaml-dummy-ruby-app A GitHub actions workflow shows the results for all affected Ruby versions: https://github.com/tombruijn/yaml-dummy-ruby-app/actions/runs/3969088933 The [example app repo](https://github.com/tombruijn/yaml-dummy-ruby-app) also has instructions to run the example app locally. Please follow the instructions in the README to see the error. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:112866] [Ruby master Feature#19528] `JSON.load` defaults are surprising (`create_additions: true`)
by byroot (Jean Boussier) 14 Apr '23

14 Apr '23

Issue #19528 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19528: `JSON.load` defaults are surprising (`create_additions: true`) https://bugs.ruby-lang.org/issues/19528 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- I'm not sure if it was actually intended, but there's some tacit naming convention for serializers in Ruby to use `load` and `dump` as methods, likely inspired from `Marshal` and `YAML`. Because of this it's extremely common to see code that uses `JSON.load` expecting a simple, no surprise, and safe JSON parsing. However that's `JSON.parse`. `JSON.load` has this very surprising behavior (albeit perfectly documented), of de-serializing more complex types: ```ruby >> JSON.load('{ "json_class": "String", "raw": [72, 101, 108, 108, 111] }') => "Hello" ``` It's particularly weird because aside from the `String` extension that is eagerly defined, for other types you have to `require "json/add/core"`. Seasoned Ruby developers know about this of course, and [it is banned by various linters](https://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Security/JSONLoad), but it keeps popping regularly in gems security releases and such. ### Proposal Assuming entirely removing this feature is not an option, I think `json 2.x` should warn when this feature is actually being used, and `json 3.x` should disable it by default and require users to explicitly use `JSON.load(str, create_additions: true)` to keep the old behavior. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:113028] [Ruby master Bug#19551] Backport commits for CI failures
by hsbt (Hiroshi SHIBATA) 14 Apr '23

14 Apr '23

Issue #19551 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19551: Backport commits for CI failures https://bugs.ruby-lang.org/issues/19551 * Author: hsbt (Hiroshi SHIBATA) * Status: Closed * Priority: Normal * Backport: 2.7: REQUIRED, 3.0: REQUIRED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- I created PRs to fix CI failure for stable or old stable branches. * [Fix CI for ruby_2_7](https://github.com/ruby/ruby/pull/7259) * [Fix CI for ruby_3_0](https://github.com/ruby/ruby/pull/7258) * [Fix CI for ruby_3_2](https://github.com/ruby/ruby/pull/7604) Please merge them before release time. ---- Can I merge PR for "Fix CI failure" myself from April 2023? I can maintain to keep CI status with my full-time job. It helps to release work for all of branch maintainers. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:113051] [Ruby master Feature#19561] ObjectSpace::WeakMap#delete and ObjectSpace::WeakKeyMap#delete
by byroot (Jean Boussier) 14 Apr '23

14 Apr '23

Issue #19561 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19561: ObjectSpace::WeakMap#delete and ObjectSpace::WeakKeyMap#delete https://bugs.ruby-lang.org/issues/19561 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- I just realized in https://bugs.ruby-lang.org/issues/19560#Ruby-shim that there's no way to remove elements from either WeakMaps. I think they could both benefit from a `delete` method. ### Use case For example, to keep track of a subset of instances of a class, you may need to remove some from the list when a condition changes: ```ruby class Connection CLOSE_ON_FORK = ObjectSpace::WeakMap.new class << self def after_fork CLOSE_ON_FORK.each_key do |connection| connection.close if connection.close_on_fork? end end end def close_on_fork=(enabled) if enabled CLOSE_ON_FORK[self] = true else CLOSE_ON_FORK.delete(self) end @close_on_fork = enabled end def close_on_fork? @close_on_fork end end module CloseOnFork IOS = ObjectSpace::WeakMap.new def _fork pid = super if pid == 0 # child ::Connection.after_fork end pid end end Process.singleton_class.prepend(CloseIOOnFork) ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:112933] [Ruby master Feature#19538] Performance warnings
by byroot (Jean Boussier) 13 Apr '23

13 Apr '23

Issue #19538 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19538: Performance warnings https://bugs.ruby-lang.org/issues/19538 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- Suggested by @Eregon. There are program behaviors that are supported, but that we know aren't good for performance, however it's hard for users to know about them. Now that we have warning categories, we could add a `:performance` category to allow the VM to emit warning in some situations. The category would be disabled by default, and users interested in optimizing their program could turn it on in development. ```ruby Warning[:performance] = true ``` -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113060] [Ruby master Bug#19564] Range.cover? fails for Range wrapped in SimpleDelegator
by PeteMichaud (Pete Michaud) 13 Apr '23

13 Apr '23

Issue #19564 has been reported by PeteMichaud (Pete Michaud). ---------------------------------------- Bug #19564: Range.cover? fails for Range wrapped in SimpleDelegator https://bugs.ruby-lang.org/issues/19564 * Author: PeteMichaud (Pete Michaud) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-darwin22] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ```ruby test 'built-in cover? method is broken for SimpleDelegator' do class TimeRangeDelegator < SimpleDelegator end t1 = Time.now t2 = t1+1.day t3 = t1+1.week r_long = t1..t3 r_short = t1..t2 assert r_long.cover?(r_short) #passes correctly, assert == true r_long_delegated = TimeRangeDelegator.new(r_long) r_short_delegated = TimeRangeDelegator.new(r_short) assert r_long_delegated.cover?(r_short_delegated) #fails incorrectly, assert == false end ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:112818] [Ruby master Misc#19525] DevMeeting-2023-04-13
by mame (Yusuke Endoh) 13 Apr '23

13 Apr '23

Issue #19525 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19525: DevMeeting-2023-04-13 https://bugs.ruby-lang.org/issues/19525 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/04/13 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/04/10. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

7 7

[ruby-core:111780] [Ruby master Bug#19331] --enable-rpath results in incorrect RPATH in Ruby 3.1.3
by Kulikjak (Jakub Kulik) 12 Apr '23

12 Apr '23

Issue #19331 has been reported by Kulikjak (Jakub Kulik). ---------------------------------------- Bug #19331: --enable-rpath results in incorrect RPATH in Ruby 3.1.3 https://bugs.ruby-lang.org/issues/19331 * Author: Kulikjak (Jakub Kulik) * Status: Open * Priority: Normal * ruby -v: 3.1.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I just updated Ruby from 3.1.2 to 3.1.3 and found out that all .so libraries from the enc directory have wrong RPATH/RUNPATH (when building with `--enable-rpath`). I first hit this on Solaris, but my testing on Linux resulted in the same thing. It can be easily reproduced with the following: ``` wget http://cache.ruby-lang.org/pub/ruby/3.1/ruby-3.1.3.tar.gz tar xzf ruby-3.1.3.tar.gz cd ruby-3.1.3 ./configure --prefix=/usr/ruby/3.1 --mandir=/usr/ruby/3.1/share/man --bindir=/usr/ruby/3.1/bin --sbindir=/usr/ruby/3.1/sbin --libdir=/usr/ruby/3.1/lib/amd64 --with-rubylibprefix=/usr/ruby/3.1/lib/ruby --enable-shared --enable-rpath /usr/bin/make -j 16 -l 32 objdump -x ./.ext/x86_64-linux/enc/windows_31j.so | grep RPATH ``` While previously this resulted in the following output: `RPATH /usr/ruby/3.1/lib/amd64` in 3.1.3 it is wrongly set to the build directory: `RPATH /home/jkulik/rubytest/ruby-3.1.3` I tried looking for some obvious change but didn't find the core of this issue yet. All I found out is that generated `enc.mk` differs in `prefix` and `libdir` ``` -prefix = /usr/ruby/3.1 +prefix = /home/jkulik/rubytest/ruby-3.1.3 exec_prefix = $(prefix) -libdir = $(exec_prefix)/lib/amd64 +libdir = /home/jkulik/rubytest/ruby-3.1.3 ``` `rbconfig.rb` is similar in both versions, but when I print out `CONFIG` in `make_encmake.rb` right after `load File.expand_path("lib/mkmf.rb", dir)`, I can see the differences from above, so it's probably something in the `lib/mkmf.rb` file that changes that. I am seeing the same issue in the current latest ruby 3.1 branch cloned from github. -- https://bugs.ruby-lang.org/

3 4