March 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112329] [Ruby master Misc#19431] DevMeeting at RubyKaigi 2023
by mame (Yusuke Endoh) 10 May '23

10 May '23

Issue #19431 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19431: DevMeeting at RubyKaigi 2023 https://bugs.ruby-lang.org/issues/19431 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- RubyKaigi 2023 will be at Matsumoto, Japan, May 11th - 13th. I would like to try to hold a face-to-face dev meeting at Matsumoto. (@matz will also participate!) Date: 2023/05/10 (Wed.) 14:00-18:00 (The day before RubyKaigi) Location: A meeting room in [Matsumoto Performing Arts Centre](https://www.mpac-en.com/about) (in the RubyKaigi venue, detailed location TBA) ### How to participate Open to any RubyKaigi attendees who have a commit bit or who have a topic they particularly want to discuss. If you would like to participate, please comment on this ticket so that we can get a headcount. ### Call for agenda If you want to give a talk at the meeting, please raise your agenda on this ticket. I recommend you to file a ticket of your proposal ahead of time. ### Meeting format Depending on the number of talks, we plan to spend approximately half the time on the talks and the remaining time triaging and discussing open tickets. (However, the schedule may change.) ### Disclaimer We may cancel the meeting depending on the situation, such as due to COVID-19. -- https://bugs.ruby-lang.org/

33 34

[ruby-core:112398] [Ruby master Feature#19435] Expose counts for each GC reason in GC.stat
by byroot (Jean Boussier) 09 May '23

09 May '23

Issue #19435 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19435: Expose counts for each GC reason in GC.stat https://bugs.ruby-lang.org/issues/19435 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context We recently tuned the GC settings on our monolith application because we were seeing some very long GC pauses (multiple seconds) during some requests. Very early we realized that we could know how often the GC was triggered, and how long it was taking, but we had no information as to why, hence no good way to know which specific configuration to tune. As of today, the only way to get this information is to compile Ruby with debug counters, but that's not really accessible for most users, and not very suitable to be deployed in production. So we patched our Ruby to expose counters for each specific reason in `GC.stat` and this data was extremely valuable. For instance we discovered that the number 1 cause of major GC was `shady` objects, which allowed us to both better tune or GC and to drive some targeted patches to Ruby. ### Proposal We'd like to merge the patch we used on our Ruby build. It expose 8 new keys in `GC.stat`: - `:major_gc_nofree_count` - `:major_gc_oldgen_count` - `:major_gc_shady_count` - `:major_gc_newobj_count` - `:major_gc_malloc_count` - `:major_gc_oldmalloc_count` - `:minor_gc_newobj_count` - `:minor_gc_malloc_count` Some very uncommon reasons like `force` etc are ignored as they're not valuable. Also note that sometimes multiple conditions can be met to trigger GC, in such case we my increment several counters, so the sum of `major_gc_*_count` can be higher than `major_gc_count`. Proposed patch: https://github.com/ruby/ruby/pull/7250 -- https://bugs.ruby-lang.org/

6 9

[ruby-core:113050] [Ruby master Feature#19560] IO#close_on_fork= and IO#close_on_fork?
by byroot (Jean Boussier) 19 Apr '23

19 Apr '23

Issue #19560 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19560: IO#close_on_fork= and IO#close_on_fork? https://bugs.ruby-lang.org/issues/19560 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context Forking setups are extremely common in the Ruby ecosystem, as they remain the primary way to get parallelism with MRI. Generally speaking it works very well, however there are two main issues library authors and application owners need to be careful of: - Restarting threads - Closing inherited connections and other file descriptors. I believe we could make the second one much easier. ### O_CLOFORK A couple years ago, [a new flag was added to the POSIX spec: `O_CLOFORK`](https://austingroupbugs.net/view.php?id=1318). Similar to `O_CLOEXEC`, this file descriptor flag make it so the file descriptor is automatically closed upon forking. Unfortunately its support is relatively limited for now. It's supported on macOS and some relatively exotic unixes, but not in Linux nor most BSDs. [The feature was discussed on Linux mailing list](https://lore.kernel.org/lkml/20200525081626.GA16796@amd/T/#m5b8b20ea6…, but it seem to have encountered some strong opposition, so it's unclear if we can hope for it to be added. That said, I don't think it would be too hard for Ruby to shim this feature by closing all IOs with `close_on_fork?` right after fork. ### Ruby shim This can be implemented as a Ruby shim starting in Ruby 3.1 using the `Process._fork` callback ```ruby class IO def close_on_fork=(enabled) if enabled ::CloseIOOnFork::IOS[self] = true end @close_on_fork = enabled end def close_on_fork? @close_on_fork end end module CloseIOOnFork IOS = ObjectSpace::WeakMap.new def _fork pid = super if pid == 0 # child ::CloseIOOnFork::IOS.each_key do |io| io.close if io.close_on_fork? end end pid end end Process.singleton_class.prepend(CloseIOOnFork) rd, rw = IO.pipe rw.close_on_fork = true pid = fork do p rw.closed? # => true end Process.wait(pid) ``` ### Usage With such feature, many network client would mostly just need to set this flag on their sockets, and just properly handle unexpectedly closed connections, which most already do. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:112183] [Ruby master Bug#19401] [Doc] Broken links in CSV documentation
by eightbitraptor (Matthew Valentine-House) 19 Apr '23

19 Apr '23

Issue #19401 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19401: [Doc] Broken links in CSV documentation https://bugs.ruby-lang.org/issues/19401 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The link "Recipes for CSV" on [this page](https://docs.ruby-lang.org/en/master/CSV.html) points to a broken link: [https://docs.ruby-lang.org/en/master/csv/recipes/recipes_rdoc.html](https:/…. It doesn't look like the intended target `doc/csv/recipes/recipes.rdoc` is being included in the rdoc output. There isn't an equivalent HTML file generated in `.ext/html` after I run `make doc` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:111990] [Ruby master Bug#19371] Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML
by tombruijn (Tom de Bruijn) 14 Apr '23

14 Apr '23

Issue #19371 has been reported by tombruijn (Tom de Bruijn). ---------------------------------------- Bug #19371: Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML https://bugs.ruby-lang.org/issues/19371 * Author: tombruijn (Tom de Bruijn) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Summary There's an issue on Ruby versions with Psych 4 installed by default (Ruby 2.6 through 3.1) after installing the Psych gem version 5. This problem occurs when a Ruby gem has a C-extension installation script that parses a YAML string. I'm reporting it here and not with on the Psych gem repo, because it looks more like an issue with which Ruby C-extension is load during other gem's C-extension installation. ## Background I have a gem that parses a YAML string in the C-extension installation script, or it calls `Gem.configuration[:http_proxy]`, which parses the `.gemrc` file as YAML. This triggers the error mentioned below. This YAML parsing is done in the gem's `ext/extconf.rb` file. An example gem can be found in this repository: https://github.com/tombruijn/yaml-dummy-gem, see the [`ext/extconf.rb` file](https://github.com/tombruijn/yaml-dummy-gem/blob/main/ext/extconf.rb#…. ## The problem On Ruby 3.1.3 Psych version 4 is installed by default. When it parses the YAML file, it will use Psych 4. When Psych 5 is also installed on Ruby 3.1.3, it is no longer be able to parse the YAML file. The following error is raised: ``` $ bundle install Fetching https://github.com/tombruijn/yaml-dummy-gem.git Resolving dependencies... Using bundler 2.3.7 Using yaml-dummy-gem 1.0.0 from https://github.com/tombruijn/yaml-dummy-gem.git (at main@a48852d) Gem::Ext::BuildError: ERROR: Failed to build gem native extension. current directory: /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d/ext /usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r ./siteconf20230123-730-rmbnnl.rb extconf.rb /usr/local/lib/ruby/3.1.0/psych.rb:459:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x0000ffff8078c7f8 @handler=#<Psych::Handlers::DocumentStream:0x0000ffff8078c910 @stack=[], @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil, @block=#<Proc:0x0000ffff8078c848 /usr/local/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError) parser.parse yaml, filename ^^^^^^ from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse' from extconf.rb:3:in `<main>' extconf failed, exit code 1 Gem files will remain installed in /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d for inspection. Results logged to /usr/local/bundle/bundler/gems/extensions/aarch64-linux/3.1.0/yaml-dummy-gem-a48852dac33d/gem_make.out /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run' /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build' /usr/local/lib/ruby/3.1.0/tempfile.rb:317:in `open' /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in build_extensions' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions' /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions' /usr/local/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions' /usr/local/lib/ruby/3.1.0/bundler/source/path/installer.rb:34:in `post_install' /usr/local/lib/ruby/3.1.0/bundler/source/path.rb:244:in `generate_bin' /usr/local/lib/ruby/3.1.0/bundler/source/git.rb:194:in `install' /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install' /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in `install_from_spec' /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in `do_install' /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in create_threads' An error occurred while installing yaml-dummy-gem (1.0.0), and Bundler cannot continue. In Gemfile: yaml-dummy-gem ``` ## Debugging results The error is raised because the `Psych::Parser#parse` method cannot be found. In Psych version 4, this method is [defined by the Psych C-extension](https://github.com/ruby/psych/blob/2c3708e0a483c6d44ebddaff0b5…. In Psych version 5 the `parse` method is [defined in the gem's Ruby code](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb52d…. This method calls a [C function registered as the private `_native_parse` method](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb5…, which is the renamed version of the `parse` C-function in Psych version 4. From what I can tell, the Psych version 4 C-extension is no longer loaded when Psych version 5 is installed in this scenario. There is a mix up in which Psych gem version's C-extension is loaded during my dummy gem's C-extension installation. It load the Psych 4 Ruby code, with the Psych 5 C-extension∂. I confirmed this by modifying the standard installed Psych gem's code on Ruby 3.1 (Docker image `ruby:3.1`), with the following the change, which prints `true` on error. This means the Psych 4 gem has the Psych 5 C-extension loaded where `_native_parse` is defined. ```diff diff --git lib/psych.rb lib/psych.rb index 42d79ef..1a690d2 100644 --- lib/psych.rb +++ lib/psych.rb @@ -452,6 +452,9 @@ def self.parser def self.parse_stream yaml, filename: nil, &block if block_given? parser = Psych::Parser.new(Handlers::DocumentStream.new(&block)) + # This returns `true`, but it should be `false`. The `_native_parse` + # method is defined in the Psych 5 C-extension, not Psych 4. + puts parser.respond_to? :_native_parse, true # => true parser.parse yaml, filename ``` This error only occurs during the gem's extension installation in `ext/extconf.rb`. If the gem parses YAML when a Ruby app is running, it will not produce the same error with Psych version 5 installed. This issue does not occur on Ruby 3.2, where Psych version 5 is installed by default. I have confirmed this error occurs on the latest patch releases of the following Ruby versions: 3.1, 3.0, 2.7 and 2.6. ``` $ gem list psych psych (5.0.2, default: 4.0.3) ``` ## Code to reproduce Here is a basic Ruby gem that only parses a YAML file during extension installation: https://github.com/tombruijn/yaml-dummy-gem Here is a small project that triggers the error: https://github.com/tombruijn/yaml-dummy-ruby-app A GitHub actions workflow shows the results for all affected Ruby versions: https://github.com/tombruijn/yaml-dummy-ruby-app/actions/runs/3969088933 The [example app repo](https://github.com/tombruijn/yaml-dummy-ruby-app) also has instructions to run the example app locally. Please follow the instructions in the README to see the error. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:112866] [Ruby master Feature#19528] `JSON.load` defaults are surprising (`create_additions: true`)
by byroot (Jean Boussier) 14 Apr '23

14 Apr '23

Issue #19528 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19528: `JSON.load` defaults are surprising (`create_additions: true`) https://bugs.ruby-lang.org/issues/19528 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- I'm not sure if it was actually intended, but there's some tacit naming convention for serializers in Ruby to use `load` and `dump` as methods, likely inspired from `Marshal` and `YAML`. Because of this it's extremely common to see code that uses `JSON.load` expecting a simple, no surprise, and safe JSON parsing. However that's `JSON.parse`. `JSON.load` has this very surprising behavior (albeit perfectly documented), of de-serializing more complex types: ```ruby >> JSON.load('{ "json_class": "String", "raw": [72, 101, 108, 108, 111] }') => "Hello" ``` It's particularly weird because aside from the `String` extension that is eagerly defined, for other types you have to `require "json/add/core"`. Seasoned Ruby developers know about this of course, and [it is banned by various linters](https://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Security/JSONLoad), but it keeps popping regularly in gems security releases and such. ### Proposal Assuming entirely removing this feature is not an option, I think `json 2.x` should warn when this feature is actually being used, and `json 3.x` should disable it by default and require users to explicitly use `JSON.load(str, create_additions: true)` to keep the old behavior. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:113028] [Ruby master Bug#19551] Backport commits for CI failures
by hsbt (Hiroshi SHIBATA) 14 Apr '23

14 Apr '23

Issue #19551 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19551: Backport commits for CI failures https://bugs.ruby-lang.org/issues/19551 * Author: hsbt (Hiroshi SHIBATA) * Status: Closed * Priority: Normal * Backport: 2.7: REQUIRED, 3.0: REQUIRED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- I created PRs to fix CI failure for stable or old stable branches. * [Fix CI for ruby_2_7](https://github.com/ruby/ruby/pull/7259) * [Fix CI for ruby_3_0](https://github.com/ruby/ruby/pull/7258) * [Fix CI for ruby_3_2](https://github.com/ruby/ruby/pull/7604) Please merge them before release time. ---- Can I merge PR for "Fix CI failure" myself from April 2023? I can maintain to keep CI status with my full-time job. It helps to release work for all of branch maintainers. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:113051] [Ruby master Feature#19561] ObjectSpace::WeakMap#delete and ObjectSpace::WeakKeyMap#delete
by byroot (Jean Boussier) 14 Apr '23

14 Apr '23

Issue #19561 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19561: ObjectSpace::WeakMap#delete and ObjectSpace::WeakKeyMap#delete https://bugs.ruby-lang.org/issues/19561 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- I just realized in https://bugs.ruby-lang.org/issues/19560#Ruby-shim that there's no way to remove elements from either WeakMaps. I think they could both benefit from a `delete` method. ### Use case For example, to keep track of a subset of instances of a class, you may need to remove some from the list when a condition changes: ```ruby class Connection CLOSE_ON_FORK = ObjectSpace::WeakMap.new class << self def after_fork CLOSE_ON_FORK.each_key do |connection| connection.close if connection.close_on_fork? end end end def close_on_fork=(enabled) if enabled CLOSE_ON_FORK[self] = true else CLOSE_ON_FORK.delete(self) end @close_on_fork = enabled end def close_on_fork? @close_on_fork end end module CloseOnFork IOS = ObjectSpace::WeakMap.new def _fork pid = super if pid == 0 # child ::Connection.after_fork end pid end end Process.singleton_class.prepend(CloseIOOnFork) ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:112933] [Ruby master Feature#19538] Performance warnings
by byroot (Jean Boussier) 13 Apr '23

13 Apr '23

Issue #19538 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19538: Performance warnings https://bugs.ruby-lang.org/issues/19538 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- Suggested by @Eregon. There are program behaviors that are supported, but that we know aren't good for performance, however it's hard for users to know about them. Now that we have warning categories, we could add a `:performance` category to allow the VM to emit warning in some situations. The category would be disabled by default, and users interested in optimizing their program could turn it on in development. ```ruby Warning[:performance] = true ``` -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113060] [Ruby master Bug#19564] Range.cover? fails for Range wrapped in SimpleDelegator
by PeteMichaud (Pete Michaud) 13 Apr '23

13 Apr '23

Issue #19564 has been reported by PeteMichaud (Pete Michaud). ---------------------------------------- Bug #19564: Range.cover? fails for Range wrapped in SimpleDelegator https://bugs.ruby-lang.org/issues/19564 * Author: PeteMichaud (Pete Michaud) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-darwin22] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ```ruby test 'built-in cover? method is broken for SimpleDelegator' do class TimeRangeDelegator < SimpleDelegator end t1 = Time.now t2 = t1+1.day t3 = t1+1.week r_long = t1..t3 r_short = t1..t2 assert r_long.cover?(r_short) #passes correctly, assert == true r_long_delegated = TimeRangeDelegator.new(r_long) r_short_delegated = TimeRangeDelegator.new(r_short) assert r_long_delegated.cover?(r_short_delegated) #fails incorrectly, assert == false end ``` -- https://bugs.ruby-lang.org/

3 2