February 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112398] [Ruby master Feature#19435] Expose counts for each GC reason in GC.stat
by byroot (Jean Boussier) 09 May '23

09 May '23

Issue #19435 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19435: Expose counts for each GC reason in GC.stat https://bugs.ruby-lang.org/issues/19435 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context We recently tuned the GC settings on our monolith application because we were seeing some very long GC pauses (multiple seconds) during some requests. Very early we realized that we could know how often the GC was triggered, and how long it was taking, but we had no information as to why, hence no good way to know which specific configuration to tune. As of today, the only way to get this information is to compile Ruby with debug counters, but that's not really accessible for most users, and not very suitable to be deployed in production. So we patched our Ruby to expose counters for each specific reason in `GC.stat` and this data was extremely valuable. For instance we discovered that the number 1 cause of major GC was `shady` objects, which allowed us to both better tune or GC and to drive some targeted patches to Ruby. ### Proposal We'd like to merge the patch we used on our Ruby build. It expose 8 new keys in `GC.stat`: - `:major_gc_nofree_count` - `:major_gc_oldgen_count` - `:major_gc_shady_count` - `:major_gc_newobj_count` - `:major_gc_malloc_count` - `:major_gc_oldmalloc_count` - `:minor_gc_newobj_count` - `:minor_gc_malloc_count` Some very uncommon reasons like `force` etc are ignored as they're not valuable. Also note that sometimes multiple conditions can be met to trigger GC, in such case we my increment several counters, so the sum of `major_gc_*_count` can be higher than `major_gc_count`. Proposed patch: https://github.com/ruby/ruby/pull/7250 -- https://bugs.ruby-lang.org/

6 9

[ruby-core:112183] [Ruby master Bug#19401] [Doc] Broken links in CSV documentation
by eightbitraptor (Matthew Valentine-House) 19 Apr '23

19 Apr '23

Issue #19401 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19401: [Doc] Broken links in CSV documentation https://bugs.ruby-lang.org/issues/19401 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The link "Recipes for CSV" on [this page](https://docs.ruby-lang.org/en/master/CSV.html) points to a broken link: [https://docs.ruby-lang.org/en/master/csv/recipes/recipes_rdoc.html](https:/…. It doesn't look like the intended target `doc/csv/recipes/recipes.rdoc` is being included in the rdoc output. There isn't an equivalent HTML file generated in `.ext/html` after I run `make doc` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:111990] [Ruby master Bug#19371] Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML
by tombruijn (Tom de Bruijn) 14 Apr '23

14 Apr '23

Issue #19371 has been reported by tombruijn (Tom de Bruijn). ---------------------------------------- Bug #19371: Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML https://bugs.ruby-lang.org/issues/19371 * Author: tombruijn (Tom de Bruijn) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Summary There's an issue on Ruby versions with Psych 4 installed by default (Ruby 2.6 through 3.1) after installing the Psych gem version 5. This problem occurs when a Ruby gem has a C-extension installation script that parses a YAML string. I'm reporting it here and not with on the Psych gem repo, because it looks more like an issue with which Ruby C-extension is load during other gem's C-extension installation. ## Background I have a gem that parses a YAML string in the C-extension installation script, or it calls `Gem.configuration[:http_proxy]`, which parses the `.gemrc` file as YAML. This triggers the error mentioned below. This YAML parsing is done in the gem's `ext/extconf.rb` file. An example gem can be found in this repository: https://github.com/tombruijn/yaml-dummy-gem, see the [`ext/extconf.rb` file](https://github.com/tombruijn/yaml-dummy-gem/blob/main/ext/extconf.rb#…. ## The problem On Ruby 3.1.3 Psych version 4 is installed by default. When it parses the YAML file, it will use Psych 4. When Psych 5 is also installed on Ruby 3.1.3, it is no longer be able to parse the YAML file. The following error is raised: ``` $ bundle install Fetching https://github.com/tombruijn/yaml-dummy-gem.git Resolving dependencies... Using bundler 2.3.7 Using yaml-dummy-gem 1.0.0 from https://github.com/tombruijn/yaml-dummy-gem.git (at main@a48852d) Gem::Ext::BuildError: ERROR: Failed to build gem native extension. current directory: /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d/ext /usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r ./siteconf20230123-730-rmbnnl.rb extconf.rb /usr/local/lib/ruby/3.1.0/psych.rb:459:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x0000ffff8078c7f8 @handler=#<Psych::Handlers::DocumentStream:0x0000ffff8078c910 @stack=[], @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil, @block=#<Proc:0x0000ffff8078c848 /usr/local/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError) parser.parse yaml, filename ^^^^^^ from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse' from extconf.rb:3:in `<main>' extconf failed, exit code 1 Gem files will remain installed in /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d for inspection. Results logged to /usr/local/bundle/bundler/gems/extensions/aarch64-linux/3.1.0/yaml-dummy-gem-a48852dac33d/gem_make.out /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run' /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build' /usr/local/lib/ruby/3.1.0/tempfile.rb:317:in `open' /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in build_extensions' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each' /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions' /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions' /usr/local/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions' /usr/local/lib/ruby/3.1.0/bundler/source/path/installer.rb:34:in `post_install' /usr/local/lib/ruby/3.1.0/bundler/source/path.rb:244:in `generate_bin' /usr/local/lib/ruby/3.1.0/bundler/source/git.rb:194:in `install' /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install' /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in `install_from_spec' /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in `do_install' /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue' /usr/local/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in create_threads' An error occurred while installing yaml-dummy-gem (1.0.0), and Bundler cannot continue. In Gemfile: yaml-dummy-gem ``` ## Debugging results The error is raised because the `Psych::Parser#parse` method cannot be found. In Psych version 4, this method is [defined by the Psych C-extension](https://github.com/ruby/psych/blob/2c3708e0a483c6d44ebddaff0b5…. In Psych version 5 the `parse` method is [defined in the gem's Ruby code](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb52d…. This method calls a [C function registered as the private `_native_parse` method](https://github.com/ruby/psych/blob/1f23e6e7f0ab4a6efab598c1ee528bb5…, which is the renamed version of the `parse` C-function in Psych version 4. From what I can tell, the Psych version 4 C-extension is no longer loaded when Psych version 5 is installed in this scenario. There is a mix up in which Psych gem version's C-extension is loaded during my dummy gem's C-extension installation. It load the Psych 4 Ruby code, with the Psych 5 C-extension∂. I confirmed this by modifying the standard installed Psych gem's code on Ruby 3.1 (Docker image `ruby:3.1`), with the following the change, which prints `true` on error. This means the Psych 4 gem has the Psych 5 C-extension loaded where `_native_parse` is defined. ```diff diff --git lib/psych.rb lib/psych.rb index 42d79ef..1a690d2 100644 --- lib/psych.rb +++ lib/psych.rb @@ -452,6 +452,9 @@ def self.parser def self.parse_stream yaml, filename: nil, &block if block_given? parser = Psych::Parser.new(Handlers::DocumentStream.new(&block)) + # This returns `true`, but it should be `false`. The `_native_parse` + # method is defined in the Psych 5 C-extension, not Psych 4. + puts parser.respond_to? :_native_parse, true # => true parser.parse yaml, filename ``` This error only occurs during the gem's extension installation in `ext/extconf.rb`. If the gem parses YAML when a Ruby app is running, it will not produce the same error with Psych version 5 installed. This issue does not occur on Ruby 3.2, where Psych version 5 is installed by default. I have confirmed this error occurs on the latest patch releases of the following Ruby versions: 3.1, 3.0, 2.7 and 2.6. ``` $ gem list psych psych (5.0.2, default: 4.0.3) ``` ## Code to reproduce Here is a basic Ruby gem that only parses a YAML file during extension installation: https://github.com/tombruijn/yaml-dummy-gem Here is a small project that triggers the error: https://github.com/tombruijn/yaml-dummy-ruby-app A GitHub actions workflow shows the results for all affected Ruby versions: https://github.com/tombruijn/yaml-dummy-ruby-app/actions/runs/3969088933 The [example app repo](https://github.com/tombruijn/yaml-dummy-ruby-app) also has instructions to run the example app locally. Please follow the instructions in the README to see the error. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:111780] [Ruby master Bug#19331] --enable-rpath results in incorrect RPATH in Ruby 3.1.3
by Kulikjak (Jakub Kulik) 12 Apr '23

12 Apr '23

Issue #19331 has been reported by Kulikjak (Jakub Kulik). ---------------------------------------- Bug #19331: --enable-rpath results in incorrect RPATH in Ruby 3.1.3 https://bugs.ruby-lang.org/issues/19331 * Author: Kulikjak (Jakub Kulik) * Status: Open * Priority: Normal * ruby -v: 3.1.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I just updated Ruby from 3.1.2 to 3.1.3 and found out that all .so libraries from the enc directory have wrong RPATH/RUNPATH (when building with `--enable-rpath`). I first hit this on Solaris, but my testing on Linux resulted in the same thing. It can be easily reproduced with the following: ``` wget http://cache.ruby-lang.org/pub/ruby/3.1/ruby-3.1.3.tar.gz tar xzf ruby-3.1.3.tar.gz cd ruby-3.1.3 ./configure --prefix=/usr/ruby/3.1 --mandir=/usr/ruby/3.1/share/man --bindir=/usr/ruby/3.1/bin --sbindir=/usr/ruby/3.1/sbin --libdir=/usr/ruby/3.1/lib/amd64 --with-rubylibprefix=/usr/ruby/3.1/lib/ruby --enable-shared --enable-rpath /usr/bin/make -j 16 -l 32 objdump -x ./.ext/x86_64-linux/enc/windows_31j.so | grep RPATH ``` While previously this resulted in the following output: `RPATH /usr/ruby/3.1/lib/amd64` in 3.1.3 it is wrongly set to the build directory: `RPATH /home/jkulik/rubytest/ruby-3.1.3` I tried looking for some obvious change but didn't find the core of this issue yet. All I found out is that generated `enc.mk` differs in `prefix` and `libdir` ``` -prefix = /usr/ruby/3.1 +prefix = /home/jkulik/rubytest/ruby-3.1.3 exec_prefix = $(prefix) -libdir = $(exec_prefix)/lib/amd64 +libdir = /home/jkulik/rubytest/ruby-3.1.3 ``` `rbconfig.rb` is similar in both versions, but when I print out `CONFIG` in `make_encmake.rb` right after `load File.expand_path("lib/mkmf.rb", dir)`, I can see the differences from above, so it's probably something in the `lib/mkmf.rb` file that changes that. I am seeing the same issue in the current latest ruby 3.1 branch cloned from github. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:111956] [Ruby master Bug#19363] Fix rb_transient_heap_mark: wrong header (T_STRUCT) segfault
by bkuhlmann (Brooke Kuhlmann) 10 Apr '23

10 Apr '23

Issue #19363 has been reported by bkuhlmann (Brooke Kuhlmann). ---------------------------------------- Bug #19363: Fix rb_transient_heap_mark: wrong header (T_STRUCT) segfault https://bugs.ruby-lang.org/issues/19363 * Author: bkuhlmann (Brooke Kuhlmann) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) +YJIT [arm64-darwin22.2.0] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ## Overview Hello. 👋 I'm hitting an issue where my build is constantly failing with a segfault. The following is a snippet taken from my local machine with YJIT enabled (see attachments for details): ``` /Users/bkuhlmann/.cache/frum/versions/3.2.0/lib/ruby/gems/3.2.0/gems/puma-6.0.2/lib/puma/runner.rb: [BUG] rb_transient_heap_mark: wrong header, T_STRUCT (0x0000000109ea98a0) ruby 3.2.0 (2022-12-25 revision a528908271) +YJIT [arm64-darwin22.2.0] ``` The closest issue I could find that might be related to this issue (but not sure) is this issue: #15358. ## Steps to Recreate You should be able to quickly recreate this issue via these steps: - Download/clone my [Hemo](https://github.com/bkuhlmann/hemo) project. - Run the setup steps. - Run the test suite by running `bin/rspec`. If you need an example of the same segfault (but not on my macOS machine), you can see the same segfault via my [Circle CI Build](https://app.circleci.com/pipelines/github/bkuhlmann/hemo/11/workflow…. My Circle CI build is using my [Docker Alpine Linux Ruby](https://www.alchemists.io/projects/docker-alpine-ruby) image which might be of interest as well. This Docker image is also built with YJIT enabled. Interestingly, is if you were to run the test suite with `bin/guard` instead of `bin/rspec` then the segfault doesn't occur. ## Environment ``` ruby 3.2.0 (2022-12-25 revision a528908271) +YJIT [arm64-darwin22.2.0] 1.43.0 (using Parser 3.2.0.0, rubocop-ast 1.24.1, running on ruby 3.2.0) [arm64-darwin22.2.0] - rubocop-performance 1.15.2 - rubocop-rake 0.6.0 - rubocop-rspec 2.18.1 - rubocop-sequel 0.3.4 - rubocop-thread_safety 0.4.4 ``` ---Files-------------------------------- segfault.txt (237 KB) ruby-2023-01-21-113841.ips (19.6 KB) -- https://bugs.ruby-lang.org/

6 8

[ruby-core:112445] [Ruby master Bug#19441] Closing an IO#dup behaviour
by stac47 (Laurent Stacul) 03 Apr '23

03 Apr '23

Issue #19441 has been reported by stac47 (Laurent Stacul). ---------------------------------------- Bug #19441: Closing an IO#dup behaviour https://bugs.ruby-lang.org/issues/19441 * Author: stac47 (Laurent Stacul) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-darwin20] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello amazing ruby folks! I classified that ticket as a "Bug" although it is perhaps something I did not understand. Here is a ruby session: ``` ruby irb(main):001:0> file = Tempfile.new => #<File:/var/folders/m2/bljzrgq160vbf0vk466k_7gw0000gn/T/20230216-39664-davfmj> irb(main):002:0> file.fileno => 9 irb(main):003:0> dup = file.dup => #<File:/var/folders/m2/bljzrgq160vbf0vk466k_7gw0000gn/T/20230216-39664-davfmj> irb(main):004:0> dup.fileno => 10 irb(main):005:0> dup.close => nil irb(main):006:0> dup.closed? => false irb(main):007:0> file.closed? => true ``` The two last lines are unexpected to me. I would have expected the converse: ```ruby irb(main):006:0> dup.closed? => true irb(main):007:0> file.closed? => false ``` I tried this scenario in latest ruby but also in ruby 2.7. Thanks in advance for helping me to understand this behaviour. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:112218] [Ruby master Bug#19414] uninitialized constant URI::WSS in 3.0.X and 3.1.X
by noraj (Alexandre ZANNI) 26 Mar '23

26 Mar '23

Issue #19414 has been reported by noraj (Alexandre ZANNI). ---------------------------------------- Bug #19414: uninitialized constant URI::WSS in 3.0.X and 3.1.X https://bugs.ruby-lang.org/issues/19414 * Author: noraj (Alexandre ZANNI) * Status: Open * Priority: Normal * ruby -v: 3.1.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I have a library called [ctf-party](https://github.com/noraj/ctf-party), [making use of URI:WSS](https://github.com/noraj/ctf-party/blob/ef3b82cc1668325b782246a9bb…, and it's making my [CI pipeline fails](https://github.com/noraj/ctf-party/actions/runs/4059895670). I'm able to reproduce it locally (`bundle exec rake test`), working with ruby 3.2.0 and not with 3.1.3. The error is: ``` irb(main):001:0> require 'uri' => true irb(main):002:0> URI::WS => URI::WS irb(main):003:0> URI::WSS (irb):3:in `<main>': uninitialized constant URI::WSS (NameError) from /home/noraj/.asdf/installs/ruby/3.1.3/lib/ruby/gems/3.1.0/gems/irb-1.4.1/exe/irb:11:in `<top (required)>' from /home/noraj/.asdf/installs/ruby/3.1.3/bin/irb:25:in `load' from /home/noraj/.asdf/installs/ruby/3.1.3/bin/irb:25:in `<main>' ``` I don't understand why is that happening since [it exists in the official documentation](https://ruby-doc.org/3.1.3/stdlibs/uri/URI/WSS.html) and the [code looks ok to me on tag 3.1.3](https://github.com/ruby/ruby/blob/v3_1_3/lib/uri/wss.rb) which is [strictly identical to the code on tag 3.2.0](https://github.com/ruby/ruby/blob/v3_2_0/lib/uri/wss.rb). It works if I require the file explicitly: ``` irb(main):004:0> require 'uri/wss' => true irb(main):005:0> URI::WSS => URI::WSS ``` The issue come from the fact that [in 3.2.0 `'uri/wss'` is required](https://github.com/ruby/ruby/blob/v3_2_0/lib/uri.rb#L104) but [not in 3.1.3](https://github.com/ruby/ruby/blob/v3_1_3/lib/uri.rb#L103). In [3.0.5](https://github.com/ruby/ruby/blob/v3_0_5/lib/uri.rb) it is even worse as neither ws and wss are required. -- https://bugs.ruby-lang.org/

5 6

[ruby-core:112433] [Ruby master Bug#19439] Marshal.load doesn't load Regexp instance variables
by andrykonchin (Andrew Konchin) 26 Mar '23

26 Mar '23

Issue #19439 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #19439: Marshal.load doesn't load Regexp instance variables https://bugs.ruby-lang.org/issues/19439 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal * ruby -v: 3.1.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hello, I've noticed this strange behaviour: ```ruby source_object = Regexp.new("a") source_object.instance_variable_set(:@foo, :bar) regexp = Marshal.load(Marshal.dump(source_object)) regexp.instance_variables # => [] ``` I would expect instance variables to be loaded -- https://bugs.ruby-lang.org/

5 7

[ruby-core:111332] [Ruby master Bug#19242] Circular cause by Marshal
by nobu (Nobuyoshi Nakada) 25 Mar '23

25 Mar '23

Issue #19242 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19242: Circular cause by Marshal https://bugs.ruby-lang.org/issues/19242 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 2.7: REQUIRED, 3.0: REQUIRED, 3.1: REQUIRED ---------------------------------------- Crafted marshaled data can creates an `Exception` object with circular `cause`. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:111333] [Ruby master Bug#19243] Windows: Dir.home returns string in wrong encoding
by larskanis (Lars Kanis) 25 Mar '23

25 Mar '23

Issue #19243 has been reported by larskanis (Lars Kanis). ---------------------------------------- Bug #19243: Windows: Dir.home returns string in wrong encoding https://bugs.ruby-lang.org/issues/19243 * Author: larskanis (Lars Kanis) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x64-mingw-ucrt] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Create a new local user on Windows called "kanisä". Then login to the new user and run in cmd: ``` C:\Users\kanisä>ruby -e "p Dir.home" "C:/Users/kanis\xE4" C:\Users\kanisä>ruby -e "p Dir.home.encoding" #<Encoding:UTF-8> C:\Users\kanisä>ruby -e "p Dir.home.encode('utf-8', 'iso-8859-15')" "C:/Users/kanisä" C:\Users\kanisä>echo %USERPROFILE% C:\Users\kanisä ``` `Dir.home` should encode the path in UTF-8. Instead the string is labeled as UTF-8 but bytes are encoded in `iso-8859-15`. All versions between ruby-3.0 and current master branch are affected. -- https://bugs.ruby-lang.org/

2 3