August 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114564] [Ruby master Feature#18440] YJIT is enabled if any YJIT tuning options are set
by byroot (Jean Boussier) 26 Aug '23

26 Aug '23

Issue #18440 has been updated by byroot (Jean Boussier). For what it's worth, I would expect `--yjit-*` options not to turn on YJIT. I agree that it's surprising, I would expect only `--yjit` or `RUBY_YJIT_ENABLE=1` to actually enable it. ---------------------------------------- Feature #18440: YJIT is enabled if any YJIT tuning options are set https://bugs.ruby-lang.org/issues/18440#change-104378 * Author: georgeclaghorn (George Claghorn) * Status: Feedback * Priority: Normal ---------------------------------------- I was testing YJIT in a Rails app with `RUBYOPT="--yjit --yjit-exec-mem-size=32"`. I saw some weird issues around Rails view caching, so I attempted to temporarily disable YJIT by removing `--yjit` and leaving the tuning options in place (`RUBYOPT="--yjit-exec-mem-size=32"`). However, YJIT remained enabled until I also removed `--yjit-exec-mem-size=32`. ``` $ ruby -e 'puts RubyVM::YJIT.runtime_stats.inspect' nil $ ruby --yjit-exec-mem-size -e 'puts RubyVM::YJIT.runtime_stats.inspect' {:inline_code_size=>378495, :outlined_code_size=>311079} ``` I expected removing `--yjit` to disable YJIT and was surprised by this. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114562] [Ruby master Bug#18789] make test-bundled-gems failed after make install
by jeremyevans0 (Jeremy Evans) 25 Aug '23

25 Aug '23

Issue #18789 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Assigned to Feedback @znz I tried with current Ruby master and I could not reproduce the issue. Is it still a problem in your environment? ---------------------------------------- Bug #18789: make test-bundled-gems failed after make install https://bugs.ruby-lang.org/issues/18789#change-104374 * Author: znz (Kazuhiro NISHIYAMA) * Status: Feedback * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * ruby -v: ruby 3.2.0dev (2022-05-17T22:12:44Z master e2bad65eab) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- On snapshot-master CI, `make test-bundled-gems` failed. https://github.com/ruby/actions/runs/6476724416?check_suite_focus=true#step… ``` ./tool/test-bundled-gems.rb:10:in `realpath': No such file or directory @ realpath_rec - /home/runner/work/actions/actions/snapshot-master/.bundle/bin/rake (Errno::ENOENT) from ./tool/test-bundled-gems.rb:10:in `<main>' make: *** [uncommon.mk:1410: test-bundled-gems-run] Error 1 ``` I investigated it, it causes after `make install`. How to reproduce: 1. Run configure 2. `make install` 3. `make test-bundled-gems` or In already built directory 1. `rm -rf path/to/srcdir/.bundle` 2. `make install` 3. `make test-bundled-gems` If I run `rm -rf path/to/srcdir/.bundle` and `make test-bundled-gems` without `make install`, `.bundle/bin/rake` exists. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114561] [Ruby master Feature#18812] Add ability to trace exit locations for YJIT
by k0kubun (Takashi Kokubun) 25 Aug '23

25 Aug '23

Issue #18812 has been updated by k0kubun (Takashi Kokubun). Status changed from Open to Closed https://github.com/ruby/ruby/pull/5970 has already been merged. ---------------------------------------- Feature #18812: Add ability to trace exit locations for YJIT https://bugs.ruby-lang.org/issues/18812#change-104370 * Author: eileencodes (Eileen Uchitelle) * Status: Closed * Priority: Normal ---------------------------------------- Currently, when running yjit with `--yjit-stats` you are able to see method call exit reasons and the top 20 most frequent exits. This is useful to know where to spend time investigating whether an exit should be fixed, but in a larger codebase like Rails, it's next to impossible to know what the Ruby code that is exiting looks like. Aaron Patterson and I aim to fix that with the addition of `--yjit-trace-exits` option and feature. When running with `--yjit-stats` turned on Ruby code can inform the user what the most common exits are. While this is useful information it doesn't tell you the source location of the code that exited or what the code that exited looks like. This change intends to fix that. To use the feature, run yjit with `--yjit-stats` and `--yjit-trace-exits`, which will record the backtrace for every exit that occurs. Users must save the output of `RubyVM::YJIT.exit_locations` to a dump file. That file can then be read by StackProf to see the code that exited and the reason. *Example usage:* Given the following script, we write to a file called `concat_array.dump` the results of `RubyVM::YJIT.exit_locations`. ```ruby def concat_array ["t", "r", *x = "u", "e"].join end 1000.times do concat_array end File.write("concat_array.dump", Marshal.dump(RubyVM::YJIT.exit_locations)) ``` When we run the file with this branch and the appropriate flags the stacktrace will be recorded. Note Stackprof needs to be installed or you need to point to the library directly. ``` ./ruby --yjit --yjit-call-threshold=1 --yjit-stats --yjit-trace-exits -I/Users/eileencodes/open_source/stackprof/lib test.rb ``` We can then read the dump file with Stackprof: ``` ./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump ``` Results will look similar to the following: ``` ================================== Mode: () Samples: 1817 (0.00% miss rate) GC: 0 (0.00%) ================================== TOTAL (pct) SAMPLES (pct) FRAME 1001 (55.1%) 1001 (55.1%) concatarray 335 (18.4%) 335 (18.4%) invokeblock 178 (9.8%) 178 (9.8%) send 140 (7.7%) 140 (7.7%) opt_getinlinecache ...etc... ``` Simply inspecting the `concatarray` method will give `SOURCE UNAVAILABLE` because the source is insns.def. ``` ./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump --method concatarray ``` Result: ``` concatarray (nonexistent.def:1) samples: 1001 self (55.1%) / 1001 total (55.1%) callers: 1000 ( 99.9%) Object#concat_array 1 ( 0.1%) Gem.suffixes callees (0 total): code: SOURCE UNAVAILABLE ``` However if we go deeper to the callee we can see the exact source of the `concatarray` exit. ``` ./ruby -I/Users/eileencodes/open_source/stackprof/lib/ /Users/eileencodes/open_source/stackprof/bin/stackprof --text concat_array.dump --method Object#concat_array ``` ``` Object#concat_array (/Users/eileencodes/open_source/rust_ruby/test.rb:1) samples: 0 self (0.0%) / 1000 total (55.0%) callers: 1000 ( 100.0%) block in <main> callees (1000 total): 1000 ( 100.0%) concatarray code: | 1 | def concat_array 1000 (55.0%) | 2 | ["t", "r", *x = "u", "e"].join | 3 | end ``` The `--walk` option is recommended for this feature as it make it easier to traverse the tree of exits. *Goals of this feature:* This feature is meant to give more information when working on YJIT. The idea is that if we know what code is exiting we can decide what areas to prioritize when fixing exits. In some cases this means adding prioritizing avoiding certain exits in yjit. In more complex cases it might mean changing the Ruby code to be more performant when run with yjit. Ultimately the more information we have about what code is exiting AND why, the better we can make yjit. *Known limitations:* * Due to tracing exits, running this on large codebases like Rails can be quite slow. * On complex methods it can still be difficult to pinpoint the exact cause of an exit. * Stackprof is a requirement to to view the backtrace information from the dump file PR https://github.com/ruby/ruby/pull/5970 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114560] [Ruby master Bug#18080] Syntax error on one-line pattern matching
by kddnewton (Kevin Newton) 25 Aug '23

25 Aug '23

Issue #18080 has been updated by kddnewton (Kevin Newton). As Mike said, there's no issue with this for YARP. ---------------------------------------- Bug #18080: Syntax error on one-line pattern matching https://bugs.ruby-lang.org/issues/18080#change-104369 * Author: ko1 (Koichi Sasada) * Status: Open * Priority: Normal * ruby -v: 3.1.0dev * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- One line pattern matching with a method return value with parameters which are not surrounded by parenthesis raises syntax error. I think it is not intentional, but nobu said it's hard to support because of parse.y limitation. ```ruby p do end => a p a #=> nil p(1) do end => a p a #=> 1 p 1 do end => a #=> # syntax error, unexpected =>, expecting end-of-input # end => a # ^~ p 1 do end in a #=> # syntax error, unexpected `in', expecting end-of-input # end in a # ^~ ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114559] [Ruby master Bug#18783] OptionParser should recognize "-" as an optional argument
by jeremyevans0 (Jeremy Evans) 25 Aug '23

25 Aug '23

Issue #18783 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Open to Closed Merged at commit:419ad1e13e6287d0b7a6ba1dfeb485d2f889bf9e ---------------------------------------- Bug #18783: OptionParser should recognize "-" as an optional argument https://bugs.ruby-lang.org/issues/18783#change-104368 * Author: konsolebox (K B) * Status: Closed * Priority: Normal * ruby -v: ruby 2.7.4p191, ruby 3.1.2p20 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- ``` ruby require 'optparse' optarg = nil OptionParser.new do |parser| parser.on("-o", "--opt [ARG]", "Option with optional argument") do |arg| optarg = arg end end.parse! puts "#{optarg.inspect}, #{ARGV.inspect}" ``` Executing the code above with `-o -` as arguments outputs `nil, ["-"]` when it should produce `"-", []` instead. OptionParser recognizes `-` as a general non-option argument but it doesn't do the same when `-` follows an option with an optional argument. In my shell script, I consider options as `-?*` (glob) instead of `-*` so `-` is also considered as a valid argument. Perhaps OptionParser should do the same. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114558] [Ruby master Bug#18601] Invalid byte sequences in Big5 encodings
by jeremyevans0 (Jeremy Evans) 25 Aug '23

25 Aug '23

Issue #18601 has been updated by jeremyevans0 (Jeremy Evans). @duerst ping. ---------------------------------------- Bug #18601: Invalid byte sequences in Big5 encodings https://bugs.ruby-lang.org/issues/18601#change-104366 * Author: janosch-x (Janosch Müller) * Status: Open * Priority: Normal * Assignee: duerst (Martin Dürst) * ruby -v: any * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- I encoded all unicode codepoints in all encodings: ``` full_string = ((0..0xD7FF).to_a + (0xE000..0x10FFFF).to_a).pack('U*'); 1 uniq_encodings = Encoding.name_list - Encoding.aliases.keys - %w[locale external filesystem internal] encoded_strings = uniq_encodings.map do |enc| full_string.encode(enc, invalid: :replace, undef: :replace, replace: '') rescue => e puts e end; 1 ``` This prints about 10 "converter not found" errors, such as `code converter not found (UTF-8 to UTF-7)`, but I guess this is expected. Some of the converters seem to output invalid strings, though: ``` encoded_strings.each do |str| str&.codepoints rescue => e puts e end; 1 ``` This will print `invalid byte sequence in {Big5HKSCS,Big5-UAO,CP950,CP951}`. Looking for example at the generated CP950 string, 8031 of its 25342 characters are invalid, spread across 2017 distinct ranges in the string. The invalid characters' codepoints are all in the range of 0x81..0xFE. Is this a bug? I would expect `String#encode` with `invalid: :replace, undef: :replace` not to create invalid byte sequences, but maybe I am misunderstanding these encodings and this is an unavoidable issue? CC @duerst -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114557] [Ruby master Bug#18454] YJIT slowing down key Discourse benchmarks
by k0kubun (Takashi Kokubun) 25 Aug '23

25 Aug '23

Issue #18454 has been updated by k0kubun (Takashi Kokubun). Yeah, this issue should be fine since they published this https://blog.discourse.org/2023/05/running-ruby-3-2s-yjit-in-production-at-…. ---------------------------------------- Bug #18454: YJIT slowing down key Discourse benchmarks https://bugs.ruby-lang.org/issues/18454#change-104365 * Author: sam.saffron (Sam Saffron) * Status: Closed * Priority: Normal * ruby -v: 3.1.0p0 * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- 4 out of 6 key Discourse benchmarks used in: https://github.com/discourse/discourse/blob/main/script/bench.rb are slower under YJIT. Vanilla (median) -> YJIT (median) Categories: 47ms -> 51 Home page: 85 -> 86 Topic: 42 -> 38 Categories Admin: 48 -> 50 Home Admin: 83 -> 84 Topic Admin: 43 -> 39 Boot: 1740ms -> 2484ms RSS: 358756 -> 640540 This was run with a simple `RUBYOPT='--enable-yjit' ruby script/bench.rb -i 100 -s` To get Discourse working with Ruby 3.1 you will need: ``` diff --git a/Gemfile b/Gemfile index 9cdfbf21a9..403c428718 100644 --- a/Gemfile +++ b/Gemfile @@ -263,3 +263,11 @@ gem 'colored2', require: false gem 'maxminddb' gem 'rails_failover', require: false + +if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.0.0") + gem 'net-pop', require: false + gem 'net-smtp', require: false + gem 'net-imap', require: false + # waiting on new version of i18n gem + gem 'psych', '3.3.2' +end ``` It is clear that memory usage and boot time have to take a hit here, but slowing down home page seems concerning. Any ideas on what we should test here and how to narrow this down? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114556] [Ruby master Bug#18457] ruby 2.7.5 fiddle/types.rb use uint32_t but fiddle/cparser.rb lacks uint32_t
by jeremyevans0 (Jeremy Evans) 25 Aug '23

25 Aug '23

Issue #18457 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Open to Closed I tested and confirmed this was fixed in Ruby 3.0. ---------------------------------------- Bug #18457: ruby 2.7.5 fiddle/types.rb use uint32_t but fiddle/cparser.rb lacks uint32_t https://bugs.ruby-lang.org/issues/18457#change-104363 * Author: YO4 (Yoshinao Muramatsu) * Status: Closed * Priority: Normal * ruby -v: ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x64-mingw32] * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- ``` C:\src\git\reline>bundle exec rake test_yamatanooroti Traceback (most recent call last): 17: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `<main>' 16: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:6:in `select' 15: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `block in <main>' 14: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/rake_test_loader.rb:21:in `require' 13: from C:/src/git/reline/test/reline/yamatanooroti/test_rendering.rb:6:in `<top (required)>' 12: from C:/src/git/yamatanooroti/lib/yamatanooroti.rb:67:in `inherited' 11: from C:/src/git/yamatanooroti/lib/yamatanooroti.rb:30:in `const_missing' 10: from C:/src/git/yamatanooroti/lib/yamatanooroti.rb:15:in `load_windows' 9: from C:/src/git/yamatanooroti/lib/yamatanooroti.rb:15:in `require' 8: from C:/src/git/yamatanooroti/lib/yamatanooroti/windows.rb:5:in `<top (required)>' 7: from C:/src/git/yamatanooroti/lib/yamatanooroti/windows.rb:57:in `<module:WindowsDefinition>' 6: from C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/import.rb:223:in `struct' 5: from C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/cparser.rb:41:in `parse_struct_signature' 4: from C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/cparser.rb:41:in `each' 3: from C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/cparser.rb:46:in `block in parse_struct_signature' 2: from C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/cparser.rb:178:in `parse_ctype' 1: from C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/cparser.rb:178:in `parse_ctype' C:/Ruby27-x64/lib/ruby/2.7.0/fiddle/cparser.rb:180:in `parse_ctype': unknown type: uint32_t (Fiddle::DLError) rake aborted! Command failed with status (1) Tasks: TOP => test_yamatanooroti (See full trace by running task with --trace) ``` related to https://github.com/ruby/ruby/commit/6601fb5672dc4c1f4bf5ee0f9b3f97a029df06cd -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114555] [Ruby master Bug#18456] rdoc non-determinism: module includes can be added once or twice to generated .ri
by jeremyevans0 (Jeremy Evans) 25 Aug '23

25 Aug '23

Issue #18456 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Open to Closed If this is still an issue, can you please file it upstream: https://github.com/ruby/rdoc/issues ---------------------------------------- Bug #18456: rdoc non-determinism: module includes can be added once or twice to generated .ri https://bugs.ruby-lang.org/issues/18456#change-104362 * Author: alex.kanavin(a)gmail.com (Alexander Kanavin) * Status: Closed * Priority: Normal * ruby -v: ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux-gnu] * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- We're observing a sporadically occurring non-determinism in generated .ri files: ``` [ak@localhost ruby-repro]$ ri --dump=./oe-reproducible-20211214-gj4vndgo/packages-excluded/reproducibleA/tmp/deploy/rpm/core2_64/usr/share/ri/3.0.0/system/Gem/DefaultUserInteraction/cdesc-DefaultUserInteraction.ri [module Gem::DefaultUserInteraction: includes: #<RDoc::Include:0x384 Gem::DefaultUserInteraction.rdoc::include Gem::Text> constants: ``` .... ``` [ak@localhost ruby-repro]$ ri --dump=./oe-reproducible-20211214-gj4vndgo/packages-excluded/reproducibleB/tmp/deploy/rpm/core2_64/usr/share/ri/3.0.0/system/Gem/DefaultUserInteraction/cdesc-DefaultUserInteraction.ri [module Gem::DefaultUserInteraction: includes: #<RDoc::Include:0x384 Gem::DefaultUserInteraction.rdoc::include Gem::Text>, #<RDoc::Include:0x398 Gem::DefaultUserInteraction.rdoc::include Gem::Text> constants: ``` This occurs in random .ri files, but the pattern seems same: the difference is always in the includes, and how many times a specific one gets listed. ---Files-------------------------------- cdesc-DefaultUserInteraction.ri (905 Bytes) cdesc-DefaultUserInteraction.ri (819 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114554] [Ruby master Bug#18454] YJIT slowing down key Discourse benchmarks
by jeremyevans0 (Jeremy Evans) 25 Aug '23

25 Aug '23

Issue #18454 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Open to Closed YJIT in Ruby 3.2 was significantly improved from Ruby 3.1, and has further been improved. If YJIT in current Ruby master is still resulting in a slowdown for Discourse, please post an update. ---------------------------------------- Bug #18454: YJIT slowing down key Discourse benchmarks https://bugs.ruby-lang.org/issues/18454#change-104361 * Author: sam.saffron (Sam Saffron) * Status: Closed * Priority: Normal * ruby -v: 3.1.0p0 * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- 4 out of 6 key Discourse benchmarks used in: https://github.com/discourse/discourse/blob/main/script/bench.rb are slower under YJIT. Vanilla (median) -> YJIT (median) Categories: 47ms -> 51 Home page: 85 -> 86 Topic: 42 -> 38 Categories Admin: 48 -> 50 Home Admin: 83 -> 84 Topic Admin: 43 -> 39 Boot: 1740ms -> 2484ms RSS: 358756 -> 640540 This was run with a simple `RUBYOPT='--enable-yjit' ruby script/bench.rb -i 100 -s` To get Discourse working with Ruby 3.1 you will need: ``` diff --git a/Gemfile b/Gemfile index 9cdfbf21a9..403c428718 100644 --- a/Gemfile +++ b/Gemfile @@ -263,3 +263,11 @@ gem 'colored2', require: false gem 'maxminddb' gem 'rails_failover', require: false + +if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("3.0.0") + gem 'net-pop', require: false + gem 'net-smtp', require: false + gem 'net-imap', require: false + # waiting on new version of i18n gem + gem 'psych', '3.3.2' +end ``` It is clear that memory usage and boot time have to take a hit here, but slowing down home page seems concerning. Any ideas on what we should test here and how to narrow this down? -- https://bugs.ruby-lang.org/

1 0