November 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118957] [Ruby master Bug#20699] On Windows, the `__dir__` keyword is garbled in paths containing Japanese characters, and `require_relative` fails as well
by flatland001 (Hayato Arai) 18 Dec '24

18 Dec '24

Issue #20699 has been reported by flatland001 (Hayato Arai). ---------------------------------------- Bug #20699: On Windows, the `__dir__` keyword is garbled in paths containing Japanese characters, and `require_relative` fails as well https://bugs.ruby-lang.org/issues/20699 * Author: flatland001 (Hayato Arai) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x64-mingw-ucrt] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In paths containing Japanese characters, such as: C:\テスト_test\test.rb Code: ``` ruby # this code -> C:\テスト_test\test.rb # external file -> C:\テスト_test\lib\foo.rb p __dir__ p __dir__.encoding p File.dirname(File.expand_path(__FILE__)) p File.dirname(File.expand_path(__FILE__)).encoding print "__dir__ == File.dirname(File.expand_path(__FILE__)): " p __dir__ == File.dirname(File.expand_path(__FILE__)) begin require_relative "lib/foo" puts "foo.rb loaded" rescue LoadError puts "LoadError: #{$!}" end ``` Results: ``` "C:/?e?X?g_test" #<Encoding:Windows-31J> "C:/\x{8365}\x{8358}\x{8367}_test" #<Encoding:Windows-31J> __dir__ == File.dirname(File.expand_path(__FILE__)): false LoadError: cannot load such file -- C:/?e?X?g_test/lib/foo ``` However, in ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x64-mingw-ucrt], the same code produces the following expected results: ``` "C:/\x{8365}\x{8358}\x{8367}_test" #<Encoding:Windows-31J> "C:/\x{8365}\x{8358}\x{8367}_test" #<Encoding:Windows-31J> __dir__ == File.dirname(File.expand_path(__FILE__)): true foo.rb loaded ``` Is this behavior expected, or could this be a bug in the Ruby interpreter on Windows? If this is my misunderstanding or a known issue, I apologize. Thank you for your consideration. Environment Information: - OS: Windows 11 Pro 23H2 (OS build 22631.4037) - Installed via: RubyInstaller - Shell: Command Prompt - File System: NTFS - System Locale: Japanese (Japan), Language: Japanese -- https://bugs.ruby-lang.org/

3 2

[ruby-core:119380] [Ruby master Misc#20774] Remove remaining locale dependent code from Windows port
by larskanis (Lars Kanis) 18 Dec '24

18 Dec '24

Issue #20774 has been reported by larskanis (Lars Kanis). ---------------------------------------- Misc #20774: Remove remaining locale dependent code from Windows port https://bugs.ruby-lang.org/issues/20774 * Author: larskanis (Lars Kanis) * Status: Open ---------------------------------------- The external_encoding of files, file names and ENV on Windows were changed from locale codepage to UTF-8 in ruby-3.0. But there are still several remaining points where locale encoding is used although there is no need to do so. The Windows port is already fully UTF-16/UTF-8 based and locale encoding is only used for historical and not for technical reasons. My proposal is to remove (most of) the locale dependent conversions from the ruby code for Windows. Before I open pull requests in this regard, I would like to confirm this direction with the ruby core team. Let me show what I mean: ``` # täst-locale-enc.rb def pr(*strs) strs.each do |str| p [str, IO===str ? str.external_encoding&.name : str.encoding.name] end end if $0==__FILE__ pr STDIN # => [#<IO:<STDIN>>, "CP850"] pr $0 # => ["ruby/t\x84st-locale-enc.rb", "CP850"] pr __FILE__ # => ["ruby/t\x84st-locale-enc.rb", "CP850"] pr __dir__ # => ["C:/Users/kanis/ruby", "CP850"] pr 'ä' # => ["ä", "UTF-8"] pr '€' # => ["€", "UTF-8"] pr $:.first # => ["C:/Users/kanis/t\xE2\x82\xACst", "ASCII-8BIT"] pr $:.last # => ["C:/Ruby33-x64/lib/ruby/3.3.0/x64-mingw-ucrt", "CP850"] require "win32/registry" pr Win32::Registry::HKEY_CURRENT_USER.open("Environment")['TMP'] # => ["C:\\Users\\kanis\\AppData\\Local\\Temp", "UTF-8"] pr Win32::Registry::HKEY_CURRENT_USER.open("\\").each_key{ break _1 } # => ["AppEvents", "CP850"] end # execute with: ruby -It€st ruby\täst-locale-enc.rb ``` I wrote the results on `ruby-3.3 x64-mingw-ucrt` right into the code. The situation is even worse when called with `-e` script: ``` $ ruby -It€st -r .\ruby\täst-locale-enc.rb -e "pr STDIN, $0, __FILE__, __dir__, 'ä', '€', $:.first, $:.last" [#<IO:<STDIN>>, "CP850"] ["-e", "CP850"] ["-e", "UTF-8"] [".", "US-ASCII"] ["\x84", "CP850"] ["?", "CP850"] ["C:/Users/kanis/t\xE2\x82\xACst", "ASCII-8BIT"] ["C:/Ruby33-x64/lib/ruby/3.3.0/x64-mingw-ucrt", "CP850"] ``` There are also some inconsistencies like that it's possible to `require` script names with characters outside of the codepage, but it fails to execute a script directly or by using `require_relative` : ``` $ ruby -r .\t€st-locale-enc.rb -e "pr STDIN" [#<IO:<STDIN>>, "CP850"] $ ruby .\t€st-locale-enc.rb ruby: Invalid argument -- ./t?st-locale-enc.rb (LoadError) ``` Maybe there are more places which are working with locale codepage - these are only the few that I remember from memory. I would like to change all the above results to be UTF-8 encoded, like it is the case on Ubuntu. Compatibility ------------- Changing the encoding of returned strings is of course an API change. IMHO it is still something we should change in a minor release of ruby. The reason is that I don't remember about only a single issue cased by the change to UTF-8 in ruby-3.0 in the company I work for. To the contrary many issues are caused by using locale codepage where some non-ASCII characters work and other characters don't. Most issue with ruby-3.0 were cased by the keyword argument changes. -- https://bugs.ruby-lang.org/

2 4

[ruby-core:119925] [Ruby master Feature#20893] random/formatter.rb: Separate from securerandom
by nobu (Nobuyoshi Nakada) 16 Dec '24

16 Dec '24

Issue #20893 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #20893: random/formatter.rb: Separate from securerandom https://bugs.ruby-lang.org/issues/20893 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- For historical reasons, random/formatter.rb is included in ruby/securerandom, but it does not contain any security-related methods as evidenced by the fact that its path is not under `securerandom/`, but only contains generic methods for PRNGs, including the standard class `Random`. So I propose to either: 1. put it back in the standard library, or 2. separate it into a new gem. Adding a new dependency may not be welcomed, as seen in the recent issues with base64. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:117657] [Ruby master Bug#20447] Ruby 3.3.1 broken on i686
by vo.x (Vit Ondruch) 15 Dec '24

15 Dec '24

Issue #20447 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #20447: Ruby 3.3.1 broken on i686 https://bugs.ruby-lang.org/issues/20447 * Author: vo.x (Vit Ondruch) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [i686-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ~~~ gcc -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -march=i686 -mtune=generic -msse2 -mfpmath=sse -mstackrealign -fasynchronous-unwind-tables -fstack-clash-protection -fPIC -m32 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -I. -I.ext/include/i386-linux -I/builddir/build/BUILD/ruby-3.3.1/include -I/builddir/build/BUILD/ruby-3.3.1 -I/builddir/build/BUILD/ruby-3.3.1/prism -I/builddir/build/BUILD/ruby-3.3.1/enc/unicode/15.0.0 -o time.o -c /builddir/build/BUILD/ruby-3.3.1/time.c make: Leaving directory '/builddir/build/BUILD/ruby-3.3.1/redhat-linux-build' /builddir/build/BUILD/ruby-3.3.1/time.c: In function ‘zone_timelocal’: /builddir/build/BUILD/ruby-3.3.1/time.c:2345:35: error: passing argument 3 of ‘split_second’ from incompatible pointer type [-Wincompatible-pointer-types] 2345 | split_second(tobj->timew, &t, &s); | ^~ | | | wideval_t * {aka long long unsigned int *} /builddir/build/BUILD/ruby-3.3.1/time.c:616:58: note: expected ‘VALUE *’ {aka ‘long unsigned int *’} but argument is of type ‘wideval_t *’ {aka ‘long long unsigned int *’} 616 | split_second(wideval_t timew, wideval_t *timew_p, VALUE *subsecx_p) | ~~~~~~~^~~~~~~~~ make: *** [Makefile:447: time.o] Error 1 make: *** Waiting for unfinished jobs.... make: Entering directory '/builddir/build/BUILD/ruby-3.3.1/redhat-linux-build' ~~~ This is presumably due to #20327 / https://github.com/ruby/ruby/pull/10332. There seems to be medicine in master commit:git|055613fd868a8c94e43893f8c58a00cdd2a81f6d or maybe the whole https://github.com/ruby/ruby/pull/10332, not sure -- https://bugs.ruby-lang.org/

8 14

[ruby-core:119852] [Ruby master Feature#20882] Provide Boolean(...)
by getajobmike (Mike Perham) 14 Dec '24

14 Dec '24

Issue #20882 has been reported by getajobmike (Mike Perham). ---------------------------------------- Feature #20882: Provide Boolean(...) https://bugs.ruby-lang.org/issues/20882 * Author: getajobmike (Mike Perham) * Status: Open ---------------------------------------- Ruby provides Integer(...) and Float(...) global methods to coerce values. Is there a similar method for Booleans? I'd like to do something like: ``` # ENV["SOME_FEATURE"] is unset Boolean(ENV["SOME_FEATURE"]) # => false # ENV["SOME_FEATURE"] is unset, but allow a default? Boolean(ENV["SOME_FEATURE"], true) # => true # explicitly disable ENV["SOME_FEATURE"] = "0" Boolean(ENV["SOME_FEATURE"], true) # => false # explicitly enable ENV["SOME_FEATURE"] = "1" Boolean(ENV["SOME_FEATURE"]) # => true ``` -- https://bugs.ruby-lang.org/

11 11

[ruby-core:119769] [Ruby master Feature#20875] Atomic initialization for Ractor local storage
by ko1 (Koichi Sasada) 12 Dec '24

12 Dec '24

Issue #20875 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Feature #20875: Atomic initialization for Ractor local storage https://bugs.ruby-lang.org/issues/20875 * Author: ko1 (Koichi Sasada) * Status: Open ---------------------------------------- ## Motivation Now there is no way to initialize Ractor local storage in multi-thread. For example, if we want to introduce per-Ractor counter, which should be protected with a per-Ractor Mutex for multi-threading support. ```ruby def init Ractor[:cnt] = 0 Ractor[:mtx] = Mutex.new end def inc init unless Ractor[:cnt] Ractor[:mtx].synchronize do Ractor[:cnt] += 1 end end ``` In this code, if `inc` was called on multiple threads, `init` can be called with multiple threads and `cnt` can not be synchronized correctly. ## Proposal Let's introduce `Ractor.local_storage_init(sym){ block }` to initialize values in Ractor local storage. If there is no slot for `sym`, synchronize with per-Ractor mutex and call `block` and the slot will be filled with the evaluation with the block result. The return value of this method will be the filled value. Otherwise, returning corresponding value will be returned. The implementation is like that (in C): ```ruby class Ractor def self.local_storage_init(sym) Ractor.per_ractor_mutex.synchronize do if Ractor.local_storage_has_key?(sym) Ractor[:sym] else Ractor[:sym] = yield end end end end ``` The above examples will be rewritten with the following code: ```ruby def inc Ractor.local_storage_init(:mtx) do Ractor[:cnt] = 0 Mutex.new end.synchronize do Ractor[:cnt] += 1 end end ``` ## Discussion ### Approach There are another approach like `pthread_atfork`, maybe like `Ractor.atcreate{ init }`. A library registers a callback which will be called when a new ractor is created. However, there are many Ractors which don't use the library, so that ``atcreate` can be huge overhead for Ractor creation. ### Naming I propose `local_storage_init`, but not sure it matches. I also proposed `Ractor.local_variable_init(sym)`, but Matz said he doesn't like this naming because it should not be a "variable". (there is a `Thread#thread_variable_get` method, though). On another aspect, `lcoal_storage_init` seems it clears all of ractor local storage slots. -- https://bugs.ruby-lang.org/

3 11

[ruby-core:120035] [Ruby master Bug#20918] Prism error indicates line number of `-e` that does not exist
by mame (Yusuke Endoh) 12 Dec '24

12 Dec '24

Issue #20918 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #20918: Prism error indicates line number of `-e` that does not exist https://bugs.ruby-lang.org/issues/20918 * Author: mame (Yusuke Endoh) * Status: Open * Assignee: prism * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ``` $ ruby -e 'foo(' -e: -e:2: syntax error found (SyntaxError) 1 | foo( > 2 | | ^ unexpected end-of-input; expected a `)` to close the arguments ``` It says `-e:2`, but there is no line 2 in `-e 'foo('`. `eval("foo(")` reports line 1, so I guess this issue is only for `-e`. -- https://bugs.ruby-lang.org/

4 5

[ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced
by alanwu (Alan Wu) 12 Dec '24

12 Dec '24

Issue #20659 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #20659: Speed regression of `parse.y` parser after numeric nodes were introduced https://bugs.ruby-lang.org/issues/20659 * Author: alanwu (Alan Wu) * Status: Open * ruby -v: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The [`mail` benchmark from yjit-bench][1] is about 20% slower on `master` compared to commit:98eeadc9 ("Development of 3.4.0 started.") as the baseline, comparing running time of the Ruby process running the benchmark for a single iteration. Much of this workload is Ruby parsing. ``` $ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby' '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb' Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 889.0 ms ± 2.0 ms [User: 776.9 ms, System: 111.7 ms] Range (min … max): 885.7 ms … 892.1 ms 10 runs Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 891.7 ms ± 1.6 ms [User: 794.5 ms, System: 96.9 ms] Range (min … max): 889.0 ms … 894.7 ms 10 runs Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 1.086 s ± 0.003 s [User: 0.951 s, System: 0.134 s] Range (min … max): 1.080 s … 1.091 s 10 runs Summary '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ran 1.00 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' 1.22 ± 0.00 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' $ for tag in 3.3-equiv master before-numeric-nodes; do echo $tag: $(~/.rubies/$tag/bin/ruby -v); done 3.3-equiv: ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [x86_64-linux] master: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux] before-numeric-nodes: ruby 3.4.0dev (2024-01-06T18:26:38Z master 76afbda5b5) [x86_64-linux] ``` Using Valgrind's DHAT reveals that commit:1b8d01136c3ff6c60325c7609d61e19ac42acd9f ("Introduce Numeric Node's") issues roughly 3 times more `malloc` calls compared to the baseline, most of them comping from strdup() calls in set_number_literal(). Comparison with `--parser=prism`, for interest: <details> ``` $ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby,master/bin/ruby --parser=prism' \ '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb' Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 889.7 ms ± 2.6 ms [User: 771.4 ms, System: 118.0 ms] Range (min … max): 885.8 ms … 894.6 ms 10 runs Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 890.4 ms ± 1.6 ms [User: 776.8 ms, System: 113.3 ms] Range (min … max): 888.4 ms … 892.9 ms 10 runs Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 1.087 s ± 0.004 s [User: 0.968 s, System: 0.119 s] Range (min … max): 1.083 s … 1.097 s 10 runs Benchmark 4: ~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb Time (mean ± σ): 826.9 ms ± 2.1 ms [User: 725.8 ms, System: 100.7 ms] Range (min … max): 823.6 ms … 830.9 ms 10 runs Summary '~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb' ran 1.08 ± 0.00 times faster than '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' 1.08 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' 1.31 ± 0.01 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ``` </details> [1]: https://github.com/Shopify/yjit-bench/blob/b8f1bd8931d4757afdae933ef6f11e07… -- https://bugs.ruby-lang.org/

2 4

[ruby-core:119632] [Ruby master Feature#20818] Allow passing a block to Hash#store (to update current value)
by furunkel (Julian Aron Prenner) 12 Dec '24

12 Dec '24

Issue #20818 has been reported by furunkel (Julian Aron Prenner). ---------------------------------------- Feature #20818: Allow passing a block to Hash#store (to update current value) https://bugs.ruby-lang.org/issues/20818 * Author: furunkel (Julian Aron Prenner) * Status: Open ---------------------------------------- I would like to propose a block form for `Hash#store`. In addition to passing a value, it should also be allowed to pass a block. If passed a block instead of a value, the block is called with the current value or, if unset, the hash's default value; the block's return value will be the value that is stored. This is similar to e.g., `computeIfPresent`/`computeIfAbsent` in Java (I think). I can think of several situations where this would be useful, in particular for caches and counters. For instance: ```ruby counts = {} elements.each do |element| counts.store(element){ (_1 || 0) + element.weight} end ``` or even more elegant with a default value: ```ruby counts = {} counts.default = 0 elements.each do |element| counts.store(element){ _1 + element.weight} end ``` Moreover, using the block form we should be able to do operations such as `h[k] ||= x`, `h[k] -= x`, `h[k] += x`, or more generally `h[k] = f(h[k])`, with a single "hashing round-trip". If I'm not mistaken, currently these involve two separate calls to `#[]` and `#[]=` (with two calls to `#hash`?). Finally, this makes `#store` a proper dual of `#fetch` which, similarly, can be passed a block. I have an experimental implementation of this (GitHub PR) at: https://github.com/ruby/ruby/pull/11956 -- https://bugs.ruby-lang.org/

3 2

[ruby-core:120050] [Ruby master Feature#20922] Should not we omit parentheses in assert calls?
by mame (Yusuke Endoh) 12 Dec '24

12 Dec '24

Issue #20922 has been reported by mame (Yusuke Endoh). ---------------------------------------- Feature #20922: Should not we omit parentheses in assert calls? https://bugs.ruby-lang.org/issues/20922 * Author: mame (Yusuke Endoh) * Status: Open ---------------------------------------- I often see the style of omitting parentheses in assert calls, but it leads to annoying warnings in the following case: ``` assert_equal -1, val #=> warning: ambiguous first argument; put parentheses or a space even after `-` operator assert_match /foo/, str #=> warning: ambiguous `/`; wrap regexp in parentheses or add a space after `/` operator ``` To deal with this warning, it is necessary to add parentheses only to *some* assert calls. ``` assert_equal 1, one assert_equal 0, zero assert_equal(-1, minus_one) ``` Obviously, this is not very cool. I feel that by these warnings, Ruby says "we should not omit parentheses in assert calls". Is this what matz intended? If it is the intent, I would like to add parentheses on all assert calls in tests of Ruby and default gems. If it is not the intent, why don't we remove these warnings? Note that, as far as I recall, I encountered this problem only with `assert_equal` and `assert_match`. I don't write parentheses in `p`, `puts`, `attr_reader`, `include`, etc., but all of them rarely accept `-1` and `//` literally as the first argument in real code. As a milder approach, I came up with an idea to stop the warning only when the method name starts with "assert". It is very ad-hoc, though. -- https://bugs.ruby-lang.org/

2 3