September 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118965] [Ruby master Bug#20701] Hash argument passed as keyword splat can be mutated inside method
by pabloh (Pablo Herrero) 02 Sep '24

02 Sep '24

Issue #20701 has been reported by pabloh (Pablo Herrero). ---------------------------------------- Bug #20701: Hash argument passed as keyword splat can be mutated inside method https://bugs.ruby-lang.org/issues/20701 * Author: pabloh (Pablo Herrero) * Status: Open * ruby -v: ruby 3.4.0dev (2024-08-27T00:39:24Z master e5295644c9) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- If you call a method with a splatted kwargs that also uses a splatted array followed by at least one positional argument, the called method can mutate the splatted kwargs. ``` ruby def foo(a, h) h[:splat_modified] = true end a = [] b = {splat_modified: false} foo(*a, :x, **b) b[:splat_modified] # Should be 'false' but is 'true' # It makes no difference if the called method expects a kwargs def foo2(a, **h) h[:splat_modified] = true end b[:splat_modified] = false foo2(*a, :x, **b) b[:splat_modified] # Should be 'false' but is 'true' # The case when a block is also pass is broken too c = proc { } b[:splat_modified] = false foo(*a, :x, **b, &c) b[:splat_modified] # Should be 'false' but is 'true' # When the extra positional argument after the array splat is removed the correct behavior is restored b[:splat_modified] = false foo(*a, **b) foo(*a, :x, **b, &c) b[:splat_modified] # Is 'false' ``` Apparently, this problem is due to the `KW_SPLAT_MUT` flag being set at the method's call site: `ruby --dump=insns,-optimize -e 'a, b, c = [4,6,4],{a: 3},:ds; foo(*a,9,**c)` outputs ``` 0000 duparray [4, 6, 4] ( 1)[Li] 0002 duphash {:a=>3} 0004 putobject :ds 0006 setlocal c@2, 0 0009 setlocal b@1, 0 0012 setlocal a@0, 0 0015 putself 0016 getlocal a@0, 0 0019 splatarray true 0021 putobject 9 0023 pushtoarray 1 0025 getlocal c@2, 0 0028 send <calldata!mid:foo, argc:2, ARGS_SPLAT|ARGS_SPLAT_MUT|FCALL|KW_SPLAT|KW_SPLAT_MUT>, nil 0031 leave ``` The error is present in both 3.3 and 3.4: - ruby 3.4.0dev (2024-08-27T00:39:24Z master e5295644c9) [x86_64-linux] - ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux] PR for test: https://github.com/ruby/ruby/pull/11467 -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118927] [Ruby master Bug#20691] Use-after-free in WeakKeyMap#clear
by peterzhu2118 (Peter Zhu) 02 Sep '24

02 Sep '24

Issue #20691 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20691: Use-after-free in WeakKeyMap#clear https://bugs.ruby-lang.org/issues/20691 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/11437 If the WeakKeyMap has been marked but sweeping hasn't started yet and we cann WeakKeyMap#clear, then there could be a use-after-free because we do not call rb_gc_remove_weak to remove the key from the GC. For example, the following code triggers use-after-free errors in Valgrind: ```ruby map = ObjectSpace::WeakKeyMap.new 1_000.times do 1_000.times do map[Object.new] = nil end map.clear end ``` Output from Valgrind: ``` ==61230== Invalid read of size 8 ==61230== at 0x25CAF8: gc_update_weak_references (default.c:5593) ==61230== by 0x25CAF8: gc_marks_finish (default.c:5641) ==61230== by 0x26031C: gc_marks_continue (default.c:5987) ==61230== by 0x26031C: gc_continue (default.c:2255) ==61230== by 0x2605FC: newobj_cache_miss (default.c:2589) ==61230== by 0x26111F: newobj_alloc (default.c:2622) ==61230== by 0x26111F: rb_gc_impl_new_obj (default.c:2701) ==61230== by 0x26111F: newobj_of (gc.c:890) ==61230== by 0x26111F: rb_wb_protected_newobj_of (gc.c:917) ==61230== by 0x2DE218: rb_class_allocate_instance (object.c:131) ==61230== by 0x2E32A8: class_call_alloc_func (object.c:2141) ==61230== by 0x2E32A8: rb_class_alloc (object.c:2113) ==61230== by 0x2E32A8: rb_class_new_instance_pass_kw (object.c:2172) ==61230== by 0x4296BC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3788) ==61230== by 0x44A9CD: vm_sendish (vm_insnhelper.c:5955) ==61230== by 0x44A9CD: vm_exec_core (insns.def:898) ==61230== by 0x43A0E4: rb_vm_exec (vm.c:2564) ==61230== by 0x2341B4: rb_ec_exec_node (eval.c:281) ==61230== by 0x236258: ruby_run_node (eval.c:319) ==61230== by 0x15D665: rb_main (main.c:43) ==61230== by 0x15D665: main (main.c:62) ==61230== Address 0x2159cb00 is 0 bytes inside a block of size 8 free'd ==61230== at 0x4849B2C: free (vg_replace_malloc.c:989) ==61230== by 0x248EF1: rb_gc_impl_free (default.c:8512) ==61230== by 0x248EF1: rb_gc_impl_free (default.c:8493) ==61230== by 0x248EF1: ruby_sized_xfree.constprop.0 (gc.c:4178) ==61230== by 0x4627EC: wkmap_free_table_i (weakmap.c:652) ==61230== by 0x3A54AF: apply_functor (st.c:1633) ==61230== by 0x3A54AF: st_general_foreach (st.c:1543) ==61230== by 0x3A54AF: rb_st_foreach (st.c:1640) ==61230== by 0x46203C: wkmap_clear (weakmap.c:973) ==61230== by 0x4296BC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3788) ==61230== by 0x44A9CD: vm_sendish (vm_insnhelper.c:5955) ==61230== by 0x44A9CD: vm_exec_core (insns.def:898) ==61230== by 0x43A0E4: rb_vm_exec (vm.c:2564) ==61230== by 0x2341B4: rb_ec_exec_node (eval.c:281) ==61230== by 0x236258: ruby_run_node (eval.c:319) ==61230== by 0x15D665: rb_main (main.c:43) ==61230== by 0x15D665: main (main.c:62) ==61230== Block was alloc'd at ==61230== at 0x484680F: malloc (vg_replace_malloc.c:446) ==61230== by 0x25C68E: rb_gc_impl_malloc (default.c:8527) ==61230== by 0x4622E9: wkmap_aset_replace (weakmap.c:817) ==61230== by 0x3A4D02: rb_st_update (st.c:1487) ==61230== by 0x4623E4: wkmap_aset (weakmap.c:854) ==61230== by 0x4296BC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3788) ==61230== by 0x44A9CD: vm_sendish (vm_insnhelper.c:5955) ==61230== by 0x44A9CD: vm_exec_core (insns.def:898) ==61230== by 0x43A0E4: rb_vm_exec (vm.c:2564) ==61230== by 0x2341B4: rb_ec_exec_node (eval.c:281) ==61230== by 0x236258: ruby_run_node (eval.c:319) ==61230== by 0x15D665: rb_main (main.c:43) ==61230== by 0x15D665: main (main.c:62) ==61230== ==61230== Invalid write of size 8 ==61230== at 0x25CB3B: gc_update_weak_references (default.c:5598) ==61230== by 0x25CB3B: gc_marks_finish (default.c:5641) ==61230== by 0x26031C: gc_marks_continue (default.c:5987) ==61230== by 0x26031C: gc_continue (default.c:2255) ==61230== by 0x2605FC: newobj_cache_miss (default.c:2589) ==61230== by 0x26111F: newobj_alloc (default.c:2622) ==61230== by 0x26111F: rb_gc_impl_new_obj (default.c:2701) ==61230== by 0x26111F: newobj_of (gc.c:890) ==61230== by 0x26111F: rb_wb_protected_newobj_of (gc.c:917) ==61230== by 0x2DE218: rb_class_allocate_instance (object.c:131) ==61230== by 0x2E32A8: class_call_alloc_func (object.c:2141) ==61230== by 0x2E32A8: rb_class_alloc (object.c:2113) ==61230== by 0x2E32A8: rb_class_new_instance_pass_kw (object.c:2172) ==61230== by 0x4296BC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3788) ==61230== by 0x44A9CD: vm_sendish (vm_insnhelper.c:5955) ==61230== by 0x44A9CD: vm_exec_core (insns.def:898) ==61230== by 0x43A0E4: rb_vm_exec (vm.c:2564) ==61230== by 0x2341B4: rb_ec_exec_node (eval.c:281) ==61230== by 0x236258: ruby_run_node (eval.c:319) ==61230== by 0x15D665: rb_main (main.c:43) ==61230== by 0x15D665: main (main.c:62) ==61230== Address 0x2159cb00 is 0 bytes inside a block of size 8 free'd ==61230== at 0x4849B2C: free (vg_replace_malloc.c:989) ==61230== by 0x248EF1: rb_gc_impl_free (default.c:8512) ==61230== by 0x248EF1: rb_gc_impl_free (default.c:8493) ==61230== by 0x248EF1: ruby_sized_xfree.constprop.0 (gc.c:4178) ==61230== by 0x4627EC: wkmap_free_table_i (weakmap.c:652) ==61230== by 0x3A54AF: apply_functor (st.c:1633) ==61230== by 0x3A54AF: st_general_foreach (st.c:1543) ==61230== by 0x3A54AF: rb_st_foreach (st.c:1640) ==61230== by 0x46203C: wkmap_clear (weakmap.c:973) ==61230== by 0x4296BC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3788) ==61230== by 0x44A9CD: vm_sendish (vm_insnhelper.c:5955) ==61230== by 0x44A9CD: vm_exec_core (insns.def:898) ==61230== by 0x43A0E4: rb_vm_exec (vm.c:2564) ==61230== by 0x2341B4: rb_ec_exec_node (eval.c:281) ==61230== by 0x236258: ruby_run_node (eval.c:319) ==61230== by 0x15D665: rb_main (main.c:43) ==61230== by 0x15D665: main (main.c:62) ==61230== Block was alloc'd at ==61230== at 0x484680F: malloc (vg_replace_malloc.c:446) ==61230== by 0x25C68E: rb_gc_impl_malloc (default.c:8527) ==61230== by 0x4622E9: wkmap_aset_replace (weakmap.c:817) ==61230== by 0x3A4D02: rb_st_update (st.c:1487) ==61230== by 0x4623E4: wkmap_aset (weakmap.c:854) ==61230== by 0x4296BC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3788) ==61230== by 0x44A9CD: vm_sendish (vm_insnhelper.c:5955) ==61230== by 0x44A9CD: vm_exec_core (insns.def:898) ==61230== by 0x43A0E4: rb_vm_exec (vm.c:2564) ==61230== by 0x2341B4: rb_ec_exec_node (eval.c:281) ==61230== by 0x236258: ruby_run_node (eval.c:319) ==61230== by 0x15D665: rb_main (main.c:43) ==61230== by 0x15D665: main (main.c:62) ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:117210] [Ruby master Bug#20344] argument stack underflow (-1)
by nobu (Nobuyoshi Nakada) 02 Sep '24

02 Sep '24

Issue #20344 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20344: argument stack underflow (-1) https://bugs.ruby-lang.org/issues/20344 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- This file produces argument stack underflow, since ruby 2.5. ```ruby proc do next if true case nil when "a" next when "b" when "c" proc {} end next unless true end ``` ``` -- raw disasm-------- trace: 100 0000 nop ( 1) <L000> [sp: 0, unremovable: 1, refcnt: 5] trace: 1 <L004> [sp: 0, unremovable: 0, refcnt: 1] adjust: [label: 0] 0001 putnil ( 2) 0002 leave ( 13) <L008> [sp: 1, unremovable: 0, refcnt: 1] adjust: [label: 0] 0003 putnil ( 6) 0004 leave ( 13) adjust: [label: 8] 0005 jump <L005> ( 5) 0007 pop ( 7) 0008 jump <L005> ( 7) * 0010 pop ( 8) trace: 1 <L011> [sp: -1, unremovable: 1, refcnt: 1] 0011 putself ( 9) 0012 send <calldata:proc, 0>, nil ( 9) <L012> [sp: -1, unremovable: 0, refcnt: 2] 0015 pop ( 9) <L005> [sp: 1, unremovable: 0, refcnt: 2] trace: 1 0016 putnil ( 12) <L001> [sp: -1, unremovable: 0, refcnt: 3] trace: 200 0017 leave ( 13) --------------------- ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118676] [Ruby master Bug#20649] Ripper fails to tokenize `def f; (x)::A =`
by tompng (tomoya ishida) 02 Sep '24

02 Sep '24

Issue #20649 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20649: Ripper fails to tokenize `def f; (x)::A =` https://bugs.ruby-lang.org/issues/20649 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-07-24T05:44:32Z master e642ddf7ae) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Ripper fails to tokenize some dynamic constant assignment syntax error code. `''::A=` `(x)::A=` `(x).y::Z=` `[]::A=` `{}::A=` ``` ``::A=``` `//::A=` `%()::A=` in method body seems to fail. Same for `for ''::A in` and `rescue => ''::A;` in method body. ~~~ruby Ripper.tokenize 'def f; (x)::A =' #=> /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:245:in 'Ripper::Lexer#on_error2': undefined method 'pos' for nil (NoMethodError) @errors.push Elem.new(elem.pos, __callee__, elem.tok, elem.state, mesg) ^^^^ from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:176:in 'Ripper#parse' from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:176:in 'Ripper::Lexer#parse' from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:150:in 'Ripper::Lexer#tokenize' from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:26:in 'Ripper.tokenize' from -e:1:in '<main>' ~~~ -- https://bugs.ruby-lang.org/

4 3

[ruby-core:118270] [Ruby master Bug#20571] Compile error on for loop
by kddnewton (Kevin Newton) 02 Sep '24

02 Sep '24

Issue #20571 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20571: Compile error on for loop https://bugs.ruby-lang.org/issues/20571 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby for _ in 1.5 or _ in [] end ``` raises a compile error for the current compiler only with ``` -- raw disasm-------- trace: 1 <L000> [sp: 0, unremovable: 1, refcnt: 1] 0000 putobject 1.5 ( 1) 0002 jump <L002> ( 1) <L006> [sp: 1, unremovable: 1, refcnt: 1] <L004> [sp: 1, unremovable: 0, refcnt: 0] 0004 pop ( 1) * 0005 pop ( 1) 0006 putobject false ( 1) <L002> [sp: 1, unremovable: 0, refcnt: 1] 0008 send <calldata:each, 0>, nil ( 1) <L001> [sp: -1, unremovable: 0, refcnt: 2] 0011 leave ( 1) --------------------- test.rb:1: argument stack underflow (-1) test.rb: compile error (SyntaxError) ``` -- https://bugs.ruby-lang.org/

4 4

[ruby-core:118681] [Ruby master Bug#20651] `false && (1 in 1)` produces argument stack underflow (-1)
by tompng (tomoya ishida) 02 Sep '24

02 Sep '24

Issue #20651 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20651: `false && (1 in 1)` produces argument stack underflow (-1) https://bugs.ruby-lang.org/issues/20651 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-07-24T19:55:04Z master 30eba40f9c) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- `false && (1 in 1)` and `true || (1 in 1)` produces argument stack underflow since ruby 3.0 ~~~ # ruby -v ruby 3.4.0dev (2024-07-24T19:55:04Z master 30eba40f9c) [x86_64-linux] # ruby -e "false && (1 in 1)" -- raw disasm-------- trace: 1 0000 putobject false ( 1) 0002 leave ( 1) <L004> [sp: 1, unremovable: 1, refcnt: 1] <L002> [sp: 1, unremovable: 0, refcnt: 0] 0003 pop ( 1) * 0004 pop ( 1) 0005 putobject false ( 1) 0007 leave ( 1) --------------------- -e:1: argument stack underflow (-1) -e: compile error (SyntaxError) ~~~ -- https://bugs.ruby-lang.org/

3 3

[ruby-core:118797] [Ruby master Bug#20668] [3.3] shareable_constant_value: literal crash
by byroot (Jean Boussier) 02 Sep '24

02 Sep '24

Issue #20668 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #20668: [3.3] shareable_constant_value: literal crash https://bugs.ruby-lang.org/issues/20668 * Author: byroot (Jean Boussier) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin23] * Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- The following test script crashes in the parser: ```ruby # shareable_constant_value: literal FORM_STRATEGIES = { 'product' => Object, 'storefront_password' => String, 'contact' => Class, } ``` The entire 3.3.x series is affected but not `master` nor 3.2. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118686] [Ruby master Bug#20653] Memory leak in String#start_with? when regexp times out
by peterzhu2118 (Peter Zhu) 02 Sep '24

02 Sep '24

Issue #20653 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20653: Memory leak in String#start_with? when regexp times out https://bugs.ruby-lang.org/issues/20653 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: WONTFIX, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/11247 This commit refactors how Onigmo handles timeout. Instead of raising a timeout error, onig_search will return a ONIGERR_TIMEOUT which the caller can free memory, and then raise a timeout error. This fixes a memory leak in String#start_with when the regexp times out. For example: ```ruby regex = Regexp.new("^#{"(a*)" * 10_000}x$", timeout: 0.000001) str = "a" * 1000000 + "x" 10.times do 100.times do str.start_with?(regex) rescue end puts `ps -o rss= -p #{$$}` end ``` Before: ``` 33216 51936 71152 81728 97152 103248 120384 133392 133520 133616 ``` After: ``` 14912 15376 15824 15824 16128 16128 16144 16144 16160 16160 ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118637] [Ruby master Bug#20641] `lib/bundled_gems.rb` makes `Kernel.require` over 100x slower
by byroot (Jean Boussier) 02 Sep '24

02 Sep '24

Issue #20641 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #20641: `lib/bundled_gems.rb` makes `Kernel.require` over 100x slower https://bugs.ruby-lang.org/issues/20641 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- I just discovered this while profiling Active Record's test suite, and I noticed 40% of the runtime was in `$LOAD_PATH.resolve_feature_path`, so much I thought it was a profiler bug. But it turns out it's real. Various APIs do call `require` late, for instance Psych calls `require 'date'` every single time it parses a date: https://github.com/ruby/psych/blob/be0ba74e5613c20f213403e15914d24944c2652d… I've put together a quick benchmark: ```ruby # frozen_string_literal: true require "bundler/inline" gemfile(true) do source "https://rubygems.org" gem "benchmark-ips" end Benchmark.ips do |x| x.report(RUBY_VERSION) { require "erb" } x.save! "/tmp/require.bench" x.compare! end ``` And the difference is massive: ``` Calculating ------------------------------------- 3.2.2 6.450M (± 1.2%) i/s - 32.822M in 5.089538s Comparison: 3.2.2: 6449899.4 i/s 3.3.3: 46996.8 i/s - 137.24x slower ``` And that's with a small `$LOAD_PATH`, the bigger the application, the worse it is, if I add: ```ruby 100.times do |i| $LOAD_PATH.unshift("/tmp/empty-#{i}") end ``` ``` Calculating ------------------------------------- 3.3.3 6.198k (± 6.1%) i/s - 30.968k in 5.018955s Comparison: 3.2.2: 3380939.6 i/s 3.3.3: 6198.4 i/s - 545.46x slower ``` I'm looking at a way to speed this up, but this is big enough that I believe we should backport the fix. cc @hsbt -- https://bugs.ruby-lang.org/

2 2