March 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:116597] [Ruby master Bug#20239] Segmentation fault when using Regex on a large String
by martinsp (Martins Polakovs) 09 Jul '24

09 Jul '24

Issue #20239 has been reported by martinsp (Martins Polakovs). ---------------------------------------- Bug #20239: Segmentation fault when using Regex on a large String https://bugs.ruby-lang.org/issues/20239 * Author: martinsp (Martins Polakovs) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Since v3.2.0 ruby crashes with segmentation fault on the following script with a `[BUG] Segmentation fault at ...` ``` ruby require "rbconfig/sizeof" ("\u{0101}" + "a" * RbConfig::LIMITS["INT_MAX"] + "b").match(/b/) ``` Crash can be reproduced on the following ruby versions: - ruby 3.2.0 (2022-12-25 revision a528908271) [aarch64-linux] - ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [aarch64-linux] - ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux] ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [aarch64-linux] works as expected It seems that call to `enclen` inside `str_lower_case_match` returns negative offset in this case https://bugs.ruby-lang.org/projects/ruby-master/repository/git/revisions/v3… -- https://bugs.ruby-lang.org/

3 2

[ruby-core:116078] [Ruby master Bug#20162] Memory leak when duplicating too complex object
by peterzhu2118 (Peter Zhu) 08 Jul '24

08 Jul '24

Issue #20162 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20162: Memory leak when duplicating too complex object https://bugs.ruby-lang.org/issues/20162 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: DONTNEED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9449 Creating a ST table then calling st_replace leaks memory because the st_replace overwrites the ST table without freeing any of the existing memory. This commit changes it to use st_copy instead. For example: ```ruby RubyVM::Shape.exhaust_shapes o = Object.new o.instance_variable_set(:@a, 0) 10.times do 100_000.times { o.dup } puts `ps -o rss= -p #{$$}` end ``` Before: ``` 23264 33600 42672 52160 61600 71728 81056 90528 100560 109840 ``` After: ``` 14752 14816 15584 15584 15664 15664 15664 15664 15664 15664 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114932] [Ruby master Bug#19907] Method calls with keyword arguments in eval leaks callcache and callinfo objects
by peterzhu2118 (Peter Zhu) 07 Jul '24

07 Jul '24

Issue #19907 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19907: Method calls with keyword arguments in eval leaks callcache and callinfo objects https://bugs.ruby-lang.org/issues/19907 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following script leaks callcache and callinfo objects: ```ruby def foo(a:); end 10.times do 10_000.times do eval(<<~RUBY) foo(a: 1) RUBY end puts "Number of live objects: #{GC.stat(:heap_live_slots)}" puts "Memory usage: #{`ps -o rss= -p #{$$}`}" puts end ``` Output is: ``` Number of live objects: 46248 Memory usage: 16160 Number of live objects: 65902 Memory usage: 19888 Number of live objects: 92656 Memory usage: 24032 Number of live objects: 126791 Memory usage: 28048 Number of live objects: 132919 Memory usage: 28816 Number of live objects: 180687 Memory usage: 32384 Number of live objects: 181957 Memory usage: 32464 Number of live objects: 227485 Memory usage: 34224 Number of live objects: 256101 Memory usage: 37200 Number of live objects: 274151 Memory usage: 38752 ``` After performing a `ObjectSpace.dump_all`, I found that it is leaking callcache and callinfo objects that is being held on by the `Object` class. ```json {"address":"0x102ecef70", "type":"CLASS", "shape_id":2, "slot_size":160, "class":"0x102ecf8d0", "variation_count":0, "superclass":"0x102ecfd30", "real_class_name":"Object", "singleton":true, "references":[ ... ]} {"address":"0x1030c90a0", "type":"IMEMO", "shape_id":0, "slot_size":40, "imemo_type":"callinfo", "mid":"foo", "memsize":40, "flags":{"wb_protected":true}} {"address":"0x102fad568", "type":"IMEMO", "shape_id":0, "slot_size":40, "imemo_type":"callcache", "memsize":40, "flags":{"wb_protected":true, "old":true, "uncollectible":true, "marked":true}} ``` -- https://bugs.ruby-lang.org/

5 4

[ruby-core:115999] [Ruby master Bug#20145] Memory leak when duplicating identhash
by peterzhu2118 (Peter Zhu) 07 Jul '24

07 Jul '24

Issue #20145 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20145: Memory leak when duplicating identhash https://bugs.ruby-lang.org/issues/20145 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9411 Both copy_compare_by_id and hash_copy will create a copy of the ST table, so the ST table created in copy_compare_by_id will be leaked. ```ruby h = { 1 => 2 }.compare_by_identity 10.times do 1_000_000.times do h.select { false } end puts `ps -o rss= -p #{$$}` end ``` Before: ``` 110736 204352 300272 395520 460704 476736 542000 604704 682624 770528 ``` After: ``` 15504 16048 16144 16256 16320 16320 16752 16752 16752 16752 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:116514] [Ruby master Bug#20228] Memory leak in Regexp timeout
by peterzhu2118 (Peter Zhu) 07 Jul '24

07 Jul '24

Issue #20228 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20228: Memory leak in Regexp timeout https://bugs.ruby-lang.org/issues/20228 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: DONTNEED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9765 If a Regexp::TimeoutError is raised, the `stk_base` and `OnigRegion` will leak. For example: ```ruby Regexp.timeout = 0.001 regex = /^(a*)*$/ str = "a" * 1000000 + "x" 10.times do 100.times do begin regex =~ str rescue end end puts `ps -o rss= -p #{$$}` end ``` Before: ``` 328800 632416 934368 1230448 1531088 1831248 2125072 2414384 2703440 2995664 ``` After: ``` 39280 47888 49024 56240 56496 56512 56592 56592 56720 56720 ``` -- https://bugs.ruby-lang.org/

5 5

[ruby-core:111306] [Ruby master Feature#19236] Allow to create hashes with a specific capacity from Ruby
by byroot (Jean Boussier) 07 Jul '24

07 Jul '24

Issue #19236 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19236: Allow to create hashes with a specific capacity from Ruby https://bugs.ruby-lang.org/issues/19236 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.3 ---------------------------------------- Followup on [Feature #18683] which added a C-API for this purpose. Various protocol parsers such as Redis `RESP3` or `msgpack`, have to create hashes, and they know the size in advance. For efficiency, it would be preferable if they could directly allocate a Hash of the necessary size, so that large hashes wouldn't cause many re-alloccations and re-hash. `String` and `Array` both already offer similar APIs: ```ruby String.new(capacity: XXX) Array.new(XX) / rb_ary_new_capa(long) ``` However there's no such public API for Hashes in Ruby land. ### Proposal I think `Hash` should have a way to create a new hash with a `capacity` parameter. The logical signature of `Hash.new(capacity: 1000)` was deemed too incompatible in [Feature #18683]. @Eregon proposed to add `Hash.create(capacity: 1000)`. -- https://bugs.ruby-lang.org/

8 14

[ruby-core:116960] [Ruby master Bug#20305] commit 1d2d25dcadda0764f303183ac091d0c87b432566 breaks grapheme_clusters
by fablestales (Fable Tales) 06 Jul '24

06 Jul '24

Issue #20305 has been reported by fablestales (Fable Tales). ---------------------------------------- Bug #20305: commit 1d2d25dcadda0764f303183ac091d0c87b432566 breaks grapheme_clusters https://bugs.ruby-lang.org/issues/20305 * Author: fablestales (Fable Tales) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- given a script: ``` #script.rb p "안녕".byteslice(0, 4).grapheme_clusters ``` The commit 1d2d25dcadda0764f303183ac091d0c87b432566 (https://github.com/ruby/ruby/commit/1d2d25dcadda0764f303183ac091d0c87b432566) breaks the grapheme_clusters method on a byte slice ``` (commit 1d2d25dcadda0764f303183ac091d0c87b432566) ((HEAD detached at 1d2d25dcad)) $ ./ruby --disable=gems script.rb ["안", "\xEB"] ((HEAD detached at 1d2d25dcad)) $ git checkout HEAD^ (114e71d06280f9c57b9859ee4405ae89a989ddb6) ((HEAD detached at 114e71d062)) $ make -j ... ((HEAD detached at 114e71d062)) $ ./ruby --disable=gems script.rb ["안"] ((HEAD detached at 114e71d062)) $ cat script.rb p "안녕".byteslice(0, 4).grapheme_clusters ``` the expected result here is almost certainly the latter output, and not the former. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:116894] [Ruby master Bug#20288] `rb_fiber_scheduler_close` exceptions are not handled in `rb_fiber_scheduler_set`.
by ioquatix (Samuel Williams) 06 Jul '24

06 Jul '24

Issue #20288 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20288: `rb_fiber_scheduler_close` exceptions are not handled in `rb_fiber_scheduler_set`. https://bugs.ruby-lang.org/issues/20288 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The code responsible for setting a new scheduler on a thread does not properly handle exceptions raised by `rb_fiber_scheduler_close`. If `rb_fiber_scheduler_close` raised an exception, the assignment `thread->scheduler = scheduler` would not be executed. This leaves the thread in an undefined state because it might still hold a reference to the old scheduler, which was supposed to be closed and replaced. ## Steps to Reproduce: 1. Define a custom fiber scheduler that raises an exception in its close method. 2. Set the custom scheduler on a thread. 3. Attempt to replace the custom scheduler with a different scheduler. 4. Observe that if the close method of the original scheduler raises an exception, the thread's scheduler reference is not updated. ## Proposed Fix: The use of `rb_ensure` can be introduced to wrap the call to `rb_fiber_scheduler_close` to ensure that, regardless of whether an exception is raised, `thread->scheduler` is set to `Qnil`, and then `thread->scheduler = scheduler` is safely executed to update the thread's scheduler reference. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:117321] [Ruby master Bug#20393] `after_fork_ruby` clears all pending interrupts for both parent and child process.
by ioquatix (Samuel Williams) 06 Jul '24

06 Jul '24

Issue #20393 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20393: `after_fork_ruby` clears all pending interrupts for both parent and child process. https://bugs.ruby-lang.org/issues/20393 * Author: ioquatix (Samuel Williams) * Status: Open * Assignee: ioquatix (Samuel Williams) * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- In the following program, the behaviour of the parent process is affected by whether `Process.fork` is invoked or not. ```ruby Thread.handle_interrupt(RuntimeError => :never) do Thread.current.raise(RuntimeError, "Queued error") puts "Pending interrupt: #{Thread.pending_interrupt?}" # true pid = Process.fork do puts "Pending interrupt (child process): #{Thread.pending_interrupt?}" Thread.handle_interrupt(RuntimeError => :immediate){} end _, status = Process.waitpid2(pid) puts "Child process status: #{status.inspect}" puts "Pending interrupt: #{Thread.pending_interrupt?}" # false end puts "Exiting..." ``` I don't think the parent process pending interrupts should be cleared by `after_fork_ruby`: ```c static void after_fork_ruby(rb_pid_t pid) { rb_threadptr_pending_interrupt_clear(GET_THREAD()); if (pid == 0) { // child clear_pid_cache(); rb_thread_atfork(); } else { // parent after_exec(); } } ``` How about this implementation: ```c static void after_fork_ruby(rb_pid_t pid) { if (pid == 0) { // child rb_threadptr_pending_interrupt_clear(GET_THREAD()); clear_pid_cache(); rb_thread_atfork(); } else { // parent after_exec(); } } ``` cc @ko1 -- https://bugs.ruby-lang.org/

3 5

[ruby-core:116890] [Ruby master Bug#20286] TracePoint does not emit `thread_end` event when thread exits with exception
by ioquatix (Samuel Williams) 06 Jul '24

06 Jul '24

Issue #20286 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20286: TracePoint does not emit `thread_end` event when thread exits with exception https://bugs.ruby-lang.org/issues/20286 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Using TracePoint to trace `thread_begin` and `thread_end` events fails to emit the `thread_end` event when an exception (e.g., Interrupt) is raised within a thread. This behavior occurs because the exception handling bypasses the internal thread finishing logic, including trace point and fiber scheduler cleanup code. This issue affects the ability to accurately monitor thread lifecycle events in scenarios involving exception handling or abrupt thread terminations. ## Steps to Reproduce: 1. Set up `TracePoint` to trace `thread_begin` and `thread_end` events. 2. Create a new thread that raises an exception. 3. Join the thread and observe that only the `thread_begin` event is emitted without a corresponding `thread_end` event. ## Expected Behavior: The `TracePoint` should emit both `thread_begin` and `thread_end` events, accurately reflecting the lifecycle of the thread, even when an exception is raised within the thread. ## Actual Behavior: The `TracePoint` emits the `thread_begin` event but fails to emit the `thread_end` event when an exception is raised within the thread, indicating an incomplete tracing of thread lifecycle events. I've confirmed this as far back as Ruby 2.6. ## Example Code ```ruby TracePoint.trace(:thread_begin, :thread_end) do |tp| p [tp.event, tp.lineno, tp.path, tp.defined_class, tp.method_id] end thread = Thread.new do raise Interrupt end thread.join ``` ## Possible Fix Changing the implementation of `thread_do_start` to have what amounts to an "ensure" block. ```c static void thread_do_start(rb_thread_t *th) { native_set_thread_name(th); VALUE result = Qundef; rb_execution_context_t *ec = th->ec; int state; EXEC_EVENT_HOOK(ec, RUBY_EVENT_THREAD_BEGIN, th->self, 0, 0, 0, Qundef); EC_PUSH_TAG(ec); if ((state = EC_EXEC_TAG()) == TAG_NONE) { switch (th->invoke_type) { case thread_invoke_type_proc: result = thread_do_start_proc(th); break; case thread_invoke_type_ractor_proc: result = thread_do_start_proc(th); rb_ractor_atexit(th->ec, result); break; case thread_invoke_type_func: result = (*th->invoke_arg.func.func)(th->invoke_arg.func.arg); break; case thread_invoke_type_none: rb_bug("unreachable"); } } EC_POP_TAG(); VALUE errinfo = ec->errinfo; if (!NIL_P(errinfo) && !RB_TYPE_P(errinfo, T_OBJECT)) { ec->errinfo = Qnil; } rb_fiber_scheduler_set(Qnil); EXEC_EVENT_HOOK(th->ec, RUBY_EVENT_THREAD_END, th->self, 0, 0, 0, Qundef); ec->errinfo = errinfo; if (state) EC_JUMP_TAG(ec, state); th->value = result; } ``` It's possible `rb_fiber_scheduler_set(Qnil);` can emit an exception itself. How do we write the code to handle that case? -- https://bugs.ruby-lang.org/

3 5