May 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:113052] [Ruby master Bug#19562] Setting default_(in|ex)ternal encoding after running an Ractor freezes ruby
by larskanis (Lars Kanis) 20 Sep '24

20 Sep '24

Issue #19562 has been reported by larskanis (Lars Kanis). ---------------------------------------- Bug #19562: Setting default_(in|ex)ternal encoding after running an Ractor freezes ruby https://bugs.ruby-lang.org/issues/19562 * Author: larskanis (Lars Kanis) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following command blocks infinitely and must be killed with "kill -9" ``` ruby -e "Ractor.new{}.take; Encoding.default_external = Encoding::ISO8859_2" ``` Backtrace is: ``` #0 futex_wait (private=0, expected=2, futex_word=0x55555555abd0) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x55555555abd0, private=0) at ./nptl/lowlevellock.c:49 #2 0x00007ffff760a082 in lll_mutex_lock_optimized (mutex=0x55555555abd0) at ./nptl/pthread_mutex_lock.c:48 #3 ___pthread_mutex_lock (mutex=mutex@entry=0x55555555abd0) at ./nptl/pthread_mutex_lock.c:93 #4 0x00007ffff7c5a40d in rb_native_mutex_lock (lock=lock@entry=0x55555555abd0) at ruby-3.2.1/thread_pthread.c:556 #5 0x00007ffff7cc5376 in vm_lock_enter (cr=0x55555555deb0, vm=0x55555555aba0, locked=locked@entry=false, no_barrier=no_barrier@entry=false, lev=0x55555555ac00) at vm_sync.c:56 #6 0x00007ffff7cc5747 in rb_vm_lock_body () at vm_sync.c:175 #7 0x00007ffff7bc02c2 in rb_vm_lock (line=1829, file=0x7ffff7e209cb "ractor.c") at ruby-3.2.1/vm_sync.h:57 #8 ractor_check_blocking (remained_thread_cnt=<optimized out>, file=0x7ffff7e209cb "ractor.c", line=1865, cr=0x55555555deb0) at ractor.c:1829 #9 ractor_check_blocking (line=1865, file=0x7ffff7e209cb "ractor.c", remained_thread_cnt=<optimized out>, cr=0x55555555deb0) at ractor.c:1811 #10 rb_ractor_blocking_threads_inc (cr=0x55555555deb0, file=file@entry=0x7ffff7e29b04 "thread.c", line=line@entry=1488) at ractor.c:1865 #11 0x00007ffff7c60977 in blocking_region_begin (th=0x55555555d070, region=0x7fffffffd220, ubf=0x7ffff7c5b300 <ubf_select>, arg=0x55555555d070, fail_if_interrupted=<optimized out>) at thread.c:1488 #12 0x00007ffff7c61edf in rb_nogvl (func=0x7ffff7ac66e0 <no_gvl_stat>, data1=0x7fffffffd2a0, ubf=<optimized out>, data2=0x55555555d070, flags=<optimized out>) at thread.c:1548 #13 0x00007ffff7acdfbc in stat_without_gvl (st=0x7fffffffd2b0, path=0x7ffff27dfe88 "ruby-3.2.1/lib/ruby/3.2.0/x86_64-linux/enc/iso_8859_2.so") at file.c:1171 #14 rb_check_realpath_internal (basedir=basedir@entry=4, path=path@entry=140737261984200, origenc=0x555555565030, mode=mode@entry=RB_REALPATH_STRICT) at file.c:4491 #15 0x00007ffff7acebaa in rb_realpath_internal (basedir=basedir@entry=4, path=path@entry=140737261984200, strict=strict@entry=1) at file.c:4536 #16 0x00007ffff7b2cdd4 in require_internal (ec=0x55555555e1f0, fname=<optimized out>, exception=1, warn=<optimized out>) at load.c:1195 #17 0x00007ffff7aa0a17 in load_encoding (name=0x5555556049b0 "ISO-8859-2") at encoding.c:779 #18 0x00007ffff7aa2ef8 in check_encoding (enc=0x5555556049d0) at encoding.c:191 #19 enc_check_encoding (obj=140737335533800) at encoding.c:202 #20 rb_to_encoding (enc=enc@entry=140737335533800) at encoding.c:307 #21 0x00007ffff7aa4049 in enc_set_default_encoding (name=0x7ffff7de6287 "external", encoding=140737335533800, def=0x7ffff7fb00c0 <default_external>) at encoding.c:1563 #22 rb_enc_set_default_external (encoding=140737335533800) at encoding.c:1637 #23 0x00007ffff7aa411f in set_default_external (klass=<optimized out>, encoding=140737335533800) at encoding.c:1658 #24 0x00007ffff7c99037 in vm_call_cfunc_with_frame (ec=0x55555555e1f0, reg_cfp=0x7ffff6f7df90, calling=<optimized out>) at ruby-3.2.1/vm_insnhelper.c:3268 #25 0x00007ffff7ca9564 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>) at ruby-3.2.1/vm_callinfo.h:367 #26 vm_exec_core (ec=0x55555555e1f0, initial=128) at ruby-3.2.1/insns.def:820 #27 0x00007ffff7caee93 in rb_vm_exec (ec=0x55555555e1f0, jit_enable_p=true) at vm.c:2374 #28 0x00007ffff7abfbf1 in rb_ec_exec_node (ec=ec@entry=0x55555555e1f0, n=n@entry=0x7ffff796c2c0) at eval.c:289 #29 0x00007ffff7ac60b3 in ruby_run_node (n=0x7ffff796c2c0) at eval.c:330 #30 0x0000555555555187 in rb_main (argv=0x7fffffffdb98, argc=3) at ./main.c:38 #31 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57 ``` When the encoding was already loaded before running `Ractor.new`, then it works and doesn't block: ``` ruby -e "''.encode(Encoding::ISO8859_2); Ractor.new{}.take; Encoding.default_internal = Encoding::ISO8859_2" ``` When the encoding is not assigned as default encoding, but used otherwise, it works too: ``` ruby -e "Ractor.new{}.take; ''.encode(Encoding::ISO8859_2)" ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:111144] [Ruby master Feature#19171] Update Unicode data to Unicode Version 15.1
by duerst 20 Sep '24

20 Sep '24

Issue #19171 has been reported by duerst (Martin Dürst). ---------------------------------------- Feature #19171: Update Unicode data to Unicode Version 15.1 https://bugs.ruby-lang.org/issues/19171 * Author: duerst (Martin Dürst) * Status: Open * Priority: Normal * Assignee: duerst (Martin Dürst) ---------------------------------------- According to http://blog.unicode.org/2022/11/the-unicode-standard-2023-release.html, Unicode plans to release Version 15.1 in September 2023. According to https://www.unicode.org/versions/beta.html, public alpha review starts Feb. 7, 2023, and ends April 4, 2023. Because alpha review may not include all the files we use, it may be difficult for us to participate. Public beta review is planned to start May 23, 2023, ending July 4, 2023. At this point, we should be able to test things. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116581] [Ruby master Bug#20237] Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread
by hanazuki (Kasumi Hanazuki) 16 Sep '24

16 Sep '24

Issue #20237 has been reported by hanazuki (Kasumi Hanazuki). ---------------------------------------- Bug #20237: Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread https://bugs.ruby-lang.org/issues/20237 * Author: hanazuki (Kasumi Hanazuki) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Backgrounds [unshare(2)](https://man7.org/linux/man-pages/man2/unshare.2.html) is a syscall in Linux to move the calling process into a fresh execution context. With `unshare(CLONE_NEWUSER)` you can move a process into a new [user_namespace(7)](https://man7.org/linux/man-pages/man7/user_namespaces.7.…, where the process gains the full capability on the resources within the namespace. This is fundamental for Linux containers to achieve privilege separation. `unshare(CLONE_NEWUSER)` requires the calling process to be single-threaded (or no background threads are running). So, it is often invoked after `fork(2)` as forking propagates only the calling thread to the child process. ## Problem It becomes a problem that Ruby 3.3 on Linux uses timer threads even for a single-`Thread`ed application. Because `Kernel#fork` spawns a thread in the child process before the control returns to the user code, there is no chance to call `unshare(CLONE_NEWUSER)` in Ruby. The following snippet is a reproducer of this problem. This program first forks and then shows the user namespace to which the process belongs before and after calling unshare(2). It also shows the threads of the child process after forking. ```ruby p(RUBY_DESCRIPTION:) require 'fiddle/import' module C extend Fiddle::Importer dlload 'libc.so.6' extern 'int unshare(int flags)' CLONE_NEWUSER = 0x10000000 def self.raise_system_call_error raise SystemCallError.new(Fiddle.last_error) end end pid = fork do system("ps -O tid -T -p #$$") system("ls -l /proc/self/ns/user") if C.unshare(C::CLONE_NEWUSER) != 0 C.raise_system_call_error # => EINVAL with Ruby 3.3 end system("ls -l /proc/self/ns/user") end p Process.wait2(pid) ``` The program successfully changes the user namespace with Ruby 3.2, but it raises EINVAL with Ruby 3.3. You can see Ruby 3.3 has two threads running after forking. ``` % rbenv shell 3.2 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585787 1585787 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' lrwxrwxrwx 1 nobody nogroup 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026532675]' [1585787, #<Process::Status: pid 1585787 exit 0>] % rbenv shell 3.3 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585849 1585849 S pts/12 00:00:00 ruby ./test.rb 1585849 1585851 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585849, #<Process::Status: pid 1585849 exit 1>] % rbenv shell master && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585965 1585965 S pts/12 00:00:00 ruby ./test.rb 1585965 1585967 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585965, #<Process::Status: pid 1585965 exit 1>] ``` ## Workaround My workaround is to rebuild ruby with `rb_thread_stop_timer_thread` and `rb_thread_start_timer_thread` exported, and use a C-ext that stops the timer thread before calling `unshare`. This seems not robust because the process cannot know when the terminated thread is reclaimed by the kernel, after which the process is considered single-threaded. ```c #define _GNU_SOURCE 1 #include <sched.h> #include <ruby/ruby.h> static VALUE Unshare_s_unshare(VALUE _self, VALUE rflags) { int const flags = NUM2INT(rflags); rb_thread_stop_timer_thread(); usleep(1000); // FIXME: It takes some time for the kernel to remove the stopped thread? int const ret = unshare(flags); rb_thread_start_timer_thread(); if(ret != 0) rb_sys_fail_str(rb_sprintf("unshare(%#x)", flags)); return Qnil; } RUBY_FUNC_EXPORTED void Init_unshare(void) { VALUE rb_mUnshare = rb_define_module("Unshare"); rb_define_singleton_method(rb_mUnshare, "unshare", Unshare_s_unshare, 1); rb_define_const(rb_mUnshare, "CLONE_NEWUSER", INT2FIX(CLONE_NEWUSER)); } ``` ## Questions - Is this a limitation of Ruby? - Is it safe (or even possible) to stop the timer thread during execution? - If so, can we export it as the public API? - But it may not so useful for this problem as explained in the workaround. - Is it guaranteed that no other threads are running after forks? - Are there any better ways to solve this issue? - Can we somehow delay the start of the timer thread after forking, or hook into `fork` to run some code in the child process immediately after it spawns. - Can they be Ruby API instead of C API? -- https://bugs.ruby-lang.org/

4 9

[ruby-core:117783] [Ruby master Bug#20473] Ractor array/hash literals
by kddnewton (Kevin Newton) 13 Sep '24

13 Sep '24

Issue #20473 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20473: Ractor array/hash literals https://bugs.ruby-lang.org/issues/20473 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- There are many optimizations in place in the compiler to ensure Ruby doesn't overflow the stack when an array or hash literal has many values. For example: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` gives the instructions: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,1564)> 0000 putobject "" ( 1)[Li] 0002 putnil 0003 dup 0004 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0006 anytostring 0007 concatstrings 2 0009 newarray 1 0011 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0013 concatarray 0014 putobject "" 0016 putnil 0017 dup 0018 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0020 anytostring 0021 concatstrings 2 0023 newarray 1 0025 concatarray 0026 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0028 concatarray 0029 putobject "" 0031 putnil 0032 dup 0033 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0035 anytostring 0036 concatstrings 2 0038 newarray 1 0040 concatarray 0041 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0043 concatarray 0044 putobject "" 0046 putnil 0047 dup 0048 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0050 anytostring 0051 concatstrings 2 0053 newarray 1 0055 concatarray 0056 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0058 concatarray 0059 leave ``` because it is smart enough to group the static elements together and use concatarray. This ensures we never have too many values on the stack. In Ruby 3.3, this worked for ractors as well. (Adding `# shareable_constant_value: literal` and `A =` to the example above doesn't change anything about the instructions above except adding a couple of `-@` calls and one call to `make_shareable`. On HEAD, all of these optimizations are now gone. With this: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm # shareable_constant_value: literal A = [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` code on master, we get: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(2,1568)> 0000 putobject RubyVM::FrozenCore ( 2)[Li] 0002 putobject "" 0004 putnil 0005 dup 0006 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0008 anytostring 0009 concatstrings 2 0011 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0013 putobject_INT2FIX_1_ ~~~~ 0139 putobject_INT2FIX_1_ 0140 putobject "" 0142 putnil 0143 dup 0144 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0146 anytostring 0147 concatstrings 2 0149 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0151 putobject_INT2FIX_1_ ~~~~ 0277 putobject_INT2FIX_1_ 0278 putobject "" 0280 putnil 0281 dup 0282 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0284 anytostring 0285 concatstrings 2 0287 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0289 putobject_INT2FIX_1_ ~~~~ 0415 putobject_INT2FIX_1_ 0416 putobject "" 0418 putnil 0419 dup 0420 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0422 anytostring 0423 concatstrings 2 0425 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0427 putobject_INT2FIX_1_ ~~~~ 0553 putobject_INT2FIX_1_ 0554 newarray 512 0556 opt_send_without_block <calldata!mid:make_shareable, argc:1, ARGS_SIMPLE> 0558 dup 0559 putspecialobject 3 0561 setconstant :A 0563 leave ``` where the `~~~~` represents another 128 `putobject_INT2FIX_1_` instructions. I assume this was done because it used to modify the AST and that got removed. However, I also assume it was not intentional to lose all of these optimizations, as it now means if Ractors are being used that large literals are vulnerable to stack overflows. Please let me know how to proceed, as I am trying to exactly match behavior in the prism compiler. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:117210] [Ruby master Bug#20344] argument stack underflow (-1)
by nobu (Nobuyoshi Nakada) 02 Sep '24

02 Sep '24

Issue #20344 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20344: argument stack underflow (-1) https://bugs.ruby-lang.org/issues/20344 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- This file produces argument stack underflow, since ruby 2.5. ```ruby proc do next if true case nil when "a" next when "b" when "c" proc {} end next unless true end ``` ``` -- raw disasm-------- trace: 100 0000 nop ( 1) <L000> [sp: 0, unremovable: 1, refcnt: 5] trace: 1 <L004> [sp: 0, unremovable: 0, refcnt: 1] adjust: [label: 0] 0001 putnil ( 2) 0002 leave ( 13) <L008> [sp: 1, unremovable: 0, refcnt: 1] adjust: [label: 0] 0003 putnil ( 6) 0004 leave ( 13) adjust: [label: 8] 0005 jump <L005> ( 5) 0007 pop ( 7) 0008 jump <L005> ( 7) * 0010 pop ( 8) trace: 1 <L011> [sp: -1, unremovable: 1, refcnt: 1] 0011 putself ( 9) 0012 send <calldata:proc, 0>, nil ( 9) <L012> [sp: -1, unremovable: 0, refcnt: 2] 0015 pop ( 9) <L005> [sp: 1, unremovable: 0, refcnt: 2] trace: 1 0016 putnil ( 12) <L001> [sp: -1, unremovable: 0, refcnt: 3] trace: 200 0017 leave ( 13) --------------------- ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114645] [Ruby master Bug#19865] Segfault when calling user signal handlers during VM shutdown
by ivoanjo (Ivo Anjo) 27 Aug '24

27 Aug '24

Issue #19865 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #19865: Segfault when calling user signal handlers during VM shutdown https://bugs.ruby-lang.org/issues/19865 * Author: ivoanjo (Ivo Anjo) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Howdy 👋! I work for Datadog [on the `ddtrace` gem](https://github.com/DataDog/dd-trace-rb) . I've found this issue while investigating a [customer crash report](https://github.com/DataDog/dd-trace-rb/issues/2980). ### Background The original issue was found in a production app. A number of things need to be in play to cause it. The [`ruby-odbc`](https://rubygems.org/gems/ruby-odbc) gem provides a way of accessing databases through the ODBC API. It wraps a database connection as a Data object, with a free function that, prior to freeing the native resources, disconnects from the database if the connection was still active. Because disconnecting from the database is a blocking operation, the gem (reasonably, in my opinion), releases the global VM lock before disconnecting. The trigger for the crash is: 1. The app in question used puma, and puma installs a `Signal.trap('TERM')` 2. The database object was still connected when the app started to shut down 3. A VM shutdown starts... 4. Half-way through shutdown, the VM received a SIGTERM signal, and queued it for processing 5. The VM calls the free function on all objects 6. The ruby-odbc gem sees there's an active database connection, and tries to release the GVL to call the blocking disconnect 7. Before releasing the GVL, the VM checks for pending interruptions 8. The VM tries to run the Ruby-level signal handler method half-way through VM shutdown, when you can no longer run Ruby code 9. Segfault ### How to reproduce (Ruby version & script) I was able to reproduce on a minimal example on Ruby 3.2.2 (`ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]`) and recent master (`ruby 3.3.0dev (2023-08-17T07:30:01Z master d26b015e83) [x86_64-linux]`). I've put the test-case up on github as well <https://github.com/DataDog/signal-bug-testcase>, but here's the important bits: `signal-bug-testcase.rb`: ```ruby require 'signal_bug_testcase' Signal.trap("TERM") { puts "Hello, world" } FOO = SignalBugTestcase.new ``` `signal_bug_testcase.c`: ```ruby #include <ruby.h> #include <ruby/thread.h> #include <signal.h> #include <unistd.h> typedef struct { int dummy; } BugTestcase; void *test_nogvl(void *unused) { fprintf(stderr, "GVL released!\n"); return NULL; } static void bug_testcase_free(void* ptr) { fprintf(stderr, "Free getting called! Sending signal...\n"); kill(getpid(), SIGTERM); fprintf(stderr, "SIGTERM signal queued, trying to release GVL...\n"); rb_thread_call_without_gvl(test_nogvl, NULL, NULL, NULL); fprintf(stderr, "After releasing GVL!\n"); free(ptr); } const rb_data_type_t bug_testcase_data_type = { .wrap_struct_name = "SignalBugTestcase", .function = { NULL, bug_testcase_free, NULL }, .flags = RUBY_TYPED_FREE_IMMEDIATELY }; VALUE bug_testcase_alloc(VALUE klass) { BugTestcase* obj = calloc(1, sizeof(BugTestcase)); return TypedData_Make_Struct(klass, BugTestcase, &bug_testcase_data_type, obj); } void Init_signal_bug_testcase(void) { VALUE cBugTestcase = rb_define_class("SignalBugTestcase", rb_cObject); rb_define_alloc_func(cBugTestcase, bug_testcase_alloc); } ``` ### Expectation and result No segfault happens. Interestingly, on Ruby 2.7, the VM exits half-way through but doesn't always segfault, but running it a few times always triggers the issue. On 3.2+ it crashes every time for me. I suspect the right thing here is to no longer accept/try to run any Ruby-level signal handlers after VM shutdown starts. Here's what I see with this test-case: ``` $ bundle exec ruby lib/signal-bug-testcase.rb Free getting called! Sending signal... SIGTERM signal queued, trying to release GVL... lib/signal-bug-testcase.rb:3: [BUG] Segmentation fault at 0x0000000000000007 ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0003 p:---- s:0011 e:000010 CFUNC :puts c:0002 p:0005 s:0006 e:000005 BLOCK lib/signal-bug-testcase.rb:3 [FINISH] c:0001 p:0000 s:0003 E:0001a0 DUMMY [FINISH] -- Ruby level backtrace information ---------------------------------------- lib/signal-bug-testcase.rb:3:in `block in <main>' lib/signal-bug-testcase.rb:3:in `puts' -- Machine register context ------------------------------------------------ RIP: 0x000070aa64cedbe7 RBP: 0x000070aa648e8fd0 RSP: 0x00007ffc5c057608 RAX: 0x0000000000004171 RBX: 0x00007ffc5c057630 RCX: 0x0000000000000001 RDX: 0x00007ffc5c057630 RDI: 0x0000000000000007 RSI: 0x0000000000004171 R8: 0x000000000000021b R9: 0x0000000000000000 R10: 0x000070aa63eff048 R11: 0x0000000000000000 R12: 0x000070aa648e8fd0 R13: 0x0000000000004171 R14: 0x0000000000000007 R15: 0x000070aa648e8ff0 EFL: 0x0000000000010202 -- C level backtrace information ------------------------------------------- ruby-3.2.2/lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x70aa64d5bb5f] ruby-3.2.2/vm_dump.c:785 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_bugreport) ruby-3.2.2/vm_dump.c:1080 ruby-3.2.2/lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf4) [0x70aa64b52164] ruby-3.2.2/error.c:813 ruby-3.2.2/lib/libruby.so.3.2(sigsegv+0x4d) [0x70aa64cab0fd] ruby-3.2.2/signal.c:964 /lib/x86_64-linux-gnu/libc.so.6(0x70aa64642520) [0x70aa64642520] ruby-3.2.2/lib/libruby.so.3.2(hash_table_index+0x0) [0x70aa64cedbe7] ruby-3.2.2/symbol.h:72 ruby-3.2.2/lib/libruby.so.3.2(rb_id_table_lookup) ruby-3.2.2/id_table.c:230 ruby-3.2.2/lib/libruby.so.3.2(cached_callable_method_entry+0x24) [0x70aa64d356bb] ruby-3.2.2/vm_method.c:1295 ruby-3.2.2/lib/libruby.so.3.2(callable_method_entry_or_negative) ruby-3.2.2/vm_method.c:1365 ruby-3.2.2/lib/libruby.so.3.2(callable_method_entry) ruby-3.2.2/vm_method.c:1402 ruby-3.2.2/lib/libruby.so.3.2(rb_callable_method_entry) ruby-3.2.2/vm_method.c:1409 ruby-3.2.2/lib/libruby.so.3.2(gccct_method_search_slowpath+0x38) [0x70aa64d36258] ruby-3.2.2/vm_eval.c:434 ruby-3.2.2/lib/libruby.so.3.2(rb_call0+0x267) [0x70aa64d4e877] ruby-3.2.2/vm_eval.c:483 ruby-3.2.2/lib/libruby.so.3.2(rb_call+0x32) [0x70aa64d4f406] ruby-3.2.2/vm_eval.c:877 ruby-3.2.2/lib/libruby.so.3.2(rb_funcallv_kw) ruby-3.2.2/vm_eval.c:1074 ruby-3.2.2/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x70aa64d30277] ruby-3.2.2/vm_insnhelper.c:3268 ruby-3.2.2/lib/libruby.so.3.2(vm_sendish+0x97) [0x70aa64d407a4] ruby-3.2.2/vm_insnhelper.c:5080 ruby-3.2.2/lib/libruby.so.3.2(vm_exec_core) ruby-3.2.2/insns.def:820 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x70aa64d460d3] ruby-3.2.2/vm.c:2374 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_invoke_proc+0x5f) [0x70aa64d4bfcf] ruby-3.2.2/vm.c:1603 ruby-3.2.2/lib/libruby.so.3.2(vm_call0_body+0x5df) [0x70aa64d4c5cf] ruby-3.2.2/vm_eval.c:274 ruby-3.2.2/lib/libruby.so.3.2(vm_call0_cc+0x77) [0x70aa64d4e7e7] ruby-3.2.2/vm_eval.c:87 ruby-3.2.2/lib/libruby.so.3.2(rb_call0) ruby-3.2.2/vm_eval.c:551 ruby-3.2.2/lib/libruby.so.3.2(rb_call+0x32) [0x70aa64d4f406] ruby-3.2.2/vm_eval.c:877 ruby-3.2.2/lib/libruby.so.3.2(rb_funcallv_kw) ruby-3.2.2/vm_eval.c:1074 ruby-3.2.2/lib/libruby.so.3.2(rb_eval_cmd_kw+0x142) [0x70aa64d4f562] ruby-3.2.2/vm_eval.c:1923 ruby-3.2.2/lib/libruby.so.3.2(signal_exec+0xf6) [0x70aa64caae16] ruby-3.2.2/signal.c:1064 ruby-3.2.2/lib/libruby.so.3.2(rb_threadptr_execute_interrupts+0x36b) [0x70aa64cf7078] ruby-3.2.2/thread.c:2334 ruby-3.2.2/lib/libruby.so.3.2(rb_threadptr_execute_interrupts) ruby-3.2.2/thread.c:2291 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_check_ints+0xb) [0x70aa64cf7ac5] ruby-3.2.2/vm_core.h:1994 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_check_ints) ruby-3.2.2/vm_core.h:1985 ruby-3.2.2/lib/libruby.so.3.2(unblock_function_set) ruby-3.2.2/thread.c:320 ruby-3.2.2/lib/libruby.so.3.2(blocking_region_begin) ruby-3.2.2/thread.c:1485 ruby-3.2.2/lib/libruby.so.3.2(rb_nogvl+0xbf) [0x70aa64cf90cf] ruby-3.2.2/thread.c:1548 signal-bug-testcase-2/lib/signal_bug_testcase.so(fprintf+0x0) [0x70aa6518f299] ../../../../ext/signal_bug_testcase/signal_bug_testcase.c:17 signal-bug-testcase-2/lib/signal_bug_testcase.so(bug_testcase_free) ../../../../ext/signal_bug_testcase/signal_bug_testcase.c:18 ruby-3.2.2/lib/libruby.so.3.2(run_final+0xf) [0x70aa64b73172] ruby-3.2.2/gc.c:4388 ruby-3.2.2/lib/libruby.so.3.2(finalize_list) ruby-3.2.2/gc.c:4407 ruby-3.2.2/lib/libruby.so.3.2(finalize_deferred_heap_pages) ruby-3.2.2/gc.c:4436 ruby-3.2.2/lib/libruby.so.3.2(rb_objspace_call_finalizer+0x350) [0x70aa64b80d70] ruby-3.2.2/gc.c:4573 ruby-3.2.2/lib/libruby.so.3.2(rb_ec_finalize+0x2a) [0x70aa64b5d6d1] ruby-3.2.2/eval.c:168 ruby-3.2.2/lib/libruby.so.3.2(rb_ec_cleanup) ruby-3.2.2/eval.c:262 ruby-3.2.2/lib/libruby.so.3.2(ruby_run_node+0x9d) [0x70aa64b5d91d] ruby-3.2.2/eval.c:330 ruby-3.2.2/bin/ruby(rb_main+0x21) [0x5d5d1295f187] ./main.c:38 ruby-3.2.2/bin/ruby(main) ./main.c:57 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_call_main+0x80) [0x70aa64629d90] ../sysdeps/nptl/libc_start_call_main.h:58 /lib/x86_64-linux-gnu/libc.so.6(call_init+0x0) [0x70aa64629e40] ../csu/libc-start.c:392 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main_impl) ../csu/libc-start.c:379 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main) (null):0 [0x5d5d1295f1d5] ``` ---Files-------------------------------- signal-bug-testcase-main.zip (7.73 KB) -- https://bugs.ruby-lang.org/

2 3

[ruby-core:117882] [Ruby master Bug#20490] Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process
by stanhu (Stan Hu) 18 Aug '24

18 Aug '24

Issue #20490 has been reported by stanhu (Stan Hu). ---------------------------------------- Bug #20490: Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process https://bugs.ruby-lang.org/issues/20490 * Author: stanhu (Stan Hu) * Status: Open * ruby -v: ruby 3.2.4 (2024-04-23 revision af471c0e01) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is a follow-up issue for a bug that I thought was fixed in https://bugs.ruby-lang.org/issues/19837 and duplicated in https://bugs.ruby-lang.org/issues/20181. The following script doesn't terminate quickly in Ruby 3.1.5 and 3.2.4, even with the patches to address https://bugs.ruby-lang.org/issues/19837. It works fine in Ruby 3.3. It appears that the `Process::WNOHANG` argument passed to `Process.wait2` causes this script to spin until the child process stops: ```ruby #!/bin/env ruby Process.spawn({}, "sh -c 'sleep 600'").tap do |pid| puts "detaching PID #{pid}" Process.detach(pid) end forked_pid = fork do loop { sleep 1 } end child_waiter = Thread.new do puts "Waiting for child process to die..." # The spawned process has to exit before this returns in Ruby 3.1 and 3.2 loop do pid, status = Process.wait2(-1, Process::WNOHANG) puts "Exited PID: #{pid}, status: #{status}" break if pid sleep 1 end end process_killer = Thread.new do puts "Killing #{forked_pid}" system("kill #{forked_pid}") end child_waiter.join process_killer.join ``` If I drop the `Process::WNOHANG` argument, it works fine. -- https://bugs.ruby-lang.org/

3 6

[ruby-core:118012] [Ruby master Misc#20509] Document importance of #to_ary and #to_hash for Array#== and Hash#==
by gettalong (Thomas Leitner) 02 Aug '24

02 Aug '24

Issue #20509 has been reported by gettalong (Thomas Leitner). ---------------------------------------- Misc #20509: Document importance of #to_ary and #to_hash for Array#== and Hash#== https://bugs.ruby-lang.org/issues/20509 * Author: gettalong (Thomas Leitner) * Status: Open ---------------------------------------- Both `Array#==` and `Hash#==` provide special behaviour in case the `other` argument is not an Array/Hash but defines the special `#to_ary`/`#to_hash` methods. Those methods are never called, they are just checked for existence. And if they exist, `other#==` is called to allow the `other` argument to decide whether the two objects are equal. I think this is worth mentioning in the documentation for `Array#==` and `Hash#==`. [Background: In my PDF library HexaPDF I have defined two classes `PDFArray` and `Dictionary` which act like Array and Hash but provide special PDF specific behaviour. For PDFArray I defined the `#to_ary` method but for Dictionary just the `#to_h` method. I have come across a bug where comparing Arrays with PDFArrays just works as it should be but comparing Hashes with Dictionaries doesn't due to the absence of `#to_hash` (it seems I removed `Dictionary#to_hash` in 2017 due to problems with automatic destructuring when passing a Dictionary as argument; from what I see that should be no problem anymore, so I will just add it back).] -- https://bugs.ruby-lang.org/

4 3

[ruby-core:118010] [Ruby master Feature#20508] Explicit access to *, **, &, and ...
by bradgessler (Brad Gessler) 01 Aug '24

01 Aug '24

Issue #20508 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Feature #20508: Explicit access to *, **, &, and ... https://bugs.ruby-lang.org/issues/20508 * Author: bradgessler (Brad Gessler) * Status: Open ---------------------------------------- I find debugging and certain meta-programming tasks challenging because there's no explicit APIs for accessing certain types of arguments in Ruby. Here's some example code: ```ruby def splats(one, *, two: nil, **, &) # These work p(*) p(**) # But the block doesn't show a proc. p(&) # This would show [:one, :two, :_] p binding.local_variables end splats(:arg, two: true) do 42 end ``` I'm not sure how to access the `&` variable, unless I name it. The same applies to endless ruby methods. I'd like to see a way to explicitly call bindings for those methods. Something like this (not sure if binding is the right place for it): ```ruby def splats(one, *, true: nil, **, &) binding.arguments # Returns an array binding.keyword_arguments # Returns a hash binding.block_argument # Returns a block end ``` These methods would give me access to the values used to call the method. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:111272] [Ruby master Bug#19231] Integer#step and Float::INFINITY - inconsistent behaviour when called with and without a block
by andrykonchin (Andrew Konchin) 01 Aug '24

01 Aug '24

Issue #19231 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #19231: Integer#step and Float::INFINITY - inconsistent behaviour when called with and without a block https://bugs.ruby-lang.org/issues/19231 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal * ruby -v: 3.1.2 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- The initial issue was reported here https://github.com/oracle/truffleruby/issues/2797. `0.step(Float::INFINITY, 10)` returns: - `Integers` when called with a block - `Floats` when called without a block I would expect `Floats` to be returned in both cases. Examples: ```ruby 0.step(100.0, 10).take(1).map(&:class) # => [Float] ``` ```ruby 0.step(Float::INFINITY, 10) { |offset| p offset.class; break } # Integer ``` When `to` argument is a finite `Float` value then calling with a block returns `Floats` as well: ```ruby 0.step(100.0, 10) { |offset| p offset.class; break } # Float ``` Wondering whether it's intentional behaviour. I've found a related issue https://bugs.ruby-lang.org/issues/15518. -- https://bugs.ruby-lang.org/

6 6