June 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:111144] [Ruby master Feature#19171] Update Unicode data to Unicode Version 15.1
by duerst 20 Sep '24

20 Sep '24

Issue #19171 has been reported by duerst (Martin Dürst). ---------------------------------------- Feature #19171: Update Unicode data to Unicode Version 15.1 https://bugs.ruby-lang.org/issues/19171 * Author: duerst (Martin Dürst) * Status: Open * Priority: Normal * Assignee: duerst (Martin Dürst) ---------------------------------------- According to http://blog.unicode.org/2022/11/the-unicode-standard-2023-release.html, Unicode plans to release Version 15.1 in September 2023. According to https://www.unicode.org/versions/beta.html, public alpha review starts Feb. 7, 2023, and ends April 4, 2023. Because alpha review may not include all the files we use, it may be difficult for us to participate. Public beta review is planned to start May 23, 2023, ending July 4, 2023. At this point, we should be able to test things. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116581] [Ruby master Bug#20237] Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread
by hanazuki (Kasumi Hanazuki) 16 Sep '24

16 Sep '24

Issue #20237 has been reported by hanazuki (Kasumi Hanazuki). ---------------------------------------- Bug #20237: Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread https://bugs.ruby-lang.org/issues/20237 * Author: hanazuki (Kasumi Hanazuki) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Backgrounds [unshare(2)](https://man7.org/linux/man-pages/man2/unshare.2.html) is a syscall in Linux to move the calling process into a fresh execution context. With `unshare(CLONE_NEWUSER)` you can move a process into a new [user_namespace(7)](https://man7.org/linux/man-pages/man7/user_namespaces.7.…, where the process gains the full capability on the resources within the namespace. This is fundamental for Linux containers to achieve privilege separation. `unshare(CLONE_NEWUSER)` requires the calling process to be single-threaded (or no background threads are running). So, it is often invoked after `fork(2)` as forking propagates only the calling thread to the child process. ## Problem It becomes a problem that Ruby 3.3 on Linux uses timer threads even for a single-`Thread`ed application. Because `Kernel#fork` spawns a thread in the child process before the control returns to the user code, there is no chance to call `unshare(CLONE_NEWUSER)` in Ruby. The following snippet is a reproducer of this problem. This program first forks and then shows the user namespace to which the process belongs before and after calling unshare(2). It also shows the threads of the child process after forking. ```ruby p(RUBY_DESCRIPTION:) require 'fiddle/import' module C extend Fiddle::Importer dlload 'libc.so.6' extern 'int unshare(int flags)' CLONE_NEWUSER = 0x10000000 def self.raise_system_call_error raise SystemCallError.new(Fiddle.last_error) end end pid = fork do system("ps -O tid -T -p #$$") system("ls -l /proc/self/ns/user") if C.unshare(C::CLONE_NEWUSER) != 0 C.raise_system_call_error # => EINVAL with Ruby 3.3 end system("ls -l /proc/self/ns/user") end p Process.wait2(pid) ``` The program successfully changes the user namespace with Ruby 3.2, but it raises EINVAL with Ruby 3.3. You can see Ruby 3.3 has two threads running after forking. ``` % rbenv shell 3.2 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585787 1585787 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' lrwxrwxrwx 1 nobody nogroup 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026532675]' [1585787, #<Process::Status: pid 1585787 exit 0>] % rbenv shell 3.3 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585849 1585849 S pts/12 00:00:00 ruby ./test.rb 1585849 1585851 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585849, #<Process::Status: pid 1585849 exit 1>] % rbenv shell master && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585965 1585965 S pts/12 00:00:00 ruby ./test.rb 1585965 1585967 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585965, #<Process::Status: pid 1585965 exit 1>] ``` ## Workaround My workaround is to rebuild ruby with `rb_thread_stop_timer_thread` and `rb_thread_start_timer_thread` exported, and use a C-ext that stops the timer thread before calling `unshare`. This seems not robust because the process cannot know when the terminated thread is reclaimed by the kernel, after which the process is considered single-threaded. ```c #define _GNU_SOURCE 1 #include <sched.h> #include <ruby/ruby.h> static VALUE Unshare_s_unshare(VALUE _self, VALUE rflags) { int const flags = NUM2INT(rflags); rb_thread_stop_timer_thread(); usleep(1000); // FIXME: It takes some time for the kernel to remove the stopped thread? int const ret = unshare(flags); rb_thread_start_timer_thread(); if(ret != 0) rb_sys_fail_str(rb_sprintf("unshare(%#x)", flags)); return Qnil; } RUBY_FUNC_EXPORTED void Init_unshare(void) { VALUE rb_mUnshare = rb_define_module("Unshare"); rb_define_singleton_method(rb_mUnshare, "unshare", Unshare_s_unshare, 1); rb_define_const(rb_mUnshare, "CLONE_NEWUSER", INT2FIX(CLONE_NEWUSER)); } ``` ## Questions - Is this a limitation of Ruby? - Is it safe (or even possible) to stop the timer thread during execution? - If so, can we export it as the public API? - But it may not so useful for this problem as explained in the workaround. - Is it guaranteed that no other threads are running after forks? - Are there any better ways to solve this issue? - Can we somehow delay the start of the timer thread after forking, or hook into `fork` to run some code in the child process immediately after it spawns. - Can they be Ruby API instead of C API? -- https://bugs.ruby-lang.org/

4 9

[ruby-core:117783] [Ruby master Bug#20473] Ractor array/hash literals
by kddnewton (Kevin Newton) 13 Sep '24

13 Sep '24

Issue #20473 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20473: Ractor array/hash literals https://bugs.ruby-lang.org/issues/20473 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- There are many optimizations in place in the compiler to ensure Ruby doesn't overflow the stack when an array or hash literal has many values. For example: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` gives the instructions: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,1564)> 0000 putobject "" ( 1)[Li] 0002 putnil 0003 dup 0004 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0006 anytostring 0007 concatstrings 2 0009 newarray 1 0011 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0013 concatarray 0014 putobject "" 0016 putnil 0017 dup 0018 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0020 anytostring 0021 concatstrings 2 0023 newarray 1 0025 concatarray 0026 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0028 concatarray 0029 putobject "" 0031 putnil 0032 dup 0033 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0035 anytostring 0036 concatstrings 2 0038 newarray 1 0040 concatarray 0041 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0043 concatarray 0044 putobject "" 0046 putnil 0047 dup 0048 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0050 anytostring 0051 concatstrings 2 0053 newarray 1 0055 concatarray 0056 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0058 concatarray 0059 leave ``` because it is smart enough to group the static elements together and use concatarray. This ensures we never have too many values on the stack. In Ruby 3.3, this worked for ractors as well. (Adding `# shareable_constant_value: literal` and `A =` to the example above doesn't change anything about the instructions above except adding a couple of `-@` calls and one call to `make_shareable`. On HEAD, all of these optimizations are now gone. With this: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm # shareable_constant_value: literal A = [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` code on master, we get: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(2,1568)> 0000 putobject RubyVM::FrozenCore ( 2)[Li] 0002 putobject "" 0004 putnil 0005 dup 0006 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0008 anytostring 0009 concatstrings 2 0011 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0013 putobject_INT2FIX_1_ ~~~~ 0139 putobject_INT2FIX_1_ 0140 putobject "" 0142 putnil 0143 dup 0144 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0146 anytostring 0147 concatstrings 2 0149 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0151 putobject_INT2FIX_1_ ~~~~ 0277 putobject_INT2FIX_1_ 0278 putobject "" 0280 putnil 0281 dup 0282 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0284 anytostring 0285 concatstrings 2 0287 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0289 putobject_INT2FIX_1_ ~~~~ 0415 putobject_INT2FIX_1_ 0416 putobject "" 0418 putnil 0419 dup 0420 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0422 anytostring 0423 concatstrings 2 0425 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0427 putobject_INT2FIX_1_ ~~~~ 0553 putobject_INT2FIX_1_ 0554 newarray 512 0556 opt_send_without_block <calldata!mid:make_shareable, argc:1, ARGS_SIMPLE> 0558 dup 0559 putspecialobject 3 0561 setconstant :A 0563 leave ``` where the `~~~~` represents another 128 `putobject_INT2FIX_1_` instructions. I assume this was done because it used to modify the AST and that got removed. However, I also assume it was not intentional to lose all of these optimizations, as it now means if Ractors are being used that large literals are vulnerable to stack overflows. Please let me know how to proceed, as I am trying to exactly match behavior in the prism compiler. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:118388] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding
by byroot (Jean Boussier) 11 Sep '24

11 Sep '24

Issue #20594 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/

12 31

[ruby-core:118180] [Ruby master Bug#20525] Percent string literal with indentation support
by bradgessler (Brad Gessler) 03 Sep '24

03 Sep '24

Issue #20525 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Bug #20525: Percent string literal with indentation support https://bugs.ruby-lang.org/issues/20525 * Author: bradgessler (Brad Gessler) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I have code that looks like this in an application: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown <<~MARKDOWN * Because its fun * Because its super-de-dooper MARKDOWN }, ``` The "squiggle" HEREDOCs strips the indentation, but the name of the HEREDOC doesn't look that great since "Markdown" appears twice. What I'd prefer is a string literal that deals with indentation, maybe it looks something like this: ``` ContentSlide(title: "Why Phlex?"){ Markdown %~{ # Why do you like markdown? * Because its fun * Because its super-de-dooper } }, ``` If Ruby doesn't want to go down the path of adding another type of literal, I'd propose adding a method to String that does the same thing as the `~` HEREDOC so something like this is possible: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown %{ # Why do you like markdown? * Because its fun * Because its super-de-dooper }.deindent }, ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:117210] [Ruby master Bug#20344] argument stack underflow (-1)
by nobu (Nobuyoshi Nakada) 02 Sep '24

02 Sep '24

Issue #20344 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20344: argument stack underflow (-1) https://bugs.ruby-lang.org/issues/20344 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- This file produces argument stack underflow, since ruby 2.5. ```ruby proc do next if true case nil when "a" next when "b" when "c" proc {} end next unless true end ``` ``` -- raw disasm-------- trace: 100 0000 nop ( 1) <L000> [sp: 0, unremovable: 1, refcnt: 5] trace: 1 <L004> [sp: 0, unremovable: 0, refcnt: 1] adjust: [label: 0] 0001 putnil ( 2) 0002 leave ( 13) <L008> [sp: 1, unremovable: 0, refcnt: 1] adjust: [label: 0] 0003 putnil ( 6) 0004 leave ( 13) adjust: [label: 8] 0005 jump <L005> ( 5) 0007 pop ( 7) 0008 jump <L005> ( 7) * 0010 pop ( 8) trace: 1 <L011> [sp: -1, unremovable: 1, refcnt: 1] 0011 putself ( 9) 0012 send <calldata:proc, 0>, nil ( 9) <L012> [sp: -1, unremovable: 0, refcnt: 2] 0015 pop ( 9) <L005> [sp: 1, unremovable: 0, refcnt: 2] trace: 1 0016 putnil ( 12) <L001> [sp: -1, unremovable: 0, refcnt: 3] trace: 200 0017 leave ( 13) --------------------- ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118270] [Ruby master Bug#20571] Compile error on for loop
by kddnewton (Kevin Newton) 02 Sep '24

02 Sep '24

Issue #20571 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20571: Compile error on for loop https://bugs.ruby-lang.org/issues/20571 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby for _ in 1.5 or _ in [] end ``` raises a compile error for the current compiler only with ``` -- raw disasm-------- trace: 1 <L000> [sp: 0, unremovable: 1, refcnt: 1] 0000 putobject 1.5 ( 1) 0002 jump <L002> ( 1) <L006> [sp: 1, unremovable: 1, refcnt: 1] <L004> [sp: 1, unremovable: 0, refcnt: 0] 0004 pop ( 1) * 0005 pop ( 1) 0006 putobject false ( 1) <L002> [sp: 1, unremovable: 0, refcnt: 1] 0008 send <calldata:each, 0>, nil ( 1) <L001> [sp: -1, unremovable: 0, refcnt: 2] 0011 leave ( 1) --------------------- test.rb:1: argument stack underflow (-1) test.rb: compile error (SyntaxError) ``` -- https://bugs.ruby-lang.org/

4 4

[ruby-core:114645] [Ruby master Bug#19865] Segfault when calling user signal handlers during VM shutdown
by ivoanjo (Ivo Anjo) 27 Aug '24

27 Aug '24

Issue #19865 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #19865: Segfault when calling user signal handlers during VM shutdown https://bugs.ruby-lang.org/issues/19865 * Author: ivoanjo (Ivo Anjo) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Howdy 👋! I work for Datadog [on the `ddtrace` gem](https://github.com/DataDog/dd-trace-rb) . I've found this issue while investigating a [customer crash report](https://github.com/DataDog/dd-trace-rb/issues/2980). ### Background The original issue was found in a production app. A number of things need to be in play to cause it. The [`ruby-odbc`](https://rubygems.org/gems/ruby-odbc) gem provides a way of accessing databases through the ODBC API. It wraps a database connection as a Data object, with a free function that, prior to freeing the native resources, disconnects from the database if the connection was still active. Because disconnecting from the database is a blocking operation, the gem (reasonably, in my opinion), releases the global VM lock before disconnecting. The trigger for the crash is: 1. The app in question used puma, and puma installs a `Signal.trap('TERM')` 2. The database object was still connected when the app started to shut down 3. A VM shutdown starts... 4. Half-way through shutdown, the VM received a SIGTERM signal, and queued it for processing 5. The VM calls the free function on all objects 6. The ruby-odbc gem sees there's an active database connection, and tries to release the GVL to call the blocking disconnect 7. Before releasing the GVL, the VM checks for pending interruptions 8. The VM tries to run the Ruby-level signal handler method half-way through VM shutdown, when you can no longer run Ruby code 9. Segfault ### How to reproduce (Ruby version & script) I was able to reproduce on a minimal example on Ruby 3.2.2 (`ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]`) and recent master (`ruby 3.3.0dev (2023-08-17T07:30:01Z master d26b015e83) [x86_64-linux]`). I've put the test-case up on github as well <https://github.com/DataDog/signal-bug-testcase>, but here's the important bits: `signal-bug-testcase.rb`: ```ruby require 'signal_bug_testcase' Signal.trap("TERM") { puts "Hello, world" } FOO = SignalBugTestcase.new ``` `signal_bug_testcase.c`: ```ruby #include <ruby.h> #include <ruby/thread.h> #include <signal.h> #include <unistd.h> typedef struct { int dummy; } BugTestcase; void *test_nogvl(void *unused) { fprintf(stderr, "GVL released!\n"); return NULL; } static void bug_testcase_free(void* ptr) { fprintf(stderr, "Free getting called! Sending signal...\n"); kill(getpid(), SIGTERM); fprintf(stderr, "SIGTERM signal queued, trying to release GVL...\n"); rb_thread_call_without_gvl(test_nogvl, NULL, NULL, NULL); fprintf(stderr, "After releasing GVL!\n"); free(ptr); } const rb_data_type_t bug_testcase_data_type = { .wrap_struct_name = "SignalBugTestcase", .function = { NULL, bug_testcase_free, NULL }, .flags = RUBY_TYPED_FREE_IMMEDIATELY }; VALUE bug_testcase_alloc(VALUE klass) { BugTestcase* obj = calloc(1, sizeof(BugTestcase)); return TypedData_Make_Struct(klass, BugTestcase, &bug_testcase_data_type, obj); } void Init_signal_bug_testcase(void) { VALUE cBugTestcase = rb_define_class("SignalBugTestcase", rb_cObject); rb_define_alloc_func(cBugTestcase, bug_testcase_alloc); } ``` ### Expectation and result No segfault happens. Interestingly, on Ruby 2.7, the VM exits half-way through but doesn't always segfault, but running it a few times always triggers the issue. On 3.2+ it crashes every time for me. I suspect the right thing here is to no longer accept/try to run any Ruby-level signal handlers after VM shutdown starts. Here's what I see with this test-case: ``` $ bundle exec ruby lib/signal-bug-testcase.rb Free getting called! Sending signal... SIGTERM signal queued, trying to release GVL... lib/signal-bug-testcase.rb:3: [BUG] Segmentation fault at 0x0000000000000007 ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0003 p:---- s:0011 e:000010 CFUNC :puts c:0002 p:0005 s:0006 e:000005 BLOCK lib/signal-bug-testcase.rb:3 [FINISH] c:0001 p:0000 s:0003 E:0001a0 DUMMY [FINISH] -- Ruby level backtrace information ---------------------------------------- lib/signal-bug-testcase.rb:3:in `block in <main>' lib/signal-bug-testcase.rb:3:in `puts' -- Machine register context ------------------------------------------------ RIP: 0x000070aa64cedbe7 RBP: 0x000070aa648e8fd0 RSP: 0x00007ffc5c057608 RAX: 0x0000000000004171 RBX: 0x00007ffc5c057630 RCX: 0x0000000000000001 RDX: 0x00007ffc5c057630 RDI: 0x0000000000000007 RSI: 0x0000000000004171 R8: 0x000000000000021b R9: 0x0000000000000000 R10: 0x000070aa63eff048 R11: 0x0000000000000000 R12: 0x000070aa648e8fd0 R13: 0x0000000000004171 R14: 0x0000000000000007 R15: 0x000070aa648e8ff0 EFL: 0x0000000000010202 -- C level backtrace information ------------------------------------------- ruby-3.2.2/lib/libruby.so.3.2(rb_print_backtrace+0xd) [0x70aa64d5bb5f] ruby-3.2.2/vm_dump.c:785 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_bugreport) ruby-3.2.2/vm_dump.c:1080 ruby-3.2.2/lib/libruby.so.3.2(rb_bug_for_fatal_signal+0xf4) [0x70aa64b52164] ruby-3.2.2/error.c:813 ruby-3.2.2/lib/libruby.so.3.2(sigsegv+0x4d) [0x70aa64cab0fd] ruby-3.2.2/signal.c:964 /lib/x86_64-linux-gnu/libc.so.6(0x70aa64642520) [0x70aa64642520] ruby-3.2.2/lib/libruby.so.3.2(hash_table_index+0x0) [0x70aa64cedbe7] ruby-3.2.2/symbol.h:72 ruby-3.2.2/lib/libruby.so.3.2(rb_id_table_lookup) ruby-3.2.2/id_table.c:230 ruby-3.2.2/lib/libruby.so.3.2(cached_callable_method_entry+0x24) [0x70aa64d356bb] ruby-3.2.2/vm_method.c:1295 ruby-3.2.2/lib/libruby.so.3.2(callable_method_entry_or_negative) ruby-3.2.2/vm_method.c:1365 ruby-3.2.2/lib/libruby.so.3.2(callable_method_entry) ruby-3.2.2/vm_method.c:1402 ruby-3.2.2/lib/libruby.so.3.2(rb_callable_method_entry) ruby-3.2.2/vm_method.c:1409 ruby-3.2.2/lib/libruby.so.3.2(gccct_method_search_slowpath+0x38) [0x70aa64d36258] ruby-3.2.2/vm_eval.c:434 ruby-3.2.2/lib/libruby.so.3.2(rb_call0+0x267) [0x70aa64d4e877] ruby-3.2.2/vm_eval.c:483 ruby-3.2.2/lib/libruby.so.3.2(rb_call+0x32) [0x70aa64d4f406] ruby-3.2.2/vm_eval.c:877 ruby-3.2.2/lib/libruby.so.3.2(rb_funcallv_kw) ruby-3.2.2/vm_eval.c:1074 ruby-3.2.2/lib/libruby.so.3.2(vm_call_cfunc_with_frame+0x127) [0x70aa64d30277] ruby-3.2.2/vm_insnhelper.c:3268 ruby-3.2.2/lib/libruby.so.3.2(vm_sendish+0x97) [0x70aa64d407a4] ruby-3.2.2/vm_insnhelper.c:5080 ruby-3.2.2/lib/libruby.so.3.2(vm_exec_core) ruby-3.2.2/insns.def:820 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_exec+0xd3) [0x70aa64d460d3] ruby-3.2.2/vm.c:2374 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_invoke_proc+0x5f) [0x70aa64d4bfcf] ruby-3.2.2/vm.c:1603 ruby-3.2.2/lib/libruby.so.3.2(vm_call0_body+0x5df) [0x70aa64d4c5cf] ruby-3.2.2/vm_eval.c:274 ruby-3.2.2/lib/libruby.so.3.2(vm_call0_cc+0x77) [0x70aa64d4e7e7] ruby-3.2.2/vm_eval.c:87 ruby-3.2.2/lib/libruby.so.3.2(rb_call0) ruby-3.2.2/vm_eval.c:551 ruby-3.2.2/lib/libruby.so.3.2(rb_call+0x32) [0x70aa64d4f406] ruby-3.2.2/vm_eval.c:877 ruby-3.2.2/lib/libruby.so.3.2(rb_funcallv_kw) ruby-3.2.2/vm_eval.c:1074 ruby-3.2.2/lib/libruby.so.3.2(rb_eval_cmd_kw+0x142) [0x70aa64d4f562] ruby-3.2.2/vm_eval.c:1923 ruby-3.2.2/lib/libruby.so.3.2(signal_exec+0xf6) [0x70aa64caae16] ruby-3.2.2/signal.c:1064 ruby-3.2.2/lib/libruby.so.3.2(rb_threadptr_execute_interrupts+0x36b) [0x70aa64cf7078] ruby-3.2.2/thread.c:2334 ruby-3.2.2/lib/libruby.so.3.2(rb_threadptr_execute_interrupts) ruby-3.2.2/thread.c:2291 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_check_ints+0xb) [0x70aa64cf7ac5] ruby-3.2.2/vm_core.h:1994 ruby-3.2.2/lib/libruby.so.3.2(rb_vm_check_ints) ruby-3.2.2/vm_core.h:1985 ruby-3.2.2/lib/libruby.so.3.2(unblock_function_set) ruby-3.2.2/thread.c:320 ruby-3.2.2/lib/libruby.so.3.2(blocking_region_begin) ruby-3.2.2/thread.c:1485 ruby-3.2.2/lib/libruby.so.3.2(rb_nogvl+0xbf) [0x70aa64cf90cf] ruby-3.2.2/thread.c:1548 signal-bug-testcase-2/lib/signal_bug_testcase.so(fprintf+0x0) [0x70aa6518f299] ../../../../ext/signal_bug_testcase/signal_bug_testcase.c:17 signal-bug-testcase-2/lib/signal_bug_testcase.so(bug_testcase_free) ../../../../ext/signal_bug_testcase/signal_bug_testcase.c:18 ruby-3.2.2/lib/libruby.so.3.2(run_final+0xf) [0x70aa64b73172] ruby-3.2.2/gc.c:4388 ruby-3.2.2/lib/libruby.so.3.2(finalize_list) ruby-3.2.2/gc.c:4407 ruby-3.2.2/lib/libruby.so.3.2(finalize_deferred_heap_pages) ruby-3.2.2/gc.c:4436 ruby-3.2.2/lib/libruby.so.3.2(rb_objspace_call_finalizer+0x350) [0x70aa64b80d70] ruby-3.2.2/gc.c:4573 ruby-3.2.2/lib/libruby.so.3.2(rb_ec_finalize+0x2a) [0x70aa64b5d6d1] ruby-3.2.2/eval.c:168 ruby-3.2.2/lib/libruby.so.3.2(rb_ec_cleanup) ruby-3.2.2/eval.c:262 ruby-3.2.2/lib/libruby.so.3.2(ruby_run_node+0x9d) [0x70aa64b5d91d] ruby-3.2.2/eval.c:330 ruby-3.2.2/bin/ruby(rb_main+0x21) [0x5d5d1295f187] ./main.c:38 ruby-3.2.2/bin/ruby(main) ./main.c:57 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_call_main+0x80) [0x70aa64629d90] ../sysdeps/nptl/libc_start_call_main.h:58 /lib/x86_64-linux-gnu/libc.so.6(call_init+0x0) [0x70aa64629e40] ../csu/libc-start.c:392 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main_impl) ../csu/libc-start.c:379 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main) (null):0 [0x5d5d1295f1d5] ``` ---Files-------------------------------- signal-bug-testcase-main.zip (7.73 KB) -- https://bugs.ruby-lang.org/

2 3

[ruby-core:118346] [Ruby master Bug#20586] Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted
by ivoanjo (Ivo Anjo) 22 Aug '24

22 Aug '24

Issue #20586 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #20586: Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted https://bugs.ruby-lang.org/issues/20586 * Author: ivoanjo (Ivo Anjo) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- #### Background Hey! I work for Datadog on the Ruby profiler part of the [`datadog` (previously `ddtrace`)](https://github.com/datadog/dd-trace-rb) gem. A customer reached [out with an issue](https://github.com/DataDog/dd-trace-rb/issues/3450) where enabling the profiler made `Dir.glob` return no files for a given folder: Without profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => ["/gcsfuse/test.html", "/gcsfuse/test.txt"] ``` With profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => [] ``` It turns out the issue is related to missing error handling in `dir.c`. The Datadog Ruby profiler, like stackprof, pf2 and vernier, uses unix signals to interrupt the currently-active thread and take a sample (usually `SIGPROF`). When some system calls get interrupted by a signal, they return an [EINTR error code](https://man7.org/linux/man-pages/man7/signal.7.html#:~:text=Interrupt… back to the caller. Consider for instance the implementation of `dir_each_entry` in `dir.c`: ```c static VALUE dir_each_entry(VALUE dir, VALUE (*each)(VALUE, VALUE), VALUE arg, int children_only) { struct dir_data *dirp; struct dirent *dp; IF_NORMALIZE_UTF8PATH(int norm_p); GetDIR(dir, dirp); rewinddir(dirp->dir); IF_NORMALIZE_UTF8PATH(norm_p = need_normalization(dirp->dir, RSTRING_PTR(dirp->path))); while ((dp = READDIR(dirp->dir, dirp->enc)) != NULL) { // ... do things } return dir; } ``` If `READDIR` returns `NULL`, then `dir_each_entry` assumes it has iterated the entire directory. But looking [at the man page for `readdir`](https://man7.org/linux/man-pages/man3/readdir.3.html) we see the following sharp edge (emphasis mine): > It returns NULL on reaching the end of the directory stream **or if an error occurred**. So what's happening in this situation is: `readdir` gets interrupted, returns `NULL` + sets errno to `EINTR`. But `dir_each_entry` doesn't check `errno`, so rather than raising an exception to flag the issue, it treats it as if the end of the directory has been reached. #### How to reproduce Reproducing this is somewhat annoying, because it's dependent on timing: the signal must arrive at the exact time the dir API is getting executed. I was able to reproduce this every time by using the [google cloud `gcsfuse`](https://cloud.google.com/storage/docs/gcs-fuse) tool. This somewhat makes sense -- a remote filesystem is much slower than a local one, so there's a much bigger window of opportunity for a signal to arrive while the system call is blocked. Here's an example I included in https://github.com/DataDog/dd-trace-rb/pull/3720: ``` # Not shown: Set up a trial google cloud account, install gcsfuse, create a cloud storage bucket and put in some test files $ gcsfuse test_fs_dd_trace_rb fuse-testing/ $ ls fuse-testing/ hello.txt test.html test.txt # Not shown: Add `datadog` gem to `Gemfile` $ DD_PROFILING_ENABLED=true DD_PROFILING_DIR_INTERRUPTION_WORKAROUND_ENABLED=false bundle exec ddprofrb exec ruby -e "Datadog::Profiling.wait_until_running; pp Dir.children('fuse-testing/')" [] ``` Let me know if you'd like me to try to create a reproducer that does not depend on the `datadog` gem. #### Additional notes I've spent quite some time looking at the `dir.c` sources, and here's the full list of APIs that suffer from issues: * `dir_each_entry` does not check `errno`; all of its users have interruption bugs * `dir_tell` will return -1 instead of the correct position (which means that passing -1 to `dir_seek`/`dir_set_pos` will cause it to not list the directory properly) * `do_opendir` an error in system calls will only sometimes be turned into a raised exception * Indirect callers that pass in rb_glob_error as errfunc: rb_glob, Dir.[], Dir.glob * Indirect callers that pass in 0 as errfunc: ruby_glob, ruby_brace_glob * `glob_opendir` does not check errno; all of its users have interruption bugs * `glob_getent` does not check errno; all of its users have interruption bugs * `nogvl_dir_empty_p` does not check errno (of readdir! it actually checks for opendir); all of its users have interruption bugs Less sure about these: * `do_stat`/`do_lstat` will turn errors into warnings (unclear if enabled or disabled by default) * `need_normalization` calls `fgetattrlist` / `getattrlist` and all errors `(ret != 0)` are treated in the same way * `rb_glob_error` is and `rb_glob_caller` leave exceptions as pending and rely on callers to raise them properly * Error handling of `rb_home_dir_of` and `rb_default_home_dir` are a bit suspicious As a workaround in the Datadog Ruby profiler, in https://github.com/DataDog/dd-trace-rb/pull/3720 I've added monkey patches to all of the Ruby-level APIs that use the above functions and mask out `SIGPROF` so these calls are never interrupted. This solution is does successfully work around the issue, although it prevents the profiler from sampling during these system calls, which will mean less visibility if e.g. these calls are taking a long time. And well, maintaining monkey patches is always problematic for future Ruby compatibility. -- https://bugs.ruby-lang.org/

5 12

[ruby-core:117882] [Ruby master Bug#20490] Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process
by stanhu (Stan Hu) 18 Aug '24

18 Aug '24

Issue #20490 has been reported by stanhu (Stan Hu). ---------------------------------------- Bug #20490: Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process https://bugs.ruby-lang.org/issues/20490 * Author: stanhu (Stan Hu) * Status: Open * ruby -v: ruby 3.2.4 (2024-04-23 revision af471c0e01) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is a follow-up issue for a bug that I thought was fixed in https://bugs.ruby-lang.org/issues/19837 and duplicated in https://bugs.ruby-lang.org/issues/20181. The following script doesn't terminate quickly in Ruby 3.1.5 and 3.2.4, even with the patches to address https://bugs.ruby-lang.org/issues/19837. It works fine in Ruby 3.3. It appears that the `Process::WNOHANG` argument passed to `Process.wait2` causes this script to spin until the child process stops: ```ruby #!/bin/env ruby Process.spawn({}, "sh -c 'sleep 600'").tap do |pid| puts "detaching PID #{pid}" Process.detach(pid) end forked_pid = fork do loop { sleep 1 } end child_waiter = Thread.new do puts "Waiting for child process to die..." # The spawned process has to exit before this returns in Ruby 3.1 and 3.2 loop do pid, status = Process.wait2(-1, Process::WNOHANG) puts "Exited PID: #{pid}, status: #{status}" break if pid sleep 1 end end process_killer = Thread.new do puts "Killing #{forked_pid}" system("kill #{forked_pid}") end child_waiter.join process_killer.join ``` If I drop the `Process::WNOHANG` argument, it works fine. -- https://bugs.ruby-lang.org/

3 6