March 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117169] [Ruby master Bug#20338] calling new has regressed allocations over past few ruby releases
by fablestales (Fable Tales) 21 Mar '24

21 Mar '24

Issue #20338 has been reported by fablestales (Fable Tales). ---------------------------------------- Bug #20338: calling new has regressed allocations over past few ruby releases https://bugs.ruby-lang.org/issues/20338 * Author: fablestales (Fable Tales) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Given this script: ``` class Foo def initialize(**kwargs) end end class Bar def initialize(y:) end end b = GC.stat[:total_allocated_objects] 1000.times { Foo.new(y:1) } p GC.stat[:total_allocated_objects]-b b = GC.stat[:total_allocated_objects] 1000.times { Bar.new(y:1) } p GC.stat[:total_allocated_objects]-b ``` On Ruby 3.2 it prints ``` (fable-3_3_dead_with_patches) $ ruby --version ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23] (fable-3_3_dead_with_patches) $ ruby hi.rb 3006 2005 ``` On Ruby 3.3 it prints: ``` ((HEAD detached at v3_3_0)) $ ./ruby --version ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23] ((HEAD detached at v3_3_0)) $ ./ruby hi.rb 3008 3005 ``` On master: ``` (master) $ git rev-parse HEAD 83618f2cfa004accdd1514de7dcbba291aa7e831 (master) $ ./ruby --version ruby 3.4.0dev (2024-03-14T12:33:30Z master 83618f2cfa) [arm64-darwin23] (master) $ ./ruby hi.rb 3006 4005 ``` so that is: since Ruby 3.2 `.new`ing an object with static keyword args has grown from 2 allocations per object to 4 allocations per object. I have a pull request which introduces a new instruction to the VM `opt_new` which basically inlines calling allocate and initialize directly, avoiding a lot of what is causing these extra objects to be allocated. My PR is here: https://github.com/ruby/ruby/pull/10254. With my patch the script prints 2006 and 1005 respectively. This is the smallest number of objects that _can_ be allocated for such calls. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:117281] [Ruby master Bug#20386] Backport CVE-2024-27281
by hsbt (Hiroshi SHIBATA) 21 Mar '24

21 Mar '24

Issue #20386 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #20386: Backport CVE-2024-27281 https://bugs.ruby-lang.org/issues/20386 * Author: hsbt (Hiroshi SHIBATA) * Status: Closed * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- I disclosed https://www.ruby-lang.org/en/news/2024/03/21/rce-rdoc-cve-2024-27281/ today. We should backport fixed RDoc to all stable version. * For 3.0: https://github.com/ruby/ruby/pull/10319 * For 3.1: https://github.com/ruby/ruby/pull/10318 * For 3.2: https://github.com/ruby/ruby/pull/10317 * For 3.3: https://github.com/ruby/ruby/pull/10316 -- https://bugs.ruby-lang.org/

2 1

[ruby-core:117280] [Ruby master Bug#20385] Backport CVE-2024-27280
by hsbt (Hiroshi SHIBATA) 21 Mar '24

21 Mar '24

Issue #20385 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #20385: Backport CVE-2024-27280 https://bugs.ruby-lang.org/issues/20385 * Author: hsbt (Hiroshi SHIBATA) * Status: Closed * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONTNEED, 3.3: DONTNEED ---------------------------------------- I disclosed https://www.ruby-lang.org/en/news/2024/03/21/buffer-overread-cve-2024-27280/ today. This StringIO versions should be backported in the next release. * For Ruby 3.0: https://github.com/ruby/ruby/pull/10320 * For Ruby 3.1: https://github.com/ruby/ruby/pull/10321 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116627] [Ruby master Bug#20246] Unexpected behavior for Regexp in Subexpression Calls on Ruby 3.3.0
by bestwebua (Vladislav Trotsenko) 21 Mar '24

21 Mar '24

Issue #20246 has been reported by bestwebua (Vladislav Trotsenko). ---------------------------------------- Bug #20246: Unexpected behavior for Regexp in Subexpression Calls on Ruby 3.3.0 https://bugs.ruby-lang.org/issues/20246 * Author: bestwebua (Vladislav Trotsenko) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Hello! A few days ago, after migration on 3.3.0 I have faced with issue inside subexpression calls. An expected result on Ruby up to 3.3.0: ```ruby '1.2.3'[/(\d+)(\.\g<1>){2}/] # => "1.2.3" ``` An actual result on Ruby 3.3.0: ```ruby '1.2.3'[/(\d+)(\.\g<1>){2}/] # => nil ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:116201] [Ruby master Bug#20184] Ruby segfaults on Fly.io with 256 MB RAM
by aalin (Andreas Alin) 20 Mar '24

20 Mar '24

Issue #20184 has been reported by aalin (Andreas Alin). ---------------------------------------- Bug #20184: Ruby segfaults on Fly.io with 256 MB RAM https://bugs.ruby-lang.org/issues/20184 * Author: aalin (Andreas Alin) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I've been running a Ruby 3.2 app on Fly.io with 256 MB RAM and it has been working fine. When I upgraded to Ruby 3.3.0, any Ruby script would crash immediately with a segfault. I haven't been able to reproduce it outside of fly.io. I created a repository with a basic rack app which crashes on fly.io. The readme includes the full output and a GDB backtrace. https://github.com/aalin/ruby-3-3-0-fly-crash -- https://bugs.ruby-lang.org/

5 6

[ruby-core:115924] [Ruby master Bug#20095] Regex lookahead behaving strangely in 3.3.0
by sisyphus_cg (Sisyphus CG) 20 Mar '24

20 Mar '24

Issue #20095 has been reported by sisyphus_cg (Sisyphus CG). ---------------------------------------- Bug #20095: Regex lookahead behaving strangely in 3.3.0 https://bugs.ruby-lang.org/issues/20095 * Author: sisyphus_cg (Sisyphus CG) * Status: Open * Priority: Normal * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Multiple regex lookaheads seem to behave strangely in 3.3.0: ```ruby p "xxx" =~ /(?=.*x)x(?=banana)/ # 2 ``` Clearly, the string `xxx` does not contain `banana`, so this regex should never match, but it does. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:116450] [Ruby master Bug#20212] Regex match method is crashing - (irb):6: [BUG] Segmentation fault at 0x0000000000000000 ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
by rubanthilak (Ruban Thilak) 20 Mar '24

20 Mar '24

Issue #20212 has been reported by rubanthilak (Ruban Thilak). ---------------------------------------- Bug #20212: Regex match method is crashing - (irb):6: [BUG] Segmentation fault at 0x0000000000000000 ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] https://bugs.ruby-lang.org/issues/20212 * Author: rubanthilak (Ruban Thilak) * Status: Open * Priority: Normal * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Tried this with ruby 3.3.0, both irb and rails console are crashing ```ruby DOMAIN_REGEX = Regexp.new( /\A((?=.*?[a-z])(?!.*--)[a-z\d]+[a-z\d-]*[a-z\d]+).((?=.*?[a-z])(?!.*--)[a-z\d]+[a-z\d-]*[a-z\d]+).((?=.*?[a-z])(?!.*--)[a-zd]+[a-zd-]*[a-zd]+).((?=.*?[a-z])(?!.*--)[a-zd]+[a-zd-]*[a-zd]+)\Z/x ) full_domain = "www.google.com" 100.times.each { full_domain.match(DOMAIN_REGEX) } ``` -- Ruby level backtrace information ---------------------------------------- /Users/rthilak/.rvm/gems/ruby-3.3.0@ams/bin/ruby_executable_hooks:22:in `<main>' /Users/rthilak/.rvm/gems/ruby-3.3.0@ams/bin/ruby_executable_hooks:22:in `eval' /Users/rthilak/.rvm/rubies/ruby-3.3.0/bin/bundle:25:in `<main>' /Users/rthilak/.rvm/rubies/ruby-3.3.0/bin/bundle:25:in `load' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/exe/bundle:20:in `<top (required)>' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/friendly_errors.rb:117:in `with_friendly_errors' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/exe/bundle:28:in `block in <top (required)>' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli.rb:28:in `start' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/base.rb:584:in `start' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli.rb:34:in `dispatch' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor.rb:527:in `dispatch' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/vendor/thor/lib/thor/command.rb:28:in `run' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli.rb:451:in `exec' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:23:in `run' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:58:in `kernel_load' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/bundler/cli/exec.rb:58:in `load' /Users/rthilak/.rvm/rubies/ruby-3.3.0/bin/irb:25:in `<top (required)>' /Users/rthilak/.rvm/rubies/ruby-3.3.0/bin/irb:25:in `load' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/gems/3.3.0/gems/irb-1.11.0/exe/irb:9:in `<top (required)>' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/irb.rb:903:in `start' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/irb.rb:1001:in `run' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/irb.rb:1001:in `catch' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/irb.rb:1002:in `block in run' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/irb.rb:1015:in `eval_input' /Users/rthilak/.rvm/rubies/ruby-3.3.0/lib/ruby/3.3.0/irb.rb:1093:in `each_top_level_statement' <internal:kernel>:187:in `loop' ---Files-------------------------------- trace.txt (61.7 KB) -- https://bugs.ruby-lang.org/

4 3

[ruby-core:116395] [Ruby master Bug#20207] Segmentation fault for a regexp containing positive and negative lookaheads
by Sundeep (Sundeep Agarwal) 20 Mar '24

20 Mar '24

Issue #20207 has been reported by Sundeep (Sundeep Agarwal). ---------------------------------------- Bug #20207: Segmentation fault for a regexp containing positive and negative lookaheads https://bugs.ruby-lang.org/issues/20207 * Author: Sundeep (Sundeep Agarwal) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I'm getting segmentation fault for the following regexp with Ruby 3.3.0: ``` ruby puts 'clan'.match?(/(?=.*a)(?!.*n)/) ``` This is the minimal example for which I was able to produce the issue. `a` and `n` in the above example are just a sample, can be other character combinations too. There has to be at least two characters before them in the input string (`cl` in the above example). The error is seen when the above code is run from a file, but only occasionally from irb. When run from a file, most of the time I get segmentation fault (see attached log file), but sometimes I get the following error: ``` ruby lookaround.rb:1:in `match?': undefined bytecode (bug): /(?=.*a)(?!.*n)/ (RegexpError) from lookaround.rb:1:in `<main>' ``` When asked in the r/ruby forum, somebody else was able to reproduce this issue. They also mentioned that the issue wasn't seen in the Ruby 3.2.2 version. ---Files-------------------------------- lookaround_segfault.log (15.9 KB) -- https://bugs.ruby-lang.org/

6 6

[ruby-core:117267] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by Eregon (Benoit Daloze) 20 Mar '24

20 Mar '24

Issue #19057 has been updated by Eregon (Benoit Daloze). Re beauty of the implementation this change is more than that, we can see in https://github.com/ruby/ruby/pull/10283/files that `rb_io_t` exposes way too much internals, which in turns prevents changing/evolving these internals without affecting extensions, can easily break ABI unintentionally, can have negative performance impact, etc. It's also rather brittle with the internal duplicate definition of rb_io_t, and could easily cause segfaults if the structs don't match (e.g. when working on this code and then wasting a bunch of time due to that, or maybe subtle enough that the CI would not catch and it would only be found later). I hope we can still land this change, if it needs to wait 3.5 or so then that sounds a good compromise. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-107392 * Author: ioquatix (Samuel Williams) * Status: Assigned * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117262] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by mame (Yusuke Endoh) 20 Mar '24

20 Mar '24

Issue #19057 has been updated by mame (Yusuke Endoh). ioquatix (Samuel Williams) wrote in #note-43: > Here is the revert PR: https://github.com/ruby/ruby/pull/10283 Thanks! Just FYI, I think the precedence in Ruby's decision is basically: * beauty of the language (conciseness and intuitiveness for users) >= compatibility > runtime performance and efficiency >>> beauty of the implementation (simplicity for the core developers). Compared to the past, compatibility has become much more of a priority, and performance has become somewhat more important. The order is sometimes reversed in rare cases, but the beauty of the implementation has always been the lowest priority. The "beauty of the implementation", such as hiding the rb_io_t implementation, can beat "compatibility" only in limited and special cases, e.g., only when there is no other way at all, or when the practical impact is very small (no affected gem is identified, or it is only an issue with minor gems). In this case, unicorn, which is undeniably an important gem and still part of the Ruby ecosystem, is affected and there are known implementation workarounds. In that case, there is no reason not to choose the workaround. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-107355 * Author: ioquatix (Samuel Williams) * Status: Assigned * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0