March 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112146] [Ruby master Bug#19394] cvars in instance of cloned class point to source class's cvars even after class_variable_set on clone
by jamescdavis (James Davis) 25 Jul '23

25 Jul '23

Issue #19394 has been reported by jamescdavis (James Davis). ---------------------------------------- Bug #19394: cvars in instance of cloned class point to source class's cvars even after class_variable_set on clone https://bugs.ruby-lang.org/issues/19394 * Author: jamescdavis (James Davis) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-darwin21] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This unexpected change in behavior happens between Ruby 3.0.x and 3.1.x. In Ruby >= 3.1, when a class with a cvar is cloned (or duped), the cvar in instances of the cloned class continues to point to the source class’s cvar after the clone has its cvar updated with `class_variable_set`. In Ruby < 3.1, the cloned class instance points to the updated cvar, as expected. It seems likely that this is a bug in the [cvar cache](https://bugs.ruby-lang.org/issues/17763) introduced in Ruby 3.1. Repro: ```rb class Foo @@bar = 'bar' def print_bar puts "#{self.class.name} (from instance): #{@@bar} #{@@bar.object_id}" end end foo_bar = Foo.class_variable_get(:@@bar) puts "Foo (class_variable_get): #{foo_bar} #{foo_bar.object_id}" Foo.new.print_bar FooClone = Foo.clone FooClone.class_variable_set(:@@bar, 'bar_clone') foo_clone_bar = FooClone.class_variable_get(:@@bar) puts "FooClone (class_variable_get): #{foo_clone_bar} #{foo_clone_bar.object_id}" FooClone.new.print_bar ``` Ruby 3.0.5: ``` Foo (class_variable_get): bar 60 Foo (from instance): bar 60 FooClone (class_variable_get): bar_clone 80 FooClone (from instance): bar_clone 80 ``` Ruby 3.1.3, 3.2.0: ``` Foo (class_variable_get): bar 60 Foo (from instance): bar 60 FooClone (class_variable_get): bar_clone 80 FooClone (from instance): bar 60 ``` Something similar happens when there are multiple clones and a cvar that the source class does not have defined is set on the clones. In this case, the cvars in instances of the clones all point to the first clone’s cvar. Repro: ```rb class Foo def print_bar puts "#{self.class.name} (from instance): #{@@bar} #{@@bar.object_id}" end end Foo1 = Foo.clone Foo2 = Foo.clone Foo3 = Foo.clone Foo1.class_variable_set(:@@bar, 'bar1') Foo2.class_variable_set(:@@bar, 'bar2') Foo3.class_variable_set(:@@bar, 'bar3') foo1_bar = Foo1.class_variable_get(:@@bar) foo2_bar = Foo2.class_variable_get(:@@bar) foo3_bar = Foo3.class_variable_get(:@@bar) puts "Foo1 (class_variable_get): #{foo1_bar} #{foo1_bar.object_id}" puts "Foo2 (class_variable_get): #{foo2_bar} #{foo2_bar.object_id}" puts "Foo3 (class_variable_get): #{foo3_bar} #{foo3_bar.object_id}" Foo1.new.print_bar Foo2.new.print_bar Foo3.new.print_bar ``` Ruby 3.0.5: ``` Foo1 (class_variable_get): bar1 60 Foo2 (class_variable_get): bar2 80 Foo3 (class_variable_get): bar3 100 Foo1 (from instance): bar1 60 Foo2 (from instance): bar2 80 Foo3 (from instance): bar3 100 ``` Ruby 3.1.3, 3.2.0: ``` Foo1 (class_variable_get): bar1 60 Foo2 (class_variable_get): bar2 80 Foo3 (class_variable_get): bar3 100 Foo1 (from instance): bar1 60 Foo2 (from instance): bar1 60 Foo3 (from instance): bar1 60 ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:111565] [Ruby master Bug#19293] The new Time.new(String) API is nice... but we still need a stricter version of this
by matsuda (Akira Matsuda) 22 Jul '23

22 Jul '23

Issue #19293 has been reported by matsuda (Akira Matsuda). ---------------------------------------- Bug #19293: The new Time.new(String) API is nice... but we still need a stricter version of this https://bugs.ruby-lang.org/issues/19293 * Author: matsuda (Akira Matsuda) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-01-01T07:39:00Z master 542e984d82) +YJIT [arm64-darwin21] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The Ruby 3.2 style `Time.new(String)` API works very well so far, but since the original `Time.new(Integer, Integer, Integer...)` API actually accepts String objects as its arguments, there's one ambiguous case as follows: `Time.new('20230123') #=> 20230123-01-01 00:00:00 +0900` Then the problem that I'm facing is that we cannot tell if `Time.new` would parse the given String as ISO8601-ish or just a year, and in order to avoid this ambiguity, we still need to somehow parse the String beforehand in our application side (like we're doing this way in Ruby on Rails https://github.com/rails/rails/blob/c49b8270/activemodel/lib/active_model/t…) then dispatch to the new `Time.new` only when the String is validated to be conforming the ISO format. Otherwise, if we just optimistically pass in given Strings to `Time.new`, we'll occasionally get a Time object with an unintended buggy value. Therefore, it unfortunately seems that my feature request on #16005 still continues... I have to keep proposing that we need either of the following: 1. A trustworthy version of ISO8601 parser method perhaps with another name than `.new` that accepts strict ISO8601-ish String only (but with the T delimiter, I still don't know what the proper name of this format is). 2. Change `Time.new(Integer-ish, Integer-ish, Integer-ish...)` not to accept Integer-ish Strings but to accept only Integers. But I can imagine that this direction is very unlikely acceptable, due to the incompatibility. -- https://bugs.ruby-lang.org/

7 7

[ruby-core:112918] [Ruby master Bug#19532] Handling of 6-byte codepoints in left_adjust_char_head in CESU-8 encoding is broken
by Eregon (Benoit Daloze) 22 Jul '23

22 Jul '23

Issue #19532 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Bug #19532: Handling of 6-byte codepoints in left_adjust_char_head in CESU-8 encoding is broken https://bugs.ruby-lang.org/issues/19532 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- Fix in https://github.com/ruby/ruby/pull/7510 -- https://bugs.ruby-lang.org/

4 3

[ruby-core:112926] [Ruby master Misc#19535] Instance variables order is unpredictable on objects with `OBJ_TOO_COMPLEX_SHAPE_ID`
by byroot (Jean Boussier) 21 Jul '23

21 Jul '23

Issue #19535 has been reported by byroot (Jean Boussier). ---------------------------------------- Misc #19535: Instance variables order is unpredictable on objects with `OBJ_TOO_COMPLEX_SHAPE_ID` https://bugs.ruby-lang.org/issues/19535 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context I've been helping the Mastodon folks in investigating a weird Marshal deserialization bug they randomly experience since they upgraded to Ruby 3.2: https://github.com/mastodon/mastodon/issues/23644 Ultimately the bug comes from a circular dependency issues in the object graph that is serialized when one call `Marshal.dump` on an `ActiveRecord::Base` object. A simplified reproduction to better explain the problem is: ```ruby class Status def normal_order @attributes = { id: 42 } @relations = { self => 1 } self end def inverse_order @relations = nil @attributes = { id: 42 } @relations = { self => 1 } self end def hash @attributes.fetch(:id) end end s = Marshal.load(Marshal.dump(Status.new.normal_order)) s = Marshal.load(Marshal.dump(Status.new.inverse_order)) ``` In short, that `Status` object is both the top level object, and is referenced as a key in a hash, in that same payload. It also defined a custom `#hash` method, that requires some other attribute to be set. It all "works" as long as `@attributes` is dumped before `@relations`. ### Problem The above micro-reproduction uses two different shapes to demonstrate the ordering issues, but in both case the ordering is predictable. However if you generate too many shapes from a single class, it will be marked as `TOO_COMPLEX` and future instance will have their instance variables backed by an `id_table`, which is unordered, and will cause a similar issue. I definitely consider this a bug on the Rails side, and I will do what I can so that Rails doesn't depend on that implicit ordering. However it's unlikely we'll be able to fix older version, and other users may run into this issue when upgrading to Ruby 3.2, so I think it may be worth to try to preserve some sort of predicable ordering, at least for a few more versions. Additionally, debugging it was made particularly difficult, because it would work fine initially, and then break after enough shapes had been generated. Generally speaking I think such semi-predictable behavior is much worse than a fully random behavior (similar to how Go randomize keys order in their maps). ### Historical behavior On Ruby 3.1 and older, the instance variables ordering was defined by the order in which each ivar appeared for the very first time: ```ruby class Foo def set @a = 1 @b = 2 @c = 3 self end def inverse_order @c = 3 @b = 2 @a = 1 self end end p Foo.new.set.instance_variables # => [:@a, :@b, :@c] p Foo.new.inverse_order.instance_variables # => [:@a, :@b, :@c] ``` This means that the order could be different from once execution of the program to another, but would remain stable inside a single process. On 3.2, it's now defined by the order in which each ivar appeared in that specific object instance: ```ruby [:@a, :@b, :@c] [:@c, :@b, :@a] ``` Except, if the object is backed by an `id_table`, in which case it's fully unpredictable. ### Possible changes I discussed this with @tenderlovemaking, and he suggested we could change the `id_table` for an `st_table` so that the ordering could be predictable again, and would behave like objects with a non-complex shape. Another possibility would be to preserve the observable behavior of 3.1 and older. Or of course we could clearly specify that the ordering is random, but if so I think it would be wise to make it always random so that this class of bugs has a much higher chance to be caught early in testing rather than in production. cc @Eregon as I presume this has implications on TruffleRuby as well. -- https://bugs.ruby-lang.org/

5 7

[ruby-core:112906] [Ruby master Bug#19531] ObjectSpace::WeakMap: replaced values still clear the key they were assigned to
by byroot (Jean Boussier) 21 Jul '23

21 Jul '23

Issue #19531 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19531: ObjectSpace::WeakMap: replaced values still clear the key they were assigned to https://bugs.ruby-lang.org/issues/19531 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: WONTFIX, 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- ### Reproduction script ```ruby wmap = ObjectSpace::WeakMap.new a = "A" b = "B" wmap[1] = a wmap[1] = b # the table entry with 1 is still in the list of entries to clear when `a` is GCed a = nil GC.start p wmap[1] # Should be `"B"`, but is `nil` ``` ### Explanation What happens is that when we set `wmap[1] = "A"`, WeakMap internally keeps a list of keys to clear when `"A"` is GCed, e.g. pseudo code: ```ruby class WeakMap def []=(key, value) @hash[key] = value @reverse[value] << key end end ``` But it doesn't clear previously kept mapping when a key is overwritten. I'll work on a fix. ### References https://github.com/protocolbuffers/protobuf/pull/12216 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:111873] [Ruby master Bug#19351] Promote bundled gems at Ruby 3.3
by hsbt (Hiroshi SHIBATA) 19 Jul '23

19 Jul '23

Issue #19351 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19351: Promote bundled gems at Ruby 3.3 https://bugs.ruby-lang.org/issues/19351 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- In Ruby 3.2, the default gems and bundled gems are changed only adding `syntax_suggest`. I and some committers are considering promote default gems to bundled gems again for Ruby 3.3+. We hope to keep the current developer experience with dependency resolution and ignore the additional work like "Put gem "xxx" into your Gemfile" for developers. ### Proposal We propose the following libraries will promote default gems to bundled gems at Ruby 3.3. They are not the dependencies of Rails and RubyGems/Bundler. ``` abbrev getoptlong optparse observable resolv resolv-replace rinda un fcntl nkf syslog win32ole ``` ### Additional works I also propose to poromote rails dependencies: ``` ostruct base64 irb rdoc tsort singleton delegate ``` and gems maintained by @kou ``` csv strscan fiddle stringio ``` But if we promote them to bundled gems, many of users need to add `gem "csv"` into their Gemfile. I'm considering to avoid this situation. Can we the specific feature of bundled gems to RubyGems or Bundler? Example, bundler have allowed list for bundled gems. So, listed gems could be require without Gemfile under the bundle exec. -- https://bugs.ruby-lang.org/

6 25

[ruby-core:112533] [Ruby master Bug#19455] Ruby 3.2: wrong Regexp encoding with non-ASCII comments
by janosch-x 17 Jul '23

17 Jul '23

Issue #19455 has been reported by janosch-x (Janosch Müller). ---------------------------------------- Bug #19455: Ruby 3.2: wrong Regexp encoding with non-ASCII comments https://bugs.ruby-lang.org/issues/19455 * Author: janosch-x (Janosch Müller) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- comments and comment groups don't trigger the correct `Regexp#encoding` on Ruby 3.2 anymore: ```ruby # ruby 3.1 /#a/x.encoding # => #<Encoding:US-ASCII> # OK /(?#a)/.encoding # => #<Encoding:US-ASCII> # OK /#ü/x.encoding # => #<Encoding:UTF-8> # OK /(?#ü)/.encoding # => #<Encoding:UTF-8> # OK # ruby 3.2 /#a/x.encoding # => #<Encoding:US-ASCII> # OK /(?#a)/.encoding # => #<Encoding:US-ASCII> # OK /#ü/x.encoding # => #<Encoding:US-ASCII> # BUG /(?#ü)/.encoding # => #<Encoding:US-ASCII> # BUG /#ü/x.inspect # => "/#\\xC3\\xBC/x" /(?#ü)/.inspect # => "/(?#\\xC3\\xBC)/" # bug is hidden if there are non-ascii chars outside comments /ü#ü/x.encoding # => #<Encoding:UTF-8> /ü(?#ü)/.encoding # => #<Encoding:UTF-8> ``` i think these changes might be the cause: https://github.com/ruby/ruby/commit/ec3542229b29ec93062e9d90e877ea29d3c1947… @jeremyevans0 JFYI -- https://bugs.ruby-lang.org/

6 6

[ruby-core:112922] [Ruby master Bug#19533] Behavior of ===/include? on a beginless/endless range (nil..nil) changed in ruby 3.2
by knu (Akinori MUSHA) 17 Jul '23

17 Jul '23

Issue #19533 has been reported by knu (Akinori MUSHA). ---------------------------------------- Bug #19533: Behavior of ===/include? on a beginless/endless range (nil..nil) changed in ruby 3.2 https://bugs.ruby-lang.org/issues/19533 * Author: knu (Akinori MUSHA) * Status: Open * Priority: Normal * Target version: 3.2 * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-darwin22] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Starting from Ruby 2.7.0 a range `nil..nil` used to match any value until 3.2.0-preview3, but 3.2.0-rc1 started to reject it. ``` % docker run -it --rm rubylang/all-ruby ./all-ruby -e 'p( (nil..nil) === 1 )' (snip) ruby-2.6.10 false ruby-2.7.0-preview1 true ... ruby-3.2.0-preview3 true ruby-3.2.0-rc1 -e:1:in `===': cannot determine inclusion in beginless/endless ranges (TypeError) p( (nil..nil) === 1 ) ^ from -e:1:in `<main>' exit 1 ... ruby-3.2.1 -e:1:in `===': cannot determine inclusion in beginless/endless ranges (TypeError) p( (nil..nil) === 1 ) ^ from -e:1:in `<main>' ``` The previous behavior was so useful because when you have optional lower and upper bounds `lbound..rbound` would always work regardless of each end being nil or not. There is no mention of this in doc/NEWS/NEWS-3.2.0.md, so I'm afraid it was caused unintentionally while fixing other problems. ``` % docker run -it --rm rubylang/all-ruby ./all-ruby -e 'p( (nil..nil).cover?(1) )' (snip) ruby-2.6.10 false ruby-2.7.0-preview1 true ... ruby-3.2.1 true ``` This was pointed out by the following blog article (written in Japanese) as a "pitfall" that you need to work around when upgrading Ruby from 3.0/3.1 to 3.2. https://blog.studysapuri.jp/entry/2023/03/16/ujihisa-ruby32#endless-range%E… -- https://bugs.ruby-lang.org/

5 5

[ruby-core:113026] [Ruby master Bug#19550] Memory leak in iclass for 32 bit systems
by peterzhu2118 (Peter Zhu) 16 Jul '23

16 Jul '23

Issue #19550 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19550: Memory leak in iclass for 32 bit systems https://bugs.ruby-lang.org/issues/19550 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- If !RCLASS_EXT_EMBEDDED (e.g. 32 bit systems) then the rb_classext_t is allocated throug malloc so it must be freed. The issue can be seen in the following script: ```ruby 20.times do 100_000.times do mod = Module.new Class.new do include mod end end # Output the Resident Set Size (memory usage, in KB) of the current Ruby process puts `ps -o rss= -p #{$$}` end ``` Before this fix, the max RSS is 280MB, while after this change, it's 30MB. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:113005] [Ruby master Bug#19546] IO::Buffer is incorrectly invoking fiber scheduler interface.
by ioquatix (Samuel Williams) 16 Jul '23

16 Jul '23

Issue #19546 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19546: IO::Buffer is incorrectly invoking fiber scheduler interface. https://bugs.ruby-lang.org/issues/19546 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The `IO::Buffer` implementation incorrectly calls rb_fiber_scheduler_io_write with the size_ts converted to VALUEs: https://github.com/ruby/ruby/blob/400ccb16eefe4e21c4e3eacab4fd0f208fc5e151/… But that function expects size_ts: https://github.com/ruby/ruby/blob/400ccb16eefe4e21c4e3eacab4fd0f208fc5e151/… (And probably the same for the other scheduler methods.) Fixed by <https://github.com/ruby/ruby/pull/7593>. -- https://bugs.ruby-lang.org/

2 2