February 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112399] [Ruby master Bug#19436] Call Cache for singleton methods can lead to "memory leaks"
by byroot (Jean Boussier) 23 Aug '23

23 Aug '23

Issue #19436 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#met…. This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

8 24

[ruby-core:111740] [Ruby master Bug#19325] Windows support lacking.
by dsisnero (Dominic Sisneros) 03 Aug '23

03 Aug '23

Issue #19325 has been reported by dsisnero (Dominic Sisneros). ---------------------------------------- Bug #19325: Windows support lacking. https://bugs.ruby-lang.org/issues/19325 * Author: dsisnero (Dominic Sisneros) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Ruby's support on windows has always been second class. With some of the recent decisions, windows support is falling even more behind. Recent developments in mjit and yjit that exclude windows are two glaring issues that should be corrected. Googling 'percent of windows vs other operating systems' and it shows windows has a share of 76%. Ceding that users to python and other programming languages has to be one of the reasons python continues get more market share from ruby. With rust having first class windows support and threading support, is there a reason why yjit is not able to work on windows? Also, windows compiler support has matured enough and vcpkg support has evolved enough that it seems it should be possible to finally get a ruby version without having to use msys2. Even Crystal language has a version that runs on windows without needing msys2. -- https://bugs.ruby-lang.org/

9 10

[ruby-core:111698] [Ruby master Bug#19318] Float#round rounds incorrectly for some cases
by Eregon (Benoit Daloze) 25 Jul '23

25 Jul '23

Issue #19318 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Bug #19318: Float#round rounds incorrectly for some cases https://bugs.ruby-lang.org/issues/19318 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- This was discovered by @aardvark179. The following spec in `spec/ruby/core/float/round_spec.rb` fails on CRuby: ```ruby ruby_bug "", ""..."3.3" do # These numbers are neighbouring floating point numbers round a # precise value. They test that the rounding modes work correctly # round that value and precision is not lost which might cause # incorrect results. it "does not lose precision during the rounding process" do 767573.1875850001.round(5, half: nil).should eql(767573.18759) 767573.1875850001.round(5, half: :up).should eql(767573.18759) 767573.1875850001.round(5, half: :down).should eql(767573.18759) 767573.1875850001.round(5, half: :even).should eql(767573.18759) -767573.1875850001.round(5, half: nil).should eql(-767573.18759) -767573.1875850001.round(5, half: :up).should eql(-767573.18759) -767573.1875850001.round(5, half: :down).should eql(-767573.18759) -767573.1875850001.round(5, half: :even).should eql(-767573.18759) 767573.187585.round(5, half: nil).should eql(767573.18759) 767573.187585.round(5, half: :up).should eql(767573.18759) 767573.187585.round(5, half: :down).should eql(767573.18758) 767573.187585.round(5, half: :even).should eql(767573.18758) -767573.187585.round(5, half: nil).should eql(-767573.18759) -767573.187585.round(5, half: :up).should eql(-767573.18759) -767573.187585.round(5, half: :down).should eql(-767573.18758) -767573.187585.round(5, half: :even).should eql(-767573.18758) 767573.1875849998.round(5, half: nil).should eql(767573.18758) 767573.1875849998.round(5, half: :up).should eql(767573.18758) 767573.1875849998.round(5, half: :down).should eql(767573.18758) 767573.1875849998.round(5, half: :even).should eql(767573.18758) -767573.1875849998.round(5, half: nil).should eql(-767573.18758) -767573.1875849998.round(5, half: :up).should eql(-767573.18758) -767573.1875849998.round(5, half: :down).should eql(-767573.18758) -767573.1875849998.round(5, half: :even).should eql(-767573.18758) end end ``` Yet this test to the best of our knowledge is correct. This was fixed on master by @mrkn in https://github.com/ruby/ruby/pull/7023 (thanks!). The question is should we backport this? I think yes. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:112146] [Ruby master Bug#19394] cvars in instance of cloned class point to source class's cvars even after class_variable_set on clone
by jamescdavis (James Davis) 25 Jul '23

25 Jul '23

Issue #19394 has been reported by jamescdavis (James Davis). ---------------------------------------- Bug #19394: cvars in instance of cloned class point to source class's cvars even after class_variable_set on clone https://bugs.ruby-lang.org/issues/19394 * Author: jamescdavis (James Davis) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-darwin21] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This unexpected change in behavior happens between Ruby 3.0.x and 3.1.x. In Ruby >= 3.1, when a class with a cvar is cloned (or duped), the cvar in instances of the cloned class continues to point to the source class’s cvar after the clone has its cvar updated with `class_variable_set`. In Ruby < 3.1, the cloned class instance points to the updated cvar, as expected. It seems likely that this is a bug in the [cvar cache](https://bugs.ruby-lang.org/issues/17763) introduced in Ruby 3.1. Repro: ```rb class Foo @@bar = 'bar' def print_bar puts "#{self.class.name} (from instance): #{@@bar} #{@@bar.object_id}" end end foo_bar = Foo.class_variable_get(:@@bar) puts "Foo (class_variable_get): #{foo_bar} #{foo_bar.object_id}" Foo.new.print_bar FooClone = Foo.clone FooClone.class_variable_set(:@@bar, 'bar_clone') foo_clone_bar = FooClone.class_variable_get(:@@bar) puts "FooClone (class_variable_get): #{foo_clone_bar} #{foo_clone_bar.object_id}" FooClone.new.print_bar ``` Ruby 3.0.5: ``` Foo (class_variable_get): bar 60 Foo (from instance): bar 60 FooClone (class_variable_get): bar_clone 80 FooClone (from instance): bar_clone 80 ``` Ruby 3.1.3, 3.2.0: ``` Foo (class_variable_get): bar 60 Foo (from instance): bar 60 FooClone (class_variable_get): bar_clone 80 FooClone (from instance): bar 60 ``` Something similar happens when there are multiple clones and a cvar that the source class does not have defined is set on the clones. In this case, the cvars in instances of the clones all point to the first clone’s cvar. Repro: ```rb class Foo def print_bar puts "#{self.class.name} (from instance): #{@@bar} #{@@bar.object_id}" end end Foo1 = Foo.clone Foo2 = Foo.clone Foo3 = Foo.clone Foo1.class_variable_set(:@@bar, 'bar1') Foo2.class_variable_set(:@@bar, 'bar2') Foo3.class_variable_set(:@@bar, 'bar3') foo1_bar = Foo1.class_variable_get(:@@bar) foo2_bar = Foo2.class_variable_get(:@@bar) foo3_bar = Foo3.class_variable_get(:@@bar) puts "Foo1 (class_variable_get): #{foo1_bar} #{foo1_bar.object_id}" puts "Foo2 (class_variable_get): #{foo2_bar} #{foo2_bar.object_id}" puts "Foo3 (class_variable_get): #{foo3_bar} #{foo3_bar.object_id}" Foo1.new.print_bar Foo2.new.print_bar Foo3.new.print_bar ``` Ruby 3.0.5: ``` Foo1 (class_variable_get): bar1 60 Foo2 (class_variable_get): bar2 80 Foo3 (class_variable_get): bar3 100 Foo1 (from instance): bar1 60 Foo2 (from instance): bar2 80 Foo3 (from instance): bar3 100 ``` Ruby 3.1.3, 3.2.0: ``` Foo1 (class_variable_get): bar1 60 Foo2 (class_variable_get): bar2 80 Foo3 (class_variable_get): bar3 100 Foo1 (from instance): bar1 60 Foo2 (from instance): bar1 60 Foo3 (from instance): bar1 60 ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:111565] [Ruby master Bug#19293] The new Time.new(String) API is nice... but we still need a stricter version of this
by matsuda (Akira Matsuda) 23 Jul '23

23 Jul '23

Issue #19293 has been reported by matsuda (Akira Matsuda). ---------------------------------------- Bug #19293: The new Time.new(String) API is nice... but we still need a stricter version of this https://bugs.ruby-lang.org/issues/19293 * Author: matsuda (Akira Matsuda) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-01-01T07:39:00Z master 542e984d82) +YJIT [arm64-darwin21] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The Ruby 3.2 style `Time.new(String)` API works very well so far, but since the original `Time.new(Integer, Integer, Integer...)` API actually accepts String objects as its arguments, there's one ambiguous case as follows: `Time.new('20230123') #=> 20230123-01-01 00:00:00 +0900` Then the problem that I'm facing is that we cannot tell if `Time.new` would parse the given String as ISO8601-ish or just a year, and in order to avoid this ambiguity, we still need to somehow parse the String beforehand in our application side (like we're doing this way in Ruby on Rails https://github.com/rails/rails/blob/c49b8270/activemodel/lib/active_model/t…) then dispatch to the new `Time.new` only when the String is validated to be conforming the ISO format. Otherwise, if we just optimistically pass in given Strings to `Time.new`, we'll occasionally get a Time object with an unintended buggy value. Therefore, it unfortunately seems that my feature request on #16005 still continues... I have to keep proposing that we need either of the following: 1. A trustworthy version of ISO8601 parser method perhaps with another name than `.new` that accepts strict ISO8601-ish String only (but with the T delimiter, I still don't know what the proper name of this format is). 2. Change `Time.new(Integer-ish, Integer-ish, Integer-ish...)` not to accept Integer-ish Strings but to accept only Integers. But I can imagine that this direction is very unlikely acceptable, due to the incompatibility. -- https://bugs.ruby-lang.org/

7 7

[ruby-core:111873] [Ruby master Bug#19351] Promote bundled gems at Ruby 3.3
by hsbt (Hiroshi SHIBATA) 19 Jul '23

19 Jul '23

Issue #19351 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19351: Promote bundled gems at Ruby 3.3 https://bugs.ruby-lang.org/issues/19351 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- In Ruby 3.2, the default gems and bundled gems are changed only adding `syntax_suggest`. I and some committers are considering promote default gems to bundled gems again for Ruby 3.3+. We hope to keep the current developer experience with dependency resolution and ignore the additional work like "Put gem "xxx" into your Gemfile" for developers. ### Proposal We propose the following libraries will promote default gems to bundled gems at Ruby 3.3. They are not the dependencies of Rails and RubyGems/Bundler. ``` abbrev getoptlong optparse observable resolv resolv-replace rinda un fcntl nkf syslog win32ole ``` ### Additional works I also propose to poromote rails dependencies: ``` ostruct base64 irb rdoc tsort singleton delegate ``` and gems maintained by @kou ``` csv strscan fiddle stringio ``` But if we promote them to bundled gems, many of users need to add `gem "csv"` into their Gemfile. I'm considering to avoid this situation. Can we the specific feature of bundled gems to RubyGems or Bundler? Example, bundler have allowed list for bundled gems. So, listed gems could be require without Gemfile under the bundle exec. -- https://bugs.ruby-lang.org/

6 25

[ruby-core:112533] [Ruby master Bug#19455] Ruby 3.2: wrong Regexp encoding with non-ASCII comments
by janosch-x 17 Jul '23

17 Jul '23

Issue #19455 has been reported by janosch-x (Janosch Müller). ---------------------------------------- Bug #19455: Ruby 3.2: wrong Regexp encoding with non-ASCII comments https://bugs.ruby-lang.org/issues/19455 * Author: janosch-x (Janosch Müller) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- comments and comment groups don't trigger the correct `Regexp#encoding` on Ruby 3.2 anymore: ```ruby # ruby 3.1 /#a/x.encoding # => #<Encoding:US-ASCII> # OK /(?#a)/.encoding # => #<Encoding:US-ASCII> # OK /#ü/x.encoding # => #<Encoding:UTF-8> # OK /(?#ü)/.encoding # => #<Encoding:UTF-8> # OK # ruby 3.2 /#a/x.encoding # => #<Encoding:US-ASCII> # OK /(?#a)/.encoding # => #<Encoding:US-ASCII> # OK /#ü/x.encoding # => #<Encoding:US-ASCII> # BUG /(?#ü)/.encoding # => #<Encoding:US-ASCII> # BUG /#ü/x.inspect # => "/#\\xC3\\xBC/x" /(?#ü)/.inspect # => "/(?#\\xC3\\xBC)/" # bug is hidden if there are non-ascii chars outside comments /ü#ü/x.encoding # => #<Encoding:UTF-8> /ü(?#ü)/.encoding # => #<Encoding:UTF-8> ``` i think these changes might be the cause: https://github.com/ruby/ruby/commit/ec3542229b29ec93062e9d90e877ea29d3c1947… @jeremyevans0 JFYI -- https://bugs.ruby-lang.org/

6 6

[ruby-core:112578] [Ruby master Bug#19463] YJIT `[BUG] Stack consistency error` under certain invalidation scenarios
by alanwu (Alan Wu) 16 Jul '23

16 Jul '23

Issue #19463 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #19463: YJIT `[BUG] Stack consistency error` under certain invalidation scenarios https://bugs.ruby-lang.org/issues/19463 * Author: alanwu (Alan Wu) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) +YJIT [arm64-darwin22] * Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- > test.rb:19: [BUG] Stack consistency error (sp: 15, bp: 16) With the following: ```ruby klass = Class.new do def self.lookup(hash, key) = hash[key] def self.foo(a, b) = [] def self.test(hash, key) [lookup(hash, key), key, "".freeze] # 05 opt_send_without_block :lookup # 07 getlocal_WC_0 :hash # 09 opt_str_freeze "" # 12 newarray 3 # 14 leave # # YJIT will put instructions (07..14) into a block. # When String#freeze is redefined from within lookup(), # the return address to the block is still on-stack. We rely # on invalidation patching the code at the return address # to service this situation correctly. end end # get YJIT to compile test() hash = { 1 => [] } 31.times { klass.test(hash, 1) } # inject invalidation into lookup() evil_hash = Hash.new do |_, key| class String undef :freeze def freeze = :ugokanai end key end p klass.test(evil_hash, 1) ``` The fix is fairly simple and I'll apply it shortly. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:111735] [Ruby master Bug#19323] Integer overflow in `Integer#<<`
by nobu (Nobuyoshi Nakada) 25 Jun '23

25 Jun '23

Issue #19323 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19323: Integer overflow in `Integer#<<` https://bugs.ruby-lang.org/issues/19323 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 2.7: REQUIRED, 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- ```ruby require 'rbconfig/sizeof' char_bit = RbConfig::LIMITS["UCHAR_MAX"].bit_length size_max = RbConfig::LIMITS["SIZE_MAX"] size_bit_max = size_max * char_bit 1 << size_bit_max ``` The above code raises an "integer overflow" error. ``` -:5:in `<<': integer overflow: 4611686018427387905 * 4 > 18446744073709551615 (ArgumentError) from -:5:in `<main>' ``` It doesn't seem like a proper or intentional exception. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:111678] [Ruby master Feature#19315] Lazy substrings in CRuby
by Eregon (Benoit Daloze) 21 Jun '23

21 Jun '23

Issue #19315 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Feature #19315: Lazy substrings in CRuby https://bugs.ruby-lang.org/issues/19315 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal ---------------------------------------- CRuby should implement lazy substrings, i.e., "abcdef"[1..3] must not copy bytes. Currently CRuby only reuse the char* if the substring is until the end of the buffer. But it should also work wherever the substring starts and ends. Yes, it means RSTRING_PTR() might need to allocate to \0-terminate, so be it, it's worth it. There is already code for this (`SHARABLE_MIDDLE_SUBSTRING`), but it's disabled by default and `RSTRING_PTR()` needs to be changed to deal with this. It seems a good idea to introduce a variant of `RSTRING_PTR` which doesn't guarantee \0-termination, so such callers can then use the existing bytes always without copy. There are countless workarounds for this missing optimization, all not worth and all less readable: * https://bugs.ruby-lang.org/issues/19314 * https://github.com/ruby/net-protocol/pull/14 * Manual lazy substrings which track string + index + length * More but I don't remember all now, feel free to comment or link more urls/tickets. -- https://bugs.ruby-lang.org/

7 10