July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112399] [Ruby master Bug#19436] Call Cache for singleton methods can lead to "memory leaks"
by byroot (Jean Boussier) 23 Aug '23

23 Aug '23

Issue #19436 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#met…. This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

8 24

[ruby-core:114290] [Ruby master Feature#19785] Deprecate `RUBY_GC_HEAP_INIT_SLOTS `
by peterzhu2118 (Peter Zhu) 23 Aug '23

23 Aug '23

Issue #19785 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19785: Deprecate `RUBY_GC_HEAP_INIT_SLOTS ` https://bugs.ruby-lang.org/issues/19785 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/8116 The `RUBY_GC_HEAP_INIT_SLOTS` environment variable is replaced by `RUBY_GC_HEAP_INIT_SIZE_%d_SLOTS` (introduced in commit [3ab3455](https://github.com/ruby/ruby/commit/3ab34551450c7a3a3e1ae0b24bf6b7…) which allows for more fine-grained tuning of size pools. So it doesn't make sense to keep the environment variable `RUBY_GC_HEAP_INIT_SLOTS`. I'm proposing to deprecate `RUBY_GC_HEAP_INIT_SLOTS` and eventually remove it. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114274] [Ruby master Feature#19783] Weak References in the GC
by peterzhu2118 (Peter Zhu) 23 Aug '23

23 Aug '23

Issue #19783 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19783: Weak References in the GC https://bugs.ruby-lang.org/issues/19783 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/8113 I'm proposing support for weak references in the Ruby garbage collector. This feature adds a new function called `void rb_gc_mark_weak(VALUE *ptr)` which marks `*ptr` as weak, meaning that if no other object strongly marks `*ptr` (using `rb_gc_mark` or `rb_gc_mark_movable`), then it will be overwritten with `*ptr = Qundef`. Weak references are implemented using a buffer in `objspace` that stores all the `ptr` in the latest marking phase. After marking has finished, we iterate over the buffer and check if the `*ptr` is a dead object. If it is, then we set `*ptr = Qundef`. Weak references are implemented on the callable method entry (CME) of callcaches, which fixes issue #19436. Weak references are also implemented on `ObjectSpace::WeakMap` and `ObjectSpace::WeakKeyMap`, which have: - Significantly simpler implementations because we no longer need to have multiple tables and do not need to define finalizers on the objects. - Support for compaction because finalizers pin objects and we no longer need to define finalizers on the objects. - Much faster performance (see [benchmarks](#microbenchmarks)). ## Benchmark results ### YJIT-bench We see largely no change in performance or memory usage after this feature. ``` -------------- --------- ---------- --------- ----------- ---------- --------- -------------- ----------- bench base (ms) stddev (%) RSS (MiB) branch (ms) stddev (%) RSS (MiB) branch 1st itr base/branch activerecord 72.3 2.2 51.9 72.9 2.2 51.9 0.99 0.99 chunky-png 889.2 0.3 43.9 874.5 0.3 42.5 1.02 1.02 erubi-rails 21.2 13.5 90.7 21.0 13.3 90.9 1.01 1.01 hexapdf 2557.0 0.8 157.1 2559.2 0.7 197.1 1.01 1.00 liquid-c 65.2 0.4 34.5 65.4 0.4 34.5 0.99 1.00 liquid-compile 62.5 0.4 30.9 62.2 0.4 31.0 1.00 1.01 liquid-render 164.6 0.4 33.1 162.6 0.3 33.1 1.01 1.01 mail 133.3 0.1 46.4 134.4 0.2 46.4 1.03 0.99 psych-load 2066.6 0.2 31.6 2083.6 0.1 31.6 0.99 0.99 railsbench 2027.0 0.5 88.8 2019.4 0.5 89.0 1.01 1.00 ruby-lsp 65.6 3.0 90.1 65.4 3.1 88.5 1.00 1.00 sequel 73.1 1.1 36.6 73.1 1.1 36.6 1.00 1.00 -------------- --------- ---------- --------- ----------- ---------- --------- -------------- ----------- ``` ### Microbenchmarks We can see signficantly improved performance in `ObjectSpace::WeakMap`, with `ObjectSpace::WeakMap#[]=` being nearly 3x faster. Base: ``` ObjectSpace::WeakMap#[]= 1.037M (± 0.5%) i/s - 5.262M in 5.072833s ObjectSpace::WeakMap#[] 12.367M (± 0.9%) i/s - 62.479M in 5.052365s ``` Branch: ``` ObjectSpace::WeakMap#[]= 3.054M (± 0.3%) i/s - 15.448M in 5.058783s ObjectSpace::WeakMap#[] 15.796M (± 4.8%) i/s - 79.245M in 5.028583s ``` Code: ```ruby require "bundler/inline" gemfile do source "https://rubygems.org" gem "benchmark-ips" end wmap = ObjectSpace::WeakMap.new key = Object.new val = Object.new wmap[key] = val Benchmark.ips do |x| x.report("ObjectSpace::WeakMap#[]=") do |times| i = 0 while i < times wmap[Object.new] = Object.new i += 1 end end x.report("ObjectSpace::WeakMap#[]") do |times| i = 0 while i < times wmap[key] wmap[val] # does not exist i += 1 end end end ``` -- https://bugs.ruby-lang.org/

4 5

[ruby-core:114168] [Ruby master Misc#19766] DevMeeting-2023-08-24
by mame (Yusuke Endoh) 22 Aug '23

22 Aug '23

Issue #19766 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19766: DevMeeting-2023-08-24 https://bugs.ruby-lang.org/issues/19766 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/08/24 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/08/21. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

8 8

[ruby-core:114113] [Ruby master Bug#19761] AddressSanitizer fails with Thread and Process
by nobu (Nobuyoshi Nakada) 20 Aug '23

20 Aug '23

Issue #19761 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19761: AddressSanitizer fails with Thread and Process https://bugs.ruby-lang.org/issues/19761 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- With AddressSanitizer on macOS, this code crashes. ```ruby system('') Thread.new {} 300_000.times do String.new(capacity: 1000) end ``` ``` ==62870==ERROR: AddressSanitizer failed to deallocate 0x8000 (32768) bytes at address 0x62d00000a400 AddressSanitizer: CHECK failed: sanitizer_posix.cpp:61 "(("unable to unmap" && 0)) != (0)" (0x0, 0x0) (tid=857549) <empty stack> Assertion Failed: ../debug/vm_core.h:1853:rb_current_execution_context:!expect_ec || ec != NULL ruby 3.3.0dev (2023-07-08T02:54:37Z tmp 0083edc3a0) [x86_64-darwin22] -- Crash Report log information -------------------------------------------- See Crash Report log file in one of the following locations: * ~/Library/Logs/DiagnosticReports * /Library/Logs/DiagnosticReports for more details. Don't forget to include the above Crash Report log file in bug reports. -- C level backtrace information ------------------------------------------- ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:114218] [Ruby master Bug#19770] TLS / Certificate Hostname Verification against IP fails with "address family must be specified"
by Schachi65 (Joachim Schachermayer) 17 Aug '23

17 Aug '23

Issue #19770 has been reported by Schachi65 (Joachim Schachermayer). ---------------------------------------- Bug #19770: TLS / Certificate Hostname Verification against IP fails with "address family must be specified" https://bugs.ruby-lang.org/issues/19770 * Author: Schachi65 (Joachim Schachermayer) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.4 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I dont't use Ruby directly but indirectly via Fluentd-1.16.1 (https://www.fluentd.org/) Fluentd uses Ruby 3.1.4. The problem is, that a TLS connection from a Fluentd Client to a Fluentd Server fails, if the connection is established using the IP address of the server and tls_verify_hostname is switched to on. In the servers certificate, in the SubjectAlternativeNames extention, there is the servers IP address set with the correct value. I get the error message " **address family must be specified** ". **Probable cause** : ext/openssl/lib/openssl/ **ssl.rb** line 288: Here the Methode IPAddr.new(hostname) is called with just one argument: return true if san.value == IPAddr.new( **hostname** ).hton lib/ **ipaddr.rb** line 594: Here the exception is raised with the observed error message: when Socket::AF_UNSPEC **raise** AddressFamilyError, " **address family must be specified** " The exception is raised, because the variable "family" has the value " **Socket::AF_UNSPEC** " because thats the default value of "family" if there is no additional parameter in the call to "initialize()". The documumentation of "initialize()" says in line 575/576/577: "Although the address family is determined automatically from a specified string, you can specify one explicitly by the optional second argument." But that automatic address determination is done too late. If no second parameter is given, it is not done at all because that exception is raised (assumption: the addr i always a string). **Probable solution** : If the address family is set to Socket::AF_UNSPEC with the call of "initialize()", dont't raise an exception now but do the in the comment announced automatic adress family detection here. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114323] [Ruby master Bug#19791] Debugger does not handle delete key correctly
by stuartb (Stuart Bury) 15 Aug '23

15 Aug '23

Issue #19791 has been reported by stuartb (Stuart Bury). ---------------------------------------- Bug #19791: Debugger does not handle delete key correctly https://bugs.ruby-lang.org/issues/19791 * Author: stuartb (Stuart Bury) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux-musl] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- When entering the debugger using `binding.debugger` or the `rdbg` command, pressing the delete key on the keyboard to delete characters at the REPL merely prints "^[[3~" to the console, and does not actually delete anything. This is **very** inconvenient. Can this be fixed somehow? I really like the new debugger, but my swearing like a sailor every time I hit "delete" is starting to really annoy my wife... It is worth noting that if I run the `pry` command and then, within pry, call `binding.debugger`, the delete key operates correctly. I believe the issue is related in some way to readline support, or the lack thereof – but my Ruby installation was compiled with readline support. I've tried everything I can think of, and spent some time on Google trying to solve it, but the problem remains. It happens in Alpine 3.18 and also in Debian 12. -- https://bugs.ruby-lang.org/

2 4

[ruby-core:114315] [Ruby master Bug#19789] YJIT + rustc 1.71.0 is broken with macOS
by hsbt (Hiroshi SHIBATA) 12 Aug '23

12 Aug '23

Issue #19789 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19789: YJIT + rustc 1.71.0 is broken with macOS https://bugs.ruby-lang.org/issues/19789 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * Priority: Normal * Assignee: yjit * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I got some segmentation fault on macOS. ``` 1) Failure: TestVMDump#test_darwin_invalid_call [/Users/hsbt/Documents/github.com/ruby/ruby/test/ruby/test_vm_dump.rb:11]: pid 35071 exit 1 | -e:1: [BUG] Bus Error at 0x0000000000000001 | ruby 3.3.0dev (2023-07-30T06:13:58Z master b5c74d5488) [arm64-darwin23] | | -- Crash Report log information -------------------------------------------- | See Crash Report log file in one of the following locations: | * ~/Library/Logs/DiagnosticReports | * /Library/Logs/DiagnosticReports | for more details. | Don't forget to include the above Crash Report log file in bug reports. | | -- Control frame information ----------------------------------------------- | c:0003 p:---- s:0010 e:000009 CFUNC :call | c:0002 p:0015 s:0006 e:000005 EVAL -e:1 [FINISH] | c:0001 p:0000 s:0003 E:000a30 DUMMY [FINISH] | | -- Ruby level backtrace information ---------------------------------------- | -e:1:in `<main>' | -e:1:in `call' | | -- Threading information --------------------------------------------------- | Total ractor count: 1 | Ruby thread count for this ractor: 1 | | -- Machine register context ------------------------------------------------ | x0: 0x0000000000000000 x1: 0x0000000000000000 x2: 0x0000000000000000 | x3: 0x0000000000000000 x4: 0x0000000000000000 x5: 0x0000000000000000 | x6: 0x0000000000000000 x7: 0x0000000000000000 x18: 0x0000000000000000 | x19: 0x0000000000000000 x20: 0x000000016faadf40 x21: 0x000000016faadf40 | x22: 0x000000016faadec0 x23: 0x0000600001256eb0 x24: 0x000000016faadc70 | x25: 0x0000000000000000 x26: 0x0000000000000000 x27: 0x0000000000000000 | x28: 0x0000000140e04f20 lr: 0x000000019b211050 fp: 0x000000016faadd30 | sp: 0x000000016faadd30 | | -- C level backtrace information ------------------------------------------- | 1360: Abbrev Number 707 not found .. ``` I filed it and other failures to https://gist.github.com/hsbt/ea3e52d9fb52aed58d25caf839eae5ff @katei found this issue caused by YJIT and rustc 1.71.0. It's not happend with ``` ../ruby/configure -C --disable-install-doc --with-openssl-dir=/opt/homebrew/opt/openssl(a)1.1 --with-readline-dir=/opt/homebrew/opt/readline --with-libyaml-dir=/opt/homebrew/opt/libyaml RUSTC='rustc +1.70.0' ``` But `RUSTC='rustc +1.71.0'` is failed with same error on macOS Sonoma(beta) and Ventura. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114221] [Ruby master Bug#19771] Backport 79a4484, abfac22
by krk (Kerem Kat) 08 Aug '23

08 Aug '23

Issue #19771 has been reported by krk (Kerem Kat). ---------------------------------------- Bug #19771: Backport 79a4484, abfac22 https://bugs.ruby-lang.org/issues/19771 * Author: krk (Kerem Kat) * Status: Open * Priority: Normal * ruby -v: 3.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- * https://github.com/ruby/ruby/commit/79a4484a072e9769b603e7b4fbdb15b1d7eccb15 * https://github.com/ruby/ruby/commit/abfac2222bd51a03ec11b64f4770179cb6d06515 Related backport PR: https://github.com/ruby/ruby/pull/8023 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:111740] [Ruby master Bug#19325] Windows support lacking.
by dsisnero (Dominic Sisneros) 03 Aug '23

03 Aug '23

Issue #19325 has been reported by dsisnero (Dominic Sisneros). ---------------------------------------- Bug #19325: Windows support lacking. https://bugs.ruby-lang.org/issues/19325 * Author: dsisnero (Dominic Sisneros) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Ruby's support on windows has always been second class. With some of the recent decisions, windows support is falling even more behind. Recent developments in mjit and yjit that exclude windows are two glaring issues that should be corrected. Googling 'percent of windows vs other operating systems' and it shows windows has a share of 76%. Ceding that users to python and other programming languages has to be one of the reasons python continues get more market share from ruby. With rust having first class windows support and threading support, is there a reason why yjit is not able to work on windows? Also, windows compiler support has matured enough and vcpkg support has evolved enough that it seems it should be possible to finally get a ruby version without having to use msys2. Even Crystal language has a version that runs on windows without needing msys2. -- https://bugs.ruby-lang.org/

9 10