[ruby-core:112399] [Ruby master Bug#19436] Call Cache for singleton methods can lead to "memory leaks"

Issue #19436 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). I think making `Class#attached_object` a weak reference is a better and easier solution. Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM, as it would mean one extra WeakReference object for each of these call caches. I suppose there could be workarounds like storing the WeakReference on the singleton class, and use a direct reference otherwise, but then it'd make method lookup more complex and slower in interpreter or when polymorphic (need to handle cached method lookup on regular and singleton class differently). FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes. That approach is probably also quite difficult to get right on CRuby, also considering ractors and multithreading when the class GCs. As a general note, creating a singleton class is not cheap, this should only be used for class objects (which always have one) and for a few rare global objects where it's convenient. Using `Object#extend` objects often/on the fast path is just "making programs slow and uncached". ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-101842 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier).
FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.
I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a `Module` or `Class`.
Using Object#extend objects often/on the fast path is just "making programs slow and uncached".
Yeah I agree. I tried to avoid this in Rails by caching these dynamic classes, unfortunately that's not possible because of the API: https://github.com/rails/rails/pull/47314. So I'll try to see if this API can be replaced by a better one and deprecated. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-101843 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier).
Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM
Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-101844 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). byroot (Jean Boussier) wrote in #note-2:
I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a `Module` or `Class`.
I missed that sorry. I think there are quite a few cases where a global object (which is not a module/class) with a singleton class is used, it'd be unfortunate and slow to not inline cache lookup in those cases. I think it makes sense to only try caching one (or a few) singleton class(es), and give up when seeing a second one or more (already happens on TruffleRuby, but for a higher limit, 8 currently). That meant we still need to make `Class#attached_object` a weak reference. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-101846 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). byroot (Jean Boussier) wrote in #note-3:
Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.
Yes, since TruffleRuby caches the the Class object in method lookup (LookupMethodNode), and currently the Class object has a strong reference to the `attached_object`. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-101847 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier).
That means we still need to make Class#attached_object a weak reference.
I agree that conceptually, it's the simplest solution. That being said, in MRI weak references generally require a finalizer, which is annoying. But I'll see if I can implement that. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-101849 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ko1 (Koichi Sasada). Matz asked me to solve this memory leaking issue, so I'll consider about it before Ruby 3.3. One idea is re-introduce class serial. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102332 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier).
One idea is re-introduce class serial.
Does that mean the method cache could be global again? ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102336 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). @ko1 I think the real solution to this category of problems, not just this specific one is to make `Class#attached_object` (internally) a weak reference. So I would strongly advise against changing the method lookup for this, it's an independent problem. For instance, if someone does some caching in a Hash based on the Class, they don't expect holding onto a Class also holds onto the singleton object forever. So `Class#attached_object` (the internal field used for it) is the leak, not the inline cache itself which needs to hold onto the class. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102346 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). Also as mentioned in https://bugs.ruby-lang.org/issues/19436#note-1 I believe TruffleRuby and JRuby (and probably more) would never use a weak reference in the inline cache, it's too much of a footprint overhead and also an extra indirection at least in the interpreter. So then the only other solution I see is `Class#attached_object` (internally) being weak. I think CRuby should follow the same idea, otherwise CRuby would be knowingly hurting compatibility between Ruby implementations here. A per-class serial as far as I understand it is no good for a JIT, if e.g. methods are defined in some class late or repeatedly, it would invalidate all inline cache of method lookup on that class and all related JITed code, needlessly. https://medium.com/graalvm/precise-method-and-constant-invalidation-in-truff... has more details on that. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102348 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ko1 (Koichi Sasada). byroot (Jean Boussier) wrote in #note-8:
One idea is re-introduce class serial.
Does that mean the method cache could be global again?
No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102349 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ko1 (Koichi Sasada). Eregon (Benoit Daloze) wrote in #note-9:
@ko1 I think the real solution to this category of problems, not just this specific one is to make `Class#attached_object` (internally) a weak reference.
```ruby o = Object.new c = o.singleton_class o = nil # do some task p c.attached_object #=> ??? ``` making weak reference, `attached_objec` may return nil for the collected object? I'm not sure it is acceptable or not. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102350 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). ko1 (Koichi Sasada) wrote in #note-12:
making weak reference, `attached_objec` may return nil for the collected object?
I'm not sure it is acceptable or not.
Yes, this is indeed a consequence. I think it's fine, `Class#attached_object` is very recent and this is necessary to solve a fundamental leak. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102354 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by Eregon (Benoit Daloze). ko1 (Koichi Sasada) wrote in #note-11:
No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.
That would mean an extra indirection/memory load on every cached method lookup, i.e. `obj->klass == CACHED_CLASS` (current) vs `obj->klass->serial == CACHED_SERIAL`. Also it would require these serials to be unique, otherwise that's not a correct check (so maybe more of a "unique ID" than serial). ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102355 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ufuk (Ufuk Kayserilioglu). Let me add a few points in this conversation:
making weak reference, attached_objec may return nil for the collected object?
I'm not sure it is acceptable or not.
I agree with @Eregon on this point that this is totally acceptable. When I was proposing `Class#attached_object` the argument was that it would be helpful in cases where the objects are still supposed to be in memory for introspection purposes. There is very little value in trying to introspect an attached object that was already garbage collected, and it would not be great to retain objects just because they had a singleton class defined for them. So, I am +1 for slightly changing the semantics of `Class#attached_object` before it becomes more widely used. I should also say, though, that the purist in me still considers that the cache should be the one holding weak references, since, after all it is a cache and a miss should be relatively inconsequential, as opposed to the link between a singleton class and its attached object. I do understand, however, that it might not be the desired implementation and pragmatism over purism might be more warranted here. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102361 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier). Assignee set to ko1 (Koichi Sasada) ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-102372 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier). @ko1 thanks for the patch, however we're running into some assertion failures on our CI: ``` Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS) /usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_dump.c:1088 /usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x563d185aeb55] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/error.c:879 /usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x563d1859d954] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_callinfo.h:339 /usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x563d1877a9b8] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_insnhelper.c:2181 ``` Which I suspect are related? ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104002 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ko1 (Koichi Sasada). Thanks. Could you give some more lines of C backtrace? ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104003 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier). @ko1 of course, here you go: ``` Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS) ruby 3.3.0dev (2023-07-30T08:05:58Z shopify b5c74d5488) [x86_64-linux] -- Threading information --------------------------------------------------- Total ractor count: 1 Ruby thread count for this ractor: 5 -- C level backtrace information ------------------------------------------- /usr/local/ruby/bin/ruby(rb_print_backtrace+0xd) [0x55dcf6947c3a] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:772 /usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:1088 /usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x55dcf6751c85] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/error.c:879 /usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x55dcf6740995] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_callinfo.h:339 /usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x55dcf691db28] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:2181 /usr/local/ruby/bin/ruby(vm_search_super_method) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:4637 /usr/local/ruby/bin/ruby(vm_sendish+0xac) [0x55dcf6938e40] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5520 /usr/local/ruby/bin/ruby(vm_exec_core) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:939 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(catch_i+0xa3) [0x55dcf692da33] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(vm_catch_protect+0xfd) [0x55dcf69266bd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2360 /usr/local/ruby/bin/ruby(rb_catch_obj+0x28) [0x55dcf692697d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2386 /usr/local/ruby/bin/ruby(rb_f_catch) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2336 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728 /usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236 /usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329 /usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(rb_funcallv_scope+0x6f) [0x55dcf6932fac] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1062 /usr/local/ruby/bin/ruby(rb_funcallv) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1077 /usr/local/ruby/bin/ruby(autoload_feature_require+0x70) [0x55dcf690bee6] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2739 /usr/local/ruby/bin/ruby(autoload_try_load) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2753 /usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009 /usr/local/ruby/bin/ruby(rb_autoload_load+0x12f) [0x55dcf690a94f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2814 /usr/local/ruby/bin/ruby(rb_const_lookup+0x0) [0x55dcf690ab9b] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2916 /usr/local/ruby/bin/ruby(rb_const_search_from) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2903 /usr/local/ruby/bin/ruby(rb_const_search+0x7) [0x55dcf690b097] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2938 /usr/local/ruby/bin/ruby(rb_const_get_0) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2865 /usr/local/ruby/bin/ruby(rb_mod_const_get+0x1f5) [0x55dcf67ec8e5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/object.c:2401 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield_values2+0xa0) [0x55dcf692dae0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(each_pair_i_fast+0x37) [0x55dcf6780cf7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3046 /usr/local/ruby/bin/ruby(RHASH_ST_TABLE+0x0) [0x55dcf678111f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1253 /usr/local/ruby/bin/ruby(hash_foreach_iter) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1255 /usr/local/ruby/bin/ruby(st_general_foreach+0x3b) [0x55dcf68ab030] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1516 /usr/local/ruby/bin/ruby(rb_st_foreach_check) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1621 /usr/local/ruby/bin/ruby(hash_foreach_call+0x3d) [0x55dcf678431d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1382 /usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009 /usr/local/ruby/bin/ruby(rb_hash_foreach+0x91) [0x55dcf6784519] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1406 /usr/local/ruby/bin/ruby(rb_hash_foreach) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1392 /usr/local/ruby/bin/ruby(rb_hash_each_pair+0x47) [0x55dcf67845b7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3081 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0x11d) [0x55dcf692e0ad] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(yield_under+0x234) [0x55dcf692ceb4] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70 /usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70 /usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70 /usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70 /usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call0_cfunc_with_frame+0xe8) [0x55dcf692e7d0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:170 /usr/local/ruby/bin/ruby(vm_call0_cfunc) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:184 /usr/local/ruby/bin/ruby(vm_call0_body) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:230 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70 /usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728 /usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236 /usr/local/ruby/bin/ruby(rb_require_string+0x35) [0x55dcf67bff94] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329 /usr/local/ruby/bin/ruby(rb_f_require_relative) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:989 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728 /usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236 /usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329 /usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728 /usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236 /usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329 /usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520 /usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107 /usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(raise_load_if_failed+0x0) [0x55dcf67bd63d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728 /usr/local/ruby/bin/ruby(rb_load_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:791 /usr/local/ruby/bin/ruby(rb_f_load+0x89) [0x55dcf67bdef9] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:864 /usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458 /usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517 /usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835 /usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399 /usr/local/ruby/bin/ruby(rb_ec_exec_node+0xc1) [0x55dcf67530b1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:287 /usr/local/ruby/bin/ruby(ruby_run_node+0x56) [0x55dcf6758406] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:328 /usr/local/ruby/bin/ruby(rb_main+0x21) [0x55dcf6752637] ./main.c:39 /usr/local/ruby/bin/ruby(main) ./main.c:58 /lib/x86_64-linux-gnu/libc.so.6(0x7f686651ad90) [0x7f686651ad90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f686651ae40] [0x55dcf6752685] ``` ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104014 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ko1 (Koichi Sasada). Thanks, Could you try current master (I hope it was fixed). ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104017 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier). @ko1 I don't see any more failures, thank you very much. So overall the patch is all these commits? ``` c330037c1a38cc257dbe21022fcc7b13159c2821 - cc->cme should not be marked. 6391132c03ac08da0483adb986ff9a54e41f9e14 - fix typo (CACH_ -> CACHE_) 6dc15cc8895b28800d5c187929d846ac4eb7cd3f - do not clear cme but invalidate cc 7a7aba755d8da34555445510dc568380d0d96791 - check liveness of cc->klass and cc->cme_ 087a2deccfb9d99961f1ce8526b80c5f72ee9a5d - check cc->* liveness strictly 36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super 280419d0e0ba3e96e19551c70cba789fbedd80e1 - calling->cd instead of calling->ci ``` I'm asking because I may test it more extensively in production which involve backporting it to our custom 3.2. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104018 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by ko1 (Koichi Sasada).
36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super
for this SEGV. But other patches can be required. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104019 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by cedricm (Cédric Maïon). Hi, Will this be backported to 3.2? I'm currently seeing this in production on a large Rails app: when YJIT is enabled, after usual initial warmup, unicorn workers memory usage continue to grows linearly until reaching a significant amount of RAM -- at which point they are either killed by a monitoring tool due to unreasonable memory usage (>650MB), or some cache gets flushed and a big chunk of memory gets released (and then it goes back for another round). Memory usage is perfectly stable without YJIT (~420MB). I believe that this is linked to this bug. Cheers ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104229 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

Issue #19436 has been updated by byroot (Jean Boussier). This bug is unrelated to YJIT, it happens with or without YJIT enabled. It thus cannot explain what you are seeing. As for memory usage with YJIT, what you are describing isn't out of the ordinary. The default `--yjit-exec-mem-size` is `64MB`, and the rule of thumb is that if you had the metadata YJIT need to track, you will find an overhead of ~4x `yjit-exec-mem-size`, so roughly 250MB. So you probably want to raise your Unicorn memory limit a bit more, or reduce the `--yjit-exec-mem-size`. ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436#change-104230 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Target version: 3.3 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-...). This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/
participants (8)
-
byroot (Jean Boussier)
-
byroot (Jean Boussier)
-
cedricm
-
Eregon (Benoit Daloze)
-
Eregon (Benoit Daloze)
-
ko1 (Koichi Sasada)
-
ko1 (Koichi Sasada)
-
ufuk (Ufuk Kayserilioglu)