[ruby-dev:52069] [Ruby master Bug#20311] Struct.new("A") memory leak?
Issue #20311 has been reported by MaxLap (Maxime Lapointe). ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by byroot (Jean Boussier). I had a quick look at this using `ObjectSpace.dump_all`, and it appears that these classes are being retained in the VM root: ```ruby require 'objspace' 3.times do puts ObjectSpace.dump(Struct.new("A")) Struct.send(:remove_const, :A) end GC.start ObjectSpace.dump_all(output: File.open("/tmp/heap.json", "w+")) ``` ``` {"address":"0x102bc33f8", "type":"CLASS", .... {"address":"0x102bc3218", "type":"CLASS", .... {"address":"0x102bc3038", "type":"CLASS", .... ``` ``` $ rg -F '"0x102bc3218"' /tmp/heap.json 1:{"type":"ROOT", "root":"vm", "references":[..., "0x102bc3218" $ rg -F '"0x102bc33f8"' /tmp/heap.json 1:{"type":"ROOT", "root":"vm", "references":[..., "0x102bc33f8" ``` ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107049 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by nobu (Nobuyoshi Nakada). Seems referred from `defined_module_hash`. `rb_const_remove` needs to take care of it when the removed constant is a class or module? ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107052 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by byroot (Jean Boussier).
rb_const_remove needs to take care of it when the removed constant is a class or module?
The problem is that it's a set: ```c st_insert(vm->defined_module_hash, (st_data_t)module, (st_data_t)module); ``` So if a module is added twice, you need to remove it from the set if it's removed twice... I'll see if I can do that. But I also wonder why we have this root in the first place, I'd think these get marked from being constants in `::Object`. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107073 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by byroot (Jean Boussier).
But I also wonder why we have this root in the first place, I'd think these get marked from being constants in ::Object.
Nevermind, if I understand correctly, it's not to mark them, it's to pin them. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107074 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by byroot (Jean Boussier). I think https://github.com/ruby/ruby/pull/10143 will do. But a good followup would also be to not pin `Struct.new("A")`. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107075 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by byroot (Jean Boussier). So https://github.com/ruby/ruby/pull/10143 is turning into a huge yak-shave and will be hard to backport. The reason is a lot of code end up relying on classes defined in C becoming immortal (in addition to being pinned). So maybe a better short term solution that is easy to backport is to just not use these API for naming structs: https://github.com/ruby/ruby/pull/10144 ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107077 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by nobu (Nobuyoshi Nakada). byroot (Jean Boussier) wrote in #note-3:
So if a module is added twice, you need to remove it from the set if it's removed twice... I'll see if I can do that.
Is there any chance for the same module to be added twice? ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107087 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- The following code gives the impression of a memory leak. ``` 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by nobu (Nobuyoshi Nakada). byroot (Jean Boussier) wrote in #note-7:
So maybe a better short term solution that is easy to backport is to just not use these API for naming structs: https://github.com/ruby/ruby/pull/10144
Seems fine, but I'm afraid what can happen when an extension library stores `rb_path2class("Struct::A")` result then remove that constant? Probably this is same for any Ruby-level defined classes/modules. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107089 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- The following code gives the impression of a memory leak. ```ruby 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by byroot (Jean Boussier).
Is there any chance for the same module to be added twice?
Yes, aliasing isn't uncommon.
Seems fine, but I'm afraid what can happen when an extension library stores rb_path2class("Struct::A") result then remove that constant? Probably this is same for any Ruby-level defined classes/modules.
Yes, my reasoning is that it's not reasonable to expect classes created in Ruby to be immortal and immovable like the ones created in C. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107090 * Author: MaxLap (Maxime Lapointe) * Status: Open * ruby -v: 3.3.0 * Backport: 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- The following code gives the impression of a memory leak. ```ruby 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by naruse (Yui NARUSE). Backport changed from 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED to 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE ruby_3_3 f79b1d1ef1f7aa64d20f0eadbb3b0f8f7084deb3 merged revision(s) e626da82eae3d437b84d4f9ead0164d436b08e1a,f3af5ae7e6c1c096bbfe46d69de825a02b1696cf. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-107404 * Author: MaxLap (Maxime Lapointe) * Status: Closed * ruby -v: 3.3.0 * Backport: 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE ---------------------------------------- The following code gives the impression of a memory leak. ```ruby 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
Issue #20311 has been updated by nagachika (Tomoyuki Chikanaga). Backport changed from 3.0: WONTFIX, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: DONE to 3.0: WONTFIX, 3.1: REQUIRED, 3.2: DONE, 3.3: DONE ruby_3_2 commit:bd5df1693c89d389471d145fc19b487c708912b1 merged revision(s) commit:e626da82eae3d437b84d4f9ead0164d436b08e1a, commit:f3af5ae7e6c1c096bbfe46d69de825a02b1696cf. ---------------------------------------- Bug #20311: Struct.new("A") memory leak? https://bugs.ruby-lang.org/issues/20311#change-108988 * Author: MaxLap (Maxime Lapointe) * Status: Closed * ruby -v: 3.3.0 * Backport: 3.0: WONTFIX, 3.1: REQUIRED, 3.2: DONE, 3.3: DONE ---------------------------------------- The following code gives the impression of a memory leak. ```ruby 10.times do 5000.times do Struct.new("A") Struct.send(:remove_const, :A) end GC.start puts `ps -o rss= -p #{$$}`.to_i end ``` ``` 27868 35324 43400 51472 58676 66144 73764 81196 88512 95752 ``` Is there another location where the struct gets set that I need to clear up for the GC free the memory? Happens in 3.2.2, 3.2.3, 3.3.0, 3.3-head, ruby-head. -- https://bugs.ruby-lang.org/
participants (5)
-
byroot (Jean Boussier) -
MaxLap (Maxime Lapointe)
-
nagachika (Tomoyuki Chikanaga)
-
naruse (Yui NARUSE)
-
nobu (Nobuyoshi Nakada)