Issue #22390 has been updated by byroot (Jean Boussier). Not 100% sure it's the exact same issue, but the discussion on https://github.com/ruby/ruby/pull/18433#issuecomment-5679306710 may be of interest to you. ---------------------------------------- Bug #22390: ruby 3.4 -> 4.0 upgrade GC changes causing >2x RSS increase https://bugs.ruby-lang.org/issues/22390#change-119259 * Author: chucke (Tiago Cardoso) * Status: Open * ruby -v: 4.0.6 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Hello, I've been trying to upgrade a fairly sizeable monolith, which has been running on ruby 3.4.10 in production, to ruby 4 (have been targeting 4.0.6 more specifically, although the below should apply the same to 4.0.7). While we didn't experience anything out of the ordinary in staging, unfortunately, we started seeing our more intensive workloads (job processors) grow in memory usage over the first 40m of running on ruby 4, up to the point where the servers would hit OOM, at which point we decided to rollback and investigate. This workload calls `Process.warmup` after booting the application and then fork multiple processes (not sure if this information is relevant). We figured out that the culprit of this change in GC behaviour was [this commit](https://github.com/ruby/ruby/commit/d4406f0627c78af31e61f9e07dda9151e109dbc4) from @peter.zhu; this commit actually another bug for which we have a mitigation in ruby 3.4, however it changed quite a few more things in the heuristics used to decide when to give empty pages back to the OS. Before the commit, `heap_pages_free_unused_pages` used `objspace->heap_pages.allocatable_slots` to decide, and this number would be reset to 0 on every sweep, and grow when expanding the heap, which resulted in every empty page being freed on every major GC. After the commit, this changed to using `heap_pages_freeable_pages`, which gets computed on every major GC, and matched against GC_HEAP_FREE_SLOTS_MAX_RATIO. In our case at least, this doesn't work out, because in the aforementioned workload, `objspace_available_slots` grows at the same pace as `heap_pages_freeable_pages`; this makes it so that the 65% max ratio is never reached for this type of steadily churning live sets; by the time the size pools settle, the remainder of empty pages is never given back. I also tried to pull claude to figure some of this out, and it mentioned that the per pool 20% reservation alongside the single (per process) empty pages management somehow plays a role, but I couldn't make sense of whether that was legit or just an hallucination. I'm attaching a script which I coded with some AI help to roughly mimic the objects being created per job iteration. Bear in mind that, in our case in production, we're talking about many more (OOM hits at ~5Gb, and we've seen the ruby 4 builds grow up to 4Gb before we rolled back), but this represents the type of objects created in a way that makes each of the pools grow independently. If you run it locally, you should see the ruby 4 example use ~2x the memory, mostly because of the empty pages being kept around; major gcs will sometimes free them back to the OS, but I suspect that the reason for it was some other condition triggering a major GC (oldmalloc perhaps?) and causing `heap_pages_freeable_pages` to be recalculated as a result. We continue experimenting with the GC knobs to see if we can find a combination which would allow us to run ruby 4 in production. We're looking at pre-allocating a bunch of slots in pools 1-5 (as per upper bounds from observations using `GC.stat_heap`), while lowering GC_HEAP_OLDOBJECT_LIMIT_FACTOR to force major GC to run more often (a tad more, not too much, not to hurt latency) and hopefully updating freeable pages stats, while also lowering GC_HEAP_FREE_SLOTS_MAX_RATIO as well to a value that makes more sense. If anyone has suggestions on how to further get to optimal values, I'm all ears. Still, while I accept that special workloads require special tuning, I don't find this one unusual enough to justify this type of memory usage increase just due to a ruby upgrade, hence why I file this as a bug. ---Files-------------------------------- repro_gc.rb (4.56 KB) -- https://bugs.ruby-lang.org/

Issue #22390 has been updated by hsbt (Hiroshi SHIBATA). I asked Claude to investigate this with the attached script. Comparing [d4406f0627c](https://github.com/ruby/ruby/commit/d4406f0627c78af31e61f9e07dda9151e109dbc4) with its parent confirms the change comes from that commit. On macOS with YJIT off and 100k iterations, the parent stays at 28MB RSS with 16303 minor GCs, while d4406f0627c reaches 50MB with 3881 minor GCs. The empty pages are not where the memory goes. The GC heap grows by only 1.3MB (41 to 62 pages), and the other 20MB is free memory kept by malloc. GC runs four times less often, so garbage strings and `JSON.parse` results live longer. The cause is that `heap_page_allocate_and_initialize` now takes pages from the empty page pool without checking or charging `allocatable_slots`. `heap_prepare` tries it before starting a GC, so every pooled page becomes room to allocate without GC. Limiting only that path to the budget brings the script back to 23MB and 30 pages. It still fails `test_thrashing_for_young_objects` by one extra page, and in the Bug #21214 script the pool keeps 394 unused pages because the free slots ratio check ignores pooled pages. Master is worse (61MB, 94 pages, 8 minor GCs over 30k iterations) and the same change does not help there. None of the `RUBY_GC_*` variables I tried, nor [#18433](https://github.com/ruby/ruby/pull/18433), fix this case. ---------------------------------------- Bug #22390: ruby 3.4 -> 4.0 upgrade GC changes causing >2x RSS increase https://bugs.ruby-lang.org/issues/22390#change-119326 * Author: chucke (Tiago Cardoso) * Status: Assigned * Assignee: peterzhu2118 (Peter Zhu) * ruby -v: 4.0.6 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Hello, I've been trying to upgrade a fairly sizeable monolith, which has been running on ruby 3.4.10 in production, to ruby 4 (have been targeting 4.0.6 more specifically, although the below should apply the same to 4.0.7). While we didn't experience anything out of the ordinary in staging, unfortunately, we started seeing our more intensive workloads (job processors) grow in memory usage over the first 40m of running on ruby 4, up to the point where the servers would hit OOM, at which point we decided to rollback and investigate. This workload calls `Process.warmup` after booting the application and then fork multiple processes (not sure if this information is relevant). We figured out that the culprit of this change in GC behaviour was [this commit](https://github.com/ruby/ruby/commit/d4406f0627c78af31e61f9e07dda9151e109dbc4) from @peter.zhu; this commit actually another bug for which we have a mitigation in ruby 3.4, however it changed quite a few more things in the heuristics used to decide when to give empty pages back to the OS. Before the commit, `heap_pages_free_unused_pages` used `objspace->heap_pages.allocatable_slots` to decide, and this number would be reset to 0 on every sweep, and grow when expanding the heap, which resulted in every empty page being freed on every major GC. After the commit, this changed to using `heap_pages_freeable_pages`, which gets computed on every major GC, and matched against GC_HEAP_FREE_SLOTS_MAX_RATIO. In our case at least, this doesn't work out, because in the aforementioned workload, `objspace_available_slots` grows at the same pace as `heap_pages_freeable_pages`; this makes it so that the 65% max ratio is never reached for this type of steadily churning live sets; by the time the size pools settle, the remainder of empty pages is never given back. I also tried to pull claude to figure some of this out, and it mentioned that the per pool 20% reservation alongside the single (per process) empty pages management somehow plays a role, but I couldn't make sense of whether that was legit or just an hallucination. I'm attaching a script which I coded with some AI help to roughly mimic the objects being created per job iteration. Bear in mind that, in our case in production, we're talking about many more (OOM hits at ~5Gb, and we've seen the ruby 4 builds grow up to 4Gb before we rolled back), but this represents the type of objects created in a way that makes each of the pools grow independently. If you run it locally, you should see the ruby 4 example use ~2x the memory, mostly because of the empty pages being kept around; major gcs will sometimes free them back to the OS, but I suspect that the reason for it was some other condition triggering a major GC (oldmalloc perhaps?) and causing `heap_pages_freeable_pages` to be recalculated as a result. We continue experimenting with the GC knobs to see if we can find a combination which would allow us to run ruby 4 in production. We're looking at pre-allocating a bunch of slots in pools 1-5 (as per upper bounds from observations using `GC.stat_heap`), while lowering GC_HEAP_OLDOBJECT_LIMIT_FACTOR to force major GC to run more often (a tad more, not too much, not to hurt latency) and hopefully updating freeable pages stats, while also lowering GC_HEAP_FREE_SLOTS_MAX_RATIO as well to a value that makes more sense. If anyone has suggestions on how to further get to optimal values, I'm all ears. Still, while I accept that special workloads require special tuning, I don't find this one unusual enough to justify this type of memory usage increase just due to a ruby upgrade, hence why I file this as a bug. ---Files-------------------------------- repro_gc.rb (4.56 KB) -- https://bugs.ruby-lang.org/