Issue #21955 has been updated by rwstauner (Randy Stauner). backport PR for 4.0: https://github.com/ruby/ruby/pull/16422 ---------------------------------------- Bug #21955: `Fiber#transfer`: machine stack not released when fiber terminates, causing `FiberError: can't set a guard page` https://bugs.ruby-lang.org/issues/21955#change-116753 * Author: ioquatix (Samuel Williams) * Status: Closed * Assignee: ioquatix (Samuel Williams) * Backport: 3.2: REQUIRED, 3.3: REQUIRED, 3.4: REQUIRED, 4.0: REQUIRED ---------------------------------------- When a fiber terminates (falls off the end of its block, or raises an unhandled exception) after being reached via `Fiber#transfer`, its machine stack is **not** returned to the fiber pool. The stack is only freed when the `Fiber` object is eventually garbage collected. In production, where major GC does not run regularly (or at all), terminated fibers accumulate unreleased stacks. Each stack allocation contains a guard page set with `mprotect(PROT_NONE)`, which splits a kernel VMA. On Linux this exhausts the per-process `vm.max_map_count` limit and raises: ``` FiberError: can't set a guard page: Cannot allocate memory ``` The symptom is confusing: the fibers are all dead (`alive? == false`), but new fibers cannot be created. ## Root cause In `cont.c`, `fiber_switch()`, the eager stack release after `fiber_store` returns is guarded by `resuming_fiber`: ```c // cont.c (affected versions) #ifndef COROUTINE_PTHREAD_CONTEXT if (resuming_fiber && FIBER_TERMINATED_P(fiber)) { RB_VM_LOCKING() { fiber_stack_release(fiber); } } #endif ``` `resuming_fiber` is only non-`NULL` when the switch was initiated by `Fiber#resume` (which passes `resuming_fiber = fiber`). `Fiber#transfer` passes `resuming_fiber = NULL`, so the condition is never true and the stack is silently leaked until GC. Additionally, `fiber_raise` on a suspended non-yielding (transferred) fiber calls `fiber_transfer_kw`, also passing `resuming_fiber = NULL`, so the same leak occurs when a transferred fiber is terminated by a raised exception. ## Reproduction Set a low map count limit, then run the attached script: ``` sudo bash -c "echo 10000 > /proc/sys/vm/max_map_count" ruby test_fiber_transfer_leak.rb ``` Restore afterwards (`sysctl vm.max_map_count` shows the default, typically 65530). ```ruby GC.disable leaked = [] count = 0 begin 10_000.times do f = Fiber.new { } # terminates immediately leaked << f # hold reference so Fiber object is not GC'd f.transfer # transfer, not resume => stack NOT released (bug) count += 1 puts "[#{count} fibers] all dead: #{leaked.none?(&:alive?)}" if count % 1000 == 0 end puts "No error — fix is applied." rescue FiberError => e puts "FiberError after #{count} fibers: #{e.message}" puts "All terminated (alive?=false): #{leaked.none?(&:alive?)}" ensure GC.enable leaked.clear GC.start end ``` Expected output on **unpatched** Ruby (vm.max_map_count=10000): ``` [1000 fibers] all dead: true [2000 fibers] all dead: true [3000 fibers] all dead: true [4000 fibers] all dead: true FiberError after 4096 fibers: can't set a guard page: Cannot allocate memory All terminated (alive?=false): false ``` Expected output on **patched** Ruby: ``` [1000 fibers] all dead: true ... [10000 fibers] all dead: true No error — fix is applied. ``` ## Fix Drop the `resuming_fiber &&` guard. After `fiber_store` returns we are executing in the caller's context — we are never on `fiber`'s stack — so releasing it is unconditionally safe. `fiber_stack_release` is already idempotent (guarded by `fiber->stack.base == NULL`), so the resume path is unaffected. ```c #ifndef COROUTINE_PTHREAD_CONTEXT if (FIBER_TERMINATED_P(fiber)) { RB_VM_LOCKING() { fiber_stack_release(fiber); } } #endif ``` Pull request with fix: https://github.com/ruby/ruby/pull/16416 -- https://bugs.ruby-lang.org/

Issue #21955 has been updated by scohen-tines (seth cohen). The backport of this fix to 3.4.10 seems to be the cause of the linked issue: https://bugs.ruby-lang.org/issues/22196. ---------------------------------------- Bug #21955: `Fiber#transfer`: machine stack not released when fiber terminates, causing `FiberError: can't set a guard page` https://bugs.ruby-lang.org/issues/21955#change-118096 * Author: ioquatix (Samuel Williams) * Status: Closed * Assignee: ioquatix (Samuel Williams) * Backport: 3.2: REQUIRED, 3.3: REQUIRED, 3.4: DONE, 4.0: DONE ---------------------------------------- When a fiber terminates (falls off the end of its block, or raises an unhandled exception) after being reached via `Fiber#transfer`, its machine stack is **not** returned to the fiber pool. The stack is only freed when the `Fiber` object is eventually garbage collected. In production, where major GC does not run regularly (or at all), terminated fibers accumulate unreleased stacks. Each stack allocation contains a guard page set with `mprotect(PROT_NONE)`, which splits a kernel VMA. On Linux this exhausts the per-process `vm.max_map_count` limit and raises: ``` FiberError: can't set a guard page: Cannot allocate memory ``` The symptom is confusing: the fibers are all dead (`alive? == false`), but new fibers cannot be created. ## Root cause In `cont.c`, `fiber_switch()`, the eager stack release after `fiber_store` returns is guarded by `resuming_fiber`: ```c // cont.c (affected versions) #ifndef COROUTINE_PTHREAD_CONTEXT if (resuming_fiber && FIBER_TERMINATED_P(fiber)) { RB_VM_LOCKING() { fiber_stack_release(fiber); } } #endif ``` `resuming_fiber` is only non-`NULL` when the switch was initiated by `Fiber#resume` (which passes `resuming_fiber = fiber`). `Fiber#transfer` passes `resuming_fiber = NULL`, so the condition is never true and the stack is silently leaked until GC. Additionally, `fiber_raise` on a suspended non-yielding (transferred) fiber calls `fiber_transfer_kw`, also passing `resuming_fiber = NULL`, so the same leak occurs when a transferred fiber is terminated by a raised exception. ## Reproduction Set a low map count limit, then run the attached script: ``` sudo bash -c "echo 10000 > /proc/sys/vm/max_map_count" ruby test_fiber_transfer_leak.rb ``` Restore afterwards (`sysctl vm.max_map_count` shows the default, typically 65530). ```ruby GC.disable leaked = [] count = 0 begin 10_000.times do f = Fiber.new { } # terminates immediately leaked << f # hold reference so Fiber object is not GC'd f.transfer # transfer, not resume => stack NOT released (bug) count += 1 puts "[#{count} fibers] all dead: #{leaked.none?(&:alive?)}" if count % 1000 == 0 end puts "No error — fix is applied." rescue FiberError => e puts "FiberError after #{count} fibers: #{e.message}" puts "All terminated (alive?=false): #{leaked.none?(&:alive?)}" ensure GC.enable leaked.clear GC.start end ``` Expected output on **unpatched** Ruby (vm.max_map_count=10000): ``` [1000 fibers] all dead: true [2000 fibers] all dead: true [3000 fibers] all dead: true [4000 fibers] all dead: true FiberError after 4096 fibers: can't set a guard page: Cannot allocate memory All terminated (alive?=false): false ``` Expected output on **patched** Ruby: ``` [1000 fibers] all dead: true ... [10000 fibers] all dead: true No error — fix is applied. ``` ## Fix Drop the `resuming_fiber &&` guard. After `fiber_store` returns we are executing in the caller's context — we are never on `fiber`'s stack — so releasing it is unconditionally safe. `fiber_stack_release` is already idempotent (guarded by `fiber->stack.base == NULL`), so the resume path is unaffected. ```c #ifndef COROUTINE_PTHREAD_CONTEXT if (FIBER_TERMINATED_P(fiber)) { RB_VM_LOCKING() { fiber_stack_release(fiber); } } #endif ``` Pull request with fix: https://github.com/ruby/ruby/pull/16416 -- https://bugs.ruby-lang.org/