[ruby-core:127406] [Ruby Bug#22532] pthread_mutex_lock: Invalid argument (EINVAL) / SEGV in rb_threadptr_join_list_wakeup when a fiber blocked in Thread#join under a Fiber scheduler is never resumed
Issue #22532 has been reported by chucke (Tiago Cardoso). ---------------------------------------- Bug #22532: pthread_mutex_lock: Invalid argument (EINVAL) / SEGV in rb_threadptr_join_list_wakeup when a fiber blocked in Thread#join under a Fiber scheduler is never resumed https://bugs.ruby-lang.org/issues/22532 * Author: chucke (Tiago Cardoso) * Status: Open * ruby -v: 4.0.9 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Thread#join links a struct rb_waiting_list waiter, allocated on the joining fiber's stack, into target_th->join_list. Only the rb_ensure clause (remove_from_join_list) ever unlinks it. Under a Fiber scheduler, a non-blocking fiber joining a thread suspends in Scheduler#block. If that fiber is never resumed, the waiter stays linked in two situations: The joining thread terminates. Its rb_thread_t may later be freed by GC, and the waiter now points at freed memory and a released fiber stack. The fiber itself is garbage-collected while its thread is still alive, so the waiter lives in a released fiber stack. When the target thread finishes, rb_threadptr_join_list_wakeup walks the stale entry. The scheduler of the dead joiner is nil, so it calls rb_threadptr_interrupt on freed memory. Observed symptoms: [BUG] pthread_mutex_lock: Invalid argument (EINVAL) (rb_threadptr_interrupt → rb_native_mutex_lock) SEGV in rb_fiberptr_self(join_list->fiber) NoMethodError: undefined method 'unblock' for false (macOS) Reproduced on 3.3, 3.4.9, 3.4.10, 4.0.5 and master (1f2a4bd575), on x86_64-linux, aarch64-linux and arm64-darwin. Reproduction (crashes within a few hundred iterations, every run): ```ruby class Scheduler def initialize = @main = Fiber.current def block(blocker, timeout = nil) = @main.transfer def unblock(blocker, fiber) = nil def kernel_sleep(duration = nil) = block(:sleep, duration) def io_wait(io, events, timeout) = block(:io_wait, timeout) def fiber_interrupt(fiber, exception) = nil def close = nil def fiber(&) = Fiber.new(blocking: false, &).tap(&:transfer) end 1000.times do target = nil Thread.new do Fiber.set_scheduler(Scheduler.new) Fiber.schedule { (target = Thread.new { sleep 0.001 }).join } end.join GC.start target.join end ``` This was found in a real test suite whose scheduler's io_select hook does Thread.new { IO.select(...) }.value. When an exception cut the scheduler's close short, the fiber parked in that join was abandoned. -- https://bugs.ruby-lang.org/
Issue #22532 has been updated by chucke (Tiago Cardoso). Proposed fix: https://github.com/ruby/ruby/pull/19826 ---------------------------------------- Bug #22532: pthread_mutex_lock: Invalid argument (EINVAL) / SEGV in rb_threadptr_join_list_wakeup when a fiber blocked in Thread#join under a Fiber scheduler is never resumed https://bugs.ruby-lang.org/issues/22532#change-119851 * Author: chucke (Tiago Cardoso) * Status: Open * ruby -v: 4.0.9 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Thread#join links a struct rb_waiting_list waiter, allocated on the joining fiber's stack, into target_th->join_list. Only the rb_ensure clause (remove_from_join_list) ever unlinks it. Under a Fiber scheduler, a non-blocking fiber joining a thread suspends in Scheduler#block. If that fiber is never resumed, the waiter stays linked in two situations: The joining thread terminates. Its rb_thread_t may later be freed by GC, and the waiter now points at freed memory and a released fiber stack. The fiber itself is garbage-collected while its thread is still alive, so the waiter lives in a released fiber stack. When the target thread finishes, rb_threadptr_join_list_wakeup walks the stale entry. The scheduler of the dead joiner is nil, so it calls rb_threadptr_interrupt on freed memory. Observed symptoms: [BUG] pthread_mutex_lock: Invalid argument (EINVAL) (rb_threadptr_interrupt → rb_native_mutex_lock) SEGV in rb_fiberptr_self(join_list->fiber) NoMethodError: undefined method 'unblock' for false (macOS) Reproduced on 3.3, 3.4.9, 3.4.10, 4.0.5 and master (1f2a4bd575), on x86_64-linux, aarch64-linux and arm64-darwin. Reproduction (crashes within a few hundred iterations, every run): ```ruby class Scheduler def initialize = @main = Fiber.current def block(blocker, timeout = nil) = @main.transfer def unblock(blocker, fiber) = nil def kernel_sleep(duration = nil) = block(:sleep, duration) def io_wait(io, events, timeout) = block(:io_wait, timeout) def fiber_interrupt(fiber, exception) = nil def close = nil def fiber(&) = Fiber.new(blocking: false, &).tap(&:transfer) end 1000.times do target = nil Thread.new do Fiber.set_scheduler(Scheduler.new) Fiber.schedule { (target = Thread.new { sleep 0.001 }).join } end.join GC.start target.join end ``` This was found in a real test suite whose scheduler's io_select hook does Thread.new { IO.select(...) }.value. When an exception cut the scheduler's close short, the fiber parked in that join was abandoned. -- https://bugs.ruby-lang.org/
Issue #22532 has been updated by luke-gru (Luke Gruber). Thanks for the report. I have a couple of old PRs where I was working on fixing this issue. It isn't an issue just with `Thread#join`, but also with Mutex, ConditionVariable, etc. https://github.com/ruby/ruby/pull/17207 - attempt to fix it at fiber scheduler level https://github.com/ruby/ruby/pull/17208 - similar to your fix but also attempts to fix mutexes I'll go over all the PRs soon and try to come up with what I think is the best fix. I may just merge yours :) ---------------------------------------- Bug #22532: pthread_mutex_lock: Invalid argument (EINVAL) / SEGV in rb_threadptr_join_list_wakeup when a fiber blocked in Thread#join under a Fiber scheduler is never resumed https://bugs.ruby-lang.org/issues/22532#change-119942 * Author: chucke (Tiago Cardoso) * Status: Open * Assignee: luke-gru (Luke Gruber) * ruby -v: 4.0.9 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Thread#join links a struct rb_waiting_list waiter, allocated on the joining fiber's stack, into target_th->join_list. Only the rb_ensure clause (remove_from_join_list) ever unlinks it. Under a Fiber scheduler, a non-blocking fiber joining a thread suspends in Scheduler#block. If that fiber is never resumed, the waiter stays linked in two situations: The joining thread terminates. Its rb_thread_t may later be freed by GC, and the waiter now points at freed memory and a released fiber stack. The fiber itself is garbage-collected while its thread is still alive, so the waiter lives in a released fiber stack. When the target thread finishes, rb_threadptr_join_list_wakeup walks the stale entry. The scheduler of the dead joiner is nil, so it calls rb_threadptr_interrupt on freed memory. Observed symptoms: [BUG] pthread_mutex_lock: Invalid argument (EINVAL) (rb_threadptr_interrupt → rb_native_mutex_lock) SEGV in rb_fiberptr_self(join_list->fiber) NoMethodError: undefined method 'unblock' for false (macOS) Reproduced on 3.3, 3.4.9, 3.4.10, 4.0.5 and master (1f2a4bd575), on x86_64-linux, aarch64-linux and arm64-darwin. Reproduction (crashes within a few hundred iterations, every run): ```ruby class Scheduler def initialize = @main = Fiber.current def block(blocker, timeout = nil) = @main.transfer def unblock(blocker, fiber) = nil def kernel_sleep(duration = nil) = block(:sleep, duration) def io_wait(io, events, timeout) = block(:io_wait, timeout) def fiber_interrupt(fiber, exception) = nil def close = nil def fiber(&) = Fiber.new(blocking: false, &).tap(&:transfer) end 1000.times do target = nil Thread.new do Fiber.set_scheduler(Scheduler.new) Fiber.schedule { (target = Thread.new { sleep 0.001 }).join } end.join GC.start target.join end ``` This was found in a real test suite whose scheduler's io_select hook does Thread.new { IO.select(...) }.value. When an exception cut the scheduler's close short, the fiber parked in that join was abandoned. -- https://bugs.ruby-lang.org/
Issue #22532 has been updated by luke-gru (Luke Gruber). Here's a related Fiber Scheduler issue: ```ruby class Scheduler def initialize = @main = Fiber.current def block(blocker, timeout = nil) = @main.transfer def unblock(blocker, fiber) = nil def kernel_sleep(duration = nil) = block(:sleep, duration) def io_wait(io, events, timeout) = block(:io_wait, timeout) def fiber_interrupt(fiber, exception) = nil def close = nil def fiber(&) = Fiber.new(blocking: false, &).tap(&:transfer) end 50.times do r, w = IO.pipe Thread.new do Fiber.set_scheduler(Scheduler.new) Fiber.schedule { r.wait_readable } # fiber is parked here end.join GC.start; GC.start 3.times { Fiber.new { a = [0] * 200 }.resume } r.close # wakeup w.close end puts "survived" ``` ---------------------------------------- Bug #22532: pthread_mutex_lock: Invalid argument (EINVAL) / SEGV in rb_threadptr_join_list_wakeup when a fiber blocked in Thread#join under a Fiber scheduler is never resumed https://bugs.ruby-lang.org/issues/22532#change-120101 * Author: chucke (Tiago Cardoso) * Status: Open * Assignee: luke-gru (Luke Gruber) * ruby -v: 4.0.9 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Thread#join links a struct rb_waiting_list waiter, allocated on the joining fiber's stack, into target_th->join_list. Only the rb_ensure clause (remove_from_join_list) ever unlinks it. Under a Fiber scheduler, a non-blocking fiber joining a thread suspends in Scheduler#block. If that fiber is never resumed, the waiter stays linked in two situations: The joining thread terminates. Its rb_thread_t may later be freed by GC, and the waiter now points at freed memory and a released fiber stack. The fiber itself is garbage-collected while its thread is still alive, so the waiter lives in a released fiber stack. When the target thread finishes, rb_threadptr_join_list_wakeup walks the stale entry. The scheduler of the dead joiner is nil, so it calls rb_threadptr_interrupt on freed memory. Observed symptoms: [BUG] pthread_mutex_lock: Invalid argument (EINVAL) (rb_threadptr_interrupt → rb_native_mutex_lock) SEGV in rb_fiberptr_self(join_list->fiber) NoMethodError: undefined method 'unblock' for false (macOS) Reproduced on 3.3, 3.4.9, 3.4.10, 4.0.5 and master (1f2a4bd575), on x86_64-linux, aarch64-linux and arm64-darwin. Reproduction (crashes within a few hundred iterations, every run): ```ruby class Scheduler def initialize = @main = Fiber.current def block(blocker, timeout = nil) = @main.transfer def unblock(blocker, fiber) = nil def kernel_sleep(duration = nil) = block(:sleep, duration) def io_wait(io, events, timeout) = block(:io_wait, timeout) def fiber_interrupt(fiber, exception) = nil def close = nil def fiber(&) = Fiber.new(blocking: false, &).tap(&:transfer) end 1000.times do target = nil Thread.new do Fiber.set_scheduler(Scheduler.new) Fiber.schedule { (target = Thread.new { sleep 0.001 }).join } end.join GC.start target.join end ``` This was found in a real test suite whose scheduler's io_select hook does Thread.new { IO.select(...) }.value. When an exception cut the scheduler's close short, the fiber parked in that join was abandoned. -- https://bugs.ruby-lang.org/
Issue #22532 has been updated by luke-gru (Luke Gruber). Draft PR using the new weak references API: https://github.com/ruby/ruby/pull/19996. This attempts to fix all the known issues with thread join list/fiber/mutex/condvar/queue, etc. Since the bug mentioned in this issue also appears on 4.0, if we go with the above approach we would need a fix for 4.0 as well, since 4.0 doesn't include the weakref API. In that case, your fix looks good to me. ---------------------------------------- Bug #22532: pthread_mutex_lock: Invalid argument (EINVAL) / SEGV in rb_threadptr_join_list_wakeup when a fiber blocked in Thread#join under a Fiber scheduler is never resumed https://bugs.ruby-lang.org/issues/22532#change-120109 * Author: chucke (Tiago Cardoso) * Status: Open * Assignee: luke-gru (Luke Gruber) * ruby -v: 4.0.9 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Thread#join links a struct rb_waiting_list waiter, allocated on the joining fiber's stack, into target_th->join_list. Only the rb_ensure clause (remove_from_join_list) ever unlinks it. Under a Fiber scheduler, a non-blocking fiber joining a thread suspends in Scheduler#block. If that fiber is never resumed, the waiter stays linked in two situations: The joining thread terminates. Its rb_thread_t may later be freed by GC, and the waiter now points at freed memory and a released fiber stack. The fiber itself is garbage-collected while its thread is still alive, so the waiter lives in a released fiber stack. When the target thread finishes, rb_threadptr_join_list_wakeup walks the stale entry. The scheduler of the dead joiner is nil, so it calls rb_threadptr_interrupt on freed memory. Observed symptoms: [BUG] pthread_mutex_lock: Invalid argument (EINVAL) (rb_threadptr_interrupt → rb_native_mutex_lock) SEGV in rb_fiberptr_self(join_list->fiber) NoMethodError: undefined method 'unblock' for false (macOS) Reproduced on 3.3, 3.4.9, 3.4.10, 4.0.5 and master (1f2a4bd575), on x86_64-linux, aarch64-linux and arm64-darwin. Reproduction (crashes within a few hundred iterations, every run): ```ruby class Scheduler def initialize = @main = Fiber.current def block(blocker, timeout = nil) = @main.transfer def unblock(blocker, fiber) = nil def kernel_sleep(duration = nil) = block(:sleep, duration) def io_wait(io, events, timeout) = block(:io_wait, timeout) def fiber_interrupt(fiber, exception) = nil def close = nil def fiber(&) = Fiber.new(blocking: false, &).tap(&:transfer) end 1000.times do target = nil Thread.new do Fiber.set_scheduler(Scheduler.new) Fiber.schedule { (target = Thread.new { sleep 0.001 }).join } end.join GC.start target.join end ``` This was found in a real test suite whose scheduler's io_select hook does Thread.new { IO.select(...) }.value. When an exception cut the scheduler's close short, the fiber parked in that join was abandoned. -- https://bugs.ruby-lang.org/
participants (2)
-
chucke (Tiago Cardoso) -
luke-gru (Luke Gruber)