[ruby-core:123923] [Ruby Bug#21716] YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop)
Issue #21716 has been reported by angelim@angelim.com.br (Alexandre Angelim). ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Open * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... `` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
Issue #21716 has been updated by ivoanjo (Ivo Anjo). 👋 I'll add that we have some limited crash telemetry at Datadog and I've seen what might be the same issue (YJIT failing somewhere inside `branch_stub_hit`) on both Ruby 3.3 and 3.4 (maybe 3.2 but unclear? if the same) on both x86-64 and arm64. If it helps, I can provide some of the native stack traces we're seeing. ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716#change-115324 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Open * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... `` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
Issue #21716 has been updated by alanwu (Alan Wu). Description updated There is two separate issues here, YJIT panicking, and the crash reporter looping during stack tracing. During development, I've seen the the crash reporter go into infinite loop on aarch64-dwarin, too. The rust unwinder seems to be working fine, and the infinite loop seems to be from ruby's own DWARF parser and unwinder in addr2line.c. Can you confirm that before the repeated stack trace lines there is a line that says "C level backtrace information"? ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716#change-115327 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Open * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... ``` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
Issue #21716 has been updated by nonster (Nony Dutton). alanwu (Alan Wu) wrote in #note-2:
The rust unwinder seems to be working fine, and the infinite loop seems to be from ruby's own DWARF parser and unwinder in addr2line.c. Can you confirm that before the repeated stack trace lines there is a line that says "C level backtrace information"?
Yep, here's more context: ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5 1: core::panicking::panic_fmt at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14 2: core::panicking::panic_display at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:150:5 3: core::panicking::panic_str at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:134:5 4: core::option::expect_failed at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/option.rs:2025:5 5: core::option::Option<T>::expect at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/option.rs:913:21 6: yjit::core::<impl yjit::codegen::JITState>::into_block::{{closure}} at /workspace/ruby-3.4.7/yjit/src/core.rs:2403:27 7: core::iter::adapters::map::map_try_fold::{{closure}} at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:91:28 8: core::iter::traits::iterator::Iterator::try_fold at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2304:21 9: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:117:9 10: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/in_place_collect.rs:257:13 11: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/in_place_collect.rs:181:19 12: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/vec/mod.rs:2712:9 13: core::iter::traits::iterator::Iterator::collect at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:1896:9 14: <alloc::boxed::Box<[I]> as core::iter::traits::collect::FromIterator<I>>::from_iter at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:2001:9 15: core::iter::traits::iterator::Iterator::collect at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:1896:9 16: yjit::core::<impl yjit::codegen::JITState>::into_block at /workspace/ruby-3.4.7/yjit/src/core.rs:2405:16 17: yjit::codegen::gen_single_block at /workspace/ruby-3.4.7/yjit/src/codegen.rs:1457:8 18: yjit::core::gen_block_series_body at /workspace/ruby-3.4.7/yjit/src/core.rs:3088:23 19: yjit::core::gen_block_series at /workspace/ruby-3.4.7/yjit/src/core.rs:3066:18 20: yjit::core::branch_stub_hit_body at /workspace/ruby-3.4.7/yjit/src/core.rs:3604:17 21: yjit::core::branch_stub_hit::{{closure}}::{{closure}} at /workspace/ruby-3.4.7/yjit/src/core.rs:3493:36 22: yjit::stats::with_compile_time at /workspace/ruby-3.4.7/yjit/src/stats.rs:1084:15 23: yjit::core::branch_stub_hit::{{closure}} at /workspace/ruby-3.4.7/yjit/src/core.rs:3493:13 24: std::panicking::try::do_call at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40 25: std::panicking::try at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19 26: std::panic::catch_unwind at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14 27: yjit::cruby::with_vm_lock at /workspace/ruby-3.4.7/yjit/src/cruby.rs:688:21 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 29: <unknown> note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] '-- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:593 c:0044 p:0062 s:0230 e:000229 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:363 c:0043 p:0003 s:0222 e:000221 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:401 c:0042 p:0025 s:0218 e:000217 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:321 c:0041 p:0029 s:0212 e:000211 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http.rb:2430 c:0040 p:0050 s:0204 e:000203 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http.rb:2384 c:0039 p:0033 s:0196 e:000195 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 c:0038 p:0029 s:0189 e:000188 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry/hub.rb:138 [FINISH] c:0037 p:0022 s:0179 e:000178 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry-ruby.rb:536 c:0036 p:0052 s:0173 e:000172 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:37 c:0035 p:0006 s:0166 e:000165 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:78 c:0034 p:0025 s:0162 e:000161 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http.rb:1632 c:0033 p:0023 s:0158 e:000157 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 c:0031 p:0018 s:0145 e:000144 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:44 c:0030 p:0024 s:0140 e:000139 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/api/instance.rb:47 c:0029 p:0005 s:0135 e:000134 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:65 c:0028 p:0020 s:0131 e:000130 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/api/endpoint.rb:24 [FINISH] c:0027 p:0055 s:0126 e:000125 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:98 c:0026 p:0029 s:0118 e:000117 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:47 c:0025 p:0028 s:0112 e:000111 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:64 c:0024 p:0006 s:0107 e:000106 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:34 c:0023 p:0014 s:0102 e:000101 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/client.rb:27 c:0022 p:0006 s:0092 e:000091 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:33 c:0021 p:0022 s:0087 e:000086 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:148 [FINISH] c:0020 p:---- s:0081 e:000080 IFUNC c:0019 p:---- s:0078 e:000077 IFUNC c:0018 p:---- s:0075 e:000074 CFUNC :<< c:0017 p:---- s:0072 E:001440 IFUNC c:0016 p:---- s:0069 E:0013e0 CFUNC :each c:0015 p:---- s:0066 e:000065 CFUNC :each c:0014 p:---- s:0063 e:000062 CFUNC :each c:0013 p:---- s:0060 E:0013c0 IFUNC c:0012 p:---- s:0057 E:0013a0 CFUNC :each c:0011 p:---- s:0054 e:000053 CFUNC :each c:0010 p:---- s:0051 E:001380 CFUNC :to_a c:0009 p:0038 s:0047 E:0009e0 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168 c:0008 p:0015 s:0039 e:000038 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/writer.rb:90 c:0007 p:0007 s:0032 e:000031 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/writer.rb:160 [FINISH] c:0006 p:0051 s:0027 e:000023 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:55 c:0005 p:0004 s:0018 e:000017 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:114 [FINISH] c:0004 p:0017 s:0015 e:000014 METHOD <internal:kernel>:168 c:0003 p:0004 s:0010 e:000009 METHOD /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:113 c:0002 p:0003 s:0006 e:000005 BLOCK /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:73 [FINISH] c:0001 p:---- s:0003 e:000002 DUMMY [FINISH] '-- Ruby level backtrace information ---------------------------------------- /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:73:in 'block (2 levels) in start' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:113:in 'perform' <internal:kernel>:168:in 'loop' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:114:in 'block in perform' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/workers.rb:55:in 'callback_traces' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/writer.rb:160:in 'block in start_worker' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/writer.rb:90:in 'send_spans' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'send_traces' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'to_a' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'each' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'each' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'each' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'each' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in 'each' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:168:in '<<' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/traces.rb:148:in 'block in send_traces' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:33:in 'send_traces_payload' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/client.rb:27:in 'send_request' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:34:in 'block in send_traces_payload' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:64:in 'send_traces' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:47:in 'send_traces' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:98:in 'call' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/api/endpoint.rb:24:in 'call' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/tracing/transport/http/traces.rb:65:in 'block in send_traces' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/api/instance.rb:47:in 'call' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:44:in 'call' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77:in 'post' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43:in 'open' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http.rb:1632:in 'start' /var/zendesk/bundle/ruby/3.4.0/gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:78:in 'block in post' /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:37:in 'request' /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry-ruby.rb:536:in 'with_child_span' /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry/hub.rb:138:in 'with_child_span' /var/zendesk/bundle/ruby/3.4.0/gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44:in 'block in request' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http.rb:2384:in 'request' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http.rb:2430:in 'transport_request' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:321:in 'reading_body' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:401:in 'body' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:363:in 'read_body' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:593:in 'read_body_0' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:588:in 'inflater' /var/zendesk/bundle/ruby/3.4.0/gems/net-http-0.6.0/lib/net/http/response.rb:604:in 'block in read_body_0' '-- Threading information --------------------------------------------------- Total ractor count: 1 Ruby thread count for this ractor: 9 '-- C level backtrace information ------------------------------------------- /usr/lib/libruby.so.3.4(fprintf+0x0) [0xf0c8b9f94d80] /workspace/ruby-3.4.7/vm_dump.c:823 /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch>, alloc::alloc::Global>, alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, core::iter::adapters::map::map_try_fold::{closure_env#0}<alloc::rc::Rc<yjit::core::PendingBranch>, core::ptr::non_null::NonNull<yjit::core::Branch>, alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, core::result::Result<alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, !>, yjit::core::{impl#19}::into_block::{closure_env#0}, alloc::vec::in_place_collect::write_in_place_with_drop::{closure_env#0}<core::ptr::non_null::NonNull<yjit::core::Branch>>>, core::result::Result<alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, !>>) /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2303 /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch>, alloc::alloc::Global>, alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, core::iter::adapters::map::map_try_fold::{closure_env#0}<alloc::rc::Rc<yjit::core::PendingBranch>, core::ptr::non_null::NonNull<yjit::core::Branch>, alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, core::result::Result<alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, !>, yjit::core::{impl#19}::into_block::{closure_env#0}, alloc::vec::in_place_collect::write_in_place_with_drop::{closure_env#0}<core::ptr::non_null::NonNull<yjit::core::Branch>>>, core::result::Result<alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, !>>) /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2303 /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch>, alloc::alloc::Global>, alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, core::iter::adapters::map::map_try_fold::{closure_env#0}<alloc::rc::Rc<yjit::core::PendingBranch>, core::ptr::non_null::NonNull<yjit::core::Branch>, alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, core::result::Result<alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, !>, yjit::core::{impl#19}::into_block::{closure_env#0}, alloc::vec::in_place_collect::write_in_place_with_drop::{closure_env#0}<core::ptr::non_null::NonNull<yjit::core::Branch>>>, core::result::Result<alloc::vec::in_place_drop::InPlaceDrop<core::ptr::non_null::NonNull<yjit::core::Branch>>, !>>) /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/traits/iterator.rs:2303 # Continues looping from here. `` ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716#change-115363 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Open * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... ``` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
Issue #21716 has been updated by k0kubun (Takashi Kokubun). Assignee set to jit ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716#change-115368 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Open * Assignee: jit * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... ``` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
Issue #21716 has been updated by alanwu (Alan Wu). Status changed from Open to Feedback Unfortunately, with the information available, there is not much we can do on our end. The PendingBranchRefs in question are transient reference counted objects in YJIT. They have clear release points that should have been reached before the point of the panic. The loop in the crash reporter is also odd, because it seems to be while traversing a singly linked list that is only appended to in one spot in the addr2line.c. This is looking like some sort of heap/stack corruption that is tripping over this point in the code. We changed the panic message to include the reference counts in 3.4.8, so that can show clearly if a memory corruption took place. No hints as to who did the memory corruption at all, though. It can be any native code in the process, including C extensions in the app. ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716#change-115835 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Feedback * Assignee: jit * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... ``` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
Issue #21716 has been updated by nonster (Nony Dutton). alanwu (Alan Wu) wrote in #note-5:
Unfortunately, with the information available, there is not much we can do on our end.
The PendingBranchRefs in question are transient reference counted objects in YJIT. They have clear release points that should have been reached before the point of the panic. The loop in the crash reporter is also odd, because it seems to be while traversing a singly linked list that is only appended to in one spot in the addr2line.c. This is looking like some sort of heap/stack corruption that is tripping over this point in the code.
We changed the panic message to include the reference counts in 3.4.8, so that can show clearly if a memory corruption took place. No hints as to who did the memory corruption at all, though. It can be any native code in the process, including C extensions in the app.
Sadly, we don't have much more information yet. We have one additional piece of the puzzle just in case it helps someone in the future. We're using `resque` along with `resque-pool` as our background job processor. We had been previously enabling `YJIT` with `RUBY_YJIT_ENABLE=true`. We changed it to lazily enable `YJIT` with: ```ruby Resque::Pool.after_prefork do |_job| RubyVM::YJIT.enable if defined?(RubyVM::YJIT) end ``` And we haven't seen the issue since (we used to see it about once a week). We'll try investigating what might be going wrong before the prefork that can lead to this. ---------------------------------------- Bug #21716: YJIT panic: "all PendingBranchRefs should be unique" in net/http response reading (infinite unwind loop) https://bugs.ruby-lang.org/issues/21716#change-116192 * Author: angelim@angelim.com.br (Alexandre Angelim) * Status: Feedback * Assignee: jit * ruby -v: ruby 3.4.7 * Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. Key Observations: The Panic: YJIT fails with all PendingBranchRefs should be unique when ready to construct a Block. The Loop: After the panic, the process fails to exit and instead logs `rust_begin_unwind` and `try_fold` iterator errors repeatedly, filling logs for hours. System Configuration Ruby Version: ruby 3.4.7 (2025-10-08 revision 7a5688e2a2) +YJIT +PRISM [aarch64-linux] Environment: Linux aarch64 (AWS Graviton/ARM) Reproduction Context While we cannot provide a standalone reproduction script yet, the crash happens intermittently during a Net::HTTP request initiated by datadog-ruby (v2.17.0) to flush traces to a Unix socket, which is wrapped by sentry-ruby (v5.26.0). Ruby Location: `/gems/net-http-0.6.0/lib/net/http/response.rb:604` Rust Location: `./yjit/src/core.rs:2403:27` Backtrace & Logs ``` ruby: YJIT has panicked. More info to follow... thread '<unnamed>' panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 stack backtrace: 0: rust_begin_unwind at /rustc/.../library/std/src/panicking.rs:578:5 ... 28: yjit::core::branch_stub_hit at /workspace/ruby-3.4.7/yjit/src/core.rs:3492:9 /gems/net-http-0.6.0/lib/net/http/response.rb:604: [BUG] YJIT: panicked at 'all PendingBranchRefs should be unique when ready to construct a Block', ./yjit/src/core.rs:2403:27 -- Control frame information ----------------------------------------------- c:0047 p:0049 s:0248 e:000247 BLOCK /gems/net-http-0.6.0/lib/net/http/response.rb:604 c:0046 p:0098 s:0243 e:000242 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:588 c:0045 p:0004 s:0235 e:000234 METHOD /gems/net-http-0.6.0/lib/net/http/response.rb:593 ... c:0039 p:0033 s:0196 e:000195 BLOCK /gems/sentry-ruby-5.26.0/lib/sentry/net/http.rb:44 ... c:0033 p:0023 s:0158 e:000157 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/unix_socket.rb:43 c:0032 p:0066 s:0152 e:000151 METHOD /gems/datadog-2.17.0/lib/datadog/core/transport/http/adapters/net.rb:77 ``` The stack suggests the crash happens during `branch_stub_hit` resolution. The presence of sentry-ruby and datadog suggests complex interaction with Net::HTTP patching. The infinite loop of logs following the crash looks like this (repeated): ``` /usr/lib/libruby.so.3.4(try_fold<alloc::vec::into_iter::IntoIter<alloc::rc::Rc<yjit::core::PendingBranch> ... ``` This suggests the panic handler itself is tripping over the corrupted `PendingBranch` state. We encountered a YJIT panic in production that caused the process to enter an infinite logging loop for several hours. The panic occurs within `yjit/src/core.rs` when handling branch references. The trigger appears to be in `net/http`, specifically when reading a response body (possibly involving chunked encoding or zlib) inside a Datadog trace transport flush. -- https://bugs.ruby-lang.org/
participants (5)
-
alanwu (Alan Wu) -
angelim＠angelim.com.br (Alexandre Angelim)
-
ivoanjo (Ivo Anjo)
-
k0kubun (Takashi Kokubun)
-
nonster (Nony Dutton)