[ruby-core:126427] [Ruby Bug#22248] `Errno::NOERROR` from `connect(2)` once a non-main Ractor has more than one Ruby thread
Issue #22248 has been reported by hornairs (Harry Brundage). ---------------------------------------- Bug #22248: `Errno::NOERROR` from `connect(2)` once a non-main Ractor has more than one Ruby thread https://bugs.ruby-lang.org/issues/22248 * Author: hornairs (Harry Brundage) * Status: Open * ruby -v: 4.0.6 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ## Summary On `x86_64-linux`, socket IO in a non-main Ractor fails spuriously as soon as that Ractor contains a second Ruby thread. The connecting thread raises `Errno::NOERROR` — errno 0 surfaced as a `SystemCallError`, whose message is literally `Success`: ``` Errno::NOERROR: Failed to open TCP connection to 127.0.0.1:38975 (Success - connect(2) for "127.0.0.1" port 38975) ``` With a raw `TCPSocket` instead of `Net::HTTP` the VM can abort outright: ``` [BUG] rb_sys_fail_path_in(io_fillbuf, fd:6 ) - errno == 0 ``` What the second thread does doesn't seem to matter. A thread that only calls `sleep`, one that only burns CPU, one doing pipe IO, and one doing socket IO all trigger it. Nothing else I varied matters: a Ractor with a single thread never fails, no matter how much socket work it does or how loaded the machine is, and a second thread in the **main** Ractor never causes it, but a second thread in a non-main Ractor does. This bug made my CI jobs flaky in roughly 10% of builds, where the "second thread" was nothing more exotic than a stub HTTP server standing in for an upstream service alongside the client under test. ## Reproduction See this gist: https://gist.github.com/airhorns/dc676921e02272cc41f8aa15da040e45 ``` ruby ractor_factors.rb interleave 200 ``` Note that the interleaving is important to more reliably trigger the bug. On a shared machine the failure rate swings enormously with host load: run in blocks, the same arrangement scored 196/200 in one build and 0/200 in the next, so blocked counts are not comparable to each other and a 0 says nothing. ## Architecture I have only reproduced this on `x86_64-linux`. On `aarch64-linux` — official `ruby:4.0`, byte-identical `revision 03b6d3f889` — roughly 1500 exchanges of the failing arrangements are clean, including 300 of the arrangement that fails 89/200 on x86, under 1- and 2-CPU quotas, with competing busy loops, and with 8 neighbour threads. `arm64-darwin` is clean too. So expect to need an x86_64 Linux host, ideally a loaded one. ## Environment ``` ruby 4.0.6 (2026-07-14 revision 03b6d3f889) +PRISM [x86_64-linux] Socket.tcp_fast_fallback = true (default; false reproduces identically) RUBY_MAX_CPU unset (=1 does not help) shared CI host, nproc 16, 1-minute load average 3.1-5.9 during the run ``` ## Relation to Bug #21195 [Bug #21195](https://bugs.ruby-lang.org/issues/21195) is the same shape — errno lost around `io_internal_wait`, fixed for 3.3 and 3.4 in 2025 — and the `errno == 0` assertion text is identical. This is a live path in 4.0.6, reached through ordinary socket connect and read, with or without Happy Eyeballs, so I am filing it separately rather than commenting there. I could not find an existing report for the Ractor variant. -- https://bugs.ruby-lang.org/
Issue #22248 has been updated by hornairs (Harry Brundage). ## What it is not * **Not `Net::HTTP` specific.** Raw `TCPSocket` fails too, and is the case that aborts the VM. * **Not "IO in a Ractor is unsupported".** A single-threaded non-main Ractor drove the identical request 200 times without a failure, in the same run, in the same load window — including one arrangement that owns the listening socket and calls `accept(2)` inside the Ractor, and one that owns both ends of the connection. * **Not the second thread's IO.** A neighbour that never touches a file descriptor fails at 89/200. * **Not CPU contention as such.** The same CPU-burning thread placed in the main Ractor instead of the non-main one gives 0/200, in the same run. * **Not Happy Eyeballs v2.** With `Socket.tcp_fast_fallback = false` the pattern reproduces unchanged: 49, 49, 32, 20, 3 bad out of 66 for the five two-thread arrangements, 0 for all six others. * **Not M:N native-thread migration** — see below. * **Not a native extension.** The reproduction is pure stdlib. I first saw this through a Rust extension's Ruby HTTP transport, but the extension is absent from the script. * **Not the connection failing for a real reason.** The stub server is listening on the port the same iteration just obtained from `TCPServer#addr`, and `connect(2)` reports *success* while raising. ## Native-thread migration is ruled out The obvious guess is that `errno`, being per-native-thread, is read on a different native thread than the one that set it — Ruby threads in non-main Ractors being scheduled M:N. Migration is directly observable: `/proc/thread-self` resolves to `<pid>/task/<tid>` for the calling native thread and is readable from a non-main Ractor (`Fiddle.dlopen` is not — it raises `Ractor::UnsafeError`). `gettid_migration.rb` is attached in the gist. Three observations kill the theory: * On the `x86_64` host where the bug fires constantly, migration was observed **0 of 6 rounds** in every setting, with both a sleeping and a CPU-burning neighbour, in the same build as the failures above. * On `aarch64-linux` migration happens readily — 3 of 4 rounds for a two-thread non-main Ractor, tids moving e.g. `16 → 17` and `17 → 18 → 19` — and the bug never reproduces there at all. * `RUBY_MAX_CPU=1` leaves one native thread to migrate between, so migration is impossible by construction (and observably stops on arm64, where it otherwise happens constantly) — the failure persists anyway, 9/200. So migration and the failure are, if anything, anti-correlated. Whatever loses errno here, it is not a thread waking on a different native thread. ---------------------------------------- Bug #22248: `Errno::NOERROR` from `connect(2)` once a non-main Ractor has more than one Ruby thread https://bugs.ruby-lang.org/issues/22248#change-118572 * Author: hornairs (Harry Brundage) * Status: Open * ruby -v: 4.0.6 * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ## Summary On `x86_64-linux`, socket IO in a non-main Ractor fails spuriously as soon as that Ractor contains a second Ruby thread. The connecting thread raises `Errno::NOERROR` — errno 0 surfaced as a `SystemCallError`, whose message is literally `Success`: ``` Errno::NOERROR: Failed to open TCP connection to 127.0.0.1:38975 (Success - connect(2) for "127.0.0.1" port 38975) ``` With a raw `TCPSocket` instead of `Net::HTTP` the VM can abort outright: ``` [BUG] rb_sys_fail_path_in(io_fillbuf, fd:6 ) - errno == 0 ``` What the second thread does doesn't seem to matter. A thread that only calls `sleep`, one that only burns CPU, one doing pipe IO, and one doing socket IO all trigger it. Nothing else I varied matters: a Ractor with a single thread never fails, no matter how much socket work it does or how loaded the machine is, and a second thread in the **main** Ractor never causes it, but a second thread in a non-main Ractor does. This bug made my CI jobs flaky in roughly 10% of builds, where the "second thread" was nothing more exotic than a stub HTTP server standing in for an upstream service alongside the client under test. ## Reproduction See this gist: https://gist.github.com/airhorns/dc676921e02272cc41f8aa15da040e45 ``` ruby ractor_factors.rb interleave 200 ``` Note that the interleaving is important to more reliably trigger the bug. On a shared machine the failure rate swings enormously with host load: run in blocks, the same arrangement scored 196/200 in one build and 0/200 in the next, so blocked counts are not comparable to each other and a 0 says nothing. ## Architecture I have only reproduced this on `x86_64-linux`. On `aarch64-linux` — official `ruby:4.0`, byte-identical `revision 03b6d3f889` — roughly 1500 exchanges of the failing arrangements are clean, including 300 of the arrangement that fails 89/200 on x86, under 1- and 2-CPU quotas, with competing busy loops, and with 8 neighbour threads. `arm64-darwin` is clean too. So expect to need an x86_64 Linux host, ideally a loaded one. ## Environment ``` ruby 4.0.6 (2026-07-14 revision 03b6d3f889) +PRISM [x86_64-linux] Socket.tcp_fast_fallback = true (default; false reproduces identically) RUBY_MAX_CPU unset (=1 does not help) shared CI host, nproc 16, 1-minute load average 3.1-5.9 during the run ``` ## Relation to Bug #21195 [Bug #21195](https://bugs.ruby-lang.org/issues/21195) is the same shape — errno lost around `io_internal_wait`, fixed for 3.3 and 3.4 in 2025 — and the `errno == 0` assertion text is identical. This is a live path in 4.0.6, reached through ordinary socket connect and read, with or without Happy Eyeballs, so I am filing it separately rather than commenting there. I could not find an existing report for the Ractor variant. -- https://bugs.ruby-lang.org/
participants (1)
-
hornairs (Harry Brundage)