Issue #22216 has been updated by ioquatix (Samuel Williams). @headius Just thinking out loud...
Switching your code to a fiber shouldn't suddenly make them disappear, or should it?
I'd gently push back on this, because switching to a *thread* already does exactly that today, and has for 20 years: ```ruby def outer "abc" =~ /(b)/ before = $1 blk = proc { "xyz" =~ /(y)/ } # lexically owned by `outer` Thread.new { blk.call }.join # invoked on another thread [before, $1] end outer # => ["b", "b"] -- the block's match never reaches outer's $1 ``` So "these variables live in the nearest method frame, regardless of how the block is invoked" isn't quite the rule we have now. It's really *method-frame + thread*: the block shares `outer`'s frame lexically, but thread-locality overrides that. Execution context already changes svar visibility, and nobody considers the thread case a bug. If `Enumerator` used a `Thread` internally instead of a `Fiber`, we'd have the same isolation — so Fiber-local doesn't introduce a novel exception, it just makes fibers behave the way threads already do. To me that's *more* consistent, not less. It also seems like "method scope" isn't really a single well-defined thing here. A lazy enumerator is, by definition, evaluated somewhere other than where it's written, so its block's svar scope is genuinely ambiguous: ```ruby def process(section_header, pairs) section_header =~ /^\[(\w+)\]$/ # $1 in process's frame == "database" section = $1 result = [] loop do k, v = pairs.next # fiber-backed pull result << "#{section}.#{k} = #{v}" result << " (still in section #{$1})" # which $1 is this? end result end pairs = lines.lazy.map { |l| l =~ /(\w+)=(\w+)/; [$1, $2] } # block defined here process("[database]", pairs) p $1, $2 # => "port", "5432" (!) ``` What is the scope of `$1`/`$2` inside that block? Does it belong to where `pairs` is defined, or where `pairs.next` is called? The answer today is neither of the obvious ones: the block's matches land in its *lexical* definition site (the top-level frame), so after the call `$1`/`$2` at the top level are `"port"`/`"5432"` — the last match performed deep inside `process`'s loop, across a fiber boundary. Meanwhile `process`'s own `$1` stayed `"database"` the whole time. So the observable result depends on where the block was *written*, not where it ran, which most people wouldn't guess. And if that same block were driven by a `Thread` instead of a `Fiber`, the top-level `$1` wouldn't be clobbered at all — so this is a fiber-specific quirk, not a universal "svars live in the method frame" guarantee. The fiber itself carries a well-defined scope for the frames on its own stack; the only ambiguity is about frames it closed over that live *outside* its stack, and I don't think those are meaningfully "parented" to the consuming method at all. Like you, I don't know the single right answer here.
I don't have any answers other than eliminating these hidden variables altogether. [...] Eliminating them is probably the best choice long-term if safe concurrency is a future priority.
I basically agree — long term, removing them is the honest fix. But that's not something we can do without longer term planning, and in the meantime we still have to pick *some* behavior. My worry with `transparent:` is that it doubles down on "fibers don't isolate" precisely when threads already established the opposite precedent, and it does so to preserve `e.next; $1`, a pattern already conceded to be non-idiomatic. So I'd suggest plain Fiber-local (or EC local more precisely) as the near-term default: it's the simplest rule, it matches how threads already behave, and it doesn't require the non-transferable-fiber restriction (which conflicts with schedulers that rely on `#transfer`). ---------------------------------------- Bug #22216: Special variables (ex. Regexp backref and IO lastline) are thread-unsafe in some cases, incompatible with Ractor https://bugs.ruby-lang.org/issues/22216#change-119069 * Author: headius (Charles Nutter) * Status: Open * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- ## Problem Several Regexp-matching methods currently write (and sometimes read) the implicit "backref" `$~` variable in the local frame (and related variables like `$'`). Several IO methods read or write the "last line" `$_` variable in the same way. In both cases, the result is a mutable object, which makes these variables already problematic for ractors. Making matters worse, the frame might be shared if a proc is captured and used across threads or ractors, and there's no static way to inspect a piece of code to know if it expects to read or write these variables. Where procs can be rejected by a proc for accessing captured state, there's no such check possible for these variables. All of these facts make the backref and lastline variables fundamentally incompatible with Ractor. ## Possible remedies A wholesale removal of these variables would solve the problem, but there's a lot of code that depends on them... much of that code without even realizing it, since they might not access the variables directly. In some cases, the dependencies are internal and part of the behavior of core methods. Hard errors when using methods that read or write these variables would avoid introducing threading problems into a Ractor, but would also break a large number of commonly-used methods. There have been experiments to make these variables both frame and thread-local, but they have never been made standard. Updates to backref and lastline are visible across threads and already can lead to concurrency issues even on CRuby. Deprecating the implicit behavior and making it opt-in (or opt-out?), perhaps with keyword arguments or file pragmas, might be a halfway measure. It would probably not be an easy transition. I don't know the right path forward, but I believe this issue needs to be discussed. ## JRuby perspective We continue to mimic CRuby behavior, which has led to our users occasionally running into issues when a proc accesses these variables across threads. Our recommendation: "don't do that". We also have had our frustrations optimizing around these variables, since they implicitly require access across calls. Because we cannot statically detect when they will be used, we essentially treat all method names that might *potentially* access them as deopt triggers. It's not ideal. I'd like to hear ideas for how to make these variables less "magic", less implicit and easier to deal with across calls (and across threads/ractors). -- https://bugs.ruby-lang.org/