[ruby-core:126598] [Ruby Feature#22300] `Ractor.check_isolation`: report Ractor isolation violations as warnings instead of raising
Issue #22300 has been reported by ufuk (Ufuk Kayserilioglu). ---------------------------------------- Feature #22300: `Ractor.check_isolation`: report Ractor isolation violations as warnings instead of raising https://bugs.ruby-lang.org/issues/22300 * Author: ufuk (Ufuk Kayserilioglu) * Status: Open * Target version: 4.1 ---------------------------------------- ## Summary We are proposing a `Ractor.check_isolation { ... }` method which runs the given block in a special non-main Ractor. Inside that Ractor, supported operations that would fail because of Ractor isolation report a `:ractor_isolation` category warning, and then continue, instead of raising an isolation error. The special Ractor that is used to run the `Ractor.check_isolation` block is a real non-main Ractor. For example, `Ractor.main?` returns `false` and `Ractor.current` returns a genuine Ractor with its own default port. This is possible by ensuring that no other Ractor executes Ruby concurrently with this special Ractor. That exclusivity makes the "warn and continue" contract safe. ### Examples A trivial code example: ```ruby class Config @settings = {} # not shareable @values = [] # also not shareable def self.settings = @settings def self.values = @values end # Today: the first violation ends the run. Ractor.new { Config.settings }.value #=> Ractor::RemoteError # whose cause is Ractor::IsolationError: can not # get unshareable values from instance variables of # classes/modules from non-main Ractors # With `check_isolation`: the violation is a warning, and the run continues. Ractor.check_isolation do Config.settings # warning, not exception Config.values # ...and the next one is also reported Ractor.main? #=> false end ``` The next example is a test suite that can collect the supported isolation violations in a single run: ```ruby # test_helper.rb ISOLATION_WARNINGS = Thread::Queue.new Warning.singleton_class.prepend(Module.new do def warn(msg, category: nil) if category == :ractor_isolation && !Thread.current[:capturing_isolation_warning] Thread.current[:capturing_isolation_warning] = true begin ISOLATION_WARNINGS << Ractor.make_shareable(msg) ensure Thread.current[:capturing_isolation_warning] = false end return end super end end) class ActiveSupport::TestCase def run(...) Ractor.check_isolation { super } end end ``` ## 1. Motivation ### 1.1 An isolation error stops the run at the first violation Migrating a large codebase to Ractor-safety is difficult, since running it inside a real Ractor gives only one violation per run, which needs to be made Ractor-safe before you can observe the next violation. A large codebase has thousands, if not tens of thousands of these Ractor isolation violations that need to be identified. Our team at Shopify started an effort to make Rails Ractor-safe earlier this year, and as part of that work we needed the full list of isolation problems and not just the first item. To tackle this problem, we used an agent assisted loop against a scaffolded Rails application which served 165 endpoints inside a Ractor. We directed the agent loop to identify and fix the source of the `Ractor::IsolationError`, and run the loop again, until all endpoints could successfully run. While this approach did end up getting us a list of isolation violations, it proved to be an cumbersome task even with agent assistance. Instead, when we built the original prototype of `Ractor.check_isolation`, our initial run reported 50,995 warnings, of which 353 were unique in a matter of minutes with no agent assistance at all. A later run against Rails `main` reported 58,315 warnings, of which 419 were unique. The same work needs 419 build/fix/rebuild cycles (performed by agents or humans) to collect the same information. ### 1.2 You cannot put an existing test suite inside a real Ractor The obvious alternative to the above is to put a `Ractor.new { ... }` around each test in the test suite of the codebase. This approach, while attractive, does not work on real codebases, for three main reasons: 1. The test code must itself be Ractor-safe. Making the tests safe is often harder than making the application safe. Most test helpers hold mutable global state on purpose. 2. The block of `Ractor.new` must be isolated. It cannot close over the test case, the fixtures or the assertion state. 3. Non-shareable arguments and return values are copied. However, copying is not possible for some objects, for example a `Proc`, and the run aborts. `Ractor.check_isolation` removes all three obstacles. The block is not isolated. Arguments, the return value and outgoing message payloads pass by reference. Moreover, any isolation violation warning reported from the test file sources can easily be discarded as irrelevant. ### 1.3 There is no way to keep a codebase Ractor-safe Given the above point, this is the argument that matters most in the long term, in our opinion. A codebase that is Ractor-safe today will not stay Ractor-safe in the future. As the codebase evolves, one new lazy memoization, or one new unfrozen constant in a dependency, will reintroduce a violation. A normal test suite cannot detect the change, because the test suite does not run in a Ractor (as explain in the previous point). A gem can therefore break its downstream users without any signal, and the failure would only appear in production. `Ractor.check_isolation` turns Ractor safety into a property that CI can measure. A gem or an application can wrap its tests in the block and fails the build when the count of unique warnings increases. In essence, our argument is that this proposal gives the ecosystem two things it does not have today: - a burn-down list, to make a codebase Ractor-safe; - a ratchet, to keep it Ractor-safe. ## 2. Proposal ```ruby Ractor.check_isolation(*args, name: nil) { |*args| ... } # -> the value of the block ``` `Ractor.check_isolation` method: - runs the block in a new, real, and special non-main Ractor; - does not isolate the block, and passes arguments by reference; - reports each supported isolation violation as a `:ractor_isolation` category warning, and continues without raising; - unlike an ordinary Ractor, returns a non-shareable block value without copying it, preserving object identity. The reported violations match the constructs that would have raised a `Ractor::IsolationError` under a normal non-main `Ractor`. They include, but are not limited to: - instance variables and class variables of classes and modules; - constants that hold non-shareable objects; - global variables; - `Ractor.make_shareable`, `Ractor.shareable_proc` and `Ractor.shareable_lambda` failures; - calls to Ractor-unsafe C methods; - `fork` from a non-main Ractor; - defining or undefining finalizers on objects owned by another Ractor; - a call to a method that `define_method` defined with a non-shareable `Proc`; - isolation of the check block when it captures outer variables or uses `yield`. The user suppresses isolation-violation warnings with `Warning[:ractor_isolation] = false` or with `-W:no-ractor_isolation`. The exclusivity of the special Ractor that run the block is a precondition of the method, not a mode that the user should select. The current patch relies on `RUBY_RACTOR_EXCLUSIVE=1` and warns when it is unavailable. However, for the final solution we propose that `Ractor.check_isolation` instead establish exclusivity itself and raise when it cannot. Section 4.3 discusses the implementation constraints regarding this. The exclusive mode mentioned here is not a stop-the-world barrier. A blocking operation still hands the run slot over, so that, for example, work that the block correctly dispatches to the main Ractor completes, and the program does not deadlock. ## 3. Implementation Our current patch is on this branch: https://github.com/Shopify/ruby/compare/master...Shopify:ruby:ractor-check-i... The branch exposes the design described here under the name `Ractor.check_isolation`. Main points: - A new warning category, `RB_WARN_CATEGORY_RACTOR_ISOLATION`, registered as `:ractor_isolation`. - A flag on the special `rb_ractor_t`. A helper, `rb_ractor_isolation_violation()`, replaces the raise at many of the covered violation sites. The helper warns when the current Ractor has the flag set, and raises otherwise. - `ractor_create0()` creates and marks the special Ractor. When `rb_thread_create_ractor()` starts it, `thread_create_core()` skips `rb_proc_isolate_bang()` and retains the argument array by reference. `rb_proc_check_isolation_warn()` reports the `Proc`-isolation violations that `Ractor.new` would raise, and leaves the closure intact. - `ractor_prepare_payload()` passes a non-shareable message payload by reference, with a warning, instead of copying it. - The bmethod call paths warn instead of raising for a cross-Ractor call to a method defined with a non-shareable `Proc`. - On builds with M:N scheduling, `ruby_mn_threads_params()` enables it and sets `max_cpu = 1` under `RUBY_RACTOR_EXCLUSIVE=1`. - The first call switches the VM into multi-ractor mode. This switch is one-way today. Two costs with the current patch are worth stating explicitly: 1. The VM keeps the multi-ractor overhead for the rest of the process, and 2. the checks sit on paths that a production process executes. We consider these costs acceptable since this feature is a development and CI tool, and should not be enabled in production mode. ## 4. Open questions We have used this feature for four months on Rails and on two large production applications at Shopify. The following questions come out of that use. We do not have a strong position on most of them. 1. **Is `warn` the right surface?**: `Warning.warn` was the cheapest thing to build, but it may be the weakest part of the design, for the following reasons: - The volume is high. In one Shopify application, the run produced more warning text than the 6 GB memory limit of the CI worker, so it was killed. The consumers want a deduplication by site, not a stream of strings. - A string throws the structure away. The consumers want the class, the variable name, the reference chain and the location as data, rather than parsing whatever information is available from the message. - Re-entrancy is a real hazard. Any code in the handler can itself trigger a violation. We hit an infinite recursion this way, and we also hit a stack overflow because formatting the message called `to_s` on a proxy module. - `Warning.warn` is global process state. A warning that the special Ractor reports is awkward to collect in the caller. The alternatives are a callback, or a reporter object: ```ruby Ractor.check_isolation(on_violation: ->(violation) { ... }) { ... } Ractor.check_isolation(reporter: MyReporter.new) { ... } ``` A `violation` object can carry the class, the name, the kind, the reference chain and the backtrace. The warning category can stay as the default when the caller gives no callback. 2. **Should the checks be a build-time option?** The checks add work to hot paths. An alternative is to compile them only into a build that configures them, in the same way as `RGENGC_CHECK_MODE`. This trades convenience for speed. We do not know which side upstream prefers, but considering that so few Ruby developers know how to build a special version of Ruby, we think this will stop the widespread adoption of the feature. 3. **How does the method establish exclusivity?** The current patch reads a boot flag, `RUBY_RACTOR_EXCLUSIVE=1`, and warns once when the flag is absent. We do not propose that as the interface for the final state. The constraint for why we have the boot flag is technical and we welcome suggestions to removing it. Two mechanisms in `thread_sched.c` decide the outcome: - `vm->ractor.sched.max_cpu` caps the number of shared native threads. The VM reads it when an NT rejoins the pool, so lowering it at runtime does not retire the NTs that already exist. They retire lazily, and `SNT_KEEP_MINIMUM` keeps some of them alive. - `enable_mn_threads` decides how each thread gets a native thread, and the VM reads it when the thread is created. A thread created while the flag was off holds a dedicated native thread. A dedicated NT is its own OS thread and ignores `max_cpu` completely. Together these mean that a switch at the time of the call cannot cover the threads that the process already has, but it can cover a process with no other living threads. We therefore propose: - `Ractor.check_isolation` switches the VM to a single run slot when it is the only living thread in the process, and runs the block; - otherwise it raises, and names the threads that prevent the guarantee. This removes the flag from the common case, and it turns a silent unsound result into an error. Ideally, we would like to find a solution where we can create a special exclusive Ractor regardless of the state of the VM, but need more guidance on how to achieve that. 4. **How does this interact with Feature #22226?** Class and module ownership changes which accesses are violations. The checker must follow the same rule, and we are prepared to improve the proposal to obey this invariant as well. Moreove, the special Ractor should also behave like how a real Ractor would with respect to the ownership of the classes/modules that the block creates. Otherwise, it would report fewer violations than a production worker would hit. 5. **Is passing by reference acceptable in a public API?** The block, the arguments, the return value and outgoing message payloads all bypass the isolation rules. This is sound because the special Ractor runs exclusively, so it is not a safety concern. 6. **What are the semantics for nested calls and for threads?** In our current patch, a nested call creates another special Ractor and resumes the calling Ractor when it returns. A thread that the block creates belongs to the special Ractor and therefore observes the check and only raises warnings. On the other hand, a Ractor created with `Ractor.new` does not inherit the check, and raises `Ractor::IsolationError` for isolation errors. We think this behaviour is right, but it should be specified. 7. **Should the VM deduplicate?** Most consumers want unique violations per source location, and they build that themselves. A `Set` of sites in the VM would cut the volume by two orders of magnitude. On the other hand, it might also hide a violation that occurs on a second path through the same line, and would make it hard to collect how many times each unique violation is seen. ## 5. Related issues - #22226: Ractor: class/module ownership. Changes which accesses are violations. - #22252: improve the "defined with an un-shareable Proc in a different Ractor" error. This feature downgrades the same check to a warning. - [ruby/ruby#15983](https://github.com/ruby/ruby/pull/15983): the reference chain in `Ractor::IsolationError#detailed_message`. Adding the same chain to the warnings would make a violation more actionable. ## 6. Acknowledgements Different parts of this patch were built by Ufuk Kayserilioglu, Edouard Chin and Hartley McGuire, using LLM assistance. @ko1 provided guidance on our original approach which was using a special Thread instead of a special Ractor. -- https://bugs.ruby-lang.org/
Issue #22300 has been updated by byroot (Jean Boussier). I believe this would be a very valuable feature (we discussed that ~1 year ago), particularly for testing environments. Right now testing that a given API is Ractor-compatible is very hard, an API like the one described would make it much easier.
Is warn the right surface?
A violation object can carry the class, the name, the kind, the reference chain and the backtrace. The warning category can stay as the default when the caller gives no callback.
I agree with the above. Warnings being the default can be useful in some use cases, like for a first look at which dependencies may be ractor incompatible.
The consumers want a deduplication by site
Note that some warnings were deduplicated in the past (keyword arguments I think?). So when in Warnings mode, it would make sense to deduplicate, not when in "report violations" mode.
Should the checks be a build-time option?
Definitely not. But if we wish to integrate this more with testing frameworks, I think the `on_violation` proposal is much better. Have you considered that `violation` would be a `Ractor::IsolationError` instance? You'd have the backtrace, etc. If we had such API, I'd like to use it to allow testing `ruby/spec` for Ractor compatibility, which would allow use to eagerly find bugs like [Bug #22293]. ---------------------------------------- Feature #22300: `Ractor.check_isolation`: report Ractor isolation violations as warnings instead of raising https://bugs.ruby-lang.org/issues/22300#change-119085 * Author: ufuk (Ufuk Kayserilioglu) * Status: Open * Target version: 4.1 ---------------------------------------- ## Summary We are proposing a `Ractor.check_isolation { ... }` method which runs the given block in a special non-main Ractor. Inside that Ractor, supported operations that would fail because of Ractor isolation report a `:ractor_isolation` category warning, and then continue, instead of raising an isolation error. The special Ractor that is used to run the `Ractor.check_isolation` block is a real non-main Ractor. For example, `Ractor.main?` returns `false` and `Ractor.current` returns a genuine Ractor with its own default port. This is possible by ensuring that no other Ractor executes Ruby concurrently with this special Ractor. That exclusivity makes the "warn and continue" contract safe. ### Examples A trivial code example: ```ruby class Config @settings = {} # not shareable @values = [] # also not shareable def self.settings = @settings def self.values = @values end # Today: the first violation ends the run. Ractor.new { Config.settings }.value #=> Ractor::RemoteError # whose cause is Ractor::IsolationError: can not # get unshareable values from instance variables of # classes/modules from non-main Ractors # With `check_isolation`: the violation is a warning, and the run continues. Ractor.check_isolation do Config.settings # warning, not exception Config.values # ...and the next one is also reported Ractor.main? #=> false end ``` The next example is a test suite that can collect the supported isolation violations in a single run: ```ruby # test_helper.rb ISOLATION_WARNINGS = Thread::Queue.new Warning.singleton_class.prepend(Module.new do def warn(msg, category: nil) if category == :ractor_isolation && !Thread.current[:capturing_isolation_warning] Thread.current[:capturing_isolation_warning] = true begin ISOLATION_WARNINGS << Ractor.make_shareable(msg) ensure Thread.current[:capturing_isolation_warning] = false end return end super end end) class ActiveSupport::TestCase def run(...) Ractor.check_isolation { super } end end ``` ## 1. Motivation ### 1.1 An isolation error stops the run at the first violation Migrating a large codebase to Ractor-safety is difficult, since running it inside a real Ractor gives only one violation per run, which needs to be made Ractor-safe before you can observe the next violation. A large codebase has thousands, if not tens of thousands of these Ractor isolation violations that need to be identified. Our team at Shopify started an effort to make Rails Ractor-safe earlier this year, and as part of that work we needed the full list of isolation problems and not just the first item. To tackle this problem, we used an agent assisted loop against a scaffolded Rails application which served 165 endpoints inside a Ractor. We directed the agent loop to identify and fix the source of the `Ractor::IsolationError`, and run the loop again, until all endpoints could successfully run. While this approach did end up getting us a list of isolation violations, it proved to be an cumbersome task even with agent assistance. Instead, when we built the original prototype of `Ractor.check_isolation`, our initial run reported 50,995 warnings, of which 353 were unique in a matter of minutes with no agent assistance at all. A later run against Rails `main` reported 58,315 warnings, of which 419 were unique. The same work needs 419 build/fix/rebuild cycles (performed by agents or humans) to collect the same information. ### 1.2 You cannot put an existing test suite inside a real Ractor The obvious alternative to the above is to put a `Ractor.new { ... }` around each test in the test suite of the codebase. This approach, while attractive, does not work on real codebases, for three main reasons: 1. The test code must itself be Ractor-safe. Making the tests safe is often harder than making the application safe. Most test helpers hold mutable global state on purpose. 2. The block of `Ractor.new` must be isolated. It cannot close over the test case, the fixtures or the assertion state. 3. Non-shareable arguments and return values are copied. However, copying is not possible for some objects, for example a `Proc`, and the run aborts. `Ractor.check_isolation` removes all three obstacles. The block is not isolated. Arguments, the return value and outgoing message payloads pass by reference. Moreover, any isolation violation warning reported from the test file sources can easily be discarded as irrelevant. ### 1.3 There is no way to keep a codebase Ractor-safe Given the above point, this is the argument that matters most in the long term, in our opinion. A codebase that is Ractor-safe today will not stay Ractor-safe in the future. As the codebase evolves, one new lazy memoization, or one new unfrozen constant in a dependency, will reintroduce a violation. A normal test suite cannot detect the change, because the test suite does not run in a Ractor (as explain in the previous point). A gem can therefore break its downstream users without any signal, and the failure would only appear in production. `Ractor.check_isolation` turns Ractor safety into a property that CI can measure. A gem or an application can wrap its tests in the block and fails the build when the count of unique warnings increases. In essence, our argument is that this proposal gives the ecosystem two things it does not have today: - a burn-down list, to make a codebase Ractor-safe; - a ratchet, to keep it Ractor-safe. ## 2. Proposal ```ruby Ractor.check_isolation(*args, name: nil) { |*args| ... } # -> the value of the block ``` `Ractor.check_isolation` method: - runs the block in a new, real, and special non-main Ractor; - does not isolate the block, and passes arguments by reference; - reports each supported isolation violation as a `:ractor_isolation` category warning, and continues without raising; - unlike an ordinary Ractor, returns a non-shareable block value without copying it, preserving object identity. The reported violations match the constructs that would have raised a `Ractor::IsolationError` under a normal non-main `Ractor`. They include, but are not limited to: - instance variables and class variables of classes and modules; - constants that hold non-shareable objects; - global variables; - `Ractor.make_shareable`, `Ractor.shareable_proc` and `Ractor.shareable_lambda` failures; - calls to Ractor-unsafe C methods; - `fork` from a non-main Ractor; - defining or undefining finalizers on objects owned by another Ractor; - a call to a method that `define_method` defined with a non-shareable `Proc`; - isolation of the check block when it captures outer variables or uses `yield`. The user suppresses isolation-violation warnings with `Warning[:ractor_isolation] = false` or with `-W:no-ractor_isolation`. The exclusivity of the special Ractor that run the block is a precondition of the method, not a mode that the user should select. The current patch relies on `RUBY_RACTOR_EXCLUSIVE=1` and warns when it is unavailable. However, for the final solution we propose that `Ractor.check_isolation` instead establish exclusivity itself and raise when it cannot. Section 4.3 discusses the implementation constraints regarding this. The exclusive mode mentioned here is not a stop-the-world barrier. A blocking operation still hands the run slot over, so that, for example, work that the block correctly dispatches to the main Ractor completes, and the program does not deadlock. ## 3. Implementation Our current patch is on this branch: https://github.com/Shopify/ruby/compare/master...Shopify:ruby:ractor-check-i... The branch exposes the design described here under the name `Ractor.check_isolation`. Main points: - A new warning category, `RB_WARN_CATEGORY_RACTOR_ISOLATION`, registered as `:ractor_isolation`. - A flag on the special `rb_ractor_t`. A helper, `rb_ractor_isolation_violation()`, replaces the raise at many of the covered violation sites. The helper warns when the current Ractor has the flag set, and raises otherwise. - `ractor_create0()` creates and marks the special Ractor. When `rb_thread_create_ractor()` starts it, `thread_create_core()` skips `rb_proc_isolate_bang()` and retains the argument array by reference. `rb_proc_check_isolation_warn()` reports the `Proc`-isolation violations that `Ractor.new` would raise, and leaves the closure intact. - `ractor_prepare_payload()` passes a non-shareable message payload by reference, with a warning, instead of copying it. - The bmethod call paths warn instead of raising for a cross-Ractor call to a method defined with a non-shareable `Proc`. - On builds with M:N scheduling, `ruby_mn_threads_params()` enables it and sets `max_cpu = 1` under `RUBY_RACTOR_EXCLUSIVE=1`. - The first call switches the VM into multi-ractor mode. This switch is one-way today. Two costs with the current patch are worth stating explicitly: 1. The VM keeps the multi-ractor overhead for the rest of the process, and 2. the checks sit on paths that a production process executes. We consider these costs acceptable since this feature is a development and CI tool, and should not be enabled in production mode. ## 4. Open questions We have used this feature for four months on Rails and on two large production applications at Shopify. The following questions come out of that use. We do not have a strong position on most of them. 1. **Is `warn` the right surface?**: `Warning.warn` was the cheapest thing to build, but it may be the weakest part of the design, for the following reasons: - The volume is high. In one Shopify application, the run produced more warning text than the 6 GB memory limit of the CI worker, so it was killed. The consumers want a deduplication by site, not a stream of strings. - A string throws the structure away. The consumers want the class, the variable name, the reference chain and the location as data, rather than parsing whatever information is available from the message. - Re-entrancy is a real hazard. Any code in the handler can itself trigger a violation. We hit an infinite recursion this way, and we also hit a stack overflow because formatting the message called `to_s` on a proxy module. - `Warning.warn` is global process state. A warning that the special Ractor reports is awkward to collect in the caller. The alternatives are a callback, or a reporter object: ```ruby Ractor.check_isolation(on_violation: ->(violation) { ... }) { ... } Ractor.check_isolation(reporter: MyReporter.new) { ... } ``` A `violation` object can carry the class, the name, the kind, the reference chain and the backtrace. The warning category can stay as the default when the caller gives no callback. 2. **Should the checks be a build-time option?** The checks add work to hot paths. An alternative is to compile them only into a build that configures them, in the same way as `RGENGC_CHECK_MODE`. This trades convenience for speed. We do not know which side upstream prefers, but considering that so few Ruby developers know how to build a special version of Ruby, we think this will stop the widespread adoption of the feature. 3. **How does the method establish exclusivity?** The current patch reads a boot flag, `RUBY_RACTOR_EXCLUSIVE=1`, and warns once when the flag is absent. We do not propose that as the interface for the final state. The constraint for why we have the boot flag is technical and we welcome suggestions to removing it. Two mechanisms in `thread_sched.c` decide the outcome: - `vm->ractor.sched.max_cpu` caps the number of shared native threads. The VM reads it when an NT rejoins the pool, so lowering it at runtime does not retire the NTs that already exist. They retire lazily, and `SNT_KEEP_MINIMUM` keeps some of them alive. - `enable_mn_threads` decides how each thread gets a native thread, and the VM reads it when the thread is created. A thread created while the flag was off holds a dedicated native thread. A dedicated NT is its own OS thread and ignores `max_cpu` completely. Together these mean that a switch at the time of the call cannot cover the threads that the process already has, but it can cover a process with no other living threads. We therefore propose: - `Ractor.check_isolation` switches the VM to a single run slot when it is the only living thread in the process, and runs the block; - otherwise it raises, and names the threads that prevent the guarantee. This removes the flag from the common case, and it turns a silent unsound result into an error. Ideally, we would like to find a solution where we can create a special exclusive Ractor regardless of the state of the VM, but need more guidance on how to achieve that. 4. **How does this interact with Feature #22226?** Class and module ownership changes which accesses are violations. The checker must follow the same rule, and we are prepared to improve the proposal to obey this invariant as well. Moreove, the special Ractor should also behave like how a real Ractor would with respect to the ownership of the classes/modules that the block creates. Otherwise, it would report fewer violations than a production worker would hit. 5. **Is passing by reference acceptable in a public API?** The block, the arguments, the return value and outgoing message payloads all bypass the isolation rules. This is sound because the special Ractor runs exclusively, so it is not a safety concern. 6. **What are the semantics for nested calls and for threads?** In our current patch, a nested call creates another special Ractor and resumes the calling Ractor when it returns. A thread that the block creates belongs to the special Ractor and therefore observes the check and only raises warnings. On the other hand, a Ractor created with `Ractor.new` does not inherit the check, and raises `Ractor::IsolationError` for isolation errors. We think this behaviour is right, but it should be specified. 7. **Should the VM deduplicate?** Most consumers want unique violations per source location, and they build that themselves. A `Set` of sites in the VM would cut the volume by two orders of magnitude. On the other hand, it might also hide a violation that occurs on a second path through the same line, and would make it hard to collect how many times each unique violation is seen. ## 5. Related issues - #22226: Ractor: class/module ownership. Changes which accesses are violations. - #22252: improve the "defined with an un-shareable Proc in a different Ractor" error. This feature downgrades the same check to a warning. - [ruby/ruby#15983](https://github.com/ruby/ruby/pull/15983): the reference chain in `Ractor::IsolationError#detailed_message`. Adding the same chain to the warnings would make a violation more actionable. ## 6. Acknowledgements Different parts of this patch were built by Ufuk Kayserilioglu, Edouard Chin and Hartley McGuire, using LLM assistance. @ko1 provided guidance on our original approach which was using a special Thread instead of a special Ractor. -- https://bugs.ruby-lang.org/
Issue #22300 has been updated by matz (Yukihiro Matsumoto). I like the idea. Ractor needs a way to find all violations in one run and to keep a codebase Ractor-safe in CI, and this gives both. But I don't like the name. This method does not check anything. It runs the block in a non-main Ractor with the rules relaxed, and reports violations on the side. A name like `trace_isolation` or `report_isolation_violations` would describe it better. Other suggestions are welcome. My answers to the open questions that need my opinion: * Passing by reference (question 5) is acceptable for a diagnostic tool, as long as exclusivity is guaranteed. The final version must raise when it cannot run exclusively, and must not continue with a warning. * For the reporting surface (question 1), I prefer `on_violation:` receiving a `Ractor::IsolationError` as @byroot suggested, with the warning as the default when no callback is given. There are still many open details, so let's introduce it as experimental in 4.1. I leave the rest (how to establish exclusivity, nesting and threads, #22226) to @ko1. Matz. ---------------------------------------- Feature #22300: `Ractor.check_isolation`: report Ractor isolation violations as warnings instead of raising https://bugs.ruby-lang.org/issues/22300#change-119347 * Author: ufuk (Ufuk Kayserilioglu) * Status: Open * Target version: 4.1 ---------------------------------------- ## Summary We are proposing a `Ractor.check_isolation { ... }` method which runs the given block in a special non-main Ractor. Inside that Ractor, supported operations that would fail because of Ractor isolation report a `:ractor_isolation` category warning, and then continue, instead of raising an isolation error. The special Ractor that is used to run the `Ractor.check_isolation` block is a real non-main Ractor. For example, `Ractor.main?` returns `false` and `Ractor.current` returns a genuine Ractor with its own default port. This is possible by ensuring that no other Ractor executes Ruby concurrently with this special Ractor. That exclusivity makes the "warn and continue" contract safe. ### Examples A trivial code example: ```ruby class Config @settings = {} # not shareable @values = [] # also not shareable def self.settings = @settings def self.values = @values end # Today: the first violation ends the run. Ractor.new { Config.settings }.value #=> Ractor::RemoteError # whose cause is Ractor::IsolationError: can not # get unshareable values from instance variables of # classes/modules from non-main Ractors # With `check_isolation`: the violation is a warning, and the run continues. Ractor.check_isolation do Config.settings # warning, not exception Config.values # ...and the next one is also reported Ractor.main? #=> false end ``` The next example is a test suite that can collect the supported isolation violations in a single run: ```ruby # test_helper.rb ISOLATION_WARNINGS = Thread::Queue.new Warning.singleton_class.prepend(Module.new do def warn(msg, category: nil) if category == :ractor_isolation && !Thread.current[:capturing_isolation_warning] Thread.current[:capturing_isolation_warning] = true begin ISOLATION_WARNINGS << Ractor.make_shareable(msg) ensure Thread.current[:capturing_isolation_warning] = false end return end super end end) class ActiveSupport::TestCase def run(...) Ractor.check_isolation { super } end end ``` ## 1. Motivation ### 1.1 An isolation error stops the run at the first violation Migrating a large codebase to Ractor-safety is difficult, since running it inside a real Ractor gives only one violation per run, which needs to be made Ractor-safe before you can observe the next violation. A large codebase has thousands, if not tens of thousands of these Ractor isolation violations that need to be identified. Our team at Shopify started an effort to make Rails Ractor-safe earlier this year, and as part of that work we needed the full list of isolation problems and not just the first item. To tackle this problem, we used an agent assisted loop against a scaffolded Rails application which served 165 endpoints inside a Ractor. We directed the agent loop to identify and fix the source of the `Ractor::IsolationError`, and run the loop again, until all endpoints could successfully run. While this approach did end up getting us a list of isolation violations, it proved to be an cumbersome task even with agent assistance. Instead, when we built the original prototype of `Ractor.check_isolation`, our initial run reported 50,995 warnings, of which 353 were unique in a matter of minutes with no agent assistance at all. A later run against Rails `main` reported 58,315 warnings, of which 419 were unique. The same work needs 419 build/fix/rebuild cycles (performed by agents or humans) to collect the same information. ### 1.2 You cannot put an existing test suite inside a real Ractor The obvious alternative to the above is to put a `Ractor.new { ... }` around each test in the test suite of the codebase. This approach, while attractive, does not work on real codebases, for three main reasons: 1. The test code must itself be Ractor-safe. Making the tests safe is often harder than making the application safe. Most test helpers hold mutable global state on purpose. 2. The block of `Ractor.new` must be isolated. It cannot close over the test case, the fixtures or the assertion state. 3. Non-shareable arguments and return values are copied. However, copying is not possible for some objects, for example a `Proc`, and the run aborts. `Ractor.check_isolation` removes all three obstacles. The block is not isolated. Arguments, the return value and outgoing message payloads pass by reference. Moreover, any isolation violation warning reported from the test file sources can easily be discarded as irrelevant. ### 1.3 There is no way to keep a codebase Ractor-safe Given the above point, this is the argument that matters most in the long term, in our opinion. A codebase that is Ractor-safe today will not stay Ractor-safe in the future. As the codebase evolves, one new lazy memoization, or one new unfrozen constant in a dependency, will reintroduce a violation. A normal test suite cannot detect the change, because the test suite does not run in a Ractor (as explain in the previous point). A gem can therefore break its downstream users without any signal, and the failure would only appear in production. `Ractor.check_isolation` turns Ractor safety into a property that CI can measure. A gem or an application can wrap its tests in the block and fails the build when the count of unique warnings increases. In essence, our argument is that this proposal gives the ecosystem two things it does not have today: - a burn-down list, to make a codebase Ractor-safe; - a ratchet, to keep it Ractor-safe. ## 2. Proposal ```ruby Ractor.check_isolation(*args, name: nil) { |*args| ... } # -> the value of the block ``` `Ractor.check_isolation` method: - runs the block in a new, real, and special non-main Ractor; - does not isolate the block, and passes arguments by reference; - reports each supported isolation violation as a `:ractor_isolation` category warning, and continues without raising; - unlike an ordinary Ractor, returns a non-shareable block value without copying it, preserving object identity. The reported violations match the constructs that would have raised a `Ractor::IsolationError` under a normal non-main `Ractor`. They include, but are not limited to: - instance variables and class variables of classes and modules; - constants that hold non-shareable objects; - global variables; - `Ractor.make_shareable`, `Ractor.shareable_proc` and `Ractor.shareable_lambda` failures; - calls to Ractor-unsafe C methods; - `fork` from a non-main Ractor; - defining or undefining finalizers on objects owned by another Ractor; - a call to a method that `define_method` defined with a non-shareable `Proc`; - isolation of the check block when it captures outer variables or uses `yield`. The user suppresses isolation-violation warnings with `Warning[:ractor_isolation] = false` or with `-W:no-ractor_isolation`. The exclusivity of the special Ractor that run the block is a precondition of the method, not a mode that the user should select. The current patch relies on `RUBY_RACTOR_EXCLUSIVE=1` and warns when it is unavailable. However, for the final solution we propose that `Ractor.check_isolation` instead establish exclusivity itself and raise when it cannot. Section 4.3 discusses the implementation constraints regarding this. The exclusive mode mentioned here is not a stop-the-world barrier. A blocking operation still hands the run slot over, so that, for example, work that the block correctly dispatches to the main Ractor completes, and the program does not deadlock. ## 3. Implementation Our current patch is on this branch: https://github.com/Shopify/ruby/compare/master...Shopify:ruby:ractor-check-i... The branch exposes the design described here under the name `Ractor.check_isolation`. Main points: - A new warning category, `RB_WARN_CATEGORY_RACTOR_ISOLATION`, registered as `:ractor_isolation`. - A flag on the special `rb_ractor_t`. A helper, `rb_ractor_isolation_violation()`, replaces the raise at many of the covered violation sites. The helper warns when the current Ractor has the flag set, and raises otherwise. - `ractor_create0()` creates and marks the special Ractor. When `rb_thread_create_ractor()` starts it, `thread_create_core()` skips `rb_proc_isolate_bang()` and retains the argument array by reference. `rb_proc_check_isolation_warn()` reports the `Proc`-isolation violations that `Ractor.new` would raise, and leaves the closure intact. - `ractor_prepare_payload()` passes a non-shareable message payload by reference, with a warning, instead of copying it. - The bmethod call paths warn instead of raising for a cross-Ractor call to a method defined with a non-shareable `Proc`. - On builds with M:N scheduling, `ruby_mn_threads_params()` enables it and sets `max_cpu = 1` under `RUBY_RACTOR_EXCLUSIVE=1`. - The first call switches the VM into multi-ractor mode. This switch is one-way today. Two costs with the current patch are worth stating explicitly: 1. The VM keeps the multi-ractor overhead for the rest of the process, and 2. the checks sit on paths that a production process executes. We consider these costs acceptable since this feature is a development and CI tool, and should not be enabled in production mode. ## 4. Open questions We have used this feature for four months on Rails and on two large production applications at Shopify. The following questions come out of that use. We do not have a strong position on most of them. 1. **Is `warn` the right surface?**: `Warning.warn` was the cheapest thing to build, but it may be the weakest part of the design, for the following reasons: - The volume is high. In one Shopify application, the run produced more warning text than the 6 GB memory limit of the CI worker, so it was killed. The consumers want a deduplication by site, not a stream of strings. - A string throws the structure away. The consumers want the class, the variable name, the reference chain and the location as data, rather than parsing whatever information is available from the message. - Re-entrancy is a real hazard. Any code in the handler can itself trigger a violation. We hit an infinite recursion this way, and we also hit a stack overflow because formatting the message called `to_s` on a proxy module. - `Warning.warn` is global process state. A warning that the special Ractor reports is awkward to collect in the caller. The alternatives are a callback, or a reporter object: ```ruby Ractor.check_isolation(on_violation: ->(violation) { ... }) { ... } Ractor.check_isolation(reporter: MyReporter.new) { ... } ``` A `violation` object can carry the class, the name, the kind, the reference chain and the backtrace. The warning category can stay as the default when the caller gives no callback. 2. **Should the checks be a build-time option?** The checks add work to hot paths. An alternative is to compile them only into a build that configures them, in the same way as `RGENGC_CHECK_MODE`. This trades convenience for speed. We do not know which side upstream prefers, but considering that so few Ruby developers know how to build a special version of Ruby, we think this will stop the widespread adoption of the feature. 3. **How does the method establish exclusivity?** The current patch reads a boot flag, `RUBY_RACTOR_EXCLUSIVE=1`, and warns once when the flag is absent. We do not propose that as the interface for the final state. The constraint for why we have the boot flag is technical and we welcome suggestions to removing it. Two mechanisms in `thread_sched.c` decide the outcome: - `vm->ractor.sched.max_cpu` caps the number of shared native threads. The VM reads it when an NT rejoins the pool, so lowering it at runtime does not retire the NTs that already exist. They retire lazily, and `SNT_KEEP_MINIMUM` keeps some of them alive. - `enable_mn_threads` decides how each thread gets a native thread, and the VM reads it when the thread is created. A thread created while the flag was off holds a dedicated native thread. A dedicated NT is its own OS thread and ignores `max_cpu` completely. Together these mean that a switch at the time of the call cannot cover the threads that the process already has, but it can cover a process with no other living threads. We therefore propose: - `Ractor.check_isolation` switches the VM to a single run slot when it is the only living thread in the process, and runs the block; - otherwise it raises, and names the threads that prevent the guarantee. This removes the flag from the common case, and it turns a silent unsound result into an error. Ideally, we would like to find a solution where we can create a special exclusive Ractor regardless of the state of the VM, but need more guidance on how to achieve that. 4. **How does this interact with Feature #22226?** Class and module ownership changes which accesses are violations. The checker must follow the same rule, and we are prepared to improve the proposal to obey this invariant as well. Moreove, the special Ractor should also behave like how a real Ractor would with respect to the ownership of the classes/modules that the block creates. Otherwise, it would report fewer violations than a production worker would hit. 5. **Is passing by reference acceptable in a public API?** The block, the arguments, the return value and outgoing message payloads all bypass the isolation rules. This is sound because the special Ractor runs exclusively, so it is not a safety concern. 6. **What are the semantics for nested calls and for threads?** In our current patch, a nested call creates another special Ractor and resumes the calling Ractor when it returns. A thread that the block creates belongs to the special Ractor and therefore observes the check and only raises warnings. On the other hand, a Ractor created with `Ractor.new` does not inherit the check, and raises `Ractor::IsolationError` for isolation errors. We think this behaviour is right, but it should be specified. 7. **Should the VM deduplicate?** Most consumers want unique violations per source location, and they build that themselves. A `Set` of sites in the VM would cut the volume by two orders of magnitude. On the other hand, it might also hide a violation that occurs on a second path through the same line, and would make it hard to collect how many times each unique violation is seen. ## 5. Related issues - #22226: Ractor: class/module ownership. Changes which accesses are violations. - #22252: improve the "defined with an un-shareable Proc in a different Ractor" error. This feature downgrades the same check to a warning. - [ruby/ruby#15983](https://github.com/ruby/ruby/pull/15983): the reference chain in `Ractor::IsolationError#detailed_message`. Adding the same chain to the warnings would make a violation more actionable. ## 6. Acknowledgements Different parts of this patch were built by Ufuk Kayserilioglu, Edouard Chin and Hartley McGuire, using LLM assistance. @ko1 provided guidance on our original approach which was using a special Thread instead of a special Ractor. -- https://bugs.ruby-lang.org/
participants (3)
-
byroot (Jean Boussier) -
matz (Yukihiro Matsumoto)
-
ufuk (Ufuk Kayserilioglu)