[ruby-core:120694] [Ruby master Bug#21039] Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks

Issue #21039 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Bug #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039 * Author: Eregon (Benoit Daloze) * Status: Open * Assignee: ko1 (Koichi Sasada) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by luke-gru (Luke Gruber). As far as I know this is intentional behavior, so even though I agree it is confusing I think this is more accurately a feature request instead of a bug. ---------------------------------------- Bug #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-111536 * Author: Eregon (Benoit Daloze) * Status: Open * Assignee: ko1 (Koichi Sasada) * ruby -v: ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). I think it's a bug, because it breaks fundamental Ruby block semantics. No Ractor method or functionality should be able to do that for an existing Proc, even more so when that Proc is called on the main Ractor. ---------------------------------------- Bug #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-111550 * Author: Eregon (Benoit Daloze) * Status: Open * Assignee: ko1 (Koichi Sasada) * ruby -v: ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by luke-gru (Luke Gruber). Okay fair enough, and it's not of much consequence either way whether it's a bug or a feature because your point still stands. ---------------------------------------- Bug #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-111553 * Author: Eregon (Benoit Daloze) * Status: Open * Assignee: ko1 (Koichi Sasada) * ruby -v: ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by matz (Yukihiro Matsumoto). Tracker changed from Bug to Feature ruby -v deleted (ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux]) Backport deleted (3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN) 1. It's intended behavior. We don't call it a bug. If we keep the non-shareable semantics, the basic principle of Ractors would fail (no shared state). 2. I understand your feeling to the new behavior. The behavior can be negotiable, but it should follow Ractor principle. Matz. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-111571 * Author: Eregon (Benoit Daloze) * Status: Open * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). I think a good solution here would be: * raise on `Ractor.make_shareable(proc)` * raise on `Ractor.make_shareable(proc, copy: true)` * allow `Ractor.make_shareable { ... }` but only with a block literal. That way, the fact that block behaves differently by copying its environment is very clear. The method could check that the block doesn't do assignments in the environment as well to avoid surprises. It could also set the `self` in the block to `nil`, and/or take a keyword argument to set the receiver, or use the block's original self and error if it cannot be made shareable. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-113845 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by matz (Yukihiro Matsumoto). I'd like to have `Ractor.shareble_proc` and `Ractor.sharable_lambda`. See the agenda from 20250710 developer meeting. Matz. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-113983 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by mame (Yusuke Endoh). https://github.com/ruby/dev-meeting-log/blob/master/2025/DevMeeting-2025-07-... * `Ractor.shareable_proc { }` returns a Proc that is shareable between ractors. In the proc, `self` is nil. * `Ractor.shareable_proc(self: 42) { }` returns a Proc that is shareable between ractors. In the proc, `self` is `42`. * `Ractor.shareable_lambda` returns a lambda-version of `Ractor.shareable_proc`. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114006 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by mame (Yusuke Endoh). The followings are also approved; changing an existing proc object to shareable should be prohibited.
* raise on `Ractor.make_shareable(proc)` * raise on `Ractor.make_shareable(proc, copy: true)`
---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114007 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). Great to hear, this makes a lot of sense and addresses the original semantics issue perfectly. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114063 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by tenderlovemaking (Aaron Patterson). I think these make sense, but I would also like to propose that `Ractor.shareable_proc` take a block that isn't a literal and returns a new proc that is shareable. For example: ``` not_shareable = ->{ ... } shareable = Ractor.shareable_proc(¬_shareable) ``` I think forcing the proc to be a literal put a very big limitation on Ractor usefulness. A simple example is a Sinatra app: ```ruby # A simple Sinatra app get "/" do "Hello world" end ``` Since the Sinatra API uses procs, we wouldn't be able to serve the Sinatra request from a Ractor. If we can pass a non-literal block to `Ractor.shareable_proc`, I think that would solve the issue. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114130 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). @tenderlovemaking The issue with that is it still breaks the block semantics as in the OP description, specifically reading of captured variables inside the block is snapshotted for Ractor-shareable-blocks: ``` $ ruby -e 'count = 0; b = nil.instance_exec { -> { count } }; p b.call; count += 1; p b.call' 0 1 $ ruby -e 'count = 0; b = nil.instance_exec { -> { count } }; b2 = Ractor.shareable_proc(&b); p b2.call; count += 1; p b2.call' 0 0 ``` It's the general guarantee in Ruby that a given literal block always behave the same, e.g. it's either a proc or lambda, but not both (except when using `send(condition ? :proc : :lambda) { ... }` but that's explicit then), and in this case it's either a block respecting updated captures variables or not. So how to keep this important guarantee (i.e. the block author can rely on captured variables to behave as they always have been for Ruby blocks and respect reassignments) and allow the flexibility you want? BTW `Ractor.make_shareable` on a Proc which assigns captured variables is an error (good, better to fail early than silently ignore the write): ``` $ ruby -e 'count = 0; b = nil.instance_exec { -> { count += 1 } }; Ractor.make_shareable b' -e:1:in 'Ractor.make_shareable': can not make a Proc shareable because it accesses outer variables (count). (ArgumentError) from -e:1:in '<main>' ``` Maybe one way would be for `Ractor.shareable_proc` to be an error if there is any code around that Proc assigning any captured variable? It can't detect `binding` and `eval` though, so that's still not complete. One way for that Sinatra case would be to write: ```ruby get "/", &Ractor.shareable_proc { "Hello world" } ``` but this only really works if there are no captured variables, or captured variables are not reassigned and the contents of captured variables is shareable, so probably in many realistic cases it doesn't work anyway. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114131 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by tenderlovemaking (Aaron Patterson). I understand your argument, but I don't agree this is an issue.
ruby -e 'count = 0; b = nil.instance_exec { -> { count } }; b2 = Ractor.shareable_proc(&b); p b2.call; count += 1; p b2.call'
In this code we've explicitly converted `b` to a shareable proc, `b2`. The value of `count` is internally consistent from the standpoint of `b2` since `shareable_proc` explicitly copied / disconnected the captured environment. I understand the problem you're pointing out, but I'm not convinced it would be a big deal in practice. ```ruby get "/", &Ractor.shareable_proc { "Hello world" } ``` This can't be a serious suggestion? It's basically saying that no existing Sinatra code could run inside a Ractor based webserver. If we had new syntax for `Ractor.shareable_proc`, I could see that being easier to swallow, but this doesn't seem acceptable (to me at least). ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114150 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). tenderlovemaking (Aaron Patterson) wrote in #note-18:
This can't be a serious suggestion? It's basically saying that no existing Sinatra code could run inside a Ractor based webserver. If we had new syntax for `Ractor.shareable_proc`, I could see that being easier to swallow, but this doesn't seem acceptable (to me at least).
Yeah, I understand your concern, and I meant this mostly as a workaround, while finding what other parts of Ractor prevents using Ractor for realistic Ruby code. OTOH I'm rather skeptical that even if `Ractor.shareable_proc` would be allowed with a non-literal block that we'd be able to run Sinatra (or Rails, etc) apps on Ractors. Not even MSpec runs on Ractor, and that's pretty simple logic, yet making it Ractor-compatible in a non-ugly-and-complicated way seems very hard. I think it would be good to have a construct (be it syntax or a Kernel or Proc method), independent of Ractor, to create a Proc which snapshots its environment, and is not allowed to write to its environment. That way, that Proc would have the same semantics whether Ractors are used or not. That concept on its own is very useful for optimizations and JITs, in fact TruffleRuby [already has this functionality internally](https://github.com/oracle/truffleruby/blob/e805fbbf231d0680cb262c5dfd2278efd...). One complication though is it's pretty expensive to do this, as on every call to that method it allocates a new Proc and potentially copies + change the bytecode to replace captured variable reads with their values (thought there might be other ways to do this). Having it as syntax or an intrinisified method (which would mean cannot be redefined and must be detectable at parse time, no metaprogramming call to it) would help. It would be useful for `define_method` too, and would mean methods defined with `define_method` could be as fast as `def` when called. To make it useful for Ractor we'd need that construct to also support setting the receiver, as in the `Ractor.shareable_proc(self: 42) { }` case from above. And it would also need to either check that captured variable values are shareable, or make them shareable. That part is a bit weird when not using Ractors though, especially making shareable. Checking for shareable seems better anyway, because making them shareable would need to copy to be safe in general, but maybe the user want to do it inplace if they know that's safe, etc. It could be something like: ```ruby captured = 7 p = Proc.isolated(self: 6) { self * captured } captured = 10 p.call # => 42 ``` Syntax seems better-defined for the semantics and probably would look cleaner, but I'm not sure what would be a good syntax, and then of course it can't work on older Ruby versions at all, even when not using Ractors. Unless we use something cheeky like `proc { |a,b; isolated| }` maybe, but that wouldn't allow setting the self (would have to be done with `instance_exec` around it), and would have different semantics on different versions which is not great. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114153 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by tenderlovemaking (Aaron Patterson). Eregon (Benoit Daloze) wrote in #note-19:
tenderlovemaking (Aaron Patterson) wrote in #note-18:
This can't be a serious suggestion? It's basically saying that no existing Sinatra code could run inside a Ractor based webserver. If we had new syntax for `Ractor.shareable_proc`, I could see that being easier to swallow, but this doesn't seem acceptable (to me at least).
Yeah, I understand your concern, and I meant this mostly as a workaround, while finding what other parts of Ractor prevents using Ractor for realistic Ruby code. OTOH I'm rather skeptical that even if `Ractor.shareable_proc` would be allowed with a non-literal block that we'd be able to run Sinatra (or Rails, etc) apps on Ractors. Not even MSpec runs on Ractor, and that's pretty simple logic, yet making it Ractor-compatible in a non-ugly-and-complicated way seems very hard.
I don't have any numbers, but my intuition is that most non-literal, global procs (procs that are reachable via the application), including ones provided by the user, don't depend on environment mutations that occur outside of the block. Outside of iterators, depending on mutations to one's captured environment would be extremely confusing and hard to track behavior, so IME most people don't do it in practice. But besides that, I'm just proposing that `Ractor.shareable_proc` be allowed to take a non-literal block. This would allow frameworks to pick and choose which lambdas should be "safe" for a Ractor. If someone had a Sinatra app that depended on env mutations like this: ```ruby counter = 0 get "/" do # assume the proc gets copied here so counter is 0 "Hello world #{counter}" counter += 1 end counter += 123 ``` I think a user running a Ractor webserver would report an issue with the webserver since it would behave "as expected" on a non-Ractor webserver.
I think it would be good to have a construct (be it syntax or a Kernel or Proc method), independent of Ractor, to create a Proc which snapshots its environment, and is not allowed to write to its environment. That way, that Proc would have the same semantics whether Ractors are used or not.
I'm not sure why it matters whether the proc can write to the captured env or not, since we can just copy the environment and attach it to the proc. To me this is similar to dup'ing an object. If I mutate one copy, I don't expect those mutations to be reflected in the other.
That concept on its own is very useful for optimizations and JITs, in fact TruffleRuby [already has this functionality internally](https://github.com/oracle/truffleruby/blob/e805fbbf231d0680cb262c5dfd2278efd...). One complication though is it's pretty expensive to do this, as on every call to that method it allocates a new Proc and potentially copies + change the bytecode to replace captured variable reads with their values (thought there might be other ways to do this). Having it as syntax or an intrinisified method (which would mean cannot be redefined and must be detectable at parse time, no metaprogramming call to it) would help. It would be useful for `define_method` too, and would mean methods defined with `define_method` could be as fast as `def` when called.
I've thought of doing this by mprotecting the escaped env and only allowing reads 😅
To make it useful for Ractor we'd need that construct to also support setting the receiver, as in the `Ractor.shareable_proc(self: 42) { }` case from above. And it would also need to either check that captured variable values are shareable, or make them shareable. That part is a bit weird when not using Ractors though, especially making shareable. Checking for shareable seems better anyway, because making them shareable would need to copy to be safe in general, but maybe the user want to do it inplace if they know that's safe, etc. It could be something like: ```ruby captured = 7 p = Proc.isolated(self: 6) { self * captured } captured = 10 p.call # => 42 ```
Syntax seems better-defined for the semantics and probably would look cleaner, but I'm not sure what would be a good syntax, and then of course it can't work on older Ruby versions at all, even when not using Ractors. Unless we use something cheeky like `proc { |a,b; isolated| }` maybe, but that wouldn't allow setting the self (would have to be done with `instance_exec` around it), and would have different semantics on different versions which is not great.
Anyway, by not allowing non-literal blocks, we can't even abstract calls to `shareable_proc` and I think that really hampers the usefulness of Ractors. I really think we should find a way to support non-literal blocks. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114154 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by tenderlovemaking (Aaron Patterson). After chatting a bit with @Eregon, I'd like to make a proposal about Ractor shareable procs. I think it's going to be very difficult to port existing code to use Ractors if we can't have some way of creating shareable procs from non-literal blocks. For example we wouldn't be able to use a Ractor based webserver with existing Sinatra applications. With regard to local variables captured in environments, I think we should have the following rules: 1. Allow as many local writes as the user wants before the lambda captures the environment 2. Disallow any writes to the "now shared" environment after the lambda captures it 3. Disallow writes to the shared environment from inside the lambda These rules should only apply to local variables declared outside the block. Here are some examples to demonstrate the rules: ```ruby ## OK foo = 123 Ractor.shareable_proc { foo } p foo ``` ```ruby # NG: Raise an exception when creating a shareable proc # The reason is because we're setting a local after the proc # is created. This can cause possible race conditions / crashes foo = 123 Ractor.shareable_proc { foo } foo = Object.new # reassignment isn't allowed ``` ```ruby # NG: Raise an exception when creating a shareable proc # The proc shouldn't be allowed to mutate a shared environment foo = 123 Ractor.shareable_proc { foo += 1 # Not allowed because other env can't see mutation } ``` ```ruby # Works, but value of `foo` may be unexpected. # The second assignment should be ignored because the env is copied foo = 123 Ractor.shareable_proc { foo } eval("foo = Object.new") ``` ```ruby # Works, but value of `foo` outside the proc may be unexpected # Proc only mutates its copied env foo = 123 Ractor.shareable_proc { eval("foo += 1") } ``` If we can enforce these rules, then I think it should be fine for `Ractor.shareable_proc` to take a block that _isn't_ a literal. I also think this would allow a much larger number of existing proc objects to work safely with Ractors. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114229 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). I agree this should make `Ractor.shareable_proc` safe enough with a non-literal block and address the semantics issue in the OP. Cases with `eval` seem not possible to know and it seems rare enough to be OK to behave with "snapshot/copy of environment" semantics in that case. The rules are quite similar to the rules for the `Ractor.new {}` block, which could use the same rules (`Ractor.new {}` is currently stricter as it does not allow reading an environment variable, but it should/could for convenience & consistency). Of course there should still be a check that environment variables the block captures are shareable, which `Ractor.make_shareable(Proc)` already does: ```ruby nil.instance_exec { a = Object.new; Ractor.make_shareable(proc { a }) } # can not make shareable Proc because it can refer unshareable object #<Object:0x00007ff7ef878360> from variable 'a' (Ractor::IsolationError) ``` And same for the `self` around the block: ```ruby a = 1; Ractor.make_shareable(proc { a }) # Proc's self is not shareable: #<Proc:0x00007faf42016f00 -e:1> (Ractor::IsolationError) ``` Regarding the reason for the 2nd example in @tenderlovemaking 's comment, it wouldn't cause race conditions / crashes because `shareable_proc` already makes a copy of the environment anyway (Aaron told me). But it would cause the semantics issue in the OP, that an assignment to outer variable is not observed and breaks Ruby block semantics, hence that case must be an exception from `Ractor.shareable_proc`, e.g. a `Ractor::IsolationError` or `ArgumentError`. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114233 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by jhawthorn (John Hawthorn). tenderlovemaking (Aaron Patterson) wrote in #note-21:
```ruby foo = 123 Ractor.shareable_proc { foo } foo = Object.new # reassignment isn't allowed ```
All the other new rules are great and improve consistency, but I don't think this one is viable or necessary. To implement it would require examining a CFG (which we don't currently build in CRuby) and find every assignment which could come after. We'd also have to scan the rescue table (ex. `begin; p = proc { e }; raise; rescue => e; end`) and all child iseqs (ex. `1.times { foo = 456 }`). It makes very little sense to me that example 4 is fine but example 2 isn't. Since in a normal program those are the same thing. We'd also have to throw in the same caveat for `binding.local_variable_{set,get}`. So it all amounts to a lot of work that's more surprising to the user. ``` foo = 123 # Standard proc, using existing mutable env regular_proc = -> { foo } # New shareable proc object, the env is copied and immutable shared_proc = Ractor.shareable_proc(®ular_proc) foo = 456 # Should be allowed # For regular proc, standard, existing behaviour, we get the new value regular_proc.call # => 456 # For shareable proc, uses copied readonly env shared_proc.call # => 123 ``` I don't think this is a significant departure from existing semantics. It's reasonable for a different object, which has been explicitly created with these semantics, behaves differently than the original. It's also not dissimilar to using, say, instance_exec to change self (only here it's a different environment). This is also much more similar to the existing `Ractor.new(foo) {|foo| }`, which prohibits reading parent's locals, but doesn't forbid assignment after creating a Ractor (which would be MORE strict). Shareable proc just makes the copies implicit. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114291 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze).
It makes very little sense to me that example 4 is fine but example 2 isn't.
Ideally example 4 would also be forbidden. It just seems difficult to do it, but maybe there are ways?
To implement it would require examining a CFG (which we don't currently build in CRuby) and find every assignment which could come after.
Yes it requires some non-trivial work (maybe this is easier if this is done in `compile.c/compile_prism.c`? They already deal with assigning indices to local vars, resolving captured vars, etc). That work is required to make this new primitive safe enough to not break things very far away in the program. If we have to choose to do some more work and be safe being vs being unsafe, I hope you agree it's worth being safe and not breaking language semantics? Using the example of https://bugs.ruby-lang.org/issues/21039#note-20: ```ruby counter = 0 get "/" do # assume the proc gets copied here so counter is 0 "Hello world #{counter}" counter += 1 end counter += 123 ``` Assuming somewhere deep in Sinatra or Puma or so `Ractor.shareable_proc` is called with that block, I think it's a clear violation of the most basic semantics of Ruby blocks if that block doesn't see updates of `counter`. If people or I see this problem, I think they would react "WTF, Ruby is broken, nothing can be relied on anymore, not even local variable assignments". So I think there are three ways forward: * Don't allow `Ractor.shareable_proc` blocks to use any variable from the environment (like current `Ractor.new { ... }`) * Allow `Ractor.shareable_proc` to use variables from the environment, but make sure they are not reassigned so there are no broken semantics. Edge cases like `eval` and `binding` seem a much smaller issue if they notice the copy, though clearly not ideal. * Only allow `Ractor.shareable_proc` with a literal block, then it's clear enough that block has special semantics different from regular blocks (like a 3rd type besides proc & lambda). What has been agreed in https://bugs.ruby-lang.org/issues/21039#note-12.
All the other new rules are great and improve consistency, but I don't think this one is viable or necessary.
Actually you already need to scan rescue and inner blocks for rule 3 (The proc shouldn't be allowed to mutate a shared environment), for cases like: ```ruby foo = 123 Ractor.shareable_proc { 1.tap { begin; expr; rescue; foo += 1; end } } ```
So it all amounts to a lot of work that's more surprising to the user.
I think it's clear the user would be surprised in far less cases.
I don't think this is a significant departure from existing semantics.
Really, you think ignoring variable assignments is not a significant departure from existing semantics? BTW there is Proc#dup, of course it doesn't copy the environment. These new semantics are clearly a huge departure from existing block semantics.
It's also not dissimilar to using, say, instance_exec to change self (only here it's a different environment).
True that there is similarity. Also `instance_exec` can already be problematic. The big assumption there is a given block must always be executed with the same kind of `self`, e.g. instances of a given class or subclasses. If it's called with unrelated `self`s, most likely things would break, e.g. any method call inside the block would break unless that method is defined on both objects. `instance_exec` is mostly used for DSLs and there the `self` is always that DSL object. That's fine, because it's predictable and consistent, a given block in the source program gets called with `self` of the same class. It's the same thing as a given block in the program is either a lambda or proc, but never both (except some artificial edge cases). It's not guaranteed for `instance_exec`, but if misused it would be correct to blame whatever code is misusing it, and the worst case is a wrong `self`, which can be debugged with just `p self`. The worse case with unsafe `Ractor.shareable_proc` is extreme confusion (very hard to debug), data loss or corruption (due to ignored assignments) and potentially even security vulnerabilities as a result. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114292 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). @jhawthorn I apologize if my reply sounded disrespectful or so, it was not my intention, I am/was genuinely surprised that you think ignoring assignments is not a significant departure from existing semantics, maybe I misunderstood what you said. I edited the comment on Redmine. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114295 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by ko1 (Koichi Sasada). My summary: * To prohibit local variable assignment, we need to change the logic around local variables. * as jhawthorn said, we can assume: * `foo = 123; Ractor.shareable_proc { foo }` as * `foo = 123; Ractor.shareable_proc(foo) {|foo| foo }` (implicitly shadow'ed) as new syntax (`Ractor.shareable_proc` is a marker) * The problem is, if we allow to accept any Proc (like Sinatra's case), we can't assume which local variables are shadow'ed * matz accept this implicit behavior https://bugs.ruby-lang.org/issues/21039#note-9 * Another idea: The issue is, new local variable semantics caused by implicitly because of `Ractor.shareable_proc(&bl)`. So introduce new shorter syntax? For example, `-->(...){ ... }` ```ruby counter = 0 get "/", --> do # assume the proc gets copied here so counter is 0 "Hello world #{counter}" end counter += 123 ``` ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114297 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by jhawthorn (John Hawthorn). Eregon (Benoit Daloze) wrote in #note-24:
```ruby counter = 0 get "/" do # assume the proc gets copied here so counter is 0 "Hello world #{counter}" counter += 1 end
counter += 123 ```
Assuming somewhere deep in Sinatra or Puma or so `Ractor.shareable_proc` is called with that block, I think it's a clear violation of the most basic semantics of Ruby blocks if that block doesn't see updates of `counter`.
Because of the `counter += 1` this falls into case 3 and raises an error. I agree case 3 is good and we should keep it, the proposed case 2 (forbidding the `counter += 123` mutation in the parent scope) is unhelpful and we should not implement it. I really do not believe it is a departure from the existing semantics. For the shareable proc, the environment is captured at the point `Ractor.shareable_proc` is called on a block, so the imagined DSL _could_ have called it at that time and seen those values. I think you are reaching with the imagined security/corruption concerns, all of which already exist with instance_eval, or just calling procs at different times (or concurrently in Threads, which is the specific situation this aims to replace). ko1 (Koichi Sasada) wrote in #note-26:
```ruby counter = 0 get "/", --> do # assume the proc gets copied here so counter is 0 "Hello world #{counter}" end
counter += 123 ```
I would prefer we didn't introduce a new syntax. Ractor adoption is quite challenging as-is, so having the ability to integrate into existing DSLs where it makes sense would be very helpful. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114298 * Author: Eregon (Benoit Daloze) * Status: Assigned * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by matz (Yukihiro Matsumoto). Status changed from Assigned to Closed We discussed at the developers' meeting, and had the conclusion that the original issue cannot be addressed if we use make_sharable(proc). Especially, I cannot accept the idea prohibiting outer assignment to local variables when a proc is made sharable. The only help to the original issue is to make `make_sharable(proc)` an error, so we can prohibit accidental make_sharable(proc). In that case, the sharable_proc() idea can be considered later. But in the different issue. Matz. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114300 * Author: Eregon (Benoit Daloze) * Status: Closed * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). Log at https://github.com/ruby/dev-meeting-log/blob/master/2025/DevMeeting-2025-08-... matz (Yukihiro Matsumoto) wrote in #note-28:
The only help to the original issue is to make `make_sharable(proc)` an error, so we can prohibit accidental make_sharable(proc). In that case, the sharable_proc() idea can be considered later. But in the different issue.
I think this would be good for now, until we find a good solution, to not have block semantics broken until then, and some code using Ractor to rely on the wrong semantics (even more broken given that `Ractor.make_sharable(proc)` mutates `proc` inplace). --- I think something we might all agree on for now is:
* Don't allow `Ractor.shareable_proc` blocks to use any variable from the environment (like current `Ractor.new`)
This is already used for `Ractor.new`. It's safe, clearly and simply defined, and more permissive than requiring a literal block. If the Proc accesses captured variables it's in most cases problematic anyway: * if the captured variables are reassigned, those assignments will be ignored silently (the main concern in this issue) * the value of these captures variables need to be shareable (otherwise it's an exception, and if it wasn't an exception it would be a segfault by leaking an unshareable object to another Ractor). This means anyway around where the block is defined you would often need some changes to make these variable values shareable (e.g. `.freeze` or `Ractor.make_sharable(...)`). In some cases it won't be possible, e.g. if the value is meant to be mutable. I think we should use concrete examples to make progress. @tenderlovemaking told me he had some places in Rails he wants to use this, could you link them here? Do they use captured variables? Is any of them assigned more than once? I think quite a few Sinatra apps wouldn't work, here are some examples: * If the Sinatra app or block-to-be-made-shareable uses state via local variables or constants, it can't work with Ractors (because `todos` must be mutable for the app to work): ```ruby todos = [] get '/' do ul { todos.map { li it } } end post '/add' do todos << params redirect '/' end ``` * If the Sinatra app or block-to-be-made-shareable uses captured variables or constants you would need to make the values of them shareable unless they are already deeply frozen, e.g.: ```ruby values = [1, 2, 3] # needs `.freeze` or `Ractor.make_sharable(...)` get '/' do values.sum end ``` * If the block-to-be-made-shareable uses singleton methods defined on the outer `self`, or `@ivar` of the outer self it won't work (that one is not an issue for Sinatra as Sinatra already changes the `self` of `get/post/...` blocks: ```ruby def self.answer 42 end @foo = 43 Ractor.shareble_proc(self: nil) do answer # NoMethodError due to having to change to a different `self` @foo # nil with Ractor due to having to change to a different `self` end ``` I think if you want to make Sinatra to work with Ractor, for most apps it's not possible to make the blocks shareable for these reasons and others (related to the general incompatibility of Ractor and state). I think a solution that would work much more often would be to combine Namespace and Ractor, and adapt Ractor semantics to be "if the Namespace in which this code was loaded is owned by the current Ractor" instead of "if main Ractor" (e.g. for whether it's allowed to assign a constant, or to accesses constants with a non-shareable value). So you'd load the code once per Ractor, and each Ractor would have a copy of the app and be able to run it, including necessary mutations. Though for that purpose sub-interpreters or forking might be superior in many ways. But still it would make Ractor a lot more usable, at the expense of having to load the code multiple times. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114352 * Author: Eregon (Benoit Daloze) * Status: Closed * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/

Issue #21039 has been updated by Eregon (Benoit Daloze). jhawthorn (John Hawthorn) wrote in #note-27:
I really do not believe it is a departure from the existing semantics.
I think it is in many ways. It seems other committers see the problem as well. Nothing in Ruby gives these semantics currently, to shallow-copy the environment of a block (just that proves it's a departure from the existing semantics, it's behavior that is impossible without the currently-broken `Ractor.make_shareable(Proc)`). And I think for good reasons, I see it as breaking lexical scoping, and something as simple as `a = []; get("/") { a }; post("/add") { a << it }` can no longer rely on the `a` inside referencing the `a` outside, even though it must because it's defined there (in the outer scope). I would be fine with a lexical block and a clearly-named method (`Ractor.shareable_proc { }` or `Proc.isolated { }`, etc), because that would be a very good hint about new special semantics, and that block would be guaranteed to only be used with those semantics and not a mix. But altering the semantics of existing blocks, only in some conditions (e.g. when using `Ractor.make_shareable`) would I believe be a very big language design mistake. ---------------------------------------- Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks https://bugs.ruby-lang.org/issues/21039#change-114353 * Author: Eregon (Benoit Daloze) * Status: Closed * Assignee: ko1 (Koichi Sasada) ---------------------------------------- ```ruby def make_counter count = 0 nil.instance_exec do [-> { count }, -> { count += 1 }] end end get, increment = make_counter reader = Thread.new { sleep 0.01 loop do p get.call sleep 0.1 end } writer = Thread.new { loop do increment.call sleep 0.1 end } ractor_thread = Thread.new { sleep 1 Ractor.make_shareable(get) } sleep 2 ``` This prints: ``` 1 2 3 4 5 6 7 8 9 10 10 10 10 10 10 10 10 10 10 10 ``` But it should print 1..20, and indeed it does when commenting out the `Ractor.make_shareable(get)`. This shows a given block/Proc instance is concurrently broken by `Ractor.make_shareable`, IOW Ractor is breaking fundamental Ruby semantics of blocks and their captured/outer variables or "environment". It's expected that `Ractor.make_shareable` can `freeze` objects and that may cause some FrozenError, but here it's not a FrozenError, it's wrong/stale values being read. I think what should happen instead is that `Ractor.make_shareable` should create a new Proc and mutate that. However, if the Proc is inside some other object and not just directly the argument, that wouldn't work (like `Ractor.make_shareable([get])`). So I think one fix would to be to only accept Procs for `Ractor.make_shareable(obj, copy: true)`. FWIW that currently doesn't allow Procs, it gives `<internal:ractor>:828:in 'Ractor.make_shareable': allocator undefined for Proc (TypeError)`. It makes sense to use `copy` here since `make_shareable` effectively takes a copy/snapshot of the Proc's environment. I think the only other way, and I think it would be a far better way would be to not support making Procs shareable with `Ractor.make_shareable`. Instead it could be some new method like `isolated { ... }` or `Proc.isolated { ... }` or `Proc.snapshot_outer_variables { ... }` or so, only accepting a literal block (to avoid mutating/breaking an existing block), and that would snapshot outer variables (or require no outer variables like Ractor.new's block, or maybe even do `Ractor.make_shareable(copy: true)` on outer variables) and possibly also set `self` since that's anyway needed. That would make such blocks with different semantics explicit, which would fix the problem of breaking the intention of who wrote that block and whoever read that code, expecting normal Ruby block semantics, which includes seeing updated outer variables. Related: #21033 https://bugs.ruby-lang.org/issues/18243#note-5 Extracted from https://bugs.ruby-lang.org/issues/21033#note-14 -- https://bugs.ruby-lang.org/
participants (7)
-
Eregon (Benoit Daloze)
-
jhawthorn (John Hawthorn)
-
ko1 (Koichi Sasada)
-
luke-gru (Luke Gruber)
-
mame (Yusuke Endoh)
-
matz (Yukihiro Matsumoto)
-
tenderlovemaking (Aaron Patterson)