[ruby-core:123042] [Ruby Feature#21550] Ractor.sharable_proc/sharable_lambda to make sharable Proc object
Issue #21550 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. Captured outer variables follow the current `Ractor.make_shareable` semantics: * If a captured outer local variable refers to a shareable object, a shareable Proc may read it. * If any captured outer variable refers to a non‑shareable object, creating the shareable Proc raises an error. ```ruby a = 42 b = "str" Ractor.sharalbe_proc{ p a #=> 42 } Ractor.sharalbe_proc{ # error when making a sharealbe proc p b #=> 42 } ``` * The captured outer local variables are copied by value when the shareable Proc is created. Subsequent modifications of those variables in the creator scope do not affect the Proc. ``` a = 42 shpr = Ractor.sharable_proc{ p a } a = 0 shpr.call #=> 42 ``` ``` * Assigning to outer local variables from within the shareable Proc is not allowed (error at creation). ```ruby a = 42 Ractor.shareable_proc{ # error when making a sharealbe proc a = 43 } ``` More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 1. No problem to change the @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by matz (Yukihiro Matsumoto). I understand the concern for future confusion, but it's a trade-off. I'd accept confusion here (option 1) to avoid complex semantics and implementation. Matz. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114368 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. Captured outer variables follow the current `Ractor.make_shareable` semantics: * If a captured outer local variable refers to a shareable object, a shareable Proc may read it. * If any captured outer variable refers to a non‑shareable object, creating the shareable Proc raises an error. ```ruby a = 42 b = "str" Ractor.sharalbe_proc{ p a #=> 42 } Ractor.sharalbe_proc{ # error when making a sharealbe proc p b #=> 42 } ``` * The captured outer local variables are copied by value when the shareable Proc is created. Subsequent modifications of those variables in the creator scope do not affect the Proc. ```ruby a = 42 shpr = Ractor.sharable_proc{ p a } a = 0 shpr.call #=> 42 ``` * Assigning to outer local variables from within the shareable Proc is not allowed (error at creation). ```ruby a = 42 Ractor.shareable_proc{ # error when making a sharealbe proc a = 43 } ``` More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). Summary: I think option 2 is great it's both flexible, clear and safe, and we should implement it. It addresses the concerns in #21039. It should be easy to implement, because if given a Proc it's the same semantics as `Ractor.new(proc_object)`, and when given a literal block it's the desired more flexible semantics which anyway both options want. What follow is some more thoughts, which might lead to some improvements later but anyway I believe it's fine to implement option 2 as-is. --- I wonder if the ability to capture outer local variables for shareable procs is even needed at all. Because that only works if the value of the captured variables is shareable, which seems not so likely (only numbers/true/false/nil/frozen-string-literals/modules/regexps are frozen/shareable without an explicit freeze/make_shareable call). I would like to see some concrete examples of blocks to be made shareable which capture outer local variables, and how they handle the case of captured variable values not already being deeply frozen (cc @tenderlovemaking you said there were some in Rails IIRC). Related to that, maybe it would be useful for ```ruby o = Object.new Ractor.shareable_proc { o } ``` to call `Ractor.make_shareable(o)`? Or maybe `Ractor.make_shareable(o, copy: true)` on `Ractor.shareable_proc(copy: true) { ... }`? Though not sure if reasonable or too surprising. It might be surprising for cases where it freezes many objects, e.g. if `o` is `Foo.new([])` the Array also gets frozen (which might break things). I think that case makes it clear this might maybe be acceptable with a literal block, but would never be acceptable with a Proc object (it would freeze things far away and be hard to debug). In some way we can see a block as an object with an extra `@ivar` being the "captured environment" which contains the captured variables. From that POV, making the Proc shareable then would rather naturally make that "captured environment" and objects inside shareable as well. --- I find it interesting that we all seem to agree on `Assigning to outer local variables from within the shareable Proc is not allowed (error at creation)` but not on `Assigning to outer local variables from outside the shareable Proc is not allowed through an exception when trying to create a shareable Proc in such a case`. IMO they are just both sides of the same issue. But I understand the second one is more tricky because e.g. it's impossible to detect assignments within eval at (file) parse time (IMO so rare of an edge case to not worry about that too much). I also agree the second one is more rarely used, though that doesn't mean it doesn't matter. --- With option 1 it would break existing blocks, so I am against it, e.g.: ```ruby visits = 0 get '/' do # assuming this block is not made shareable (because that is forbidden, see `Assigning to outer local variables from within the shareable Proc is not allowed (error at creation)`) visits += 1 "Hello" end get '/visits' do # if this block is made shareable and called, it will behave incorrectly and use a snapshot of `visits` visits.to_s end ``` Concretely, assuming the second block is made shareable at load time, if the block is used only in non-main Ractors but not in main Ractors: * if the request hits the main Ractor it would be correct (the number of visits of `/`) * if it hits a non-main Ractor it would be incorrect (0). That's pretty bad for obvious reasons. If the shared block is instead used by all Ractors, including the main one then it would always be incorrect (0). Still clearly breaking the program/intended semantics. So standard Ruby code, specifically blocks (and their intended logic) would be broken "just because a Proc has been made Ractor-shareable". This is worse than just `instance_exec`, because a block called with `instance_exec` is typically always called with the same kind of receiver. And the worse case with `instance_exec` is a NoMethodError or calling the wrong method (very unlikely). The worse case with option 1 is reading a stale/outdated/inconsistent value, very much like a stale read which is usually a race condition/multithreading problem, but here Ractor would actually introduce this issue even though Ractor is meant to prevent such issues. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114378 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. Captured outer variables follow the current `Ractor.make_shareable` semantics: * If a captured outer local variable refers to a shareable object, a shareable Proc may read it. * If any captured outer variable refers to a non‑shareable object, creating the shareable Proc raises an error. ```ruby a = 42 b = "str" Ractor.sharalbe_proc{ p a #=> 42 } Ractor.sharalbe_proc{ # error when making a sharealbe proc p b #=> 42 } ``` * The captured outer local variables are copied by value when the shareable Proc is created. Subsequent modifications of those variables in the creator scope do not affect the Proc. ```ruby a = 42 shpr = Ractor.sharable_proc{ p a } a = 0 shpr.call #=> 42 ``` * Assigning to outer local variables from within the shareable Proc is not allowed (error at creation). ```ruby a = 42 Ractor.shareable_proc{ # error when making a sharealbe proc a = 43 } ``` More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by ko1 (Koichi Sasada). Eregon (Benoit Daloze) wrote in #note-6:
I wonder if the ability to capture outer local variables for shareable procs is even needed at all. Because that only works if the value of the captured variables is shareable, which seems not so likely (only numbers/true/false/nil/frozen-string-literals/modules/regexps are frozen/shareable without an explicit freeze/make_shareable call). I would like to see some concrete examples of blocks to be made shareable which capture outer local variables, and how they handle the case of captured variable values not already being deeply frozen (cc @tenderlovemaking you said there were some in Rails IIRC).
I have two examples. One is `define_method`. ```ruby def define_foo suffix, ivar_name # should be in Symbol define_method "foo_#{mid} do instance_variable_get ivar_name end end ``` This method define a new method by using a outer local variable `ivar_name`. For example, this pattern is used: ```ruby # lib/delegate.rb def Delegator.delegating_block(mid) # :nodoc: lambda do |*args, &block| target = self.__getobj__ target.__send__(mid, *args, &block) end.ruby2_keywords end # lib/bundler/errors.rb def self.status_code(code) define_method(:status_code) { code } # repl_type_completor/test/repl_type_completor/test_repl_type_completor.rb def with_failing_method(klass, method_name, message) ... # message should be marked as sharable klass.define_method(method_name) do |*, **| raise Exception.new(message) end # debug/test/console/config_fork_test.rb ['fork', 'Process.fork', 'Kernel.fork'].each{|fork_method| c = Class.new ConsoleTestCase do ... # fork_method shouldbe marked as sharable define_method :fork_method do fork_method end end ``` The first motivation is how to handle `define_method`. Another example to make a task object which should be run in another Ractor. result = Ractor::Port.new Ractor.sharable_proc do result << task() end p result.receive ``` I have already written the following pattern many times: ```ruby port = Ractor::Port.new Ractor.new port do |port| port << ... end ``` and it is somewhat cumbersome to write. ---- At least we have no objection to introduce `Ractor.shareable_proc(&bl)` if `bl` doesn't have references to the outer variable. I'll merge it. ---- I have another idea. Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. *1: ``` # {name: foo}, 3] means shadowed name "foo" is used 3 times. [[[:FILES, 3_068_818], [:FAILED_PARSE, 14_928]], [[:lvar, 78], [{name: :glark}, 6], [[:FAILED, SystemStackError], 5], [{name: :bl}, 4], [{name: :a}, 4], [{name: :options}, 3], [{name: :name}, 3], [{name: :bytes}, 2], [{name: :extra}, 2], [{name: :shape}, 2], [{name: :b}, 2], [{name: :x}, 2], [{name: :new_root}, 2], [{name: :bl2}, 2], [{name: :ifc_spc}, 2], [{name: :m}, 2], [{name: :foo}, 2], [{name: :key}, 2], [{name: :md}, 2], [{name: :d}, 2], [{name: :c}, 2], [{name: :out}, 2], [{name: :bar}, 2], [{name: :req}, 1], [{name: :resp}, 1], [{name: :spec}, 1], [{name: :world}, 1], [{name: :u}, 1], [{name: :onto}, 1], [{name: :rbname}, 1], [{name: :r}, 1], [{name: :h}, 1], [{name: :bug9605}, 1], [{name: :expected}, 1], [{name: :result}, 1], [{name: :services}, 1], [{name: :step}, 1], [{name: :obj}, 1], [{name: :ancestor}, 1], [{name: :count}, 1], [{name: :path}, 1], [{name: :nav}, 1], [{name: :safe_position_in_input}, 1], [{name: :safe_count}, 1], [{name: :v}, 1], [{name: :k}, 1], [{name: :indent}, 1], [{name: :y}, 1], [{name: :z_diff}, 1]]] ``` ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114397 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. Captured outer variables follow the current `Ractor.make_shareable` semantics: * If a captured outer local variable refers to a shareable object, a shareable Proc may read it. * If any captured outer variable refers to a non‑shareable object, creating the shareable Proc raises an error. ```ruby a = 42 b = "str" Ractor.sharalbe_proc{ p a #=> 42 } Ractor.sharalbe_proc{ # error when making a sharealbe proc p b #=> 42 } ``` * The captured outer local variables are copied by value when the shareable Proc is created. Subsequent modifications of those variables in the creator scope do not affect the Proc. ```ruby a = 42 shpr = Ractor.sharable_proc{ p a } a = 0 shpr.call #=> 42 ``` * Assigning to outer local variables from within the shareable Proc is not allowed (error at creation). ```ruby a = 42 Ractor.shareable_proc{ # error when making a sharealbe proc a = 43 } ``` More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by ko1 (Koichi Sasada). I made a patch to disallow any outer variables and I found that I can't write ostruct case: ```ruby # ostruct-0.6.3/lib/ostruct.rb def new_ostruct_member!(name) # :nodoc: unless @table.key?(name) || is_method_protected!(name) if defined?(::Ractor.shareable_proc) getter_proc = Ractor.shareable_proc { @table[name] } setter_proc = Ractor.shareable_proc {|x| @table[name] = x} elsif defined?(::Ractor) getter_proc = nil.instance_eval{ Proc.new { @table[name] } } setter_proc = nil.instance_eval{ Proc.new {|x| @table[name] = x} } ::Ractor.make_shareable(getter_proc) ::Ractor.make_shareable(setter_proc) else getter_proc = Proc.new { @table[name] } setter_proc = Proc.new {|x| @table[name] = x} end define_singleton_method!(name, &getter_proc) define_singleton_method!("#{name}=", &setter_proc) end end ``` So I think it is needed to allow the way to access outer variables. Option 2 (allow accesses only on lexical blocks) is rejected by Matz, so I want to introduce 1 or 3. 1 is better for me... ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114401 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by tenderlovemaking (Aaron Patterson). ko1 (Koichi Sasada) wrote in #note-9:
I made a patch to disallow accessing to any outer variables and I found that I can't write ostruct case:
```ruby # ostruct-0.6.3/lib/ostruct.rb
def new_ostruct_member!(name) # :nodoc: unless @table.key?(name) || is_method_protected!(name) if defined?(::Ractor.shareable_proc) getter_proc = Ractor.shareable_proc { @table[name] } setter_proc = Ractor.shareable_proc {|x| @table[name] = x} elsif defined?(::Ractor) getter_proc = nil.instance_eval{ Proc.new { @table[name] } } setter_proc = nil.instance_eval{ Proc.new {|x| @table[name] = x} } ::Ractor.make_shareable(getter_proc) ::Ractor.make_shareable(setter_proc) else getter_proc = Proc.new { @table[name] } setter_proc = Proc.new {|x| @table[name] = x} end define_singleton_method!(name, &getter_proc) define_singleton_method!("#{name}=", &setter_proc) end end ```
So I think it is needed to allow the way to access outer variables. Option 2 (allow accesses only on lexical blocks) is rejected by Matz, so I want to introduce 1 or 3. 1 is better for me...
I think 3 is a good idea, but I think we could introduce it in later versions of Ruby and slowly migrate code (if the syntax change is acceptable). Option 1 lets us more easily port existing code without syntax changes, so I like it for now. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114402 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). ko1 (Koichi Sasada) wrote in #note-9:
Option 2 (allow accesses only on lexical blocks) is rejected by Matz, so I want to introduce 1 or 3.
Where did matz say that? In https://bugs.ruby-lang.org/issues/21550#note-5 he said:
I'd accept confusion here (option 1) to avoid complex semantics and implementation.
`I'd accept` sounds like a preference, not rejection. As I clarified in my comment after that, I see no concern about complex implementation, it's the same effort anyway:
It should be easy to implement, because if given a Proc it's the same semantics as `Ractor.new(proc_object)`, and when given a literal block it's the desired more flexible semantics which anyway both options want.
And regarding complex semantics (due to different behavior with literal vs non-literal block) it's necessary to avoid breaking block semantics, which are way more fundamental than `Ractor.shareable_proc` semantics. `Kernel#lambda` already had such semantics BTW (no effect if not a literal block), and in fact for similar reasons and concerns (don't break the author of the block's intent). So there is precedent for doing exactly this kind of switching on literal block vs Proc. Option 2 seems the obvious and correct solution for the ostruct case. Though it seems that `define_method` is a frequent issue that keeps coming up with Ractor, and that's quite a specific case where we know the Proc `self` will be changed anyway. So I think `define_method(name) { @table[name] }` / `define_singleton_method(name) { @table[name] }` should just work, with and without Ractors. That would mean that `define_method` would automatically shallow-copy the environment, Ractor or not, for consistency. That's a semantic change but it seems very compatible. And `define_method` knows `self` will be the instance so no need to check if `self` is shareable. And the defined method would automatically be available to Ractors, as long as the captured variables values are shareable. If they are not, then the method can't be called on a Ractor, that's impossible anyway, same error as currently. That would mean `define_method` would just work with Ractor, vs needing pretty messy code as in your example. "if Ractor" code should be avoided as much as possible. If we are too concerned about changing `define_method` semantics we could instead make that behavior opt-in e.g. via a keyword argument to `define_method` like `define_method(name, ractor/shareable/make_shareable: true) { @table[name] }`. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114403 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). Option 3 doesn't seem good because it would break the block if the block is run on Ruby 3.4 and before, as the value of `a` would be `nil`. One would need to have 2 copies of the block which is clearly inconvenient. I think Ractor should be able to use captured variables, because this is one of the most elegant ways to pass data/input in Ruby. But it should be safe, and that means not breaking normal Ruby block semantics (at least for blocks which don't explicitly opt-in to the environment copy behavior). So I think Option 4 would still be the best, and that's the option proposed in #21039: * `Ractor.sharable_proc` is the same for literal and non-literal blocks * `Ractor.sharable_proc` raises if it captures a variable which is reassigned inside or after the block/Proc given to `Ractor.sharable_proc`. Notice we already agree on "`Ractor.sharable_proc` raises if it captures a variable which is reassigned inside the block/Proc given to `Ractor.sharable_proc`". So it's just adding `or after`, which makes it fully safe and compatible with regular block semantics. I don't think that check is very difficult to implement, in fact I can help implementing it. Ruby code to make it clear: ```ruby # OK because does not change the semantics of b def example a = 1 a += 2 b = proc { a } Ractor.sharable_proc(&b) end # OK because does not change the semantics of the block def example a = 1 a += 2 Ractor.sharable_proc { a } end # error (that everyone seems to agree on) def example a = 1 b = proc { a = 2 } Ractor.sharable_proc(&b) end # error (that everyone seems to agree on) def example a = 1 Ractor.sharable_proc { a = 2 } end # error (the case we discussing about): the code clearly assumes it can reassigns `a` but the `sharable_proc` would not respect it, i.e. `sharable_proc` would break Ruby block semantics # Also note the Ractor.sharable_proc call might be far away from the block, so one can't tell when looking at the block that it would be broken by `sharable_proc` (if no error for this case) def example a = 1 b = proc { a } Ractor.sharable_proc(&proc { a }) a = 2 end # I think should be error too, semantics are ill-defined in such a case, the code clearly assumes it can reassigns `a` but the `sharable_proc` would not respect it def example a = 1 Ractor.sharable_proc { a } a = 2 end ``` That check can be more strict for convenience (e.g. raise if the assignments are not trivially all before the block/Proc), I think that's fine because it's fairly rare to reassign a variable after a block captures it, so won't be a practical limitation anyway. But still, such cases should still behave correctly according to block semantics, hence should be forbidden for `Ractor.sharable_proc`. Re `eval` and `binding` it's so rare and such a corner case in combination with reassigning a variable after a block captures it that I think it is acceptable to expose that `Ractor.sharable_proc` copies the environment for those extremely rare cases. BTW, Proc#binding is already not supported for a `sharable_proc`: ``` $ ruby -e 'nil.instance_exec { a = 1; b = proc { a }; b2 = Ractor.make_shareable(b); p b2.binding }' -e:1:in `binding': Can't create Binding from isolated Proc (ArgumentError) ``` So `binding`/`eval` is in general already not fully respected with Ractor anyway. Some edge cases for clarity: ```ruby # OK, we cannot detect it, extreme corner case unlikely to appear in any real code. The sharable_proc will capture 1, b will capture 2. def example a = 0 b = proc { a } p = Ractor.sharable_proc(&proc { a }) eval("a = 2") # or binding.local_variable_set(:a, 2), or b.binding.local_variable_set(:a, 2) [b, p] end # error, `a` is reassigned after the block def example a = 0 while condition b = proc { a } Ractor.sharable_proc(&proc { a }) a += 1 end end # error, `a` might be reassigned (if condition is true twice or more, but we have to analyze statically so be conservative) def example while condition a = rand b = proc { a } Ractor.sharable_proc(&proc { a }) end end ``` As said before, I'm OK with option 2, but it's less flexible than option 4. We could also have a mix and allow everything for literal block, and option 4 for Proc. I remain strongly against option 1, I think it is a language design mistake we won't be able to fix later. And for `define_method` I think we should have something more convenient than `Ractor.shareable_proc` (see previous comment). ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114404 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). Regarding `define_method`, it's as-if it would try `Ractor.shareable_proc` with option 4 semantics automatically on the block/Proc, and if that succeeds use that, if it fails then make the method not callable from a Ractor and use block/Proc as-is. So that would mean no compatibility issue if a captured variable is reassigned inside or after the block, and for all other cases it just works with Ractor. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114405 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by ko1 (Koichi Sasada). @Eregon let's me clarify your proposal. * Option 4 is your proposal on https://bugs.ruby-lang.org/issues/21039#note-21, which prohibit any writing to the captured outer variables from inside/outside of the block * This proposal is clearly rejected by Matz https://bugs.ruby-lang.org/issues/21039#note-28 * About `define_method` * > That would mean that define_method would automatically shallow-copy the environment, Ractor or not, for consistency. That's a semantic change but it seems very compatible. * I think it is not acceptable because it will break compatibility if the block uses outer local variables as a storage, like that: ```ruby def counters suffix cnt = 0 define_method("get_#{suffix}){ cnt } define_method("inc_#{suffix}){ cnt += 1} end ``` I didn't count such cases. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114412 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by ko1 (Koichi Sasada). I confirmed with Matz that option 2 is too complex and therefore rejected. I also confirmed with Matz that option 1 is preferable: not the best, but the better one among these. (I met with Matz today and discussed) So I will merge option 1 soon. Please reopen if there are further points. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114416 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). ko1 (Koichi Sasada) wrote in #note-14:
@Eregon let's me clarify your proposal.
* Option 4 is your proposal on https://bugs.ruby-lang.org/issues/21039#note-21, which prohibit any writing to the captured outer variables from inside/outside of the block
I think there is a misunderstanding there: Option 4 does *not* prohibit writing to captured variables, it never proposed that. Ruby code can write to all local variables, always. Maybe the text in that comment is confusing (e.g. "Disallow writes" is not to literally disallow writing, it's to prevent making a shareable proc in such a context). Please read https://bugs.ruby-lang.org/issues/21550#note-12 and ignored that older comment, it should be very clear what is proposed with all the examples. It prevents making a `shareable_proc` for a block which uses captured variables which are reassigned *after* the block, as I showed in the examples above: ```ruby def example a = 1 b = proc { a } Ractor.shareable_proc(&b) # should be Ractor::IsolationError: cannot isolate a block because it accesses outer variables (a) which are reassigned inside or after the block a = 2 end ``` All options already prevent making a `shareable_proc` for a block which uses captured variables which are reassigned *inside* the block. ```ruby # error (that everyone seems to agree on) def example a = 1 b = proc { a = 2 } Ractor.shareable_proc(&b) end ``` So option 4 does the same but also prevents making a shareable proc when reassigned *after*.
* This proposal is clearly rejected by Matz https://bugs.ruby-lang.org/issues/21039#note-28
I think the proposal was misunderstood, I tried to clarify in https://bugs.ruby-lang.org/issues/21039#note-33 as well. "prohibiting outer assignment to local variables when a proc is made sharable." was never proposed (because that's clearly too difficult and the wrong place to prevent/prohibit).
* About `define_method` * > That would mean that define_method would automatically shallow-copy the environment, Ractor or not, for consistency. That's a semantic change but it seems very compatible. * I think it is not acceptable because it will break compatibility if the block uses outer local variables as a storage, like that:
My last comment explains how to fix this. Since it's not clear I will make a separate proposal which does not rely on any previous comment to minimize confusion. ---------------------------------------- Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114419 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (sharable_proc) worker << task task = (sharable_proc) worker << task task = (sharable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.sharable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). ko1 (Koichi Sasada) wrote in #note-15:
So I will merge option 1 soon. Please reopen if there are further points.
Please wait, I believe you and matz did not understand my proposal. I will make a clear proposal ASAP. ---------------------------------------- Feature #21550: Ractor.shareable_proc/shareable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114421 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (shareable_proc) worker << task task = (shareable_proc) worker << task task = (shareable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.shareable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by ko1 (Koichi Sasada).
I think there is a misunderstanding there: Option 4 does not prohibit writing to captured variables, it never proposed that. Ruby code can write to all local variables, always.
Ok I misunderstood. ```ruby def sh(bl) Ractor.sharable_proc(&bl) #=> raise an error because threre is `a = 2` line. end a = 1 shpr = sh proc{ a } a = 2 ``` Is it correct? To make it we need to track local assignment outside of the block. Is that feasible...? ---------------------------------------- Feature #21550: Ractor.shareable_proc/shareable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114423 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (shareable_proc) worker << task task = (shareable_proc) worker << task task = (shareable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.shareable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). Yes, that's the idea. Yes it's feasible via static analysis (e.g. in `compile.c`). It would be somewhat conservative but I think that's good enough, I'll give more details in my proposal. I think a possible simple way is to use "if a captured variable is assigned more than once, `Ractor.sharable_proc` is `Ractor::IsolationError` on a block using that captured variable". ---------------------------------------- Feature #21550: Ractor.shareable_proc/shareable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114424 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (shareable_proc) worker << task task = (shareable_proc) worker << task task = (shareable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.shareable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
Issue #21550 has been updated by Eregon (Benoit Daloze). I wrote my detailed proposal in #21557. ---------------------------------------- Feature #21550: Ractor.shareable_proc/shareable_lambda to make sharable Proc object https://bugs.ruby-lang.org/issues/21550#change-114466 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Target version: 3.5 ---------------------------------------- Let's introduce a way to make a sharable Proc. * `Ractor.shareable_proc(self: nil, &block)` makes proc. * `Ractor.shareable_lambda(self: nil, &block)` makes lambda. See also: https://bugs.ruby-lang.org/issues/21039 ## Background ### Motivation Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor: ```ruby worker = Ractor.new do while task = Ractor.receive task.call(...) end end task = (shareable_proc) worker << task task = (shareable_proc) worker << task task = (shareable_proc) worker << task ``` There are various ways to represent a task, but using a Proc is straightforward. However, to make a Proc shareable today, self must also be shareable, which leads to patterns like: ```ruby nil.instance_eval{ Proc.new{ ... } } ``` This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly. ## Specification * `Ractor.shareable_proc(self: nil, &block)` makes a proc. * `Ractor.shareable_lambda(self: nil, &block)` makes a lambda. Both methods create the Proc/lambda with the given self and make the resulting object shareable. (changed) Accessing outer variables are not allowed. An error is raised at the creation. More about outer-variable handling are discussed below. In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants. This proposal does not change the semantics of Ractor.make_shareable() itself. ## Discussion about outer local variables [Feature #21039] discusses how captured variables should be handled. I propose two options. ### 0. Disallow accessing to the outer-variables It is simple and no confusion. ### 1. No problem to change the outer-variable semantics @Eregon noted that the current behavior of `Ractor.make_shareable(proc_obj)` can surprise users. While that is understandable, Ruby already has similar *surprises*. For instance: ```ruby RSpec.describe 'foo' do p self #=> RSpec::ExampleGroups::Foo end ``` Here, `self` is implicitly replaced, likely via `instance_exec`. This can be surprising if one does not know self can change, yet it is accepted in Ruby. We view the current situation as a similar kind of surprise. ### 2. Enforce a strict rule for non‑lexical usage The difficulty is that it is hard to know which block will become shareable unless it is lexically usage. ```ruby # (1) On this code, it is clear that the block will be shareable block: a = 42 Ractor.shareable_proc{ p a } # (2) On this code, it is not clear that the block becomes sharable or not get path do p a end # (3) On this code, it has no problem because get '/hello' do "world" end ``` The idea is to allow accessing captured outer variables only for lexically explicit uses of `Ractor.shareable_proc` as in (1), and to raise an error for non‑lexical cases as in (2). So the example (3) is allowed if the block becomes sharable or not. The strict rule is same as `Ractor.new` block rule. ### 3. Adding new rules (quoted from https://bugs.ruby-lang.org/issues/21550#note-7) Returning to the issue: we want a way to express that, within a block, an outer variable is shadowed while preserving its current value. We already have syntax to shadow an outer variable using `|i; a|`, where `a` is shadowed in the block and initialized to `nil` (just like a normal local variable). ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> nil ``` What if we instead initialized the shadowed variable to the outer variable's current value? ```ruby a = 42 pr = proc{|;a| p a} a = 43 pr.call #=> 42 ``` For example, we can write the port example like that: ```ruby port = Ractor::Port.new Ractor.new do |;port| port << ... end ``` and it is better (shorter). Maybe only few people know this spec and I checked that there are few lines in rubygems (78 cases in 3M files)(*1). So I think there is a few compatibility impact. -- https://bugs.ruby-lang.org/
participants (4)
-
Eregon (Benoit Daloze) -
ko1 (Koichi Sasada)
-
matz (Yukihiro Matsumoto)
-
tenderlovemaking (Aaron Patterson)