December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115909] [Ruby master Bug#20087] Uninitialized instance variable warning - obsolete documentation
by semaperepelitsa (Simon Perepelitsa) 28 Dec '23

28 Dec '23

Issue #20087 has been reported by semaperepelitsa (Simon Perepelitsa). ---------------------------------------- Bug #20087: Uninitialized instance variable warning - obsolete documentation https://bugs.ruby-lang.org/issues/20087 * Author: semaperepelitsa (Simon Perepelitsa) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This warning got removed from Ruby 3.0: https://bugs.ruby-lang.org/issues/17055 But the documentation (ruby/doc/syntax/assignment.rdoc@master) still says: > An uninitialized instance variable has a value of +nil+. If you run Ruby with > warnings enabled, you will get a warning when accessing an uninitialized > instance variable. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115969] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by ioquatix (Samuel Williams). > This proposal breaks Fiber semantics so fundamentally I don't disagree with you, and I don't have a strong opinion one way or another, but there is nothing about coroutine semantics which prevents them from being passed between threads which is what the MN implementation is already doing (it uses the same coroutines). So, I'd like to experiment with it. Alternatively, maybe we can re-implement `Enumerator` with `Thread+Queue` which is more flexible, at least that deals with one of the problems that is often brought up in these discussions. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105928 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115931] [Ruby master Bug#20097] Regexp#match? with empty capture group repeat is inconsistent
by tompng (tomoya ishida) 28 Dec '23

28 Dec '23

Issue #20097 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20097: Regexp#match? with empty capture group repeat is inconsistent https://bugs.ruby-lang.org/issues/20097 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) +MN [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- It sometimes matches and sometimes does not, depend on repeat quantifier number. ~~~ruby # empty capture group repeat p /(a(){1,4}){2}/.match?('aa') #=> true p /(a(){1,5}){2}/.match?('aa') #=> false # repeat of capture group that match to empty string p /(a(|){1,2}){2}/.match?('aa') #=> true p /(a(|){1,3}){2}/.match?('aa') #=> false p /(a(x?y?|z){1,2}){2}/.match?('aa') #=> false ~~~ non-capturing group seems ok. ~~~ruby p /(a(?:){1,4}){2}/.match?('aa') #=> true p /(a(?:){1,5}){2}/.match?('aa') #=> true p /(a(?:|){1,2}){2}/.match?('aa') #=> true p /(a(?:|){1,3}){2}/.match?('aa') #=> true p /(a(?:x?y?|z){1,2}){2}/.match?('aa') #=> true ~~~ Same behavior in ruby 3.0, 3.1, 3.2 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115965] [Ruby master Bug#20103] rb_internal_thread_remove_event_hook() hangs when called from within a Thread hook
by osyoyu (Daisuke Aritomo) 28 Dec '23

28 Dec '23

Issue #20103 has been reported by osyoyu (Daisuke Aritomo). ---------------------------------------- Bug #20103: rb_internal_thread_remove_event_hook() hangs when called from within a Thread hook https://bugs.ruby-lang.org/issues/20103 * Author: osyoyu (Daisuke Aritomo) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2023-12-28T14:53:15Z master e81a5453e3) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Problem The Ruby process hangs when `rb_internal_thread_remove_event_hook()` is called from within a Thread hook registered by `rb_internal_thread_add_event_hook()`. The thread is waiting for the thread event execution lock (`pthread_rwlock_wrlock(&rb_internal_thread_event_hooks_rw_lock))`), which is acquired by its caller, `rb_thread_execute_hooks()`. https://github.com/ruby/ruby/blob/e81a5453e3c76c4348da042d86debde7689254fe/… This situation would occur when one wants to register a *oneshot* hook that performs something on a Thread for only a single time. The hook would remove itself after its main procedure. In my case, I'm doing a timer_create(3) on each pthread for profiling purposes. ## Expected behavior The process won't hang, or at least print some kind of warning. I think there some options here: 1. Discourage this kind of definition. Thread hooks should not add/remove other Thread hooks, or otherwise do so through other async methods, such as postponed jobs. 2. Find a way to make this work. Maybe inner calls to add/remove_event_hook() can share locks with the outer call, but I'm not sure if that's possible as I'm not familiar with pthread_rwlocks. ## Repro ```c #include "ruby.h" #include "ruby/thread.h" rb_internal_thread_event_hook_t *hook; void event_hook_callback(rb_event_flag_t flag, const rb_internal_thread_event_data_t *data, void *custom_data) { rb_internal_thread_remove_event_hook(hook); // hangs } RUBY_FUNC_EXPORTED void Init_mycext(void) { hook = rb_internal_thread_add_event_hook( event_hook_callback, RUBY_INTERNAL_THREAD_EVENT_RESUMED, NULL ); } ``` ``` ruby -rmycext -e 'Thread.new { loop { 1 + 1 } }' ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115958] [Ruby master Feature#15667] Introduce malloc_trim(0) in full gc cycles
by hongli (Hongli Lai) 28 Dec '23

28 Dec '23

Issue #15667 has been updated by hongli (Hongli Lai). It looks like this ticket can be closed. I see that Ruby 3.3 makes use of malloc_trim. ---------------------------------------- Feature #15667: Introduce malloc_trim(0) in full gc cycles https://bugs.ruby-lang.org/issues/15667#change-105915 * Author: sam.saffron (Sam Saffron) * Status: Open * Priority: Normal ---------------------------------------- Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-… I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation. ---Files-------------------------------- ruby_gc_malloc_trim.patch (1011 Bytes) Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB) crash.png (85.9 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115957] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by Eregon (Benoit Daloze) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by Eregon (Benoit Daloze). bascule (Tony Arcieri) wrote in #note-10: > There's a simple solution to this: track if a given fiber is holding mutexes (e.g. keep a count of them) and if it is, make Fiber#resume raise an exception if it is resumed in a different thread from the one where it was originally yielded. > > That way you eliminate the nasty edge case, but still allow fibers which aren't holding mutexes (or whatever other synchronization primitives you're worried about) to be resumed in a different thread. This may work for ::Mutex, but it won't work for any other synchronization. For example any of the locks/barriers/countdown latch in concurrent-ruby. Or even locks in C extensions. > The same solution could work for thread-local storage: disallow fiber cross-thread fiber resumption if thread local storage is in use. I think this will be so limiting that it makes creating a Fiber which can migrate between threads almost always useless in practice as such a Fiber can't e.g. call a gem as that might use thread-local storage or synchronization internally. @shan Regarding Enumerator, only next/peek create a Fiber, all other methods are fine and do not need a Fiber. Using an Enumerator in a next/peek way from multiple threads is something that is fundamentally very tricky, and that Ruby currently does not really have a concept to support it. Basically you'd want a thread-safe Java Iterator(#hasNext/#next), but there is no such protocol in Ruby. That's really tricky, and few data structures can actually support that. I think it has very limited value, because next/peek is rare, and doing next/peek in threads concurrently seems asking for concurrency bugs and data races. A Queue seems a much better thing to use to get some elements on one thread and some on another. @ioquatix > I'll put together a PR so we can experiment. This proposal breaks Fiber semantics so fundamentally I think if such a feature is added it should have another name than Fiber, to make it clear it can migrate threads and is subject to all thread races. In fact since it would have the exact same concurrency semantics as Thread, so I think it should include thread in the name. BTW, now CRuby has M-N threads (#19842), is there any reason to try this over using `RUBY_MN_THREADS=1`? One could also simulate this by creating every Fiber in a Thread with `RUBY_MN_THREADS=1` as threads should be cheaper that way. That also avoids all the problems of a Fiber changing threads dynamically, while not restricting to run a Fiber on a given thread. Migrating a Fiber to another threads seems to also have little value on CRuby due to the GVL, as it won't improve parallelism. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105914 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115956] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by ioquatix (Samuel Williams). One issue was the use of the copy-coroutine. As we've removed this and replaced it with a pthread fallback, we can now potentially allow fibers to be resumed across threads. I'll put together a PR so we can experiment. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105913 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115953] [Ruby master Feature#18035] Introduce general model/semantic for immutability.
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #18035 has been updated by ioquatix (Samuel Williams). There is a PR here that mostly just works: https://github.com/ruby/ruby/pull/4879 However, additional work would be required to: 1. Tidy up the implementation and 2. Mark some core classes as Immutable as outlined in https://gist.github.com/eregon/bce555fa9e9133ed27fbfc1deb181573 ---------------------------------------- Feature #18035: Introduce general model/semantic for immutability. https://bugs.ruby-lang.org/issues/18035#change-105910 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- It would be good to establish some rules around mutability, immutability, frozen, and deep frozen in Ruby. I see time and time again, incorrect assumptions about how this works in production code. Constants that aren't really constant, people using `#freeze` incorrectly, etc. I don't have any particular preference but: - We should establish consistent patterns where possible, e.g. - Objects created by `new` are mutable. - Objects created by literal are immutable. We have problems with how `freeze` works on composite data types, e.g. `Hash#freeze` does not impact children keys/values, same for Array. Do we need to introduce `freeze(true)` or `#deep_freeze` or some other method? Because of this, frozen does not necessarily correspond to immutable. This is an issue which causes real world problems. I also propose to codify this where possible, in terms of "this class of object is immutable" should be enforced by the language/runtime, e.g. ```ruby module Immutable def new(...) super.freeze end end class MyImmutableObject extend Immutable def initialize(x) @x = x end def freeze return self if frozen? @x.freeze super end end o = MyImmutableObject.new([1, 2, 3]) puts o.frozen? ``` Finally, this area has an impact to thread and fiber safe programming, so it is becoming more relevant and I believe that the current approach which is rather adhoc is insufficient. I know that it's non-trivial to retrofit existing code, but maybe it can be done via magic comment, etc, which we already did for frozen string literals. Proposed PR: https://github.com/ruby/ruby/pull/4879 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112446] [Ruby master Bug#19442] Remove USE_RINCGC flag
by eightbitraptor (Matthew Valentine-House) 28 Dec '23

28 Dec '23

Issue #19442 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19442: Remove USE_RINCGC flag https://bugs.ruby-lang.org/issues/19442 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- [GitHub PR #7317](https://github.com/ruby/ruby/pull/7313) Ruby doesn't compile when USE_RINCGC is disabled. It's also not being tested in CI. As @nobu has pointed out in comments on the PR, fixing it is simple. I think there are 2 approaches we could take: 1. Remove `USE_RINCGC` entirely and always run with incremental GC enabled. We have a precedent for this: `USE_RGENGC=0` was removed [in this commit](https://github.com/ruby/ruby/commit/62c2b8c74e47652fc5bbaf6150f4acd… almost 3 years ago. `USE_RINCGC=0` is in a similar state. It has been broken for almost a year (since [this commit](https://github.com/ruby/ruby/commit/dde164e968e382d50b07ad455946888…) and [disabled in CI for more than 2 years](https://github.com/ruby/ruby/commit/46d3ea2c2569e2e5a9ee3e7e206f07f0…. 2. We could fix `USE_RINCGC`. If we do this we should re-enable the CI task so that we continue to keep it working in the future. I am very much in favour of option 1, because I don't think this flag is being actively used, and removing it will simplify the code and reduce the cognitive overhead of working with the GC. However I am happy to defer to the team here. -- https://bugs.ruby-lang.org/

6 9

[ruby-core:115879] [Ruby master Bug#20081] Transfered Fiber doesn't return to Fiber that started it
by rmosolgo (Robert Mosolgo) 27 Dec '23

27 Dec '23

Issue #20081 has been reported by rmosolgo (Robert Mosolgo). ---------------------------------------- Bug #20081: Transfered Fiber doesn't return to Fiber that started it https://bugs.ruby-lang.org/issues/20081 * Author: rmosolgo (Robert Mosolgo) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi! I'm trying to figure out how to make sure that Fibers started with `.transfer` end up _terminated_, not just suspended. (If they're suspended, Rails thinks they're still alive, and they continue to hold onto database connections, see: https://github.com/rmosolgo/graphql-ruby/issues/4739#issuecomment-1866930914.) So, I'm looking for way to make sure that any Fiber I start with `.transfer` will be properly terminated. But what I noticed is that when a transfer-based Fiber terminates, it gives control back to the top-most Fiber, not the Fiber which transfered to it. Is this intended? Here's a script to replicate the issue: ```ruby manager = Fiber.new do parent = Fiber.current worker = Fiber.new do puts "2. Begin Worker" parent.transfer puts "4. End Worker" end puts "1. Transfer 1" worker.transfer puts "3. Transfer 2" worker.transfer puts "5. Finished manager" end manager.transfer puts "6. Finished script" ``` I expect the steps to print in order: ``` 1. Transfer 1 2. Begin Worker 3. Transfer 2 4. End Worker 5. Finished manager 6. Finished script ``` But instead, `5. ...` is skipped: ``` 1. Transfer 1 2. Begin Worker 3. Transfer 2 4. End Worker 6. Finished script ``` I think that's because my `worker` fiber terminates and passes control back to the top-level Fiber. Should it have passed control back to the `manager`? Or is there another way to make sure `worker` is terminated, and `manager` gets control? -- https://bugs.ruby-lang.org/

2 4