December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115980] [Ruby master Misc#17815] Snapcraft Ruby plugin
by paddor (Patrik Wenger) 31 Dec '23

31 Dec '23

Issue #17815 has been updated by paddor (Patrik Wenger). This can be closed. I've implemented a Ruby build plugin on v1 and v2. ---------------------------------------- Misc #17815: Snapcraft Ruby plugin https://bugs.ruby-lang.org/issues/17815#change-105943 * Author: paddor (Patrik Wenger) * Status: Open * Priority: Normal ---------------------------------------- I'm working on a Ruby build plugin for the Snapcraft v2 plugin AP, since the v1 Ruby plugin does not work on the Ubuntu `core20` image. The v2 API only allows to influence the build step, which means there's no nice way for the plugin to set environment variables that apply during runtime. The problem is that the final paths to Ruby's stdlib are unknown during compile time. I've got it to work by renaming the `ruby` executable to `ruby.bare` and creating a wrapper script `ruby` that sets the `$RUBYLIB` and `$GEM_PATH` env variables based on `$SNAP` before `exec`ing `ruby.bare`. I don't like that it requires a wrapper script though, since it would have to awkwardly determine the arch-specific stdlib directory to support any platform. The current wrapper script is this: ```sh #!/bin/sh export RUBYLIB="$SNAP/lib/ruby/snap:$SNAP/lib/ruby/snap/x86_64-linux:$RUBYLIB" export GEM_PATH="$SNAP/lib/ruby/gems/snap:$GEM_PATH" exec `dirname $0`/ruby.bare "$@" ``` What possible solutions are there? I could maybe fix the shebangs of Ruby executables like `gem` and `bundle`, but not `ruby` itself since it's binary. Also, I'm not sure if env variable expansion even works in shebang. Can the configure script's `--with-search-path` be made to support env variable expansion during runtime? That still wouldn't solve the issue about the arch-specific directory. Python apparently determines its load paths relative to the executable being run. Does Ruby support anything like that? Any other ideas? Current state: https://github.com/paddor/snapcraft-ruby-plugin-v2 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112010] [Ruby master Bug#19375] File objects are currently shareable
by luke-gru (Luke Gruber) 29 Dec '23

29 Dec '23

Issue #19375 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19375: File objects are currently shareable https://bugs.ruby-lang.org/issues/19375 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I don't know the internals of file.c but I don't think files are thread-safe. -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115919] [Ruby master Feature#20093] Syntax or keyword to reopen existing classs/modules, never to define new classs/modules
by tagomoris (Satoshi Tagomori) 29 Dec '23

29 Dec '23

Issue #20093 has been reported by tagomoris (Satoshi Tagomori). ---------------------------------------- Feature #20093: Syntax or keyword to reopen existing classs/modules, never to define new classs/modules https://bugs.ruby-lang.org/issues/20093 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- `class A` and `module B` will reopen existing class A or module B to add/re-define methods if A/B exists. Otherwise, these will define the new class/module A/B. But, in my opinion, the code of `class A` for patching existing classes doesn't work expectedly when `A` is not defined beforehand. It expects other codes to define `A` before being called. For example: ```ruby # string_exclude.rb class String def exclude?(string) !include?(string) end end ``` This code expects that there is the `String` class, and it has the `include?` method. This code doesn't work if the file is loaded in the way below: ```ruby load('string_exclude.rb', true) ``` This code doesn't raise errors and will define an almost empty class (only with a method `exclude?` to raise NameError). It should be unexpected for every user. So, I want to propose a new syntax to reopen the existing class/module or raise errors if the specified class/module is not defined. ```ruby class extension String def exclude?(string) !include?(string) end end # adds #exclude? to String class class extension Stroooong def exclude?(string) !include?(string) end end # will raise NameError (or something else) ``` Some additional things: * `class extension String` (and `module extension String`) causes a compile error (SyntaxError) on Ruby 3.3. So we have space to add a keyword between class/module and the class/module name. * I don't have a strong opinion about the keyword name `extension`. An alternative idea is `reopen`. -- https://bugs.ruby-lang.org/

8 10

[ruby-core:115909] [Ruby master Bug#20087] Uninitialized instance variable warning - obsolete documentation
by semaperepelitsa (Simon Perepelitsa) 28 Dec '23

28 Dec '23

Issue #20087 has been reported by semaperepelitsa (Simon Perepelitsa). ---------------------------------------- Bug #20087: Uninitialized instance variable warning - obsolete documentation https://bugs.ruby-lang.org/issues/20087 * Author: semaperepelitsa (Simon Perepelitsa) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This warning got removed from Ruby 3.0: https://bugs.ruby-lang.org/issues/17055 But the documentation (ruby/doc/syntax/assignment.rdoc@master) still says: > An uninitialized instance variable has a value of +nil+. If you run Ruby with > warnings enabled, you will get a warning when accessing an uninitialized > instance variable. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115969] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by ioquatix (Samuel Williams). > This proposal breaks Fiber semantics so fundamentally I don't disagree with you, and I don't have a strong opinion one way or another, but there is nothing about coroutine semantics which prevents them from being passed between threads which is what the MN implementation is already doing (it uses the same coroutines). So, I'd like to experiment with it. Alternatively, maybe we can re-implement `Enumerator` with `Thread+Queue` which is more flexible, at least that deals with one of the problems that is often brought up in these discussions. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105928 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115931] [Ruby master Bug#20097] Regexp#match? with empty capture group repeat is inconsistent
by tompng (tomoya ishida) 28 Dec '23

28 Dec '23

Issue #20097 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20097: Regexp#match? with empty capture group repeat is inconsistent https://bugs.ruby-lang.org/issues/20097 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) +MN [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- It sometimes matches and sometimes does not, depend on repeat quantifier number. ~~~ruby # empty capture group repeat p /(a(){1,4}){2}/.match?('aa') #=> true p /(a(){1,5}){2}/.match?('aa') #=> false # repeat of capture group that match to empty string p /(a(|){1,2}){2}/.match?('aa') #=> true p /(a(|){1,3}){2}/.match?('aa') #=> false p /(a(x?y?|z){1,2}){2}/.match?('aa') #=> false ~~~ non-capturing group seems ok. ~~~ruby p /(a(?:){1,4}){2}/.match?('aa') #=> true p /(a(?:){1,5}){2}/.match?('aa') #=> true p /(a(?:|){1,2}){2}/.match?('aa') #=> true p /(a(?:|){1,3}){2}/.match?('aa') #=> true p /(a(?:x?y?|z){1,2}){2}/.match?('aa') #=> true ~~~ Same behavior in ruby 3.0, 3.1, 3.2 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115965] [Ruby master Bug#20103] rb_internal_thread_remove_event_hook() hangs when called from within a Thread hook
by osyoyu (Daisuke Aritomo) 28 Dec '23

28 Dec '23

Issue #20103 has been reported by osyoyu (Daisuke Aritomo). ---------------------------------------- Bug #20103: rb_internal_thread_remove_event_hook() hangs when called from within a Thread hook https://bugs.ruby-lang.org/issues/20103 * Author: osyoyu (Daisuke Aritomo) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2023-12-28T14:53:15Z master e81a5453e3) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Problem The Ruby process hangs when `rb_internal_thread_remove_event_hook()` is called from within a Thread hook registered by `rb_internal_thread_add_event_hook()`. The thread is waiting for the thread event execution lock (`pthread_rwlock_wrlock(&rb_internal_thread_event_hooks_rw_lock))`), which is acquired by its caller, `rb_thread_execute_hooks()`. https://github.com/ruby/ruby/blob/e81a5453e3c76c4348da042d86debde7689254fe/… This situation would occur when one wants to register a *oneshot* hook that performs something on a Thread for only a single time. The hook would remove itself after its main procedure. In my case, I'm doing a timer_create(3) on each pthread for profiling purposes. ## Expected behavior The process won't hang, or at least print some kind of warning. I think there some options here: 1. Discourage this kind of definition. Thread hooks should not add/remove other Thread hooks, or otherwise do so through other async methods, such as postponed jobs. 2. Find a way to make this work. Maybe inner calls to add/remove_event_hook() can share locks with the outer call, but I'm not sure if that's possible as I'm not familiar with pthread_rwlocks. ## Repro ```c #include "ruby.h" #include "ruby/thread.h" rb_internal_thread_event_hook_t *hook; void event_hook_callback(rb_event_flag_t flag, const rb_internal_thread_event_data_t *data, void *custom_data) { rb_internal_thread_remove_event_hook(hook); // hangs } RUBY_FUNC_EXPORTED void Init_mycext(void) { hook = rb_internal_thread_add_event_hook( event_hook_callback, RUBY_INTERNAL_THREAD_EVENT_RESUMED, NULL ); } ``` ``` ruby -rmycext -e 'Thread.new { loop { 1 + 1 } }' ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115958] [Ruby master Feature#15667] Introduce malloc_trim(0) in full gc cycles
by hongli (Hongli Lai) 28 Dec '23

28 Dec '23

Issue #15667 has been updated by hongli (Hongli Lai). It looks like this ticket can be closed. I see that Ruby 3.3 makes use of malloc_trim. ---------------------------------------- Feature #15667: Introduce malloc_trim(0) in full gc cycles https://bugs.ruby-lang.org/issues/15667#change-105915 * Author: sam.saffron (Sam Saffron) * Status: Open * Priority: Normal ---------------------------------------- Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-… I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation. ---Files-------------------------------- ruby_gc_malloc_trim.patch (1011 Bytes) Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB) crash.png (85.9 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115957] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by Eregon (Benoit Daloze) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by Eregon (Benoit Daloze). bascule (Tony Arcieri) wrote in #note-10: > There's a simple solution to this: track if a given fiber is holding mutexes (e.g. keep a count of them) and if it is, make Fiber#resume raise an exception if it is resumed in a different thread from the one where it was originally yielded. > > That way you eliminate the nasty edge case, but still allow fibers which aren't holding mutexes (or whatever other synchronization primitives you're worried about) to be resumed in a different thread. This may work for ::Mutex, but it won't work for any other synchronization. For example any of the locks/barriers/countdown latch in concurrent-ruby. Or even locks in C extensions. > The same solution could work for thread-local storage: disallow fiber cross-thread fiber resumption if thread local storage is in use. I think this will be so limiting that it makes creating a Fiber which can migrate between threads almost always useless in practice as such a Fiber can't e.g. call a gem as that might use thread-local storage or synchronization internally. @shan Regarding Enumerator, only next/peek create a Fiber, all other methods are fine and do not need a Fiber. Using an Enumerator in a next/peek way from multiple threads is something that is fundamentally very tricky, and that Ruby currently does not really have a concept to support it. Basically you'd want a thread-safe Java Iterator(#hasNext/#next), but there is no such protocol in Ruby. That's really tricky, and few data structures can actually support that. I think it has very limited value, because next/peek is rare, and doing next/peek in threads concurrently seems asking for concurrency bugs and data races. A Queue seems a much better thing to use to get some elements on one thread and some on another. @ioquatix > I'll put together a PR so we can experiment. This proposal breaks Fiber semantics so fundamentally I think if such a feature is added it should have another name than Fiber, to make it clear it can migrate threads and is subject to all thread races. In fact since it would have the exact same concurrency semantics as Thread, so I think it should include thread in the name. BTW, now CRuby has M-N threads (#19842), is there any reason to try this over using `RUBY_MN_THREADS=1`? One could also simulate this by creating every Fiber in a Thread with `RUBY_MN_THREADS=1` as threads should be cheaper that way. That also avoids all the problems of a Fiber changing threads dynamically, while not restricting to run a Fiber on a given thread. Migrating a Fiber to another threads seems to also have little value on CRuby due to the GVL, as it won't improve parallelism. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105914 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115956] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by ioquatix (Samuel Williams). One issue was the use of the copy-coroutine. As we've removed this and replaced it with a pthread fallback, we can now potentially allow fibers to be resumed across threads. I'll put together a PR so we can experiment. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105913 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0