December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115958] [Ruby master Feature#15667] Introduce malloc_trim(0) in full gc cycles
by hongli (Hongli Lai) 28 Dec '23

28 Dec '23

Issue #15667 has been updated by hongli (Hongli Lai). It looks like this ticket can be closed. I see that Ruby 3.3 makes use of malloc_trim. ---------------------------------------- Feature #15667: Introduce malloc_trim(0) in full gc cycles https://bugs.ruby-lang.org/issues/15667#change-105915 * Author: sam.saffron (Sam Saffron) * Status: Open * Priority: Normal ---------------------------------------- Per Hongli's excellent article it looks like malloc_trim can help tremendously with memory bloat issues. https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-… I would like to get this patch tested side-by-side at Discourse, GitHub and Shopify. If it looks good I think this is both a great candidate for 2.7 and and 2.4,2.5,2.6 backports. Will coordinate with Shopify and GitHub to see if we can get numbers posted here, I will run tests on a live Discourse instance over the next week and report numbers here. Koichi, what are your thoughts, to me this looks like an incredibly safe patch, the amount of work added to major GCs is tiny compared to the potential benefit, walking all pages is a very cheap operation. ---Files-------------------------------- ruby_gc_malloc_trim.patch (1011 Bytes) Screenshot_2019-03-28 Grafana - Compare Discourse Perf.png (530 KB) crash.png (85.9 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115957] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by Eregon (Benoit Daloze) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by Eregon (Benoit Daloze). bascule (Tony Arcieri) wrote in #note-10: > There's a simple solution to this: track if a given fiber is holding mutexes (e.g. keep a count of them) and if it is, make Fiber#resume raise an exception if it is resumed in a different thread from the one where it was originally yielded. > > That way you eliminate the nasty edge case, but still allow fibers which aren't holding mutexes (or whatever other synchronization primitives you're worried about) to be resumed in a different thread. This may work for ::Mutex, but it won't work for any other synchronization. For example any of the locks/barriers/countdown latch in concurrent-ruby. Or even locks in C extensions. > The same solution could work for thread-local storage: disallow fiber cross-thread fiber resumption if thread local storage is in use. I think this will be so limiting that it makes creating a Fiber which can migrate between threads almost always useless in practice as such a Fiber can't e.g. call a gem as that might use thread-local storage or synchronization internally. @shan Regarding Enumerator, only next/peek create a Fiber, all other methods are fine and do not need a Fiber. Using an Enumerator in a next/peek way from multiple threads is something that is fundamentally very tricky, and that Ruby currently does not really have a concept to support it. Basically you'd want a thread-safe Java Iterator(#hasNext/#next), but there is no such protocol in Ruby. That's really tricky, and few data structures can actually support that. I think it has very limited value, because next/peek is rare, and doing next/peek in threads concurrently seems asking for concurrency bugs and data races. A Queue seems a much better thing to use to get some elements on one thread and some on another. @ioquatix > I'll put together a PR so we can experiment. This proposal breaks Fiber semantics so fundamentally I think if such a feature is added it should have another name than Fiber, to make it clear it can migrate threads and is subject to all thread races. In fact since it would have the exact same concurrency semantics as Thread, so I think it should include thread in the name. BTW, now CRuby has M-N threads (#19842), is there any reason to try this over using `RUBY_MN_THREADS=1`? One could also simulate this by creating every Fiber in a Thread with `RUBY_MN_THREADS=1` as threads should be cheaper that way. That also avoids all the problems of a Fiber changing threads dynamically, while not restricting to run a Fiber on a given thread. Migrating a Fiber to another threads seems to also have little value on CRuby due to the GVL, as it won't improve parallelism. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105914 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115956] [Ruby master Feature#13821] Allow fibers to be resumed across threads
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #13821 has been updated by ioquatix (Samuel Williams). One issue was the use of the copy-coroutine. As we've removed this and replaced it with a pthread fallback, we can now potentially allow fibers to be resumed across threads. I'll put together a PR so we can experiment. ---------------------------------------- Feature #13821: Allow fibers to be resumed across threads https://bugs.ruby-lang.org/issues/13821#change-105913 * Author: cremes (Chuck Remes) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) ---------------------------------------- Given a Fiber created in ThreadA, Ruby 2.4.1 (and earlier releases) raise a FiberError if the fiber is resumed in ThreadB or any other thread other than the one that created the original Fiber. Sample code attached to demonstrate problem. If Fibers are truly encapsulating all of the data for the continuation, we should be allowed to move them between Threads and resume their operation. Why? One use-case is to support the async-await asynchronous programming model. In that model, a method marked async runs *synchronously* until the #await method is encountered. At that point the method is suspended and control is returned to the caller. When the #await method completes (asynchronously) then it may resume the suspended method and continue. The only way to capture this program state, suspend and resume, is via a Fiber. example: ``` class Wait include AsyncAwait def dofirst async do puts 'Synchronously print dofirst.' result = await { dosecond } puts 'dosecond is complete' result end end def dosecond async do puts 'Synchronously print dosecond from async task.' slept = await { sleep 3 } puts 'Sleep complete' slept end end def run task = dofirst puts 'Received task' p AsyncAwait::Task.await(task) end end Wait.new.run ``` ``` # Expected output: # Synchronous print dofirst. # Received task # Synchronously print dosecond from async task. # Sleep complete # dosecond is complete # 3 ``` Right now the best way to accomplish suspension of the #dofirst and #dosecond commands and allow them to run asynchronously is by passing those blocks to *another thread* (other than the callers thread) so they can be encapsulated in a new Fiber and then yielded. When it's time to resume after #await completes, that other thread must lookup the fiber and resume it. This is lots of extra code and logic to make sure that fibers are only resumed on the threads that created them. Allowing Fibers to migrate between threads would eliminate this problem. ---Files-------------------------------- fiber_across_threads.rb (377 Bytes) wait.rb (728 Bytes) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111789] [Ruby master Feature#19333] Setting (Fiber Local|Thread Local|Fiber Storage) to nil should delete value in order to avoid memory leaks.
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #19333 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19333: Setting (Fiber Local|Thread Local|Fiber Storage) to nil should delete value in order to avoid memory leaks. https://bugs.ruby-lang.org/issues/19333 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As it stands, Fiber Locals, Thread Locals and Fiber Storage have no way of deleting key-value associations. ```ruby 100.times do |i| name = :"variable-#{i}" Thread.current[name] = 10 end ``` Because of this, dynamically generated associations can leak over time. This is worse for things like Threads that might be pooled (or maybe an argument against user-space pooling). In any case, having a way to delete those associations would allow application code to at least delete the associations when they no longer make sense. I propose that assigning `nil` to "locals" or "storage" should effectively delete them. e.g. ```ruby 100.times do |i| name = :"variable-#{i}" Thread.current[name] = 10 Thread.current[name] = nil # delete association end ``` A more invasive alternative would be to define new interfaces like `Thread::Local`, `Fiber::Local` and `Fiber::Storage::Local` (or something) which correctly clean up on GC. -- https://bugs.ruby-lang.org/

3 8

[ruby-core:115953] [Ruby master Feature#18035] Introduce general model/semantic for immutability.
by ioquatix (Samuel Williams) 28 Dec '23

28 Dec '23

Issue #18035 has been updated by ioquatix (Samuel Williams). There is a PR here that mostly just works: https://github.com/ruby/ruby/pull/4879 However, additional work would be required to: 1. Tidy up the implementation and 2. Mark some core classes as Immutable as outlined in https://gist.github.com/eregon/bce555fa9e9133ed27fbfc1deb181573 ---------------------------------------- Feature #18035: Introduce general model/semantic for immutability. https://bugs.ruby-lang.org/issues/18035#change-105910 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- It would be good to establish some rules around mutability, immutability, frozen, and deep frozen in Ruby. I see time and time again, incorrect assumptions about how this works in production code. Constants that aren't really constant, people using `#freeze` incorrectly, etc. I don't have any particular preference but: - We should establish consistent patterns where possible, e.g. - Objects created by `new` are mutable. - Objects created by literal are immutable. We have problems with how `freeze` works on composite data types, e.g. `Hash#freeze` does not impact children keys/values, same for Array. Do we need to introduce `freeze(true)` or `#deep_freeze` or some other method? Because of this, frozen does not necessarily correspond to immutable. This is an issue which causes real world problems. I also propose to codify this where possible, in terms of "this class of object is immutable" should be enforced by the language/runtime, e.g. ```ruby module Immutable def new(...) super.freeze end end class MyImmutableObject extend Immutable def initialize(x) @x = x end def freeze return self if frozen? @x.freeze super end end o = MyImmutableObject.new([1, 2, 3]) puts o.frozen? ``` Finally, this area has an impact to thread and fiber safe programming, so it is becoming more relevant and I believe that the current approach which is rather adhoc is insufficient. I know that it's non-trivial to retrofit existing code, but maybe it can be done via magic comment, etc, which we already did for frozen string literals. Proposed PR: https://github.com/ruby/ruby/pull/4879 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112446] [Ruby master Bug#19442] Remove USE_RINCGC flag
by eightbitraptor (Matthew Valentine-House) 28 Dec '23

28 Dec '23

Issue #19442 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19442: Remove USE_RINCGC flag https://bugs.ruby-lang.org/issues/19442 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- [GitHub PR #7317](https://github.com/ruby/ruby/pull/7313) Ruby doesn't compile when USE_RINCGC is disabled. It's also not being tested in CI. As @nobu has pointed out in comments on the PR, fixing it is simple. I think there are 2 approaches we could take: 1. Remove `USE_RINCGC` entirely and always run with incremental GC enabled. We have a precedent for this: `USE_RGENGC=0` was removed [in this commit](https://github.com/ruby/ruby/commit/62c2b8c74e47652fc5bbaf6150f4acd… almost 3 years ago. `USE_RINCGC=0` is in a similar state. It has been broken for almost a year (since [this commit](https://github.com/ruby/ruby/commit/dde164e968e382d50b07ad455946888…) and [disabled in CI for more than 2 years](https://github.com/ruby/ruby/commit/46d3ea2c2569e2e5a9ee3e7e206f07f0…. 2. We could fix `USE_RINCGC`. If we do this we should re-enable the CI task so that we continue to keep it working in the future. I am very much in favour of option 1, because I don't think this flag is being actively used, and removing it will simplify the code and reduce the cognitive overhead of working with the GC. However I am happy to defer to the team here. -- https://bugs.ruby-lang.org/

6 9

[ruby-core:115879] [Ruby master Bug#20081] Transfered Fiber doesn't return to Fiber that started it
by rmosolgo (Robert Mosolgo) 27 Dec '23

27 Dec '23

Issue #20081 has been reported by rmosolgo (Robert Mosolgo). ---------------------------------------- Bug #20081: Transfered Fiber doesn't return to Fiber that started it https://bugs.ruby-lang.org/issues/20081 * Author: rmosolgo (Robert Mosolgo) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Hi! I'm trying to figure out how to make sure that Fibers started with `.transfer` end up _terminated_, not just suspended. (If they're suspended, Rails thinks they're still alive, and they continue to hold onto database connections, see: https://github.com/rmosolgo/graphql-ruby/issues/4739#issuecomment-1866930914.) So, I'm looking for way to make sure that any Fiber I start with `.transfer` will be properly terminated. But what I noticed is that when a transfer-based Fiber terminates, it gives control back to the top-most Fiber, not the Fiber which transfered to it. Is this intended? Here's a script to replicate the issue: ```ruby manager = Fiber.new do parent = Fiber.current worker = Fiber.new do puts "2. Begin Worker" parent.transfer puts "4. End Worker" end puts "1. Transfer 1" worker.transfer puts "3. Transfer 2" worker.transfer puts "5. Finished manager" end manager.transfer puts "6. Finished script" ``` I expect the steps to print in order: ``` 1. Transfer 1 2. Begin Worker 3. Transfer 2 4. End Worker 5. Finished manager 6. Finished script ``` But instead, `5. ...` is skipped: ``` 1. Transfer 1 2. Begin Worker 3. Transfer 2 4. End Worker 6. Finished script ``` I think that's because my `worker` fiber terminates and passes control back to the top-level Fiber. Should it have passed control back to the `manager`? Or is there another way to make sure `worker` is terminated, and `manager` gets control? -- https://bugs.ruby-lang.org/

2 4

[ruby-core:115938] [Ruby master Feature#20100] [Feature] Better / easier support for (optional) colours, on the commandline, in case a parsing error happened?
by rubyFeedback (robert heiler) 27 Dec '23

27 Dec '23

Issue #20100 has been reported by rubyFeedback (robert heiler). ---------------------------------------- Feature #20100: [Feature] Better / easier support for (optional) colours, on the commandline, in case a parsing error happened? https://bugs.ruby-lang.org/issues/20100 * Author: rubyFeedback (robert heiler) * Status: Open * Priority: Normal ---------------------------------------- This suggestion is primarily in regard to (optional) colour support, on the commandline, in case a parsing error happened. I'll explain what I mean with this in a moment. Let me first show a partial screenshot (image), to indicate this problem. https://i.imgur.com/FylLvsY.png Ruby evaluates the content and indicates the error here, while also helpfully showing the line numbers WHERE the error happened (on the left hand side). This is nice. But, I would actually prefer if some colours could be shown here, at the user's discretion. For instance, I would like to have the line numbers show in red or a red-colour; or any colour for that matter. That would give me a visual cue as to the primary problem (sometimes more than one error is shown, or occurs, in a given .rb file, or distributed via several .rb files). The keyword or type-error, such as SyntaxError, should ideally also use another colour. I may also want to see the specific error in yet another colour, e. g. the part ", "unexpected 'end". The error here was that I copy/pasted some code (I am porting old code and checking it as I do right now) and left behind a trailing ',' there. Hopefully I could convince the dear reader that colours may be important. Perhaps colour support is already possible, and I (and perhaps others) just don't know about it. So the question, then, would be: how to enable simple colours to be shown? Well, a couple of ways would be possible. A simple one may be to put an environment variable, and document it clearly, such as RUBY_ENABLE_COLOURS or RUBY_COLOURS, and a value of 1 to indicate that we want to use colours. Or any other name, these are just suggestions; can be longer names, to indicate on-error situations such as RUBY_SHOW_COLOURED_ERRORS or RUBY_COLOURED_ERRORS (or COLORED, I don't mind the UK or US spelling, even though I got used to the UK spelling due to UK being close to central Europe). There could also be an internal API we could call; not sure which one, perhaps via RbConfig, e. g. RbConfig.enable_colours or RbConfig::COLOURS_ON_WARNING = true or something like that. Or another way where this may reside, although an ENV variable may be simpler. A few other options may also be possible - either way the primary objective would be to make it super-easy and super-simple to have colours support, perhaps even for all of ruby, without needing to micro-manage ENV variables. Ideally https://docs.ruby-lang.org/en/ should also get a dedicate page about colours in general. On Windows colours may not always work well (I have had different success there) but on linux, bash, KDE konsole and so forth, colours work really well for me. At any rate this was the suggestion - thank you for reading this. PS: I searched for "colo" on the docs.ruby-lang.org webpage; results often involved e. g. IRB like https://docs.ruby-lang.org/en/3.3/IRB/Color.html, which is useful if someone wants to modify IRB's behaviour, but not so useful when it comes to general things concerning MRI. There is really a distinct lack of top-down overview-articles in regards to ruby on the official homepage. How are new users supposed to find things like that? It would be nice if a long-term strategy for ruby would also be to provide higher "overarching" documentation for ruby, via a central place (MRI). At any rate, this is a separate issue - my feature request here is concerning about enabling colours support on the commandline, in case of an error. The ruby parser probably already has this information, since it can show where an error happens, so all that may be necessary would be to tie this towards colour-output; and then also a way for people to tap into that (which is why I think an ENV variable, even though not super-sophisticated, may be the simplest way to go, but that should also be clearly documented in a very-easy-to-find, and accessible, manner). -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115889] [Ruby master Bug#20084] Breaking change with Regexp.new on 3.3.0
by paddor (Patrik Wenger) 27 Dec '23

27 Dec '23

Issue #20084 has been reported by paddor (Patrik Wenger). ---------------------------------------- Bug #20084: Breaking change with Regexp.new on 3.3.0 https://bugs.ruby-lang.org/issues/20084 * Author: paddor (Patrik Wenger) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Regexp.new used to take up to 3 arguments, but now only 1..2 are allowed: ``` [1] pry(main)> Regexp.new 'abc', 'i', 'n' ArgumentError: wrong number of arguments (given 3, expected 1..2) from (pry):1:in `initialize' ``` On 3.2.2: ``` [1] pry(main)> Regexp.new 'abc', 'i', 'n' => /abc/in ``` -- https://bugs.ruby-lang.org/

3 3

[ruby-core:115918] [Ruby master Bug#20092] Warnings for anonymous parameters do not display in Ruby 3.4dev
by koic (Koichi ITO) 27 Dec '23

27 Dec '23

Issue #20092 has been reported by koic (Koichi ITO). ---------------------------------------- Bug #20092: Warnings for anonymous parameters do not display in Ruby 3.4dev https://bugs.ruby-lang.org/issues/20092 * Author: koic (Koichi ITO) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2023-12-26T22:32:38Z master be92bf4ff4) [x86_64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I encountered no expected warnings in Ruby 3.4dev. ## Reproduction code ```ruby def anonymous(*, **, &) rest_block do foo(*) end kwrest_block do bar(**) end block_block do baz(&) end end ``` ## Expected Behavior (Ruby 3.3.0) Warnings are displayed: ```ruby $ ruby -vc example.rb ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] example.rb: example.rb:3: anonymous rest parameter is also used within block (SyntaxError) example.rb:7: anonymous keyword rest parameter is also used within block example.rb:11: anonymous block parameter is also used within block ``` ## Actual Behavior (Ruby 3.4dev) No warning displayed: ```ruby $ ruby -vc example.rb ruby 3.4.0dev (2023-12-26T22:32:38Z master be92bf4ff4) [x86_64-darwin22] Syntax OK ``` -- https://bugs.ruby-lang.org/

2 1