March 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:116827] [Ruby master Feature#20276] Introduce Fiber interfaces for Ractors
by forthoney (Seong-Heon Jung) 18 Mar '24

18 Mar '24

Issue #20276 has been reported by forthoney (Seong-Heon Jung). ---------------------------------------- Feature #20276: Introduce Fiber interfaces for Ractors https://bugs.ruby-lang.org/issues/20276 * Author: forthoney (Seong-Heon Jung) * Status: Open * Priority: Normal ---------------------------------------- ## Motivation I am trying to build a web server with Ractors. The lifecycle for a request in the current implementation is 1. main ractor buffers request 2. main ractor sends request to worker ractor 3. worker ractor sends response to main ractor 4. main ractor writes response 5. repeat The main ractor utilizes the Async gem (specifically async-http) to handle connections concurrently, meaning each request is handled on a separate fiber. The issue I am running into is after I send a request to a worker ractor, I need to do a blocking wait until I receive a response. While I am waiting for the response, I cannot take any more connections. ## Solution If the fiber scheduler had a hook for `Ractor.receive` or `Ractor#take` (both of which are blocking), the main ractor can send the message, handle other connections while the worker processes the request. When the worker produces a message, it will then take the reqeust and write it in the socket. -- https://bugs.ruby-lang.org/

3 7

[ruby-core:117088] [Ruby master Feature#20329] Clean up `--dump` sub-options
by nobu (Nobuyoshi Nakada) 18 Mar '24

18 Mar '24

Issue #20329 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #20329: Clean up `--dump` sub-options https://bugs.ruby-lang.org/issues/20329 * Author: nobu (Nobuyoshi Nakada) * Status: Open ---------------------------------------- Currently we have 5 options for `--dump` command line option. * insns * insns_without_opt * yydebug(+error-tolerant) * parsetree(+error-tolerant) * parsetree_with_comment(+error-tolerant) Among these, `insns_without_opt` is a variant of `insns`, and `parsetree_with_comment` is a variant of `parsetree`. However, there is now another way to specify variants (e.g. `+error-tolerant`). How about unifying the two so that the former can also be specified in the same form, such as `--dump=parsetree+comment+error-tolerant`? It also will be able to abbreviate as parse+comm+err` or more. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:117208] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by ioquatix (Samuel Williams) 18 Mar '24

18 Mar '24

Issue #19057 has been updated by ioquatix (Samuel Williams). I don't use `unicorn` so thanks for the clarification. Maybe someone who does know, can write some documentation about the current state of affairs and how to use the current head. My trivial `gem build` and `gem install` seemed to work, but I didn't test it beyond that. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-107297 * Author: ioquatix (Samuel Williams) * Status: Assigned * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117207] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by byroot (Jean Boussier) 18 Mar '24

18 Mar '24

Issue #19057 has been updated by byroot (Jean Boussier). > are you able to point it at the git head? You can't just do that because Unicorn has a bunch of files that need to be generated and are not compiled. So you need a custom fork: https://github.com/k0kubun/unicorn/commit/6215d4cad7a964bf4b8bdef48edadf334… ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-107296 * Author: ioquatix (Samuel Williams) * Status: Assigned * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117206] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by ioquatix (Samuel Williams) 18 Mar '24

18 Mar '24

Issue #19057 has been updated by ioquatix (Samuel Williams). @mame are you able to point it at the git head? e.g. ```ruby gem "unicorn", git: "https://yhbt.net/unicorn.git" # or gem "unicorn", git: "https://github.com/socketry/unicorn.git" ``` That should get things moving again. I hope Eric will release an updated `unicorn` soon. However, if that doesn't happen, I see a couple of options: - We can release `unicorn2` gem. - Companies with a vested interest could fork unicorn for internal use. - Companies could contact Eric and offer incentives for him to make a release. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-107295 * Author: ioquatix (Samuel Williams) * Status: Assigned * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117204] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by mame (Yusuke Endoh) 18 Mar '24

18 Mar '24

Issue #19057 has been updated by mame (Yusuke Endoh). This change has broken our internal CI. Because our CI contributes to assure the quality of Ruby master, it is a shame that it will not work until the release of unicorn, which we do not know when (and whether) it will be released. I hope to postpone the change at least until unicorn etc. is released. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-107293 * Author: ioquatix (Samuel Williams) * Status: Assigned * Assignee: ioquatix (Samuel Williams) * Target version: 3.4 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117202] [Ruby master Bug#20343] Ripper.sexp in Ruby 3.4 returns nil
by koic (Koichi ITO) 17 Mar '24

17 Mar '24

Issue #20343 has been reported by koic (Koichi ITO). ---------------------------------------- Bug #20343: Ripper.sexp in Ruby 3.4 returns nil https://bugs.ruby-lang.org/issues/20343 * Author: koic (Koichi ITO) * Status: Open * ruby -v: ruby 3.4.0dev (2024-03-17T10:09:37Z master 5fd6b461c7) [x86_64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In Ruby 3.4.0's Ripper, there is the following incompatible behavior: ## Expected (Ruby 3.3) It is expected that tokens will be returned. ```console $ ruby -rripper -ve "p Ripper.sexp('foo[bar, &baz] += 1')" ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [:program, [[:opassign, [:aref_field, [:vcall, [:@ident, "foo", [1, 0]]], [:args_add_block, [[:vcall, [:@ident, "bar", [1, 4]]]], [:vcall, [:@ident, "baz", [1, 10]]]]], [:@op, "+=", [1, 15]], [:@int, "1", [1, 18]]]]] $ ruby -rripper -ve "p Ripper.sexp_raw('foo[bar, &baz] += 1')" ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] [:program, [:stmts_add, [:stmts_new], [:opassign, [:aref_field, [:vcall, [:@ident, "foo", [1, 0]]], [:args_add_block, [:args_add, [:args_new], [:vcall, [:@ident, "bar", [1, 4]]]], [:vcall, [:@ident, "baz", [1, 10]]]]], [:@op, "+=", [1, 15]], [:@int, "1", [1, 18]]]]] ``` ## Actual (Ruby 3.4) `nil` is returned. ```console $ ruby -rripper -ve "p Ripper.sexp('foo[bar, &baz] += 1')" ruby 3.4.0dev (2024-03-17T10:09:37Z master 5fd6b461c7) [x86_64-darwin23] nil $ ruby -rripper -ve "p Ripper.sexp_raw('foo[bar, &baz] += 1')" ruby 3.4.0dev (2024-03-17T10:09:37Z master 5fd6b461c7) [x86_64-darwin23] nil ``` The same applies to the following cases. ```ruby foo[bar, &baz] += 1 foo[bar, &baz] ||= 1 foo[bar, &baz] &&= 1 foo.foo[bar, &baz] += 1 foo.foo[bar, &baz] ||= 1 foo.foo[bar, &baz] &&= 1 ``` This was discovered in a failure of Prism's CI: https://github.com/ruby/prism/actions/runs/8317131929/job/22757514162?pr=26… -- https://bugs.ruby-lang.org/

2 1

[ruby-core:114309] [Ruby master Feature#19787] Add Enumerable#uniq_map, Enumerable::Lazy#uniq_map, Array#uniq_map and Array#uniq_map!
by joshuay03 (Joshua Young) 16 Mar '24

16 Mar '24

Issue #19787 has been reported by joshuay03 (Joshua Young). ---------------------------------------- Feature #19787: Add Enumerable#uniq_map, Enumerable::Lazy#uniq_map, Array#uniq_map and Array#uniq_map! https://bugs.ruby-lang.org/issues/19787 * Author: joshuay03 (Joshua Young) * Status: Open * Priority: Normal ---------------------------------------- I would like to propose a collection of new methods, `Enumerable#uniq_map`, `Enumerable::Lazy#uniq_map`, `Array#uniq_map` and `Array#uniq_map!`. TL;DR: It's a drop in replacement for `.map { ... }.uniq`, with better performance. I've quite often had to map over an array and get its unique elements. It occurred to me when doing so recently that Ruby doesn't have a short form method for doing that, similar to how `flat_map { ... }` replaces `.map { ... }.flatten` and `filter_map { ... }` replaces `.map { ... }.compact` (with minor differences). I think these new methods could be beneficial both in terms of better performance and writing more succinct code. I have already got a draft PR up with some initial benchmarks in the description: https://github.com/ruby/ruby/pull/8140. -- https://bugs.ruby-lang.org/

4 6

[ruby-core:117197] [Ruby master Bug#20341] Parser segfault with shareable constant with array of more than 1 element
by kddnewton (Kevin Newton) 15 Mar '24

15 Mar '24

Issue #20341 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20341: Parser segfault with shareable constant with array of more than 1 element https://bugs.ruby-lang.org/issues/20341 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby # shareable_constant_value: literal C = [foo, bar] ``` results in ``` ./ruby: [BUG] unexpected node: NODE_LIT ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117115] [Ruby master Feature#20331] Should parser warn hash duplication and when clause?
by yui-knk (Kaneko Yuichiro) 15 Mar '24

15 Mar '24

Issue #20331 has been reported by yui-knk (Kaneko Yuichiro). ---------------------------------------- Feature #20331: Should parser warn hash duplication and when clause? https://bugs.ruby-lang.org/issues/20331 * Author: yui-knk (Kaneko Yuichiro) * Status: Open ---------------------------------------- # Background Right now, parser warns duplicated hash keys (#1) and when clause (#2). For example, ```ruby {1 => :a, 1 => :b} # => warning: key 1 is duplicated and overwritten on line 1 ``` ```ruby case 2 when 1, 1 else end # => test.rb:2: warning: duplicated `when' clause with line 2 is ignored ``` The parser compares different cardinality numbers. ```ruby { 1 => :a, 0x1 => :b, 0b1 => :b, 0d1 => :b, 0o1 => :b, } # => test.rb:2: warning: key 1 is duplicated and overwritten on line 3 # => test.rb:3: warning: key 1 is duplicated and overwritten on line 4 # => test.rb:4: warning: key 1 is duplicated and overwritten on line 5 # => test.rb:5: warning: key 1 is duplicated and overwritten on line 6 ``` # Problem Currently this is implemeted by converting string like `"123"` to Ruby Object and compare them. It's needed to remove Ruby Object from parse.y for Universal Parser. I created PR https://github.com/ruby/ruby/pull/10079 which implements bignum for parse.y without dependency on Ruby Object, however nobu and mame express concern about the cost and benefit of implmenting bignum for parser. I want to discuss which is the best approach for this problem. By the way, it's needed to calculate irreducible fraction for Rational key if we will keep warning messages. ```ruby $ ruby -wc -e '{10.2r => :a, 10.2r => :b}' -e:1: warning: key (51/5) is duplicated and overwritten on line 1 -e:1: warning: unused literal ignored Syntax OK ``` # Options ## 1. Warnings on parser Pros: * Users of Universal Parser don't need to implement warnings by themselves. I guess developers of other Ruby implementation may get benefit of reducing their effort. * Warnings are shown by `ruby -wc`. Cons: * We need to maintain bignum implementation for parser. There are two approaches for this option. ### 1-1. Implement bignum for parser The PR is this approach, implementing sub set of Ruby bignum for parser. ### 1-2. Extract existing bignum implementation then use it Make existing bignum implementation to be independent of Ruby Object and use it from both bignum.c and parse.y. ## 2. Moving warnings logic into compile phase We can use Ruby Object in compile.c. Then moving the logic into compile.c solves this problem. Pros: * No need to implement bignum for parser. Cons: * Users of Universal Parser need to implement warnings by themselves. * Warnings are not shown by `ruby -wc`. -- https://bugs.ruby-lang.org/

7 10