December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115845] [Ruby master Bug#20078] StringIO cannot be moved between Ractors
by forthoney (Seong-Heon Jung) 11 Jan '24

11 Jan '24

Issue #20078 has been reported by forthoney (Seong-Heon Jung). ---------------------------------------- Bug #20078: StringIO cannot be moved between Ractors https://bugs.ruby-lang.org/issues/20078 * Author: forthoney (Seong-Heon Jung) * Status: Open * Priority: Normal * ruby -v: 3.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following code will raise an unusual Ractor error. ``` ruby require 'stringio' r = Ractor.new { loop { Ractor.receive } } Ractor.shareable?(StringIO.new) #=> false r.send(StringIO.new) # passing it via copy works r.send(StringIO.new, move: true) # <internal:ractor>:587:in `send': can not move StringIO object. (Ractor::Error) ``` I'm not 100% sure but I believe that this probably not the intended behavior considering * The error raised is a generic `Ractor::Error` rather than something specific like Ractor::MovedError or Ractor::IsolationError * It can be copied * [No documentation](https://docs.ruby-lang.org/en/master/ractor_md.html) exists for this error * I have yet to seen this happen on any other class instance * Typo (can not -> cannot) -- https://bugs.ruby-lang.org/

3 6

[ruby-core:115736] [Ruby master Bug#20064] Inconsistent behavior between array splat *nil and hash splat **nil
by zeke (Zeke Gabrielse) 11 Jan '24

11 Jan '24

Issue #20064 has been reported by zeke (Zeke Gabrielse). ---------------------------------------- Bug #20064: Inconsistent behavior between array splat *nil and hash splat **nil https://bugs.ruby-lang.org/issues/20064 * Author: zeke (Zeke Gabrielse) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This has been discussed in #8507 and #9291 but both were closed because they lacked a clear use case. I think the following code should work, showing a clear use case: ```ruby invitation = if params.key?(:inviter_id) { invitation_attributes: params.slice(:inviter_id) } end User.create( email: 'john.doe(a)ruby.example', first_name: 'John', first_name: 'Doe', **invitation, ) ``` Per the previous discussions, this is because `*` uses explicit conversion to array (`to_a`, not `to_ary`), while `**` uses implicit conversion to hash (`to_hash`, not `to_h`). I find it confusing that you can splat nil into an array, but not nil into a hash. It would make sense for `**` to use explicit conversion. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:115491] [Ruby master Feature#20024] SyntaxError subclasses
by kddnewton (Kevin Newton) 10 Jan '24

10 Jan '24

Issue #20024 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Feature #20024: SyntaxError subclasses https://bugs.ruby-lang.org/issues/20024 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- There are many places around the Ruby ecosystem that handle syntax errors in different ways. Some provide highlighting, others provide recovery of some form, still more provide LSP metadata. In order to provide more rich information, most of them switch on the message of the error being returned, as in: https://github.com/ruby/irb/blob/f86d9dbe2fc05ed62332069a27f4aacc59ba9634/l… Within ruby/spec, specific error messages are required for these kinds of messages in order to support this implicit interface that syntax errors have a hidden type, which is only expressed through their message. For example: https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… It's not clear from these specs or from the parser itself which error messages are permanent/guaranteed versus which are changeable. Either way, relying on the error message itself as opposed to the type of the error is brittle at best. I would like to suggest instead we implement subclasses on `SyntaxError` that would allow tools that depend on specific syntax errors to rescue those subclasses instead of parsing the message. In addition to alleviating the need to parse error messages with regex, this would also allow for the possibility that the error messages could change in the future without breaking external tooling. Allowing these to change would allow them to be potentially enhanced or changed by other tools - for example by providing recovery information or translating them. This is particularly important for Prism since we are getting down to individual spec failures and some of the failures are related to the fact that we have messages like `"Numbered parameter is already used in outer scope"` where the spec requires `/numbered parameter is already used in/`. Even this case-sensitivity is causing failures, which seems like we're testing the wrong thing. -- https://bugs.ruby-lang.org/

9 16

[ruby-core:113788] [Ruby master Bug#19717] `ConditionVariable#signal` is not fair when the wakeup is consistently spurious.
by ioquatix (Samuel Williams) 10 Jan '24

10 Jan '24

Issue #19717 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19717: `ConditionVariable#signal` is not fair when the wakeup is consistently spurious. https://bugs.ruby-lang.org/issues/19717 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- For background, see this issue <https://github.com/socketry/async/issues/99>. It looks like `ConditionVariable#signal` is not fair, if the calling thread immediately reacquires the resource. I've given a detailed reproduction here as it's non-trivial: <https://github.com/ioquatix/ruby-condition-variable-timeout>. Because the spurious wakeup occurs, the thread is pushed to the back of the waitq, which means any other waiting thread will acquire the resource, and that thread will perpetually be at the back of the queue. I believe the solution is to change `ConditionVarialbe#signal` should only remove the thread from the waitq if it's possible to acquire the lock. Otherwise it should be left in place, so that the order is retained, this should result in fair scheduling. -- https://bugs.ruby-lang.org/

3 12

[ruby-core:115684] [Ruby master Feature#20057] Change behaviour of rb_register_postponed_job for Ruby 3.3
by kjtsanaktsidis (KJ Tsanaktsidis) 10 Jan '24

10 Jan '24

Issue #20057 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #20057: Change behaviour of rb_register_postponed_job for Ruby 3.3 https://bugs.ruby-lang.org/issues/20057 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- This ticket is to discuss some changes to `rb_register_postponed_job` that @ko1 and myself propose to make for Ruby 3.3. The motivation for this work is to fix a bug in the current implementation, which can cause the registered functions to be called with the wrong `data` argument (https://bugs.ruby-lang.org/issues/19991) There's a long discussion on the associated PR (https://github.com/ruby/ruby/pull/9041) but in the end we came to the conclusion that the best way to fix this bug involved actually changing the current semantics of `rb_register_postponed_job`. I'm opening this issue to get feedback on this approach and to see if anybody knows of a reason why we should not release this for Ruby 3.3. ## Current behaviour in Ruby 3.2 Currently, Ruby has two functions for interacting with postponed jobs. These jobs can be enqueued from anywhere (including signal handlers), and will be executed next time Ruby checks for `RUBY_VM_CHECK_INTS()`. * `rb_postponed_job_register(func, data)`: Schedules `func(data)` to be executed the next time `RUBY_VM_CHECK_INTS` is checked. * `rb_postponed_job_register_once(func, data)`: Works like `rb_postponed_job_register`, _except_ if `func` is already scheduled to be executed (either with this `data` or with different `data`), in which case it does nothing. The postponed jobs are stored in a fixed sized array (of length 1024), so it's possible that enqueuing them could fail if the buffer is full. In this case, they signal this by returning `0` (otherwise, they return `1` for successful enqueue or `2` because `rb_postponed_job_register_once` did nothing because `func` was already in the queue). Unfortunately, as I mentioned before, the implementation of these functions are subject to a race condition because `func` and `data` are not written into the postponed job buffer together atomically (they are two separate variables and CPUs tend not to have double-word atomic instructions). Again, see https://bugs.ruby-lang.org/issues/19991 for the full details. ## What we have done Whilst working on this issue, we had a look at all of the in-the-wild usages of these APIs on rubygems. The only real usage of these APIs is for profiling tools, and the following was true for essentially all of them: * Each gem only is registering a single callback function, * Almost all of the usages either make no use of the `data` argument at all, or pass some kind of never-changing global context into it. * There are only a very small handful of gems using these APIs at all Thus, we concluded that the current behaviour of allowing scheduling and execution of arbitrary `(func, data)` pairs is actually not really needed. Instead, we could offer a more limited API which would meet the needs of all current users, whilst making it easy to avoid the race conditions in the current implementation. The new API is as follows: * `rb_postponed_job_preregister(func, data)`: This function registers `func`/`data` into a small, fixed-size table, and return a handle to this registration. Subsequent calls to this function with the same `func` will return the same handle, and overwrite the `data` with new data if it is different. The size of the table is 32 entries on most systems, which is still enough to use literally every gem on rubygems that actually uses these APIs at the same time. The intention is that libraries would call this function in their initialization routines, storing the handle for later. * `rb_postponed_job_trigger(handle)`: This function takes the handle from `rb_postponed_job_preregister` and schedules it for execution the next time `RUBY_VM_CHECK_INTS` is called. If the handle is already scheduled, this will not cause it to be scheduled twice; each `func` can only be called a maximum of one time for each call to `RUBY_VM_CHECK_INTS`, essentially. All of the usages of the old `rb_postponed_job_register{,_once}` functions in the Ruby tree have been replaced by calls to the above two functions, and these two old functions have been marked with the deprecated attribute. They have also been re-implemented in terms of the new functions; both `rb_postponed_job_register` and `rb_postponed_job_register_once` are now both equivalent to `rb_postponed_job_trigger(rb_postponed_job_prereigster(func, data))`. This means that: * `rb_postponed_job_register` now works like `rb_postponed_job_register_once` i.e. `func` can only be executed one time per `RUBY_VM_CHECK_INTS`, no matter how many times it is registered * They are also called with the _last_ `data` to be registered, not the first (which is how `rb_postponed_job_register_once` previously worked) I verified that stackprof still builds & works correctly with the new implementation of `rb_postponed_job_register`. ## What else we tried I tried a couple of things to keep the current semantics of `rb_postponed_job_register{,_once}` intact, without introducing new APIs. * First, I tried protecting postponed job buffer by masking signals around the critical section & using a POSIX semaphore instead of a pthread mutex: https://github.com/ruby/ruby/pull/8856. However, there was a concern that this would be too slow (since `RUBY_VM_CHECK_INTS` is called very often, and both the semaphore and the signal mask require calling into the kernel). * Then, I implemented a lock-free ringbuffer to store the postponed job queue: https://github.com/ruby/ruby/compare/master...KJTsanaktsidis:ruby:old_circu…. However, the concern with this implementation was that it was too complex. ## Ruby 3.3 As of right now, we have merged these changes (from https://github.com/ruby/ruby/pull/9041) and @ko1 plans for them to go out in 3.3-rc1. The point of opening this issue is to ask: does anybody foresee any problem with our approach? -- https://bugs.ruby-lang.org/

3 7

[ruby-core:115947] [Ruby master Bug#20101] rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set
by kjtsanaktsidis (KJ Tsanaktsidis) 10 Jan '24

10 Jan '24

Issue #20101 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20101: rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set https://bugs.ruby-lang.org/issues/20101 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When opening a file with `File.open`, as long as `'b'` is not set in the mode, Ruby will perform CRLF -> LF conversion on Windows when reading text files - i.e. CRLF line endings on disk get converted to Ruby strings with only "\n" in them. If you explicitly set the encoding with `IO#set_encoding`, this still works properly. If you open the file in C with either the `rb_io_fdopen` or `rb_file_open` APIs in text mode, CRLF -> LF conversion also works. However, if you then call `IO#set_encoding` on this file, the CRLF -> LF conversion stops happening. Concretely, this means that the conversion doesn't happen in the following circumstances: * When loading ruby files with require (that calls `rb_io_fdopen`) * When parsing ruuby files with RubyVM::AbstractSyntaxTree (that calls `rb_file_open`). This then causes the ErrorHighlight tests to fail on windows if git has checked them out with CRLF line endings - the error messages it's testing wind up with literal \r\n sequences in them because the iseq text from the parser contains un-newline-converted strings. This seems to happen because, in `File.open`, the file's encflags get the flag `ECONV_DEFAULT_NEWLINE_DECORATOR` in `rb_io_extract_modeenc`; however, this method isn't called for `rb_io_fdopen` or `rb_file_open`, so `encflags` doesn't get set to `ECONV_DEFAULT_NEWLINE_DECORATOR`. Without that flag, the underlying file descriptor's mode gets changed to binary mode by the `NEED_NEWLINE_DECORATOR_ON_READ_CHECK` macro. -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115104] [Ruby master Feature#19965] Make the name resolution interruptible
by mame (Yusuke Endoh) 09 Jan '24

09 Jan '24

Issue #19965 has been reported by mame (Yusuke Endoh). ---------------------------------------- Feature #19965: Make the name resolution interruptible https://bugs.ruby-lang.org/issues/19965 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- ## Problem Currently, Ruby name resolution is not interruptible. ``` $ cat /etc/resolv.conf nameserver 198.51.100.1 $ ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C^C^C^C ``` If you set a non-responsive IP as the nameserver, you cannot stop `Addrinfo.getaddrinfo` by pressing Ctrl+C. Note that `Timeout.timeout` does not work either. This is because there is no way to cancel `getaddrinfo(3)`. ## Proposal I wrote a patch to make `getaddrinfo(3)` work in a separate pthread. https://github.com/ruby/ruby/pull/8695 Whenever it needs name resolution, it creates a worker pthread, and executes `getaddrinfo(3)` in it. The caller thread waits for the worker to complete. When an interrupt occurs, the caller thread leaves stop waiting and leaves the worker pthread. The detached worker pthread will exit after `getaddrinfo(3)` completes (or name resolution times out). ## Evaluation By applying this patch, name resolution is now interruptible. ``` $ ./local/bin/ruby -rsocket -e 'pp Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C-e:1:in `getaddrinfo': Interrupt from -e:1:in `<main>' ``` As a drawback, name resolution performance will be degraded. ``` 10000.times { Addrinfo.getaddrinfo("www.ruby-lang.org", 80) } # Before patch: 2.3 sec. # After ptach: 3.0 sec. ``` However, I think that name resolution is typically short enough for the application's runtime. For example, the difference is small for the performance of `URI.open`. ``` 100.times { URI.open("https://www.ruby-lang.org").read } # Before patch: 3.36 sec. # After ptach: 3.40 sec. ``` ## Alternative approaches I proposed using c-ares to resolve this issue (#19430). However, there was an opinion that it would be a problem that c-ares does not respect the platform-dependent own name resolution. ## Room for improvement * Currently, this patch works only when pthread is available. * It might be possible to force to stop the worker threads by using `pthread_cancel`. However, `pthread_cancel` with `getaddrinfo(3)` seems still premature; there seems to be a bug in glibc until recently: https://bugzilla.redhat.com/show_bug.cgi?id=1405071 https://sourceware.org/bugzilla/show_bug.cgi?id=20975 * It would be more efficient to pool worker pthreads instead of creating them each time. -- https://bugs.ruby-lang.org/

3 6

[ruby-core:112208] [Ruby master Bug#19409] Object's shape is reset after a ractor move
by luke-gru (Luke Gruber) 04 Jan '24

04 Jan '24

Issue #19409 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19409: Object's shape is reset after a ractor move https://bugs.ruby-lang.org/issues/19409 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I believe an object should have the same shape after being moved from 1 ractor to another. ```ruby class Obj attr_accessor :a, :b, :c, :d def initialize @a = 1 @b = 2 @c = 3 end end r = Ractor.new do obj = receive #p RubyVM::Shape.of(obj) obj.d = 4 p obj.a, obj.b, obj.c, obj.d # gets wrong values due to object shape id being reset on object end obj = Obj.new #p RubyVM::Shape.of(obj) r.send(obj, move: true) r.take ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112211] [Ruby master Bug#19411] GC issue with moved objects
by luke-gru (Luke Gruber) 04 Jan '24

04 Jan '24

Issue #19411 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19411: GC issue with moved objects https://bugs.ruby-lang.org/issues/19411 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This crashes: ```ruby class Obj def initialize @obj = 3 end end GC.stress = true r = Ractor.new do obj = receive p obj end obj = Obj.new r.send(obj, move: true) r.take ``` It only crashes with nested objects, if you remove the ivar set in `initialize` it works fine. Maybe missing `RB_GC_GUARD`? -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115671] [Ruby master Feature#20054] Replace the use of `def` in endless method definitions with a new sigil
by sawa (Tsuyoshi Sawada) 04 Jan '24

04 Jan '24

Issue #20054 has been reported by sawa (Tsuyoshi Sawada). ---------------------------------------- Feature #20054: Replace the use of `def` in endless method definitions with a new sigil https://bugs.ruby-lang.org/issues/20054 * Author: sawa (Tsuyoshi Sawada) * Status: Open * Priority: Normal ---------------------------------------- I propose to remove the use of keyword `def` from the syntax of endless method definition, and introduce a new sigil instead of it. There are several possibilities for what character to use as the sigil, but the most seemingly promising one to me at this point is the colon. So, instead of: ```rb def foo = method_body ``` I propose to write ```rb :foo = method_body ``` There a few reasons to dispense with `def` in endless method definition. First, the current syntax for endless method definition looks too similar to conventional method definition. Without endless method definition, we could already define a method in a single line: ```rb def foo; method_body end ``` and compared to this, what the endless method definition does is that, it only saves you from typing the `end` keyword just by replacing the semicolon with an equal sign. This actually had not made much sense to me. Just saving you from typing the keyword `end` looks too small of a change for introducing new syntax. In order for endless method definition syntax to be justified (as a shorthand for conventional method definition), it needs to save more typing. Second, in #19392, some people are claiming to change the precedence involving endless method definition. I agree with Matz and other developers who support the current precedence in which: ```rb def foo = bar and baz ``` is interpreted as: ```rb (def foo = bar) and baz ``` and I understand that the controversy is due to the look and feel of the keyword `def`. `def` has lower precedence than `and` in conventional method definition, although `=` has higher precedence than `and` in variable/constant assignment. Mixing the low-precedence `def` and the high-precedence `=` into a single syntax was the cause of the trouble, according to my opinion. Thence, we should get rid of `def`. Once we do so, we need to distinguish endless method definition from variable/constant assignment in a new way. What came to my mind was to use a single character: a sigil. Especially, using the colon seems to make sense to me for several reasons: Most importantly, assignment to a symbol is impossible, and it currently raises a syntax error, so it would not conflict with variable/constant assignment syntax. Within Ruby syntax, symbol is naturally used to represent a method name. For example, in `foo(&:bar)` constructions, users are used to passing a method name as a symbol. Also, a method definition returns a symbol representing the method name. So, making the endless method definition syntax look superficially like an "assignment to a symbol" would make sense. -- https://bugs.ruby-lang.org/

6 6