December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113788] [Ruby master Bug#19717] `ConditionVariable#signal` is not fair when the wakeup is consistently spurious.
by ioquatix (Samuel Williams) 10 Jan '24

10 Jan '24

Issue #19717 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19717: `ConditionVariable#signal` is not fair when the wakeup is consistently spurious. https://bugs.ruby-lang.org/issues/19717 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- For background, see this issue <https://github.com/socketry/async/issues/99>. It looks like `ConditionVariable#signal` is not fair, if the calling thread immediately reacquires the resource. I've given a detailed reproduction here as it's non-trivial: <https://github.com/ioquatix/ruby-condition-variable-timeout>. Because the spurious wakeup occurs, the thread is pushed to the back of the waitq, which means any other waiting thread will acquire the resource, and that thread will perpetually be at the back of the queue. I believe the solution is to change `ConditionVarialbe#signal` should only remove the thread from the waitq if it's possible to acquire the lock. Otherwise it should be left in place, so that the order is retained, this should result in fair scheduling. -- https://bugs.ruby-lang.org/

3 12

[ruby-core:115684] [Ruby master Feature#20057] Change behaviour of rb_register_postponed_job for Ruby 3.3
by kjtsanaktsidis (KJ Tsanaktsidis) 10 Jan '24

10 Jan '24

Issue #20057 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #20057: Change behaviour of rb_register_postponed_job for Ruby 3.3 https://bugs.ruby-lang.org/issues/20057 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- This ticket is to discuss some changes to `rb_register_postponed_job` that @ko1 and myself propose to make for Ruby 3.3. The motivation for this work is to fix a bug in the current implementation, which can cause the registered functions to be called with the wrong `data` argument (https://bugs.ruby-lang.org/issues/19991) There's a long discussion on the associated PR (https://github.com/ruby/ruby/pull/9041) but in the end we came to the conclusion that the best way to fix this bug involved actually changing the current semantics of `rb_register_postponed_job`. I'm opening this issue to get feedback on this approach and to see if anybody knows of a reason why we should not release this for Ruby 3.3. ## Current behaviour in Ruby 3.2 Currently, Ruby has two functions for interacting with postponed jobs. These jobs can be enqueued from anywhere (including signal handlers), and will be executed next time Ruby checks for `RUBY_VM_CHECK_INTS()`. * `rb_postponed_job_register(func, data)`: Schedules `func(data)` to be executed the next time `RUBY_VM_CHECK_INTS` is checked. * `rb_postponed_job_register_once(func, data)`: Works like `rb_postponed_job_register`, _except_ if `func` is already scheduled to be executed (either with this `data` or with different `data`), in which case it does nothing. The postponed jobs are stored in a fixed sized array (of length 1024), so it's possible that enqueuing them could fail if the buffer is full. In this case, they signal this by returning `0` (otherwise, they return `1` for successful enqueue or `2` because `rb_postponed_job_register_once` did nothing because `func` was already in the queue). Unfortunately, as I mentioned before, the implementation of these functions are subject to a race condition because `func` and `data` are not written into the postponed job buffer together atomically (they are two separate variables and CPUs tend not to have double-word atomic instructions). Again, see https://bugs.ruby-lang.org/issues/19991 for the full details. ## What we have done Whilst working on this issue, we had a look at all of the in-the-wild usages of these APIs on rubygems. The only real usage of these APIs is for profiling tools, and the following was true for essentially all of them: * Each gem only is registering a single callback function, * Almost all of the usages either make no use of the `data` argument at all, or pass some kind of never-changing global context into it. * There are only a very small handful of gems using these APIs at all Thus, we concluded that the current behaviour of allowing scheduling and execution of arbitrary `(func, data)` pairs is actually not really needed. Instead, we could offer a more limited API which would meet the needs of all current users, whilst making it easy to avoid the race conditions in the current implementation. The new API is as follows: * `rb_postponed_job_preregister(func, data)`: This function registers `func`/`data` into a small, fixed-size table, and return a handle to this registration. Subsequent calls to this function with the same `func` will return the same handle, and overwrite the `data` with new data if it is different. The size of the table is 32 entries on most systems, which is still enough to use literally every gem on rubygems that actually uses these APIs at the same time. The intention is that libraries would call this function in their initialization routines, storing the handle for later. * `rb_postponed_job_trigger(handle)`: This function takes the handle from `rb_postponed_job_preregister` and schedules it for execution the next time `RUBY_VM_CHECK_INTS` is called. If the handle is already scheduled, this will not cause it to be scheduled twice; each `func` can only be called a maximum of one time for each call to `RUBY_VM_CHECK_INTS`, essentially. All of the usages of the old `rb_postponed_job_register{,_once}` functions in the Ruby tree have been replaced by calls to the above two functions, and these two old functions have been marked with the deprecated attribute. They have also been re-implemented in terms of the new functions; both `rb_postponed_job_register` and `rb_postponed_job_register_once` are now both equivalent to `rb_postponed_job_trigger(rb_postponed_job_prereigster(func, data))`. This means that: * `rb_postponed_job_register` now works like `rb_postponed_job_register_once` i.e. `func` can only be executed one time per `RUBY_VM_CHECK_INTS`, no matter how many times it is registered * They are also called with the _last_ `data` to be registered, not the first (which is how `rb_postponed_job_register_once` previously worked) I verified that stackprof still builds & works correctly with the new implementation of `rb_postponed_job_register`. ## What else we tried I tried a couple of things to keep the current semantics of `rb_postponed_job_register{,_once}` intact, without introducing new APIs. * First, I tried protecting postponed job buffer by masking signals around the critical section & using a POSIX semaphore instead of a pthread mutex: https://github.com/ruby/ruby/pull/8856. However, there was a concern that this would be too slow (since `RUBY_VM_CHECK_INTS` is called very often, and both the semaphore and the signal mask require calling into the kernel). * Then, I implemented a lock-free ringbuffer to store the postponed job queue: https://github.com/ruby/ruby/compare/master...KJTsanaktsidis:ruby:old_circu…. However, the concern with this implementation was that it was too complex. ## Ruby 3.3 As of right now, we have merged these changes (from https://github.com/ruby/ruby/pull/9041) and @ko1 plans for them to go out in 3.3-rc1. The point of opening this issue is to ask: does anybody foresee any problem with our approach? -- https://bugs.ruby-lang.org/

3 7

[ruby-core:115947] [Ruby master Bug#20101] rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set
by kjtsanaktsidis (KJ Tsanaktsidis) 10 Jan '24

10 Jan '24

Issue #20101 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20101: rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set https://bugs.ruby-lang.org/issues/20101 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When opening a file with `File.open`, as long as `'b'` is not set in the mode, Ruby will perform CRLF -> LF conversion on Windows when reading text files - i.e. CRLF line endings on disk get converted to Ruby strings with only "\n" in them. If you explicitly set the encoding with `IO#set_encoding`, this still works properly. If you open the file in C with either the `rb_io_fdopen` or `rb_file_open` APIs in text mode, CRLF -> LF conversion also works. However, if you then call `IO#set_encoding` on this file, the CRLF -> LF conversion stops happening. Concretely, this means that the conversion doesn't happen in the following circumstances: * When loading ruby files with require (that calls `rb_io_fdopen`) * When parsing ruuby files with RubyVM::AbstractSyntaxTree (that calls `rb_file_open`). This then causes the ErrorHighlight tests to fail on windows if git has checked them out with CRLF line endings - the error messages it's testing wind up with literal \r\n sequences in them because the iseq text from the parser contains un-newline-converted strings. This seems to happen because, in `File.open`, the file's encflags get the flag `ECONV_DEFAULT_NEWLINE_DECORATOR` in `rb_io_extract_modeenc`; however, this method isn't called for `rb_io_fdopen` or `rb_file_open`, so `encflags` doesn't get set to `ECONV_DEFAULT_NEWLINE_DECORATOR`. Without that flag, the underlying file descriptor's mode gets changed to binary mode by the `NEED_NEWLINE_DECORATOR_ON_READ_CHECK` macro. -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115104] [Ruby master Feature#19965] Make the name resolution interruptible
by mame (Yusuke Endoh) 09 Jan '24

09 Jan '24

Issue #19965 has been reported by mame (Yusuke Endoh). ---------------------------------------- Feature #19965: Make the name resolution interruptible https://bugs.ruby-lang.org/issues/19965 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- ## Problem Currently, Ruby name resolution is not interruptible. ``` $ cat /etc/resolv.conf nameserver 198.51.100.1 $ ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C^C^C^C ``` If you set a non-responsive IP as the nameserver, you cannot stop `Addrinfo.getaddrinfo` by pressing Ctrl+C. Note that `Timeout.timeout` does not work either. This is because there is no way to cancel `getaddrinfo(3)`. ## Proposal I wrote a patch to make `getaddrinfo(3)` work in a separate pthread. https://github.com/ruby/ruby/pull/8695 Whenever it needs name resolution, it creates a worker pthread, and executes `getaddrinfo(3)` in it. The caller thread waits for the worker to complete. When an interrupt occurs, the caller thread leaves stop waiting and leaves the worker pthread. The detached worker pthread will exit after `getaddrinfo(3)` completes (or name resolution times out). ## Evaluation By applying this patch, name resolution is now interruptible. ``` $ ./local/bin/ruby -rsocket -e 'pp Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C-e:1:in `getaddrinfo': Interrupt from -e:1:in `<main>' ``` As a drawback, name resolution performance will be degraded. ``` 10000.times { Addrinfo.getaddrinfo("www.ruby-lang.org", 80) } # Before patch: 2.3 sec. # After ptach: 3.0 sec. ``` However, I think that name resolution is typically short enough for the application's runtime. For example, the difference is small for the performance of `URI.open`. ``` 100.times { URI.open("https://www.ruby-lang.org").read } # Before patch: 3.36 sec. # After ptach: 3.40 sec. ``` ## Alternative approaches I proposed using c-ares to resolve this issue (#19430). However, there was an opinion that it would be a problem that c-ares does not respect the platform-dependent own name resolution. ## Room for improvement * Currently, this patch works only when pthread is available. * It might be possible to force to stop the worker threads by using `pthread_cancel`. However, `pthread_cancel` with `getaddrinfo(3)` seems still premature; there seems to be a bug in glibc until recently: https://bugzilla.redhat.com/show_bug.cgi?id=1405071 https://sourceware.org/bugzilla/show_bug.cgi?id=20975 * It would be more efficient to pool worker pthreads instead of creating them each time. -- https://bugs.ruby-lang.org/

3 6

[ruby-core:112208] [Ruby master Bug#19409] Object's shape is reset after a ractor move
by luke-gru (Luke Gruber) 04 Jan '24

04 Jan '24

Issue #19409 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19409: Object's shape is reset after a ractor move https://bugs.ruby-lang.org/issues/19409 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I believe an object should have the same shape after being moved from 1 ractor to another. ```ruby class Obj attr_accessor :a, :b, :c, :d def initialize @a = 1 @b = 2 @c = 3 end end r = Ractor.new do obj = receive #p RubyVM::Shape.of(obj) obj.d = 4 p obj.a, obj.b, obj.c, obj.d # gets wrong values due to object shape id being reset on object end obj = Obj.new #p RubyVM::Shape.of(obj) r.send(obj, move: true) r.take ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112211] [Ruby master Bug#19411] GC issue with moved objects
by luke-gru (Luke Gruber) 04 Jan '24

04 Jan '24

Issue #19411 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19411: GC issue with moved objects https://bugs.ruby-lang.org/issues/19411 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * ruby -v: 3.2.0 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This crashes: ```ruby class Obj def initialize @obj = 3 end end GC.stress = true r = Ractor.new do obj = receive p obj end obj = Obj.new r.send(obj, move: true) r.take ``` It only crashes with nested objects, if you remove the ivar set in `initialize` it works fine. Maybe missing `RB_GC_GUARD`? -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115671] [Ruby master Feature#20054] Replace the use of `def` in endless method definitions with a new sigil
by sawa (Tsuyoshi Sawada) 04 Jan '24

04 Jan '24

Issue #20054 has been reported by sawa (Tsuyoshi Sawada). ---------------------------------------- Feature #20054: Replace the use of `def` in endless method definitions with a new sigil https://bugs.ruby-lang.org/issues/20054 * Author: sawa (Tsuyoshi Sawada) * Status: Open * Priority: Normal ---------------------------------------- I propose to remove the use of keyword `def` from the syntax of endless method definition, and introduce a new sigil instead of it. There are several possibilities for what character to use as the sigil, but the most seemingly promising one to me at this point is the colon. So, instead of: ```rb def foo = method_body ``` I propose to write ```rb :foo = method_body ``` There a few reasons to dispense with `def` in endless method definition. First, the current syntax for endless method definition looks too similar to conventional method definition. Without endless method definition, we could already define a method in a single line: ```rb def foo; method_body end ``` and compared to this, what the endless method definition does is that, it only saves you from typing the `end` keyword just by replacing the semicolon with an equal sign. This actually had not made much sense to me. Just saving you from typing the keyword `end` looks too small of a change for introducing new syntax. In order for endless method definition syntax to be justified (as a shorthand for conventional method definition), it needs to save more typing. Second, in #19392, some people are claiming to change the precedence involving endless method definition. I agree with Matz and other developers who support the current precedence in which: ```rb def foo = bar and baz ``` is interpreted as: ```rb (def foo = bar) and baz ``` and I understand that the controversy is due to the look and feel of the keyword `def`. `def` has lower precedence than `and` in conventional method definition, although `=` has higher precedence than `and` in variable/constant assignment. Mixing the low-precedence `def` and the high-precedence `=` into a single syntax was the cause of the trouble, according to my opinion. Thence, we should get rid of `def`. Once we do so, we need to distinguish endless method definition from variable/constant assignment in a new way. What came to my mind was to use a single character: a sigil. Especially, using the colon seems to make sense to me for several reasons: Most importantly, assignment to a symbol is impossible, and it currently raises a syntax error, so it would not conflict with variable/constant assignment syntax. Within Ruby syntax, symbol is naturally used to represent a method name. For example, in `foo(&:bar)` constructions, users are used to passing a method name as a symbol. Also, a method definition returns a symbol representing the method name. So, making the endless method definition syntax look superficially like an "assignment to a symbol" would make sense. -- https://bugs.ruby-lang.org/

6 6

[ruby-core:115980] [Ruby master Misc#17815] Snapcraft Ruby plugin
by paddor (Patrik Wenger) 31 Dec '23

31 Dec '23

Issue #17815 has been updated by paddor (Patrik Wenger). This can be closed. I've implemented a Ruby build plugin on v1 and v2. ---------------------------------------- Misc #17815: Snapcraft Ruby plugin https://bugs.ruby-lang.org/issues/17815#change-105943 * Author: paddor (Patrik Wenger) * Status: Open * Priority: Normal ---------------------------------------- I'm working on a Ruby build plugin for the Snapcraft v2 plugin AP, since the v1 Ruby plugin does not work on the Ubuntu `core20` image. The v2 API only allows to influence the build step, which means there's no nice way for the plugin to set environment variables that apply during runtime. The problem is that the final paths to Ruby's stdlib are unknown during compile time. I've got it to work by renaming the `ruby` executable to `ruby.bare` and creating a wrapper script `ruby` that sets the `$RUBYLIB` and `$GEM_PATH` env variables based on `$SNAP` before `exec`ing `ruby.bare`. I don't like that it requires a wrapper script though, since it would have to awkwardly determine the arch-specific stdlib directory to support any platform. The current wrapper script is this: ```sh #!/bin/sh export RUBYLIB="$SNAP/lib/ruby/snap:$SNAP/lib/ruby/snap/x86_64-linux:$RUBYLIB" export GEM_PATH="$SNAP/lib/ruby/gems/snap:$GEM_PATH" exec `dirname $0`/ruby.bare "$@" ``` What possible solutions are there? I could maybe fix the shebangs of Ruby executables like `gem` and `bundle`, but not `ruby` itself since it's binary. Also, I'm not sure if env variable expansion even works in shebang. Can the configure script's `--with-search-path` be made to support env variable expansion during runtime? That still wouldn't solve the issue about the arch-specific directory. Python apparently determines its load paths relative to the executable being run. Does Ruby support anything like that? Any other ideas? Current state: https://github.com/paddor/snapcraft-ruby-plugin-v2 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112010] [Ruby master Bug#19375] File objects are currently shareable
by luke-gru (Luke Gruber) 29 Dec '23

29 Dec '23

Issue #19375 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19375: File objects are currently shareable https://bugs.ruby-lang.org/issues/19375 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I don't know the internals of file.c but I don't think files are thread-safe. -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115919] [Ruby master Feature#20093] Syntax or keyword to reopen existing classs/modules, never to define new classs/modules
by tagomoris (Satoshi Tagomori) 29 Dec '23

29 Dec '23

Issue #20093 has been reported by tagomoris (Satoshi Tagomori). ---------------------------------------- Feature #20093: Syntax or keyword to reopen existing classs/modules, never to define new classs/modules https://bugs.ruby-lang.org/issues/20093 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- `class A` and `module B` will reopen existing class A or module B to add/re-define methods if A/B exists. Otherwise, these will define the new class/module A/B. But, in my opinion, the code of `class A` for patching existing classes doesn't work expectedly when `A` is not defined beforehand. It expects other codes to define `A` before being called. For example: ```ruby # string_exclude.rb class String def exclude?(string) !include?(string) end end ``` This code expects that there is the `String` class, and it has the `include?` method. This code doesn't work if the file is loaded in the way below: ```ruby load('string_exclude.rb', true) ``` This code doesn't raise errors and will define an almost empty class (only with a method `exclude?` to raise NameError). It should be unexpected for every user. So, I want to propose a new syntax to reopen the existing class/module or raise errors if the specified class/module is not defined. ```ruby class extension String def exclude?(string) !include?(string) end end # adds #exclude? to String class class extension Stroooong def exclude?(string) !include?(string) end end # will raise NameError (or something else) ``` Some additional things: * `class extension String` (and `module extension String`) causes a compile error (SyntaxError) on Ruby 3.3. So we have space to add a keyword between class/module and the class/module name. * I don't have a strong opinion about the keyword name `extension`. An alternative idea is `reopen`. -- https://bugs.ruby-lang.org/

8 10