- ruby-core - ml.ruby-lang.org

[ruby-core:113788] [Ruby master Bug#19717] `ConditionVariable#signal` is not fair when the wakeup is consistently spurious.
by ioquatix (Samuel Williams) 10 Jan '24

10 Jan '24

Issue #19717 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19717: `ConditionVariable#signal` is not fair when the wakeup is consistently spurious. https://bugs.ruby-lang.org/issues/19717 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- For background, see this issue <https://github.com/socketry/async/issues/99>. It looks like `ConditionVariable#signal` is not fair, if the calling thread immediately reacquires the resource. I've given a detailed reproduction here as it's non-trivial: <https://github.com/ioquatix/ruby-condition-variable-timeout>. Because the spurious wakeup occurs, the thread is pushed to the back of the waitq, which means any other waiting thread will acquire the resource, and that thread will perpetually be at the back of the queue. I believe the solution is to change `ConditionVarialbe#signal` should only remove the thread from the waitq if it's possible to acquire the lock. Otherwise it should be left in place, so that the order is retained, this should result in fair scheduling. -- https://bugs.ruby-lang.org/

3 12

[ruby-core:115684] [Ruby master Feature#20057] Change behaviour of rb_register_postponed_job for Ruby 3.3
by kjtsanaktsidis (KJ Tsanaktsidis) 10 Jan '24

10 Jan '24

Issue #20057 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Feature #20057: Change behaviour of rb_register_postponed_job for Ruby 3.3 https://bugs.ruby-lang.org/issues/20057 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal ---------------------------------------- This ticket is to discuss some changes to `rb_register_postponed_job` that @ko1 and myself propose to make for Ruby 3.3. The motivation for this work is to fix a bug in the current implementation, which can cause the registered functions to be called with the wrong `data` argument (https://bugs.ruby-lang.org/issues/19991) There's a long discussion on the associated PR (https://github.com/ruby/ruby/pull/9041) but in the end we came to the conclusion that the best way to fix this bug involved actually changing the current semantics of `rb_register_postponed_job`. I'm opening this issue to get feedback on this approach and to see if anybody knows of a reason why we should not release this for Ruby 3.3. ## Current behaviour in Ruby 3.2 Currently, Ruby has two functions for interacting with postponed jobs. These jobs can be enqueued from anywhere (including signal handlers), and will be executed next time Ruby checks for `RUBY_VM_CHECK_INTS()`. * `rb_postponed_job_register(func, data)`: Schedules `func(data)` to be executed the next time `RUBY_VM_CHECK_INTS` is checked. * `rb_postponed_job_register_once(func, data)`: Works like `rb_postponed_job_register`, _except_ if `func` is already scheduled to be executed (either with this `data` or with different `data`), in which case it does nothing. The postponed jobs are stored in a fixed sized array (of length 1024), so it's possible that enqueuing them could fail if the buffer is full. In this case, they signal this by returning `0` (otherwise, they return `1` for successful enqueue or `2` because `rb_postponed_job_register_once` did nothing because `func` was already in the queue). Unfortunately, as I mentioned before, the implementation of these functions are subject to a race condition because `func` and `data` are not written into the postponed job buffer together atomically (they are two separate variables and CPUs tend not to have double-word atomic instructions). Again, see https://bugs.ruby-lang.org/issues/19991 for the full details. ## What we have done Whilst working on this issue, we had a look at all of the in-the-wild usages of these APIs on rubygems. The only real usage of these APIs is for profiling tools, and the following was true for essentially all of them: * Each gem only is registering a single callback function, * Almost all of the usages either make no use of the `data` argument at all, or pass some kind of never-changing global context into it. * There are only a very small handful of gems using these APIs at all Thus, we concluded that the current behaviour of allowing scheduling and execution of arbitrary `(func, data)` pairs is actually not really needed. Instead, we could offer a more limited API which would meet the needs of all current users, whilst making it easy to avoid the race conditions in the current implementation. The new API is as follows: * `rb_postponed_job_preregister(func, data)`: This function registers `func`/`data` into a small, fixed-size table, and return a handle to this registration. Subsequent calls to this function with the same `func` will return the same handle, and overwrite the `data` with new data if it is different. The size of the table is 32 entries on most systems, which is still enough to use literally every gem on rubygems that actually uses these APIs at the same time. The intention is that libraries would call this function in their initialization routines, storing the handle for later. * `rb_postponed_job_trigger(handle)`: This function takes the handle from `rb_postponed_job_preregister` and schedules it for execution the next time `RUBY_VM_CHECK_INTS` is called. If the handle is already scheduled, this will not cause it to be scheduled twice; each `func` can only be called a maximum of one time for each call to `RUBY_VM_CHECK_INTS`, essentially. All of the usages of the old `rb_postponed_job_register{,_once}` functions in the Ruby tree have been replaced by calls to the above two functions, and these two old functions have been marked with the deprecated attribute. They have also been re-implemented in terms of the new functions; both `rb_postponed_job_register` and `rb_postponed_job_register_once` are now both equivalent to `rb_postponed_job_trigger(rb_postponed_job_prereigster(func, data))`. This means that: * `rb_postponed_job_register` now works like `rb_postponed_job_register_once` i.e. `func` can only be executed one time per `RUBY_VM_CHECK_INTS`, no matter how many times it is registered * They are also called with the _last_ `data` to be registered, not the first (which is how `rb_postponed_job_register_once` previously worked) I verified that stackprof still builds & works correctly with the new implementation of `rb_postponed_job_register`. ## What else we tried I tried a couple of things to keep the current semantics of `rb_postponed_job_register{,_once}` intact, without introducing new APIs. * First, I tried protecting postponed job buffer by masking signals around the critical section & using a POSIX semaphore instead of a pthread mutex: https://github.com/ruby/ruby/pull/8856. However, there was a concern that this would be too slow (since `RUBY_VM_CHECK_INTS` is called very often, and both the semaphore and the signal mask require calling into the kernel). * Then, I implemented a lock-free ringbuffer to store the postponed job queue: https://github.com/ruby/ruby/compare/master...KJTsanaktsidis:ruby:old_circu…. However, the concern with this implementation was that it was too complex. ## Ruby 3.3 As of right now, we have merged these changes (from https://github.com/ruby/ruby/pull/9041) and @ko1 plans for them to go out in 3.3-rc1. The point of opening this issue is to ask: does anybody foresee any problem with our approach? -- https://bugs.ruby-lang.org/

3 7

[ruby-core:115947] [Ruby master Bug#20101] rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set
by kjtsanaktsidis (KJ Tsanaktsidis) 10 Jan '24

10 Jan '24

Issue #20101 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20101: rb_file_open and rb_io_fdopen don't perform CRLF -> LF conversion when encoding is set https://bugs.ruby-lang.org/issues/20101 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When opening a file with `File.open`, as long as `'b'` is not set in the mode, Ruby will perform CRLF -> LF conversion on Windows when reading text files - i.e. CRLF line endings on disk get converted to Ruby strings with only "\n" in them. If you explicitly set the encoding with `IO#set_encoding`, this still works properly. If you open the file in C with either the `rb_io_fdopen` or `rb_file_open` APIs in text mode, CRLF -> LF conversion also works. However, if you then call `IO#set_encoding` on this file, the CRLF -> LF conversion stops happening. Concretely, this means that the conversion doesn't happen in the following circumstances: * When loading ruby files with require (that calls `rb_io_fdopen`) * When parsing ruuby files with RubyVM::AbstractSyntaxTree (that calls `rb_file_open`). This then causes the ErrorHighlight tests to fail on windows if git has checked them out with CRLF line endings - the error messages it's testing wind up with literal \r\n sequences in them because the iseq text from the parser contains un-newline-converted strings. This seems to happen because, in `File.open`, the file's encflags get the flag `ECONV_DEFAULT_NEWLINE_DECORATOR` in `rb_io_extract_modeenc`; however, this method isn't called for `rb_io_fdopen` or `rb_file_open`, so `encflags` doesn't get set to `ECONV_DEFAULT_NEWLINE_DECORATOR`. Without that flag, the underlying file descriptor's mode gets changed to binary mode by the `NEED_NEWLINE_DECORATOR_ON_READ_CHECK` macro. -- https://bugs.ruby-lang.org/

1 2

[ruby-core:115104] [Ruby master Feature#19965] Make the name resolution interruptible
by mame (Yusuke Endoh) 09 Jan '24

09 Jan '24

Issue #19965 has been reported by mame (Yusuke Endoh). ---------------------------------------- Feature #19965: Make the name resolution interruptible https://bugs.ruby-lang.org/issues/19965 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- ## Problem Currently, Ruby name resolution is not interruptible. ``` $ cat /etc/resolv.conf nameserver 198.51.100.1 $ ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C^C^C^C ``` If you set a non-responsive IP as the nameserver, you cannot stop `Addrinfo.getaddrinfo` by pressing Ctrl+C. Note that `Timeout.timeout` does not work either. This is because there is no way to cancel `getaddrinfo(3)`. ## Proposal I wrote a patch to make `getaddrinfo(3)` work in a separate pthread. https://github.com/ruby/ruby/pull/8695 Whenever it needs name resolution, it creates a worker pthread, and executes `getaddrinfo(3)` in it. The caller thread waits for the worker to complete. When an interrupt occurs, the caller thread leaves stop waiting and leaves the worker pthread. The detached worker pthread will exit after `getaddrinfo(3)` completes (or name resolution times out). ## Evaluation By applying this patch, name resolution is now interruptible. ``` $ ./local/bin/ruby -rsocket -e 'pp Addrinfo.getaddrinfo("www.ruby-lang.org", 80)' ^C-e:1:in `getaddrinfo': Interrupt from -e:1:in `<main>' ``` As a drawback, name resolution performance will be degraded. ``` 10000.times { Addrinfo.getaddrinfo("www.ruby-lang.org", 80) } # Before patch: 2.3 sec. # After ptach: 3.0 sec. ``` However, I think that name resolution is typically short enough for the application's runtime. For example, the difference is small for the performance of `URI.open`. ``` 100.times { URI.open("https://www.ruby-lang.org").read } # Before patch: 3.36 sec. # After ptach: 3.40 sec. ``` ## Alternative approaches I proposed using c-ares to resolve this issue (#19430). However, there was an opinion that it would be a problem that c-ares does not respect the platform-dependent own name resolution. ## Room for improvement * Currently, this patch works only when pthread is available. * It might be possible to force to stop the worker threads by using `pthread_cancel`. However, `pthread_cancel` with `getaddrinfo(3)` seems still premature; there seems to be a bug in glibc until recently: https://bugzilla.redhat.com/show_bug.cgi?id=1405071 https://sourceware.org/bugzilla/show_bug.cgi?id=20975 * It would be more efficient to pool worker pthreads instead of creating them each time. -- https://bugs.ruby-lang.org/

3 6

[ruby-core:116125] [Ruby master Bug#20171] ri -i ruby interactive pressing tab does not autocomplete
by Mystorium (Josh Goldfarb) 09 Jan '24

09 Jan '24

Issue #20171 has been reported by Mystorium (Josh Goldfarb). ---------------------------------------- Bug #20171: ri -i ruby interactive pressing tab does not autocomplete https://bugs.ruby-lang.org/issues/20171 * Author: Mystorium (Josh Goldfarb) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When typing a Object or Object with a method, the tab button does not pull up autocomplete options. This issue is not present in ruby 3.2.2 Ex Type - ri - i - String.cap Press. - "Tab" What is expected - autocomplete options to populate What is happening - Nothing Happens when pressing tab. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116122] [Ruby master Feature#18948] Add `with_private_method` option to `private_constant`
by okuramasafumi (Masafumi OKURA) 09 Jan '24

09 Jan '24

Issue #18948 has been updated by okuramasafumi (Masafumi OKURA). I'm convinced by Jeremy's argument. Please someone close this issue. ---------------------------------------- Feature #18948: Add `with_private_method` option to `private_constant` https://bugs.ruby-lang.org/issues/18948#change-106123 * Author: okuramasafumi (Masafumi OKURA) * Status: Open * Priority: Normal ---------------------------------------- ## Problem I have the following code: ```ruby module M1 CONST = 'CONST'.freeze end module M2 def self.included(base) base.include M1 base.include InstanceMethods end module InstanceMethods def const?(value) M1::CONST == value end end end class C include M2 end C.new.const?('CONST') # => true ``` `M2` module "inherits" `M1` module using `included` hook. This code works, but I prefer making constants private. However, when I add `private_constant :CONST` to `M1` then the code doesn't work anymore since we cannot refer `M1::CONST`. ## Workaround One way to solve this is to define method that just returns the private constant. ```ruby # Refined version of M1 with `const` method module M1 CONST = 'CONST'.freeze private_constant :CONST private def const CONST end end # Refined version of M2, using `const` method inherited from M1 module M2 def self.included(base) base.include M1 base.include InstanceMethods end module InstanceMethods def const?(value) const == value end end end ``` This works anyway, but defining private method for every private constant could be cumbersome. ## Solution So my suggestion here is to add `with_private_method` option to `private_constant` method. The auto-generated private method has the same name as the given constant and just returns it. So we can do the following: ```ruby module M1 CONST = 'CONST'.freeze private_constant :CONST, with_private_method: true end ``` Now we can access `CONST` with `const` method. This makes is possible to refer private constants through private methods more easily, that helps some code like my example. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116063] [Ruby master Bug#20159] Prism assertion failure even if it is executed with --parser=parse.y
by tompng (tomoya ishida) 09 Jan '24

09 Jan '24

Issue #20159 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20159: Prism assertion failure even if it is executed with --parser=parse.y https://bugs.ruby-lang.org/issues/20159 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-01-07T16:23:58Z master 8b86d6f0c1) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Prism fails to parse this code with assertion failed. Reported at https://github.com/ruby/prism/issues/1616 ~~~ruby <<A+% A ~~~ Ruby exits with assertion failed even if it's run with option `--parser=parse.y` ~~~ # ruby -v ruby 3.4.0dev (2024-01-07T16:23:58Z master 8b86d6f0c1) [x86_64-linux] # (echo "<<A+%"; echo "A") > a.rb # ruby --parser=parse.y a.rb ruby: prism/util/pm_newline_list.c:42: pm_newline_list_append: Assertion `list->size == 0 || newline_offset > list->offsets[list->size - 1]' failed. qemu: uncaught target signal 6 (Aborted) - core dumped Aborted ~~~ `eval` will correclty result in SyntaxError, `require` and `load` exits with assertion failed. ~~~ruby # irb irb(main):001> eval File.read('a.rb') (irb):1:in `eval': (eval at (irb):1):1: unterminated string meets end of file (SyntaxError) irb(main):002> load 'a.rb' prism/util/pm_newline_list.c:42: pm_newline_list_append: Assertion `list->size == 0 || newline_offset > list->offsets[list->size - 1]' failed. qemu: uncaught target signal 6 (Aborted) - core dumped Aborted ~~~ -- https://bugs.ruby-lang.org/

3 5

[ruby-core:116080] shadow.so: undefined symbol: rb_tainted_str_new2
by Jay Mav 09 Jan '24

09 Jan '24

Hello Ruby-Shadow owners and Ruby Dev Team, On Linux, using Ruby 3.2.2, getting shadow.so: undefined symbol: rb_tainted_str_new2 with ruby-shadow-ruby32-2.5.1.gem. The following code in this gem uses rb_tainted_str_new2: pwd/shadow.c:59: rb_tainted_str_new2(entry->pw_name), /* sp_namp */ pwd/shadow.c:60: rb_tainted_str_new2(entry->pw_passwd), /* sp_pwdp, encryped password */ pwd/shadow.c:69: rb_tainted_str_new2(entry->pw_class), /* sp_loginclass, user access class */ shadow/shadow.c:37: rb_tainted_str_new2(entry->sp_namp), shadow/shadow.c:38: rb_tainted_str_new2(entry->sp_pwdp), It needs to be fixed similar to ruby-odbc's: https://github.com/vhermecz/ruby-odbc/commit/c38d182863b907f9ce55d2dfeb9bd8… Can we get the ruby-shadow-ruby32 patched to resolve this issue? Thanks.

2 1

[ruby-core:116104] [Ruby master Bug#20167] Code execution isn't recorded in Ractor
by shia (Sangyong Sim) 09 Jan '24

09 Jan '24

Issue #20167 has been reported by shia (Sangyong Sim). ---------------------------------------- Bug #20167: Code execution isn't recorded in Ractor https://bugs.ruby-lang.org/issues/20167 * Author: shia (Sangyong Sim) * Status: Open * Priority: Normal * ruby -v: 3.3.0 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## reproduction code ```ruby # frozen-string-literal: true require "coverage" Coverage.start require_relative "./some_lib" # # some_lib.rb # class C # def hoge(i) # i # end # end r = Ractor.new do loop do v = Ractor.receive ret_v = C.new.hoge(v) Ractor.yield ret_v end end 2.times do |i| r << i r.take end Coverage.result.each do |file, lines| if file.include?("some_ractor_lib") if lines == [1, 1, 2, nil, nil] puts "OK" else puts "expected: [1, 1, 2, nil, nil]" puts "actual: #{lines.inspect}" end end end ``` ## Target ruby version 3.2.0+ affected -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116105] [Ruby master Feature#16137] Add === to UnboundMethod
by okuramasafumi (Masafumi OKURA) 09 Jan '24

09 Jan '24

Issue #16137 has been updated by okuramasafumi (Masafumi OKURA). Now that we have had pattern match for a while, I agree with Eregon, we can utilize it more rather than adding new methods. I cannot change its status so I hope someone will do so instead of me. ---------------------------------------- Feature #16137: Add === to UnboundMethod https://bugs.ruby-lang.org/issues/16137#change-106100 * Author: okuramasafumi (Masafumi OKURA) * Status: Open * Priority: Normal ---------------------------------------- # Abstract `UnboundMethod` class should have `=== ` so that it can be used in case statement. # Background `Method` class has `===` method so that we can do something like: ``` ruby require 'prime' case 11 when Prime.method(:prime?) then :prime end ``` However, we cannot do something like this: ```ruby positive = Integer.instance_method(:positive?) case 11 when positive then :positive end ``` # Proposal Add `===` method to `UnboundMethod` class. # Implementation Minimal implementation in Ruby could be: ```ruby class UnboundMethod def ===(other) bind(other).call end end ``` # Summary `===` for `UnboundMethod` can improve readability in case statement. -- https://bugs.ruby-lang.org/

1 0