November 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:119783] [Ruby master Bug#20877] Introduce (public) debug assertion for holding the GVL.
by ioquatix (Samuel Williams) 20 Nov '24

20 Nov '24

Issue #20877 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20877: Introduce (public) debug assertion for holding the GVL. https://bugs.ruby-lang.org/issues/20877 * Author: ioquatix (Samuel Williams) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I found issues with `zlib.c` calling `rb_` functions without holding the GVL: https://bugs.ruby-lang.org/issues/20863 ## Background The GVL must be held before many of Ruby's C functions can be called. However, few functions enforce this. Even thought it may work in many scenarios, it may break unexpectedly (SEGFAULT etc). ## Proposal I think we should (1) add more debug checks within CRuby for detecting this invalid usage and (2) expose a public macro that can be used by Ruby's native extensions so that they may do the same. The current internal implementation looks like this: ```c RUBY_ASSERT(ruby_thread_has_gvl_p()); ``` While fixing `zlib.c`, I made a simple PR to demonstrate the change: https://github.com/ruby/ruby/pull/11975 `ruby_thread_has_gvl_p` is not a public interface at this time, so we can't use it in native extensions. I would like to consider adding a public macro to allow for this, e.g. `RUBY_DEBUG_ASSERT_GVL` or something similar. Finally, I propose to add this macro to all methods that should be executed with the GVL so that we catch invalid usage. If the name `RUBY_DEBUG_ASSERT_GVL` is not suitable / too specific, maybe some other ideas: - `RUBY_DEBUG_ASSERT_THREAD` - `RUBY_DEBUG_ASSERT_CURRENT_THREAD` or `RUBY_DEBUG_ASSERT_CURRENT_THREAD_P` - `RUBY_DEBUG_ASSERT_GVL` or `RUBY_DEBUG_ASSERT_GVL_P` - `RUBY_DEBUG_ASSERT_INTERPRETER_LOCKED` I don't have a strong opinion on naming, but I have a strong opinion about preventing invalid usage. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:113489] [Ruby master Bug#19642] Remove vectored read/write from `io.c`.
by ioquatix (Samuel Williams) 20 Nov '24

20 Nov '24

Issue #19642 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19642: Remove vectored read/write from `io.c`. https://bugs.ruby-lang.org/issues/19642 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- https://github.com/socketry/async/issues/228#issuecomment-1546789910 is a comment from the kernel developer who tells us that `writev` is always worse than `write` system call. A large amount of complexity in `io.c` comes from optional support from `writev`. So, I'd like to remove support for `writev`. I may try to measure the performance before/after. However it may not show much difference, except that the implementation in `io.c` can be much simpler. -- https://bugs.ruby-lang.org/

5 9

[ruby-core:119972] [Ruby master Feature#20902] Allow `IO::Buffer#copy` to release the GVL.
by ioquatix (Samuel Williams) 20 Nov '24

20 Nov '24

Issue #20902 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #20902: Allow `IO::Buffer#copy` to release the GVL. https://bugs.ruby-lang.org/issues/20902 * Author: ioquatix (Samuel Williams) * Status: Open ---------------------------------------- Related to <https://bugs.ruby-lang.org/issues/20876>. ## Background `IO::Buffer#copy` execution time is proportional to the length of the data copied. As such, large copies can take a long time (100ms+). Currently, the GVL is not released, which can stall the Ruby interpreter. ## Proposal Pull Request: https://github.com/ruby/ruby/pull/12021 If the size of the data to be copied is larger than a specific amount (heuristic), we will perform `memmove` using `rb_nogvl`. The initial size heuristic is set to 1MiB. This won't be perfect for every system, but should be good enough to avoid ms+ stalls. ## Results I measured the difference: | GVL | Threads | Buffer Size | Total Duration | Throughput (MB/s) | |-----|---------|-------------|----------------|-------------------| | Yes | 1 | 1 | 0.12ms | 8393.09 | | Yes | 1 | 5 | 0.51ms | 9857.7 | | Yes | 1 | 10 | 1.12ms | 8937.54 | | Yes | 1 | 20 | 2.22ms | 9015.95 | | Yes | 2 | 1 | 0.24ms | 8307.07 | | Yes | 2 | 5 | 1.13ms | 8819.58 | | Yes | 2 | 10 | 1.49ms | 13385.35 | | Yes | 2 | 20 | 5.63ms | 7110.8 | | Yes | 4 | 1 | 0.92ms | 4360.18 | | Yes | 4 | 5 | 2.08ms | 9606.58 | | Yes | 4 | 10 | 4.51ms | 8863.13 | | Yes | 4 | 20 | 9.3ms | 8601.41 | | Yes | 8 | 1 | 1.22ms | 6574.93 | | Yes | 8 | 5 | 3.56ms | 11239.27 | | Yes | 8 | 10 | 7.31ms | 10943.68 | | Yes | 8 | 20 | 15.57ms | 10274.99 | | Yes | 16 | 1 | 1.95ms | 8220.16 | | Yes | 16 | 5 | 5.51ms | 14518.05 | | Yes | 16 | 10 | 13.77ms | 11618.96 | | Yes | 16 | 20 | 27.21ms | 11759.43 | | Yes | 32 | 1 | 3.24ms | 9891.05 | | Yes | 32 | 5 | 11.42ms | 14007.41 | | Yes | 32 | 10 | 21.64ms | 14786.48 | | Yes | 32 | 20 | 45.52ms | 14060.25 | | No | 1 | 1 | 0.13ms | 7582.85 | | No | 1 | 5 | 0.44ms | 11248.55 | | No | 1 | 10 | 1.11ms | 9029.91 | | No | 1 | 20 | 2.43ms | 8228.42 | | No | 2 | 1 | 0.18ms | 11245.61 | | No | 2 | 5 | 0.96ms | 10396.76 | | No | 2 | 10 | 1.9ms | 10501.59 | | No | 2 | 20 | 3.16ms | 12656.77 | | No | 4 | 1 | 0.69ms | 5827.76 | | No | 4 | 5 | 1.15ms | 17440.54 | | No | 4 | 10 | 2.31ms | 17307.79 | | No | 4 | 20 | 4.11ms | 19483.68 | | No | 8 | 1 | 0.67ms | 11954.1 | | No | 8 | 5 | 1.3ms | 30713.68 | | No | 8 | 10 | 2.05ms | 38990.98 | | No | 8 | 20 | 4.15ms | 38552.37 | | No | 16 | 1 | 0.96ms | 16698.03 | | No | 16 | 5 | 1.46ms | 54782.47 | | No | 16 | 10 | 2.74ms | 58295.64 | | No | 16 | 20 | 4.89ms | 65482.43 | | No | 32 | 1 | 1.82ms | 17554.27 | | No | 32 | 5 | 2.68ms | 59673.59 | | No | 32 | 10 | 3.87ms | 82733.34 | | No | 32 | 20 | 6.93ms | 92297.47 | In the base case, the performance is about the same, but in the best case, the throughput is significantly better: 15GiB/s vs 92GiB/s (32 threads copying 20MiB of data). -- https://bugs.ruby-lang.org/

1 1

[ruby-core:119958] [Ruby master Bug#20900] Net.send(:remove_const, :HTTPSession) does not emit warnings
by yahonda (Yasuo Honda) 19 Nov '24

19 Nov '24

Issue #20900 has been reported by yahonda (Yasuo Honda). ---------------------------------------- Bug #20900: Net.send(:remove_const, :HTTPSession) does not emit warnings https://bugs.ruby-lang.org/issues/20900 * Author: yahonda (Yasuo Honda) * Status: Open * ruby -v: ruby 3.3.6 (2024-11-05 revision 75015d4c1f) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- After https://github.com/ruby/net-http/pull/189 has been merged and cherry-picked to Ruby master branch, `Net.send(:remove_const, :HTTPSession)` gets the NameError. This code should emit warnings against Ruby 3.3 or earlier versions of Ruby but actually it does not. ``` $ ruby -v ruby 3.4.0dev (2024-11-18T15:04:50Z :detached: 4a7ac694e5) +PRISM [x86_64-linux] $ irb -w ... snip ... irb(main):001> require 'net/http' => true irb(main):002> Net.send(:remove_const, :HTTPSession) (irb):2:in 'Module#remove_const': constant Net::HTTPSession not defined (NameError) from (irb):2:in '<main>' from <internal:kernel>:168:in 'Kernel#loop' from /home/yahonda/.rbenv/versions/trunk/lib/ruby/gems/3.4.0+1/gems/irb-1.14.1/exe/irb:9:in '<top (required)>' from /home/yahonda/.rbenv/versions/trunk/bin/irb:25:in 'Kernel#load' from /home/yahonda/.rbenv/versions/trunk/bin/irb:25:in '<main>' irb(main):003> ``` ### Steps to reproduce Here are the steps to see if the warnings are emitted against Ruby 3.3. $ ruby -v ruby 3.3.6 (2024-11-05 revision 75015d4c1f) [x86_64-linux] $ irb -w ``` require 'net/http' Net::HTTPSession Net.send(:remove_const, :HTTPSession) ``` ### Expected behavior Both of `Net::HTTPSession` and `Net.send(:remove_const, :HTTPSession)` should emit the `warning: constant Net::HTTPSession is deprecated` . ### Actual behavior Only `Net::HTTPSession` emits the warnings. `Net.send(:remove_const, :HTTPSession)` does not emit any warnings. ``` $ irb -w irb(main):001> require 'net/http' irb(main):002> => true irb(main):003> Net::HTTPSession (irb):3: warning: constant Net::HTTPSession is deprecated => Net::HTTP irb(main):004> Net.send(:remove_const, :HTTPSession) => Net::HTTP irb(main):005> ``` -- https://bugs.ruby-lang.org/

3 5

[ruby-core:119772] [Ruby master Feature#20876] Introduce `Fiber::Scheduler#blocking_operation_wait` to avoid stalling the event loop.
by ioquatix (Samuel Williams) 19 Nov '24

19 Nov '24

Issue #20876 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #20876: Introduce `Fiber::Scheduler#blocking_operation_wait` to avoid stalling the event loop. https://bugs.ruby-lang.org/issues/20876 * Author: ioquatix (Samuel Williams) * Status: Open ---------------------------------------- This is an evolution of the previous proposal: https://bugs.ruby-lang.org/issues/20855 ## Background The current Fiber Scheduler performance can be significantly impacted by blocking operations that cannot be deferred to the event loop, particularly in high-concurrency environments where Fibers rely on non-blocking operations for efficient task execution. ## Proposal Pull Request: https://github.com/ruby/ruby/pull/12016 We will introduce a new fiber scheduler hook called `blocking_operation_work`: ```ruby class MySchduler # ... def blocking_operation_wait(work) # Example implementation: Thread.new(&work).join end end ``` We introduce a new flag for `rb_nogvl`: `RB_NOGVL_BLOCKING_OPERATION` which indicates that `rb_nogvl(func, ...)` is a blocking operation that is safe to execute on a different thread or thread pool. When a C extension invokes `rb_nogvl(..., RB_NOGVL_BLOCKING_OPERATION)`, and a fiber scheduler is available, all the arguments will be saved into a instance of a callable object (at this time a `Proc`) called `work`. When `work` is `#call`ed, it will execute `rb_nogvl` again with all the same arguments. The fiber scheduler can decide how to execute that work, e.g. on a separate thread, to mitigate the performance impact of the blocking operation on the event loop. ![](clipboard-202411070126-fbqpn.png) ## Example Using the branch of `async` gem: https://github.com/socketry/async/pull/352/files and enabling zlib deflate to use this feature, the following performance improvement was achieved: ```ruby require "zlib" require "async" require "benchmark" DATA = Random.new.bytes(1024*1024*100) duration = Benchmark.measure do Async do 10.times do Async do Zlib.deflate(DATA) end end end end # Ruby 3.3.4: ~16 seconds # Ruby 3.4.0 + PR: ~2 seconds. ``` ---Files-------------------------------- clipboard-202411070126-fbqpn.png (135 KB) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:111966] [Ruby master Feature#19366] Rename/alias Refinedment#refined_class => #refined_module
by zverok (Victor Shepelev) 19 Nov '24

19 Nov '24

Issue #19366 has been reported by zverok (Victor Shepelev). ---------------------------------------- Feature #19366: Rename/alias Refinedment#refined_class => #refined_module https://bugs.ruby-lang.org/issues/19366 * Author: zverok (Victor Shepelev) * Status: Open * Priority: Normal ---------------------------------------- In #12737, `Refinement#refined_class` is introduced. As "module" is more generic concept than "class", the name misleadingly implies that either this method doesn't returns refined _modules_, or modules can't be refined. This is obviously not true and trivially disproved: ```ruby module Refs refine Enumerable do def foo = puts 'foo' end end Refs.refinements[0].refined_class #=> Enumerable. Which is, well, not a class. # The refinement is usable, so it is not a mute concept: using Refs [1, 2, 3].foo # prints "foo" successfully ``` I believe we refer to "modules" when some feature applies to modules and classes. Unless there is some deeper consideration for the current naming (I don't see justification in #12737, but I might miss something), the method should be renamed or aliased. PS: Sorry for a huge amount of "nitpicky" issues after the version release. Typically, those concerns emerge while I am working on the [changelog](https://rubyreferences.github.io/rubychanges/), and I am usually trying to start the work at least a month before the release. Unfortunately, due to you-know-what, I was unable to do it in time this year. After the 3.2 changelog, I plan to switch to the "master changelog" approach (following the development with the "current unreleased changes" document) this year. Unless I'll be unable to due to reasons not in my control. -- https://bugs.ruby-lang.org/

4 3

[ruby-core:119960] [Ruby master Feature#20901] What's status of Refinement#refined_class?
by hsbt (Hiroshi SHIBATA) 19 Nov '24

19 Nov '24

Issue #20901 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Feature #20901: What's status of Refinement#refined_class? https://bugs.ruby-lang.org/issues/20901 * Author: hsbt (Hiroshi SHIBATA) * Status: Assigned * Assignee: shugo (Shugo Maeda) ---------------------------------------- I found the following code in `eval.c`. ``` rb_warn_deprecated_to_remove("3.4", "Refinement#refined_class", "Refinement#target"); ``` @shugo Should we remove `Refinement#refined_class` at Ruby 3.4? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119940] [Ruby master Bug#20896] io/nonblock operations not ractor-safe
by chucke (Tiago Cardoso) 16 Nov '24

16 Nov '24

Issue #20896 has been reported by chucke (Tiago Cardoso). ---------------------------------------- Bug #20896: io/nonblock operations not ractor-safe https://bugs.ruby-lang.org/issues/20896 * Author: chucke (Tiago Cardoso) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This prevents from calling `socket.nonblock = true` in a non-main ractor. https://github.com/ruby/io-nonblock/pull/14 -- https://bugs.ruby-lang.org/

2 1

[ruby-core:119462] [Ruby master Bug#20785] Should `a in b, and c` `a in b, or c` `a in b, rescue c` be syntax ok?
by tompng (tomoya ishida) 16 Nov '24

16 Nov '24

Issue #20785 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20785: Should `a in b, and c` `a in b, or c` `a in b, rescue c` be syntax ok? https://bugs.ruby-lang.org/issues/20785 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-10-04T03:22:53Z master 939ec9f080) +YJIT +MN +PRISM [arm64-darwin22] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This code is accepted in parse.y but rejected in prism ~~~ruby tap do a in b, and c a in b, or c a in b, rescue c end # parsed as tap do (a in b,;) and c (a in b,;) or c a in b,; rescue c end ~~~ I think these should be rejected like prism (parse.y accepts) ~~~ruby a in b, and c a in b, and c tap do a in b, rescue c end ~~~ I think these should be accepted like parse.y (prism rejects) ~~~ruby tap do a in b, end tap do a in b, rescue end ~~~ -- https://bugs.ruby-lang.org/

7 9

[ruby-core:119934] [Ruby master Bug#20895] Network related test failures
by vo.x (Vit Ondruch) 14 Nov '24

14 Nov '24

Issue #20895 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #20895: Network related test failures https://bugs.ruby-lang.org/issues/20895 * Author: vo.x (Vit Ondruch) * Status: Open * ruby -v: ruby 3.4.0dev (2024-11-14 master 54ff9d3525) +PRISM [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Testing recent Ruby in Fedora Rawhide, I have started to observe following errors: ~~~ 183) Failure: TestSocket_TCPSocket#test_initialize_resolv_timeout_with_connection_failure [/builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:284]: [Errno::ETIMEDOUT] exception expected, not #<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 41615>. 184) Failure: TestSocket_TCPSocket#test_initialize_with_hostname_resolution_failure_after_connection_failure [/builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:303]: [Socket::ResolutionError] exception expected, not #<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 43017>. 185) Failure: TestNetHTTP_v1_2#test_timeout_during_HTTP_session [/builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/net/http/test_http.rb:625]: [Net::ReadTimeout] exception expected, not #<Errno::ECONNREFUSED: Failed to open TCP connection to localhost:43017 (Connection refused - connect(2) for "localhost" port 43017)>. 186) Error: TestSocket_TCPSocket#test_initialize_v6_hostname_resolution_failed_and_v4_hostname_resolution_is_success: Socket::ResolutionError: getaddrinfo(3): Non-recoverable failure in name resolution /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:266:in 'TCPSocket#initialize' /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:266:in 'IO.new' /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:266:in '<main>' /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:259:in 'TestSocket_TCPSocket#test_initialize_v6_hostname_resolution_failed_and_v4_hostname_resolution_is_success' 187) Error: TestSocket_TCPSocket#test_initialize_v6_hostname_resolved_earlier_and_v6_server_is_not_listening: Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 39421 /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:222:in 'TCPSocket#initialize' /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:222:in 'IO.new' /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:222:in '<main>' /builddir/build/BUILD/ruby-3.4.0_20241114git54ff9d3525-build/ruby-3.4.0-54ff9d3525/test/socket/test_tcp.rb:213:in 'TestSocket_TCPSocket#test_initialize_v6_hostname_resolved_earlier_and_v6_server_is_not_listening' ~~~ Please note that the build system does not have internet access. At least some of the test failures are likely related to #20782 / https://github.com/ruby/ruby/pull/11653 -- https://bugs.ruby-lang.org/

1 1