Issue #19588 has been reported by kyanagi (Kouhei Yanagita).
----------------------------------------
Feature #19588: Allow Comparable#clamp(min, max) to accept nil as a specification
https://bugs.ruby-lang.org/issues/19588
* Author: kyanagi (Kouhei Yanagita)
* Status: Open
* Priority: Normal
----------------------------------------
`Comparable#clamp(min, max)` (with two arguments) accepts `nil`. This behaves the same as beginless/endless Range.
~~~ruby
5.clamp(nil, 0) # => 0
5.clamp(10, nil) # => 10
5.clamp(..0) # => 0
5.clamp(10..) # => 10
~~~
This behavior is not documented. Presumably, this was not introduced intentionally.
The old Rubies did not accept accept a `Range` argument.
In Ruby 2.7, accepting `Range` as an argument was introduced.
At that time, the approach of passing `nil` as a two-argument method was also discussed but not adopted,
and using Range was chosen instead. https://bugs.ruby-lang.org/issues/14784
However, in Ruby 3.0, the behavior of `clamp` has changed to accept `nil`.
This change is not documented in the NEWS or the documentation for `clamp`,
and I believe that it was not an intentional change.
~~~
% docker run -it --rm rubylang/all-ruby env ALL_RUBY_SINCE=ruby-2.4.0 ./all-ruby -e "p 5.clamp(0, nil)"
ruby-2.4.0 -e:1:in `clamp': comparison of Integer with nil failed (ArgumentError)
from -e:1:in `<main>'
exit 1
...
ruby-2.7.8 -e:1:in `clamp': comparison of Integer with nil failed (ArgumentError)
from -e:1:in `<main>'
exit 1
ruby-3.0.0-preview1 5
...
ruby-3.2.2 5
~~~
It seems that https://github.com/ruby/ruby/commit/a93da4970be44a473b7b42e7516eb2663dece2c3 brought about this change.
How about making the current behavior a specification?
It has been three years since the behavior changed, and I don't see much point in prohibiting `nil` now.
--
https://bugs.ruby-lang.org/
Issue #19744 has been reported by tagomoris (Satoshi TAGOMORI).
----------------------------------------
Feature #19744: Namespace on read
https://bugs.ruby-lang.org/issues/19744
* Author: tagomoris (Satoshi TAGOMORI)
* Status: Open
* Priority: Normal
----------------------------------------
# What is the "Namespace on read"
This proposes a new feature to define virtual top-level namespaces in Ruby. Those namespaces can require/load libraries (either .rb or native extension) separately from the global namespace. Dependencies of required/loaded libraries are also required/loaded in the namespace.
### Motivation
The "namespace on read" can solve the 2 problems below, and can make a path to solve another problem:
The details of those motivations are described in the below section ("Motivation details").
#### Avoiding name conflicts between libraries
Applications can require two different libraries safely which use the same module name.
#### Avoiding unexpected globally shared modules/objects
Applications can make an independent/unshared module instance.
#### (In the future) Multiple versions of gems can be required
Application developers will have fewer version conflicts between gem dependencies if rubygems/bundler will support the namespace on read.
### Example code with this feature
```ruby
# your_module
module YourModule
end
# my_module.rb
require 'your_module'
module MyModule
end
# example.rb
namespace1 = NameSpace.new
namespace1.require('my_module') #=> true
namespace1::MyModule #=> #<Module:0x00000001027ea650>::MyModule (or #<NameSpace:0x00...>::MyModule ?)
namespace1::YourModule # similar to the above
MyModule # NameError
YourModule # NameError
namespace2 = NameSpace.new # Any number of namespaces can be defined
namespace2.require('my_module') # Different library "instance" from namespace1
require 'my_module' # require in the global namespace
MyModule.object_id != namespace1::MyModule.object_id #=> true
namespace1::MyModule.object_id != namespace2::MyModule.object_id
```
The required/loaded libraries will define different "instances" of modules/classes in those namespaces (just like the "wrapper" 2nd argument of `Kernel.load`). This doesn't introduce compatibility problems if all libraries use relative name resolution (without forced top-level reference like `::Name`).
# "On read": optional, user-driven feature
"On read" is a key thing of this feature. That means:
* No changes are required in existing/new libraries (except for limited cases, described below)
* No changes are required in applications if it doesn't need namespaces
* Users can enable/use namespaces just for limited code in the whole library/application
Users can start using this feature step by step (if they want it) without any big jumps.
## Motivation details
This feature can solve multiple problems I have in writing/executing Ruby code. Those are from the 3 problems I mentioned above: name conflicts, globally shared modules, and library version conflicts between dependencies. I'll describe 4 scenarios about those problems.
### Running multiple applications on a Ruby process
Modern computers have many CPU cores and large memory spaces. We sometimes want to have many separate applications (either micro-service architecture or modular monolith). Currently, running those applications require different processes. It requires additional computation costs (especially in developing those applications).
If we have isolated namespaces and can load applications in those namespaces, we'll be able to run apps on a process, with less overhead.
(I want to run many AWS Lambda applications on a process in isolated namespaces.)
### Running tests in isolated namespaces
Tests that require external libraries need many hacks to:
* require a library multiple times
* require many different 3rd party libraries into isolated spaces (those may conflict with each other)
Software with plugin systems (for example, Fluentd) will get benefit from namespaces.
In addition to it, application tests can avoid unexpected side effects if tests are executed in isolated namespaces.
### Safely isolated library instances
Libraries may have globally shared states. For example, [Oj](https://github.com/ohler55/oj) has a global `Obj.default_options` object to change the library behavior. Those options may be changed by any dependency libraries or applications, and it changes the behavior of `Oj` globally, unexpectedly.
For such libraries, we'll be able to instantiate a safe library instance in an isolated namespace.
### Avoiding dependency hells
Modern applications use many libraries, and those libraries require much more dependencies. Those dependencies will cause version conflicts very often. In such cases, application developers should resolve those by updating each libraries, or should just wait for the new release of libraries to conflict those libraries. Sometimes, library maintainers don't release updated versions, and application developers can do nothing.
If namespaces can require/load a library multiple times, it also enables to require/load different versions of a library in a process. It requires the support of rubygems, but namespaces should be a good fundamental of it.
## Expected problems
### Use of top-level references
In my expectation, `::Name` should refer the top-level `Name` in the global namespace. I expect that `::ENV` should contain the environment variables. But it may cause compatibility problems if library code uses `::MyLibrary` to refer themselves in their deeply nested library code.
### Additional memory consumption
An extension library (dynamically linked library) may be loaded multiple times (by `dlopen` for temporarily copied dll files) to load isolated library "instances" if different namespaces require the same extension library. That consumes additional memory.
In my opinion, additional memory consumption is a minimum cost to realize loading extension libraries multiple times without compatibility issues.
This occurs only when programmers use namespaces. And it's only about libraries that are used in 2 or more namespaces.
### The change of `dlopen` flag about extension libraries
To load an extension library multiple times without conflicting symbols, all extensions should stop sharing symbols globally. Libraries referring symbols from other extension libraries will have to change code & dependencies.
(About the things about extension libraries, [Naruse also wrote an entry](https://naruse.hateblo.jp/entry/2023/05/22/193411).)
# Misc
The proof-of-concept branch is here: https://github.com/tagomoris/ruby/pull/1
It's still work-in-progress branch, especially for extension libraries.
--
https://bugs.ruby-lang.org/
Issue #19160 has been reported by kaiquekandykoga (Kaíque Koga).
----------------------------------------
Bug #19160: cmp_clamp arguments
https://bugs.ruby-lang.org/issues/19160
* Author: kaiquekandykoga (Kaíque Koga)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-freebsd13.1]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
If clamp receives min higher than max, it will raise an exception. The message says *min argument must be smaller than max argument* , but min can actually be equal to max too.
Patch https://github.com/ruby/ruby/pull/6802.
--
https://bugs.ruby-lang.org/
Issue #19576 has been reported by jprokop (Jarek Prokop).
----------------------------------------
Bug #19576: Backport request: Gemfile.lock resolving is broken with bundler shipped with Ruby 3.1.4
https://bugs.ruby-lang.org/issues/19576
* Author: jprokop (Jarek Prokop)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
This is a backport request for bundler, that regressed in 2.3.36 in a specific situation. Newer and older bundler versions that ship with Ruby are not problematic, only the version that ships with Ruby version >= 3.1.3.
A few weeks ago, we discovered a bug in resolving in bundler shipped with Ruby 3.1.3 and 3.1.4:
Bundler version:
```
$ bundler --version
Bundler version 2.3.26
```
Affected rubies `ruby -v`:
First:
```
$ ruby -v
ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
```
Second:
```
$ruby -v
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
```
Initial bug report with reproducer and more in-depth description can be found here: https://github.com/sclorg/s2i-ruby-container/issues/469
Using the following Gemfile for a rails app:
https://github.com/sclorg/rails-ex/blob/67b7a61eae9efa1088ff3f634ae316e1022…
bundler locks up in trying to resolve Nokogiri for Ruby 3.1, but keeps failing because it keeps using incompatible built binary gem instead of falling back to installing and building the binary extension of Nokogiri locally.
We craft this Gemfile to be usable from Ruby 2.5 up to Ruby 3.1, as the app is used mainly for testing.
I have created a patch to fix the situation, see the attached files. There are 2 of them, one contains the fix and the other one contains the test from the rubygems repo PR#6225.
full commit available here: https://src.fedoraproject.org/fork/jackorp/rpms/ruby/c/5ef600a8f40b76de5636…
The patches are created from the following upstream changes in bundler:
https://github.com/rubygems/rubygems/pull/6225
and adapted:
https://github.com/rubygems/rubygems/commit/7b64c64262a7a980c0eb23b96ea56cf…
for the PR#6225.
With the fix applied I no longer have issues doing `bundle install` with our Gemfile.lock.
---Files--------------------------------
rubygem-bundler-2.3.26-Tests-from-bundler-PR-6225.patch (1.82 KB)
rubygem-bundler-2.3.26-Provide-fix-for-bundler-Gemfile-resolving-regression.patch (5.21 KB)
--
https://bugs.ruby-lang.org/
Issue #19842 has been reported by ko1 (Koichi Sasada).
----------------------------------------
Feature #19842: Intorduce M:N threads
https://bugs.ruby-lang.org/issues/19842
* Author: ko1 (Koichi Sasada)
* Status: Open
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
----------------------------------------
This ticket proposes to introduce M:N threads to improve Threads/Ractors performance.
## Background
Ruby threads (RT in short) are implemented from old Ruby versions and they have the following features:
* Can be created with simple notation `Thread.new{}`
* Can be switched to another ready Ruby thread by:
* Time-slice.
* I/O blocking.
* Synchronization such as Mutex features.
* And other blocking reasons.
* Can be interruptible by:
* OS-deliver signals (only for the main thread).
* `Thread#kill`.
* Can be terminated by:
* the end of each Ruby thread.
* the end of the main thread (and other Ruby threads are killed).
Ruby 1.8 and erlier versions uses M:1 threads (green threads, user level threads, .... the word 1:N threads is more popular but to make this explanation consistent I use "M:1" term here) which manages multiple Ruby threads on 1 native thread.
(Native threads are provided by C interfaces such as Pthreads. In many cases, native threads are OS threads, but there are also user-level implementations, such as user-level pthread libraries in theory. Therefore, they are referred to as native threads in this article and NT in short)
If a Ruby thread T1 blocked because of a I/O operation, Ruby interpreter switches to the next ready Ruby thread T2. The I/O operation will be monitors by a `select()` (or similar) functionality and if the I/O is ready, T1 is marked as a ready thread and T1 will be resumed soon. However, when a Ruby thread issues some other blocking operations such as `gethostbyname()`, Ruby interpreter can not swtich to any other Ruby thread while `gethostbyname()` is not finished.
We named two types blocking operations:
* Managed blocking operations
* I/O (most of read/write)
* manage by I/O multiplexing API (select, poll, epoll, kqueue, IOCP, io_uring, ...)
* Sleeping
* Synchronization (Mutex, Queue, ...)
* Unmanaged operations
* All other blocking operations not listed above, written in C
* Huge number calculation like `Bignum#*`
* DNS lookup
* I/O (can not detect block-able or not by multiplexing API)
* open on FIFO, close on NFS, ...
* flock and other locking mechanism
* library call which uses blocking operations
* `libfoo` has `foo_func()` and `foo_func()` waits DNS lookup. A Ruby extension `foo-ruby` can call `foo_func()`.
With these terms we can say that M:1 threads can suport managed blocking operations but can not support unmanaged operations (can not make progress other Ruby threads) without further tricks.
Note that if the `select()`-like system calls say a `fd` is ready, but the I/O opeartion for `fd` can be blocked because of some contention (read by another thread or process, for example).
M:1 threads has another disadvantage that it can not run in parallel because only a native thread is used.
From Ruby 1.9 we had implemented 1:1 thread which means a Ruby thread has a corresponding native thread. To make implementation easy we also introduced a GVL. Only a Ruby thread acquires GVL can run. With 1:1 model, we can support managed blocking oprations and unmanaged blocking operations by releasing GVL. When a Ruby thread want to issue a blocking operation, the Ruby thread releases GVL and another ready Ruby threads continue to run. We don't care the blocking operation is managed or unmanaged.
(We can not make some of unmanaged blocking operations interruptible (stop by Ctrl-C for example)).
Advantages of 1:1 threads to the M:1 threads is:
* Easy to handle blocking operations by releasing GVL.
* We can utilize parallelism with multiple native threads by releasing GVL.
Disadvantages of 1:1 threads to the M:1 threads is:
* Overhead to make many native threads for many Ruby threads
* We can not make huge number of Ruby threads and Ractors on 1:1 threads.
* Thread switching overhead by GVL because inter-core communication is needed.
From Ruby 3.0 we introduced fiber scheduler mechanism to maintain multiple fibers
Differences between Ruby 1.8 M:1 threads are:
* No timeslice (only switch fibers by managed blocking operations)
* Ruby users can make own schedulers for apps with favorite underlying mechanism
Disadvantages are similar to M:1 threads. Another disadvantages is we need to consider about Fiber's behavior.
From Ruby 3.0 we also introduced Ractors. Ractors can run in parallel because of separating most of objects. 1 Ractor creates 1 Ruby thread, so Ractors has same disadvantages of 1:1 threads. For example, we can not make huge number of Ractors.
## Goal
Our goal is making lightweight Ractors on lightweight Ruby threads. To enable this goal we propose to implement M:N threads on MRI.
M:N threads manages M Ruby threads on N native threads, with limited N (~= CPU core numbers for example).
Advantages of M:N threads are:
1. We can run N ractors on N native threads simultaneously if the machine has N cores.
2. We can make huge number of Ruby threads or Ractors because we don't need huge number of native threads
3. We can support unmanaged blocking operations by locking a native thread to a Ruby thread which issues an unmanaged blocking operation.
4. We can make our own Ruby threads or Ractors scheduler instead of the native thread (OS) scheduler.
Disadvantages of M:N threads are:
1. It is complex implmentation and it can be hard.
2. It can introduce incompatibility especaially on TLS (Thread local storage).
3. We need to maitain our own scheduler.
Without using multiple Ractors, it is similar to Ruby 1.8 M:1 threads. The difference with M:1 threads are locking NT mechanism to support unmanaged blocking operations. Another advantage is that it is easy to fallback to 1:1 threads by locking all of corresponding native threads to Ruby threads.
## Proposed design
### User facing changes
If a program only has a main Ractor (i.e., most Ruby programs), the user will not face any changes by default.
On main Ractor, all threads are 1:1 threads by default and there is no compatibility issue.
`RUBY_MN_THREADS=1` envrionment variable is given, main Ractor enables M:N threads.
Note that the main thread locks NT by default because the initial NT is special in some case. I'm not sure we can relax this limitation.
On the multiple Ractors, N (+ alpha) native threads run M ractors. Now there is no way to disable M:N threads on multiple Ractors because there are only a few multi-Ractor programs and no compatibility issues.
Maximum number of N can be specified by `RUBY_MAX_PROC=N`. 8 by default but this value should be specified with the number of CPU processors (cores).
### TLS issue
On M:N threads a Ruby thread (RT1) migrates from a native thread (NT1) to NT2, ... so that TLS on native code can be a problem.
For example, RT1 calls a library function `foo()` and it set TLS1 on NT1. After migrating RT1 to NT2, RT1 calls `foo()` again but there is no TLS1 record because TLS1 is recorded only on NT1.
On this case, RT1 should be run on NT1 while using native library foo. To avoid such prbolem, we need the following features:
* 1:1 threads on main Ractor by default
* functionality to lock the NT for RT, maybe `Thread#lock_native_thread` and `Thread#unlock_native_thread` API is needed. For example, Go language has `runtime.LockOSThread()` and `runtime.UnlockOSThread()` for this purpose.
* Or C-API only for this purpose? (not fixed yet)
Thankfully, the same problem can occur with Fiber scheduler (and of course Ruby 1.8 M:1 threads), but I have not heard of it being much of a problem, so I expect that TLS will not be much of an issue.
### Unmanaged blocking operations
From Ruby 1.9 (1:1 threads), the `nogvl(func)` API is used for most blocking operations to keep the threading system healthy. In other words, `nogvl(func)` represents that the given function is blocking operation. To support unmanaged blocking operations, we lock a native thread for the Ruby thread which issues blocking operation.
If the blocking operations doesn't finish soon, other Ruby threads can not run because a RT locks NT. In this case, another system monitoring thread named "Timer thread" (historical name and TT in short) creates another NT to run ready other Ruby threads.
This TT's behavior is the same as the behavior of "sysmon" in the Go language.
We named locked NT as dedicated native threads (DNT) and other NT as shared native threads (SNT). The upper bound by `RUBY_MAX_PROC` affects the number of SNT. In other words, the number of DNT is not limited (it is same that the number of NT on 1:1 threads are not limited).
### Managed blocking operations
Managed blocking operations are multiplexing by `select()`-like functions on the Timer thread.. Now only `epoll()` is supported.
I/O operation flow (read on fd1) on Ruby thread RT1:
1. check the ready-ness of fd1 by `poll(timeout = 0)`, goto step 4.
2. register fd1 to Timer thread (TT) epoll and resume another ready Ruby thread.
3. If TT detects that the fd1 is ready, make RT1 as ready thread.
4. When RT1 is resumed, then do `read()` by locking corresponding NT1.
`sleep(n)` operation flow on Ruby thread RT1:
1. register timeout of RT1 to TT epoll.
2. If TT detects the timeout of RT1 (n seconds), TT makes RT1 as a ready Ruby thread.
### Internal design
* 2 level scheduling
* Ruby threads of a Ractor is managed by 1:N threads
* Ruby threads of different Ractors are managed by M:N threads
* Timer thread has several duties
1. Monitoring I/O (or other event) ready-ness
2. Monitoring timeout
3. Produce timeslice signals
4. Help OS signal delivering
(On pthread environment) recent Ruby doesn't make timer thread but MaNy implementation makes TT anytime. it can be improved.
## Implementation
The code name is MaNy project, it is from MN threads.
https://github.com/ko1/ruby/tree/many2
The implementation is not matured (debugging now).
## Measurements
See RubyKaigi 2023 slides: https://atdot.net/~ko1/activities/2023_rubykaigi2023.pdf
## Discussion
* Enable/disable
* default behavior
* how to switch the behavior
* Should we lock the NT for main thread anytime?
* Ruby/C API to lock the native threads
## Misc
This description will be improved more later.
--
https://bugs.ruby-lang.org/
Issue #19754 has been reported by nobu (Nobuyoshi Nakada).
----------------------------------------
Bug #19754: `IO::Buffer#get_string` raises unsuitable exception for too large offset
https://bugs.ruby-lang.org/issues/19754
* Author: nobu (Nobuyoshi Nakada)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
The message seems suitable when both of `offset` and `length` are given.
```ruby
IO::Buffer.for("test").get_string(5, 0) #=> Specified offset+length exceeds buffer size! (ArgumentError)
```
But without `length`, it doesn't seem to check `offset` properly.
```ruby
IO::Buffer.for("test").get_string(5) #=> negative string size (or size too big) (ArgumentError)
```
--
https://bugs.ruby-lang.org/
Issue #19362 has been reported by zverok (Victor Shepelev).
----------------------------------------
Bug #19362: #dup on Proc doesn't call initialize_dup
https://bugs.ruby-lang.org/issues/19362
* Author: zverok (Victor Shepelev)
* Status: Open
* Priority: Normal
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
In #17545, `#dup` had changed to create an instance of the subclass.
It, though, doesn't invoke `initialize_dup` of the subclass, unlike other standard classes.
```ruby
class MyAry < Array
def initialize_dup(...)
p(self.class, ...)
super
end
end
class MyString < String
def initialize_dup(...)
p(self.class, ...)
super
end
end
class MyProc < Proc
def initialize_dup(...)
p(self.class, ...)
super
end
end
MyString.new('test').dup # prints MyString, "test"
MyAry.new(['test']).dup # prints MyAry, ["test"]
MyProc.new { 'test' }.dup # doesn't print anything
```
This makes the change in #17545 useless: while inheriting from core classes is indeed marginal, one of author's intention might be carrying additional information with the Proc instance, and bypassing `#initialize_dup` makes it impossible to maintain this information.
It seems that actually `#initialize_dup` is also invoked on the core classes themselves, but ignored on `Proc`.
```ruby
class Array
def initialize_dup(...)
p(self.class, ...)
super
end
end
class String
def initialize_dup(...)
p(self.class, ...)
super
end
end
class Proc
def initialize_dup(...)
p(self.class, ...)
super
end
end
'test'.dup # prints String, "test"
['test'].dup # prints Array, ["test"]
Proc.new { 'test' }.dup # doesn't print anything
```
Which is an even more marginal problem but still an inconsistency.
--
https://bugs.ruby-lang.org/
Issue #19541 has been reported by kjtsanaktsidis (KJ Tsanaktsidis).
----------------------------------------
Feature #19541: Proposal: Generate frame unwinding info for YJIT code
https://bugs.ruby-lang.org/issues/19541
* Author: kjtsanaktsidis (KJ Tsanaktsidis)
* Status: Open
* Priority: Normal
----------------------------------------
## What is being propsed?
Currently, Ruby crashes with yjit generated code on the stack, `rb_print_backtrace()` is unable to actually show any frames underneath the yjit code. For example, if you send SIGSEGV to a Ruby process running yjit, this is what you see:
```
/ruby/miniruby(rb_print_backtrace+0xc) [0xaaaad0276884] /ruby/vm_dump.c:785
/ruby/miniruby(rb_vm_bugreport) /ruby/vm_dump.c:1093
/ruby/miniruby(rb_bug_for_fatal_signal+0xd0) [0xaaaad0075580] /ruby/error.c:813
/ruby/miniruby(sigsegv+0x5c) [0xaaaad01bedac] /ruby/signal.c:919
linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffff91a3e8bc]
/ruby/miniruby(map<(usize, yjit::backend::ir::Insn), (usize, yjit::backend::ir::Insn), yjit::backend::ir::{impl#17}::next_mapped::{closure_env#0}>+0x8c) [0xaaaad03b8b00] /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/option.rs:929
/ruby/miniruby(next_mapped+0x3c) [0xaaaad0291dc0] src/backend/ir.rs:1225
/ruby/miniruby(arm64_split+0x114) [0xaaaad0287744] src/backend/arm64/mod.rs:359
/ruby/miniruby(compile_with_regs+0x80) [0xaaaad028bf84] src/backend/arm64/mod.rs:1106
/ruby/miniruby(compile+0xc4) [0xaaaad0291ae0] src/backend/ir.rs:1158
/ruby/miniruby(gen_single_block+0xe44) [0xaaaad02b1f88] src/codegen.rs:854
/ruby/miniruby(gen_block_series_body+0x9c) [0xaaaad03b0250] src/core.rs:1698
/ruby/miniruby(gen_block_series+0x50) [0xaaaad03b0100] src/core.rs:1676
/ruby/miniruby(branch_stub_hit_body+0x80c) [0xaaaad03b1f68] src/core.rs:2021
/ruby/miniruby({closure#0}+0x28) [0xaaaad02eb86c] src/core.rs:1924
/ruby/miniruby(do_call<yjit::core::branch_stub_hit::{closure_env#0}, *const u8>+0x98) [0xaaaad035ba3c] /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492
[0xaaaad035c9b4]
```
(n.b. - I compiled Ruby with `-fasynchronous-unwind-tables –rdynamic –g` in cflags to make sure gcc generates appropriate unwind info & keeps the symbol tables).
Likewise, if you attach gdb to a Ruby process with yjit enabled, gdb can't show thread backtraces through yjit-generated code either.
My proposal is that YJIT generate sufficient unwinding and debug information on all platforms to allow both `rb_print_backtrace()` and the platform's debugger (gdb/lldb/WinDbg) to show:
* Full stack traces all the way back to `main`. That is, it should be possible to see frames _underneath_ `[0xaaaad035c9b4]` from the backtrace above.
* Names for the dynamically generated yjit blocks (e.g. instead of `[0xaaaad035c9b4]`, we should see something like `yjit$$name_of_ruby_method`, where `name_of_ruby_method` is the `label` for the iseq this is JIT'd code for).
## Motivation
I have a few motivations for wanting this. Firstly, I feel this functionality is independently useful. When Ruby crashes, the more information we can get, the more likely we are to find the root cause. Likewise, the same principle applies to debugging with gdb - you can get a fuller understanding of what the process is doing if you see the whole stack.
I have often found attaching gdb to the Ruby interpreter helps in understanding problems in Ruby code or C extensions and is something I do relatively frequently; yjit breaking that will definitely be inconvenient for me!
## Implementation
I have a draft implementation here on how I'd implement this: . It's currently missing tests & platform support (it only works on Linux aarch64). Also, it implements unwind info generation, so unwinding can work _through_ yjit code, but it does not currently emit symbols to give _names_ to those yjit frames.
My PR contains a document which explains how the Linux interfaces for registering unwind info for JIT'd code work, so I won't duplicate that information here.
The biggest implementation question I had is around the use of Rust crates. Currently, I prototyped my implementation using the gimli & object crates, for generating DWARF info and ELF binaries. However, the yjit build does purposefully does not use cargo & external crates for release builds. There are a few different ways we could go here:
* Don't use the gimli & object crates; instead, re-implement all debug info & object file generation code in yjit.
* Don't use the crates; instead, link againt C libraries to provide this functionality & call them from Rust (perhaps some combination of libelf, libdw, libbfd, or llvm might do what we need)
* Use cargo after all for the release build & download the crates at build-time
* Use cargo for the release build, but vendor everything, so the build doesn't need to download anything
* Only make unwind info generation available in dev mode where cargo is used, and so mark the gimli/object dependencies as optional in Cargo.toml.
We'd need to decide on one of these approaches for this proposal to work. I don't really have a strong sense of the pros/cons of each.
(Side note - my PR actually depends on a _fork_ of gimli - I've been discussing adding the needed interfaces upstream here: https://github.com/gimli-rs/gimli/issues/648).
## Benchmarks
I ran the yit-bench suite on my branch and compared it to Ruby master:
* My branch: https://gist.github.com/KJTsanaktsidis/5741a9f64e5cd75cdf5fedd846091a4f
* Ruby master: https://gist.github.com/KJTsanaktsidis/592d3ebcf98f6745dfa3efbd30a25acf
This is a (very simple) comparison:
```
-------------- ------------ ------------ ---------------
bench yjit (ms) branch (ms) branch/yjit (%)
activerecord 97.5 98.5 101.03%
hexapdf 2415.3 2458.2 101.78%
liquid-c 61.9 63.1 101.94%
liquid-render 135.3 135.0 99.78%
mail 104.6 105.5 100.86%
psych-load 1887.1 1922.0 101.85%
railsbench 1544.4 1556.0 100.75%
ruby-lsp 88.4 89.5 101.24%
sequel 147.5 151.1 102.44%
binarytrees 303 305.6 100.86%
chunky_png 1075.8 1079.4 100.33%
erubi 392.9 392.3 99.85%
erubi_rails 14.7 14.7 100.00%
etanni 792.3 791.4 99.89%
fannkuchredux 3815.9 3813.6 99.94%
lee 1030.2 1039.2 100.87%
nbody 49.2 49.3 100.20%
optcarrot 4142 4143.3 100.03%
ruby-json 2860.7 2874.0 100.46%
rubykon 7906.6 7904.2 99.97%
30k_ifelse 348.7 345.4 99.05%
30k_methods 828.6 831.8 100.39%
cfunc_itself 28.8 28.9 100.35%
fib 34.4 34.5 100.29%
getivar 115.5 109.7 94.98%
keyword_args 37.7 38.0 100.80%
respond_to 26 26.1 100.38%
setivar 33.8 33.5 99.11%
setivar_object 208.7 194.3 93.10%
str_concat 52.6 52.2 99.24%
throw 23.8 24.1 101.26%
-------------- ------------ ------------ ---------------
```
It seems like the performance impact of generating and registering the debug info is marginal.
--
https://bugs.ruby-lang.org/
Issue #19430 has been reported by mame (Yusuke Endoh).
----------------------------------------
Feature #19430: Contribution wanted: DNS lookup by c-ares library
https://bugs.ruby-lang.org/issues/19430
* Author: mame (Yusuke Endoh)
* Status: Open
* Priority: Normal
----------------------------------------
## Problem
At the present time, Ruby uses `getaddrinfo(3)` to resolve names. Because this function is synchronous, we cannot interrupt the thread performing name resolution until the DNS server returns a response.
We can see this behavior by setting blackhole.webpagetest.org (72.66.115.13) as a DNS server, which swallows all packets, and resolving any name:
```
# cat /etc/resolv.conf
nameserver 72.66.115.13
# ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)'
^C^C^C^C
```
As we see, Ctrl+C does not stop ruby.
The current workaround that users can take is to do name resolution in a Ruby thread.
```ruby
Thread.new { Addrinfo.getaddrinfo("www.ruby-lang.org", 80) }.value
```
The thread that calls this code is interruptible. (Note that the newly created thread itself will be stuck until the DNS lookup exceeds the time out.)
## Proposal
We can solve this problem by using c-ares, which is an asynchronous name resolver, as a backend of `Addrinfo.getaddrinfo`, etc. (@sorah told me about this library, thanks!)
https://c-ares.org/
I have created a PoC patch.
https://github.com/mame/ruby/commit/547806146993bbc25984011d423dcc0f913b211c
By applying this patch, we can interrupt `Addrinfo.getaddrinfo` by Ctrl+C.
```
# cat /etc/resolv.conf
nameserver 72.66.115.13
# ./local/bin/ruby -rsocket -e 'Addrinfo.getaddrinfo("www.ruby-lang.org", 80)'
^C-e:1:in `getaddrinfo': Interrupt
from -e:1:in `<main>'
```
## Discussion
### About c-ares
According to the site of c-ares, some major tools including libcurl, Wireshark, and Apache Arrow are already using c-ares. In the language interpreter, node.js seems to be using c-ares.
I am honestly not sure about the compatibility of c-ares with `getaddrinfo(3)`. I guess there is no major incompatibility because I have not experienced any name resolution problem of curl. @akr (who is the author and maintainer of Ruby's socket library) suggested to check if OS-specific name resolution, e.g., WINS on Windows, NIS on Solaris, etc., is supported. He also said that it may be acceptable even if they are not supported.
Whether to bundle c-ares source code with ruby would require further discussion. If this proposal is accepted, then c-ares will become a de facto essential dependency for practical use, like gmp, in my opinion. Incidentally, node.js bundles c-ares: https://github.com/nodejs/node/tree/main/deps/cares
### Alternative approaches
Recent glibc provides `getaddrinfo_a(3)` which performs asynchronous name resolution. However, this function has a fatal problem of being incompatible with `fork(2)`, which is heavily used in the Ruby ecosystem. In fact, the attempt to use `getaddrinfo_a(3)` (#17134) has been revert because it fails rails tests. (#17220)
Another alternative is to have a worker pthread inside Ruby that calls getaddrinfo(3). Instead of calling getaddrinfo(3) directly, `Addrinfo.getaddrinfo` would ask the worker to resolve a name and wait for a response. This method should be able to implement cancellation. (Simply put, this means reimplementation of getaddrinfo_a(3) on our own, taking into account of `fork(2).)
This has the advantages: not adding dependencies on external libraries and not having compatibility issues with `getaddrinfo(3)`. However, it is considerably more difficult to implement and maintain. An internal pthread may have a non-trivial impact on the execution efficiency and memory usage. Also, we may need to implement a mechanism to dynamically change the number of workers depending on the load.
It would be ideal if we could try and evaluate both approaches. But my current impression is that using c-ares is the quickest and best compromise.
## Contribution wanted
I have made it up to the PoC, but don't have much time to complete this. @naruse suggested me to create a ticket asking for contributions. Is anyone interested in this?
* This patch changes `rsock_getaddrinfo` to accept a timeout argument. There are several places where Qnil is passed as a timeout (where I add `// TODO` in the PoC). We need to consider what timeout we should pass.
* This cares only `getaddrinfo`, but we also need to care `getnameinfo` (and something else if any). There may be some issues I'm not aware of.
* I have not yet tested this PoC seriously. It would be great if we could evaluate it with some real apps.
Also, it would be great to hear from someone who knows more about c-ares.
--
https://bugs.ruby-lang.org/