- ruby-core - ml.ruby-lang.org

[ruby-core:116894] [Ruby master Bug#20288] `rb_fiber_scheduler_close` exceptions are not handled in `rb_fiber_scheduler_set`.
by ioquatix (Samuel Williams) 06 Jul '24

06 Jul '24

Issue #20288 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20288: `rb_fiber_scheduler_close` exceptions are not handled in `rb_fiber_scheduler_set`. https://bugs.ruby-lang.org/issues/20288 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The code responsible for setting a new scheduler on a thread does not properly handle exceptions raised by `rb_fiber_scheduler_close`. If `rb_fiber_scheduler_close` raised an exception, the assignment `thread->scheduler = scheduler` would not be executed. This leaves the thread in an undefined state because it might still hold a reference to the old scheduler, which was supposed to be closed and replaced. ## Steps to Reproduce: 1. Define a custom fiber scheduler that raises an exception in its close method. 2. Set the custom scheduler on a thread. 3. Attempt to replace the custom scheduler with a different scheduler. 4. Observe that if the close method of the original scheduler raises an exception, the thread's scheduler reference is not updated. ## Proposed Fix: The use of `rb_ensure` can be introduced to wrap the call to `rb_fiber_scheduler_close` to ensure that, regardless of whether an exception is raised, `thread->scheduler` is set to `Qnil`, and then `thread->scheduler = scheduler` is safely executed to update the thread's scheduler reference. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:117321] [Ruby master Bug#20393] `after_fork_ruby` clears all pending interrupts for both parent and child process.
by ioquatix (Samuel Williams) 06 Jul '24

06 Jul '24

Issue #20393 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20393: `after_fork_ruby` clears all pending interrupts for both parent and child process. https://bugs.ruby-lang.org/issues/20393 * Author: ioquatix (Samuel Williams) * Status: Open * Assignee: ioquatix (Samuel Williams) * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- In the following program, the behaviour of the parent process is affected by whether `Process.fork` is invoked or not. ```ruby Thread.handle_interrupt(RuntimeError => :never) do Thread.current.raise(RuntimeError, "Queued error") puts "Pending interrupt: #{Thread.pending_interrupt?}" # true pid = Process.fork do puts "Pending interrupt (child process): #{Thread.pending_interrupt?}" Thread.handle_interrupt(RuntimeError => :immediate){} end _, status = Process.waitpid2(pid) puts "Child process status: #{status.inspect}" puts "Pending interrupt: #{Thread.pending_interrupt?}" # false end puts "Exiting..." ``` I don't think the parent process pending interrupts should be cleared by `after_fork_ruby`: ```c static void after_fork_ruby(rb_pid_t pid) { rb_threadptr_pending_interrupt_clear(GET_THREAD()); if (pid == 0) { // child clear_pid_cache(); rb_thread_atfork(); } else { // parent after_exec(); } } ``` How about this implementation: ```c static void after_fork_ruby(rb_pid_t pid) { if (pid == 0) { // child rb_threadptr_pending_interrupt_clear(GET_THREAD()); clear_pid_cache(); rb_thread_atfork(); } else { // parent after_exec(); } } ``` cc @ko1 -- https://bugs.ruby-lang.org/

3 5

[ruby-core:116890] [Ruby master Bug#20286] TracePoint does not emit `thread_end` event when thread exits with exception
by ioquatix (Samuel Williams) 06 Jul '24

06 Jul '24

Issue #20286 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20286: TracePoint does not emit `thread_end` event when thread exits with exception https://bugs.ruby-lang.org/issues/20286 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Using TracePoint to trace `thread_begin` and `thread_end` events fails to emit the `thread_end` event when an exception (e.g., Interrupt) is raised within a thread. This behavior occurs because the exception handling bypasses the internal thread finishing logic, including trace point and fiber scheduler cleanup code. This issue affects the ability to accurately monitor thread lifecycle events in scenarios involving exception handling or abrupt thread terminations. ## Steps to Reproduce: 1. Set up `TracePoint` to trace `thread_begin` and `thread_end` events. 2. Create a new thread that raises an exception. 3. Join the thread and observe that only the `thread_begin` event is emitted without a corresponding `thread_end` event. ## Expected Behavior: The `TracePoint` should emit both `thread_begin` and `thread_end` events, accurately reflecting the lifecycle of the thread, even when an exception is raised within the thread. ## Actual Behavior: The `TracePoint` emits the `thread_begin` event but fails to emit the `thread_end` event when an exception is raised within the thread, indicating an incomplete tracing of thread lifecycle events. I've confirmed this as far back as Ruby 2.6. ## Example Code ```ruby TracePoint.trace(:thread_begin, :thread_end) do |tp| p [tp.event, tp.lineno, tp.path, tp.defined_class, tp.method_id] end thread = Thread.new do raise Interrupt end thread.join ``` ## Possible Fix Changing the implementation of `thread_do_start` to have what amounts to an "ensure" block. ```c static void thread_do_start(rb_thread_t *th) { native_set_thread_name(th); VALUE result = Qundef; rb_execution_context_t *ec = th->ec; int state; EXEC_EVENT_HOOK(ec, RUBY_EVENT_THREAD_BEGIN, th->self, 0, 0, 0, Qundef); EC_PUSH_TAG(ec); if ((state = EC_EXEC_TAG()) == TAG_NONE) { switch (th->invoke_type) { case thread_invoke_type_proc: result = thread_do_start_proc(th); break; case thread_invoke_type_ractor_proc: result = thread_do_start_proc(th); rb_ractor_atexit(th->ec, result); break; case thread_invoke_type_func: result = (*th->invoke_arg.func.func)(th->invoke_arg.func.arg); break; case thread_invoke_type_none: rb_bug("unreachable"); } } EC_POP_TAG(); VALUE errinfo = ec->errinfo; if (!NIL_P(errinfo) && !RB_TYPE_P(errinfo, T_OBJECT)) { ec->errinfo = Qnil; } rb_fiber_scheduler_set(Qnil); EXEC_EVENT_HOOK(th->ec, RUBY_EVENT_THREAD_END, th->self, 0, 0, 0, Qundef); ec->errinfo = errinfo; if (state) EC_JUMP_TAG(ec, state); th->value = result; } ``` It's possible `rb_fiber_scheduler_set(Qnil);` can emit an exception itself. How do we write the code to handle that case? -- https://bugs.ruby-lang.org/

3 5

[ruby-core:118458] [Ruby master Bug#18061] Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found
by kjtsanaktsidis (KJ Tsanaktsidis) 06 Jul '24

06 Jul '24

Issue #18061 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). > So either the metadata should signal just IBT in the meanwhile I've done this in https://github.com/ruby/ruby/pull/11112 ---------------------------------------- Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found https://bugs.ruby-lang.org/issues/18061#change-108972 * Author: jaruga (Jun Aruga) * Status: Assigned * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED ---------------------------------------- I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2]. ``` Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found ``` Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb, ``` $ autoconf $ ./configure --enable-shared $ make $ ls libruby.so.3.1.0 libruby.so.3.1.0* ``` If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this. ``` $ sudo dnf -y install annobin-annocheck ``` ``` $ sudo yum -y install annobin-annocheck ``` Then ``` $ annocheck libruby.so.3.1.0 ``` If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared. Prepare the following `Dockerfile`. ``` $ cat Dockerfile FROM docker.io/fedora:34 RUN cat /etc/fedora-release RUN dnf -y install annobin-annocheck WORKDIR /work ``` Then build the container image with the `Dockerfile` and run the annocheck command for the `libruby.so.3.1.0` on your host environment. The `-v` is an option for bind mount between host and container environment. ``` $ docker build --rm -t fedora-annocheck . $ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0 annocheck: Version 9.79. Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found Hardened: Rerun annocheck with --verbose to see more information on the tests. ``` The message `Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found` is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build. Asking a colleague, I was told that the `coroutine/*/Context.S` files such as [coroutine/x86/Context.S](https://github.com/ruby/ruby/blob/master/coroutine… cause the failure. Do you have any idea how to fix this? Thanks. * [1] https://sourceware.org/annobin/ * [2] You can see `man annocheck` or https://www.mankier.com/1/annocheck . ---Files-------------------------------- 0001-Add-.note.gnu.property-sections.patch (2.64 KB) 0001-Add-.note.gnu.property-sections.patch (3.69 KB) config-pie.log (11.4 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118457] [Ruby master Bug#18061] Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found
by kjtsanaktsidis (KJ Tsanaktsidis) 05 Jul '24

05 Jul '24

Issue #18061 has been updated by kjtsanaktsidis (KJ Tsanaktsidis). I wonder if distributors who want shadow-stack support should just compile Ruby with `--with-coroutine=ucontext` to use the `swapcontext(3)` based implementation of fibers instead of the assembly language one. I've spent the morning looking at the glibc sources trying to figure out how CET shadow stacks are supposed to work, but... we also can just use the glibc implementation instead of copying it :) ---------------------------------------- Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found https://bugs.ruby-lang.org/issues/18061#change-108971 * Author: jaruga (Jun Aruga) * Status: Assigned * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED ---------------------------------------- I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2]. ``` Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found ``` Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb, ``` $ autoconf $ ./configure --enable-shared $ make $ ls libruby.so.3.1.0 libruby.so.3.1.0* ``` If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this. ``` $ sudo dnf -y install annobin-annocheck ``` ``` $ sudo yum -y install annobin-annocheck ``` Then ``` $ annocheck libruby.so.3.1.0 ``` If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared. Prepare the following `Dockerfile`. ``` $ cat Dockerfile FROM docker.io/fedora:34 RUN cat /etc/fedora-release RUN dnf -y install annobin-annocheck WORKDIR /work ``` Then build the container image with the `Dockerfile` and run the annocheck command for the `libruby.so.3.1.0` on your host environment. The `-v` is an option for bind mount between host and container environment. ``` $ docker build --rm -t fedora-annocheck . $ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0 annocheck: Version 9.79. Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found Hardened: Rerun annocheck with --verbose to see more information on the tests. ``` The message `Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found` is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build. Asking a colleague, I was told that the `coroutine/*/Context.S` files such as [coroutine/x86/Context.S](https://github.com/ruby/ruby/blob/master/coroutine… cause the failure. Do you have any idea how to fix this? Thanks. * [1] https://sourceware.org/annobin/ * [2] You can see `man annocheck` or https://www.mankier.com/1/annocheck . ---Files-------------------------------- 0001-Add-.note.gnu.property-sections.patch (2.64 KB) 0001-Add-.note.gnu.property-sections.patch (3.69 KB) config-pie.log (11.4 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117453] [Ruby master Bug#20413] Enumerator can block fiber scheduler.
by ioquatix (Samuel Williams) 05 Jul '24

05 Jul '24

Issue #20413 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20413: Enumerator can block fiber scheduler. https://bugs.ruby-lang.org/issues/20413 * Author: ioquatix (Samuel Williams) * Status: Closed * Assignee: ioquatix (Samuel Williams) * Backport: 3.0: UNKNOWN, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- Using `Enumerator` in the event loop can cause problems as the fiber created by `rb_fiber_new` is blocking by default. It should be non-blocking. See <https://github.com/ruby/ruby/pull/10478> for the fix. -- https://bugs.ruby-lang.org/

4 5

[ruby-core:117458] [Ruby master Bug#20414] `Fiber#raise` should recurse to `resumed_fiber` rather than failing.
by ioquatix (Samuel Williams) 05 Jul '24

05 Jul '24

Issue #20414 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #20414: `Fiber#raise` should recurse to `resumed_fiber` rather than failing. https://bugs.ruby-lang.org/issues/20414 * Author: ioquatix (Samuel Williams) * Status: Open * Assignee: ioquatix (Samuel Williams) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The following program will fail with `FiberError`, and is difficult to properly clean up: ```ruby root_fiber = Fiber.current f1 = Fiber.new do root_fiber.transfer end f2 = Fiber.new do f1.resume end f2.transfer f2.raise("error") # => `raise': attempt to transfer to a resuming fiber (FiberError) ``` This program deliberately set's up a scenario where `f2` is resuming `f1`. Trying to raise an exception on `f2` is impossible, because the only way control flow can return to it, is when `f1` yields or exits. We can avoid this problem, by raising the exception on f1, and we can do this automatically using the following logic: ```c static VALUE fiber_raise(rb_fiber_t *fiber, VALUE exception) { // Add this recursive step: if (fiber->resuming_fiber) { return fiber_raise(fiber->resuming_fiber, exception); } // Existing code ... else if (FIBER_SUSPENDED_P(fiber) && !fiber->yielding) { return fiber_transfer_kw(fiber, -1, &exception, RB_NO_KEYWORDS); } else { return fiber_resume_kw(fiber, -1, &exception, RB_NO_KEYWORDS); } } ``` This makes `Fiber#raise` much more robust and useful for the purpose of stopping fibers, without knowing exactly what they are doing. -- https://bugs.ruby-lang.org/

4 9

[ruby-core:117947] [Ruby master Bug#20499] Ruby builds on macOS store absolute paths for AR and NM in rbconfig since Ruby 3.2.3/3.3.0
by Bo98 (Bo Anderson) 05 Jul '24

05 Jul '24

Issue #20499 has been reported by Bo98 (Bo Anderson). ---------------------------------------- Bug #20499: Ruby builds on macOS store absolute paths for AR and NM in rbconfig since Ruby 3.2.3/3.3.0 https://bugs.ruby-lang.org/issues/20499 * Author: Bo98 (Bo Anderson) * Status: Open * ruby -v: ruby 3.2.4 (2024-04-23 revision af471c0e01) [arm64-darwin23] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is a regression introduced by https://github.com/ruby/ruby/commit/038f9ade3c4d965415e4956561975454cf9eeb21 and causes various downstream problems such as https://github.com/ruby/prism/issues/2716. Development tools on macOS are relocatable - Xcode.app can exist anywhere on the system so it cannot be assumed that the location on the build machine matches the location on the running machine. If an absolute path must be used, `/usr/bin/nm` and `/usr/bin/ar` are acceptable. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118431] [Ruby master Bug#20606] Thread#thread_variable_get, Thread#thread_variable? and Thread#[] methods handle non-String/Symbol parameter values differently
by andrykonchin (Andrew Konchin) 05 Jul '24

05 Jul '24

Issue #20606 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #20606: Thread#thread_variable_get, Thread#thread_variable? and Thread#[] methods handle non-String/Symbol parameter values differently https://bugs.ruby-lang.org/issues/20606 * Author: andrykonchin (Andrew Konchin) * Status: Open * ruby -v: 3.2.4 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The `Thread#thread_variable_get`, `Thread#thread_variable?` and `Thread#[]` methods handle the `key` parameter that is not a String or a Symbol in different way but I would expect them to be consistent. When no thread-local variables were assigned to a thread the `Thread#thread_variable_get` and `Thread#thread_variable?` methods don't raise `TypeError` when argument is of incorrect type. But `Thread#[]` does raise `TypeError` exception: ```ruby t = Thread.new {}.join puts t.thread_variable_get(123).inspect # nil puts t.thread_variable?(123).inspect # false t[123] # `[]': 123 is not a symbol nor a string (TypeError) ``` -- https://bugs.ruby-lang.org/

3 4

[ruby-core:118444] [Ruby master Bug#18061] Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found
by vo.x (Vit Ondruch) 04 Jul '24

04 Jul '24

Issue #18061 has been updated by vo.x (Vit Ondruch). fweimer (Florian Weimer) wrote in #note-48: > The change in https://github.com/ruby/ruby/pull/11081/files does not look correct to me because it enables shadow stack and indirect branch tracking in the ELF markup. But the changes to `Context.S` merely enable indirect branch tracking markup. The shadow stack is not context-switched, so this will crash on the `ret` instruction. So either the metadata should signal just IBT or the shadow stack needs to be implemented, right? ---------------------------------------- Bug #18061: Execshield test: libruby.so.N.N.N: FAIL: property-note test because no .note.gnu.property section found https://bugs.ruby-lang.org/issues/18061#change-108959 * Author: jaruga (Jun Aruga) * Status: Assigned * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 2.6: UNKNOWN, 2.7: REQUIRED, 3.0: REQUIRED ---------------------------------------- I found an issue in our company's internal test called "execshield" by a security tool annobin - annocheck command [1][2]. ``` Hardened: libruby.so.2.7.4: FAIL: property-note test because no .note.gnu.property section found ``` Here is the reproducer on the upstream latest master, commit is 5f2987d6c2ae9ace3178ac3e1bbb4ac7079101eb, ``` $ autoconf $ ./configure --enable-shared $ make $ ls libruby.so.3.1.0 libruby.so.3.1.0* ``` If you are using Red Hat based Linux distro, it's easy to install by the RPM package like this. ``` $ sudo dnf -y install annobin-annocheck ``` ``` $ sudo yum -y install annobin-annocheck ``` Then ``` $ annocheck libruby.so.3.1.0 ``` If you are using other Linux distros such as Ubuntu, you can use it by a container I prepared. Prepare the following `Dockerfile`. ``` $ cat Dockerfile FROM docker.io/fedora:34 RUN cat /etc/fedora-release RUN dnf -y install annobin-annocheck WORKDIR /work ``` Then build the container image with the `Dockerfile` and run the annocheck command for the `libruby.so.3.1.0` on your host environment. The `-v` is an option for bind mount between host and container environment. ``` $ docker build --rm -t fedora-annocheck . $ docker run --rm -t -v $(pwd):/work fedora-annocheck annocheck /work/libruby.so.3.1.0 annocheck: Version 9.79. Hardened: libruby.so.3.1.0: FAIL: bind-now test because not linked with -Wl,-z,now Hardened: libruby.so.3.1.0: FAIL: notes test because gaps were detected in the annobin coverage Hardened: libruby.so.3.1.0: FAIL: cf-protection test because no .note.gnu.property section = no control flow information Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found Hardened: Rerun annocheck with --verbose to see more information on the tests. ``` The message `Hardened: libruby.so.3.1.0: FAIL: property-note test because no .note.gnu.property section found` is what I found in our internal test. For other FAIL messages, maybe it can be fixed by changing how to build. Asking a colleague, I was told that the `coroutine/*/Context.S` files such as [coroutine/x86/Context.S](https://github.com/ruby/ruby/blob/master/coroutine… cause the failure. Do you have any idea how to fix this? Thanks. * [1] https://sourceware.org/annobin/ * [2] You can see `man annocheck` or https://www.mankier.com/1/annocheck . ---Files-------------------------------- 0001-Add-.note.gnu.property-sections.patch (2.64 KB) 0001-Add-.note.gnu.property-sections.patch (3.69 KB) config-pie.log (11.4 KB) -- https://bugs.ruby-lang.org/

1 0