November 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113782] [Ruby master Bug#19716] SystemStackError occurs too easily on Alpine Linux (due to small stack size reported by pthread_attr_getstacksize on musl libc)
by alexdowad (Alex Dowad) 27 Mar '24

27 Mar '24

Issue #19716 has been reported by alexdowad (Alex Dowad). ---------------------------------------- Bug #19716: SystemStackError occurs too easily on Alpine Linux (due to small stack size reported by pthread_attr_getstacksize on musl libc) https://bugs.ruby-lang.org/issues/19716 * Author: alexdowad (Alex Dowad) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux-musl] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This is the same problem previously reported against Ruby 2.5 in https://bugs.ruby-lang.org/issues/14387. I just ran into the same problem on Ruby 3.1.4, built on Alpine Linux 3.16. @hsbt stated in the previous thread (https://bugs.ruby-lang.org/issues/14387#note-28) > If you have this issue with Ruby 3.2, please file it with another issue. I hacked `stack_check` in gc.c to print the values of `STACK_START` and `STACK_END` on stack overflow; on the Alpine 3.16 host where this problem just occurred, the values printed were: > Start=0x7ffd0bf4f000, End=0x7ffd0bf32530 ...which shows that Ruby thinks the stack size is only 131072 bytes. On the other hand, `ulimit -s` shows a stack size limit of 8192kb. This Ruby 3.1.4 was built from unmodified source code downloaded from https://cache.ruby-lang.org; the build was configured using `CFLAGS='-march=native' ./configure --disable-install-doc`. The invocation of Ruby which blew the stack was `bundle exec rake db:migrate`, on a mid-sized Rails project. Regarding @ncopa's patch from #14387, @wanabe listed some things which should be done before it is merged into mainline Ruby: > Okay, The patch needs one or more proofs of its behaviour, like that: > > Original issue [ruby-dev:50421] has gone away. > Standard test codes run well. > test-all > ruby/spec > getrlimit works on some situations like: > on single thread > with multiple threads > with RLIMIT_STACK environment variable > getrlimit code of musl is implemented correctly as expected. > (But It's doubtful whether it can be. I guess that a proof of code soundness is very difficult.) > Some "real world" applications can work. > I think it is better example that that application(s) can't work without the patch. I am happy to help cover some of these points if the Ruby development team is still interested in merging @ncopa's patch. -- https://bugs.ruby-lang.org/

4 5

[ruby-core:115538] [Ruby master Misc#20028] I'd like my commit bit back
by zenspider (Ryan Davis) 26 Mar '24

26 Mar '24

Issue #20028 has been reported by zenspider (Ryan Davis). ---------------------------------------- Misc #20028: I'd like my commit bit back https://bugs.ruby-lang.org/issues/20028 * Author: zenspider (Ryan Davis) * Status: Open * Priority: Normal ---------------------------------------- It's been a while, in the svn -> git shuffle I lost commit privs. I'd like to help out more actively, triage issues, clean doco, etc. Not sure who's doing admin work so I'm not sure who to assign to to expedite. -- https://bugs.ruby-lang.org/

4 6

[ruby-core:111075] [Ruby master Bug#19161] Cannot compile 3.0.5 or 3.1.3 on Red Hat 7
by werebus (Matt Moretti) 25 Mar '24

25 Mar '24

Issue #19161 has been reported by werebus (Matt Moretti). ---------------------------------------- Bug #19161: Cannot compile 3.0.5 or 3.1.3 on Red Hat 7 https://bugs.ruby-lang.org/issues/19161 * Author: werebus (Matt Moretti) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- When attempting to run `make` on either the 3.0.5 or 3.1.3 release, I get the following error (I included the whole output as it's pretty short): ``` BASERUBY = /opt/ruby/bin/ruby --disable=gems CC = gcc -std=gnu11 LD = ld LDSHARED = gcc -std=gnu11 -shared CFLAGS = -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wdeprecated-declarations -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-packed-bitfield-compat -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wsuggest-attribute=format -Wsuggest-attribute=noreturn -Wunused-variable XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -fPIE -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/13.0.0 CPPFLAGS = DLDFLAGS = -Wl,--compress-debug-sections=zlib -fstack-protector-strong -pie SOLIBS = -lz -lpthread -lrt -lrt -ldl -lcrypt -lm LANG = en_US.UTF-8 LC_ALL = LC_CTYPE = MFLAGS = gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. compiling ./main.c compiling dmydln.c compiling miniinit.c In file included from vm_core.h:83:0, from iseq.h:14, from mini_builtin.c:3, from miniinit.c:51: thread_pthread.h:108:43: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘struct’ RUBY_EXTERN RB_THREAD_LOCAL_SPECIFIER struct rb_execution_context_struct *ruby_current_ec; ^ In file included from iseq.h:14:0, from mini_builtin.c:3, from miniinit.c:51: vm_core.h: In function ‘rb_current_execution_context’: vm_core.h:1870:34: error: ‘ruby_current_ec’ undeclared (first use in this function) rb_execution_context_t *ec = ruby_current_ec; ^ vm_core.h:1870:34: note: each undeclared identifier is reported only once for each function it appears in miniinit.c: At top level: cc1: warning: unrecognized command line option "-Wno-tautological-compare" [enabled by default] cc1: warning: unrecognized command line option "-Wno-self-assign" [enabled by default] cc1: warning: unrecognized command line option "-Wno-parentheses-equality" [enabled by default] cc1: warning: unrecognized command line option "-Wno-constant-logical-operand" [enabled by default] cc1: warning: unrecognized command line option "-Wno-cast-function-type" [enabled by default] make: *** [miniinit.o] Error 1 ``` Both Ruby 3.0.4 and 3.1.2 build without error and pass (all but one of) the `make check` tests. I know RHEL 7 is getting to be pretty old; I suspect a factor here are the ancient build tools available to me. But... EOL for Ruby 2.7 comes before the one for RHEL 7, so I'm trying to prioritize. -- https://bugs.ruby-lang.org/

10 14

[ruby-core:114309] [Ruby master Feature#19787] Add Enumerable#uniq_map, Enumerable::Lazy#uniq_map, Array#uniq_map and Array#uniq_map!
by joshuay03 (Joshua Young) 15 Mar '24

15 Mar '24

Issue #19787 has been reported by joshuay03 (Joshua Young). ---------------------------------------- Feature #19787: Add Enumerable#uniq_map, Enumerable::Lazy#uniq_map, Array#uniq_map and Array#uniq_map! https://bugs.ruby-lang.org/issues/19787 * Author: joshuay03 (Joshua Young) * Status: Open * Priority: Normal ---------------------------------------- I would like to propose a collection of new methods, `Enumerable#uniq_map`, `Enumerable::Lazy#uniq_map`, `Array#uniq_map` and `Array#uniq_map!`. TL;DR: It's a drop in replacement for `.map { ... }.uniq`, with better performance. I've quite often had to map over an array and get its unique elements. It occurred to me when doing so recently that Ruby doesn't have a short form method for doing that, similar to how `flat_map { ... }` replaces `.map { ... }.flatten` and `filter_map { ... }` replaces `.map { ... }.compact` (with minor differences). I think these new methods could be beneficial both in terms of better performance and writing more succinct code. I have already got a draft PR up with some initial benchmarks in the description: https://github.com/ruby/ruby/pull/8140. -- https://bugs.ruby-lang.org/

4 6

[ruby-core:113966] [Ruby master Bug#19742] Introduce `Module#anonymous?`
by ioquatix (Samuel Williams) 12 Mar '24

12 Mar '24

Issue #19742 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19742: Introduce `Module#anonymous?` https://bugs.ruby-lang.org/issues/19742 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- As a follow-on <from https://bugs.ruby-lang.org/issues/19521>, I'd like propose we introduce `Module#anonymous?`. In some situations, like logging/formatting, serialisation/deserialization, debugging or meta-programming, we might like to know if a class is a proper constant or not. However, this brings about some other issues which might need to be discussed. After assigning a constant, then removing it, the internal state of Ruby still believes that the class name is permanent, even thought it's no longer true. e.g. ``` m = Module.new m.anonymous? # true M = m m.anonyomous # false Object.send(:remove_const, :M) M # uninitialized constant M (NameError) M.anonymous? # false ``` Because RCLASS data structure is not updated after the constant is removed, internally the state still has a "permanent class name". I want to use this proposal to discuss this issue and whether there is anything we should do about such behaviour (or even if it's desirable). Proposed PR: https://github.com/ruby/ruby/pull/7966 cc @fxn -- https://bugs.ruby-lang.org/

9 46

[ruby-core:115169] [Ruby master Bug#19973] Duplicate keyword argument names don't always warn
by tenderlovemaking (Aaron Patterson) 24 Feb '24

24 Feb '24

Issue #19973 has been reported by tenderlovemaking (Aaron Patterson). ---------------------------------------- Bug #19973: Duplicate keyword argument names don't always warn https://bugs.ruby-lang.org/issues/19973 * Author: tenderlovemaking (Aaron Patterson) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-10-24T19:38:50Z cleanup 3525a9bd22) [arm64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Calling a method with duplicate keyword arguments doesn't warn when it could: ```ruby def bar a:, b: a + b end # Warning bar(a: 1, b: 3, a: 2) z = { b: 123 } # No warning bar(a: 1, **z, a: 2) ``` The first call to `bar` gives a warning about duplicate keyword args, but the second call doesn't. I think both cases should emit a warning. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:111965] [Ruby master Bug#19365] Ractors can access non-shareable values through enumerators
by luke-gru (Luke Gruber) 16 Feb '24

16 Feb '24

Issue #19365 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19365: Ractors can access non-shareable values through enumerators https://bugs.ruby-lang.org/issues/19365 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I don't think enumerators should be able to be passed to `Ractor.new` ```ruby obj = Object.new # unshareable value p obj Ractor.new([obj].each) {|f| p f.first }.take ``` -- https://bugs.ruby-lang.org/

2 3

[ruby-core:112155] [Ruby master Bug#19395] Process forking within non-main Ractor creates child stuck in busy loop
by luke-gru (Luke Gruber) 16 Feb '24

16 Feb '24

Issue #19395 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19395: Process forking within non-main Ractor creates child stuck in busy loop https://bugs.ruby-lang.org/issues/19395 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ```ruby def test_fork_in_ractor r2 = Ractor.new do pid = fork do exit Ractor.count end pid end pid = r2.take puts "Process #{Process.pid} waiting for #{pid}" _pid, status = Process.waitpid2(pid) # stuck forever if status.exitstatus != 1 raise "status is #{status.exitstatus}" end end test_fork_in_ractor() ``` $ top # shows CPU usage is high for child process -- https://bugs.ruby-lang.org/

2 5

[ruby-core:114915] [Ruby master Feature#19905] Introduce `Queue#peek`
by hi＠joaofernandes.me (Joao Fernandes) 13 Feb '24

13 Feb '24

Issue #19905 has been reported by hi(a)joaofernandes.me (Joao Fernandes). ---------------------------------------- Feature #19905: Introduce `Queue#peek` https://bugs.ruby-lang.org/issues/19905 * Author: hi(a)joaofernandes.me (Joao Fernandes) * Status: Open * Priority: Normal ---------------------------------------- This ticket proposes the introduction of the `Queue#peek` method, similar to what we can find in other object oriented languages such as Java and C#. This method is similar to `Queue#pop`, but does not change the data, nor does it require a lock. ``` q = Queue.new([1,2,3]) => #<Thread::Queue:0x00000001065d7148> q.peek => 1 q.peek => 1 ``` I have felt the need of this for debugging, but I think that it can also be of practical use for presentation. I believe that the only drawback could be that newcomers could misuse it in multi-threaded work without taking into account that this method is not thread safe. I also volunteer myself to implement this method. -- https://bugs.ruby-lang.org/

6 7

[ruby-core:115286] [Ruby master Bug#19991] rb_register_postponed_job async-signal-unsafety causes crash in GC
by kjtsanaktsidis (KJ Tsanaktsidis) 13 Feb '24

13 Feb '24

Issue #19991 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #19991: rb_register_postponed_job async-signal-unsafety causes crash in GC https://bugs.ruby-lang.org/issues/19991 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Our production application experienced an interpreter crash today that I’m fairly sure is the result of a bug in the `rb_register_postponed_job` infrastructure. ## Diagnosis I’ve attached a more complete version of the crash dump, but the key information I think is the following backtrace: ``` <internal:gc>:35: [BUG] Segmentation fault at 0x00000000000000c8 .... /usr/lib/libruby.so.3.1(rb_vm_bugreport+0x60c) [0xffffaaf57a6c] vm_dump.c:759 /usr/lib/libruby.so.3.1(rb_bug_for_fatal_signal+0xd8) [0xffffaad6d120] error.c:821 /usr/lib/libruby.so.3.1(sigsegv+0x58) [0xffffaaeb4350] signal.c:964 /usr/lib/libruby.so.3.1(sigill) (null):0 linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffffab20f78c] /usr/lib/libruby.so.3.1(rbimpl_atomic_exchange+0x0) [0xffffaad94018] gc.c:4081 /usr/lib/libruby.so.3.1(gc_finalize_deferred) gc.c:4081 /usr/lib/libruby.so.3.1(rb_postponed_job_flush+0x218) [0xffffaaf5dd60] vm_trace.c:1728 /usr/lib/libruby.so.3.1(rb_threadptr_execute_interrupts+0x328) [0xffffaaef7520] thread.c:2444 /usr/lib/libruby.so.3.1(vm_exec_core+0x4e54) [0xffffaaf41c6c] vm.inc:588 /usr/lib/libruby.so.3.1(rb_vm_exec+0x144) [0xffffaaf427cc] vm.c:2211 ``` I disassembled `gc_finalize_deferred` in GDB (different execution, so the addresses don’t line up - this is NOT from a core dump): ``` (gdb) disassemble gc_finalize_deferred Dump of assembler code for function gc_finalize_deferred: 0x0000fffff7cc3ff8 <+0>: stp x29, x30, [sp, #-48]! 0x0000fffff7cc3ffc <+4>: mov w1, #0x1 // #1 0x0000fffff7cc4000 <+8>: mov x29, sp 0x0000fffff7cc4004 <+12>: stp x19, x20, [sp, #16] 0x0000fffff7cc4008 <+16>: mov x19, x0 0x0000fffff7cc400c <+20>: add x20, x0, #0xc8 0x0000fffff7cc4010 <+24>: ldaxr w0, [x20] .... continues .... ``` Based on the line numbers from the debuginfo, the faulting instruction is likely to be `gc_finalize_deferred+24` (which is the inlined `rbimpl_atomic_exchange`). That would attempt a load of 0xc8 if x0 was zero - i.e. if `gc_finalize_deferred` was called with a NULL objspace. The enqueuing of `gc_finalize_deferred` with postponed job only happens in one place (in gc_sweep_page, [here](https://github.com/ruby/ruby/blob/5cff4c5aa375787924e2df5c0b981dd922b…) and if `objspace` was null there then the crash would have had to have already happened in `gc_sweep_page`. Thus, I think that `gc_finalize_deferred` was _enqueued_ into the postponed job list with a not-NULL argument, but the argument was corrupted whilst it was in `vm->postponed_job_buffer`, and if objspace was null there then the crash would have had to have already happened in `gc_sweep_page`. Thus, I think that `gc_finalize_deferred` was enqueued into the postponed job list with a not-NULL argument, but the argument was corrupted whilst it was in `vm->postponed_job_buffer`. I had a look at the postponed job code, which is of course very tricky because it needs to be async-signal-safe. More specifically: * It needs to work if run from a thread not holding the GVL * It needs to work if run from a thread, whilst another thread is actually executing `rb_postponed_job_flush` * It needs to work if run from a signal caught in a thread that is currently executing `rb_postponed_job_flush` (this rules out trivial mutex-based solutions) We use the Datadog continuous profiler in our application (CC: @ivoanjo ;), which calls `rb_postponed_job_register` to capture profiling samples. Thus, I think our application is likely to hit all of those scenarios semi-reguarly. My best guess at a plausible sequence of events, to generate this particular crash, is that: 1. `rb_postponed_job_flush` was running on thread T1. 2. There is a queued call to gc_finalize_deferred sitting in `vm->postponed_job_buffer[vm->postponed_job_index-1]`. 3. T1 executed the `ATOMIC_CAS` at vm_trace.c:1800, decrementing `vm->postponed_job_index` (which now equals `index - 1`) and determining that a job at index index should be run. 4. Thread T2 received a signal, and the Datadog continuous profiler called `rb_postponed_job_register_one(0, sample_from_postponed_job, NULL)` 5. T2 executed the `ATOMIC_CAS` at vm_trace.c:1679, re-incrementing `vm->postponed_job_index`. It’s now equal to `index` from T1 again. 6. T2 then executes the sets on `pjob->func` and `pjob->data` at vm_trace.c:1687. It sets `->func` to `sample_from_postponed_job` (from ddtrace), and `->data` to 0. 7. T1 then goes to call `(*pjob->func)(pjob->data)` at vm_trace.c:1802 8. Since there is no memory barrier between 6 & 7, T1 is allowed to see the effects of the set on `pjob->data` and not see that of `pjob->func`. 9, T1 thus calls `gc_finalize_deferred` (which it was meant to do) with an argument of 0 (which it was not). ## Solution Managing a thread-safe list of too-big-to-be-atomic objects (like `rb_postponed_job_t`) is really tricky. I think it might be best for this code if we use a combination of masking signals (to prevent manipulation of the postponed job list occurring during `rb_postponed_job_flush`, and using a semaphore to protect the list (since semaphores are required to be async-signal-safe on POSIX systems). I've implemented this in a PR here: https://github.com/ruby/ruby/pull/8856 It seems _slightly_ slower to do it this way - semaphores require system calls in the uncontended case, which is why they're async-signal-safe but also makes them more expensive than pthread mutexes, which don't, on most systems. I ran my branch through yjit bench: With my patch: ``` interp: ruby 3.3.0dev (2023-11-07T08:14:09Z ktsanaktsidis/post.. 342f30f566) [arm64-darwin22] yjit: ruby 3.3.0dev (2023-11-07T08:14:09Z ktsanaktsidis/post.. 342f30f566) +YJIT [arm64-darwin22] -------------- ----------- ---------- --------- ---------- ------------ ----------- bench interp (ms) stddev (%) yjit (ms) stddev (%) yjit 1st itr interp/yjit activerecord 31.2 3.4 17.0 5.7 1.29 1.83 chunky-png 543.5 0.5 367.0 0.7 1.40 1.48 erubi-rails 1044.2 0.6 564.7 1.3 1.69 1.85 hexapdf 1517.6 3.1 917.6 1.2 1.46 1.65 liquid-c 37.1 1.3 28.9 1.4 0.89 1.29 liquid-compile 39.0 1.4 29.9 1.6 0.76 1.30 liquid-render 89.9 1.8 39.6 1.4 1.37 2.27 lobsters 598.2 1.7 435.4 5.2 0.63 1.37 mail 79.8 3.1 52.5 1.0 0.79 1.52 psych-load 1441.5 1.7 885.4 0.5 1.60 1.63 railsbench 1010.8 1.0 609.3 1.3 1.24 1.66 ruby-lsp 40.9 3.4 29.2 30.0 0.66 1.40 sequel 39.8 1.8 33.0 2.4 1.18 1.21 -------------- ----------- ---------- --------- ---------- ------------ ----------- ``` Without the patch: ``` interp: ruby 3.3.0dev (2023-11-07T07:56:43Z master 5a2779d40f) [arm64-darwin22] yjit: ruby 3.3.0dev (2023-11-07T07:56:43Z master 5a2779d40f) +YJIT [arm64-darwin22] -------------- ----------- ---------- --------- ---------- ------------ ----------- bench interp (ms) stddev (%) yjit (ms) stddev (%) yjit 1st itr interp/yjit activerecord 31.3 3.3 16.7 5.5 1.36 1.88 chunky-png 521.6 0.6 348.8 0.7 1.40 1.50 erubi-rails 1038.9 0.9 566.3 1.2 1.70 1.83 hexapdf 1501.9 1.1 951.7 3.9 1.42 1.58 liquid-c 36.7 1.2 29.3 1.7 0.86 1.25 liquid-compile 38.8 1.1 29.7 3.7 0.73 1.31 liquid-render 92.2 0.9 38.3 1.0 1.47 2.40 lobsters 582.5 2.0 429.8 5.6 0.59 1.36 mail 77.9 1.3 54.8 0.9 0.76 1.42 psych-load 1419.1 0.7 887.7 0.5 1.60 1.60 railsbench 1017.8 1.1 609.9 1.2 1.24 1.67 ruby-lsp 41.0 2.2 28.8 28.8 0.64 1.43 sequel 36.0 1.5 30.4 1.8 1.11 1.18 -------------- ----------- ---------- --------- ---------- ------------ ----------- ``` Maybe this is within the noise floor, but I thought I should bring it up. -- https://bugs.ruby-lang.org/

2 3