November 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115286] [Ruby master Bug#19991] rb_register_postponed_job async-signal-unsafety causes crash in GC
by kjtsanaktsidis (KJ Tsanaktsidis) 13 Feb '24

13 Feb '24

Issue #19991 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #19991: rb_register_postponed_job async-signal-unsafety causes crash in GC https://bugs.ruby-lang.org/issues/19991 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Our production application experienced an interpreter crash today that I’m fairly sure is the result of a bug in the `rb_register_postponed_job` infrastructure. ## Diagnosis I’ve attached a more complete version of the crash dump, but the key information I think is the following backtrace: ``` <internal:gc>:35: [BUG] Segmentation fault at 0x00000000000000c8 .... /usr/lib/libruby.so.3.1(rb_vm_bugreport+0x60c) [0xffffaaf57a6c] vm_dump.c:759 /usr/lib/libruby.so.3.1(rb_bug_for_fatal_signal+0xd8) [0xffffaad6d120] error.c:821 /usr/lib/libruby.so.3.1(sigsegv+0x58) [0xffffaaeb4350] signal.c:964 /usr/lib/libruby.so.3.1(sigill) (null):0 linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffffab20f78c] /usr/lib/libruby.so.3.1(rbimpl_atomic_exchange+0x0) [0xffffaad94018] gc.c:4081 /usr/lib/libruby.so.3.1(gc_finalize_deferred) gc.c:4081 /usr/lib/libruby.so.3.1(rb_postponed_job_flush+0x218) [0xffffaaf5dd60] vm_trace.c:1728 /usr/lib/libruby.so.3.1(rb_threadptr_execute_interrupts+0x328) [0xffffaaef7520] thread.c:2444 /usr/lib/libruby.so.3.1(vm_exec_core+0x4e54) [0xffffaaf41c6c] vm.inc:588 /usr/lib/libruby.so.3.1(rb_vm_exec+0x144) [0xffffaaf427cc] vm.c:2211 ``` I disassembled `gc_finalize_deferred` in GDB (different execution, so the addresses don’t line up - this is NOT from a core dump): ``` (gdb) disassemble gc_finalize_deferred Dump of assembler code for function gc_finalize_deferred: 0x0000fffff7cc3ff8 <+0>: stp x29, x30, [sp, #-48]! 0x0000fffff7cc3ffc <+4>: mov w1, #0x1 // #1 0x0000fffff7cc4000 <+8>: mov x29, sp 0x0000fffff7cc4004 <+12>: stp x19, x20, [sp, #16] 0x0000fffff7cc4008 <+16>: mov x19, x0 0x0000fffff7cc400c <+20>: add x20, x0, #0xc8 0x0000fffff7cc4010 <+24>: ldaxr w0, [x20] .... continues .... ``` Based on the line numbers from the debuginfo, the faulting instruction is likely to be `gc_finalize_deferred+24` (which is the inlined `rbimpl_atomic_exchange`). That would attempt a load of 0xc8 if x0 was zero - i.e. if `gc_finalize_deferred` was called with a NULL objspace. The enqueuing of `gc_finalize_deferred` with postponed job only happens in one place (in gc_sweep_page, [here](https://github.com/ruby/ruby/blob/5cff4c5aa375787924e2df5c0b981dd922b…) and if `objspace` was null there then the crash would have had to have already happened in `gc_sweep_page`. Thus, I think that `gc_finalize_deferred` was _enqueued_ into the postponed job list with a not-NULL argument, but the argument was corrupted whilst it was in `vm->postponed_job_buffer`, and if objspace was null there then the crash would have had to have already happened in `gc_sweep_page`. Thus, I think that `gc_finalize_deferred` was enqueued into the postponed job list with a not-NULL argument, but the argument was corrupted whilst it was in `vm->postponed_job_buffer`. I had a look at the postponed job code, which is of course very tricky because it needs to be async-signal-safe. More specifically: * It needs to work if run from a thread not holding the GVL * It needs to work if run from a thread, whilst another thread is actually executing `rb_postponed_job_flush` * It needs to work if run from a signal caught in a thread that is currently executing `rb_postponed_job_flush` (this rules out trivial mutex-based solutions) We use the Datadog continuous profiler in our application (CC: @ivoanjo ;), which calls `rb_postponed_job_register` to capture profiling samples. Thus, I think our application is likely to hit all of those scenarios semi-reguarly. My best guess at a plausible sequence of events, to generate this particular crash, is that: 1. `rb_postponed_job_flush` was running on thread T1. 2. There is a queued call to gc_finalize_deferred sitting in `vm->postponed_job_buffer[vm->postponed_job_index-1]`. 3. T1 executed the `ATOMIC_CAS` at vm_trace.c:1800, decrementing `vm->postponed_job_index` (which now equals `index - 1`) and determining that a job at index index should be run. 4. Thread T2 received a signal, and the Datadog continuous profiler called `rb_postponed_job_register_one(0, sample_from_postponed_job, NULL)` 5. T2 executed the `ATOMIC_CAS` at vm_trace.c:1679, re-incrementing `vm->postponed_job_index`. It’s now equal to `index` from T1 again. 6. T2 then executes the sets on `pjob->func` and `pjob->data` at vm_trace.c:1687. It sets `->func` to `sample_from_postponed_job` (from ddtrace), and `->data` to 0. 7. T1 then goes to call `(*pjob->func)(pjob->data)` at vm_trace.c:1802 8. Since there is no memory barrier between 6 & 7, T1 is allowed to see the effects of the set on `pjob->data` and not see that of `pjob->func`. 9, T1 thus calls `gc_finalize_deferred` (which it was meant to do) with an argument of 0 (which it was not). ## Solution Managing a thread-safe list of too-big-to-be-atomic objects (like `rb_postponed_job_t`) is really tricky. I think it might be best for this code if we use a combination of masking signals (to prevent manipulation of the postponed job list occurring during `rb_postponed_job_flush`, and using a semaphore to protect the list (since semaphores are required to be async-signal-safe on POSIX systems). I've implemented this in a PR here: https://github.com/ruby/ruby/pull/8856 It seems _slightly_ slower to do it this way - semaphores require system calls in the uncontended case, which is why they're async-signal-safe but also makes them more expensive than pthread mutexes, which don't, on most systems. I ran my branch through yjit bench: With my patch: ``` interp: ruby 3.3.0dev (2023-11-07T08:14:09Z ktsanaktsidis/post.. 342f30f566) [arm64-darwin22] yjit: ruby 3.3.0dev (2023-11-07T08:14:09Z ktsanaktsidis/post.. 342f30f566) +YJIT [arm64-darwin22] -------------- ----------- ---------- --------- ---------- ------------ ----------- bench interp (ms) stddev (%) yjit (ms) stddev (%) yjit 1st itr interp/yjit activerecord 31.2 3.4 17.0 5.7 1.29 1.83 chunky-png 543.5 0.5 367.0 0.7 1.40 1.48 erubi-rails 1044.2 0.6 564.7 1.3 1.69 1.85 hexapdf 1517.6 3.1 917.6 1.2 1.46 1.65 liquid-c 37.1 1.3 28.9 1.4 0.89 1.29 liquid-compile 39.0 1.4 29.9 1.6 0.76 1.30 liquid-render 89.9 1.8 39.6 1.4 1.37 2.27 lobsters 598.2 1.7 435.4 5.2 0.63 1.37 mail 79.8 3.1 52.5 1.0 0.79 1.52 psych-load 1441.5 1.7 885.4 0.5 1.60 1.63 railsbench 1010.8 1.0 609.3 1.3 1.24 1.66 ruby-lsp 40.9 3.4 29.2 30.0 0.66 1.40 sequel 39.8 1.8 33.0 2.4 1.18 1.21 -------------- ----------- ---------- --------- ---------- ------------ ----------- ``` Without the patch: ``` interp: ruby 3.3.0dev (2023-11-07T07:56:43Z master 5a2779d40f) [arm64-darwin22] yjit: ruby 3.3.0dev (2023-11-07T07:56:43Z master 5a2779d40f) +YJIT [arm64-darwin22] -------------- ----------- ---------- --------- ---------- ------------ ----------- bench interp (ms) stddev (%) yjit (ms) stddev (%) yjit 1st itr interp/yjit activerecord 31.3 3.3 16.7 5.5 1.36 1.88 chunky-png 521.6 0.6 348.8 0.7 1.40 1.50 erubi-rails 1038.9 0.9 566.3 1.2 1.70 1.83 hexapdf 1501.9 1.1 951.7 3.9 1.42 1.58 liquid-c 36.7 1.2 29.3 1.7 0.86 1.25 liquid-compile 38.8 1.1 29.7 3.7 0.73 1.31 liquid-render 92.2 0.9 38.3 1.0 1.47 2.40 lobsters 582.5 2.0 429.8 5.6 0.59 1.36 mail 77.9 1.3 54.8 0.9 0.76 1.42 psych-load 1419.1 0.7 887.7 0.5 1.60 1.60 railsbench 1017.8 1.1 609.9 1.2 1.24 1.67 ruby-lsp 41.0 2.2 28.8 28.8 0.64 1.43 sequel 36.0 1.5 30.4 1.8 1.11 1.18 -------------- ----------- ---------- --------- ---------- ------------ ----------- ``` Maybe this is within the noise floor, but I thought I should bring it up. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:115143] [Ruby master Bug#19970] Eval leaks callcache and callinfo objects on arm32 (linux)
by larsin (Lars Ingjer) 12 Feb '24

12 Feb '24

Issue #19970 has been reported by larsin (Lars Ingjer). ---------------------------------------- Bug #19970: Eval leaks callcache and callinfo objects on arm32 (linux) https://bugs.ruby-lang.org/issues/19970 * Author: larsin (Lars Ingjer) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [armv7l-linux-eabihf] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following script demonstrates a memory leak on arm 32 (linux): ``` ruby def gcdiff(n) GC.start if @last_gc_stat puts "GC.stat #{n} diff old_objects: #{GC.stat(:old_objects) - @last_gc_stat}" end @last_gc_stat = GC.stat(:old_objects) end def foo end 10.times do |i| 10_000.times do eval 'foo' end gcdiff(i) puts "Number of live objects: #{GC.stat(:heap_live_slots)}" puts "Memory usage: #{`ps -o rss= -p #{$$}`}" puts end ``` Output: ``` Number of live objects: 41303 Memory usage: 11900 GC.stat 1 diff old_objects: 20037 Number of live objects: 61317 Memory usage: 13604 GC.stat 2 diff old_objects: 20001 Number of live objects: 81317 Memory usage: 14880 GC.stat 3 diff old_objects: 20000 Number of live objects: 101317 Memory usage: 16596 GC.stat 4 diff old_objects: 20000 Number of live objects: 121317 Memory usage: 17248 GC.stat 5 diff old_objects: 20000 Number of live objects: 141317 Memory usage: 18760 GC.stat 6 diff old_objects: 20000 Number of live objects: 161317 Memory usage: 19540 GC.stat 7 diff old_objects: 20000 Number of live objects: 181317 Memory usage: 21752 GC.stat 8 diff old_objects: 20000 Number of live objects: 201317 Memory usage: 21828 GC.stat 9 diff old_objects: 20000 Number of live objects: 221317 Memory usage: 24896 ``` ObjectSpace.count_imemo_objects shows that imemo_callcache and imemo_callinfo are leaking. The issue does not occur on arm64 mac or x86_64 linux with the same ruby version. The issue has also been reproduced with the latest 3.2.2 snapshot (2023-09-30). -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112947] [Ruby master Bug#19542] Operations on zero-sized IO::Buffer are raising
by hanazuki (Kasumi Hanazuki) 05 Feb '24

05 Feb '24

Issue #19542 has been reported by hanazuki (Kasumi Hanazuki). ---------------------------------------- Bug #19542: Operations on zero-sized IO::Buffer are raising https://bugs.ruby-lang.org/issues/19542 * Author: hanazuki (Kasumi Hanazuki) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I found that IO::Buffer of zero length is not cloneable. ``` % ruby -v ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux] % ruby -e 'p IO::Buffer.for("").dup' -e:1:in `initialize_copy': The buffer is not allocated! (IO::Buffer::AllocationError) from -e:1:in `initialize_dup' from -e:1:in `dup' from -e:1:in `<main>' % ruby -e 'p IO::Buffer.new(0).dup' -e:1: warning: IO::Buffer is experimental and both the Ruby and C interface may change in the future! -e:1:in `initialize_copy': The buffer is not allocated! (IO::Buffer::AllocationError) from -e:1:in `initialize_dup' from -e:1:in `dup' from -e:1:in `<main>' ``` It seems `IO::Buffer.new(0)` allocates no memory for buffer on object creation and thus prohibits reading from or writing to it. So `#dup` method copying zero bytes into the new IO::Buffer raises the exception. Empty buffers, however, often appear in corner cases of usual operations (encrypting an empty string, encoding an empty list of items into binary, etc.) and it would be easy if such cases could be handled consistently. Other operations on NULL IO::Buffers are also useful but currently raising. ``` IO::Buffer.new(0) <=> IO::Buffer.new(1) IO::Buffer.new(0).each(:U8).to_a IO::Buffer.new(0).get_values([], 0) IO::Buffer.new(0).set_values([], 0, []) ``` I'm not sure this is a bug or by design, but at least I don't want cloning and comparison to raise. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:115338] [Ruby master Bug#19999] Backport: .travis.yml and fixed commits
by jaruga (Jun Aruga) 31 Jan '24

31 Jan '24

Issue #19999 has been reported by jaruga (Jun Aruga). ---------------------------------------- Bug #19999: Backport: .travis.yml and fixed commits https://bugs.ruby-lang.org/issues/19999 * Author: jaruga (Jun Aruga) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: REQUIRED ---------------------------------------- This is a backport suggestion to make Travis CI stable on the Ruby stable branches ruby_3_2 and etc. Travis CI for Ruby master branch became stable recently with arm64, ppc64le, s390x and arm32 cases. All the CI cases are running without `allow_failures` option. And more importantly, I simplified the `.travis.yml` to maintain it easily without sacrificing performance. So, I think it may be a good time to backport the Travis CI configuration file `.travis.yml` without some commits to fix some issues. https://app.travis-ci.com/github/ruby/ruby/builds/267166336 I can see ruby_3_2 and ruby_3_1 branches on the Travis CI page below. There are no ruby_3_0 and ruby_27 branches there. I am not sure how the branches are used. https://app.travis-ci.com/github/ruby/ruby/branches But it's beneficial to fix at least Travis CI for ruby_3_2 branch to save the Travis infra resource. Seeing the ruby_3_2 log, the ppc64le is already timeout after running maximum 50 minutes. https://app.travis-ci.com/github/ruby/ruby/builds/267156055 https://app.travis-ci.com/github/ruby/ruby/jobs/613043898#L2325 ``` [1/2] TestFiberQueue#test_pop_with_timeout_and_value = 0.00 s [2/2] TestFiberQueue#test_pop_with_timeout====[ 540 seconds still running ]==== ====[ 1080 seconds still running ]==== ====[ 1620 seconds still running ]==== ====[ 2160 seconds still running ]==== ``` First you can make the tests fail rather than stucking by the commit below. https://github.com/ruby/ruby/commit/3eaae72855b23158e2148566bb8a7667bfb395cb As this stucking issue happened with the combination of the `optflags=-O1` on ppc64le, I think you avoid the issue by porting the `.travis.yml` from the `master` branch where the `optflags=-O1` is not used in ppc64le. -- https://bugs.ruby-lang.org/

2 6

[ruby-core:114089] [Ruby master Misc#19758] Statically link ext/json
by MyCo (Maik Menz) 30 Jan '24

30 Jan '24

Issue #19758 has been reported by MyCo (Maik Menz). ---------------------------------------- Misc #19758: Statically link ext/json https://bugs.ruby-lang.org/issues/19758 * Author: MyCo (Maik Menz) * Status: Open * Priority: Normal ---------------------------------------- Hi, I'm building Ruby both as dynamic and static library with MSVC for a project. Everything appears to work fine, but now I'm trying to use the json ext, and it only works with the dynamically linked version. In the statically linked version it says it's missing json/pure but on closer inspection the reason it says that is because it can't find json/ext/parser (and probably also json/ext/generator) in the first place. I can see that both parser & generator created static libs in the build directory but they aren't linked into the ruby lib. With my limited knowledge of the building processes, my first attempt was to add both of those libs into `LOCAL_LIBS` and now they apear in the linking process. But this still doesn't change anything. It's still not finding those 2 libs in the statically linked Ruby build. What am I missing? What do I have to do, to get those linked into the static lib? Regards Maik -- https://bugs.ruby-lang.org/

4 6

[ruby-core:114997] [Ruby master Bug#19918] Should `a[&b]=c` be syntax valid?
by tompng (tomoya ishida) 19 Jan '24

19 Jan '24

Issue #19918 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #19918: Should `a[&b]=c` be syntax valid? https://bugs.ruby-lang.org/issues/19918 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-10-11T04:46:58Z master 40ab7b8c24) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- These codes are syntax valid now. Prism parses it as syntax error. ~~~ruby a[&b]=c a[&b]+=c a[&b]&&=c a[&b]||=c ~~~ Is this syntax intentional or should be error? Issue of Prism https://github.com/ruby/prism/issues/1636 It's added in test_parse.rb https://github.com/ruby/ruby/blob/40ab7b8c244de20007cb45846f41de3a01f7ea0c/… -- https://bugs.ruby-lang.org/

5 6

[ruby-core:111342] [Ruby master Bug#19246] Rebuilding the loaded feature index much slower in Ruby 3.1
by thomthom (Thomas Thomassen) 17 Jan '24

17 Jan '24

Issue #19246 has been reported by thomthom (Thomas Thomassen). ---------------------------------------- Bug #19246: Rebuilding the loaded feature index much slower in Ruby 3.1 https://bugs.ruby-lang.org/issues/19246 * Author: thomthom (Thomas Thomassen) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Some background to this issue: (This is a case that is unconventional usage of Ruby, but I hope you bear with me.) We ship the Ruby interpreter with our desktop applications for plugin support in our application (SketchUp). One feature we have had since, at least 2006 (maybe earlier-hard to track history beyond that) is that we had a custom alternate `require` method: `Sketchup.require`. This allows the users of our API to load encrypted Ruby files. This originally used `rb_provide` to add the path to the encrypted file into the list of loaded feature. However, somewhere between Ruby 2.2 and 2.5 there was some string optimisations made and the function `rb_provide` would not use a copy of the string passed to it. Instead it just held on to a pointer reference. In our case that string came from user-land, being passed in from `Sketchup.require` and would eventually be garbage collected and cause access violation crashes. To work around that we changed our custom `Sketchup.require` to push to `$LOADED_FEATURES` directly. There was a small penalty to the index being rebuilt after that, but it was negligible. Recently we tried to upgrade the Ruby interpreter in our application from 2.7 to 3.1 and found a major performance reduction when using our `Sketchup.require. As in, a plugin that would load in half a second would now spend 30 seconds. From https://bugs.ruby-lang.org/issues/18452 it sounds like there is _some_ expected extra penalty due to changes in how the index is built. But should it really be this much? Example minimal repro to simulate the issue: ``` # frozen_string_literal: true require 'benchmark' iterations = 200 foo_files = iterations.times.map { |i| "#{__dir__}/tmp/foo-#{i}.rb" } foo_files.each { |f| File.write(f, "") } bar_files = iterations.times.map { |i| "#{__dir__}/tmp/bar-#{i}.rb" } bar_files.each { |f| File.write(f, "") } biz_files = iterations.times.map { |i| "#{__dir__}/tmp/biz-#{i}.rb" } biz_files.each { |f| File.write(f, "") } Benchmark.bm do |x| x.report('normal') { foo_files.each { |file| require file } } x.report('loaded_features') { foo_files.each { |file| require file $LOADED_FEATURES << "#{file}-fake.rb" } } x.report('normal again') { biz_files.each { |file| require file } } end ``` ``` C:\Users\Thomas\SourceTree\ruby-perf>ruby27.bat ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x64-mingw32] C:\Users\Thomas\SourceTree\ruby-perf>ruby test-require.rb user system total real normal 0.000000 0.031000 0.031000 ( 0.078483) loaded_features 0.015000 0.000000 0.015000 ( 0.038759) normal again 0.016000 0.032000 0.048000 ( 0.076940) ``` ``` C:\Users\Thomas\SourceTree\ruby-perf>ruby30.bat ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x64-mingw32] C:\Users\Thomas\SourceTree\ruby-perf>ruby test-require.rb user system total real normal 0.000000 0.031000 0.031000 ( 0.074733) loaded_features 0.032000 0.000000 0.032000 ( 0.038898) normal again 0.000000 0.047000 0.047000 ( 0.076343) ``` ``` C:\Users\Thomas\SourceTree\ruby-perf>ruby31.bat ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x64-mingw-ucrt] C:\Users\Thomas\SourceTree\ruby-perf>ruby test-require.rb user system total real normal 0.016000 0.031000 0.047000 ( 0.132633) loaded_features 1.969000 11.500000 13.469000 ( 18.395761) normal again 0.031000 0.125000 0.156000 ( 0.249130) ``` Right now we're exploring options to deal with this. Because the performance degradation is a blocker for us upgrading. We also have 16 years of plugins created by third party developer that makes it impossible for us to drop this feature. Some options as-is, none of which are ideal: 1. We revert to using `rb_provide` but ensure the string passed in is not owned by Ruby, instead building a list of strings that we keep around for the duration of the application process. The problem is that some of our plugin developers have on occasion released plugins that will touch `$LOADED_FEATURES`, and if such a plugin is installed on a user machine it might cause the application to become unresponsive for minutes. The other non-ideal issue with using `rb_provide` is that we're also using that in ways it wasn't really intended (from that I understand). And it's not an official API? 2. We create a separate way for our `Sketchup.require` to keep track of it's loaded features, but then that would diverge even more from the behaviour of `require`. Replicating `require` functionality is not trivial and would be prone to subtle errors and possible diverge. It also doesn't address our issue that there is code out there in existing plugins that touches `$LOADED_FEATURES`. (And it's not something we can just ask people to clean up. From previous experience old versions stick around for a long time and is very hard to purge from circulation.) I have two questions for the Ruby mantainers: 1. Would it be reasonable to see an API for adding/removing/checking `$LOADED_FEATURE` that would allow for a more ideal implementation of a custom `require` functionality? 2. Is the performance difference in rebuilding the loaded feature index really expected to be as high as what we're seeing? An increase of nearly 100 times? Is there something there that might be addressed to make the rebuild to be less expensive against? (This would really help to address our challenges with third party plugins occasionally touching the global.) -- https://bugs.ruby-lang.org/

9 21

[ruby-core:115181] [Ruby master Bug#19975] ISeq#to_binary loses hidden local variable indices
by kddnewton (Kevin Newton) 12 Jan '24

12 Jan '24

Issue #19975 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #19975: ISeq#to_binary loses hidden local variable indices https://bugs.ruby-lang.org/issues/19975 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- When you call `RubyVM::InstructionSequence#to_a`, you get hidden local variables as the index in the stack from the top: ``` c if (rb_id2str(lid)) { rb_ary_push(locals, ID2SYM(lid)); } else { /* hidden variable from id_internal() */ rb_ary_push(locals, ULONG2NUM(iseq_body->local_table_size-i+1)); } ``` ``` rb RubyVM::InstructionSequence.compile("for foo in bar; end").to_a[13][4][2][10] # => [2] ``` When you call `RubyVM::InstructionSequence#to_binary`, it dumps hidden local variables as 0: ``` c if (id == 0 || rb_id2name(id) == NULL) { return 0; } return ibf_dump_object(dump, rb_id2sym(id)); ``` When it reads that back in and then you call `to_a`, you get `:#arg_rest`: ``` ruby RubyVM::InstructionSequence.load_from_binary(RubyVM::InstructionSequence.compile("for foo in bar; end").to_binary).to_a[13][4][2][10] # => [:"#arg_rest"] ``` This means you end up not being able to consistently look at the locals. Instead, when reading back in from binary it could replace it with the index so that it matches up with the value before it is dumped to binary. Could we do that so that `#to_a` is consistent no matter where the iseq came from? -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115212] [Ruby master Bug#19983] Nested * seems incorrect
by Eregon (Benoit Daloze) 11 Jan '24

11 Jan '24

Issue #19983 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Bug #19983: Nested * seems incorrect https://bugs.ruby-lang.org/issues/19983 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-10-30T09:27:06Z master 14fa5e39d7) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ``` $ ruby -v -e 'def m(*); ->(*) { p(*) }; end; m(1).call(2)' ruby 3.3.0dev (2023-10-30T09:27:06Z master 14fa5e39d7) [x86_64-linux] 1 ``` But I would expect `2`. Much like: ``` $ ruby -e 'def m(a); ->(a) { p(a) }; end; m(1).call(2)' ruby 3.3.0dev (2023-10-30T09:27:06Z master 14fa5e39d7) [x86_64-linux] 2 ``` i.e. the inner variable should win. -- https://bugs.ruby-lang.org/

4 8

[ruby-core:115491] [Ruby master Feature#20024] SyntaxError subclasses
by kddnewton (Kevin Newton) 10 Jan '24

10 Jan '24

Issue #20024 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Feature #20024: SyntaxError subclasses https://bugs.ruby-lang.org/issues/20024 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- There are many places around the Ruby ecosystem that handle syntax errors in different ways. Some provide highlighting, others provide recovery of some form, still more provide LSP metadata. In order to provide more rich information, most of them switch on the message of the error being returned, as in: https://github.com/ruby/irb/blob/f86d9dbe2fc05ed62332069a27f4aacc59ba9634/l… Within ruby/spec, specific error messages are required for these kinds of messages in order to support this implicit interface that syntax errors have a hidden type, which is only expressed through their message. For example: https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… https://github.com/ruby/spec/blob/c3206f644325c026fc5b700f0ea75ce9bd2e9d02/… It's not clear from these specs or from the parser itself which error messages are permanent/guaranteed versus which are changeable. Either way, relying on the error message itself as opposed to the type of the error is brittle at best. I would like to suggest instead we implement subclasses on `SyntaxError` that would allow tools that depend on specific syntax errors to rescue those subclasses instead of parsing the message. In addition to alleviating the need to parse error messages with regex, this would also allow for the possibility that the error messages could change in the future without breaking external tooling. Allowing these to change would allow them to be potentially enhanced or changed by other tools - for example by providing recovery information or translating them. This is particularly important for Prism since we are getting down to individual spec failures and some of the failures are related to the fact that we have messages like `"Numbered parameter is already used in outer scope"` where the spec requires `/numbered parameter is already used in/`. Even this case-sensitivity is causing failures, which seems like we're testing the wrong thing. -- https://bugs.ruby-lang.org/

9 16