February 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:111663] [Ruby master Bug#19313] Leaked file descriptors while running test-all
by Eregon (Benoit Daloze) 27 Feb '23

27 Feb '23

Issue #19313 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Bug #19313: Leaked file descriptors while running test-all https://bugs.ruby-lang.org/issues/19313 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-01-05T18:06:01Z master 54950a78e3) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ``` $ make test-all TESTOPTS="-j8" Run options: --seed=25187 "--ruby=./miniruby -I./lib -I. -I.ext/common ./tool/runruby.rb --extout=.ext -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/ -j8 # Running tests: Skipping Gem::PackageTask tests. rake not found.s_test_2 227226=test_mjit 227227=test_signal 227228=test_gem_comman Leaked file descriptor: TestFileUtils#test_chown_error: 11 #<File::Stat dev=0xfd00, ino=16777261, mode=0100664, nlink=1, uid=0, gid=0, rdev=0x0, size=9253600, blksize=4096, blocks=18080, atime=2023-01-05 18:16:02.96509764 +0100, mtime=2022-02-23 13:12:06.095215536 +0100, ctime=2022-02-23 13:12:06.095215536 +0100> Leaked file descriptor: TestFileUtils#test_chown_error: 12 #<File::Stat dev=0xfd00, ino=16777274, mode=0100664, nlink=1, uid=0, gid=0, rdev=0x0, size=6940392, blksize=4096, blocks=13560, atime=2023-01-05 18:16:04.425114636 +0100, mtime=2022-02-23 13:13:58.955685143 +0100, ctime=2022-02-23 13:13:58.955685143 +0100> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 227230 eregon 11r REG 253,0 9253600 16777261 /var/lib/sss/mc/passwd ruby 227230 eregon 12r REG 253,0 6940392 16777274 /var/lib/sss/mc/group Leaked file descriptor: Fiddle::TestFunc#test_qsort1: 12 #<File::Stat dev=0x1, ino=9463, mode=0100777, nlink=0, uid=1000, gid=1000, rdev=0x0, size=4096, blksize=4096, blocks=8, atime=2023-01-05 19:50:42.892803183 +0100, mtime=2023-01-05 19:50:42.892803183 +0100, ctime=2023-01-05 19:50:42.892803183 +0100> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 227225 eregon 12u REG 0,1 4096 9463 /memfd:libffi (deleted) Leaked file descriptor: TestFileExhaustive#test_expand_path_home_dir_string: 9 #<File::Stat dev=0xfd00, ino=16777261, mode=0100664, nlink=1, uid=0, gid=0, rdev=0x0, size=9253600, blksize=4096, blocks=18080, atime=2023-01-05 18:16:02.96509764 +0100, mtime=2022-02-23 13:12:06.095215536 +0100, ctime=2022-02-23 13:12:06.095215536 +0100> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 227226 eregon 9r REG 253,0 9253600 16777261 /var/lib/sss/mc/passwd Leaked file descriptor: TestDir#test_home: 12 #<File::Stat dev=0xfd00, ino=16777261, mode=0100664, nlink=1, uid=0, gid=0, rdev=0x0, size=9253600, blksize=4096, blocks=18080, atime=2023-01-05 18:16:02.96509764 +0100, mtime=2022-02-23 13:12:06.095215536 +0100, ctime=2022-02-23 13:12:06.095215536 +0100> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 227224 eregon 12r REG 253,0 9253600 16777261 /var/lib/sss/mc/passwd Leaked file descriptor: Fiddle::TestFunction#test_argument_count: 13 #<File::Stat dev=0x1, ino=13688, mode=0100777, nlink=0, uid=1000, gid=1000, rdev=0x0, size=4096, blksize=4096, blocks=8, atime=2023-01-05 19:50:49.882901778 +0100, mtime=2023-01-05 19:50:49.882901778 +0100, ctime=2023-01-05 19:50:49.882901778 +0100> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 227230 eregon 13u REG 0,1 4096 13688 /memfd:libffi (deleted) Leaked file descriptor: TestProcess#test_uid_from_name: 12 #<File::Stat dev=0xfd00, ino=16777261, mode=0100664, nlink=1, uid=0, gid=0, rdev=0x0, size=9253600, blksize=4096, blocks=18080, atime=2023-01-05 18:16:02.96509764 +0100, mtime=2022-02-23 13:12:06.095215536 +0100, ctime=2022-02-23 13:12:06.095215536 +0100> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ruby 227229 eregon 12r REG 253,0 9253600 16777261 /var/lib/sss/mc/passwd Retrying... Finished tests in 64.982686s, 360.4191 tests/s, 86076.9433 assertions/s. 23421 tests, 5593511 assertions, 0 failures, 0 errors, 156 skips ruby -v: ruby 3.3.0dev (2023-01-05T18:06:01Z master 54950a78e3) [x86_64-linux] ``` On Fedora 37. If this does not happen on Debian-based OS or macOS I guess it could be a glibc or maybe even a OS bug. This doesn't cause test failures but makes the output quite noisy. I have a few leaked descriptors for a while on Fedora when running test-all, also on previous Fedora versions. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:112624] [Ruby master Bug#17998] ractor: process hanging (with ractors initialized, but not being used)
by luke-gru (Luke Gruber) 27 Feb '23

27 Feb '23

Issue #17998 has been updated by luke-gru (Luke Gruber). Is this still an issue with current ruby @xtkoba? I checkout out your branch and with a few modifications for Ractor::IsolationErrors, it worked for me. It worked on: 3.3.0-dev and 3.2.0 (2022-12-25 revision a528908271) ---------------------------------------- Bug #17998: ractor: process hanging (with ractors initialized, but not being used) https://bugs.ruby-lang.org/issues/17998#change-102072 * Author: chucke (Tiago Cardoso) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * ruby -v: 3.0.1 * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- I couldn't figure out how to reproduce this in a more contained way, so I'll share what I'm doing [in this minitest branch](https://github.com/HoneyryderChuck/minitest/tree/issue-872). I'm trying to make minitest's parallel mode use ractors. If you look at the last commit of the branch, I'm: * replacing the parallel executor with a ractor-based one; * I'm defining the ractor executor, where I have a ractor pipe that a pool will consume work from * I'm turning off parallel subset of tests (to reproduce the bug that I'll be describing). When I run `rake test` in my Mac (BigSur 11.4), the process hangs. I can see that the ractor threads are executing and running, but the test process doesn't respond to the INFO signal interrupt (which should tell me where the process is hanging). This seems like a bug in the VM, as no work is being sent to the parallel executor, i.e. all ractors should be sleeping (I've `puts`'d also the executor shutdown process, and it never reaches it). If I replace the ractor-based executor back with the thread based executor, everything works as expected. ---Files-------------------------------- bug17998-0001.log (58.3 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112622] [Ruby master Bug#18572] Performance regression when invoking refined methods
by alanwu (Alan Wu) 27 Feb '23

27 Feb '23

Issue #18572 has been updated by alanwu (Alan Wu). Yes, the perf regression is unfortunate. The current per-call-site cache (`CC`) in the interpreter is keyed on `[recever_class, call_target_method_entry_validity]` -- the lexical scope that is used to resolve refinement definition for each call site is *not* cached. The way the system resolves calls to refinement definitions is through a special type of method entry (`VM_METHOD_TYPE_REFINED`) that triggers a secondary uncached lookup which takes into account the lexical scope of the call site. Any refinement definition for a module/class `M`, even ones that are never activated through `Module#using`, makes all calls to that method name in `M` do this uncached secondary lookup. This is the issue the OP showed in the `no-op {original,refined}` benchmark pair. The fact that there is a full heap walk during every call to `Module#using` is a separate issue. If you believe what I said about the secondary lookup always happening for refined method, and believe that the secondary lookup is always uncached, then this heap walk to invalidate all `CC`s should be unnecessary. This was what I noticed when I sent https://github.com/ruby/ruby/pull/4323 in 2021. The change itself is simple and we can merge it if we can get consensus that my assertions are correct. Reworking the cache system to fix the regression for the benchmark in the OP would be much more invovled. > Maybe we should deprecate refinements: my main concern about them is it's a > ton of complexity in Ruby implementations ... Their semantics definitely make the minimum required implementation complexity fairly high, and make cach design tricky. Even the current relatively simple cache design had bugs (e.g. #17806) when it first landed. I don't think they should be deprecated, though, since they've been around long enough that removing them would make many users unhappy. However, I would advise against introducing _any_ new additional complications to their semantics (e.g. #16461, #19277). We've already seen how this kind of complexity can make it more challenging for new developments like Ractors and YJIT. ---------------------------------------- Bug #18572: Performance regression when invoking refined methods https://bugs.ruby-lang.org/issues/18572#change-102070 * Author: palkan (Vladimir Dementyev) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Since Ruby 3.0, defining a refinement for a method slows down its execution even if we do not activate the refinement: ```ruby require "benchmark_driver" source = <<~RUBY class Hash def symbolize_keys transform_keys { |key| key.to_sym rescue key } end def refined_symbolize_keys transform_keys { |key| key.to_sym rescue key } end end module HashRefinements refine Hash do def refined_symbolize_keys raise "never called" end end end HASH = {foo: 1, bar: 2, baz: 3} class Foo def original end def refined end end module FooRefinements refine Foo do def refined raise "never called" end end end FOO = Foo.new RUBY Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "#symbolize_keys original", %{ HASH.symbolize_keys } x.report "#symbolize_keys refined", %{ HASH.refined_symbolize_keys } end Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "no-op original", %{ FOO.original } x.report "no-op refined", %{ FOO.refined } end ``` The results for Ruby 3.1: ```sh ... Comparison: #symbolize_keys original: 2372420.1 i/s #symbolize_keys refined: 1941019.0 i/s - 1.22x slower ... Comparison: no-op original: 51790974.2 i/s no-op refined: 14456518.9 i/s - 3.58x slower ``` For Ruby 2.6 and 2.7: ```sh Comparison: #symbolize_keys original: 2278339.7 i/s #symbolize_keys refined: 2264153.1 i/s - 1.01x slower ... Comparison: no-op refined: 64178338.5 i/s no-op original: 63357980.1 i/s - 1.01x slower ``` You can find the full code and more results in this [gist](https://gist.github.com/palkan/637dc83edd86d70b5dbf72f2a4d702e5). P.S. The problem was originally noticed by @byroot, see https://github.com/ruby-i18n/i18n/pull/573 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112621] [Ruby master Bug#18731] Parallel test-all sometimes does not run at all some tests
by luke-gru (Luke Gruber) 27 Feb '23

27 Feb '23

Issue #18731 has been updated by luke-gru (Luke Gruber). I've been working a bit on the parallel runner and I've noticed this problem too. ``` Raise if the check Test::Unit::TestCase.test_suites-suites (which is the bug) returns an empty Array. It won't catch all issues though when e.g. 2nd test class is defined in the same file. ``` Unfortunately we can't do this because some test classes are guarded by if statements, and the array can be empty in those cases too. I'll take a look at the `method_added` solution, incrementing a number on the class and checking it matches the previous one. ---------------------------------------- Bug #18731: Parallel test-all sometimes does not run at all some tests https://bugs.ruby-lang.org/issues/18731#change-102069 * Author: Eregon (Benoit Daloze) * Status: Closed * Priority: Normal * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- In TruffleRuby I've noticed that some CRuby tests sometimes run or not, non-deterministically. The TruffleRuby CI currently runs CRuby tests with `-j4`. Today I investigated and I think I've found the reason for it. One occurrence of this bug is for: * https://github.com/ruby/ruby/blob/master/test/ruby/test_method.rb * https://github.com/ruby/ruby/blob/master/test/ruby/test_inlinecache.rb Both define a test class `TestMethod`. The parallel runner distributes files across processes. If the same worker process gets both files, then all tests in the second file are not run at all! (the file is still loaded). The reason seems this line: https://github.com/ruby/ruby/blob/8751c5c2672d1391c73d9dec590063d27bed7e4c/… (it's also possible to reproduce with just these 2 files, `-j2` and not the full test suite, and having one worker start very slowly so the same worker gets both files) `Test::Unit::TestCase.test_suites` is an array of classes, and so here because `TestMethod` was already defined by the first file loaded, the result of `Test::Unit::TestCase.test_suites-suites` is empty, and nothing gets run. This doesn't seem an issue when not running in parallel/using `-j`. But that's rare/impractical because test-all is very slow without `-j`. For this specific case it seems we should rename the class in `test_inlinecache.rb`, but I suspect there are more name conflicts and the parallel runner should be fixed to handle this. --- <rant> FWIW, this test/unit code seems pretty messy, very long lines, [duplication](https://github.com/ruby/ruby/blob/8751c5c2672d1391c73d9dec5900…, hard to follow with the mix of minitest/test-unit, etc. I don't understand how this code is any better than `mspec`. It seems far more complex and hacky. And that's probably why Ruby implementations except CRuby run MRI tests only when they have no choice, it's so annoying to work with, so many unnecessary subprocesses (very slow to run, annoying to debug), many CRuby-specific tests, so many unreliable tests, many metaprogramming-defined tests which are very brittle, lots of global state and coupling, etc. </rant> -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112619] [Ruby master Bug#18119] Ractor crashes when instantiating classes
by luke-gru (Luke Gruber) 27 Feb '23

27 Feb '23

Issue #18119 has been updated by luke-gru (Luke Gruber). I can't reproduce this on 3.2.0 or 3.3.0-dev (Ubuntu 22.04 x86-64), even with GC.stress = true ---------------------------------------- Bug #18119: Ractor crashes when instantiating classes https://bugs.ruby-lang.org/issues/18119#change-102067 * Author: peterzhu2118 (Peter Zhu) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * ruby -v: ruby 3.1.0dev (2021-08-19T08:44:48Z master 6963f8f743) [x86_64-linux] * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN ---------------------------------------- The following script crashes with a segfault (tested on Ubuntu 20.04 and macOS 11.5.2): ```ruby workers = (0...8).map do Ractor.new do loop do 100.times.map { Class.new } Ractor.yield nil end end end 100.times { Ractor.select(*workers) } ``` Crash error: ``` <internal:ractor>:267: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues. test.rb:4: [BUG] Segmentation fault at 0x0000000000000040 ruby 3.1.0dev (2021-08-19T08:44:48Z master 6963f8f743) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0010 p:---- s:0033 e:000032 CFUNC :new c:0009 p:0011 s:0029 e:000028 BLOCK test.rb:4 [FINISH] c:0008 p:---- s:0026 e:000025 IFUNC c:0007 p:---- s:0023 e:000022 CFUNC :times c:0006 p:---- s:0020 e:000019 CFUNC :each c:0005 p:---- s:0017 e:000016 CFUNC :map c:0004 p:0007 s:0013 e:000012 BLOCK test.rb:4 [FINISH] c:0003 p:---- s:0010 e:000009 CFUNC :loop c:0002 p:0004 s:0006 e:000005 BLOCK test.rb:3 [FINISH] c:0001 p:---- s:0003 e:000002 (none) [FINISH] -- Ruby level backtrace information ---------------------------------------- test.rb:3:in `block (2 levels) in <main>' test.rb:3:in `loop' test.rb:4:in `block (3 levels) in <main>' test.rb:4:in `map' test.rb:4:in `each' test.rb:4:in `times' test.rb:4:in `block (4 levels) in <main>' test.rb:4:in `new' -- Machine register context ------------------------------------------------ RIP: 0x0000562c1f9cd2cb RBP: 0x00007f6c3736d378 RSP: 0x00007f6c368285f0 RAX: 0x00007f6c1c00e208 RBX: 0x00007f6c3736d378 RCX: 0x0000562c20ed8330 RDX: 0x0000000000000000 RDI: 0x00007f6c100095c0 RSI: 0x0000000000000000 R8: 0x0000000000000007 R9: 0x0000562c20ed8120 R10: 0x0000000000000022 R11: 0x0000562c21180760 R12: 0x0000000000000000 R13: 0x00007f6c3736c000 R14: 0x0000000000000000 R15: 0x00007f6c3736d378 EFL: 0x0000000000010202 -- C level backtrace information ------------------------------------------- /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_print_backtrace+0x11) [0x562c1f995e38] ../vm_dump.c:759 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_vm_bugreport) ../vm_dump.c:1041 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_bug_for_fatal_signal+0xec) [0x562c1f78a0bc] ../error.c:815 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(sigsegv+0x4d) [0x562c1f8ebcbd] ../signal.c:961 /lib/x86_64-linux-gnu/libpthread.so.0(__restore_rt+0x0) [0x7f6c3b2c63c0] /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_class_remove_from_super_subclasses+0x2b) [0x562c1f9cd2cb] ../class.c:99 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(obj_free+0x37a) [0x562c1f7ae95a] ../gc.c:3123 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_plane_sweep+0x21) [0x562c1f7aef3d] ../gc.c:5322 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_page_sweep) ../gc.c:5464 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_sweep_step) ../gc.c:5630 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_heap_prepare_minimum_pages+0x0) [0x562c1f7afd94] ../gc.c:5834 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_sweep) ../gc.c:5837 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_marks+0x1c0) [0x562c1f7b3df8] ../gc.c:8144 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(gc_start) ../gc.c:9013 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(heap_prepare+0x2f) [0x562c1f7b8b6f] ../gc.c:2131 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(heap_next_freepage) ../gc.c:2422 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(ractor_cache_slots) ../gc.c:2454 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(newobj_slowpath) ../gc.c:2495 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(newobj_slowpath_wb_protected) ../gc.c:2519 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(newobj_of0+0x5) [0x562c1f7b8ebd] ../gc.c:2562 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(newobj_of) ../gc.c:2572 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_wb_protected_newobj_of) ../gc.c:2596 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(class_alloc+0x5) [0x562c1f9cd49e] ../class.c:185 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_class_boot) ../class.c:230 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(class_call_alloc_func+0x5) [0x562c1f84e5d3] ../object.c:2075 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_class_alloc) ../object.c:2047 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_class_new_instance_pass_kw) ../object.c:2120 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_cfp_consistent_p+0x0) [0x562c1f96d6bc] ../vm_insnhelper.c:2989 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call_cfunc_with_frame) ../vm_insnhelper.c:2991 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_sendish+0x303) [0x562c1f978393] ../vm_insnhelper.c:4562 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_exec_core+0xcd) [0x562c1f98316d] ../insns.def:775 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_vm_exec+0x197) [0x562c1f978fc7] ../vm.c:2164 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(collect_i+0x12) [0x562c1fa27bf2] ../enum.c:608 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_vm_pop_frame+0x0) [0x562c1f976ba8] ../vm_insnhelper.c:3795 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_yield_with_cfunc) ../vm_insnhelper.c:3796 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(invoke_block_from_c_bh+0x10) [0x562c1f97d0d3] ../vm.c:1359 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_yield) ../vm.c:1399 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_yield_0) ../vm_eval.c:1350 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_yield_1) ../vm_eval.c:1356 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(int_dotimes+0x5c) [0x562c1f83a49c] ../numeric.c:5014 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_cfp_consistent_p+0x0) [0x562c1f97dd4f] ../vm_eval.c:135 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call0_cfunc_with_frame) ../vm_eval.c:137 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call0_cfunc) ../vm_eval.c:149 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call0_body) ../vm_eval.c:182 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_call0+0x1ea) [0x562c1f9812fa] ../vm_eval.c:72 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(iterate_method+0x3b) [0x562c1f981e9b] ../vm_eval.c:847 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_iterate0+0x101) [0x562c1f973001] ../vm_eval.c:1534 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_block_call_kw+0x76) [0x562c1f9731f6] ../vm_eval.c:1566 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(enumerator_block_call+0x59) [0x562c1fa358e9] ../enumerator.c:553 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_cfp_consistent_p+0x0) [0x562c1f97dd4f] ../vm_eval.c:135 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call0_cfunc_with_frame) ../vm_eval.c:137 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call0_cfunc) ../vm_eval.c:149 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call0_body) ../vm_eval.c:182 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_call0+0x1ea) [0x562c1f9812fa] ../vm_eval.c:72 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(iterate_method+0x3b) [0x562c1f981e9b] ../vm_eval.c:847 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_iterate0+0x101) [0x562c1f973001] ../vm_eval.c:1534 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_lambda_call+0x75) [0x562c1f973295] ../vm_eval.c:1633 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(enum_collect+0x5b) [0x562c1fa29acb] ../enum.c:647 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_cfp_consistent_p+0x0) [0x562c1f96d6bc] ../vm_insnhelper.c:2989 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call_cfunc_with_frame) ../vm_insnhelper.c:2991 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_sendish+0x303) [0x562c1f978393] ../vm_insnhelper.c:4562 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_exec_core+0x130) [0x562c1f9831d0] ../insns.def:756 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_vm_exec+0x197) [0x562c1f978fc7] ../vm.c:2164 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(invoke_block_from_c_bh+0x130) [0x562c1f97c85a] ../vm.c:1264 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_yield) ../vm.c:1399 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_yield_0) ../vm_eval.c:1350 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(loop_i) ../vm_eval.c:1449 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_vrescue2+0x114) [0x562c1f794694] ../eval.c:1023 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_rescue2+0x8e) [0x562c1f79490e] ../eval.c:1000 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_cfp_consistent_p+0x0) [0x562c1f96d6bc] ../vm_insnhelper.c:2989 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_call_cfunc_with_frame) ../vm_insnhelper.c:2991 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_sendish+0x303) [0x562c1f978393] ../vm_insnhelper.c:4562 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(vm_exec_core+0x130) [0x562c1f9831d0] ../insns.def:756 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_vm_exec+0x197) [0x562c1f978fc7] ../vm.c:2164 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(thread_do_start_proc+0x294) [0x562c1f930f24] ../thread.c:716 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(thread_do_start+0xc) [0x562c1f9336fc] ../thread.c:760 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(thread_start_func_2) ../thread.c:835 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(rb_native_cond_initialize+0x0) [0x562c1f933a09] ../thread_pthread.c:1051 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(register_cached_thread_and_wait) ../thread_pthread.c:1103 /home/spin/src/github.com/Shopify/ruby-master/install/bin/ruby(thread_start_func_1) ../thread_pthread.c:1058 /lib/x86_64-linux-gnu/libpthread.so.0(0x9609) [0x7f6c3b2ba609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f6c3b044293] ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112618] [Ruby master Bug#18572] Performance regression when invoking refined methods
by Eregon (Benoit Daloze) 27 Feb '23

27 Feb '23

Issue #18572 has been updated by Eregon (Benoit Daloze). Potential fix: https://github.com/ruby/ruby/pull/4323 ---------------------------------------- Bug #18572: Performance regression when invoking refined methods https://bugs.ruby-lang.org/issues/18572#change-102065 * Author: palkan (Vladimir Dementyev) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Since Ruby 3.0, defining a refinement for a method slows down its execution even if we do not activate the refinement: ```ruby require "benchmark_driver" source = <<~RUBY class Hash def symbolize_keys transform_keys { |key| key.to_sym rescue key } end def refined_symbolize_keys transform_keys { |key| key.to_sym rescue key } end end module HashRefinements refine Hash do def refined_symbolize_keys raise "never called" end end end HASH = {foo: 1, bar: 2, baz: 3} class Foo def original end def refined end end module FooRefinements refine Foo do def refined raise "never called" end end end FOO = Foo.new RUBY Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "#symbolize_keys original", %{ HASH.symbolize_keys } x.report "#symbolize_keys refined", %{ HASH.refined_symbolize_keys } end Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "no-op original", %{ FOO.original } x.report "no-op refined", %{ FOO.refined } end ``` The results for Ruby 3.1: ```sh ... Comparison: #symbolize_keys original: 2372420.1 i/s #symbolize_keys refined: 1941019.0 i/s - 1.22x slower ... Comparison: no-op original: 51790974.2 i/s no-op refined: 14456518.9 i/s - 3.58x slower ``` For Ruby 2.6 and 2.7: ```sh Comparison: #symbolize_keys original: 2278339.7 i/s #symbolize_keys refined: 2264153.1 i/s - 1.01x slower ... Comparison: no-op refined: 64178338.5 i/s no-op original: 63357980.1 i/s - 1.01x slower ``` You can find the full code and more results in this [gist](https://gist.github.com/palkan/637dc83edd86d70b5dbf72f2a4d702e5). P.S. The problem was originally noticed by @byroot, see https://github.com/ruby-i18n/i18n/pull/573 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112606] [Ruby master Bug#18572] Performance regression when invoking refined methods
by Eregon (Benoit Daloze) 25 Feb '23

25 Feb '23

Issue #18572 has been updated by Eregon (Benoit Daloze). palkan (Vladimir Dementyev) wrote in #note-4: > As far as I understand, this line is responsible for "the 13 seconds" boot time overhead: https://github.com/ruby/ruby/blob/master/eval.c#L1342 (I was able to achieve similar results by adding tons of `using` to the source code). The line was added in the same commit as mentioned above, so the root cause is the same for sure. Oh yeah, good find, clearing all method lookup caches on every call to `using` with `ObjectSpace.each_object` is terrible for performance. It likely throws away all JITed code too. A bit like `extend` used to blow up every method lookup inline cache a long time ago. I guess we need to wait for @ko1 to fix that. This is probably fixable but might require some changes to the overall method lookup design, which is then quite involved (from looking at similar things in TruffleRuby). My concern above is mainly the whole complexity of refinements in Ruby implementations and some parts of the semantics are/were unsound (e.g. `super` in refinements can cause infinite loops, this is being addressed by preventing `include`/`prepend` in a refinement and replacing by `import_methods`, a welcome simplification). ---------------------------------------- Bug #18572: Performance regression when invoking refined methods https://bugs.ruby-lang.org/issues/18572#change-102053 * Author: palkan (Vladimir Dementyev) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Since Ruby 3.0, defining a refinement for a method slows down its execution even if we do not activate the refinement: ```ruby require "benchmark_driver" source = <<~RUBY class Hash def symbolize_keys transform_keys { |key| key.to_sym rescue key } end def refined_symbolize_keys transform_keys { |key| key.to_sym rescue key } end end module HashRefinements refine Hash do def refined_symbolize_keys raise "never called" end end end HASH = {foo: 1, bar: 2, baz: 3} class Foo def original end def refined end end module FooRefinements refine Foo do def refined raise "never called" end end end FOO = Foo.new RUBY Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "#symbolize_keys original", %{ HASH.symbolize_keys } x.report "#symbolize_keys refined", %{ HASH.refined_symbolize_keys } end Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "no-op original", %{ FOO.original } x.report "no-op refined", %{ FOO.refined } end ``` The results for Ruby 3.1: ```sh ... Comparison: #symbolize_keys original: 2372420.1 i/s #symbolize_keys refined: 1941019.0 i/s - 1.22x slower ... Comparison: no-op original: 51790974.2 i/s no-op refined: 14456518.9 i/s - 3.58x slower ``` For Ruby 2.6 and 2.7: ```sh Comparison: #symbolize_keys original: 2278339.7 i/s #symbolize_keys refined: 2264153.1 i/s - 1.01x slower ... Comparison: no-op refined: 64178338.5 i/s no-op original: 63357980.1 i/s - 1.01x slower ``` You can find the full code and more results in this [gist](https://gist.github.com/palkan/637dc83edd86d70b5dbf72f2a4d702e5). P.S. The problem was originally noticed by @byroot, see https://github.com/ruby-i18n/i18n/pull/573 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:112585] [Ruby master Feature#19466] Class.new takes a block, why doesn't Module.new take a block?
by ioquatix (Samuel Williams) 25 Feb '23

25 Feb '23

Issue #19466 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19466: Class.new takes a block, why doesn't Module.new take a block? https://bugs.ruby-lang.org/issues/19466 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- ```ruby Class.new do #... equivalent to class_eval end ``` So, why don't we introduce: ```ruby Module.new do #... equivalent to class_eval end ``` -- https://bugs.ruby-lang.org/

2 3

[ruby-core:112517] [Ruby master Feature#19453] Move `Fiber.current` into core.
by ioquatix (Samuel Williams) 24 Feb '23

24 Feb '23

Issue #19453 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19453: Move `Fiber.current` into core. https://bugs.ruby-lang.org/issues/19453 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ioquatix (Samuel Williams) ---------------------------------------- i.e. don't `require 'fiber'` to use `Fiber.current`. Are there other methods we should consider too? -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112579] [Ruby master Bug#18572] Performance regression when invoking refined methods
by palkan (Vladimir Dementyev) 24 Feb '23

24 Feb '23

Issue #18572 has been updated by palkan (Vladimir Dementyev). Eregon (Benoit Daloze) wrote in #note-3: > Interesting, maybe this is partly the cause for the `13 seconds` mentioned in https://shopify.engineering/the-case-against-monkey-patching. As far as I understand, this line is responsible for "the 13 seconds" boot time overhead: https://github.com/ruby/ruby/blob/master/eval.c#L1342 (I was able to achieve similar results by adding tons of `using` to the source code). The line was added in the same commit as mentioned above, so the root cause is the same for sure. > Maybe we should deprecate refinements: my main concern about them is it's a ton of complexity in Ruby implementations (in method lookup and even worse in super method lookup) and it's not really used much or gains much, so it does not seem worth it. It does make every uncached/cache-filling method lookup slower. > It shouldn't make cached method lookup slower though when no refinements are active in that scope, at least it does not currently in TruffleRuby. But that wasn't the case in Ruby <3; refined methods were fast (b/c cached), and had no effect on the original method calls, like today. I believe, something just got lost along the introduction of disposable call-caches. The problem with "it's not really used much or gains much" is that it took many years for refinements to reach some sort of stability (2.6 and 2.7 releases were pretty good) and finally developers were convinced not to be afraid of the "experimentality" of the feature, and with Ruby 3 we made a huge step back. Especially now, when Shopify engineers claim that "refinements are really slow!" (which is only partially true), we, for sure, should expect the community to turn backs on refinements, again. It's a two-edged sword 🤷‍♂️ ---------------------------------------- Bug #18572: Performance regression when invoking refined methods https://bugs.ruby-lang.org/issues/18572#change-102028 * Author: palkan (Vladimir Dementyev) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Since Ruby 3.0, defining a refinement for a method slows down its execution even if we do not activate the refinement: ```ruby require "benchmark_driver" source = <<~RUBY class Hash def symbolize_keys transform_keys { |key| key.to_sym rescue key } end def refined_symbolize_keys transform_keys { |key| key.to_sym rescue key } end end module HashRefinements refine Hash do def refined_symbolize_keys raise "never called" end end end HASH = {foo: 1, bar: 2, baz: 3} class Foo def original end def refined end end module FooRefinements refine Foo do def refined raise "never called" end end end FOO = Foo.new RUBY Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "#symbolize_keys original", %{ HASH.symbolize_keys } x.report "#symbolize_keys refined", %{ HASH.refined_symbolize_keys } end Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "no-op original", %{ FOO.original } x.report "no-op refined", %{ FOO.refined } end ``` The results for Ruby 3.1: ```sh ... Comparison: #symbolize_keys original: 2372420.1 i/s #symbolize_keys refined: 1941019.0 i/s - 1.22x slower ... Comparison: no-op original: 51790974.2 i/s no-op refined: 14456518.9 i/s - 3.58x slower ``` For Ruby 2.6 and 2.7: ```sh Comparison: #symbolize_keys original: 2278339.7 i/s #symbolize_keys refined: 2264153.1 i/s - 1.01x slower ... Comparison: no-op refined: 64178338.5 i/s no-op original: 63357980.1 i/s - 1.01x slower ``` You can find the full code and more results in this [gist](https://gist.github.com/palkan/637dc83edd86d70b5dbf72f2a4d702e5). P.S. The problem was originally noticed by @byroot, see https://github.com/ruby-i18n/i18n/pull/573 -- https://bugs.ruby-lang.org/

1 0