June 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117781] [Ruby master Bug#20472] Ractor support for ** into hash literal
by kddnewton (Kevin Newton) 30 Sep '24

30 Sep '24

Issue #20472 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20472: Ractor support for ** into hash literal https://bugs.ruby-lang.org/issues/20472 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby # shareable_constant_value: experimental_copy a = { "foo" => "bar" } A = { **a } ``` In this example, Ruby fails an assertion and crashes. I think we should either provide support for this, or we should explicitly provide a syntax error indicating that ** isn't allowed in constant writes when ractor pragmas are enabled. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:117161] [Ruby master Bug#20337] Complex#inspect mutates the string returned by `real.inspect`
by Eregon (Benoit Daloze) 21 Sep '24

21 Sep '24

Issue #20337 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Bug #20337: Complex#inspect mutates the string returned by `real.inspect` https://bugs.ruby-lang.org/issues/20337 * Author: Eregon (Benoit Daloze) * Status: Open * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ``` irb(main):001:0> n = Numeric.new => #<Numeric:0x00007f81b2308578> irb(main):004:0> class Numeric; def inspect = super.freeze; end => :inspect irb(main):006:0> Complex(n, 1).inspect (irb):6:in `inspect': can't modify frozen String: "#<Numeric:0x00007f81b2308578>" (FrozenError) from (irb):6:in `<main>' from /home/eregon/.rubies/ruby-3.2.2/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>' from /home/eregon/.rubies/ruby-3.2.2/bin/irb:25:in `load' from /home/eregon/.rubies/ruby-3.2.2/bin/irb:25:in `<main>' ``` It feels wrong to mutate the result of inspect at least in general, for instance `true.inspect` is frozen. Discovered by https://github.com/ruby/spec/pull/1142 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:116799] [Ruby master Bug#20271] Issue with moving embedded string across ractors
by luke-gru (Luke Gruber) 20 Sep '24

20 Sep '24

Issue #20271 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #20271: Issue with moving embedded string across ractors https://bugs.ruby-lang.org/issues/20271 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When embedded strings are moved, the embedded bytes aren't copied over to the new embedded string. This shows itself when an embedded string has more than 16 characters, and can also be a problem when the MovedObject has been garbage collected and another object has taken its place in the ruby heap. For example: ```ruby r = Ractor.new { foo = receive puts foo foo } obj = "12345678" * 3 puts obj r.send(obj, move: true) r.take ``` ``` 123456781234567812345678 1234567812345678 ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:116779] [Ruby master Bug#20267] Hashes that use ar_table aren't moved properly across ractors
by luke-gru (Luke Gruber) 20 Sep '24

20 Sep '24

Issue #20267 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #20267: Hashes that use ar_table aren't moved properly across ractors https://bugs.ruby-lang.org/issues/20267 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This crashes in debug build and gives weird results in non debug build, because the ar_table values aren't copied over to the new "embedded" hash during a move across ractors. ```ruby r = Ractor.new { hash = receive puts hash } obj = { "1" => 1, "2" => 2, "3" => 3, } puts obj r.send(obj, move: true) r.take ``` Crash log: ``` /home/lukeg/workspace/ruby-build/miniruby(sigsegv+0x4d) [0x5641e67d105d] ../ruby/signal.c:926 /lib/x86_64-linux-gnu/libc.so.6(0x7fb4baa42520) [0x7fb4baa42520] /home/lukeg/workspace/ruby-build/miniruby(RB_BUILTIN_TYPE+0x0) [0x5641e6778db0] ../ruby/ractor.c:3128 /home/lukeg/workspace/ruby-build/miniruby(rbimpl_RB_TYPE_P_fastpath) ../ruby/include/ruby/internal/value_type.h:351 /home/lukeg/workspace/ruby-build/miniruby(RB_TYPE_P) ../ruby/include/ruby/internal/value_type.h:378 /home/lukeg/workspace/ruby-build/miniruby(RB_FL_ABLE) ../ruby/include/ruby/internal/fl_type.h:449 /home/lukeg/workspace/ruby-build/miniruby(RB_FL_TEST_RAW) ../ruby/include/ruby/internal/fl_type.h:471 /home/lukeg/workspace/ruby-build/miniruby(rb_ractor_shareable_p) ../ruby/include/ruby/ractor.h:256 /home/lukeg/workspace/ruby-build/miniruby(reset_belonging_enter) ../ruby/ractor.c:3121 ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:113052] [Ruby master Bug#19562] Setting default_(in|ex)ternal encoding after running an Ractor freezes ruby
by larskanis (Lars Kanis) 20 Sep '24

20 Sep '24

Issue #19562 has been reported by larskanis (Lars Kanis). ---------------------------------------- Bug #19562: Setting default_(in|ex)ternal encoding after running an Ractor freezes ruby https://bugs.ruby-lang.org/issues/19562 * Author: larskanis (Lars Kanis) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following command blocks infinitely and must be killed with "kill -9" ``` ruby -e "Ractor.new{}.take; Encoding.default_external = Encoding::ISO8859_2" ``` Backtrace is: ``` #0 futex_wait (private=0, expected=2, futex_word=0x55555555abd0) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x55555555abd0, private=0) at ./nptl/lowlevellock.c:49 #2 0x00007ffff760a082 in lll_mutex_lock_optimized (mutex=0x55555555abd0) at ./nptl/pthread_mutex_lock.c:48 #3 ___pthread_mutex_lock (mutex=mutex@entry=0x55555555abd0) at ./nptl/pthread_mutex_lock.c:93 #4 0x00007ffff7c5a40d in rb_native_mutex_lock (lock=lock@entry=0x55555555abd0) at ruby-3.2.1/thread_pthread.c:556 #5 0x00007ffff7cc5376 in vm_lock_enter (cr=0x55555555deb0, vm=0x55555555aba0, locked=locked@entry=false, no_barrier=no_barrier@entry=false, lev=0x55555555ac00) at vm_sync.c:56 #6 0x00007ffff7cc5747 in rb_vm_lock_body () at vm_sync.c:175 #7 0x00007ffff7bc02c2 in rb_vm_lock (line=1829, file=0x7ffff7e209cb "ractor.c") at ruby-3.2.1/vm_sync.h:57 #8 ractor_check_blocking (remained_thread_cnt=<optimized out>, file=0x7ffff7e209cb "ractor.c", line=1865, cr=0x55555555deb0) at ractor.c:1829 #9 ractor_check_blocking (line=1865, file=0x7ffff7e209cb "ractor.c", remained_thread_cnt=<optimized out>, cr=0x55555555deb0) at ractor.c:1811 #10 rb_ractor_blocking_threads_inc (cr=0x55555555deb0, file=file@entry=0x7ffff7e29b04 "thread.c", line=line@entry=1488) at ractor.c:1865 #11 0x00007ffff7c60977 in blocking_region_begin (th=0x55555555d070, region=0x7fffffffd220, ubf=0x7ffff7c5b300 <ubf_select>, arg=0x55555555d070, fail_if_interrupted=<optimized out>) at thread.c:1488 #12 0x00007ffff7c61edf in rb_nogvl (func=0x7ffff7ac66e0 <no_gvl_stat>, data1=0x7fffffffd2a0, ubf=<optimized out>, data2=0x55555555d070, flags=<optimized out>) at thread.c:1548 #13 0x00007ffff7acdfbc in stat_without_gvl (st=0x7fffffffd2b0, path=0x7ffff27dfe88 "ruby-3.2.1/lib/ruby/3.2.0/x86_64-linux/enc/iso_8859_2.so") at file.c:1171 #14 rb_check_realpath_internal (basedir=basedir@entry=4, path=path@entry=140737261984200, origenc=0x555555565030, mode=mode@entry=RB_REALPATH_STRICT) at file.c:4491 #15 0x00007ffff7acebaa in rb_realpath_internal (basedir=basedir@entry=4, path=path@entry=140737261984200, strict=strict@entry=1) at file.c:4536 #16 0x00007ffff7b2cdd4 in require_internal (ec=0x55555555e1f0, fname=<optimized out>, exception=1, warn=<optimized out>) at load.c:1195 #17 0x00007ffff7aa0a17 in load_encoding (name=0x5555556049b0 "ISO-8859-2") at encoding.c:779 #18 0x00007ffff7aa2ef8 in check_encoding (enc=0x5555556049d0) at encoding.c:191 #19 enc_check_encoding (obj=140737335533800) at encoding.c:202 #20 rb_to_encoding (enc=enc@entry=140737335533800) at encoding.c:307 #21 0x00007ffff7aa4049 in enc_set_default_encoding (name=0x7ffff7de6287 "external", encoding=140737335533800, def=0x7ffff7fb00c0 <default_external>) at encoding.c:1563 #22 rb_enc_set_default_external (encoding=140737335533800) at encoding.c:1637 #23 0x00007ffff7aa411f in set_default_external (klass=<optimized out>, encoding=140737335533800) at encoding.c:1658 #24 0x00007ffff7c99037 in vm_call_cfunc_with_frame (ec=0x55555555e1f0, reg_cfp=0x7ffff6f7df90, calling=<optimized out>) at ruby-3.2.1/vm_insnhelper.c:3268 #25 0x00007ffff7ca9564 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>) at ruby-3.2.1/vm_callinfo.h:367 #26 vm_exec_core (ec=0x55555555e1f0, initial=128) at ruby-3.2.1/insns.def:820 #27 0x00007ffff7caee93 in rb_vm_exec (ec=0x55555555e1f0, jit_enable_p=true) at vm.c:2374 #28 0x00007ffff7abfbf1 in rb_ec_exec_node (ec=ec@entry=0x55555555e1f0, n=n@entry=0x7ffff796c2c0) at eval.c:289 #29 0x00007ffff7ac60b3 in ruby_run_node (n=0x7ffff796c2c0) at eval.c:330 #30 0x0000555555555187 in rb_main (argv=0x7fffffffdb98, argc=3) at ./main.c:38 #31 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57 ``` When the encoding was already loaded before running `Ractor.new`, then it works and doesn't block: ``` ruby -e "''.encode(Encoding::ISO8859_2); Ractor.new{}.take; Encoding.default_internal = Encoding::ISO8859_2" ``` When the encoding is not assigned as default encoding, but used otherwise, it works too: ``` ruby -e "Ractor.new{}.take; ''.encode(Encoding::ISO8859_2)" ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:111144] [Ruby master Feature#19171] Update Unicode data to Unicode Version 15.1
by duerst 20 Sep '24

20 Sep '24

Issue #19171 has been reported by duerst (Martin Dürst). ---------------------------------------- Feature #19171: Update Unicode data to Unicode Version 15.1 https://bugs.ruby-lang.org/issues/19171 * Author: duerst (Martin Dürst) * Status: Open * Priority: Normal * Assignee: duerst (Martin Dürst) ---------------------------------------- According to http://blog.unicode.org/2022/11/the-unicode-standard-2023-release.html, Unicode plans to release Version 15.1 in September 2023. According to https://www.unicode.org/versions/beta.html, public alpha review starts Feb. 7, 2023, and ends April 4, 2023. Because alpha review may not include all the files we use, it may be difficult for us to participate. Public beta review is planned to start May 23, 2023, ending July 4, 2023. At this point, we should be able to test things. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116581] [Ruby master Bug#20237] Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread
by hanazuki (Kasumi Hanazuki) 16 Sep '24

16 Sep '24

Issue #20237 has been reported by hanazuki (Kasumi Hanazuki). ---------------------------------------- Bug #20237: Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread https://bugs.ruby-lang.org/issues/20237 * Author: hanazuki (Kasumi Hanazuki) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Backgrounds [unshare(2)](https://man7.org/linux/man-pages/man2/unshare.2.html) is a syscall in Linux to move the calling process into a fresh execution context. With `unshare(CLONE_NEWUSER)` you can move a process into a new [user_namespace(7)](https://man7.org/linux/man-pages/man7/user_namespaces.7.…, where the process gains the full capability on the resources within the namespace. This is fundamental for Linux containers to achieve privilege separation. `unshare(CLONE_NEWUSER)` requires the calling process to be single-threaded (or no background threads are running). So, it is often invoked after `fork(2)` as forking propagates only the calling thread to the child process. ## Problem It becomes a problem that Ruby 3.3 on Linux uses timer threads even for a single-`Thread`ed application. Because `Kernel#fork` spawns a thread in the child process before the control returns to the user code, there is no chance to call `unshare(CLONE_NEWUSER)` in Ruby. The following snippet is a reproducer of this problem. This program first forks and then shows the user namespace to which the process belongs before and after calling unshare(2). It also shows the threads of the child process after forking. ```ruby p(RUBY_DESCRIPTION:) require 'fiddle/import' module C extend Fiddle::Importer dlload 'libc.so.6' extern 'int unshare(int flags)' CLONE_NEWUSER = 0x10000000 def self.raise_system_call_error raise SystemCallError.new(Fiddle.last_error) end end pid = fork do system("ps -O tid -T -p #$$") system("ls -l /proc/self/ns/user") if C.unshare(C::CLONE_NEWUSER) != 0 C.raise_system_call_error # => EINVAL with Ruby 3.3 end system("ls -l /proc/self/ns/user") end p Process.wait2(pid) ``` The program successfully changes the user namespace with Ruby 3.2, but it raises EINVAL with Ruby 3.3. You can see Ruby 3.3 has two threads running after forking. ``` % rbenv shell 3.2 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585787 1585787 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' lrwxrwxrwx 1 nobody nogroup 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026532675]' [1585787, #<Process::Status: pid 1585787 exit 0>] % rbenv shell 3.3 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585849 1585849 S pts/12 00:00:00 ruby ./test.rb 1585849 1585851 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585849, #<Process::Status: pid 1585849 exit 1>] % rbenv shell master && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585965 1585965 S pts/12 00:00:00 ruby ./test.rb 1585965 1585967 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585965, #<Process::Status: pid 1585965 exit 1>] ``` ## Workaround My workaround is to rebuild ruby with `rb_thread_stop_timer_thread` and `rb_thread_start_timer_thread` exported, and use a C-ext that stops the timer thread before calling `unshare`. This seems not robust because the process cannot know when the terminated thread is reclaimed by the kernel, after which the process is considered single-threaded. ```c #define _GNU_SOURCE 1 #include <sched.h> #include <ruby/ruby.h> static VALUE Unshare_s_unshare(VALUE _self, VALUE rflags) { int const flags = NUM2INT(rflags); rb_thread_stop_timer_thread(); usleep(1000); // FIXME: It takes some time for the kernel to remove the stopped thread? int const ret = unshare(flags); rb_thread_start_timer_thread(); if(ret != 0) rb_sys_fail_str(rb_sprintf("unshare(%#x)", flags)); return Qnil; } RUBY_FUNC_EXPORTED void Init_unshare(void) { VALUE rb_mUnshare = rb_define_module("Unshare"); rb_define_singleton_method(rb_mUnshare, "unshare", Unshare_s_unshare, 1); rb_define_const(rb_mUnshare, "CLONE_NEWUSER", INT2FIX(CLONE_NEWUSER)); } ``` ## Questions - Is this a limitation of Ruby? - Is it safe (or even possible) to stop the timer thread during execution? - If so, can we export it as the public API? - But it may not so useful for this problem as explained in the workaround. - Is it guaranteed that no other threads are running after forks? - Are there any better ways to solve this issue? - Can we somehow delay the start of the timer thread after forking, or hook into `fork` to run some code in the child process immediately after it spawns. - Can they be Ruby API instead of C API? -- https://bugs.ruby-lang.org/

4 9

[ruby-core:117783] [Ruby master Bug#20473] Ractor array/hash literals
by kddnewton (Kevin Newton) 13 Sep '24

13 Sep '24

Issue #20473 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20473: Ractor array/hash literals https://bugs.ruby-lang.org/issues/20473 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- There are many optimizations in place in the compiler to ensure Ruby doesn't overflow the stack when an array or hash literal has many values. For example: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` gives the instructions: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,1564)> 0000 putobject "" ( 1)[Li] 0002 putnil 0003 dup 0004 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0006 anytostring 0007 concatstrings 2 0009 newarray 1 0011 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0013 concatarray 0014 putobject "" 0016 putnil 0017 dup 0018 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0020 anytostring 0021 concatstrings 2 0023 newarray 1 0025 concatarray 0026 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0028 concatarray 0029 putobject "" 0031 putnil 0032 dup 0033 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0035 anytostring 0036 concatstrings 2 0038 newarray 1 0040 concatarray 0041 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0043 concatarray 0044 putobject "" 0046 putnil 0047 dup 0048 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0050 anytostring 0051 concatstrings 2 0053 newarray 1 0055 concatarray 0056 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0058 concatarray 0059 leave ``` because it is smart enough to group the static elements together and use concatarray. This ensures we never have too many values on the stack. In Ruby 3.3, this worked for ractors as well. (Adding `# shareable_constant_value: literal` and `A =` to the example above doesn't change anything about the instructions above except adding a couple of `-@` calls and one call to `make_shareable`. On HEAD, all of these optimizations are now gone. With this: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm # shareable_constant_value: literal A = [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` code on master, we get: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(2,1568)> 0000 putobject RubyVM::FrozenCore ( 2)[Li] 0002 putobject "" 0004 putnil 0005 dup 0006 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0008 anytostring 0009 concatstrings 2 0011 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0013 putobject_INT2FIX_1_ ~~~~ 0139 putobject_INT2FIX_1_ 0140 putobject "" 0142 putnil 0143 dup 0144 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0146 anytostring 0147 concatstrings 2 0149 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0151 putobject_INT2FIX_1_ ~~~~ 0277 putobject_INT2FIX_1_ 0278 putobject "" 0280 putnil 0281 dup 0282 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0284 anytostring 0285 concatstrings 2 0287 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0289 putobject_INT2FIX_1_ ~~~~ 0415 putobject_INT2FIX_1_ 0416 putobject "" 0418 putnil 0419 dup 0420 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0422 anytostring 0423 concatstrings 2 0425 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0427 putobject_INT2FIX_1_ ~~~~ 0553 putobject_INT2FIX_1_ 0554 newarray 512 0556 opt_send_without_block <calldata!mid:make_shareable, argc:1, ARGS_SIMPLE> 0558 dup 0559 putspecialobject 3 0561 setconstant :A 0563 leave ``` where the `~~~~` represents another 128 `putobject_INT2FIX_1_` instructions. I assume this was done because it used to modify the AST and that got removed. However, I also assume it was not intentional to lose all of these optimizations, as it now means if Ractors are being used that large literals are vulnerable to stack overflows. Please let me know how to proceed, as I am trying to exactly match behavior in the prism compiler. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:118388] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding
by byroot (Jean Boussier) 11 Sep '24

11 Sep '24

Issue #20594 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/

12 31

[ruby-core:118180] [Ruby master Bug#20525] Percent string literal with indentation support
by bradgessler (Brad Gessler) 03 Sep '24

03 Sep '24

Issue #20525 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Bug #20525: Percent string literal with indentation support https://bugs.ruby-lang.org/issues/20525 * Author: bradgessler (Brad Gessler) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I have code that looks like this in an application: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown <<~MARKDOWN * Because its fun * Because its super-de-dooper MARKDOWN }, ``` The "squiggle" HEREDOCs strips the indentation, but the name of the HEREDOC doesn't look that great since "Markdown" appears twice. What I'd prefer is a string literal that deals with indentation, maybe it looks something like this: ``` ContentSlide(title: "Why Phlex?"){ Markdown %~{ # Why do you like markdown? * Because its fun * Because its super-de-dooper } }, ``` If Ruby doesn't want to go down the path of adding another type of literal, I'd propose adding a method to String that does the same thing as the `~` HEREDOC so something like this is possible: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown %{ # Why do you like markdown? * Because its fun * Because its super-de-dooper }.deindent }, ``` -- https://bugs.ruby-lang.org/

5 7