July 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:113052] [Ruby master Bug#19562] Setting default_(in|ex)ternal encoding after running an Ractor freezes ruby
by larskanis (Lars Kanis) 20 Sep '24

20 Sep '24

Issue #19562 has been reported by larskanis (Lars Kanis). ---------------------------------------- Bug #19562: Setting default_(in|ex)ternal encoding after running an Ractor freezes ruby https://bugs.ruby-lang.org/issues/19562 * Author: larskanis (Lars Kanis) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The following command blocks infinitely and must be killed with "kill -9" ``` ruby -e "Ractor.new{}.take; Encoding.default_external = Encoding::ISO8859_2" ``` Backtrace is: ``` #0 futex_wait (private=0, expected=2, futex_word=0x55555555abd0) at ../sysdeps/nptl/futex-internal.h:146 #1 __GI___lll_lock_wait (futex=futex@entry=0x55555555abd0, private=0) at ./nptl/lowlevellock.c:49 #2 0x00007ffff760a082 in lll_mutex_lock_optimized (mutex=0x55555555abd0) at ./nptl/pthread_mutex_lock.c:48 #3 ___pthread_mutex_lock (mutex=mutex@entry=0x55555555abd0) at ./nptl/pthread_mutex_lock.c:93 #4 0x00007ffff7c5a40d in rb_native_mutex_lock (lock=lock@entry=0x55555555abd0) at ruby-3.2.1/thread_pthread.c:556 #5 0x00007ffff7cc5376 in vm_lock_enter (cr=0x55555555deb0, vm=0x55555555aba0, locked=locked@entry=false, no_barrier=no_barrier@entry=false, lev=0x55555555ac00) at vm_sync.c:56 #6 0x00007ffff7cc5747 in rb_vm_lock_body () at vm_sync.c:175 #7 0x00007ffff7bc02c2 in rb_vm_lock (line=1829, file=0x7ffff7e209cb "ractor.c") at ruby-3.2.1/vm_sync.h:57 #8 ractor_check_blocking (remained_thread_cnt=<optimized out>, file=0x7ffff7e209cb "ractor.c", line=1865, cr=0x55555555deb0) at ractor.c:1829 #9 ractor_check_blocking (line=1865, file=0x7ffff7e209cb "ractor.c", remained_thread_cnt=<optimized out>, cr=0x55555555deb0) at ractor.c:1811 #10 rb_ractor_blocking_threads_inc (cr=0x55555555deb0, file=file@entry=0x7ffff7e29b04 "thread.c", line=line@entry=1488) at ractor.c:1865 #11 0x00007ffff7c60977 in blocking_region_begin (th=0x55555555d070, region=0x7fffffffd220, ubf=0x7ffff7c5b300 <ubf_select>, arg=0x55555555d070, fail_if_interrupted=<optimized out>) at thread.c:1488 #12 0x00007ffff7c61edf in rb_nogvl (func=0x7ffff7ac66e0 <no_gvl_stat>, data1=0x7fffffffd2a0, ubf=<optimized out>, data2=0x55555555d070, flags=<optimized out>) at thread.c:1548 #13 0x00007ffff7acdfbc in stat_without_gvl (st=0x7fffffffd2b0, path=0x7ffff27dfe88 "ruby-3.2.1/lib/ruby/3.2.0/x86_64-linux/enc/iso_8859_2.so") at file.c:1171 #14 rb_check_realpath_internal (basedir=basedir@entry=4, path=path@entry=140737261984200, origenc=0x555555565030, mode=mode@entry=RB_REALPATH_STRICT) at file.c:4491 #15 0x00007ffff7acebaa in rb_realpath_internal (basedir=basedir@entry=4, path=path@entry=140737261984200, strict=strict@entry=1) at file.c:4536 #16 0x00007ffff7b2cdd4 in require_internal (ec=0x55555555e1f0, fname=<optimized out>, exception=1, warn=<optimized out>) at load.c:1195 #17 0x00007ffff7aa0a17 in load_encoding (name=0x5555556049b0 "ISO-8859-2") at encoding.c:779 #18 0x00007ffff7aa2ef8 in check_encoding (enc=0x5555556049d0) at encoding.c:191 #19 enc_check_encoding (obj=140737335533800) at encoding.c:202 #20 rb_to_encoding (enc=enc@entry=140737335533800) at encoding.c:307 #21 0x00007ffff7aa4049 in enc_set_default_encoding (name=0x7ffff7de6287 "external", encoding=140737335533800, def=0x7ffff7fb00c0 <default_external>) at encoding.c:1563 #22 rb_enc_set_default_external (encoding=140737335533800) at encoding.c:1637 #23 0x00007ffff7aa411f in set_default_external (klass=<optimized out>, encoding=140737335533800) at encoding.c:1658 #24 0x00007ffff7c99037 in vm_call_cfunc_with_frame (ec=0x55555555e1f0, reg_cfp=0x7ffff6f7df90, calling=<optimized out>) at ruby-3.2.1/vm_insnhelper.c:3268 #25 0x00007ffff7ca9564 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>) at ruby-3.2.1/vm_callinfo.h:367 #26 vm_exec_core (ec=0x55555555e1f0, initial=128) at ruby-3.2.1/insns.def:820 #27 0x00007ffff7caee93 in rb_vm_exec (ec=0x55555555e1f0, jit_enable_p=true) at vm.c:2374 #28 0x00007ffff7abfbf1 in rb_ec_exec_node (ec=ec@entry=0x55555555e1f0, n=n@entry=0x7ffff796c2c0) at eval.c:289 #29 0x00007ffff7ac60b3 in ruby_run_node (n=0x7ffff796c2c0) at eval.c:330 #30 0x0000555555555187 in rb_main (argv=0x7fffffffdb98, argc=3) at ./main.c:38 #31 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57 ``` When the encoding was already loaded before running `Ractor.new`, then it works and doesn't block: ``` ruby -e "''.encode(Encoding::ISO8859_2); Ractor.new{}.take; Encoding.default_internal = Encoding::ISO8859_2" ``` When the encoding is not assigned as default encoding, but used otherwise, it works too: ``` ruby -e "Ractor.new{}.take; ''.encode(Encoding::ISO8859_2)" ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:111144] [Ruby master Feature#19171] Update Unicode data to Unicode Version 15.1
by duerst 20 Sep '24

20 Sep '24

Issue #19171 has been reported by duerst (Martin Dürst). ---------------------------------------- Feature #19171: Update Unicode data to Unicode Version 15.1 https://bugs.ruby-lang.org/issues/19171 * Author: duerst (Martin Dürst) * Status: Open * Priority: Normal * Assignee: duerst (Martin Dürst) ---------------------------------------- According to http://blog.unicode.org/2022/11/the-unicode-standard-2023-release.html, Unicode plans to release Version 15.1 in September 2023. According to https://www.unicode.org/versions/beta.html, public alpha review starts Feb. 7, 2023, and ends April 4, 2023. Because alpha review may not include all the files we use, it may be difficult for us to participate. Public beta review is planned to start May 23, 2023, ending July 4, 2023. At this point, we should be able to test things. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116581] [Ruby master Bug#20237] Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread
by hanazuki (Kasumi Hanazuki) 16 Sep '24

16 Sep '24

Issue #20237 has been reported by hanazuki (Kasumi Hanazuki). ---------------------------------------- Bug #20237: Unable to unshare(CLONE_NEWUSER) in Linux because of timer thread https://bugs.ruby-lang.org/issues/20237 * Author: hanazuki (Kasumi Hanazuki) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ## Backgrounds [unshare(2)](https://man7.org/linux/man-pages/man2/unshare.2.html) is a syscall in Linux to move the calling process into a fresh execution context. With `unshare(CLONE_NEWUSER)` you can move a process into a new [user_namespace(7)](https://man7.org/linux/man-pages/man7/user_namespaces.7.…, where the process gains the full capability on the resources within the namespace. This is fundamental for Linux containers to achieve privilege separation. `unshare(CLONE_NEWUSER)` requires the calling process to be single-threaded (or no background threads are running). So, it is often invoked after `fork(2)` as forking propagates only the calling thread to the child process. ## Problem It becomes a problem that Ruby 3.3 on Linux uses timer threads even for a single-`Thread`ed application. Because `Kernel#fork` spawns a thread in the child process before the control returns to the user code, there is no chance to call `unshare(CLONE_NEWUSER)` in Ruby. The following snippet is a reproducer of this problem. This program first forks and then shows the user namespace to which the process belongs before and after calling unshare(2). It also shows the threads of the child process after forking. ```ruby p(RUBY_DESCRIPTION:) require 'fiddle/import' module C extend Fiddle::Importer dlload 'libc.so.6' extern 'int unshare(int flags)' CLONE_NEWUSER = 0x10000000 def self.raise_system_call_error raise SystemCallError.new(Fiddle.last_error) end end pid = fork do system("ps -O tid -T -p #$$") system("ls -l /proc/self/ns/user") if C.unshare(C::CLONE_NEWUSER) != 0 C.raise_system_call_error # => EINVAL with Ruby 3.3 end system("ls -l /proc/self/ns/user") end p Process.wait2(pid) ``` The program successfully changes the user namespace with Ruby 3.2, but it raises EINVAL with Ruby 3.3. You can see Ruby 3.3 has two threads running after forking. ``` % rbenv shell 3.2 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585787 1585787 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' lrwxrwxrwx 1 nobody nogroup 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026532675]' [1585787, #<Process::Status: pid 1585787 exit 0>] % rbenv shell 3.3 && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585849 1585849 S pts/12 00:00:00 ruby ./test.rb 1585849 1585851 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585849, #<Process::Status: pid 1585849 exit 1>] % rbenv shell master && ruby ./test.rb {:RUBY_DESCRIPTION=>"ruby 3.4.0dev (2024-02-04T16:05:02Z master 8bc6fff322) [x86_64-linux]"} PID TID S TTY TIME COMMAND 1585965 1585965 S pts/12 00:00:00 ruby ./test.rb 1585965 1585967 S pts/12 00:00:00 ruby ./test.rb lrwxrwxrwx 1 kasumi kasumi 0 Feb 5 02:25 /proc/self/ns/user -> 'user:[4026531837]' ./test.rb:10:in `raise_system_call_error': Invalid argument (Errno::EINVAL) from ./test.rb:24:in `block in <main>' from ./test.rb:19:in `fork' from ./test.rb:19:in `<main>' [1585965, #<Process::Status: pid 1585965 exit 1>] ``` ## Workaround My workaround is to rebuild ruby with `rb_thread_stop_timer_thread` and `rb_thread_start_timer_thread` exported, and use a C-ext that stops the timer thread before calling `unshare`. This seems not robust because the process cannot know when the terminated thread is reclaimed by the kernel, after which the process is considered single-threaded. ```c #define _GNU_SOURCE 1 #include <sched.h> #include <ruby/ruby.h> static VALUE Unshare_s_unshare(VALUE _self, VALUE rflags) { int const flags = NUM2INT(rflags); rb_thread_stop_timer_thread(); usleep(1000); // FIXME: It takes some time for the kernel to remove the stopped thread? int const ret = unshare(flags); rb_thread_start_timer_thread(); if(ret != 0) rb_sys_fail_str(rb_sprintf("unshare(%#x)", flags)); return Qnil; } RUBY_FUNC_EXPORTED void Init_unshare(void) { VALUE rb_mUnshare = rb_define_module("Unshare"); rb_define_singleton_method(rb_mUnshare, "unshare", Unshare_s_unshare, 1); rb_define_const(rb_mUnshare, "CLONE_NEWUSER", INT2FIX(CLONE_NEWUSER)); } ``` ## Questions - Is this a limitation of Ruby? - Is it safe (or even possible) to stop the timer thread during execution? - If so, can we export it as the public API? - But it may not so useful for this problem as explained in the workaround. - Is it guaranteed that no other threads are running after forks? - Are there any better ways to solve this issue? - Can we somehow delay the start of the timer thread after forking, or hook into `fork` to run some code in the child process immediately after it spawns. - Can they be Ruby API instead of C API? -- https://bugs.ruby-lang.org/

4 9

[ruby-core:117783] [Ruby master Bug#20473] Ractor array/hash literals
by kddnewton (Kevin Newton) 13 Sep '24

13 Sep '24

Issue #20473 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20473: Ractor array/hash literals https://bugs.ruby-lang.org/issues/20473 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- There are many optimizations in place in the compiler to ensure Ruby doesn't overflow the stack when an array or hash literal has many values. For example: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` gives the instructions: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,1564)> 0000 putobject "" ( 1)[Li] 0002 putnil 0003 dup 0004 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0006 anytostring 0007 concatstrings 2 0009 newarray 1 0011 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0013 concatarray 0014 putobject "" 0016 putnil 0017 dup 0018 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0020 anytostring 0021 concatstrings 2 0023 newarray 1 0025 concatarray 0026 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0028 concatarray 0029 putobject "" 0031 putnil 0032 dup 0033 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0035 anytostring 0036 concatstrings 2 0038 newarray 1 0040 concatarray 0041 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0043 concatarray 0044 putobject "" 0046 putnil 0047 dup 0048 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0050 anytostring 0051 concatstrings 2 0053 newarray 1 0055 concatarray 0056 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0058 concatarray 0059 leave ``` because it is smart enough to group the static elements together and use concatarray. This ensures we never have too many values on the stack. In Ruby 3.3, this worked for ractors as well. (Adding `# shareable_constant_value: literal` and `A =` to the example above doesn't change anything about the instructions above except adding a couple of `-@` calls and one call to `make_shareable`. On HEAD, all of these optimizations are now gone. With this: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm # shareable_constant_value: literal A = [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` code on master, we get: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(2,1568)> 0000 putobject RubyVM::FrozenCore ( 2)[Li] 0002 putobject "" 0004 putnil 0005 dup 0006 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0008 anytostring 0009 concatstrings 2 0011 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0013 putobject_INT2FIX_1_ ~~~~ 0139 putobject_INT2FIX_1_ 0140 putobject "" 0142 putnil 0143 dup 0144 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0146 anytostring 0147 concatstrings 2 0149 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0151 putobject_INT2FIX_1_ ~~~~ 0277 putobject_INT2FIX_1_ 0278 putobject "" 0280 putnil 0281 dup 0282 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0284 anytostring 0285 concatstrings 2 0287 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0289 putobject_INT2FIX_1_ ~~~~ 0415 putobject_INT2FIX_1_ 0416 putobject "" 0418 putnil 0419 dup 0420 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0422 anytostring 0423 concatstrings 2 0425 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0427 putobject_INT2FIX_1_ ~~~~ 0553 putobject_INT2FIX_1_ 0554 newarray 512 0556 opt_send_without_block <calldata!mid:make_shareable, argc:1, ARGS_SIMPLE> 0558 dup 0559 putspecialobject 3 0561 setconstant :A 0563 leave ``` where the `~~~~` represents another 128 `putobject_INT2FIX_1_` instructions. I assume this was done because it used to modify the AST and that got removed. However, I also assume it was not intentional to lose all of these optimizations, as it now means if Ractors are being used that large literals are vulnerable to stack overflows. Please let me know how to proceed, as I am trying to exactly match behavior in the prism compiler. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:118388] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding
by byroot (Jean Boussier) 11 Sep '24

11 Sep '24

Issue #20594 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/

12 31

[ruby-core:118745] [Ruby master Misc#20658] Dropping the "Arm Neoverse (Ubuntu) server on RubyCI
by jaruga (Jun Aruga) 06 Sep '24

06 Sep '24

Issue #20658 has been reported by jaruga (Jun Aruga). ---------------------------------------- Misc #20658: Dropping the "Arm Neoverse (Ubuntu) server on RubyCI https://bugs.ruby-lang.org/issues/20658 * Author: jaruga (Jun Aruga) * Status: Open ---------------------------------------- ## Context On July 26 2024, 4 days ago, the person at Arm sponsoring a SSH-accessing Arm server via Works on Arm program told me that they could no longer sponsor the server **after 10th August 2024**. We have been using Equinix's Arm-based virtual machine on their cloud infrastructure[2] for the server right now. The person at Arm told me that Arm is actually scaling down their servers at Equinix within the Works on Arm program. While the person at Arm suggested using their cloud offerings listed on the [1] for paid or free tier, I need to take time to investigate how to continue to use Arm-based SSH-accessing servers continuously within the Works on Arm program if we want to continue to use it. At least we need to drop the Arm-based server that is named as "Arm Neoverse (Ubuntu)" on RubyCI until 10th August 2024.[3] ## Note for Ruby project commiters using the "Arm Neoverse (Ubuntu)" server Dear commiters, Please back up your files on the "Arm Neoverse (Ubuntu)" server if you have the files. After 10th August 2024, we cannot access to the server by SSH any more. ## Steps to drop the "Arm Neoverse (Ubuntu)" I plan to do the following tasks to drop the "Arm Neoverse (Ubuntu)". * Drop the "Arm Neoverse (Ubuntu)" on RubyCI. * Remove the content about the "Ubuntu arm neoverse" on [our Wiki page](https://github.com/ruby/ruby/wiki/CI-Servers). * Drop the content about Works on Arm on www.ruby-lang.org, which I added by the pull-request <https://github.com/ruby/www.ruby-lang.org/pull/3183>. Please let me know if you find any other task about this topic. [1] Works on Arm: https://www.arm.com/markets/computing-infrastructure/works-on-arm [2] Equinix: https://www.equinix.com/ [3] RubyCI: https://rubyci.org/ -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118542] [Ruby master Bug#20627] `require` on Ractor should run on the main Ractor
by ko1 (Koichi Sasada) 05 Sep '24

05 Sep '24

Issue #20627 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Bug #20627: `require` on Ractor should run on the main Ractor https://bugs.ruby-lang.org/issues/20627 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Now `require` on main Ractor is not allowed (raising error) but it is hard, especially for `autoload`. So let's allow `require` by running it on the main Ractor. ## Background On many libraries it is needed to run loading on the main Ractors because: 1. Setup constants with unshareable objects (such as `C = []`) are not allowed on non main Ractors. 2. Setup global variables and class variables are not allowed. `$LOADED_FEATURES` is also untouchable. 3. (maybe more reasons) So the `require` on non main Ractors is not allowed. However it is hard to program especially on `autoload`. Also dynamic `require` (`require` in methods) are not allowed too (`pp` method, for example). ## Proposal Allow `require` on non main Ractors by running `require` process on the main Ractor. ![](clipboard-202407111201-kaiem.png) (quoted on my talk at RubyKaigi 2024) ### `rb_ractor_interrupt_exec(target_ractor, func)` C-API Make a thread on `target_ractor` and run `func` (C function) on it. I think it is safe to expose on Ruby API because running `func` on a *newly created* thread (do not disturb running target threads). But now it is proposed as only (hidden) C-API. ### New Ractor methods - `Rcator.main?` returns Ractors - `Ractor.require(feature)` do `require` on the main Ractor These new methods are useful for users who override `require` method like [RubyGems](https://github.com/ruby/ruby/blob/master/lib/rubygems/core_ext/ke…. ```ruby alias orig_require require def require feature return Rator.require(feature) unless Ractor.main? # overriding require code end ``` Or we can prepend a module like: ``` Module.new do def require(feature) return Rator.require(feature) unless Ractor.main? super(feature) end Kernel.prepend self end ``` will support ractors for all overgrinding methods. But not sure it is acceptable to add additional one modules in ancestors by `prepend`. Also this technique doesn't support require overriding by prepending. ## Implementation https://github.com/ruby/ruby/pull/11142 (not matured yet) ---Files-------------------------------- clipboard-202407111201-kaiem.png (70.8 KB) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118180] [Ruby master Bug#20525] Percent string literal with indentation support
by bradgessler (Brad Gessler) 03 Sep '24

03 Sep '24

Issue #20525 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Bug #20525: Percent string literal with indentation support https://bugs.ruby-lang.org/issues/20525 * Author: bradgessler (Brad Gessler) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I have code that looks like this in an application: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown <<~MARKDOWN * Because its fun * Because its super-de-dooper MARKDOWN }, ``` The "squiggle" HEREDOCs strips the indentation, but the name of the HEREDOC doesn't look that great since "Markdown" appears twice. What I'd prefer is a string literal that deals with indentation, maybe it looks something like this: ``` ContentSlide(title: "Why Phlex?"){ Markdown %~{ # Why do you like markdown? * Because its fun * Because its super-de-dooper } }, ``` If Ruby doesn't want to go down the path of adding another type of literal, I'd propose adding a method to String that does the same thing as the `~` HEREDOC so something like this is possible: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown %{ # Why do you like markdown? * Because its fun * Because its super-de-dooper }.deindent }, ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:117210] [Ruby master Bug#20344] argument stack underflow (-1)
by nobu (Nobuyoshi Nakada) 02 Sep '24

02 Sep '24

Issue #20344 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20344: argument stack underflow (-1) https://bugs.ruby-lang.org/issues/20344 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- This file produces argument stack underflow, since ruby 2.5. ```ruby proc do next if true case nil when "a" next when "b" when "c" proc {} end next unless true end ``` ``` -- raw disasm-------- trace: 100 0000 nop ( 1) <L000> [sp: 0, unremovable: 1, refcnt: 5] trace: 1 <L004> [sp: 0, unremovable: 0, refcnt: 1] adjust: [label: 0] 0001 putnil ( 2) 0002 leave ( 13) <L008> [sp: 1, unremovable: 0, refcnt: 1] adjust: [label: 0] 0003 putnil ( 6) 0004 leave ( 13) adjust: [label: 8] 0005 jump <L005> ( 5) 0007 pop ( 7) 0008 jump <L005> ( 7) * 0010 pop ( 8) trace: 1 <L011> [sp: -1, unremovable: 1, refcnt: 1] 0011 putself ( 9) 0012 send <calldata:proc, 0>, nil ( 9) <L012> [sp: -1, unremovable: 0, refcnt: 2] 0015 pop ( 9) <L005> [sp: 1, unremovable: 0, refcnt: 2] trace: 1 0016 putnil ( 12) <L001> [sp: -1, unremovable: 0, refcnt: 3] trace: 200 0017 leave ( 13) --------------------- ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118676] [Ruby master Bug#20649] Ripper fails to tokenize `def f; (x)::A =`
by tompng (tomoya ishida) 02 Sep '24

02 Sep '24

Issue #20649 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20649: Ripper fails to tokenize `def f; (x)::A =` https://bugs.ruby-lang.org/issues/20649 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-07-24T05:44:32Z master e642ddf7ae) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Ripper fails to tokenize some dynamic constant assignment syntax error code. `''::A=` `(x)::A=` `(x).y::Z=` `[]::A=` `{}::A=` ``` ``::A=``` `//::A=` `%()::A=` in method body seems to fail. Same for `for ''::A in` and `rescue => ''::A;` in method body. ~~~ruby Ripper.tokenize 'def f; (x)::A =' #=> /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:245:in 'Ripper::Lexer#on_error2': undefined method 'pos' for nil (NoMethodError) @errors.push Elem.new(elem.pos, __callee__, elem.tok, elem.state, mesg) ^^^^ from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:176:in 'Ripper#parse' from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:176:in 'Ripper::Lexer#parse' from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:150:in 'Ripper::Lexer#tokenize' from /opt/ruby/lib/ruby/3.4.0+0/ripper/lexer.rb:26:in 'Ripper.tokenize' from -e:1:in '<main>' ~~~ -- https://bugs.ruby-lang.org/

4 3