September 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117783] [Ruby master Bug#20473] Ractor array/hash literals
by kddnewton (Kevin Newton) 13 Sep '24

13 Sep '24

Issue #20473 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20473: Ractor array/hash literals https://bugs.ruby-lang.org/issues/20473 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- There are many optimizations in place in the compiler to ensure Ruby doesn't overflow the stack when an array or hash literal has many values. For example: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` gives the instructions: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(1,1564)> 0000 putobject "" ( 1)[Li] 0002 putnil 0003 dup 0004 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0006 anytostring 0007 concatstrings 2 0009 newarray 1 0011 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0013 concatarray 0014 putobject "" 0016 putnil 0017 dup 0018 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0020 anytostring 0021 concatstrings 2 0023 newarray 1 0025 concatarray 0026 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0028 concatarray 0029 putobject "" 0031 putnil 0032 dup 0033 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0035 anytostring 0036 concatstrings 2 0038 newarray 1 0040 concatarray 0041 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0043 concatarray 0044 putobject "" 0046 putnil 0047 dup 0048 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0050 anytostring 0051 concatstrings 2 0053 newarray 1 0055 concatarray 0056 putobject [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 0058 concatarray 0059 leave ``` because it is smart enough to group the static elements together and use concatarray. This ensures we never have too many values on the stack. In Ruby 3.3, this worked for ractors as well. (Adding `# shareable_constant_value: literal` and `A =` to the example above doesn't change anything about the instructions above except adding a couple of `-@` calls and one call to `make_shareable`. On HEAD, all of these optimizations are now gone. With this: ```ruby puts RubyVM::InstructionSequence.compile(<<~RUBY).disasm # shareable_constant_value: literal A = [#{Array.new(512) { |index| index % 128 == 0 ? "\"\#{nil}\"" : "1" }.join(", ")}] RUBY ``` code on master, we get: ``` == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(2,1568)> 0000 putobject RubyVM::FrozenCore ( 2)[Li] 0002 putobject "" 0004 putnil 0005 dup 0006 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0008 anytostring 0009 concatstrings 2 0011 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0013 putobject_INT2FIX_1_ ~~~~ 0139 putobject_INT2FIX_1_ 0140 putobject "" 0142 putnil 0143 dup 0144 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0146 anytostring 0147 concatstrings 2 0149 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0151 putobject_INT2FIX_1_ ~~~~ 0277 putobject_INT2FIX_1_ 0278 putobject "" 0280 putnil 0281 dup 0282 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0284 anytostring 0285 concatstrings 2 0287 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0289 putobject_INT2FIX_1_ ~~~~ 0415 putobject_INT2FIX_1_ 0416 putobject "" 0418 putnil 0419 dup 0420 objtostring <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE> 0422 anytostring 0423 concatstrings 2 0425 opt_send_without_block <calldata!mid:-@, argc:0, ARGS_SIMPLE> 0427 putobject_INT2FIX_1_ ~~~~ 0553 putobject_INT2FIX_1_ 0554 newarray 512 0556 opt_send_without_block <calldata!mid:make_shareable, argc:1, ARGS_SIMPLE> 0558 dup 0559 putspecialobject 3 0561 setconstant :A 0563 leave ``` where the `~~~~` represents another 128 `putobject_INT2FIX_1_` instructions. I assume this was done because it used to modify the AST and that got removed. However, I also assume it was not intentional to lose all of these optimizations, as it now means if Ractors are being used that large literals are vulnerable to stack overflows. Please let me know how to proceed, as I am trying to exactly match behavior in the prism compiler. -- https://bugs.ruby-lang.org/

2 3

[ruby-core:119165] [Ruby master Bug#20737] Accidentally changed warning target on Ruby 3.3
by hsbt (Hiroshi SHIBATA) 13 Sep '24

13 Sep '24

Issue #20737 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #20737: Accidentally changed warning target on Ruby 3.3 https://bugs.ruby-lang.org/issues/20737 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I heard that concern from yhonda. I request to backport bundled_gems.rb that is same version of the current master. After that, Ruby 3.3.5 show warnings when some default gems was loaded. https://github.com/ruby/ruby/blob/ruby_3_3/lib/bundled_gems.rb#L29-L34 I should warn them for future update. But this change shouldn't introduce tiny/patch update from 3.3.4. @k0kubun What's your opinion about that? I will create pull-request if we suppress warns for Ruby 3.5. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:119175] [Ruby master Misc#20739] Test suite does not carry over CLI options
by kddnewton (Kevin Newton) 13 Sep '24

13 Sep '24

Issue #20739 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20739: Test suite does not carry over CLI options https://bugs.ruby-lang.org/issues/20739 * Author: kddnewton (Kevin Newton) * Status: Open ---------------------------------------- I'm not sure if anything needs to change here, but I wanted to report this in case anyone had any particular ideas. When running the ruby/ruby test suite, lots of tests call `EnvUtil.invoke_ruby` directly or indirectly through `assert_in_out_err`. That spawns a subprocess that runs Ruby and reports its status back through status, stdout, and stderr. The issue is, the subprocess does not always use the same options, which means there are lots of different ways it can be run. For RJIT/YJIT, we have `-DRJIT_FORCE_ENABLE` and `-DYJIT_FORCE_ENABLE`, which forces the subprocess to have it enabled. Still, this isn't always used, so when running the test suite for RJIT/YJIT, a lot of tests are not actually running with those compilers. Prism and parse.y have a similar problem. When running in the same process, it will respect the `--parser` option. But when running in a subprocess it will fall back to whatever the default parser is. This presents a problem for some tests that assert against RUBY_DESCRIPTION in a subprocess, as it may have `+PRISM` in the description. We have tried to fix it before by adding RUN_OPTS/RUBYOPT to the subprocess options, but this causes lots of other problems. I am not sure of the correct solution, or if we should even bother fixing this. Please let me know what you think. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:119161] [Ruby master Bug#20735] prism emits "ambiguous `&`" warning
by mame (Yusuke Endoh) 13 Sep '24

13 Sep '24

Issue #20735 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #20735: prism emits "ambiguous `&`" warning https://bugs.ruby-lang.org/issues/20735 * Author: mame (Yusuke Endoh) * Status: Open * Assignee: kddnewton (Kevin Newton) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ``` $ ./local/bin/ruby -w --parser=parse.y -e 'def foo(&) = nil; foo &:bar' $ ./local/bin/ruby -w --parser=prism -e 'def foo(&) = nil; foo &:bar' -e:1: warning: ambiguous `&` has been interpreted as an argument prefix ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:119180] [Ruby master Bug#20740] prism_compile coding issue for PM_FORWARDING_SUPER_NODE
by luke-gru (Luke Gruber) 13 Sep '24

13 Sep '24

Issue #20740 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #20740: prism_compile coding issue for PM_FORWARDING_SUPER_NODE https://bugs.ruby-lang.org/issues/20740 * Author: luke-gru (Luke Gruber) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I found a coding issue in prism_compile.c but I can't find a bug that would result from it. The `current_block` is not set back when a block is given. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:119152] [Ruby master Bug#20731] warning: mismatched indentations at 'when' appears with prism enabled ruby
by yahonda (Yasuo Honda) 13 Sep '24

13 Sep '24

Issue #20731 has been reported by yahonda (Yasuo Honda). ---------------------------------------- Bug #20731: warning: mismatched indentations at 'when' appears with prism enabled ruby https://bugs.ruby-lang.org/issues/20731 * Author: yahonda (Yasuo Honda) * Status: Open * ruby -v: ruby 3.4.0dev (2024-09-12T20:03:28Z master 0fc8422a05) +PRISM [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ### Steps to reproduce 1. Create a `case_sample.rb` file ``` foo = :bar case foo when :bar p 'foo is :bar' when :baz # do something else # do something end ``` 2. Run this file against the latest Ruby 3.4.0dev with -w option to show warnings ``` $ ruby -w case_sample.rb ``` ### Expected behavior It should show `"foo is :bar"` without any warnings as it runs against Ruby 3.3.5 or Ruby 3.4.0dev before prism is used by default ``` $ ruby -w case_sample.rb "foo is :bar" $ ruby -v ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-linux] $ ``` ``` $ ruby -w case_sample.rb "foo is :bar" $ ruby -v ruby 3.4.0dev (2024-09-12T17:04:10Z master f2919bd11c) [x86_64-linux] $ ``` ### Actual behavior It raises `warning: mismatched indentations at 'when' with 'case' ` warnings as follows. ``` $ ruby -w case_sample.rb case_sample.rb:3: warning: mismatched indentations at 'when' with 'case' at 2 case_sample.rb:5: warning: mismatched indentations at 'when' with 'case' at 2 "foo is :bar" $ ruby -v ruby 3.4.0dev (2024-09-12T20:03:28Z master 0fc8422a05) +PRISM [x86_64-linux] $ ``` ### Background This repro is based on the tzinfo code below. https://github.com/tzinfo/tzinfo/blob/d5893c99bbcd842af86023f02a84fac1659b3… https://github.com/tzinfo/tzinfo/blob/d5893c99bbcd842af86023f02a84fac1659b3… -- https://bugs.ruby-lang.org/

4 5

[ruby-core:119154] [Ruby master Bug#20732] prism emits "mismatched indentations" warning in eval
by mame (Yusuke Endoh) 13 Sep '24

13 Sep '24

Issue #20732 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #20732: prism emits "mismatched indentations" warning in eval https://bugs.ruby-lang.org/issues/20732 * Author: mame (Yusuke Endoh) * Status: Open * Assignee: kddnewton (Kevin Newton) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby eval(<<-EOC) if @check end EOC ``` ``` $ ruby --parser=parse.y -w t.rb $ ruby --parser=prism -w t.rb (eval at t.rb:1):2: warning: mismatched indentations at 'end' with 'if' at 1 ``` This issue actually occurs in rss gem. https://github.com/ruby/rss/blob/8a5b056eab591525e9c14ae1cd6e97ebae386be8/l… https://github.com/ruby/rss/blob/8a5b056eab591525e9c14ae1cd6e97ebae386be8/l… -- https://bugs.ruby-lang.org/

2 1

[ruby-core:119132] Segfault using ruby C on MacOS (Intel Catalina and M2 Sonoma)
by martin.kufner＠quiz.baby 13 Sep '24

13 Sep '24

Hey guys, I was experienced at C about 20 years ago, since 15 years I'm coding with Ruby. I now need to write a Ruby C extension, and I fail with just calling simple commands like rb_p. I try to integrate a fast C websocket server into my quiz website (Rails cable is too slow) In order to communicate I need a UNIX socket in C to receive commands from the Rails server. I used the xcode standard compiler, and also tried homebrew gcc-14 It compiles fine all set up with extconf.rb When I call QbGameServer.init, it creates the server, and after a second wait tries to connect. The c thread is created and gets the message, However when I call rb_p or rb_funcall it hangs (M2) or segfauls (Intel) What do I do wrong? Thanks a lot! Martin I created a QbGameServerModule, extending my QbGameServer and call module QbGameServer SOCKET_PATH="/tmp/sock/qb-game-server" extend QbGameServerModule def self.init # Start the server (non-blocking) self.start_rpc(SOCKET_PATH) do |message| puts "Received from client: #{message}" end sleep 1 send_socket end def self.send_socket # Create a new Unix socket socket = UNIXSocket.new(SOCKET_PATH) # Example data to send to the server (Ruby hash converted to JSON) data_to_send = { message: "Hello, server!", timestamp: Time.now.to_s }.to_json # Send the JSON data to the server puts "Sent to server: #{data_to_send}" socket.puts(data_to_send) # Read the response from the server response = socket.gets.chomp # Read the response from the socket puts "Received from server: #{response}" ensure # Close the socket connection after the communication is done socket.close if socket end end On the C side I have the ext/qb_game_server/qb_game_server.c #include "ruby.h" #include <sys/types.h> #include <sys/socket.h> #include <sys/un.h> #include <unistd.h> #include <pthread.h> #include <string.h> #define MAX_CLIENTS 10 #define BUFFER_SIZE 1024 VALUE QbGameServerModule = Qnil; static VALUE rpc_proc = Qnil; static char socket_file_path[108]; // Maximum length for UNIX domain socket path void *client_handler(void *sock) { int client_socket = *(int *)sock; char buffer[BUFFER_SIZE]; ssize_t bytes_read; while ((bytes_read = read(client_socket, buffer, sizeof(buffer) - 1)) > 0) { buffer[bytes_read] = '\0'; printf("Received C %s\n", buffer); VALUE arg = rb_str_new_cstr(buffer); // rb_gc_register_address(&arg); printf("1\n"); // DOES REACH // rb_gc(); printf("2\n"); // DOES REACH rb_p(arg); // ***** DOES NOT REACH HERE***** // On the osx sonoma M2 it just hangs, on osx Intel Catalina it shows a segfault printf("3\n"); // rb_gc_unregister_address(&arg); // rb_funcall(rpc_proc, rb_intern("call"), 1, arg); } close(client_socket); return NULL; } void *server_thread(void *arg) { int server_socket = socket(AF_UNIX, SOCK_STREAM, 0); struct sockaddr_un server_addr; int client_sockets[MAX_CLIENTS]; pthread_t threads[MAX_CLIENTS]; int client_count = 0; memset(&server_addr, 0, sizeof(server_addr)); server_addr.sun_family = AF_UNIX; strncpy(server_addr.sun_path, socket_file_path, sizeof(server_addr.sun_path) - 1); unlink(socket_file_path); bind(server_socket, (struct sockaddr *)&server_addr, sizeof(server_addr)); listen(server_socket, MAX_CLIENTS); while (1) { client_sockets[client_count] = accept(server_socket, NULL, NULL); pthread_create(&threads[client_count], NULL, client_handler, &client_sockets[client_count]); client_count++; } close(server_socket); return NULL; } static VALUE start_rpc(VALUE self, VALUE socket_file) { Check_Type(socket_file, T_STRING); strncpy(socket_file_path, StringValueCStr(socket_file), sizeof(socket_file_path) - 1); if (rb_block_given_p()) { rpc_proc = rb_block_proc(); } else { rb_raise(rb_eArgError, "A block must be provided as a callback"); } pthread_t server_tid; pthread_create(&server_tid, NULL, server_thread, NULL); return Qnil; } void Init_qb_game_server() { printf("Init QbGameServerModule\n"); QbGameServerModule = rb_define_module("QbGameServerModule"); rb_define_method(QbGameServerModule, "start_rpc", start_rpc, 1); }

1 3

[ruby-core:119136] [Ruby master Bug#20725] Inconsistent handling of doubly-unbounded ranges in Range#overlap?
by jeromepl (Jerome Parent-Levesque) 13 Sep '24

13 Sep '24

Issue #20725 has been reported by jeromepl (Jerome Parent-Levesque). ---------------------------------------- Bug #20725: Inconsistent handling of doubly-unbounded ranges in Range#overlap? https://bugs.ruby-lang.org/issues/20725 * Author: jeromepl (Jerome Parent-Levesque) * Status: Open * ruby -v: ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [arm64-darwin23] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- The new `Range#overlap?` method does not seem to have a clear definition of how it behaves when passed ranges with nil begin **and** end `(nil..)`. In the following two cases, it seems to treat this type of range as an "infinite" range that covers the entire space and therefore overlaps with anything: ``` (nil..nil).overlap?(nil..) # => true (nil..nil).overlap?(3..) # => true ``` However it behaves differently in this case: ``` (nil..nil).overlap?(..3) # => false ``` Note that this also behaves differently than the ActiveSupport implementation of #overlap? in a Ruby version < 3.3 such as 3.2: ``` require "active_support" (nil..nil).overlap?(..3) # => true ``` I think `(nil..nil)` ranges should always be considered to be completely unbounded, i.e. infinite and covering the whole space. They should therefore always return `true` when overlapping with any non-empty range. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:119135] CPU-bound load affects the timeout thread
by yuri.kanivetsky＠gmail.com 12 Sep '24

12 Sep '24

Hi, I'm not sure if it's a bug. Take a look at the following gist: https://gist.github.com/x-yuri/253f76df6287441f64b5eaee418813c0 It's supposedly a minimal reproducible example of what I ran into a couple of days ago. In my case it was basically a sort of a background jobs service. It received messages, processed them and sent the result to another service. Every message was processed and sent in a separate thread. And it made a number of http requests as part of processing messages (net/http which uses timeout()). It worked normally for a couple of hours (10-20 threads), but at some point the number of threads reached the thread pool's max_threads (320), http requests started to time out (later then they should, after 60-100 seconds instead of 3 seconds), and probably things started to get slow. Then I noticed the following line: patterns.any? { |pattern| quote.match(pattern['regex']) } ? 1 : 0 It was performed for every quote (10 times) in every thread, there were 300+ patterns and they (pattern['regex']) were strings. I moved converting strings to regexps to initialization (before starting to accept messages and creating threads) and it seems to work now with ~10 active threads. I'm trying to understand what exactly happened and how to avoid it or what awaits me in the future. Is there some critical load that breaks things? I guess in theory the number of threads should increase linearly with the load. But it looks like in this case there's some critical load that just makes things stop working. Or maybe for some time some negative effects get accumulated and then things break. My conjecture is that it has something to do with GIL. But what exactly happens? Or what can I do to further investigate the issue? I'm going to try to run ruby with RUBY_DEBUG_LOG and try to examine the debug output. I guess I need to figure out why the timeout thread doesn't always get enough time. Does something block the threads? How are they scheduled? When do they switch? Any pointers or links to the code are welcome. Regards, Yuri

2 2