January 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:116491] [Ruby master Bug#20225] Inconsistent behavior of regex matching for a regex has a null loop
by make_now_just (Hiroya Fujinami) 15 Mar '24

15 Mar '24

Issue #20225 has been reported by make_now_just (Hiroya Fujinami). ---------------------------------------- Bug #20225: Inconsistent behavior of regex matching for a regex has a null loop https://bugs.ruby-lang.org/issues/20225 * Author: make_now_just (Hiroya Fujinami) * Status: Open * Priority: Normal * Assignee: make_now_just (Hiroya Fujinami) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Usually, in Ruby (Onigmo), when a null loop (a loop consuming no characters) occurs on regex matching, this loop is terminated. But, if a loop has a capture and some complex condition is satisfied, this causes backtracking. This behavior invokes unexpected results, for example, ```ruby p /(?:.B.(?<a>(?:[C-Z]|.)*)+){2}/ =~ "ABCABC" # => nil p /(?:.B.(?:(?:[C-Z]|.)*)+){2}/ =~ "ABCABC" # => 0 ``` Because the above regex has a capture and the below does not, different matching results are returned. It is not very intuitive that the presence of a capture changes the matching result. The detailed condition for changing the null-loop behavior is 1) a previous capture in this loop holds the empty string, and 2) this capture's position is different from the current matching position. This condition is checked in `STACK_NULL_CHECK_MEMST` (https://github.com/ruby/ruby/blob/bbb7ab906ec64b963bd4b5d37e47b14796d64371/…) Perhaps, you cannot understand what this condition means. Don't worry, I also cannot understand. This condition has been introduced for at least 20 years, and no one may remember the reason for this necessity. (If you know, please tell me!) Even if there is a reason, I believe that there is no reasonable authority for allowing counter-intuitive behavior, such as the above example. This behavior can also cause memoization to be buggy. Memoization relies on the fact that backtracking only depends on positions and states (byte-code offsets of a regex). However, this condition additionally refers to captures, and the memoization is broken. My proposal is to **correct this inconsistent behavior**. Specifically, a null loop should be determined solely on the basis of whether the matching position has changed, without referring to captures. This fix changes the behavior of regex matching, but I believe that the probability that this will actually cause backward compatibility problems is remarkably low. This is because I have never seen any mention of this puzzling behavior before. -- https://bugs.ruby-lang.org/

4 9

[ruby-core:116016] [Ruby master Bug#20150] Memory leak in grapheme clusters
by peterzhu2118 (Peter Zhu) 14 Mar '24

14 Mar '24

Issue #20150 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20150: Memory leak in grapheme clusters https://bugs.ruby-lang.org/issues/20150 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9414 String#grapheme_cluters and String#each_grapheme_cluster leaks memory because if the string is not UTF-8, then the created regex will not be freed. For example: ```ruby str = "hello world".encode(Encoding::UTF_32LE) 10.times do 1_000.times do str.grapheme_clusters end puts `ps -o rss= -p #{$$}` end ``` Before: ``` 26000 42256 59008 75792 92528 109232 125936 142672 159392 176160 ``` After: ``` 9264 9504 9808 10000 10128 10224 10352 10544 10704 10896 ``` -- https://bugs.ruby-lang.org/

5 6

[ruby-core:115912] [Ruby master Bug#20090] Anonymous arguments are now syntax errors in unambiguous cases
by willcosgrove (Will Cosgrove) 14 Mar '24

14 Mar '24

Issue #20090 has been reported by willcosgrove (Will Cosgrove). ---------------------------------------- Bug #20090: Anonymous arguments are now syntax errors in unambiguous cases https://bugs.ruby-lang.org/issues/20090 * Author: willcosgrove (Will Cosgrove) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- It looks like the changes that were made in #19370 may have gone further than intended. It's also possible I'm misunderstanding what decision was made. But it was my understanding that the goal was to make ambiguous cases a syntax error. The test cases added are all testing the ambiguous cases: ```rb assert_syntax_error("def b(&) ->(&) {c(&)} end", /anonymous block parameter is also used/) # ... assert_syntax_error("def b(*) ->(*) {c(*)} end", /anonymous rest parameter is also used/) assert_syntax_error("def b(a, *) ->(*) {c(1, *)} end", /anonymous rest parameter is also used/) assert_syntax_error("def b(*) ->(a, *) {c(*)} end", /anonymous rest parameter is also used/) # ... assert_syntax_error("def b(**) ->(**) {c(**)} end", /anonymous keyword rest parameter is also used/) assert_syntax_error("def b(k:, **) ->(**) {c(k: 1, **)} end", /anonymous keyword rest parameter is also used/) assert_syntax_error("def b(**) ->(k:, **) {c(**)} end", /anonymous keyword rest parameter is also used/) ``` However it is now also producing syntax errors in all of these cases: ```rb def b(&) -> { c(&) } end def b(*) -> { c(*) } end def b(a, *) -> { c(1, *) } end def b(*) ->(a) { c(a, *) } end def b(**) -> { c(**) } end def b(k:, **) -> { c(k: 1, **) } end def b(**) ->(k:) { c(k:, **) } end ``` Again, it's possible I misunderstood the scope of the previous change. But it would be sad to lose the unambiguous case, as I've used that pattern quite a bit in my own projects. This is my first time opening an issue here, so I apologize in advance if I've done anything non-standard. -- https://bugs.ruby-lang.org/

6 7

[ruby-core:116356] [Ruby master Bug#20198] Threaded DNS resolver does not propagate errno to the calling thread
by kjtsanaktsidis (KJ Tsanaktsidis) 14 Mar '24

14 Mar '24

Issue #20198 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20198: Threaded DNS resolver does not propagate errno to the calling thread https://bugs.ruby-lang.org/issues/20198 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- If we get a return value of `EAI_SYSTEM` from `getaddrinfo`, we transform that into an appropriate `Errno::` exception on the Ruby side. However, because we now run the actual call to `getaddrinfo` in a thread, we lose that `errno` value (because `errno` is thread-local). So, what we actually raise in case of `EAI_SYSTEM` is just the last error which happened on the calling thread - e.g. this `ECHILD` which presumably got set in the bowels of pthreads somewhere: ``` 1) Socket::IPSocket#getaddress raises an error on unknown hostnames ERROR Expected SocketError but got: Errno::ECHILD (No child processes - getaddrinfo) /home/runner/work/ruby/ruby/src/spec/ruby/library/socket/ipsocket/getaddress_spec.rb:22:in `getaddress' /home/runner/work/ruby/ruby/src/spec/ruby/library/socket/ipsocket/getaddress_spec.rb:22:in `block (3 levels) in <top (required)>' /home/runner/work/ruby/ruby/src/spec/ruby/library/socket/ipsocket/getaddress_spec.rb:21:in `block (2 levels) in <top (required)>' /home/runner/work/ruby/ruby/src/spec/ruby/library/socket/ipsocket/getaddress_spec.rb:4:in `<top (required)>' ``` -- https://bugs.ruby-lang.org/

2 3

[ruby-core:116073] [Ruby master Bug#20161] Memory leak in regexp grapheme clusters
by peterzhu2118 (Peter Zhu) 14 Mar '24

14 Mar '24

Issue #20161 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20161: Memory leak in regexp grapheme clusters https://bugs.ruby-lang.org/issues/20161 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/9447 The cc->mbuf gets overwritten, so we need to free it to not leak memory. For example: ```ruby str = "hello world".encode(Encoding::UTF_32LE) 10.times do 1_000.times do str.grapheme_clusters end puts `ps -o rss= -p #{$$}` end ``` Before: ``` 15536 15760 15920 16144 16304 16480 16640 16784 17008 17280 ``` After: ``` 15584 15584 15760 15824 15888 15888 15888 15888 16048 16112 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:116489] [Ruby master Bug#20224] Backport https://github.com/ruby/ruby/pull/9634 to Ruby 3.3
by yui-knk (Kaneko Yuichiro) 14 Mar '24

14 Mar '24

Issue #20224 has been reported by yui-knk (Kaneko Yuichiro). ---------------------------------------- Bug #20224: Backport https://github.com/ruby/ruby/pull/9634 to Ruby 3.3 https://bugs.ruby-lang.org/issues/20224 * Author: yui-knk (Kaneko Yuichiro) * Status: Closed * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- It's nice if we can backport https://github.com/ruby/ruby/pull/9634 (3d19409637de1462b6790d2a92344bf0a10d8c52) to Ruby 3.3 branch. Even so previous codes work as expected but it depends on how parse.c works and misleading. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:113966] [Ruby master Bug#19742] Introduce `Module#anonymous?`
by ioquatix (Samuel Williams) 12 Mar '24

12 Mar '24

Issue #19742 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Bug #19742: Introduce `Module#anonymous?` https://bugs.ruby-lang.org/issues/19742 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- As a follow-on <from https://bugs.ruby-lang.org/issues/19521>, I'd like propose we introduce `Module#anonymous?`. In some situations, like logging/formatting, serialisation/deserialization, debugging or meta-programming, we might like to know if a class is a proper constant or not. However, this brings about some other issues which might need to be discussed. After assigning a constant, then removing it, the internal state of Ruby still believes that the class name is permanent, even thought it's no longer true. e.g. ``` m = Module.new m.anonymous? # true M = m m.anonyomous # false Object.send(:remove_const, :M) M # uninitialized constant M (NameError) M.anonymous? # false ``` Because RCLASS data structure is not updated after the constant is removed, internally the state still has a "permanent class name". I want to use this proposal to discuss this issue and whether there is anything we should do about such behaviour (or even if it's desirable). Proposed PR: https://github.com/ruby/ruby/pull/7966 cc @fxn -- https://bugs.ruby-lang.org/

9 46

[ruby-core:115925] [Ruby master Bug#20096] Ruby 3.2.2 win32/registry: Junk appended to Windows Registry String Value
by jay4rubydev (Jay M) 07 Mar '24

07 Mar '24

Issue #20096 has been reported by jay4rubydev (Jay M). ---------------------------------------- Bug #20096: Ruby 3.2.2 win32/registry: Junk appended to Windows Registry String Value https://bugs.ruby-lang.org/issues/20096 * Author: jay4rubydev (Jay M) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mswin64_140] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Ruby Version: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mswin64_140] Compiler: MSVC 2019 - Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30147 for x64. Issue: win32/registry adds junk to Windows Registry String Value Code: require 'win32/registry' win_oracle_key = "SOFTWARE\\MicroSoft" reg=Win32::Registry::HKEY_LOCAL_MACHINE.open(win_oracle_key, Win32::Registry::KEY_ALL_ACCESS) inst_loc_key = "inst_loc" inv_dir="C:\\Program Files\\Tester\\ModuleInfo" reg[inst_loc_key] = inv_dir Result: Registry contains: C:\Program Files\Tester\ModuleInfo爀 Observation: Looks like memory overread Expected Result - without the junk: C:\Program Files\Tester\ModuleInfo After changing the code in registry.rb: From: def write(name, type, data) termsize = 0 case type when REG_SZ, REG_EXPAND_SZ data = data.encode(WCHAR) termsize = WCHAR_SIZE To: def write(name, type, data) termsize = 0 case type when REG_SZ, REG_EXPAND_SZ enc_data = data.encode(WCHAR) # Add NULL WCHAR for string data and don't set the termsize because # enc_data.bytesize will now include the size of the NULL character. enc_data += WCHAR_NUL # termsize = WCHAR_SIZE ... -- https://bugs.ruby-lang.org/

4 7

[ruby-core:115892] [Ruby master Bug#20085] Fiber.new{ }.resume causes Segmentation fault for Ruby 3.3.0 on aarch64-linux
by oleksii (Oleksii Leonov) 29 Feb '24

29 Feb '24

Issue #20085 has been reported by oleksii (Oleksii Leonov). ---------------------------------------- Bug #20085: Fiber.new{ }.resume causes Segmentation fault for Ruby 3.3.0 on aarch64-linux https://bugs.ruby-lang.org/issues/20085 * Author: oleksii (Oleksii Leonov) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ``` ruby -e "Fiber.new{}.resume" 0.170 -e:1: [BUG] Segmentation fault at 0x0036ffffb4f110f0 0.170 ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux] 0.170 0.170 -- Control frame information ----------------------------------------------- 0.170 c:0003 p:---- s:0010 e:000009 CFUNC :resume 0.170 c:0002 p:0007 s:0006 E:0005e0 EVAL -e:1 [FINISH] 0.170 c:0001 p:0000 s:0003 E:000b50 DUMMY [FINISH] 0.170 0.170 -- Ruby level backtrace information ---------------------------------------- 0.170 -e:1:in `<main>' 0.170 -e:1:in `resume' 0.170 0.170 -- Threading information --------------------------------------------------- 0.170 Total ractor count: 1 0.170 Ruby thread count for this ractor: 1 0.170 0.170 -- Machine register context ------------------------------------------------ 0.170 x0: 0x0000aaab11b4f570 x1: 0x0000aaab11de4a40 x2: 0x0000ffffe5571990 0.170 x3: 0x0000ffff9ac5ff60 x4: 0x0000ffff9ac60018 x5: 0x0000ffff9ac80000 0.170 x6: 0x0000ffffb5458b88 x7: 0x0000000000000000 x18: 0x00000000007fffff 0.170 x19: 0x0000000000000000 x20: 0x0000000000000000 x21: 0x0000000000000000 0.170 x22: 0x0000000000000000 x23: 0x0000000000000000 x24: 0x0000000000000000 0.170 x25: 0x0000000000000000 x26: 0x0000000000000000 x27: 0x0000000000000000 0.170 x28: 0x0000000000000000 x29: 0x0000000000000000 sp: 0x0000ffff9ac60000 0.170 fau: 0x0036ffffb4f110f0 0.170 0.170 -- C level backtrace information ------------------------------------------- 0.171 Segmentation fault ``` - Ruby 3.3.0-rc1 and 3.2.2 work without Segmentation fault on both amd64 and arm64. - Ruby 3.3.0 works on amd64, but fails with Segmentation fault on arm64 (aarch64-linux) on Ubuntu 22.04 and Debian Bookworm. Dockerfile to reproduce: ``` FROM debian:bookworm # Set UTF-8 locale by default. ENV RBENV_ROOT=/usr/local/rbenv \ PATH=/usr/local/rbenv/bin:/usr/local/rbenv/shims:$PATH RUN ( \ export DEBIAN_FRONTEND=noninteractive \ && apt-get update \ && apt-get install -y --no-install-recommends \ ca-certificates curl git \ # standard dependencies for rbenv autoconf \ patch \ build-essential \ rustc \ libssl-dev \ libyaml-dev \ libreadline6-dev \ zlib1g-dev \ libgmp-dev \ libncurses5-dev \ libffi-dev \ libgdbm6 \ libgdbm-dev \ libdb-dev \ uuid-dev \ ) ENV RBENV_VERSION=3.3.0 RUN ( \ export DEBIAN_FRONTEND=noninteractive \ # Install rbenv & ruby-build && git clone https://github.com/rbenv/rbenv.git /usr/local/rbenv \ && git clone https://github.com/rbenv/ruby-build.git /usr/local/rbenv/plugins/ruby-build \ && /usr/local/rbenv/plugins/ruby-build/install.sh \ && echo 'export RBENV_ROOT=/usr/local/rbenv' >> /etc/profile.d/rbenv.sh \ && echo 'export PATH=/usr/local/rbenv/bin:$PATH' >> /etc/profile.d/rbenv.sh \ && echo 'eval "$(rbenv init -)"' >> /etc/profile.d/rbenv.sh \ && echo 'export RBENV_ROOT=/usr/local/rbenv' >> /root/.bashrc \ && echo 'export PATH=/usr/local/rbenv/bin:$PATH' >> /root/.bashrc \ && echo 'eval "$(rbenv init -)"' >> /root/.bashrc \ && eval "$(rbenv init -)"; rbenv install $RBENV_VERSION \ && eval "$(rbenv init -)"; rbenv global $RBENV_VERSION \ ) RUN RUN ruby -e "Fiber.new{}.resume" ``` `docker build --platform linux/arm64 .` ---Files-------------------------------- Dockerfile (1.53 KB) -- https://bugs.ruby-lang.org/

13 26

[ruby-core:116387] [Ruby master Bug#20206] PTY.spawn seems to fail to capture the output of "echo foo" once in a while
by lacostej (Jerome Lacoste) 26 Feb '24

26 Feb '24

Issue #20206 has been reported by lacostej (Jerome Lacoste). ---------------------------------------- Bug #20206: PTY.spawn seems to fail to capture the output of "echo foo" once in a while https://bugs.ruby-lang.org/issues/20206 * Author: lacostej (Jerome Lacoste) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- We use PTY.spawn to call "echo foo", and on Mac it seems to randomly fail, capturing an empty output every now and then. On Linux, the failure doesn't seem to happen. The following code 1. contains 2 ways of capturing the output from PTY.spawn. Both seem to show the same issue (`run_command` and `run_command2`) 2. invokes the external `stress` program. This helps to trigger the issue more often. ``` ruby require 'pty' require 'expect' def run_command(command) output = [] PTY.spawn(command) do |command_stdout, command_stdin, pid| begin command_stdout.each do |l| line = l.chomp output << line end rescue Errno::EIO # This is expected on some linux systems, that indicates that the subcommand finished # and we kept trying to read, ignore it ensure command_stdout.close command_stdin.close Process.wait(pid) end end raise "#{$?.exited?} #{$?.stopped?} #{$?.signaled?} - #{$?.stopsig} - #{$?.termsig} -" unless $?.exitstatus == 0 [$?.exitstatus, output.join("\n")] end def run_command2(command) output = [] PTY.spawn(command) do |command_stdout, command_stdin, pid| output = "" begin a = command_stdout.expect(/foo.*/, 5) output = a[0] if a ensure command_stdout.close command_stdin.close Process.wait(pid) end end raise "#{$?.exited?} #{$?.stopped?} #{$?.signaled?} - #{$?.stopsig} - #{$?.termsig} -" unless $?.exitstatus == 0 [$?.exitstatus, output] end def test_spawn(command) status, output = run_command(command) errors = [] errors << "status was '#{status}'" unless status == 0 errors << "output was '#{output}'" unless output == "foo" raise errors.join(" - ") unless errors.empty? end t = nil pid = nil if ENV['STRESS'] t = Thread.new do |t| puts "Spawning stress" pid = spawn("stress -c 16 -t 99", pgroup: true) puts "Waiting #{pid}" Process.wait(pid) puts "#{pid} DONE" end end command = "echo foo" if ARGV.count == 1 command = ARGV[0] end puts "Will run command: '#{command}'" errors = 0 2000.times do |i| begin test_spawn(command) rescue => e puts "ERROR #{i}: #{e}" errors += 1 end end if t begin Process.kill(:SIGKILL, -pid) rescue Errno::ESRCH # already dead, ignore end t.join end raise "Failed #{errors} times" unless errors == 0 ``` Here are some ways of reproducing the issue ``` ruby test_pty.rb STRESS=y ruby test_pty.rb ``` Here's an example of how it fails on circleci. https://app.circleci.com/pipelines/github/lacostej/cienvs/33/workflows/d6d8… Tested on ruby 2.6 to ruby 3.3.0 on Mac. -- https://bugs.ruby-lang.org/

1 1