June 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117210] [Ruby master Bug#20344] argument stack underflow (-1)
by nobu (Nobuyoshi Nakada) 02 Sep '24

02 Sep '24

Issue #20344 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20344: argument stack underflow (-1) https://bugs.ruby-lang.org/issues/20344 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- This file produces argument stack underflow, since ruby 2.5. ```ruby proc do next if true case nil when "a" next when "b" when "c" proc {} end next unless true end ``` ``` -- raw disasm-------- trace: 100 0000 nop ( 1) <L000> [sp: 0, unremovable: 1, refcnt: 5] trace: 1 <L004> [sp: 0, unremovable: 0, refcnt: 1] adjust: [label: 0] 0001 putnil ( 2) 0002 leave ( 13) <L008> [sp: 1, unremovable: 0, refcnt: 1] adjust: [label: 0] 0003 putnil ( 6) 0004 leave ( 13) adjust: [label: 8] 0005 jump <L005> ( 5) 0007 pop ( 7) 0008 jump <L005> ( 7) * 0010 pop ( 8) trace: 1 <L011> [sp: -1, unremovable: 1, refcnt: 1] 0011 putself ( 9) 0012 send <calldata:proc, 0>, nil ( 9) <L012> [sp: -1, unremovable: 0, refcnt: 2] 0015 pop ( 9) <L005> [sp: 1, unremovable: 0, refcnt: 2] trace: 1 0016 putnil ( 12) <L001> [sp: -1, unremovable: 0, refcnt: 3] trace: 200 0017 leave ( 13) --------------------- ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118270] [Ruby master Bug#20571] Compile error on for loop
by kddnewton (Kevin Newton) 02 Sep '24

02 Sep '24

Issue #20571 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20571: Compile error on for loop https://bugs.ruby-lang.org/issues/20571 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby for _ in 1.5 or _ in [] end ``` raises a compile error for the current compiler only with ``` -- raw disasm-------- trace: 1 <L000> [sp: 0, unremovable: 1, refcnt: 1] 0000 putobject 1.5 ( 1) 0002 jump <L002> ( 1) <L006> [sp: 1, unremovable: 1, refcnt: 1] <L004> [sp: 1, unremovable: 0, refcnt: 0] 0004 pop ( 1) * 0005 pop ( 1) 0006 putobject false ( 1) <L002> [sp: 1, unremovable: 0, refcnt: 1] 0008 send <calldata:each, 0>, nil ( 1) <L001> [sp: -1, unremovable: 0, refcnt: 2] 0011 leave ( 1) --------------------- test.rb:1: argument stack underflow (-1) test.rb: compile error (SyntaxError) ``` -- https://bugs.ruby-lang.org/

4 4

[ruby-core:118346] [Ruby master Bug#20586] Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted
by ivoanjo (Ivo Anjo) 22 Aug '24

22 Aug '24

Issue #20586 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #20586: Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted https://bugs.ruby-lang.org/issues/20586 * Author: ivoanjo (Ivo Anjo) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- #### Background Hey! I work for Datadog on the Ruby profiler part of the [`datadog` (previously `ddtrace`)](https://github.com/datadog/dd-trace-rb) gem. A customer reached [out with an issue](https://github.com/DataDog/dd-trace-rb/issues/3450) where enabling the profiler made `Dir.glob` return no files for a given folder: Without profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => ["/gcsfuse/test.html", "/gcsfuse/test.txt"] ``` With profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => [] ``` It turns out the issue is related to missing error handling in `dir.c`. The Datadog Ruby profiler, like stackprof, pf2 and vernier, uses unix signals to interrupt the currently-active thread and take a sample (usually `SIGPROF`). When some system calls get interrupted by a signal, they return an [EINTR error code](https://man7.org/linux/man-pages/man7/signal.7.html#:~:text=Interrupt… back to the caller. Consider for instance the implementation of `dir_each_entry` in `dir.c`: ```c static VALUE dir_each_entry(VALUE dir, VALUE (*each)(VALUE, VALUE), VALUE arg, int children_only) { struct dir_data *dirp; struct dirent *dp; IF_NORMALIZE_UTF8PATH(int norm_p); GetDIR(dir, dirp); rewinddir(dirp->dir); IF_NORMALIZE_UTF8PATH(norm_p = need_normalization(dirp->dir, RSTRING_PTR(dirp->path))); while ((dp = READDIR(dirp->dir, dirp->enc)) != NULL) { // ... do things } return dir; } ``` If `READDIR` returns `NULL`, then `dir_each_entry` assumes it has iterated the entire directory. But looking [at the man page for `readdir`](https://man7.org/linux/man-pages/man3/readdir.3.html) we see the following sharp edge (emphasis mine): > It returns NULL on reaching the end of the directory stream **or if an error occurred**. So what's happening in this situation is: `readdir` gets interrupted, returns `NULL` + sets errno to `EINTR`. But `dir_each_entry` doesn't check `errno`, so rather than raising an exception to flag the issue, it treats it as if the end of the directory has been reached. #### How to reproduce Reproducing this is somewhat annoying, because it's dependent on timing: the signal must arrive at the exact time the dir API is getting executed. I was able to reproduce this every time by using the [google cloud `gcsfuse`](https://cloud.google.com/storage/docs/gcs-fuse) tool. This somewhat makes sense -- a remote filesystem is much slower than a local one, so there's a much bigger window of opportunity for a signal to arrive while the system call is blocked. Here's an example I included in https://github.com/DataDog/dd-trace-rb/pull/3720: ``` # Not shown: Set up a trial google cloud account, install gcsfuse, create a cloud storage bucket and put in some test files $ gcsfuse test_fs_dd_trace_rb fuse-testing/ $ ls fuse-testing/ hello.txt test.html test.txt # Not shown: Add `datadog` gem to `Gemfile` $ DD_PROFILING_ENABLED=true DD_PROFILING_DIR_INTERRUPTION_WORKAROUND_ENABLED=false bundle exec ddprofrb exec ruby -e "Datadog::Profiling.wait_until_running; pp Dir.children('fuse-testing/')" [] ``` Let me know if you'd like me to try to create a reproducer that does not depend on the `datadog` gem. #### Additional notes I've spent quite some time looking at the `dir.c` sources, and here's the full list of APIs that suffer from issues: * `dir_each_entry` does not check `errno`; all of its users have interruption bugs * `dir_tell` will return -1 instead of the correct position (which means that passing -1 to `dir_seek`/`dir_set_pos` will cause it to not list the directory properly) * `do_opendir` an error in system calls will only sometimes be turned into a raised exception * Indirect callers that pass in rb_glob_error as errfunc: rb_glob, Dir.[], Dir.glob * Indirect callers that pass in 0 as errfunc: ruby_glob, ruby_brace_glob * `glob_opendir` does not check errno; all of its users have interruption bugs * `glob_getent` does not check errno; all of its users have interruption bugs * `nogvl_dir_empty_p` does not check errno (of readdir! it actually checks for opendir); all of its users have interruption bugs Less sure about these: * `do_stat`/`do_lstat` will turn errors into warnings (unclear if enabled or disabled by default) * `need_normalization` calls `fgetattrlist` / `getattrlist` and all errors `(ret != 0)` are treated in the same way * `rb_glob_error` is and `rb_glob_caller` leave exceptions as pending and rely on callers to raise them properly * Error handling of `rb_home_dir_of` and `rb_default_home_dir` are a bit suspicious As a workaround in the Datadog Ruby profiler, in https://github.com/DataDog/dd-trace-rb/pull/3720 I've added monkey patches to all of the Ruby-level APIs that use the above functions and mask out `SIGPROF` so these calls are never interrupted. This solution is does successfully work around the issue, although it prevents the profiler from sampling during these system calls, which will mean less visibility if e.g. these calls are taking a long time. And well, maintaining monkey patches is always problematic for future Ruby compatibility. -- https://bugs.ruby-lang.org/

5 12

[ruby-core:117882] [Ruby master Bug#20490] Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process
by stanhu (Stan Hu) 18 Aug '24

18 Aug '24

Issue #20490 has been reported by stanhu (Stan Hu). ---------------------------------------- Bug #20490: Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process https://bugs.ruby-lang.org/issues/20490 * Author: stanhu (Stan Hu) * Status: Open * ruby -v: ruby 3.2.4 (2024-04-23 revision af471c0e01) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is a follow-up issue for a bug that I thought was fixed in https://bugs.ruby-lang.org/issues/19837 and duplicated in https://bugs.ruby-lang.org/issues/20181. The following script doesn't terminate quickly in Ruby 3.1.5 and 3.2.4, even with the patches to address https://bugs.ruby-lang.org/issues/19837. It works fine in Ruby 3.3. It appears that the `Process::WNOHANG` argument passed to `Process.wait2` causes this script to spin until the child process stops: ```ruby #!/bin/env ruby Process.spawn({}, "sh -c 'sleep 600'").tap do |pid| puts "detaching PID #{pid}" Process.detach(pid) end forked_pid = fork do loop { sleep 1 } end child_waiter = Thread.new do puts "Waiting for child process to die..." # The spawned process has to exit before this returns in Ruby 3.1 and 3.2 loop do pid, status = Process.wait2(-1, Process::WNOHANG) puts "Exited PID: #{pid}, status: #{status}" break if pid sleep 1 end end process_killer = Thread.new do puts "Killing #{forked_pid}" system("kill #{forked_pid}") end child_waiter.join process_killer.join ``` If I drop the `Process::WNOHANG` argument, it works fine. -- https://bugs.ruby-lang.org/

3 6

[ruby-core:118012] [Ruby master Misc#20509] Document importance of #to_ary and #to_hash for Array#== and Hash#==
by gettalong (Thomas Leitner) 01 Aug '24

01 Aug '24

Issue #20509 has been reported by gettalong (Thomas Leitner). ---------------------------------------- Misc #20509: Document importance of #to_ary and #to_hash for Array#== and Hash#== https://bugs.ruby-lang.org/issues/20509 * Author: gettalong (Thomas Leitner) * Status: Open ---------------------------------------- Both `Array#==` and `Hash#==` provide special behaviour in case the `other` argument is not an Array/Hash but defines the special `#to_ary`/`#to_hash` methods. Those methods are never called, they are just checked for existence. And if they exist, `other#==` is called to allow the `other` argument to decide whether the two objects are equal. I think this is worth mentioning in the documentation for `Array#==` and `Hash#==`. [Background: In my PDF library HexaPDF I have defined two classes `PDFArray` and `Dictionary` which act like Array and Hash but provide special PDF specific behaviour. For PDFArray I defined the `#to_ary` method but for Dictionary just the `#to_h` method. I have come across a bug where comparing Arrays with PDFArrays just works as it should be but comparing Hashes with Dictionaries doesn't due to the absence of `#to_hash` (it seems I removed `Dictionary#to_hash` in 2017 due to problems with automatic destructuring when passing a Dictionary as argument; from what I see that should be no problem anymore, so I will just add it back).] -- https://bugs.ruby-lang.org/

4 3

[ruby-core:118010] [Ruby master Feature#20508] Explicit access to *, **, &, and ...
by bradgessler (Brad Gessler) 01 Aug '24

01 Aug '24

Issue #20508 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Feature #20508: Explicit access to *, **, &, and ... https://bugs.ruby-lang.org/issues/20508 * Author: bradgessler (Brad Gessler) * Status: Open ---------------------------------------- I find debugging and certain meta-programming tasks challenging because there's no explicit APIs for accessing certain types of arguments in Ruby. Here's some example code: ```ruby def splats(one, *, two: nil, **, &) # These work p(*) p(**) # But the block doesn't show a proc. p(&) # This would show [:one, :two, :_] p binding.local_variables end splats(:arg, two: true) do 42 end ``` I'm not sure how to access the `&` variable, unless I name it. The same applies to endless ruby methods. I'd like to see a way to explicitly call bindings for those methods. Something like this (not sure if binding is the right place for it): ```ruby def splats(one, *, true: nil, **, &) binding.arguments # Returns an array binding.keyword_arguments # Returns a hash binding.block_argument # Returns a block end ``` These methods would give me access to the values used to call the method. -- https://bugs.ruby-lang.org/

5 4

[ruby-core:111272] [Ruby master Bug#19231] Integer#step and Float::INFINITY - inconsistent behaviour when called with and without a block
by andrykonchin (Andrew Konchin) 01 Aug '24

01 Aug '24

Issue #19231 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #19231: Integer#step and Float::INFINITY - inconsistent behaviour when called with and without a block https://bugs.ruby-lang.org/issues/19231 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal * ruby -v: 3.1.2 * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- The initial issue was reported here https://github.com/oracle/truffleruby/issues/2797. `0.step(Float::INFINITY, 10)` returns: - `Integers` when called with a block - `Floats` when called without a block I would expect `Floats` to be returned in both cases. Examples: ```ruby 0.step(100.0, 10).take(1).map(&:class) # => [Float] ``` ```ruby 0.step(Float::INFINITY, 10) { |offset| p offset.class; break } # Integer ``` When `to` argument is a finite `Float` value then calling with a block returns `Floats` as well: ```ruby 0.step(100.0, 10) { |offset| p offset.class; break } # Float ``` Wondering whether it's intentional behaviour. I've found a related issue https://bugs.ruby-lang.org/issues/15518. -- https://bugs.ruby-lang.org/

6 6

[ruby-core:118237] [Ruby master Bug#20563] Drop support for old Windows
by nobu (Nobuyoshi Nakada) 30 Jul '24

30 Jul '24

Issue #20563 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20563: Drop support for old Windows https://bugs.ruby-lang.org/issues/20563 * Author: nobu (Nobuyoshi Nakada) * Status: Assigned * Assignee: usa (Usaku NAKAMURA) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is the table of APIs that we are calling dynamically, and the available Windows versions. DLL | API | Client | Server | ---------|--------------------------------|--------|--------| kernel32 | CreateSymbolicLinkW | Vista | 2008 | kernel32 | GetFileInformationByHandleEx | Vista | 2008 | kernel32 | GetFinalPathNameByHandleW | Vista | 2008 | kernel32 | SetFileInformationByHandle | Vista | 2008 | shell32 | SHGetPathFromIDListEx | Vista | 2008 | ws2_32 | inet_ntop | Vista | 2008 | ws2_32 | inet_pton | Vista | 2008 | kernel32 | GetSystemTimePreciseAsFileTime | 8 | 2012 | kernel32 | SetThreadDescription | 10 | 2016 | However, already random.c directly uses `BCryptGenRandom` that is available since Windows Vista/Server 2008. So I propose explicitly dropping support for Windows older than Vista/2008. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:118109] [Ruby master Bug#20514] Open3#capture3 does not receive correct exit code from Heroku but Kernel#system does
by lakehs (Ashley Lake) 30 Jul '24

30 Jul '24

Issue #20514 has been reported by lakehs (Ashley Lake). ---------------------------------------- Bug #20514: Open3#capture3 does not receive correct exit code from Heroku but Kernel#system does https://bugs.ruby-lang.org/issues/20514 * Author: lakehs (Ashley Lake) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [arm64-darwin23] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Heroku has a command-line switch for returning the exit code from a detached process to the calling terminal via their CLI. However, Open3 doesn't appear to receive this exit code properly: ``` ruby irb(main):007> command = 'heroku run "rails db:migrate:status | grep \"down \"" --exit-code -a remote_app' => "heroku run \"rails db:migrate:status | grep \\\"down \\\"\" --exit-code -a remote_app" irb(main):008> params = {} => {} irb(main):009> stdout_str, stderr_str, status = Open3.capture3(command, params) => ["", "Running rails db:migrate:status | grep \"down \" on remote_app... provisioning, run.2070 (Shield-M)\n", #<Process::Status: pid 34912 exit 0>] irb(main):010> status => #<Process::Status: pid 34912 exit 0> irb(main):011> status.exitstatus => 0 ``` In this case, since I know that there are no migrations with status "down", grep should return an exit code of 1, not 0. Running the same command with system gives the expected result: ``` ruby irb(main):025> command = 'heroku run "rails db:migrate:status | grep \"down \"" --exit-code -a remote_app' => "heroku run \"rails db:migrate:status | grep \\\"down \\\"\" --exit-code -a remote_app" irb(main):026> res = system(command) Running rails db:migrate:status | grep "down " on ⬢ remote_app... provisioning, run.9202 (Shield-M) # intermediate output snipped › Error: Process exited with code 1 › Code: 1 => false ``` Switching "up" for "down" in the above grep statement yields `true` for exit code 0 when called with Kernel#system. This doesn't happen locally, but since Kernel#system gives the right results but Open3#capture3 does not, I don't think the problem is on Heroku's end. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:115910] [Ruby master Bug#20088] Ruby 3.3.0 does not cross-complie on arm64-darwin
by stanhu (Stan Hu) 28 Jul '24

28 Jul '24

Issue #20088 has been reported by stanhu (Stan Hu). ---------------------------------------- Bug #20088: Ruby 3.3.0 does not cross-complie on arm64-darwin https://bugs.ruby-lang.org/issues/20088 * Author: stanhu (Stan Hu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- To reproduce: ```shell docker run -it ghcr.io/rake-compiler/rake-compiler-dock-image:1.4.0.rc2-mri-arm64-darwin bash cd /tmp export CPPFLAGS='' export CFLAGS='-O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong ' export LDFLAGS='-pipe ' export CC=aarch64-apple-darwin-clang export MAKE='make V=1' rake-compiler cross-ruby VERSION=3.3.0 HOST=aarch64-apple-darwin ``` This errors out with: ```shell aarch64-apple-darwin-clang -O1 -fno-omit-frame-pointer -fno-fast-math -fstack-protector-strong -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -Wmisleading-indentation -Wundef -pipe -march=arm64 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-overflow -fvisibility=hidden -DRUBY_EXPORT -fPIE -I. -I.ext/include/arm64-darwin -I/usr/local/rake-compiler/sources/ruby-3.3.0/include -I/usr/local/rake-compiler/sources/ruby-3.3.0 -I/usr/local/rake-compiler/sources/ruby-3.3.0/prism -I/usr/local/rake-compiler/sources/ruby-3.3.0/enc/unicode/15.0.0 -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT -o main.o -c /usr/local/rake-compiler/sources/ruby-3.3.0/main.c clang: error: the clang compiler does not support '-march=arm64' make: *** [uncommon.mk:1291: main.o] Error 1 ``` This appears to be a regression due to https://github.com/ruby/ruby/pull/9250. `RUBY_DEFAULT_ARCH("$target_cpu")` is called, and this `case` statement is hit: ```ac AS_CASE([$build_os], [darwin*], [ARCH_FLAG="-arch "$1], [ARCH_FLAG=-march=$1] ``` I'm not sure this test is correct, since `$build_os` appears to be `linux-gnu`. This results in `-march=arm64` being used, which is not a valid option even in `clang` 15.0.0. Should `$target_cpu` and `$target_os` be considered instead? The `configure` output shows: ``` Configuration summary for ruby version 3.3.0 * Installation prefix: /usr/local/rake-compiler/ruby/aarch64-apple-darwin/ruby-3.3.0 * exec prefix: ${prefix} * arch: arm64-darwin * site arch: ${arch} * RUBY_BASE_NAME: ruby * enable shared: no * ruby lib prefix: ${libdir}/${RUBY_BASE_NAME} * site libraries path: ${rubylibprefix}/${sitearch} * vendor path: ${rubylibprefix}/vendor_ruby * target OS: darwin * compiler: aarch64-apple-darwin-clang * with thread: pthread * with coroutine: arm64 * enable shared libs: no * dynamic library ext: bundle * CFLAGS: -fdeclspec ${optflags} ${debugflags} ${warnflags} * LDFLAGS: -L. -pipe -fstack-protector-strong * DLDFLAGS: -pipe -Wl,-multiply_defined,suppress -Wl,-undefined,dynamic_lookup * optflags: -O3 -fno-fast-math * debugflags: -ggdb3 * warnflags: -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition \ -Wmissing-noreturn -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter \ -Wno-unused-value -Wunused-variable -Wmisleading-indentation -Wundef * strip command: aarch64-apple-darwin-strip -A -n * install doc: no * YJIT support: no * RJIT support: no * man page type: man * BASERUBY -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] ``` The `configure` statement for reference: ``` ./configure --host=aarch64-apple-darwin --target=aarch64-apple-darwin --build=x86_64-pc-linux-gnu --enable-install-static-library --disable-jit-support --disable-install-doc --with-ext= --enable-static --disable-shared --prefix=/usr/local/rake-compiler/ruby/aarch64-apple-darwin/ruby-3.3.0 ``` This was first reported in a CI failure in https://github.com/rake-compiler/rake-compiler-dock/pull/112#issuecomment-1…. -- https://bugs.ruby-lang.org/

3 3