July 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118535] [Ruby master Feature#20625] Object#chain_of
by matheusrich (Matheus Richard) 24 Aug '24

24 Aug '24

Issue #20625 has been reported by matheusrich (Matheus Richard). ---------------------------------------- Feature #20625: Object#chain_of https://bugs.ruby-lang.org/issues/20625 * Author: matheusrich (Matheus Richard) * Status: Open ---------------------------------------- ## Motivation It's often common to traverse a tree/list-like structure in order to get a chain of elements. This proposal is to add a method to `Object` that allows collecting a chain of elements by applying a block to each element. It doesn't require the root element to be an instance of a specific class or respond to a specific protocol. I think this method could be useful in many cases, since hierarchies like this are common in codebases (e.g. file systems, organizations structures, commit histories, breadcrumbs in web apps, configuration hierarchies, etc.). Here are some examples extracted from real codebases (simplified for the sake of the example/privacy): ```rb # Given a file system structure, get the breadcrumbs of a file or directory def breadcrumbs(root) crumbs = [] current = root while current crumbs << current current = current.parent_dir end crumbs end # Given an employee, get the hierarchy of managers def hierarchy(employee) hierarchy = [] current = employee while current hierarchy << current current = current.manager end hierarchy end ``` ## Implementation The implementation in Ruby could look like this: ```rb class Object def chain_of(&block) chain = [] current = self while current chain << current current = block.call(current) end chain end end ``` Here's an example use: ```rb class ListNode attr_accessor :value, :parent def initialize(value, parent = nil) @value = value @parent = parent end def ancestors chain_of(&:parent).shift end end root = ListNode.new("root") child1 = ListNode.new("child1", root) child2 = ListNode.new("child2", child1) puts child2.ancestors.map(&:value) # => ["child1", "root"] ``` The examples from the motivation section could be rewritten as: ```rb breadcrumbs = root.chain_of(&:parent_directory) hierarchy = employee.chain_of(&:manager) ``` ## Considerations - While I'm including the object by default in the chain, it could be more intuitive to exclude it. In any case, it's easy to remove or add it with `shift`/`unshift`. - On a different note, the method could be named differently (I do like `chain_of`, though). Some alternatives I've considered are `map_chain`, `traverse`, and `trace_path`. - The method assumes that the traversal will finish at some point. If the user has a cyclic structure, it will loop indefinitely. We could stop looping if we find the same element twice. I don't think it's worth the extra complexity. - I'm not sure `Object` is the best place for this method. While it's very general, I think it gives power to the user to decide how to traverse a chain like this without having to rely on a specific class. Maybe a mixin (`Traversable`/`Chainable`) would be more appropriate? Could this fit in `Enumerable`, somehow? Of course, I'm open to suggestions and feedback. Thanks for reading! -- https://bugs.ruby-lang.org/

3 4

[ruby-core:118346] [Ruby master Bug#20586] Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted
by ivoanjo (Ivo Anjo) 22 Aug '24

22 Aug '24

Issue #20586 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #20586: Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted https://bugs.ruby-lang.org/issues/20586 * Author: ivoanjo (Ivo Anjo) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- #### Background Hey! I work for Datadog on the Ruby profiler part of the [`datadog` (previously `ddtrace`)](https://github.com/datadog/dd-trace-rb) gem. A customer reached [out with an issue](https://github.com/DataDog/dd-trace-rb/issues/3450) where enabling the profiler made `Dir.glob` return no files for a given folder: Without profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => ["/gcsfuse/test.html", "/gcsfuse/test.txt"] ``` With profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => [] ``` It turns out the issue is related to missing error handling in `dir.c`. The Datadog Ruby profiler, like stackprof, pf2 and vernier, uses unix signals to interrupt the currently-active thread and take a sample (usually `SIGPROF`). When some system calls get interrupted by a signal, they return an [EINTR error code](https://man7.org/linux/man-pages/man7/signal.7.html#:~:text=Interrupt… back to the caller. Consider for instance the implementation of `dir_each_entry` in `dir.c`: ```c static VALUE dir_each_entry(VALUE dir, VALUE (*each)(VALUE, VALUE), VALUE arg, int children_only) { struct dir_data *dirp; struct dirent *dp; IF_NORMALIZE_UTF8PATH(int norm_p); GetDIR(dir, dirp); rewinddir(dirp->dir); IF_NORMALIZE_UTF8PATH(norm_p = need_normalization(dirp->dir, RSTRING_PTR(dirp->path))); while ((dp = READDIR(dirp->dir, dirp->enc)) != NULL) { // ... do things } return dir; } ``` If `READDIR` returns `NULL`, then `dir_each_entry` assumes it has iterated the entire directory. But looking [at the man page for `readdir`](https://man7.org/linux/man-pages/man3/readdir.3.html) we see the following sharp edge (emphasis mine): > It returns NULL on reaching the end of the directory stream **or if an error occurred**. So what's happening in this situation is: `readdir` gets interrupted, returns `NULL` + sets errno to `EINTR`. But `dir_each_entry` doesn't check `errno`, so rather than raising an exception to flag the issue, it treats it as if the end of the directory has been reached. #### How to reproduce Reproducing this is somewhat annoying, because it's dependent on timing: the signal must arrive at the exact time the dir API is getting executed. I was able to reproduce this every time by using the [google cloud `gcsfuse`](https://cloud.google.com/storage/docs/gcs-fuse) tool. This somewhat makes sense -- a remote filesystem is much slower than a local one, so there's a much bigger window of opportunity for a signal to arrive while the system call is blocked. Here's an example I included in https://github.com/DataDog/dd-trace-rb/pull/3720: ``` # Not shown: Set up a trial google cloud account, install gcsfuse, create a cloud storage bucket and put in some test files $ gcsfuse test_fs_dd_trace_rb fuse-testing/ $ ls fuse-testing/ hello.txt test.html test.txt # Not shown: Add `datadog` gem to `Gemfile` $ DD_PROFILING_ENABLED=true DD_PROFILING_DIR_INTERRUPTION_WORKAROUND_ENABLED=false bundle exec ddprofrb exec ruby -e "Datadog::Profiling.wait_until_running; pp Dir.children('fuse-testing/')" [] ``` Let me know if you'd like me to try to create a reproducer that does not depend on the `datadog` gem. #### Additional notes I've spent quite some time looking at the `dir.c` sources, and here's the full list of APIs that suffer from issues: * `dir_each_entry` does not check `errno`; all of its users have interruption bugs * `dir_tell` will return -1 instead of the correct position (which means that passing -1 to `dir_seek`/`dir_set_pos` will cause it to not list the directory properly) * `do_opendir` an error in system calls will only sometimes be turned into a raised exception * Indirect callers that pass in rb_glob_error as errfunc: rb_glob, Dir.[], Dir.glob * Indirect callers that pass in 0 as errfunc: ruby_glob, ruby_brace_glob * `glob_opendir` does not check errno; all of its users have interruption bugs * `glob_getent` does not check errno; all of its users have interruption bugs * `nogvl_dir_empty_p` does not check errno (of readdir! it actually checks for opendir); all of its users have interruption bugs Less sure about these: * `do_stat`/`do_lstat` will turn errors into warnings (unclear if enabled or disabled by default) * `need_normalization` calls `fgetattrlist` / `getattrlist` and all errors `(ret != 0)` are treated in the same way * `rb_glob_error` is and `rb_glob_caller` leave exceptions as pending and rely on callers to raise them properly * Error handling of `rb_home_dir_of` and `rb_default_home_dir` are a bit suspicious As a workaround in the Datadog Ruby profiler, in https://github.com/DataDog/dd-trace-rb/pull/3720 I've added monkey patches to all of the Ruby-level APIs that use the above functions and mask out `SIGPROF` so these calls are never interrupted. This solution is does successfully work around the issue, although it prevents the profiler from sampling during these system calls, which will mean less visibility if e.g. these calls are taking a long time. And well, maintaining monkey patches is always problematic for future Ruby compatibility. -- https://bugs.ruby-lang.org/

5 12

[ruby-core:117882] [Ruby master Bug#20490] Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process
by stanhu (Stan Hu) 18 Aug '24

18 Aug '24

Issue #20490 has been reported by stanhu (Stan Hu). ---------------------------------------- Bug #20490: Process.waitpid2(-1, Process::WNOHANG) misbehaves on Ruby 3.1 & 3.2 with detached process https://bugs.ruby-lang.org/issues/20490 * Author: stanhu (Stan Hu) * Status: Open * ruby -v: ruby 3.2.4 (2024-04-23 revision af471c0e01) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- This is a follow-up issue for a bug that I thought was fixed in https://bugs.ruby-lang.org/issues/19837 and duplicated in https://bugs.ruby-lang.org/issues/20181. The following script doesn't terminate quickly in Ruby 3.1.5 and 3.2.4, even with the patches to address https://bugs.ruby-lang.org/issues/19837. It works fine in Ruby 3.3. It appears that the `Process::WNOHANG` argument passed to `Process.wait2` causes this script to spin until the child process stops: ```ruby #!/bin/env ruby Process.spawn({}, "sh -c 'sleep 600'").tap do |pid| puts "detaching PID #{pid}" Process.detach(pid) end forked_pid = fork do loop { sleep 1 } end child_waiter = Thread.new do puts "Waiting for child process to die..." # The spawned process has to exit before this returns in Ruby 3.1 and 3.2 loop do pid, status = Process.wait2(-1, Process::WNOHANG) puts "Exited PID: #{pid}, status: #{status}" break if pid sleep 1 end end process_killer = Thread.new do puts "Killing #{forked_pid}" system("kill #{forked_pid}") end child_waiter.join process_killer.join ``` If I drop the `Process::WNOHANG` argument, it works fine. -- https://bugs.ruby-lang.org/

3 6

[ruby-core:118699] [Ruby master Bug#20654] Floor and ceil have unexpected behaviour when ndigits is large
by peterzhu2118 (Peter Zhu) 18 Aug '24

18 Aug '24

Issue #20654 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20654: Floor and ceil have unexpected behaviour when ndigits is large https://bugs.ruby-lang.org/issues/20654 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/11257 There is unexpected behaviour in Integer#floor, Float#floor, Integer#ceil, Float#ceil when ndigits is large such that 10**ndigits is a bignum. Previously, it would return 0 in such cases. However, this would cause unexpected behaviour such as: ```ruby puts -1.floor(-5) # => -100000 puts -1.floor(-10) # => -10000000000 puts -1.floor(-20) # => 0 ``` ```ruby puts 1.ceil(-5) # => 100000 puts 1.ceil(-10) # => 10000000000 puts 1.ceil(-20) # => 0 ``` The PR changes the last result so that it will return -100000000000000000000 and 100000000000000000000, respectively. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118588] [Ruby master Bug#20634] ruby uses 1/2 (7 out of 16) cores for 16 ractors with RUBY_MAX_CPU=16 environment variable set
by skorobogatydmitry (Dmitry Skorobogaty) 09 Aug '24

09 Aug '24

Issue #20634 has been reported by skorobogatydmitry (Dmitry Skorobogaty). ---------------------------------------- Bug #20634: ruby uses 1/2 (7 out of 16) cores for 16 ractors with RUBY_MAX_CPU=16 environment variable set https://bugs.ruby-lang.org/issues/20634 * Author: skorobogatydmitry (Dmitry Skorobogaty) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- It's a cont for #20618, feel free to close this one and reopen the original bug. According to `top`, the below code uses 7/16 cores (700%) of my AMD Ryzen 7 5800HS: #!/usr/bin/env ruby 16.times { Ractor.new { 10_000.downto(1) { |i| 100_000.downto(1) { |j| i * j } } } } sleep 30 ``` Command I ran is `RUBY_MAX_CPU=16 ruby test.rb`. I also tried to set `RUBY_MN_THREADS=1` as an addition and the script starts to occupy 8 cores instead of 7 (possibly, due to N:M algo allocating an extra core for the blocked main Ractor). The same code occupies all 16 cores on ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux] The problem reproduces on ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x86_64-linux] (ruby from the latest docker image). -- https://bugs.ruby-lang.org/

1 2

[ruby-core:118520] [Ruby master Feature#20621] Check libruby.so hardening by annocheck
by vo.x (Vit Ondruch) 07 Aug '24

07 Aug '24

Issue #20621 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Feature #20621: Check libruby.so hardening by annocheck https://bugs.ruby-lang.org/issues/20621 * Author: vo.x (Vit Ondruch) * Status: Open ---------------------------------------- As part of #18061, there was implemented annocheck test case. However, the test covers just `ruby` (which is just thin shell), while leaves out `libruby.so` (which contains most of the Ruby code). This PR tries to improve that: https://github.com/ruby/ruby/pull/11123 BTW the question is if all *.so files or even all *.o files should be covered. -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118621] [Ruby master Misc#20639] Request to host C API docs on docs.ruby-lang.org
by alanwu (Alan Wu) 02 Aug '24

02 Aug '24

Issue #20639 has been reported by alanwu (Alan Wu). ---------------------------------------- Misc #20639: Request to host C API docs on docs.ruby-lang.org https://bugs.ruby-lang.org/issues/20639 * Author: alanwu (Alan Wu) * Status: Open ---------------------------------------- Having a first-party documentation site benefits readers and helps contributors improve the contents. The Ruby standard library API benefits from being hosted on <https://docs.ruby-lang.org> and it would be nice if the C API documentation produced by Doxygen can be hosted on the same site as well. I'm not familiar with the logistics of hosting the documentation site, and traffic is not free. But I think this is worth considering for the benefit of Ruby extension authors and maintainers. At the moment, <https://github.com/ruby/actions> generates the documentation with Doxygen but it's not hosted anywhere as far as I can tell. Searching for "Ruby C API documentation" on Google lead to guides about the API and no results that provide a comprehensive up-to-date reference. -- https://bugs.ruby-lang.org/

3 5

[ruby-core:118012] [Ruby master Misc#20509] Document importance of #to_ary and #to_hash for Array#== and Hash#==
by gettalong (Thomas Leitner) 02 Aug '24

02 Aug '24

Issue #20509 has been reported by gettalong (Thomas Leitner). ---------------------------------------- Misc #20509: Document importance of #to_ary and #to_hash for Array#== and Hash#== https://bugs.ruby-lang.org/issues/20509 * Author: gettalong (Thomas Leitner) * Status: Open ---------------------------------------- Both `Array#==` and `Hash#==` provide special behaviour in case the `other` argument is not an Array/Hash but defines the special `#to_ary`/`#to_hash` methods. Those methods are never called, they are just checked for existence. And if they exist, `other#==` is called to allow the `other` argument to decide whether the two objects are equal. I think this is worth mentioning in the documentation for `Array#==` and `Hash#==`. [Background: In my PDF library HexaPDF I have defined two classes `PDFArray` and `Dictionary` which act like Array and Hash but provide special PDF specific behaviour. For PDFArray I defined the `#to_ary` method but for Dictionary just the `#to_h` method. I have come across a bug where comparing Arrays with PDFArrays just works as it should be but comparing Hashes with Dictionaries doesn't due to the absence of `#to_hash` (it seems I removed `Dictionary#to_hash` in 2017 due to problems with automatic destructuring when passing a Dictionary as argument; from what I see that should be no problem anymore, so I will just add it back).] -- https://bugs.ruby-lang.org/

4 3

[ruby-core:118634] [Ruby master Bug#20640] Evaluation Order Issue in f(**h, &h.delete(key))
by jeremyevans0 (Jeremy Evans) 01 Aug '24

01 Aug '24

Issue #20640 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Bug #20640: Evaluation Order Issue in f(**h, &h.delete(key)) https://bugs.ruby-lang.org/issues/20640 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Since Ruby 3.0, there is an evaluation order issue when passing a single keyword splat and a block pass expression that modifies the keyword splat: ```ruby def f(*a, **kw) kw[:a].class end h = {a: ->{}} f(**h, &h.delete(:a)) # Ruby 2.0 - 2.7: Proc # Ruby 3.0 - 3.4: NilClass ``` For single keyword splats followed by positional argument splats, this has been an issue since 3.3: ```ruby def f(*a, **kw) kw[:a].class end h = {a: ->{}} a = [] f(*a, **h, &h.delete(:a)) # Ruby 2.0 - 3.2: Proc # Ruby 3.3 - 3.4: NilClass ``` Ruby handles these issues for positional splats, duplicating the splatted array before evaluating post, keyword, or block argument expressions: ```ruby f(*a, a.pop) # post argument f(*a, **a.pop) # keyword splat argument f(*a, a: a.pop) # keyword argument f(*a, &a.pop) # block argument ``` So it should handle the case for a keyword splat that could potentially be modified by block argument expression. I'll submit a pull request shortly to fix this issue. -- https://bugs.ruby-lang.org/

4 5

[ruby-core:118010] [Ruby master Feature#20508] Explicit access to *, **, &, and ...
by bradgessler (Brad Gessler) 01 Aug '24

01 Aug '24

Issue #20508 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Feature #20508: Explicit access to *, **, &, and ... https://bugs.ruby-lang.org/issues/20508 * Author: bradgessler (Brad Gessler) * Status: Open ---------------------------------------- I find debugging and certain meta-programming tasks challenging because there's no explicit APIs for accessing certain types of arguments in Ruby. Here's some example code: ```ruby def splats(one, *, two: nil, **, &) # These work p(*) p(**) # But the block doesn't show a proc. p(&) # This would show [:one, :two, :_] p binding.local_variables end splats(:arg, two: true) do 42 end ``` I'm not sure how to access the `&` variable, unless I name it. The same applies to endless ruby methods. I'd like to see a way to explicitly call bindings for those methods. Something like this (not sure if binding is the right place for it): ```ruby def splats(one, *, true: nil, **, &) binding.arguments # Returns an array binding.keyword_arguments # Returns a hash binding.block_argument # Returns a block end ``` These methods would give me access to the values used to call the method. -- https://bugs.ruby-lang.org/

5 4