- ruby-core - ml.ruby-lang.org

[ruby-core:118535] [Ruby master Feature#20625] Object#chain_of
by matheusrich (Matheus Richard) 24 Aug '24

24 Aug '24

Issue #20625 has been reported by matheusrich (Matheus Richard). ---------------------------------------- Feature #20625: Object#chain_of https://bugs.ruby-lang.org/issues/20625 * Author: matheusrich (Matheus Richard) * Status: Open ---------------------------------------- ## Motivation It's often common to traverse a tree/list-like structure in order to get a chain of elements. This proposal is to add a method to `Object` that allows collecting a chain of elements by applying a block to each element. It doesn't require the root element to be an instance of a specific class or respond to a specific protocol. I think this method could be useful in many cases, since hierarchies like this are common in codebases (e.g. file systems, organizations structures, commit histories, breadcrumbs in web apps, configuration hierarchies, etc.). Here are some examples extracted from real codebases (simplified for the sake of the example/privacy): ```rb # Given a file system structure, get the breadcrumbs of a file or directory def breadcrumbs(root) crumbs = [] current = root while current crumbs << current current = current.parent_dir end crumbs end # Given an employee, get the hierarchy of managers def hierarchy(employee) hierarchy = [] current = employee while current hierarchy << current current = current.manager end hierarchy end ``` ## Implementation The implementation in Ruby could look like this: ```rb class Object def chain_of(&block) chain = [] current = self while current chain << current current = block.call(current) end chain end end ``` Here's an example use: ```rb class ListNode attr_accessor :value, :parent def initialize(value, parent = nil) @value = value @parent = parent end def ancestors chain_of(&:parent).shift end end root = ListNode.new("root") child1 = ListNode.new("child1", root) child2 = ListNode.new("child2", child1) puts child2.ancestors.map(&:value) # => ["child1", "root"] ``` The examples from the motivation section could be rewritten as: ```rb breadcrumbs = root.chain_of(&:parent_directory) hierarchy = employee.chain_of(&:manager) ``` ## Considerations - While I'm including the object by default in the chain, it could be more intuitive to exclude it. In any case, it's easy to remove or add it with `shift`/`unshift`. - On a different note, the method could be named differently (I do like `chain_of`, though). Some alternatives I've considered are `map_chain`, `traverse`, and `trace_path`. - The method assumes that the traversal will finish at some point. If the user has a cyclic structure, it will loop indefinitely. We could stop looping if we find the same element twice. I don't think it's worth the extra complexity. - I'm not sure `Object` is the best place for this method. While it's very general, I think it gives power to the user to decide how to traverse a chain like this without having to rely on a specific class. Maybe a mixin (`Traversable`/`Chainable`) would be more appropriate? Could this fit in `Enumerable`, somehow? Of course, I'm open to suggestions and feedback. Thanks for reading! -- https://bugs.ruby-lang.org/

3 4

[ruby-core:118941] [Ruby master Bug#20697] `Rational("1r")` fails while `Complex("1i")` succeeds
by nobu (Nobuyoshi Nakada) 23 Aug '24

23 Aug '24

Issue #20697 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20697: `Rational("1r")` fails while `Complex("1i")` succeeds https://bugs.ruby-lang.org/issues/20697 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- @mame reported: ```ruby Rational("1r") #=> invalid value for convert(): "1r" (ArgumentError) ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118940] [Ruby master Bug#20696] `IO.read` ignores `err:` option
by nobu (Nobuyoshi Nakada) 23 Aug '24

23 Aug '24

Issue #20696 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20696: `IO.read` ignores `err:` option https://bugs.ruby-lang.org/issues/20696 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I expected the following code captures the error message, but `err:` option is ignored. ```shell-session $ ruby -e 'p IO.read("|ruby -e raise", err: %i[child out])' -e:1:in '<main>': unhandled exception "" ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118939] [Ruby master Bug#20695] Elevated GC allocations in parse.y parser
by alanwu (Alan Wu) 23 Aug '24

23 Aug '24

Issue #20695 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #20695: Elevated GC allocations in parse.y parser https://bugs.ruby-lang.org/issues/20695 * Author: alanwu (Alan Wu) * Status: Open * ruby -v: ruby 3.4.0dev (2024-08-23T14:49:27Z master 3f6be01bfc) [arm64-darwin23] * Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: DONTNEED ---------------------------------------- Testing on the `lobsters` benchmark from [`yjit-bench`](https://github.com/shopify/yjit-bench), the latest `master` parser is allocating more objects compared to commit:98eeadc932 ("Development of 3.4.0 started."). The following patch shows allocation count after code loading: ```diff diff --git a/benchmarks/lobsters/benchmark.rb b/benchmarks/lobsters/benchmark.rb index 240c50c..6cdd0ac 100644 --- a/benchmarks/lobsters/benchmark.rb +++ b/benchmarks/lobsters/benchmark.rb @@ -7,6 +7,8 @@ Dir.chdir __dir__ use_gemfile require_relative 'config/environment' +printf "allocated_after_load=%d\n", GC.stat(:total_allocated_objects) +exit require_relative "route_generator" # For an in-mem DB, we need to load all data on every boot ``` ``` $ ruby benchmarks/lobsters/benchmark.rb ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [arm64-darwin23] <snip> allocated_after_load=1747084 $ chruby master $ ruby benchmarks/lobsters/benchmark.rb ruby 3.4.0dev (2024-08-23T14:49:27Z master 3f6be01bfc) [arm64-darwin23] <snip> allocated_after_load=2163031 $ ruby --parser=prism benchmarks/lobsters/benchmark.rb ruby 3.4.0dev (2024-08-23T14:49:27Z master 3f6be01bfc) +PRISM [arm64-darwin23] <snip> allocated_after_load=932571 $ math 2163031 / 1747084 1.238081 ``` --- Profiling shows extra GC allocations coming from set_number_literal(); set_yylval_node() expands to a call to rb_enc_str_new(). So this issue seems related to #20659. A surprising side effect of these extra allocations is that they seem to impact the speed of the benchmark, not only the loading speed, even though the body of benchmark *does not do any Ruby parsing* after warmup iterations. The speed impact seem to be due to poorer data locality, as perf(1) shows elevated levels of `CYCLE_ACTIVITY.STALLS_L2_MISS` and related events. Using Prism speeds up benchmark iterations, probably since it allocates a lot fewer during code loading, leaving the heap in a better layout in terms of locality. With Prism, fewer cycles stall on data cache misses. ``` $ ruby run_benchmarks.rb --chruby 'master;mstr+p::master --parser=prism' lobsters <snip> master: ruby 3.4.0dev (2024-08-23T14:49:27Z master 3f6be01bfc) [arm64-darwin23] mstr+p: ruby 3.4.0dev (2024-08-23T14:49:27Z master 3f6be01bfc) +PRISM [arm64-darwin23] -------- ----------- ---------- ----------- ---------- -------------- ------------- bench master (ms) stddev (%) mstr+p (ms) stddev (%) mstr+p 1st itr master/mstr+p lobsters 567.9 0.7 557.0 0.8 1.00 1.02 -------- ----------- ---------- ----------- ---------- -------------- ------------- Legend: - mstr+p 1st itr: ratio of master/mstr+p time for the first benchmarking iteration. - master/mstr+p: ratio of master/mstr+p time. Higher is better for mstr+p. Above 1 represents a speedup. ``` In hindsight, it's clear how that the parser's allocation pattern can have a significant impact on heap layout, since typical app boot with Kernel#require alternates between running app code and running the parser. Parser allocations essentially end up as gaps between app objects. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118929] [Ruby master Feature#18368] Range#step semantics for non-Numeric ranges
by Dan0042 (Daniel DeLorme) 22 Aug '24

22 Aug '24

Issue #18368 has been updated by Dan0042 (Daniel DeLorme). matz (Yukihiro Matsumoto) wrote in #note-31: > It seems this change breaks step over string ranges (e.g. `"a".."z").step(3)`). We need to handle string ranges specifically. Thank you Matz, I am truly overjoyed the one in charge still cares about backward compatibility. knu (Akinori MUSHA) wrote in #note-33: > If we take this compatibility as important and go with it, should we give up `("a".."aaaa").step("a")` or support it? I'd say definitely support it. That way Range#step behavior remains largely consistent, with just one little special extra for compat. BTW not sure if it's important but `(:a .. :z).step(3)` is also supported. ---------------------------------------- Feature #18368: Range#step semantics for non-Numeric ranges https://bugs.ruby-lang.org/issues/18368#change-109498 * Author: zverok (Victor Shepelev) * Status: Open ---------------------------------------- I am sorry if the question had already been discussed, can't find the relevant topic. "Intuitively", this looks (for me) like a meaningful statement: ```ruby (Time.parse('2021-12-01')..Time.parse('2021-12-24')).step(1.day).to_a # ^^^^^ or just 24*60*60 ``` Unfortunately, it doesn't work with "TypeError (can't iterate from Time)". Initially it looked like a bug for me, but after digging a bit into code/docs, I understood that `Range#step` has an odd semantics of "advance the begin N times with `#succ`, and yield the result", with N being always integer: ```ruby ('a'..'z').step(3).first(5) # => ["a", "d", "g", "j", "m"] ``` The fact that semantic is "odd" is confirmed by the fact that for Float it is redefined to do what I "intuitively" expected: ```ruby (1.0..7.0).step(0.3).first(5) # => [1.0, 1.3, 1.6, 1.9, 2.2] ``` (Like with [`Range#===` some time ago](https://bugs.ruby-lang.org/issues/14575), I believe that to be a strong proof of the wrong generic semantics, if for numbers the semantics needed to be redefined completely.) Another thing to note is that "skip N elements" seem to be rather "generically Enumerable-related" yet it isn't defined on `Enumerable` (because nobody needs this semantics, typically!) Hence, two questions: * Can we redefine generic `Range#step` to new semantics (of using `begin + step` iteratively)? It is hard to imagine the amount of actual usage of the old behavior (with String?.. to what end?) in the wild * If the answer is "no", can we define a new method with new semantics, like, IDK, `Range#over(span)`? **UPD:** More examples of useful behavior (it is NOT only about core `Time` class): ```ruby require 'active_support/all' (1.minute..20.minutes).step(2.minutes).to_a #=> [1 minute, 3 minutes, 5 minutes, 7 minutes, 9 minutes, 11 minutes, 13 minutes, 15 minutes, 17 minutes, 19 minutes] require 'tod' (Tod::TimeOfDay.parse("8am")..Tod::TimeOfDay.parse("10am")).step(30.minutes).to_a #=> [#<Tod::TimeOfDay 08:00:00>, #<Tod::TimeOfDay 08:30:00>, #<Tod::TimeOfDay 09:00:00>, #<Tod::TimeOfDay 09:30:00>, #<Tod::TimeOfDay 10:00:00>] require 'matrix' (Vector[1, 2, 3]..).step(Vector[1, 1, 1]).take(3) #=> [Vector[1, 2, 3], Vector[2, 3, 4], Vector[3, 4, 5]] require 'unitwise' (Unitwise(0, 'km')..Unitwise(1, 'km')).step(Unitwise(100, 'm')).map(&:to_s) #=> ["0 km", "1/10 km", "1/5 km", "3/10 km", "2/5 km", "0.5 km", "3/5 km", "7/10 km", "4/5 km", "9/10 km", "1 km"] ``` **UPD:** Responding to discussion points: **Q:** Matz is concerned that the proposed simple definition will be confusing with the classes where `+` is redefined as concatenation. **A:** I believe that simplicity of semantics and ease of explaining ("it just uses `+` underneath, whatever `+` does, will be performed") will make the confusion minimal. **Q:** Why not introduce new API requirement (like "class of range's `begin` should implement `increment` method, and then it will be used in `step`) **A:** require *every* gem author to change *every* of their objects' behavior. For that, they should be aware of the change, consider it important enough to care, clearly understand the necessary semantics of implementation, have a resource to release a new version... Then all users of all such gems would be required to upgrade. The feature would be DOA (dead-on-arrival). The two alternative ways I am suggesting: change the behavior of `#step` or introduce a new method with desired behavior: 1. Easy to explain and announce 2. Require no other code changes to immediately become useful 3. With something like [backports](https://github.com/marcandre/backports) or [ruby-next](https://github.com/ruby-next/ruby-next) easy to start using even in older Ruby version, making the code more expressive even before it would be possible for some particular app/compny to upgrade to (say) 3.2 All examples of behavior from the code above are real `irb` output with monkey-patched `Range#step`, demonstrating how little change will be needed to code outside of the `Range`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118910] [Ruby master Bug#20688] Use-after-free in WeakMap and WeakKeyMap
by peterzhu2118 (Peter Zhu) 22 Aug '24

22 Aug '24

Issue #20688 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20688: Use-after-free in WeakMap and WeakKeyMap https://bugs.ruby-lang.org/issues/20688 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/11421 We cannot free the weakmap_entry before the ST_DELETE because it could hash the key which would read the weakmap_entry and would cause a use-after-free. Instead, we store the entry and free it on the next iteration. For example, the following script triggers a use-after-free in Valgrind: ```ruby weakmap = ObjectSpace::WeakMap.new 10_000.times { weakmap[Object.new] = Object.new } ``` ``` ==25795== Invalid read of size 8 ==25795== at 0x462297: wmap_cmp (weakmap.c:165) ==25795== by 0x3A2B1C: find_table_bin_ind (st.c:930) ==25795== by 0x3A5EAA: st_general_foreach (st.c:1599) ==25795== by 0x3A5EAA: rb_st_foreach (st.c:1640) ==25795== by 0x25C991: gc_mark_children (default.c:4870) ==25795== by 0x25C991: gc_marks_wb_unprotected_objects_plane (default.c:5565) ==25795== by 0x25C991: rgengc_rememberset_mark_plane (default.c:5557) ==25795== by 0x25C991: rgengc_rememberset_mark (default.c:6233) ==25795== by 0x25C991: gc_marks_start (default.c:6057) ==25795== by 0x25C991: gc_marks (default.c:6077) ==25795== by 0x25C991: gc_start (default.c:6723) ==25795== by 0x260F96: heap_prepare (default.c:2282) ==25795== by 0x260F96: heap_next_free_page (default.c:2489) ==25795== by 0x260F96: newobj_cache_miss (default.c:2598) ==25795== by 0x26197F: newobj_alloc (default.c:2622) ==25795== by 0x26197F: rb_gc_impl_new_obj (default.c:2701) ==25795== by 0x26197F: newobj_of (gc.c:890) ==25795== by 0x26197F: rb_wb_protected_newobj_of (gc.c:917) ==25795== by 0x2DEA88: rb_class_allocate_instance (object.c:131) ==25795== by 0x2E3B18: class_call_alloc_func (object.c:2141) ==25795== by 0x2E3B18: rb_class_alloc (object.c:2113) ==25795== by 0x2E3B18: rb_class_new_instance_pass_kw (object.c:2172) ==25795== by 0x429DDC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3786) ==25795== by 0x44B08D: vm_sendish (vm_insnhelper.c:5953) ==25795== by 0x44B08D: vm_exec_core (insns.def:898) ==25795== by 0x43A7A4: rb_vm_exec (vm.c:2564) ==25795== by 0x234914: rb_ec_exec_node (eval.c:281) ==25795== Address 0x21603710 is 0 bytes inside a block of size 16 free'd ==25795== at 0x4849B2C: free (vg_replace_malloc.c:989) ==25795== by 0x249651: rb_gc_impl_free (default.c:8527) ==25795== by 0x249651: rb_gc_impl_free (default.c:8508) ==25795== by 0x249651: ruby_sized_xfree.constprop.0 (gc.c:4178) ==25795== by 0x4626EC: ruby_sized_xfree_inlined (gc.h:277) ==25795== by 0x4626EC: wmap_free_entry (weakmap.c:45) ==25795== by 0x4626EC: wmap_mark_weak_table_i (weakmap.c:61) ==25795== by 0x3A5CEF: apply_functor (st.c:1633) ==25795== by 0x3A5CEF: st_general_foreach (st.c:1543) ==25795== by 0x3A5CEF: rb_st_foreach (st.c:1640) ==25795== by 0x25C991: gc_mark_children (default.c:4870) ==25795== by 0x25C991: gc_marks_wb_unprotected_objects_plane (default.c:5565) ==25795== by 0x25C991: rgengc_rememberset_mark_plane (default.c:5557) ==25795== by 0x25C991: rgengc_rememberset_mark (default.c:6233) ==25795== by 0x25C991: gc_marks_start (default.c:6057) ==25795== by 0x25C991: gc_marks (default.c:6077) ==25795== by 0x25C991: gc_start (default.c:6723) ==25795== by 0x260F96: heap_prepare (default.c:2282) ==25795== by 0x260F96: heap_next_free_page (default.c:2489) ==25795== by 0x260F96: newobj_cache_miss (default.c:2598) ==25795== by 0x26197F: newobj_alloc (default.c:2622) ==25795== by 0x26197F: rb_gc_impl_new_obj (default.c:2701) ==25795== by 0x26197F: newobj_of (gc.c:890) ==25795== by 0x26197F: rb_wb_protected_newobj_of (gc.c:917) ==25795== by 0x2DEA88: rb_class_allocate_instance (object.c:131) ==25795== by 0x2E3B18: class_call_alloc_func (object.c:2141) ==25795== by 0x2E3B18: rb_class_alloc (object.c:2113) ==25795== by 0x2E3B18: rb_class_new_instance_pass_kw (object.c:2172) ==25795== by 0x429DDC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3786) ==25795== by 0x44B08D: vm_sendish (vm_insnhelper.c:5953) ==25795== by 0x44B08D: vm_exec_core (insns.def:898) ==25795== by 0x43A7A4: rb_vm_exec (vm.c:2564) ==25795== Block was alloc'd at ==25795== at 0x484680F: malloc (vg_replace_malloc.c:446) ==25795== by 0x25CE9E: rb_gc_impl_malloc (default.c:8542) ==25795== by 0x462A39: wmap_aset_replace (weakmap.c:423) ==25795== by 0x3A5542: rb_st_update (st.c:1487) ==25795== by 0x462B8E: wmap_aset (weakmap.c:452) ==25795== by 0x429DDC: vm_call_cfunc_with_frame_ (vm_insnhelper.c:3786) ==25795== by 0x44B08D: vm_sendish (vm_insnhelper.c:5953) ==25795== by 0x44B08D: vm_exec_core (insns.def:898) ==25795== by 0x43A7A4: rb_vm_exec (vm.c:2564) ==25795== by 0x234914: rb_ec_exec_node (eval.c:281) ==25795== by 0x2369B8: ruby_run_node (eval.c:319) ==25795== by 0x15D675: rb_main (main.c:43) ==25795== by 0x15D675: main (main.c:62) ``` -- https://bugs.ruby-lang.org/

1 1

[ruby-core:118926] [Ruby master Feature#18368] Range#step semantics for non-Numeric ranges
by zverok (Victor Shepelev) 22 Aug '24

22 Aug '24

Issue #18368 has been updated by zverok (Victor Shepelev). @mame It doesn’t take long, but I’ll have time for that on the weekend (most probably Sunday). I hope that some clarifications might happen in a few days before that. ---------------------------------------- Feature #18368: Range#step semantics for non-Numeric ranges https://bugs.ruby-lang.org/issues/18368#change-109495 * Author: zverok (Victor Shepelev) * Status: Open ---------------------------------------- I am sorry if the question had already been discussed, can't find the relevant topic. "Intuitively", this looks (for me) like a meaningful statement: ```ruby (Time.parse('2021-12-01')..Time.parse('2021-12-24')).step(1.day).to_a # ^^^^^ or just 24*60*60 ``` Unfortunately, it doesn't work with "TypeError (can't iterate from Time)". Initially it looked like a bug for me, but after digging a bit into code/docs, I understood that `Range#step` has an odd semantics of "advance the begin N times with `#succ`, and yield the result", with N being always integer: ```ruby ('a'..'z').step(3).first(5) # => ["a", "d", "g", "j", "m"] ``` The fact that semantic is "odd" is confirmed by the fact that for Float it is redefined to do what I "intuitively" expected: ```ruby (1.0..7.0).step(0.3).first(5) # => [1.0, 1.3, 1.6, 1.9, 2.2] ``` (Like with [`Range#===` some time ago](https://bugs.ruby-lang.org/issues/14575), I believe that to be a strong proof of the wrong generic semantics, if for numbers the semantics needed to be redefined completely.) Another thing to note is that "skip N elements" seem to be rather "generically Enumerable-related" yet it isn't defined on `Enumerable` (because nobody needs this semantics, typically!) Hence, two questions: * Can we redefine generic `Range#step` to new semantics (of using `begin + step` iteratively)? It is hard to imagine the amount of actual usage of the old behavior (with String?.. to what end?) in the wild * If the answer is "no", can we define a new method with new semantics, like, IDK, `Range#over(span)`? **UPD:** More examples of useful behavior (it is NOT only about core `Time` class): ```ruby require 'active_support/all' (1.minute..20.minutes).step(2.minutes).to_a #=> [1 minute, 3 minutes, 5 minutes, 7 minutes, 9 minutes, 11 minutes, 13 minutes, 15 minutes, 17 minutes, 19 minutes] require 'tod' (Tod::TimeOfDay.parse("8am")..Tod::TimeOfDay.parse("10am")).step(30.minutes).to_a #=> [#<Tod::TimeOfDay 08:00:00>, #<Tod::TimeOfDay 08:30:00>, #<Tod::TimeOfDay 09:00:00>, #<Tod::TimeOfDay 09:30:00>, #<Tod::TimeOfDay 10:00:00>] require 'matrix' (Vector[1, 2, 3]..).step(Vector[1, 1, 1]).take(3) #=> [Vector[1, 2, 3], Vector[2, 3, 4], Vector[3, 4, 5]] require 'unitwise' (Unitwise(0, 'km')..Unitwise(1, 'km')).step(Unitwise(100, 'm')).map(&:to_s) #=> ["0 km", "1/10 km", "1/5 km", "3/10 km", "2/5 km", "0.5 km", "3/5 km", "7/10 km", "4/5 km", "9/10 km", "1 km"] ``` **UPD:** Responding to discussion points: **Q:** Matz is concerned that the proposed simple definition will be confusing with the classes where `+` is redefined as concatenation. **A:** I believe that simplicity of semantics and ease of explaining ("it just uses `+` underneath, whatever `+` does, will be performed") will make the confusion minimal. **Q:** Why not introduce new API requirement (like "class of range's `begin` should implement `increment` method, and then it will be used in `step`) **A:** require *every* gem author to change *every* of their objects' behavior. For that, they should be aware of the change, consider it important enough to care, clearly understand the necessary semantics of implementation, have a resource to release a new version... Then all users of all such gems would be required to upgrade. The feature would be DOA (dead-on-arrival). The two alternative ways I am suggesting: change the behavior of `#step` or introduce a new method with desired behavior: 1. Easy to explain and announce 2. Require no other code changes to immediately become useful 3. With something like [backports](https://github.com/marcandre/backports) or [ruby-next](https://github.com/ruby-next/ruby-next) easy to start using even in older Ruby version, making the code more expressive even before it would be possible for some particular app/compny to upgrade to (say) 3.2 All examples of behavior from the code above are real `irb` output with monkey-patched `Range#step`, demonstrating how little change will be needed to code outside of the `Range`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118925] [Ruby master Feature#18368] Range#step semantics for non-Numeric ranges
by mame (Yusuke Endoh) 22 Aug '24

22 Aug '24

Issue #18368 has been updated by mame (Yusuke Endoh). @zverok Please fix the incompatibility first before having such a discussion. If it takes long, I will have to revert the change. ---------------------------------------- Feature #18368: Range#step semantics for non-Numeric ranges https://bugs.ruby-lang.org/issues/18368#change-109494 * Author: zverok (Victor Shepelev) * Status: Open ---------------------------------------- I am sorry if the question had already been discussed, can't find the relevant topic. "Intuitively", this looks (for me) like a meaningful statement: ```ruby (Time.parse('2021-12-01')..Time.parse('2021-12-24')).step(1.day).to_a # ^^^^^ or just 24*60*60 ``` Unfortunately, it doesn't work with "TypeError (can't iterate from Time)". Initially it looked like a bug for me, but after digging a bit into code/docs, I understood that `Range#step` has an odd semantics of "advance the begin N times with `#succ`, and yield the result", with N being always integer: ```ruby ('a'..'z').step(3).first(5) # => ["a", "d", "g", "j", "m"] ``` The fact that semantic is "odd" is confirmed by the fact that for Float it is redefined to do what I "intuitively" expected: ```ruby (1.0..7.0).step(0.3).first(5) # => [1.0, 1.3, 1.6, 1.9, 2.2] ``` (Like with [`Range#===` some time ago](https://bugs.ruby-lang.org/issues/14575), I believe that to be a strong proof of the wrong generic semantics, if for numbers the semantics needed to be redefined completely.) Another thing to note is that "skip N elements" seem to be rather "generically Enumerable-related" yet it isn't defined on `Enumerable` (because nobody needs this semantics, typically!) Hence, two questions: * Can we redefine generic `Range#step` to new semantics (of using `begin + step` iteratively)? It is hard to imagine the amount of actual usage of the old behavior (with String?.. to what end?) in the wild * If the answer is "no", can we define a new method with new semantics, like, IDK, `Range#over(span)`? **UPD:** More examples of useful behavior (it is NOT only about core `Time` class): ```ruby require 'active_support/all' (1.minute..20.minutes).step(2.minutes).to_a #=> [1 minute, 3 minutes, 5 minutes, 7 minutes, 9 minutes, 11 minutes, 13 minutes, 15 minutes, 17 minutes, 19 minutes] require 'tod' (Tod::TimeOfDay.parse("8am")..Tod::TimeOfDay.parse("10am")).step(30.minutes).to_a #=> [#<Tod::TimeOfDay 08:00:00>, #<Tod::TimeOfDay 08:30:00>, #<Tod::TimeOfDay 09:00:00>, #<Tod::TimeOfDay 09:30:00>, #<Tod::TimeOfDay 10:00:00>] require 'matrix' (Vector[1, 2, 3]..).step(Vector[1, 1, 1]).take(3) #=> [Vector[1, 2, 3], Vector[2, 3, 4], Vector[3, 4, 5]] require 'unitwise' (Unitwise(0, 'km')..Unitwise(1, 'km')).step(Unitwise(100, 'm')).map(&:to_s) #=> ["0 km", "1/10 km", "1/5 km", "3/10 km", "2/5 km", "0.5 km", "3/5 km", "7/10 km", "4/5 km", "9/10 km", "1 km"] ``` **UPD:** Responding to discussion points: **Q:** Matz is concerned that the proposed simple definition will be confusing with the classes where `+` is redefined as concatenation. **A:** I believe that simplicity of semantics and ease of explaining ("it just uses `+` underneath, whatever `+` does, will be performed") will make the confusion minimal. **Q:** Why not introduce new API requirement (like "class of range's `begin` should implement `increment` method, and then it will be used in `step`) **A:** require *every* gem author to change *every* of their objects' behavior. For that, they should be aware of the change, consider it important enough to care, clearly understand the necessary semantics of implementation, have a resource to release a new version... Then all users of all such gems would be required to upgrade. The feature would be DOA (dead-on-arrival). The two alternative ways I am suggesting: change the behavior of `#step` or introduce a new method with desired behavior: 1. Easy to explain and announce 2. Require no other code changes to immediately become useful 3. With something like [backports](https://github.com/marcandre/backports) or [ruby-next](https://github.com/ruby-next/ruby-next) easy to start using even in older Ruby version, making the code more expressive even before it would be possible for some particular app/compny to upgrade to (say) 3.2 All examples of behavior from the code above are real `irb` output with monkey-patched `Range#step`, demonstrating how little change will be needed to code outside of the `Range`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118346] [Ruby master Bug#20586] Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted
by ivoanjo (Ivo Anjo) 22 Aug '24

22 Aug '24

Issue #20586 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #20586: Some filesystem calls in dir.c are missing error handling and can return incorrect results if interrupted https://bugs.ruby-lang.org/issues/20586 * Author: ivoanjo (Ivo Anjo) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- #### Background Hey! I work for Datadog on the Ruby profiler part of the [`datadog` (previously `ddtrace`)](https://github.com/datadog/dd-trace-rb) gem. A customer reached [out with an issue](https://github.com/DataDog/dd-trace-rb/issues/3450) where enabling the profiler made `Dir.glob` return no files for a given folder: Without profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => ["/gcsfuse/test.html", "/gcsfuse/test.txt"] ``` With profiler: ``` irb(main):002:0> Dir.glob('/gcsfuse/t*') => [] ``` It turns out the issue is related to missing error handling in `dir.c`. The Datadog Ruby profiler, like stackprof, pf2 and vernier, uses unix signals to interrupt the currently-active thread and take a sample (usually `SIGPROF`). When some system calls get interrupted by a signal, they return an [EINTR error code](https://man7.org/linux/man-pages/man7/signal.7.html#:~:text=Interrupt… back to the caller. Consider for instance the implementation of `dir_each_entry` in `dir.c`: ```c static VALUE dir_each_entry(VALUE dir, VALUE (*each)(VALUE, VALUE), VALUE arg, int children_only) { struct dir_data *dirp; struct dirent *dp; IF_NORMALIZE_UTF8PATH(int norm_p); GetDIR(dir, dirp); rewinddir(dirp->dir); IF_NORMALIZE_UTF8PATH(norm_p = need_normalization(dirp->dir, RSTRING_PTR(dirp->path))); while ((dp = READDIR(dirp->dir, dirp->enc)) != NULL) { // ... do things } return dir; } ``` If `READDIR` returns `NULL`, then `dir_each_entry` assumes it has iterated the entire directory. But looking [at the man page for `readdir`](https://man7.org/linux/man-pages/man3/readdir.3.html) we see the following sharp edge (emphasis mine): > It returns NULL on reaching the end of the directory stream **or if an error occurred**. So what's happening in this situation is: `readdir` gets interrupted, returns `NULL` + sets errno to `EINTR`. But `dir_each_entry` doesn't check `errno`, so rather than raising an exception to flag the issue, it treats it as if the end of the directory has been reached. #### How to reproduce Reproducing this is somewhat annoying, because it's dependent on timing: the signal must arrive at the exact time the dir API is getting executed. I was able to reproduce this every time by using the [google cloud `gcsfuse`](https://cloud.google.com/storage/docs/gcs-fuse) tool. This somewhat makes sense -- a remote filesystem is much slower than a local one, so there's a much bigger window of opportunity for a signal to arrive while the system call is blocked. Here's an example I included in https://github.com/DataDog/dd-trace-rb/pull/3720: ``` # Not shown: Set up a trial google cloud account, install gcsfuse, create a cloud storage bucket and put in some test files $ gcsfuse test_fs_dd_trace_rb fuse-testing/ $ ls fuse-testing/ hello.txt test.html test.txt # Not shown: Add `datadog` gem to `Gemfile` $ DD_PROFILING_ENABLED=true DD_PROFILING_DIR_INTERRUPTION_WORKAROUND_ENABLED=false bundle exec ddprofrb exec ruby -e "Datadog::Profiling.wait_until_running; pp Dir.children('fuse-testing/')" [] ``` Let me know if you'd like me to try to create a reproducer that does not depend on the `datadog` gem. #### Additional notes I've spent quite some time looking at the `dir.c` sources, and here's the full list of APIs that suffer from issues: * `dir_each_entry` does not check `errno`; all of its users have interruption bugs * `dir_tell` will return -1 instead of the correct position (which means that passing -1 to `dir_seek`/`dir_set_pos` will cause it to not list the directory properly) * `do_opendir` an error in system calls will only sometimes be turned into a raised exception * Indirect callers that pass in rb_glob_error as errfunc: rb_glob, Dir.[], Dir.glob * Indirect callers that pass in 0 as errfunc: ruby_glob, ruby_brace_glob * `glob_opendir` does not check errno; all of its users have interruption bugs * `glob_getent` does not check errno; all of its users have interruption bugs * `nogvl_dir_empty_p` does not check errno (of readdir! it actually checks for opendir); all of its users have interruption bugs Less sure about these: * `do_stat`/`do_lstat` will turn errors into warnings (unclear if enabled or disabled by default) * `need_normalization` calls `fgetattrlist` / `getattrlist` and all errors `(ret != 0)` are treated in the same way * `rb_glob_error` is and `rb_glob_caller` leave exceptions as pending and rely on callers to raise them properly * Error handling of `rb_home_dir_of` and `rb_default_home_dir` are a bit suspicious As a workaround in the Datadog Ruby profiler, in https://github.com/DataDog/dd-trace-rb/pull/3720 I've added monkey patches to all of the Ruby-level APIs that use the above functions and mask out `SIGPROF` so these calls are never interrupted. This solution is does successfully work around the issue, although it prevents the profiler from sampling during these system calls, which will mean less visibility if e.g. these calls are taking a long time. And well, maintaining monkey patches is always problematic for future Ruby compatibility. -- https://bugs.ruby-lang.org/

5 12

[ruby-core:118916] [Ruby master Feature#18368] Range#step semantics for non-Numeric ranges
by zverok (Victor Shepelev) 22 Aug '24

22 Aug '24

Issue #18368 has been updated by zverok (Victor Shepelev). @matz I can certainly implement the specialization, but just to clarify: are there any evidence that people use `("a".."z").step(3)` in any meaningful way? (Unfortunately, I have no way of doing the investigation myself, but probably those with access to “search through all existing gems code” server might shed some light on it?) What I am concerned about is that having it specialized just for strings has an intuition-breaking consequences and would certainly be perceived like a “bug” somewhere down the road. We had the same with switching from `include?` to `cover?` as a `Range#===` implementation: * In 2.6, strings were excluded from the change * ...but after several reports, it turned out that it is better to make string ranges behave consistently with everything else, which happened in 2.7. In general, Range already have a history of such specializations (like specifically considering `Time` “linear” value in 2.3 to fix `case Time.now ...` behavior, before generic switching to `cover?` fixed it consistenlty), and I believe it is mostly confusing to the language’s users: some core class behaving a “special” way that can’t be imitated by any other class, breaking all and every “duck typing” intuition. So, a big question for me is if the specialization is justified by some existing use cases, or by a generic caution against breaking things (and in the latter case, who guarantees that in some codebase nobody have relied on some _custom_ object range and integer step behavior?.. IDK) So, I see these options here: 1. Leave the “consistent first” option: no specialization for `String` (unless there is a strong proof that some popular gem or a widespread idiom is hurt by this); 2. Specialize it for strings to only work with numeric steps, and document that “strings are special, deal with it” 3. Specialize it for strings to work with both numeric _and_ string steps (by explicit type-checking “if the range is string & the step is integer”)—that’s what @knu asks about in #18368#note-33. Maybe even avoid documenting the special behavior, or document it is discouraged (so the old code—if it exists—wouldn’t break, but the new code wouldn’t rely on it) 4. Maybe make it always work with integer steps as a fallback (though there is no way to reliably check “whether this particular class supports `obj+int` directly, or we need to switch back to the ‘number of steps’ behavior”) 5. Rethink the decision in general (again, if there is a firm ground for concerns over the compatibility), and introduce _another_ method that will behave like the currently implemented `#step` relying on `#+`, reverting `#step` to the old behavior. Of those, I would honestly prefer to avoid (2) (very confusing for new users, teaching, and general language’s image) and (5); I don’t think (4) is generally possible; which leaves us with (1) (how is it now) and (3) (additional specialization for strings which _also_ supports numeric steps on the basis of range begin and step class, in addition to supporting string steps). TBH, I still believe there would be very low amount of incompatibility in existing code, but I might genuinely miss some common knowledge. ---------------------------------------- Feature #18368: Range#step semantics for non-Numeric ranges https://bugs.ruby-lang.org/issues/18368#change-109485 * Author: zverok (Victor Shepelev) * Status: Open ---------------------------------------- I am sorry if the question had already been discussed, can't find the relevant topic. "Intuitively", this looks (for me) like a meaningful statement: ```ruby (Time.parse('2021-12-01')..Time.parse('2021-12-24')).step(1.day).to_a # ^^^^^ or just 24*60*60 ``` Unfortunately, it doesn't work with "TypeError (can't iterate from Time)". Initially it looked like a bug for me, but after digging a bit into code/docs, I understood that `Range#step` has an odd semantics of "advance the begin N times with `#succ`, and yield the result", with N being always integer: ```ruby ('a'..'z').step(3).first(5) # => ["a", "d", "g", "j", "m"] ``` The fact that semantic is "odd" is confirmed by the fact that for Float it is redefined to do what I "intuitively" expected: ```ruby (1.0..7.0).step(0.3).first(5) # => [1.0, 1.3, 1.6, 1.9, 2.2] ``` (Like with [`Range#===` some time ago](https://bugs.ruby-lang.org/issues/14575), I believe that to be a strong proof of the wrong generic semantics, if for numbers the semantics needed to be redefined completely.) Another thing to note is that "skip N elements" seem to be rather "generically Enumerable-related" yet it isn't defined on `Enumerable` (because nobody needs this semantics, typically!) Hence, two questions: * Can we redefine generic `Range#step` to new semantics (of using `begin + step` iteratively)? It is hard to imagine the amount of actual usage of the old behavior (with String?.. to what end?) in the wild * If the answer is "no", can we define a new method with new semantics, like, IDK, `Range#over(span)`? **UPD:** More examples of useful behavior (it is NOT only about core `Time` class): ```ruby require 'active_support/all' (1.minute..20.minutes).step(2.minutes).to_a #=> [1 minute, 3 minutes, 5 minutes, 7 minutes, 9 minutes, 11 minutes, 13 minutes, 15 minutes, 17 minutes, 19 minutes] require 'tod' (Tod::TimeOfDay.parse("8am")..Tod::TimeOfDay.parse("10am")).step(30.minutes).to_a #=> [#<Tod::TimeOfDay 08:00:00>, #<Tod::TimeOfDay 08:30:00>, #<Tod::TimeOfDay 09:00:00>, #<Tod::TimeOfDay 09:30:00>, #<Tod::TimeOfDay 10:00:00>] require 'matrix' (Vector[1, 2, 3]..).step(Vector[1, 1, 1]).take(3) #=> [Vector[1, 2, 3], Vector[2, 3, 4], Vector[3, 4, 5]] require 'unitwise' (Unitwise(0, 'km')..Unitwise(1, 'km')).step(Unitwise(100, 'm')).map(&:to_s) #=> ["0 km", "1/10 km", "1/5 km", "3/10 km", "2/5 km", "0.5 km", "3/5 km", "7/10 km", "4/5 km", "9/10 km", "1 km"] ``` **UPD:** Responding to discussion points: **Q:** Matz is concerned that the proposed simple definition will be confusing with the classes where `+` is redefined as concatenation. **A:** I believe that simplicity of semantics and ease of explaining ("it just uses `+` underneath, whatever `+` does, will be performed") will make the confusion minimal. **Q:** Why not introduce new API requirement (like "class of range's `begin` should implement `increment` method, and then it will be used in `step`) **A:** require *every* gem author to change *every* of their objects' behavior. For that, they should be aware of the change, consider it important enough to care, clearly understand the necessary semantics of implementation, have a resource to release a new version... Then all users of all such gems would be required to upgrade. The feature would be DOA (dead-on-arrival). The two alternative ways I am suggesting: change the behavior of `#step` or introduce a new method with desired behavior: 1. Easy to explain and announce 2. Require no other code changes to immediately become useful 3. With something like [backports](https://github.com/marcandre/backports) or [ruby-next](https://github.com/ruby-next/ruby-next) easy to start using even in older Ruby version, making the code more expressive even before it would be possible for some particular app/compny to upgrade to (say) 3.2 All examples of behavior from the code above are real `irb` output with monkey-patched `Range#step`, demonstrating how little change will be needed to code outside of the `Range`. -- https://bugs.ruby-lang.org/

1 0