July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113888] [Ruby master Bug#19726] Script loadde twice when requiring self
by johannes (Johannes Krude) 14 Jul '23

14 Jul '23

Issue #19726 has been reported by johannes (Johannes Krude). ---------------------------------------- Bug #19726: Script loadde twice when requiring self https://bugs.ruby-lang.org/issues/19726 * Author: johannes (Johannes Krude) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The executed program is not added to `$LOADED_FEATURES` unexpectedly resulting in the program possibly being loaded twice: a.rb: ``` #!/usr/bin/env ruby require "./b" C = 42 return unless __FILE__ == $0 puts C ``` b.rb: ``` require "./a" ``` This results in: ``` $ ./a.rb ./a.rb:5: warning: already initialized constant C /home/johannes/t/a.rb:5: warning: previous definition of C was here 42 ``` Ruby version: ``` $ sudo docker run ruby:3.3.0-preview1 ruby --version ruby 3.3.0preview1 (2023-05-12 master a1b01e7701) [x86_64-linux] ``` **This could be solved by always adding the executed program to `$LOADED_FEATURES`.** As a workaround, executed programs can be prepended with: ``` require "pathname" if __FILE__ == $0 $LOADED_FEATURES << Pathname.new(__FILE__).expand_path.to_s end ``` -- https://bugs.ruby-lang.org/

5 5

[ruby-core:114185] [Ruby master Bug#18743] Enumerator#next / peek re-use each others stacktraces
by marcper (Marcelo Pereira) 14 Jul '23

14 Jul '23

Issue #18743 has been updated by marcper (Marcelo Pereira). Hello again, @ko1. Please let me know if this patch is acceptable. I followed the suggestions from @nobu already. Best ---------------------------------------- Bug #18743: Enumerator#next / peek re-use each others stacktraces https://bugs.ruby-lang.org/issues/18743#change-103872 * Author: sos4nt (Stefan Schüßler) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- I encountered an odd behavior. If I rescue the `StopIteration` exception from `peek` and call `next` afterwards: (or vice-versa) ```ruby # enum.rb # 1 # 2 enum = [].each # 3 enum.peek rescue nil # 4 enum.next # 5 ``` it will show the stacktrace from the rescued `peek` call: ``` $ ruby enum.rb enum.rb:4:in `peek': iteration reached an end (StopIteration) from enum.rb:4:in `<main>' ``` Whereas the error should refer to `next` on line number 5. The same happens when calling `peek` after `next` or when having muliple `peek` / `next` calls: ```ruby # enum.rb # 1 # 2 enum = [].each # 3 enum.peek rescue nil # 4 enum.next rescue nil # 5 enum.peek rescue nil # 6 puts "line #{__LINE__}" # 7 enum.next # 8 ``` The stacktrace from the first (rescued) `peek` or `next` call will be shown which doesn't reflect the actual error location: ``` $ ruby enum.rb line 7 enum.rb:4:in `peek': iteration reached an end (StopIteration) from enum.rb:4:in `<main>' ``` This is very confusing when debugging code. ---Files-------------------------------- 01-Recreate-stacktrace-enumerator.patch (1.29 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113864] [Ruby master Bug#19723] [RFC] Deprecate/disallow passing `"|command..." values to open-uri's URI.open() method
by postmodern (Hal Brodigan) 14 Jul '23

14 Jul '23

Issue #19723 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Bug #19723: [RFC] Deprecate/disallow passing `"|command..." values to open-uri's URI.open() method https://bugs.ruby-lang.org/issues/19723 * Author: postmodern (Hal Brodigan) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Due to `Kernel.open()` supporting opening pipe-commands (ex: `"|command-here..."`) this has led to multiple [1] security [2] vulnerabilities [3], where malicious user-input eventually is passed to `Kernel.open()`. One of the code-paths that malicious user-input can reach `Kernel.open()` is via open-uri's `URI.open()` method. RuboCop even recommends avoiding using `URI.open()` in favor of `uri = URI.parse(...); uri.open` to avoid accidentally opening malicious `"|command..."` inputs. I propose that `URI.open()` should not accept pipe-commands, as they are neither URIs nor files. One could even argue that `URI.open()` should only accept URIs and never fallback to `Kernel.open()`. [1]: https://45w1nkv.medium.com/ruby-code-vulnerability-analysis-confirmsnssubsc… [2]: https://bishopfox.com/blog/ruby-vulnerabilities-exploits [3]: https://blog.heroku.com/identifying-ruby-ftp-cve -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114064] [Ruby master Feature#19752] Allow `--backtrace-limit` to appear in RUBYOPT
by tomstuart (Tom Stuart) 13 Jul '23

13 Jul '23

Issue #19752 has been reported by tomstuart (Tom Stuart). ---------------------------------------- Feature #19752: Allow `--backtrace-limit` to appear in RUBYOPT https://bugs.ruby-lang.org/issues/19752 * Author: tomstuart (Tom Stuart) * Status: Open * Priority: Normal ---------------------------------------- The `--backtrace-limit` option was added in #8661 but it is not currently supported in the `RUBYOPT` environment variable, which appears to be a mistake. Unlike other long options which are not allowed in `RUBYOPT` (e.g. `--copyright`, `--version`, `--dump` and `--help`) it does not cause the interpreter to terminate, and cannot cause harm if read from the environment. During the initial discussion about the `--backtrace-limit` feature, Matz [suggested](https://bugs.ruby-lang.org/issues/8661#note-27) that he expected `RUBYOPT` to allow this option. Patch here, which also documents `--backtrace-limit` on Ruby’s `man` page: https://github.com/ruby/ruby/pull/8010 -- https://bugs.ruby-lang.org/

4 7

[ruby-core:113251] [Ruby master Feature#19600] Method `clamp?`
by sawa (Tsuyoshi Sawada) 13 Jul '23

13 Jul '23

Issue #19600 has been reported by sawa (Tsuyoshi Sawada). ---------------------------------------- Feature #19600: Method `clamp?` https://bugs.ruby-lang.org/issues/19600 * Author: sawa (Tsuyoshi Sawada) * Status: Open * Priority: Normal ---------------------------------------- Currently, we have pairs of non-predicate and predicate methods like `String#match` and `String#match?`. They are brain-friendly, and make programmers happier. Along this line, I propose the following: 1. Since by #19588, `Comparable#clamp`'s behavior is made the same as `Range#cover?` for range arguments, alias `Range#cover?` as `Range#clamp?`. 2. Synchronize the specification of `Comparable#between?` with `Comparable#clamp`, i.e., a. Allow `Comparable#between?` to take range arguments, and b. Allow `Comparable#between?` to take `nil` as either or both of its arguments, or as either or both ends of its range argument. 3. Alias `Comparable#between?` as `Comparable#clamp?` -- https://bugs.ruby-lang.org/

4 3

[ruby-core:114149] [Ruby master Bug#18572] Performance regression when invoking refined methods
by shugo (Shugo Maeda) 13 Jul '23

13 Jul '23

Issue #18572 has been updated by shugo (Shugo Maeda). shugo (Shugo Maeda) wrote in #note-10: > Eregon (Benoit Daloze) wrote in #note-9: > > From https://bugs.ruby-lang.org/issues/14083#note-3 it seems part of the problem at least is CRuby currently implements what I would call invalid usages of refinements (different set of activates refinements over time for a given call site), and that basically means extra checks and overhead, e.g., for a method which was refined once, even if there are no refinements active in the current scope. > > If it's needed to fix the performance issue, it may be acceptable to raise exceptions in some cases (but I don't know the exact condition of such cases). > @ko1, @matz What do you think? > > It's simple to prohibit using calls in blocks, but it will break backward compatibility, e.g., using in module_eval. I and @ko1 discussed it at today's developer meeting, and concluded that there is no need to change the behavior to fix the performance regression. @eregon Could you create another issue if you would like to change the behavior? ---------------------------------------- Bug #18572: Performance regression when invoking refined methods https://bugs.ruby-lang.org/issues/18572#change-103827 * Author: palkan (Vladimir Dementyev) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Since Ruby 3.0, defining a refinement for a method slows down its execution even if we do not activate the refinement: ```ruby require "benchmark_driver" source = <<~RUBY class Hash def symbolize_keys transform_keys { |key| key.to_sym rescue key } end def refined_symbolize_keys transform_keys { |key| key.to_sym rescue key } end end module HashRefinements refine Hash do def refined_symbolize_keys raise "never called" end end end HASH = {foo: 1, bar: 2, baz: 3} class Foo def original end def refined end end module FooRefinements refine Foo do def refined raise "never called" end end end FOO = Foo.new RUBY Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "#symbolize_keys original", %{ HASH.symbolize_keys } x.report "#symbolize_keys refined", %{ HASH.refined_symbolize_keys } end Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "no-op original", %{ FOO.original } x.report "no-op refined", %{ FOO.refined } end ``` The results for Ruby 3.1: ```sh ... Comparison: #symbolize_keys original: 2372420.1 i/s #symbolize_keys refined: 1941019.0 i/s - 1.22x slower ... Comparison: no-op original: 51790974.2 i/s no-op refined: 14456518.9 i/s - 3.58x slower ``` For Ruby 2.6 and 2.7: ```sh Comparison: #symbolize_keys original: 2278339.7 i/s #symbolize_keys refined: 2264153.1 i/s - 1.01x slower ... Comparison: no-op refined: 64178338.5 i/s no-op original: 63357980.1 i/s - 1.01x slower ``` You can find the full code and more results in this [gist](https://gist.github.com/palkan/637dc83edd86d70b5dbf72f2a4d702e5). P.S. The problem was originally noticed by @byroot, see https://github.com/ruby-i18n/i18n/pull/573 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113693] [Ruby master Feature#19698] Need an idiomatic way to define an empty range
by shouichi (Shouichi KAMIYA) 12 Jul '23

12 Jul '23

Issue #19698 has been reported by shouichi (Shouichi KAMIYA). ---------------------------------------- Feature #19698: Need an idiomatic way to define an empty range https://bugs.ruby-lang.org/issues/19698 * Author: shouichi (Shouichi KAMIYA) * Status: Open * Priority: Normal ---------------------------------------- It seems there is no idiomatic way to define an empty range. We can define empty ranges in several ways such as ``` ruby irb(main):001:0> (0...0).none? => true irb(main):002:0> (1...-1).none? => true ``` I think the above examples do not communicate the intention well. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:113899] [Ruby master Feature#19730] Remove transient heap
by peterzhu2118 (Peter Zhu) 11 Jul '23

11 Jul '23

Issue #19730 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19730: Remove transient heap https://bugs.ruby-lang.org/issues/19730 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/7942 I'm proposing to remove the transient heap because Variable Width Allocation now supports all the types that were on the transient heap. Arrays, objects, hashes, and structs were on the transient heap. After Variable Width Allocation: - Arrays: allocated on the transient heap for very large arrays (>78 elements) or when elements are added and is too big for embedded. - Objects: allocated on the transient heap for very large objects (>78 instance variables) or when new instance variables are added and is too big for embedded. However, classes store an estimate of the number of instance variables objects will have, which means that objects are usually allocated with the correct capacity. - Hashes: no longer on the transient heap. - Structs: allocated on the transient heap for very large structs (>78 elements). As such, the cases where allocations are on the transient heap become rarer and thus no longer provide the performance benefits it once did. This is supported by the benchmark results below, where we see no performance change with the transient heap removed. However, we do see around a 5-10% decrease in memory usage for most benchmarks. ``` -------------- ----------- ---------- --------- ---------------------- ---------- --------- ------------------------- ------------------------ bench master (ms) stddev (%) RSS (MiB) no transient heap (ms) stddev (%) RSS (MiB) no transient heap 1st itr master/no transient heap activerecord 71.9 2.4 58.2 71.9 2.2 53.4 1.00 1.00 chunky_png 822.7 0.2 45.7 837.0 0.2 47.4 0.98 0.98 erubi_rails 20.5 13.5 95.1 20.3 13.3 90.6 1.00 1.01 hexapdf 2525.8 0.7 173.1 2546.9 0.8 162.4 1.00 0.99 liquid-c 66.2 0.8 38.4 65.9 0.2 34.3 1.01 1.00 liquid-compile 60.2 1.7 36.6 60.4 1.4 32.6 1.02 1.00 liquid-render 165.0 0.2 37.2 164.7 0.2 33.1 1.01 1.00 mail 138.4 0.1 50.4 136.6 0.2 46.5 1.03 1.01 optcarrot 5232.2 0.7 62.9 5151.2 0.7 58.6 1.00 1.02 psych-load 2082.8 0.1 37.1 2059.4 0.1 32.2 1.01 1.01 railsbench 2066.2 0.1 103.6 2051.7 0.4 99.3 1.01 1.01 ruby-lsp 6.4 6.5 45.8 6.4 6.2 41.4 1.01 0.99 sequel 73.0 1.1 40.5 73.1 1.0 36.1 1.00 1.00 -------------- ----------- ---------- --------- ---------------------- ---------- --------- ------------------------- ------------------------ ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112988] [Ruby master Bug#19544] Custom quotes inconsistency
by hurricup (Alexandr Evstigneev) 11 Jul '23

11 Jul '23

Issue #19544 has been reported by hurricup (Alexandr Evstigneev). ---------------------------------------- Bug #19544: Custom quotes inconsistency https://bugs.ruby-lang.org/issues/19544 * Author: hurricup (Alexandr Evstigneev) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- According to `parse_percent` in `parse.y`: For `%` we should be able use ascii non alphanumeric delimiters. But, `p % jim beam ;` does not work. Presumably because of some look-ahead distincting `%` from percent-string starter. Seems it is the same for any `\s` after `%` For the explicit `%` construction, with type specifier `%[QqWwIisrx]` we should be able to use non-alphanumeric ascii characters, which includes `\s`. But: This works: ``` p %s jim beam ``` And this does not: ``` p %w jim\ beam ``` with unterminated list error. First seems unavoidable, but second looks like obvious error of finding terminating `\n` -- https://bugs.ruby-lang.org/

3 4

[ruby-core:113898] [Ruby master Feature#19729] Store object age in a bitmap
by eightbitraptor (Matthew Valentine-House) 11 Jul '23

11 Jul '23

Issue #19729 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Feature #19729: Store object age in a bitmap https://bugs.ruby-lang.org/issues/19729 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal ---------------------------------------- ## Github PR [Store object age in a bitmap #7938](https://github.com/ruby/ruby/pull/7938) ## Abstract Ruby currently uses 2 bits of the flags in every object to track how many GC events the object has survived. Objects are created at age 0 and can currently grow to age 3, at which point they are considered "old". Similar to the work carried out for [bitmap marking in Ruby 2.0](https://bugs.ruby-lang.org/issues/5839) which moved the mark bit from the flags to a bitmap attached to a heap page; This PR moves the age bits out of the object flags and stores them in a bitmap. ## Description This PR creates a new bitmap `age_bits` on each heap page. the size of the bitmap is controlled by `HEAP_PAGE_BITMAP_LIMIT`, which roughly indicates how many objects need to be considered by that bitmap, and `RVALUE_AGE_BITS_SIZE` which varies how many bits we use per object to store the age. We also introduce functions `RVALUE_AGE_SET`, `RVALUE_AGE_GET`, `RVALUE_AGE_RESET` and `RVALUE_AGE_INC` to manipulate the age of an object pointed to by a VALUE. ## Impact **Benefits:** * Improved CoW performance, because GC no longer has to mutate the object header for objects that age. * Allow configuration of the age at which objects are considered old. Because the number of bits used is configurable, we can support arbitrary numbers of GC events before an object is considered old. We can use this to change major/minor GC timings for workloads with unusually high/low object churn. * Free a flag in each object that can be repurposed. Object flags are a precious resource, we should prefer to store data outside the flags where possible. * Remove GC related concerns from the object structure. This is important for initiatives like a generic GC interface and MMTk in order to keep GC related code as isolated as possible. **Concerns:** * Slightly increased RSS, because we now allocate an extra 2 bits per heap page slot, to create the age_bits bitmap on VM bootup. On my machine `sizeof(struct heap_page)` has increased from 1312 to 1728 bytes. ## Benchmarking Railsbench and Optcarrot benchmarked using `yjit-bench`. Showing small memory increase, but comparable performance. ``` master: ruby 3.3.0dev (2023-06-08T11:22:43Z master 3fe09eba9d) [x86_64-linux] mvh-rvalue-age-bitmap: ruby 3.3.0dev (2023-06-13T06:59:40Z master c74f42a4fb) [x86_64-linux] ---------- ----------- ---------- --------- -------------------------- ---------- --------- ----------------------------- ---------------------------- bench master (ms) stddev (%) RSS (MiB) mvh-rvalue-age-bitmap (ms) stddev (%) RSS (MiB) mvh-rvalue-age-bitmap 1st itr master/mvh-rvalue-age-bitmap railsbench 2154.3 0.5 101.4 2124.9 0.5 101.5 1.01 1.01 optcarrot 5372.2 0.6 55.0 5282.3 0.6 55.1 1.02 1.02 ---------- ----------- ---------- --------- -------------------------- ---------- --------- ----------------------------- ---------------------------- Legend: - mvh-rvalue-age-bitmap 1st itr: ratio of master/mvh-rvalue-age-bitmap time for the first benchmarking iteration. - master/mvh-rvalue-age-bitmap: ratio of master/mvh-rvalue-age-bitmap time. Higher is better for mvh-rvalue-age-bitmap. Above 1 represents a speedup. ``` ## Notes **FL_PROMOTED** We still use one of the two age bits. The original `FL_PROMOTED0` has been renamed to `FL_PROMOTED` and has been repurposed to indicate an objects old/young status. We need this because correctly tracking references from old to young objects relies on a write barrier, that's triggered whenever a field on an object is written to. Because this code is a very hot path it needs to be fast. Looking up the heap page and then calculating the old/young status based on the age bits would slow down this part of the code too much. Instead we set `FL_PROMOTED` whenever the object crosses the threshold into the old gen, and unset it if that object ever gets demoted back into the young gen. This way the write barrier can quickly tell whether the object is old or not and whether to add it to the rememberset. **rb_age_reset** I expose a function from gc.c to reset the object age. This is only used from one place: `Init_VM` in `vm.c`. We create a hidden class during boot for the VM's "Frozen Core". This is created using `rb_class_new` as a T_CLASS, then it has it's class path set, and then the flags are overwritten, forcing the type to T_ICLASS. Classes are allocated with age 2, and `rb_set_class_path` allocates, which may trigger GC. If this happens then the class will immediately become old, and have its `FL_PROMOTED` bit set. This will then be immediately wiped over when the flags are forced to T_ICLASS. This will result in the `FL_PROMOTED` bit and the object age being out of sync. I don't know why this code is using `rb_class_new` rather than `rb_include_class_new` but I will follow up this PR with an investigation. -- https://bugs.ruby-lang.org/

2 1