June 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:112723] [Ruby master Bug#19483] rubyzip not behaving well with YJIT
by k0kubun (Takashi Kokubun) 16 Jul '23

16 Jul '23

Issue #19483 has been reported by k0kubun (Takashi Kokubun). ---------------------------------------- Bug #19483: rubyzip not behaving well with YJIT https://bugs.ruby-lang.org/issues/19483 * Author: k0kubun (Takashi Kokubun) * Status: Open * Priority: Normal * Assignee: yjit * Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- Copied from https://github.com/Shopify/yjit/issues/310 for backports. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:112578] [Ruby master Bug#19463] YJIT `[BUG] Stack consistency error` under certain invalidation scenarios
by alanwu (Alan Wu) 16 Jul '23

16 Jul '23

Issue #19463 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #19463: YJIT `[BUG] Stack consistency error` under certain invalidation scenarios https://bugs.ruby-lang.org/issues/19463 * Author: alanwu (Alan Wu) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.1 (2023-02-08 revision 31819e82c8) +YJIT [arm64-darwin22] * Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED ---------------------------------------- > test.rb:19: [BUG] Stack consistency error (sp: 15, bp: 16) With the following: ```ruby klass = Class.new do def self.lookup(hash, key) = hash[key] def self.foo(a, b) = [] def self.test(hash, key) [lookup(hash, key), key, "".freeze] # 05 opt_send_without_block :lookup # 07 getlocal_WC_0 :hash # 09 opt_str_freeze "" # 12 newarray 3 # 14 leave # # YJIT will put instructions (07..14) into a block. # When String#freeze is redefined from within lookup(), # the return address to the block is still on-stack. We rely # on invalidation patching the code at the return address # to service this situation correctly. end end # get YJIT to compile test() hash = { 1 => [] } 31.times { klass.test(hash, 1) } # inject invalidation into lookup() evil_hash = Hash.new do |_, key| class String undef :freeze def freeze = :ugokanai end key end p klass.test(evil_hash, 1) ``` The fix is fairly simple and I'll apply it shortly. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:113779] [Ruby master Feature#19714] Add Refinement#refined_module
by shugo (Shugo Maeda) 15 Jul '23

15 Jul '23

Issue #19714 has been reported by shugo (Shugo Maeda). ---------------------------------------- Feature #19714: Add Refinement#refined_module https://bugs.ruby-lang.org/issues/19714 * Author: shugo (Shugo Maeda) * Status: Assigned * Priority: Normal * Assignee: matz (Yukihiro Matsumoto) * Target version: 3.3 ---------------------------------------- Eregon (Benoit Daloze) wrote in https://bugs.ruby-lang.org/issues/12737#note-15: > `Refinement#refined_class` is a bit strange given it can return a module. > How about adding `Refinement#refined_module` as an alias for clarity? I'm for it. What do you think, Matz? -- https://bugs.ruby-lang.org/

4 6

[ruby-core:113888] [Ruby master Bug#19726] Script loadde twice when requiring self
by johannes (Johannes Krude) 14 Jul '23

14 Jul '23

Issue #19726 has been reported by johannes (Johannes Krude). ---------------------------------------- Bug #19726: Script loadde twice when requiring self https://bugs.ruby-lang.org/issues/19726 * Author: johannes (Johannes Krude) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The executed program is not added to `$LOADED_FEATURES` unexpectedly resulting in the program possibly being loaded twice: a.rb: ``` #!/usr/bin/env ruby require "./b" C = 42 return unless __FILE__ == $0 puts C ``` b.rb: ``` require "./a" ``` This results in: ``` $ ./a.rb ./a.rb:5: warning: already initialized constant C /home/johannes/t/a.rb:5: warning: previous definition of C was here 42 ``` Ruby version: ``` $ sudo docker run ruby:3.3.0-preview1 ruby --version ruby 3.3.0preview1 (2023-05-12 master a1b01e7701) [x86_64-linux] ``` **This could be solved by always adding the executed program to `$LOADED_FEATURES`.** As a workaround, executed programs can be prepended with: ``` require "pathname" if __FILE__ == $0 $LOADED_FEATURES << Pathname.new(__FILE__).expand_path.to_s end ``` -- https://bugs.ruby-lang.org/

5 5

[ruby-core:113864] [Ruby master Bug#19723] [RFC] Deprecate/disallow passing `"|command..." values to open-uri's URI.open() method
by postmodern (Hal Brodigan) 14 Jul '23

14 Jul '23

Issue #19723 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Bug #19723: [RFC] Deprecate/disallow passing `"|command..." values to open-uri's URI.open() method https://bugs.ruby-lang.org/issues/19723 * Author: postmodern (Hal Brodigan) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Due to `Kernel.open()` supporting opening pipe-commands (ex: `"|command-here..."`) this has led to multiple [1] security [2] vulnerabilities [3], where malicious user-input eventually is passed to `Kernel.open()`. One of the code-paths that malicious user-input can reach `Kernel.open()` is via open-uri's `URI.open()` method. RuboCop even recommends avoiding using `URI.open()` in favor of `uri = URI.parse(...); uri.open` to avoid accidentally opening malicious `"|command..."` inputs. I propose that `URI.open()` should not accept pipe-commands, as they are neither URIs nor files. One could even argue that `URI.open()` should only accept URIs and never fallback to `Kernel.open()`. [1]: https://45w1nkv.medium.com/ruby-code-vulnerability-analysis-confirmsnssubsc… [2]: https://bishopfox.com/blog/ruby-vulnerabilities-exploits [3]: https://blog.heroku.com/identifying-ruby-ftp-cve -- https://bugs.ruby-lang.org/

3 3

[ruby-core:113251] [Ruby master Feature#19600] Method `clamp?`
by sawa (Tsuyoshi Sawada) 13 Jul '23

13 Jul '23

Issue #19600 has been reported by sawa (Tsuyoshi Sawada). ---------------------------------------- Feature #19600: Method `clamp?` https://bugs.ruby-lang.org/issues/19600 * Author: sawa (Tsuyoshi Sawada) * Status: Open * Priority: Normal ---------------------------------------- Currently, we have pairs of non-predicate and predicate methods like `String#match` and `String#match?`. They are brain-friendly, and make programmers happier. Along this line, I propose the following: 1. Since by #19588, `Comparable#clamp`'s behavior is made the same as `Range#cover?` for range arguments, alias `Range#cover?` as `Range#clamp?`. 2. Synchronize the specification of `Comparable#between?` with `Comparable#clamp`, i.e., a. Allow `Comparable#between?` to take range arguments, and b. Allow `Comparable#between?` to take `nil` as either or both of its arguments, or as either or both ends of its range argument. 3. Alias `Comparable#between?` as `Comparable#clamp?` -- https://bugs.ruby-lang.org/

4 3

[ruby-core:113693] [Ruby master Feature#19698] Need an idiomatic way to define an empty range
by shouichi (Shouichi KAMIYA) 12 Jul '23

12 Jul '23

Issue #19698 has been reported by shouichi (Shouichi KAMIYA). ---------------------------------------- Feature #19698: Need an idiomatic way to define an empty range https://bugs.ruby-lang.org/issues/19698 * Author: shouichi (Shouichi KAMIYA) * Status: Open * Priority: Normal ---------------------------------------- It seems there is no idiomatic way to define an empty range. We can define empty ranges in several ways such as ``` ruby irb(main):001:0> (0...0).none? => true irb(main):002:0> (1...-1).none? => true ``` I think the above examples do not communicate the intention well. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:113899] [Ruby master Feature#19730] Remove transient heap
by peterzhu2118 (Peter Zhu) 11 Jul '23

11 Jul '23

Issue #19730 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19730: Remove transient heap https://bugs.ruby-lang.org/issues/19730 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/7942 I'm proposing to remove the transient heap because Variable Width Allocation now supports all the types that were on the transient heap. Arrays, objects, hashes, and structs were on the transient heap. After Variable Width Allocation: - Arrays: allocated on the transient heap for very large arrays (>78 elements) or when elements are added and is too big for embedded. - Objects: allocated on the transient heap for very large objects (>78 instance variables) or when new instance variables are added and is too big for embedded. However, classes store an estimate of the number of instance variables objects will have, which means that objects are usually allocated with the correct capacity. - Hashes: no longer on the transient heap. - Structs: allocated on the transient heap for very large structs (>78 elements). As such, the cases where allocations are on the transient heap become rarer and thus no longer provide the performance benefits it once did. This is supported by the benchmark results below, where we see no performance change with the transient heap removed. However, we do see around a 5-10% decrease in memory usage for most benchmarks. ``` -------------- ----------- ---------- --------- ---------------------- ---------- --------- ------------------------- ------------------------ bench master (ms) stddev (%) RSS (MiB) no transient heap (ms) stddev (%) RSS (MiB) no transient heap 1st itr master/no transient heap activerecord 71.9 2.4 58.2 71.9 2.2 53.4 1.00 1.00 chunky_png 822.7 0.2 45.7 837.0 0.2 47.4 0.98 0.98 erubi_rails 20.5 13.5 95.1 20.3 13.3 90.6 1.00 1.01 hexapdf 2525.8 0.7 173.1 2546.9 0.8 162.4 1.00 0.99 liquid-c 66.2 0.8 38.4 65.9 0.2 34.3 1.01 1.00 liquid-compile 60.2 1.7 36.6 60.4 1.4 32.6 1.02 1.00 liquid-render 165.0 0.2 37.2 164.7 0.2 33.1 1.01 1.00 mail 138.4 0.1 50.4 136.6 0.2 46.5 1.03 1.01 optcarrot 5232.2 0.7 62.9 5151.2 0.7 58.6 1.00 1.02 psych-load 2082.8 0.1 37.1 2059.4 0.1 32.2 1.01 1.01 railsbench 2066.2 0.1 103.6 2051.7 0.4 99.3 1.01 1.01 ruby-lsp 6.4 6.5 45.8 6.4 6.2 41.4 1.01 0.99 sequel 73.0 1.1 40.5 73.1 1.0 36.1 1.00 1.00 -------------- ----------- ---------- --------- ---------------------- ---------- --------- ------------------------- ------------------------ ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112988] [Ruby master Bug#19544] Custom quotes inconsistency
by hurricup (Alexandr Evstigneev) 11 Jul '23

11 Jul '23

Issue #19544 has been reported by hurricup (Alexandr Evstigneev). ---------------------------------------- Bug #19544: Custom quotes inconsistency https://bugs.ruby-lang.org/issues/19544 * Author: hurricup (Alexandr Evstigneev) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- According to `parse_percent` in `parse.y`: For `%` we should be able use ascii non alphanumeric delimiters. But, `p % jim beam ;` does not work. Presumably because of some look-ahead distincting `%` from percent-string starter. Seems it is the same for any `\s` after `%` For the explicit `%` construction, with type specifier `%[QqWwIisrx]` we should be able to use non-alphanumeric ascii characters, which includes `\s`. But: This works: ``` p %s jim beam ``` And this does not: ``` p %w jim\ beam ``` with unterminated list error. First seems unavoidable, but second looks like obvious error of finding terminating `\n` -- https://bugs.ruby-lang.org/

3 4

[ruby-core:113898] [Ruby master Feature#19729] Store object age in a bitmap
by eightbitraptor (Matthew Valentine-House) 11 Jul '23

11 Jul '23

Issue #19729 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Feature #19729: Store object age in a bitmap https://bugs.ruby-lang.org/issues/19729 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal ---------------------------------------- ## Github PR [Store object age in a bitmap #7938](https://github.com/ruby/ruby/pull/7938) ## Abstract Ruby currently uses 2 bits of the flags in every object to track how many GC events the object has survived. Objects are created at age 0 and can currently grow to age 3, at which point they are considered "old". Similar to the work carried out for [bitmap marking in Ruby 2.0](https://bugs.ruby-lang.org/issues/5839) which moved the mark bit from the flags to a bitmap attached to a heap page; This PR moves the age bits out of the object flags and stores them in a bitmap. ## Description This PR creates a new bitmap `age_bits` on each heap page. the size of the bitmap is controlled by `HEAP_PAGE_BITMAP_LIMIT`, which roughly indicates how many objects need to be considered by that bitmap, and `RVALUE_AGE_BITS_SIZE` which varies how many bits we use per object to store the age. We also introduce functions `RVALUE_AGE_SET`, `RVALUE_AGE_GET`, `RVALUE_AGE_RESET` and `RVALUE_AGE_INC` to manipulate the age of an object pointed to by a VALUE. ## Impact **Benefits:** * Improved CoW performance, because GC no longer has to mutate the object header for objects that age. * Allow configuration of the age at which objects are considered old. Because the number of bits used is configurable, we can support arbitrary numbers of GC events before an object is considered old. We can use this to change major/minor GC timings for workloads with unusually high/low object churn. * Free a flag in each object that can be repurposed. Object flags are a precious resource, we should prefer to store data outside the flags where possible. * Remove GC related concerns from the object structure. This is important for initiatives like a generic GC interface and MMTk in order to keep GC related code as isolated as possible. **Concerns:** * Slightly increased RSS, because we now allocate an extra 2 bits per heap page slot, to create the age_bits bitmap on VM bootup. On my machine `sizeof(struct heap_page)` has increased from 1312 to 1728 bytes. ## Benchmarking Railsbench and Optcarrot benchmarked using `yjit-bench`. Showing small memory increase, but comparable performance. ``` master: ruby 3.3.0dev (2023-06-08T11:22:43Z master 3fe09eba9d) [x86_64-linux] mvh-rvalue-age-bitmap: ruby 3.3.0dev (2023-06-13T06:59:40Z master c74f42a4fb) [x86_64-linux] ---------- ----------- ---------- --------- -------------------------- ---------- --------- ----------------------------- ---------------------------- bench master (ms) stddev (%) RSS (MiB) mvh-rvalue-age-bitmap (ms) stddev (%) RSS (MiB) mvh-rvalue-age-bitmap 1st itr master/mvh-rvalue-age-bitmap railsbench 2154.3 0.5 101.4 2124.9 0.5 101.5 1.01 1.01 optcarrot 5372.2 0.6 55.0 5282.3 0.6 55.1 1.02 1.02 ---------- ----------- ---------- --------- -------------------------- ---------- --------- ----------------------------- ---------------------------- Legend: - mvh-rvalue-age-bitmap 1st itr: ratio of master/mvh-rvalue-age-bitmap time for the first benchmarking iteration. - master/mvh-rvalue-age-bitmap: ratio of master/mvh-rvalue-age-bitmap time. Higher is better for mvh-rvalue-age-bitmap. Above 1 represents a speedup. ``` ## Notes **FL_PROMOTED** We still use one of the two age bits. The original `FL_PROMOTED0` has been renamed to `FL_PROMOTED` and has been repurposed to indicate an objects old/young status. We need this because correctly tracking references from old to young objects relies on a write barrier, that's triggered whenever a field on an object is written to. Because this code is a very hot path it needs to be fast. Looking up the heap page and then calculating the old/young status based on the age bits would slow down this part of the code too much. Instead we set `FL_PROMOTED` whenever the object crosses the threshold into the old gen, and unset it if that object ever gets demoted back into the young gen. This way the write barrier can quickly tell whether the object is old or not and whether to add it to the rememberset. **rb_age_reset** I expose a function from gc.c to reset the object age. This is only used from one place: `Init_VM` in `vm.c`. We create a hidden class during boot for the VM's "Frozen Core". This is created using `rb_class_new` as a T_CLASS, then it has it's class path set, and then the flags are overwritten, forcing the type to T_ICLASS. Classes are allocated with age 2, and `rb_set_class_path` allocates, which may trigger GC. If this happens then the class will immediately become old, and have its `FL_PROMOTED` bit set. This will then be immediately wiped over when the flags are forced to T_ICLASS. This will result in the `FL_PROMOTED` bit and the object age being out of sync. I don't know why this code is using `rb_class_new` rather than `rb_include_class_new` but I will follow up this PR with an investigation. -- https://bugs.ruby-lang.org/

2 1