June 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113864] [Ruby master Bug#19723] [RFC] Deprecate/disallow passing `"|command..." values to open-uri's URI.open() method
by postmodern (Hal Brodigan) 14 Jul '23

14 Jul '23

Issue #19723 has been reported by postmodern (Hal Brodigan). ---------------------------------------- Bug #19723: [RFC] Deprecate/disallow passing `"|command..." values to open-uri's URI.open() method https://bugs.ruby-lang.org/issues/19723 * Author: postmodern (Hal Brodigan) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Due to `Kernel.open()` supporting opening pipe-commands (ex: `"|command-here..."`) this has led to multiple [1] security [2] vulnerabilities [3], where malicious user-input eventually is passed to `Kernel.open()`. One of the code-paths that malicious user-input can reach `Kernel.open()` is via open-uri's `URI.open()` method. RuboCop even recommends avoiding using `URI.open()` in favor of `uri = URI.parse(...); uri.open` to avoid accidentally opening malicious `"|command..."` inputs. I propose that `URI.open()` should not accept pipe-commands, as they are neither URIs nor files. One could even argue that `URI.open()` should only accept URIs and never fallback to `Kernel.open()`. [1]: https://45w1nkv.medium.com/ruby-code-vulnerability-analysis-confirmsnssubsc… [2]: https://bishopfox.com/blog/ruby-vulnerabilities-exploits [3]: https://blog.heroku.com/identifying-ruby-ftp-cve -- https://bugs.ruby-lang.org/

3 3

[ruby-core:113251] [Ruby master Feature#19600] Method `clamp?`
by sawa (Tsuyoshi Sawada) 13 Jul '23

13 Jul '23

Issue #19600 has been reported by sawa (Tsuyoshi Sawada). ---------------------------------------- Feature #19600: Method `clamp?` https://bugs.ruby-lang.org/issues/19600 * Author: sawa (Tsuyoshi Sawada) * Status: Open * Priority: Normal ---------------------------------------- Currently, we have pairs of non-predicate and predicate methods like `String#match` and `String#match?`. They are brain-friendly, and make programmers happier. Along this line, I propose the following: 1. Since by #19588, `Comparable#clamp`'s behavior is made the same as `Range#cover?` for range arguments, alias `Range#cover?` as `Range#clamp?`. 2. Synchronize the specification of `Comparable#between?` with `Comparable#clamp`, i.e., a. Allow `Comparable#between?` to take range arguments, and b. Allow `Comparable#between?` to take `nil` as either or both of its arguments, or as either or both ends of its range argument. 3. Alias `Comparable#between?` as `Comparable#clamp?` -- https://bugs.ruby-lang.org/

4 3

[ruby-core:113693] [Ruby master Feature#19698] Need an idiomatic way to define an empty range
by shouichi (Shouichi KAMIYA) 12 Jul '23

12 Jul '23

Issue #19698 has been reported by shouichi (Shouichi KAMIYA). ---------------------------------------- Feature #19698: Need an idiomatic way to define an empty range https://bugs.ruby-lang.org/issues/19698 * Author: shouichi (Shouichi KAMIYA) * Status: Open * Priority: Normal ---------------------------------------- It seems there is no idiomatic way to define an empty range. We can define empty ranges in several ways such as ``` ruby irb(main):001:0> (0...0).none? => true irb(main):002:0> (1...-1).none? => true ``` I think the above examples do not communicate the intention well. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:113899] [Ruby master Feature#19730] Remove transient heap
by peterzhu2118 (Peter Zhu) 11 Jul '23

11 Jul '23

Issue #19730 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19730: Remove transient heap https://bugs.ruby-lang.org/issues/19730 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/7942 I'm proposing to remove the transient heap because Variable Width Allocation now supports all the types that were on the transient heap. Arrays, objects, hashes, and structs were on the transient heap. After Variable Width Allocation: - Arrays: allocated on the transient heap for very large arrays (>78 elements) or when elements are added and is too big for embedded. - Objects: allocated on the transient heap for very large objects (>78 instance variables) or when new instance variables are added and is too big for embedded. However, classes store an estimate of the number of instance variables objects will have, which means that objects are usually allocated with the correct capacity. - Hashes: no longer on the transient heap. - Structs: allocated on the transient heap for very large structs (>78 elements). As such, the cases where allocations are on the transient heap become rarer and thus no longer provide the performance benefits it once did. This is supported by the benchmark results below, where we see no performance change with the transient heap removed. However, we do see around a 5-10% decrease in memory usage for most benchmarks. ``` -------------- ----------- ---------- --------- ---------------------- ---------- --------- ------------------------- ------------------------ bench master (ms) stddev (%) RSS (MiB) no transient heap (ms) stddev (%) RSS (MiB) no transient heap 1st itr master/no transient heap activerecord 71.9 2.4 58.2 71.9 2.2 53.4 1.00 1.00 chunky_png 822.7 0.2 45.7 837.0 0.2 47.4 0.98 0.98 erubi_rails 20.5 13.5 95.1 20.3 13.3 90.6 1.00 1.01 hexapdf 2525.8 0.7 173.1 2546.9 0.8 162.4 1.00 0.99 liquid-c 66.2 0.8 38.4 65.9 0.2 34.3 1.01 1.00 liquid-compile 60.2 1.7 36.6 60.4 1.4 32.6 1.02 1.00 liquid-render 165.0 0.2 37.2 164.7 0.2 33.1 1.01 1.00 mail 138.4 0.1 50.4 136.6 0.2 46.5 1.03 1.01 optcarrot 5232.2 0.7 62.9 5151.2 0.7 58.6 1.00 1.02 psych-load 2082.8 0.1 37.1 2059.4 0.1 32.2 1.01 1.01 railsbench 2066.2 0.1 103.6 2051.7 0.4 99.3 1.01 1.01 ruby-lsp 6.4 6.5 45.8 6.4 6.2 41.4 1.01 0.99 sequel 73.0 1.1 40.5 73.1 1.0 36.1 1.00 1.00 -------------- ----------- ---------- --------- ---------------------- ---------- --------- ------------------------- ------------------------ ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:112988] [Ruby master Bug#19544] Custom quotes inconsistency
by hurricup (Alexandr Evstigneev) 11 Jul '23

11 Jul '23

Issue #19544 has been reported by hurricup (Alexandr Evstigneev). ---------------------------------------- Bug #19544: Custom quotes inconsistency https://bugs.ruby-lang.org/issues/19544 * Author: hurricup (Alexandr Evstigneev) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- According to `parse_percent` in `parse.y`: For `%` we should be able use ascii non alphanumeric delimiters. But, `p % jim beam ;` does not work. Presumably because of some look-ahead distincting `%` from percent-string starter. Seems it is the same for any `\s` after `%` For the explicit `%` construction, with type specifier `%[QqWwIisrx]` we should be able to use non-alphanumeric ascii characters, which includes `\s`. But: This works: ``` p %s jim beam ``` And this does not: ``` p %w jim\ beam ``` with unterminated list error. First seems unavoidable, but second looks like obvious error of finding terminating `\n` -- https://bugs.ruby-lang.org/

3 4

[ruby-core:113898] [Ruby master Feature#19729] Store object age in a bitmap
by eightbitraptor (Matthew Valentine-House) 11 Jul '23

11 Jul '23

Issue #19729 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Feature #19729: Store object age in a bitmap https://bugs.ruby-lang.org/issues/19729 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal ---------------------------------------- ## Github PR [Store object age in a bitmap #7938](https://github.com/ruby/ruby/pull/7938) ## Abstract Ruby currently uses 2 bits of the flags in every object to track how many GC events the object has survived. Objects are created at age 0 and can currently grow to age 3, at which point they are considered "old". Similar to the work carried out for [bitmap marking in Ruby 2.0](https://bugs.ruby-lang.org/issues/5839) which moved the mark bit from the flags to a bitmap attached to a heap page; This PR moves the age bits out of the object flags and stores them in a bitmap. ## Description This PR creates a new bitmap `age_bits` on each heap page. the size of the bitmap is controlled by `HEAP_PAGE_BITMAP_LIMIT`, which roughly indicates how many objects need to be considered by that bitmap, and `RVALUE_AGE_BITS_SIZE` which varies how many bits we use per object to store the age. We also introduce functions `RVALUE_AGE_SET`, `RVALUE_AGE_GET`, `RVALUE_AGE_RESET` and `RVALUE_AGE_INC` to manipulate the age of an object pointed to by a VALUE. ## Impact **Benefits:** * Improved CoW performance, because GC no longer has to mutate the object header for objects that age. * Allow configuration of the age at which objects are considered old. Because the number of bits used is configurable, we can support arbitrary numbers of GC events before an object is considered old. We can use this to change major/minor GC timings for workloads with unusually high/low object churn. * Free a flag in each object that can be repurposed. Object flags are a precious resource, we should prefer to store data outside the flags where possible. * Remove GC related concerns from the object structure. This is important for initiatives like a generic GC interface and MMTk in order to keep GC related code as isolated as possible. **Concerns:** * Slightly increased RSS, because we now allocate an extra 2 bits per heap page slot, to create the age_bits bitmap on VM bootup. On my machine `sizeof(struct heap_page)` has increased from 1312 to 1728 bytes. ## Benchmarking Railsbench and Optcarrot benchmarked using `yjit-bench`. Showing small memory increase, but comparable performance. ``` master: ruby 3.3.0dev (2023-06-08T11:22:43Z master 3fe09eba9d) [x86_64-linux] mvh-rvalue-age-bitmap: ruby 3.3.0dev (2023-06-13T06:59:40Z master c74f42a4fb) [x86_64-linux] ---------- ----------- ---------- --------- -------------------------- ---------- --------- ----------------------------- ---------------------------- bench master (ms) stddev (%) RSS (MiB) mvh-rvalue-age-bitmap (ms) stddev (%) RSS (MiB) mvh-rvalue-age-bitmap 1st itr master/mvh-rvalue-age-bitmap railsbench 2154.3 0.5 101.4 2124.9 0.5 101.5 1.01 1.01 optcarrot 5372.2 0.6 55.0 5282.3 0.6 55.1 1.02 1.02 ---------- ----------- ---------- --------- -------------------------- ---------- --------- ----------------------------- ---------------------------- Legend: - mvh-rvalue-age-bitmap 1st itr: ratio of master/mvh-rvalue-age-bitmap time for the first benchmarking iteration. - master/mvh-rvalue-age-bitmap: ratio of master/mvh-rvalue-age-bitmap time. Higher is better for mvh-rvalue-age-bitmap. Above 1 represents a speedup. ``` ## Notes **FL_PROMOTED** We still use one of the two age bits. The original `FL_PROMOTED0` has been renamed to `FL_PROMOTED` and has been repurposed to indicate an objects old/young status. We need this because correctly tracking references from old to young objects relies on a write barrier, that's triggered whenever a field on an object is written to. Because this code is a very hot path it needs to be fast. Looking up the heap page and then calculating the old/young status based on the age bits would slow down this part of the code too much. Instead we set `FL_PROMOTED` whenever the object crosses the threshold into the old gen, and unset it if that object ever gets demoted back into the young gen. This way the write barrier can quickly tell whether the object is old or not and whether to add it to the rememberset. **rb_age_reset** I expose a function from gc.c to reset the object age. This is only used from one place: `Init_VM` in `vm.c`. We create a hidden class during boot for the VM's "Frozen Core". This is created using `rb_class_new` as a T_CLASS, then it has it's class path set, and then the flags are overwritten, forcing the type to T_ICLASS. Classes are allocated with age 2, and `rb_set_class_path` allocates, which may trigger GC. If this happens then the class will immediately become old, and have its `FL_PROMOTED` bit set. This will then be immediately wiped over when the flags are forced to T_ICLASS. This will result in the `FL_PROMOTED` bit and the object age being out of sync. I don't know why this code is using `rb_class_new` rather than `rb_include_class_new` but I will follow up this PR with an investigation. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:113835] [Ruby master Misc#19722] DevMeeting-2023-07-13
by mame (Yusuke Endoh) 10 Jul '23

10 Jul '23

Issue #19722 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19722: DevMeeting-2023-07-13 https://bugs.ruby-lang.org/issues/19722 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/07/13 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/07/10. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

9 8

[ruby-core:112773] [Ruby master Feature#19521] Support for `Module#name=` and `Class#name=`.
by ioquatix (Samuel Williams) 02 Jul '23

02 Jul '23

Issue #19521 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19521: Support for `Module#name=` and `Class#name=`. https://bugs.ruby-lang.org/issues/19521 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- See https://bugs.ruby-lang.org/issues/19450 for previous discussion and motivation. This proposal introduces `Class#name=` to set the temporary class name. This is an alternative or addition to <https://bugs.ruby-lang.org/issues/19520>. ```ruby c = Class.new do self.name = "fake name" end c = Class.new c.name = "fake name" ``` Alternatively, we could use `set_name`: ```ruby Class.new do set_name "fake_name" end ``` ## Example usage The current Ruby test suite has code which shows the usefulness of this new method: ```ruby def labeled_module(name, &block) Module.new do singleton_class.class_eval { define_method(:to_s) {name} alias inspect to_s alias name to_s } class_eval(&block) if block end end module_function :labeled_module def labeled_class(name, superclass = Object, &block) Class.new(superclass) do singleton_class.class_eval { define_method(:to_s) {name} alias inspect to_s alias name to_s } class_eval(&block) if block end end module_function :labeled_class ``` The updated code would look like this: ```ruby def labeled_module(name, &block) Module.new do self.name = name class_eval(&block) if block end end def labeled_class(name, superclass = Object, &block) Class.new(superclass) do self.name = name class_eval(&block) if block end end module_function :labeled_class ``` Because the name cannot be set as part of `.new`, we have to have a separate block to set the name, before calling `class_eval`. I think the ergonomics and performance of this are slightly worse than the [counter proposal](https://bugs.ruby-lang.org/issues/19520). -- https://bugs.ruby-lang.org/

10 30

[ruby-core:114032] [Ruby master Misc#19747] Propose Kevin Newton and Jemma Issroff as core committers
by k0kubun (Takashi Kokubun) 29 Jun '23

29 Jun '23

Issue #19747 has been reported by k0kubun (Takashi Kokubun). ---------------------------------------- Misc #19747: Propose Kevin Newton and Jemma Issroff as core committers https://bugs.ruby-lang.org/issues/19747 * Author: k0kubun (Takashi Kokubun) * Status: Open * Priority: Normal ---------------------------------------- # Proposal I'd like to propose Kevin Newton (@kddnewton) and Jemma Issroff (@jemmai) as core committers. Now that YARP [Feature #19741] is merged, I want them to work on YARP's CRuby integration without asking other committers to merge their patches. # Background They have not only authored YARP but also contributed various other things to CRuby. I'll include their message for this proposal. ## Kevin Newton I have been working on [ruby/ruby](https://github.com/ruby/ruby/commits?author=kddnewton) since early 2021. My main contributions so far have been: * **YJIT:** I helped design the IR and then implemented the backend for ARM64 support. * **Constants:** I designed and built a new system for constant caches to improve copy-on-write performance and help invalidate less JIT-ed code. * **YARP:** I designed and built YARP from the ground up, which was recently mirrored into CRuby. I want to continue to contribute to ruby/ruby to help maintain the work I've done so far and build more going forward. Being a committer would make it easier to maintain YARP and more easily respond to questions/concerns. ## Jemma Issroff I have been [an active contributor to ruby/ruby](https://github.com/ruby/ruby/pulls?page=1&q=author%3Ajemmaissrof…. Here are a few highlights of my contributions so far: * **Object Shapes:** I implemented the object shapes technique which fundamentally changed how instance variable access worked, to simplify cache checks, increase cache hits, and decrease runtime checks on instance variables. [I spoke about this work](https://www.youtube.com/watch?v=So-KvN3p-eE) at RubyKaigi 2022. https://bugs.ruby-lang.org/issues/18776 * **YARP:** I made [significant contributions to YARP](https://github.com/ruby/yarp/pulls?q=author%3Ajemmaissroff), including writing the code to mirror it into ruby/ruby. https://bugs.ruby-lang.org/issues/18776 * **Documentation:** I have worked on improving the documentation within ruby/ruby, and migrating the Wiki to make it easier to edit, avoid duplication, and stay up to date. [I spoke about contributing to CRuby](https://www.youtube.com/watch?v=7y6DhXQU4wU) at RubyKaigi 2023. https://bugs.ruby-lang.org/issues/19679 I would like to continue to contribute to Ruby. Specifically, being a Ruby committer will reduce friction as I work to compile YARP's AST into ISeqs and fully integrate YARP into ruby/ruby. # Onboarding Aaron and I have helped them work on the CRuby integration. We will continue to help them as they onboard as Ruby committers. -- https://bugs.ruby-lang.org/

7 7

[ruby-core:114029] [Ruby master Bug#19745] Confirm correct behaviour when attaching private method with `#define_method`/`#define_singleton_method`
by itarato (Peter Arato) 28 Jun '23

28 Jun '23

Issue #19745 has been reported by itarato (Peter Arato). ---------------------------------------- Bug #19745: Confirm correct behaviour when attaching private method with `#define_method`/`#define_singleton_method` https://bugs.ruby-lang.org/issues/19745 * Author: itarato (Peter Arato) * Status: Open * Priority: Normal * ruby -v: HEAD * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Should dynamically added private methods be accessible publicly? See the following example? ```ruby private def bar; end foo = Object.new foo.define_singleton_method(:bar, method(:bar)) foo.bar # No error. ``` The script above runs fine on latest Ruby HEAD. Is this correct to ignore the fact that the added method (`method(:bar)`) is private? This came up during a TruffleRuby investigation (https://github.com/oracle/truffleruby/issues/3134) where the result for the same script is: `private method 'bar' called for #<Object:0xc8> (NoMethodError)` -- https://bugs.ruby-lang.org/

3 2