August 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:118969] [Ruby master Feature#20702] Add `Array#fetch_values`
by toy (Ivan Kuchin) 05 Sep '24

05 Sep '24

Issue #20702 has been reported by toy (Ivan Kuchin). ---------------------------------------- Feature #20702: Add `Array#fetch_values` https://bugs.ruby-lang.org/issues/20702 * Author: toy (Ivan Kuchin) * Status: Open ---------------------------------------- `Array` and `Hash` have matching methods to fetch: * One value using `[]` (`Array` also allows fetching multiple with range or two arguments) * One value with default or exception using `fetch` method * Multiple values using `values_at` method But only `Hash` has method `fetch_values` to fetch multiple values with fallback or raising an exception (see #10017). ```ruby hash = {a: 1, b: 2, c: 3} array = [1, 2, 3] hash[:b] # => 2 hash[:d] # => nil array[1] # => 2 array[4] # => nil hash.fetch(:b) # => 2 hash.fetch(:d) # => IndexError hash.fetch(:d){ 42 } # => 42 array.fetch(1) # => 2 array.fetch(4) # => IndexError array.fetch(4){ 42 } # => 42 hash.values_at(:b, :c) # => [2, 3] hash.values_at(:b, :d) # => [2, nil] array.values_at(1, 2) # => [2, 3] array.values_at(1, 4) # => [2, nil] hash.fetch_values(:b, :c) # => [2, 3] hash.fetch_values(:b, :d) # => IndexError hash.fetch_values(:b, :d){ 42 } # => [2, 42] # missing array.fetch_values(1, 2) # => [2, 3] array.fetch_values(1, 4) # => IndexError array.fetch_values(1, 4){ 42 } # => [2, 42] ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118970] [Ruby master Feature#20703] Alias StringIO#string to StringIO#to_s
by sebyx07 (Sebastian Buza) 05 Sep '24

05 Sep '24

Issue #20703 has been reported by sebyx07 (Sebastian Buza). ---------------------------------------- Feature #20703: Alias StringIO#string to StringIO#to_s https://bugs.ruby-lang.org/issues/20703 * Author: sebyx07 (Sebastian Buza) * Status: Open ---------------------------------------- # Description Allow to use `StringIO.new('my string').to_s` to be the same as `StringIO.new('my string').string` IMO nobody really uses the current StringIO#to_s, so I don't think it will break other apps. That way you can also do interpolation easily ```ruby my_sio = StringIO.new('my string') puts "value: #{my_sio}" # value: my string ``` - issue: https://github.com/ruby/stringio/issues - pr: https://github.com/ruby/stringio/pull/103 -- https://bugs.ruby-lang.org/

4 4

[ruby-core:118994] [Ruby master Feature#20707] Move `Time#xmlschema` into core
by byroot (Jean Boussier) 05 Sep '24

05 Sep '24

Issue #20707 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20707: Move `Time#xmlschema` into core https://bugs.ruby-lang.org/issues/20707 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Performance Converting `Time` into `RFC3339 / ISO8601` representation is an significant hotspot for application that serialize data in JSON, XML or other formats. Right now this feature is currently available through the `time` default gem, so it rely on `Time#strftime`. While [`strftime` could certainly be optimized](https://github.com/ruby/ruby/pull/11508), ultimately it's a very generic API so can't make a lot of assumptions about the output. Whereas `ISO8601` is a very strict format, with very few dynamic part, so it's much easier for dedicated code to be well optimized, as the final size of the string can be computed upfront etc. [I experimented with this and was able to get a ~5x speedup with a fairly straightforward implementation](https://github.com/ruby/ruby/pull/11510). ``` compare-ruby: ruby 3.4.0dev (2024-08-29T13:11:40Z master 6b08a50a62) +YJIT [arm64-darwin23] built-ruby: ruby 3.4.0dev (2024-08-30T13:17:32Z native-xmlschema 34041ff71f) +YJIT [arm64-darwin23] warming up...... | |compare-ruby|built-ruby| |:-----------------------|-----------:|---------:| |time._xmlschema | 1.087M| 5.190M| | | -| 4.78x| |utc_time._xmlschema | 1.464M| 6.848M| | | -| 4.68x| |time._xmlschema(6) | 859.960k| 4.646M| | | -| 5.40x| |utc_time._xmlschema(6) | 1.080M| 5.917M| | | -| 5.48x| |time._xmlschema(9) | 893.909k| 4.668M| | | -| 5.22x| |utc_time._xmlschema(9) | 1.056M| 5.707M| | | -| 5.40x| ``` ### Usability Aside from the performance reason, `ISO8601` is so common that having this functionality built-in without needing to require anything would make sense to me. `xmlschema` is the most common reason why I require `time`, and in many case having it core would allow not to load it at all. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118891] [Ruby master Feature#20684] Add optimized instructions for frozen literal Hash and Array
by etienne 05 Sep '24

05 Sep '24

Issue #20684 has been reported by etienne (Étienne Barrié). ---------------------------------------- Feature #20684: Add optimized instructions for frozen literal Hash and Array https://bugs.ruby-lang.org/issues/20684 * Author: etienne (Étienne Barrié) * Status: Open ---------------------------------------- # Context Methods that take empty arrays or empty hashes as default values allocate a new object each time the method is called without the argument. Often they don't mutate the parameter. To prevent an allocation, in performance critical sections, a constant is defined that holds a frozen hash or array, and the constant is defined as the default value for the parameter. Here are some examples: Rails: https://github.com/rails/rails/blob/607d61e884237c223c24c6f47efa0b561dd8b63… Roda: https://github.com/jeremyevans/roda/blob/102926a02dcabc9a31674e3cf98f049139… dry-rb: https://github.com/dry-rb/dry-container/blob/1ee41bb109455d06bf22ebcbd94b05… and many other gems: https://gist.github.com/casperisfine/47f22243d4ad203855256ef5bfae7979 Additionally when defining a frozen literal constant, we're currently inefficient because we store the literal in the bytecode, we dup it just to freeze it again. It doesn't amount to much but would be nice to avoid. # Proposal Introduce 2 new optimized instructions `opt_ary_freeze` and `opt_hash_freeze` that behave like `opt_str_freeze` for their respective types. If the freeze method hasn't been redefined, they simply push the frozen literal value on the stack. Like for `opt_str_freeze`, these instructions are added by the peephole optimizer when applicable. In the specific case of empty array and empty hash, we use a pre-allocated global empty frozen object to avoid retaining a distinct empty object each time. This will allow code like this: https://github.com/ruby/ruby/blob/566f2eb501d94d4047a9aad4af0d74c6a96f34a9/… to be shortened and simplified like this: ```diff diff --git i/lib/rubygems/resolver/api_set/gem_parser.rb w/lib/rubygems/resolver/api_set/gem_parser.rb index 643b857107..34146fd426 100644 --- i/lib/rubygems/resolver/api_set/gem_parser.rb +++ w/lib/rubygems/resolver/api_set/gem_parser.rb @@ -1,15 +1,12 @@ # frozen_string_literal: true class Gem::Resolver::APISet::GemParser - EMPTY_ARRAY = [].freeze - private_constant :EMPTY_ARRAY - def parse(line) version_and_platform, rest = line.split(" ", 2) version, platform = version_and_platform.split("-", 2) dependencies, requirements = rest.split("|", 2).map! {|s| s.split(",") } if rest - dependencies = dependencies ? dependencies.map! {|d| parse_dependency(d) } : EMPTY_ARRAY - requirements = requirements ? requirements.map! {|d| parse_dependency(d) } : EMPTY_ARRAY + dependencies = dependencies ? dependencies.map! {|d| parse_dependency(d) } : [].freeze + requirements = requirements ? requirements.map! {|d| parse_dependency(d) } : [].freeze [version, platform, dependencies, requirements] end ``` Overall it's a minor optimization but also a very simple patch and makes code nicer. PR pending. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118542] [Ruby master Bug#20627] `require` on Ractor should run on the main Ractor
by ko1 (Koichi Sasada) 04 Sep '24

04 Sep '24

Issue #20627 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Bug #20627: `require` on Ractor should run on the main Ractor https://bugs.ruby-lang.org/issues/20627 * Author: ko1 (Koichi Sasada) * Status: Open * Assignee: ko1 (Koichi Sasada) * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Now `require` on main Ractor is not allowed (raising error) but it is hard, especially for `autoload`. So let's allow `require` by running it on the main Ractor. ## Background On many libraries it is needed to run loading on the main Ractors because: 1. Setup constants with unshareable objects (such as `C = []`) are not allowed on non main Ractors. 2. Setup global variables and class variables are not allowed. `$LOADED_FEATURES` is also untouchable. 3. (maybe more reasons) So the `require` on non main Ractors is not allowed. However it is hard to program especially on `autoload`. Also dynamic `require` (`require` in methods) are not allowed too (`pp` method, for example). ## Proposal Allow `require` on non main Ractors by running `require` process on the main Ractor. ![](clipboard-202407111201-kaiem.png) (quoted on my talk at RubyKaigi 2024) ### `rb_ractor_interrupt_exec(target_ractor, func)` C-API Make a thread on `target_ractor` and run `func` (C function) on it. I think it is safe to expose on Ruby API because running `func` on a *newly created* thread (do not disturb running target threads). But now it is proposed as only (hidden) C-API. ### New Ractor methods - `Rcator.main?` returns Ractors - `Ractor.require(feature)` do `require` on the main Ractor These new methods are useful for users who override `require` method like [RubyGems](https://github.com/ruby/ruby/blob/master/lib/rubygems/core_ext/ke…. ```ruby alias orig_require require def require feature return Rator.require(feature) unless Ractor.main? # overriding require code end ``` Or we can prepend a module like: ``` Module.new do def require(feature) return Rator.require(feature) unless Ractor.main? super(feature) end Kernel.prepend self end ``` will support ractors for all overgrinding methods. But not sure it is acceptable to add additional one modules in ancestors by `prepend`. Also this technique doesn't support require overriding by prepending. ## Implementation https://github.com/ruby/ruby/pull/11142 (not matured yet) ---Files-------------------------------- clipboard-202407111201-kaiem.png (70.8 KB) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118773] [Ruby master Misc#20660] DevMeeting-2024-09-05
by mame (Yusuke Endoh) 04 Sep '24

04 Sep '24

Issue #20660 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #20660: DevMeeting-2024-09-05 https://bugs.ruby-lang.org/issues/20660 * Author: mame (Yusuke Endoh) * Status: Open ---------------------------------------- # The next dev meeting **Date: 2024/09/05 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2024/09/02. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

6 6

[ruby-core:118996] [Ruby master Bug#20708] EINTR while opening fifo isn't retried
by martin.dorey＠hds.com (Martin Dorey) 03 Sep '24

03 Sep '24

Issue #20708 has been reported by martin.dorey(a)hds.com (Martin Dorey). ---------------------------------------- Bug #20708: EINTR while opening fifo isn't retried https://bugs.ruby-lang.org/issues/20708 * Author: martin.dorey(a)hds.com (Martin Dorey) * Status: Open * ruby -v: ruby 3.4.0dev (2024-08-30T20:36:28Z master eef2121cf3) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I don't think this should fail: ``` martind@stormy:~/download/ruby$ rm -f /tmp/badger && mkfifo /tmp/badger && timeout 2 ./ruby --disable-gems -we 'fork() {}; IO.read("/tmp/badger")' -e:1:in 'IO.read': Interrupted system call @ rb_sysopen - /tmp/badger (Errno::EINTR) from -e:1:in '<main>' martind@stormy:~/download/ruby$ ``` The behavior changed with 2.6.0-preview3 compared to 2.6.0-preview2: ``` martind@stormy:~/download/ruby-2.6.0-preview2$ rm -f /tmp/badger && mkfifo /tmp/badger && timeout 2 ./ruby --disable-gems -we 'fork() {}; IO.read("/tmp/badger")' martind@stormy:~/download/ruby-2.6.0-preview3$ rm -f /tmp/badger && mkfifo /tmp/badger && timeout 2 ./ruby --disable-gems -we 'fork() {}; IO.read("/tmp/badger")' Traceback (most recent call last): 1: from -e:1:in `<main>' -e:1:in `read': Interrupted system call @ rb_sysopen - /tmp/badger (Errno::EINTR) martind@stormy:~/download/ruby-2.6.0-preview3$ ``` I don't know what the consequences would be, so I don't mean this as a serious suggestion: ``` martind@stormy:~/download/ruby$ git diff diff --git a/io.c b/io.c index 54b60b7357..590e8d11d7 100644 --- a/io.c +++ b/io.c @@ -6956,7 +6956,10 @@ sysopen_func(void *ptr) static inline int rb_sysopen_internal(struct sysopen_struct *data) { - int fd = IO_WITHOUT_GVL_INT(sysopen_func, data); + int fd; + do { + fd = IO_WITHOUT_GVL_INT(sysopen_func, data); + } while (fd < 0 && errno == EINTR); if (0 <= fd) rb_update_max_fd(fd); return fd; martind@stormy:~/download/ruby$ ``` ... but it fixes this particular test case, for me, with today's Ruby from github. There were negligible differences between 2.6.0-preview{2,3} in io.c but the SIGCHLD handling was changed. The test case is reliable for me on at least: Linux stormy 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux Linux moritz 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux I hope it's independent of the kernel, as: https://man7.org/linux/man-pages/man7/signal.7.html ... teaches: > otherwise the call fails with the error EINTR: ... > • open(2), if it can block (e.g., when opening a FIFO; see > fifo(7)). -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118998] [Ruby master Feature#20709] Improve String#rindex performance on OSX
by zack.ref＠gmail.com (Zack Deveau) 03 Sep '24

03 Sep '24

Issue #20709 has been reported by zack.ref(a)gmail.com (Zack Deveau). ---------------------------------------- Feature #20709: Improve String#rindex performance on OSX https://bugs.ruby-lang.org/issues/20709 * Author: zack.ref(a)gmail.com (Zack Deveau) * Status: Open ---------------------------------------- `String#rindex` is much slower on OSX than on Linux hosts. This appears due to the lack of a `memrchr` implementation on OSX. The [fallback solution](https://github.com/ruby/ruby/blob/e69945fc57efba5c8895c21235e1091… currently performs a `memcmp` on each character in the search string looking for a substring match. ```c while (s) { if (memcmp(s, t, slen) == 0) { return s - sbeg; } if (s <= sbeg) break; s = rb_enc_prev_char(sbeg, s, e, enc); } ``` ``` ruby-master | 10 char substring (long) near the end of a search string | osx search string enc: UTF-8 search string size: 8010 user system total real index (long substring) 0.000075 0.000000 0.000075 ( 0.000074) rindex 0.000445 0.000000 0.000445 ( 0.000448) ``` Proposed improvement is available on this pull request: This patch proposes using an implementation of `memrchr` made for the interpreter on hosts that lack a native implementation. Below are benchmark results which demonstrate that this approach either improves or maintains performance of `String#rindex` for all cases we could come up with. ```c static void* rb_memrchr(const char *search_str, int chr, long search_len) { const char *ptr = search_str + search_len; do { if (*(--ptr) == chr) return (void *)ptr; } while (ptr >= search_str); return ((void *)0); } ``` Special thanks to @jhawthorn who provided some helpful feedback on this solution during development! # Benchmarks Each of the following were made with a variant of the following general purpose benchmark. Note that `String#index` speed can be [impacted by the size of the provided substring](https://github.com/ruby/ruby/blob/669d1f79d800d554918c7d5647b73a… (Denoted as long/short substring in the results, assume long substring when none is specified). ``` require "benchmark" substring = "98279827AA" the_string = "" 800.times{ the_string.concat(('a'..'z').to_a.shuffle[0,8].join)} the_string.concat(substring) 200.times{ the_string.concat(('a'..'z').to_a.shuffle[0,8].join)} puts the_string.encoding puts the_string.size Benchmark.bm do |x| x.report(:index) { 100.times { the_string.index(substring) } } x.report(:rindex) { 100.times { the_string.rindex(substring) } } end ``` ``` ruby-master | substring near the end | osx https://github.com/ruby/ruby/blob/669d1f79d800d554918c7d5647b73a6431920ea8/… UTF-8 user system total real index 0.000075 0.000000 0.000075 ( 0.000074) rindex 0.000445 0.000000 0.000445 ( 0.000448) ------------------------------------------------------ ruby-patched | substring near the end | osx https://github.com/ruby/ruby/blob/669d1f79d800d554918c7d5647b73a6431920ea8/… UTF-8 user system total real index (short substring) 0.000400 0.000001 0.000401 ( 0.000399) index (long substring) 0.000080 0.000000 0.000080 ( 0.000080) rindex 0.000057 0.000001 0.000058 ( 0.000057) ``` ``` ruby-master | substring in the middle | osx UTF-8 user system total real index 0.000031 0.000000 0.000031 ( 0.000030) rindex 0.001086 0.000002 0.001088 ( 0.001088) ------------------------------------------------------ ruby-patched | substring in the middle | osx UTF-8 user system total real index 0.000075 0.000000 0.000075 ( 0.000075) rindex 0.000130 0.000000 0.000130 ( 0.000133) ``` ``` ruby-master | substring in the middle with a short substring | osx https://github.com/ruby/ruby/blob/669d1f79d800d554918c7d5647b73a6431920ea8/… UTF-8 user system total real index (short substring) 0.000243 0.000004 0.000247 ( 0.000246) rindex 0.001176 0.000005 0.001181 ( 0.001182) ------------------------------------------------------ ruby-patched | substring in the middle with a short substring | osx https://github.com/ruby/ruby/blob/669d1f79d800d554918c7d5647b73a6431920ea8/… UTF-8 user system total real index (short substring) 0.000255 0.000005 0.000260 ( 0.000259) rindex 0.000130 0.000000 0.000130 ( 0.000133) ``` ``` ruby-master | substring at the end | osx UTF-8 user system total real index 0.000049 0.000001 0.000050 ( 0.000048) rindex 0.000008 0.000001 0.000009 ( 0.000007) ------------------------------------------------------ ruby-patched | substring at the end | osx UTF-8 user system total real index 0.000075 0.000000 0.000075 ( 0.000074) rindex 0.000006 0.000000 0.000006 ( 0.000006) ``` ``` ruby-master | substring at the start | osx UTF-8 user system total real index 0.000008 0.000001 0.000009 ( 0.000007) rindex 0.002444 0.000008 0.002452 ( 0.002452) ------------------------------------------------------ ruby-patched | substring at the start | osx UTF-8 user system total real index 0.000015 0.000000 0.000015 ( 0.000015) rindex 0.000280 0.000001 0.000281 ( 0.000280) ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:118180] [Ruby master Bug#20525] Percent string literal with indentation support
by bradgessler (Brad Gessler) 03 Sep '24

03 Sep '24

Issue #20525 has been reported by bradgessler (Brad Gessler). ---------------------------------------- Bug #20525: Percent string literal with indentation support https://bugs.ruby-lang.org/issues/20525 * Author: bradgessler (Brad Gessler) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I have code that looks like this in an application: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown <<~MARKDOWN * Because its fun * Because its super-de-dooper MARKDOWN }, ``` The "squiggle" HEREDOCs strips the indentation, but the name of the HEREDOC doesn't look that great since "Markdown" appears twice. What I'd prefer is a string literal that deals with indentation, maybe it looks something like this: ``` ContentSlide(title: "Why Phlex?"){ Markdown %~{ # Why do you like markdown? * Because its fun * Because its super-de-dooper } }, ``` If Ruby doesn't want to go down the path of adding another type of literal, I'd propose adding a method to String that does the same thing as the `~` HEREDOC so something like this is possible: ```ruby ContentSlide(title: "Why Phlex?"){ Markdown %{ # Why do you like markdown? * Because its fun * Because its super-de-dooper }.deindent }, ``` -- https://bugs.ruby-lang.org/

5 7

[ruby-core:118892] [Ruby master Misc#20685] Ruby 3.2.4 tag mentions unrelated changes (CVE-2024-27280)
by kenhys (Kentaro Hayashi) 03 Sep '24

03 Sep '24

Issue #20685 has been reported by kenhys (Kentaro Hayashi). ---------------------------------------- Misc #20685: Ruby 3.2.4 tag mentions unrelated changes (CVE-2024-27280) https://bugs.ruby-lang.org/issues/20685 * Author: kenhys (Kentaro Hayashi) * Status: Open ---------------------------------------- # Problem According to https://github.com/ruby/ruby/releases/tag/v3_2_4, it mentions "CVE-2024-27280: Buffer overread vulnerability in StringIO" as a security fix, but https://www.ruby-lang.org/en/news/2024/03/21/buffer-overread-cve-2024-27280/ explicitly describe that the following: > This vulnerability is not affected StringIO 3.0.3 and later, and Ruby 3.2.x and later. so, it is a bit strange that CVE-2023-27280 was mentioned as security fix for 3.2.x, IMHO. Please correct me if I'm wrongly interpreted it. # Expected The problematic description was removed from tags and release note. # Additional Information * https://github.com/ruby/ruby/releases/tag/v3_2_4 * mention it as security fix * https://www.ruby-lang.org/ja/news/2024/04/23/ruby-3-2-4-released/ * mention it as security fix * https://www.ruby-lang.org/en/news/2024/04/23/ruby-3-2-4-released/ * mention it as security fix -- https://bugs.ruby-lang.org/

3 2