June 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:113839] [Ruby master Bug#18572] Performance regression when invoking refined methods
by shugo (Shugo Maeda) 08 Jun '23

08 Jun '23

Issue #18572 has been updated by shugo (Shugo Maeda). Eregon (Benoit Daloze) wrote in #note-9: > From https://bugs.ruby-lang.org/issues/14083#note-3 it seems part of the problem at least is CRuby currently implements what I would call invalid usages of refinements (different set of activates refinements over time for a given call site), and that basically means extra checks and overhead, e.g., for a method which was refined once, even if there are no refinements active in the current scope. If it's needed to fix the performance issue, it may be acceptable to raise exceptions in some cases (but I don't know the exact condition of such cases). @ko1, @matz What do you think? It's simple to prohibit using calls in blocks, but it will break backward compatibility, e.g., using in module_eval. ---------------------------------------- Bug #18572: Performance regression when invoking refined methods https://bugs.ruby-lang.org/issues/18572#change-103487 * Author: palkan (Vladimir Dementyev) * Status: Assigned * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Since Ruby 3.0, defining a refinement for a method slows down its execution even if we do not activate the refinement: ```ruby require "benchmark_driver" source = <<~RUBY class Hash def symbolize_keys transform_keys { |key| key.to_sym rescue key } end def refined_symbolize_keys transform_keys { |key| key.to_sym rescue key } end end module HashRefinements refine Hash do def refined_symbolize_keys raise "never called" end end end HASH = {foo: 1, bar: 2, baz: 3} class Foo def original end def refined end end module FooRefinements refine Foo do def refined raise "never called" end end end FOO = Foo.new RUBY Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "#symbolize_keys original", %{ HASH.symbolize_keys } x.report "#symbolize_keys refined", %{ HASH.refined_symbolize_keys } end Benchmark.driver do |x| x.prelude %Q{ #{source} } x.report "no-op original", %{ FOO.original } x.report "no-op refined", %{ FOO.refined } end ``` The results for Ruby 3.1: ```sh ... Comparison: #symbolize_keys original: 2372420.1 i/s #symbolize_keys refined: 1941019.0 i/s - 1.22x slower ... Comparison: no-op original: 51790974.2 i/s no-op refined: 14456518.9 i/s - 3.58x slower ``` For Ruby 2.6 and 2.7: ```sh Comparison: #symbolize_keys original: 2278339.7 i/s #symbolize_keys refined: 2264153.1 i/s - 1.01x slower ... Comparison: no-op refined: 64178338.5 i/s no-op original: 63357980.1 i/s - 1.01x slower ``` You can find the full code and more results in this [gist](https://gist.github.com/palkan/637dc83edd86d70b5dbf72f2a4d702e5). P.S. The problem was originally noticed by @byroot, see https://github.com/ruby-i18n/i18n/pull/573 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113829] [Ruby master Bug#19721] IO#timeout= can be called without required argument
by andrykonchin (Andrew Konchin) 08 Jun '23

08 Jun '23

Issue #19721 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #19721: IO#timeout= can be called without required argument https://bugs.ruby-lang.org/issues/19721 * Author: andrykonchin (Andrew Konchin) * Status: Open * Priority: Normal * ruby -v: 3.2.1 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ```ruby f = File.open("a.txt", "w") f.timeout=() # => nil ``` `IO#timeout=` requires an argument (or it's supposed to require it) but if it's called as a method it seems the check is skipped and missing argument is treated as `nil` value. If it's called with `#send` - then argument presence is checked: ```ruby f.send :"timeout=" # ...:in `timeout=': wrong number of arguments (given 0, expected 1) (ArgumentError) ``` -- https://bugs.ruby-lang.org/

3 3

[ruby-core:113716] [Ruby master Bug#19702] Promote racc as bundled gems
by hsbt (Hiroshi SHIBATA) 08 Jun '23

08 Jun '23

Issue #19702 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19702: Promote racc as bundled gems https://bugs.ruby-lang.org/issues/19702 * Author: hsbt (Hiroshi SHIBATA) * Status: Assigned * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- In the past, we need to bundle `racc` runtime for RD parser of RDoc. Now, We have racc compiler and runtime in ruby repository. I'm not sure we should bundle and maintain `racc` as stdlib in the future. Can we promote `racc` to bundled gems? -- https://bugs.ruby-lang.org/

2 5

[ruby-core:113789] [Ruby master Feature#19718] Extend `-0` option
by nobu (Nobuyoshi Nakada) 08 Jun '23

08 Jun '23

Issue #19718 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Feature #19718: Extend `-0` option https://bugs.ruby-lang.org/issues/19718 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal ---------------------------------------- Recently I learned that Perl's `-0` option is extended to accept a hexadecimal Unicode codepoint. However it uses `-0x`, and since `-x` is used for shebang and cd, it would cause a backward incompatibility if we will incorporate it as it is. So I propose that `-0uCODEPOINT` instead. This can be extended to comma- or colon-separated codepoint list. Also, another idea is `-0:sepaerator` which specifies the separator as-is. Considering many of modern shells provide escaped string form (e.g., `$'\uHHHH'), this may be a more modern answer, except for NUL cannot be represented. https://github.com/ruby/ruby/pull/7914 -- https://bugs.ruby-lang.org/

3 5

[ruby-core:112770] [Ruby master Feature#19520] Support for `Module.new(name)` and `Class.new(superclass, name)`.
by ioquatix (Samuel Williams) 07 Jun '23

07 Jun '23

Issue #19520 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #19520: Support for `Module.new(name)` and `Class.new(superclass, name)`. https://bugs.ruby-lang.org/issues/19520 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- See <https://bugs.ruby-lang.org/issues/19450> for previous discussion and motivation. [This proposal](https://github.com/ruby/ruby/pull/7376) introduces the `name` parameter to `Class.new` and `Module.new`: ```ruby Class.new(superclass, name) Module.new(name) ``` As a slight change, we could use keyword arguments instead. -- https://bugs.ruby-lang.org/

8 41

[ruby-core:113805] [Ruby master Bug#11704] Refinements only get "used" once in loop
by matz (Yukihiro Matsumoto) 07 Jun '23

07 Jun '23

Issue #11704 has been updated by matz (Yukihiro Matsumoto). Status changed from Assigned to Rejected The behavior is intended. Matz. ---------------------------------------- Bug #11704: Refinements only get "used" once in loop https://bugs.ruby-lang.org/issues/11704#change-103451 * Author: danielpclark (Daniel P. Clark) * Status: Rejected * Priority: Normal * Assignee: matz (Yukihiro Matsumoto) * Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN, 2.2: UNKNOWN ---------------------------------------- Same results on Ruby 2.2.2 through Ruby 2.3.0dev (2015-11-18 trunk 52625) [x86_64-linux] I wrote a benchmark for testing different ways of implementing a `uniq` method and I chose to do it using refinements. I looped over the results in the benchmark and refined the method with 6 different refinements which each worked. But the next iteration when `using` is called it doesn't re-refine with previously used refinements. Example benchmark output on first iteration: ~~~ Array.new(200) !self.dup.uniq! 1.770000 0.010000 1.780000 ( 1.778248) Array.new(200) == uniq.length 1.860000 0.000000 1.860000 ( 1.866862) Array.new(200) == uniq.sort 2.580000 0.010000 2.590000 ( 2.584515) Array.new(200) each index 41.450000 0.080000 41.530000 ( 41.626149) Array.new(200) combination(2) 23.460000 0.060000 23.520000 ( 23.568865) Array.new(200) == self.uniq 1.900000 0.010000 1.910000 ( 1.909466) ~~~ After that the same methods did not get refined. ~~~ Array.new(210) !self.dup.uniq! 1.990000 0.000000 1.990000 ( 2.004269) Array.new(210) == uniq.length 2.030000 0.010000 2.040000 ( 2.032602) Array.new(210) == uniq.sort 1.990000 0.000000 1.990000 ( 1.999509) Array.new(210) each index 1.990000 0.010000 2.000000 ( 2.000181) Array.new(210) combination(2) 2.000000 0.000000 2.000000 ( 2.010159) Array.new(210) == self.uniq 2.000000 0.010000 2.010000 ( 2.009117) Array.new(220) !self.dup.uniq! 2.100000 0.010000 2.110000 ( 2.113701) Array.new(220) == uniq.length 2.070000 0.000000 2.070000 ( 2.075249) Array.new(220) == uniq.sort 2.090000 0.010000 2.100000 ( 2.102771) Array.new(220) each index 2.070000 0.000000 2.070000 ( 2.077393) Array.new(220) combination(2) 2.070000 0.010000 2.080000 ( 2.079561) Array.new(220) == self.uniq 2.100000 0.010000 2.110000 ( 2.110839) Array.new(230) !self.dup.uniq! 2.210000 0.000000 2.210000 ( 2.236008) Array.new(230) == uniq.length 2.160000 0.010000 2.170000 ( 2.166484) Array.new(230) == uniq.sort 2.140000 0.000000 2.140000 ( 2.150384) Array.new(230) each index 2.130000 0.010000 2.140000 ( 2.134572) Array.new(230) combination(2) 2.120000 0.000000 2.120000 ( 2.129683) Array.new(230) == self.uniq 2.130000 0.010000 2.140000 ( 2.137515) ~~~ I found no way to inspect what code was being used as refinements currently don't allow introspection (I have read the Refinement Specs https://bugs.ruby-lang.org/projects/ruby-trunk/wiki/RefinementsSpec ). But I figure if I wanted to see what the refinement was I could write an additional method to document the current refinement in use. ~~~ruby module BooleanUniqWithEqualLength refine Array do def uniq? self.length == self.uniq.length end def which_refinement_uniq? "self.length == self.uniq.length" end end end ~~~ But introspection is not the issue I'm raising here. The issue is that you can only use the refinement once within the scope of the loop. Look at the benchmark results and see the first 6 are correct, and the following 3 times the output is lying about what refinement is being used. Here's the full benchmark code: ~~~ruby require 'securerandom' require 'benchmark' # written to allow 65,536 unique array items array_of_x = lambda {|x| SecureRandom.hex(x*2).scan(/..../) } module Refinements module BooleanUniqWithDupUniqBang refine Array do def uniq? !self.dup.uniq! end end end module BooleanUniqWithEqualLength refine Array do def uniq? self.length == self.uniq.length end end end module BooleanUniqWithEqualSort refine Array do def uniq? self.sort == self.uniq.sort end end end module BooleanUniqWithEachIndex refine Array do def uniq? self.each_with_index { |a,b| self.each_with_index { |c,d| next if b == d return false if a == c } } true end end end module BooleanUniqWithCombination refine Array do def uniq? self.combination(2).each {|a,b| return false if a == b} true end end end module BooleanUniqWithUniq refine Array do def uniq? self == self.uniq end end end end bench_reps = 10_000 bench = Benchmark.benchmark("\nTesting various ways of implementing :uniq? on Array\n(smaller numbers are better)\n\n",34) do |x| # Note doing anymore than one test per test type seems to wash the results into all things being equal. # Only the first test gives realistic numbers. [500].each do |qty| x.report("Array.new(#{qty}) !self.dup.uniq!") { using Refinements::BooleanUniqWithDupUniqBang bench_reps.times do array_of_x.(qty).uniq? end } x.report("Array.new(#{qty}) == uniq.length") { using Refinements::BooleanUniqWithEqualLength bench_reps.times do array_of_x.(qty).uniq? end } x.report("Array.new(#{qty}) == uniq.sort") { using Refinements::BooleanUniqWithEqualSort bench_reps.times do array_of_x.(qty).uniq? end } x.report("Array.new(#{qty}) each index") { using Refinements::BooleanUniqWithEachIndex bench_reps.times do array_of_x.(qty).uniq? end } x.report("Array.new(#{qty}) combination(2)") { using Refinements::BooleanUniqWithCombination bench_reps.times do array_of_x.(qty).uniq? end } x.report("Array.new(#{qty}) == self.uniq") { using Refinements::BooleanUniqWithUniq bench_reps.times do array_of_x.(qty).uniq? end } end end ~~~ -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113802] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by ioquatix (Samuel Williams) 07 Jun '23

07 Jun '23

Issue #19057 has been updated by ioquatix (Samuel Williams). Okay, here is a PR to introduce deprecations: https://github.com/ruby/ruby/pull/7916 ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-103448 * Author: ioquatix (Samuel Williams) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113587] [Ruby master Misc#19684] DevMeeting-2023-06-08
by mame (Yusuke Endoh) 07 Jun '23

07 Jun '23

Issue #19684 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19684: DevMeeting-2023-06-08 https://bugs.ruby-lang.org/issues/19684 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/06/08 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/06/05. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

6 5

[ruby-core:113786] [Ruby master Bug#18743] Enumerator#next / peek re-use each others stacktraces
by nobu (Nobuyoshi Nakada) 06 Jun '23

06 Jun '23

Issue #18743 has been updated by nobu (Nobuyoshi Nakada). Your PR makes `stop_exc` to be overwritten for each `StopIteration`. IMO, it should not change and the `cause` of all subsequent `StopIteration`s should be the same first exception. ---------------------------------------- Bug #18743: Enumerator#next / peek re-use each others stacktraces https://bugs.ruby-lang.org/issues/18743#change-103434 * Author: sos4nt (Stefan Schüßler) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- I encountered an odd behavior. If I rescue the `StopIteration` exception from `peek` and call `next` afterwards: (or vice-versa) ```ruby # enum.rb # 1 # 2 enum = [].each # 3 enum.peek rescue nil # 4 enum.next # 5 ``` it will show the stacktrace from the rescued `peek` call: ``` $ ruby enum.rb enum.rb:4:in `peek': iteration reached an end (StopIteration) from enum.rb:4:in `<main>' ``` Whereas the error should refer to `next` on line number 5. The same happens when calling `peek` after `next` or when having muliple `peek` / `next` calls: ```ruby # enum.rb # 1 # 2 enum = [].each # 3 enum.peek rescue nil # 4 enum.next rescue nil # 5 enum.peek rescue nil # 6 puts "line #{__LINE__}" # 7 enum.next # 8 ``` The stacktrace from the first (rescued) `peek` or `next` call will be shown which doesn't reflect the actual error location: ``` $ ruby enum.rb line 7 enum.rb:4:in `peek': iteration reached an end (StopIteration) from enum.rb:4:in `<main>' ``` This is very confusing when debugging code. ---Files-------------------------------- 01-Recreate-stacktrace-enumerator.patch (1.29 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:113714] [Ruby master Bug#19700] TestProcess#test_execopts_redirect_open_fifo_interrupt_print is flaky on macOS
by kjtsanaktsidis (KJ Tsanaktsidis) 06 Jun '23

06 Jun '23

Issue #19700 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #19700: TestProcess#test_execopts_redirect_open_fifo_interrupt_print is flaky on macOS https://bugs.ruby-lang.org/issues/19700 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- The test `TestProcess#test_execopts_redirect_open_fifo_interrupt_print` in `test_process.rb` is flaky on macOS for me. Sometimes, it just hangs forever. This test is testing what happens when: * You have two processes * One is blocked opening a FIFO for reading (to redirect to a child process) * The other one sends a signal to that process * And then opens the FIFO for writing (which should unblock the child process) When this test hangs forever, for me, * The child process is blocked on opening the FIFO (i.e. it's waiting for a writer) * But the parent process successfully wrote data to the FIFO already somehow (this shouldn't be possible - the parent should have been blocked opening the FIFO until the open succeeded in the child). Actually I believe this is a bug in macOS. The following program will fail with `write (parent): Broken pipe` on macOS when run in a loop, but works correctly on Linux: https://gist.github.com/KJTsanaktsidis/fc84b006cfff1bb0b55a2571df825d80 I'm going to open a PR to skip this test on macos because I believe the operating system is broken here, and as far as I can tell Ruby is doing the correct thing. -- https://bugs.ruby-lang.org/

3 4