April 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117497] [Ruby master Bug#20424] ZLib::GZipReader always double allocates strings when passed outbuf, significantly increasing memory usage
by martinemde (Martin Emde) 14 May '24

14 May '24

Issue #20424 has been reported by martinemde (Martin Emde). ---------------------------------------- Bug #20424: ZLib::GZipReader always double allocates strings when passed outbuf, significantly increasing memory usage https://bugs.ruby-lang.org/issues/20424 * Author: martinemde (Martin Emde) * Status: Open * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In trying to improve the memory performance during the install of rubygems, we previously found a bug in `eof?`. Further investigation into the memory usage during the fix for this bug found wasteful allocating of strings in readpartial and read. In ZLib, when reading with readpartial or read, a new string is always created for the bytes read from the buffer. The current approach allocates a string no matter if there is an outbuf passed. ``` # vastly simplified psuedo implementation def readpartial(len, dst=nil) if (buffer.empty?) buffer = gzipfile.readpartial(len, dst) # adds inflated bytes into dst if passed end dst = allocate_new_string(len) # make a new string for the destination dst << buffer.read(len) # read from the buffer into the destination end ``` The result is that readpartial always allocated at least double the bytes necessary. Samuel Giddins submitted, and I have tested and reviewed, [a pull request, zlib#61](https://github.com/ruby/zlib/pull/61) that resolves the issue and vastly improves the memory usage and increases the speed of GZipReader by avoiding excess memcpy and rb_str_new calls that were wasted. This PR also adds an outbuf to GZipReader#read for improvement memory management, very similar to [IO#read](https://ruby-doc.org/core-2.5.1/IO.html#method-i-read) We appreciate your attention to this performance improvement. We believe it will further improve the performance of rubygems gem installs. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:117494] [Ruby master Bug#20421] String#index and String#byteindex don't clear `$~` when offset > size (or bytesize)
by andrykonchin (Andrew Konchin) 14 May '24

14 May '24

Issue #20421 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Bug #20421: String#index and String#byteindex don't clear `$~` when offset > size (or bytesize) https://bugs.ruby-lang.org/issues/20421 * Author: andrykonchin (Andrew Konchin) * Status: Open * ruby -v: 3.2.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When String#index and String#byteindex are called with Regexp and offset they clear `$~` (that contains result of the last Regexp matching) when offset < -size. But they don't clear `$~` when offset > size what seems not consistent: ```ruby "a".index(/a/) p $~ # => #<MatchData "a"> string = "blablabla" string.index(/bla/, string.length + 1) p $~ # => #<MatchData "a"> ``` -- https://bugs.ruby-lang.org/

5 6

[ruby-core:117588] [Ruby master Misc#20436] DevMeeting at RubyKaigi 2024
by ko1 (Koichi Sasada) 13 May '24

13 May '24

Issue #20436 has been reported by ko1 (Koichi Sasada). ---------------------------------------- Misc #20436: DevMeeting at RubyKaigi 2024 https://bugs.ruby-lang.org/issues/20436 * Author: ko1 (Koichi Sasada) * Status: Open ---------------------------------------- RubyKaigi 2024 will be at Okinawa, Japan, May 15th - 17th. We would like to try to hold a face-to-face dev meeting at Okinawa. (@matz will also participate!) * Date: 2023/05/14 (Tue.) 14:00-18:00 (The day before RubyKaigi) * Location: SAKURA innobase Okinawa, Haseko Naha Building 1F, 1-2-13 Matsuyama, Naha City, Okinawa * Near by RubyKaigi venue (not a RubyKaigi venue). * Japanese page: https://www.sakura.ad.jp/innobase/ ### Program * 1:45pm door open * 2:00pm opening and self introduction * 2:30pm discuss topics * 5:00pm closing and free time (closing time is depends on topics) * 5:50pm door close There are several RubyKaigi related events after that so let's continue on the party: * [RubyKaigi 2024 前夜祭 Asakusa.rb Welcome Drinkup - connpass](https://andpad.connpass.com/event/313590/) ### How to participate Open to any RubyKaigi attendees who have a commit bit or who have a topic they particularly want to discuss. Please write your name on this spreadsheet to count the number of participants. https://docs.google.com/spreadsheets/d/1qXuvXvigP80CFNwseJDKm2YNkpQmzSg9WSI… ### Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/05/10. I'll ask Matz which should be discussed on this meeting and reorder them. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

10 13

[ruby-core:116940] [Ruby master Feature#20300] Hash: set value and get pre-existing value in one call
by AMomchilov (Alexander Momchilov) 11 May '24

11 May '24

Issue #20300 has been reported by AMomchilov (Alexander Momchilov). ---------------------------------------- Feature #20300: Hash: set value and get pre-existing value in one call https://bugs.ruby-lang.org/issues/20300 * Author: AMomchilov (Alexander Momchilov) * Status: Open ---------------------------------------- When using a Hash, sometimes you want to set a new value, *and* see what was already there. Today, you **have** to do this in two steps: ```ruby h = { k: "old value" } # 1. Do a look-up for `:k`. old_value = h[:k] # 2. Do another look-up for `:k`, even though we just did that! h[:k] = "new value" use(old_value) ``` This requires two separate `Hash` look-ups for `:k`. This is fine for symbols, but is expensive if computing `#hash` or `#eql?` is expensive for the key. It's impossible to work around this today from pure Ruby code. I propose adding `Hash#update_value`, which has semantics are similar to this Ruby snippet: ```ruby class Hash def update_value(key, new_value) old_value = self[key] self[key] = new_value old_value end end ``` ... except it'll be implemented in C, with modifications to `tbl_update` that achieves this with a hash-lookup. Here's a PR with a PoC implementation: https://github.com/ruby/ruby/pull/10092 -- https://bugs.ruby-lang.org/

8 18

[ruby-core:117680] [Ruby master Bug#20451] Bad Ruby 3.1.5 backport causes fiddle to fail to build
by Bo98 (Bo Anderson) 01 May '24

01 May '24

Issue #20451 has been reported by Bo98 (Bo Anderson). ---------------------------------------- Bug #20451: Bad Ruby 3.1.5 backport causes fiddle to fail to build https://bugs.ruby-lang.org/issues/20451 * Author: Bo98 (Bo Anderson) * Status: Open * ruby -v: 3.1.5p252 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Ruby 3.1.5 shipped with the following backport: https://github.com/ruby/ruby/commit/84f2aabd272a54e79979795d2d405090704a1d07 However this backport directly breaks the build: ``` closure.c:279:60: error: use of undeclared identifier 'data' result = ffi_prep_closure(pcl, cif, callback, (void *)(data->self)); ^ ``` The original commit (https://github.com/ruby/fiddle/commit/2530496602) was updating the second branch to match the change in the first branch a couple lines up. However that change in the other branch does not exist in Ruby 3.1. The commit in question requires a previous commit of https://github.com/ruby/fiddle/commit/81a8a56239973ab7559229830a449d201955b…. The backport should either be reverted or an other commit should also be backported. Note that these commits were in a series of many commits made to fix an upstream issue https://github.com/ruby/fiddle/issues/102 so I cannot vouch whether or not the two commits are sufficient to fix the originally reported issue. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:117714] [Ruby master Bug#20457] Final `return` is eliminated from the AST
by tenderlovemaking (Aaron Patterson) 30 Apr '24

30 Apr '24

Issue #20457 has been reported by tenderlovemaking (Aaron Patterson). ---------------------------------------- Bug #20457: Final `return` is eliminated from the AST https://bugs.ruby-lang.org/issues/20457 * Author: tenderlovemaking (Aaron Patterson) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Given the following code: ```ruby def foo a = 1 return a end ``` If you parse this with RubyVM::AbstractSyntaxTree, the AST will be missing the `return` node. Of course the `return` node isn't necessary for compilation, but would be required for building an LSP for example. Here's a full program to demonstrate: ```ruby ast = RubyVM::AbstractSyntaxTree.parse DATA.read pp ast # Output is like this: # # (SCOPE@1:0-4:3 # tbl: [] # args: nil # body: # (DEFN@1:0-4:3 # mid: :foo # body: # (SCOPE@1:0-4:3 # tbl: [:a] # args: (ARGS@1:7-1:7 pre_num: 0 pre_init: nil opt: nil first_post: nil post_num: 0 post_init: nil rest: nil kw: nil kwrest: nil block: nil) # body: (BLOCK@2:2-3:10 (LASGN@2:2-2:7 :a (INTEGER@2:6-2:7 1)) (LVAR@3:9-3:10 :a))))) __END__ def foo a = 1 return a end ``` Btw, I'm happy to write failing tests for this type of stuff I'm just not sure where to put it! :) -- https://bugs.ruby-lang.org/

3 3

[ruby-core:117706] [Ruby master Bug#20454] IRB echoes excessive input in dumb terminal
by zonuexe (Kenta USAMI) 30 Apr '24

30 Apr '24

Issue #20454 has been reported by zonuexe (Kenta USAMI). ---------------------------------------- Bug #20454: IRB echoes excessive input in dumb terminal https://bugs.ruby-lang.org/issues/20454 * Author: zonuexe (Kenta USAMI) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [arm64-darwin23] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When IRB is started on a terminal with the environment variable TERM=dumb, excessive output is generated as shown below. A simple terminal such as Emacs shell-mode or comint-mode is assumed, but it should be possible to reproduce the following with a rich terminal such as iTerm. Type 1[RET], 12[RET], 123[RET] on your keyboard in this order. % TERM=dumb irb irb(main):001> 1 irb(main):001> 1=> 1 irb(main):002> 12 irb(main):002> 1irb(main):002> 12=> 12 irb(main):003> 123 irb(main):003> 1irb(main):003> 12irb(main):003> 123=> 123 As arton-san says, you can avoid the problem by not using ReadlineInputMethod. https://twitter.com/arton/status/1783008921000804630 -- https://bugs.ruby-lang.org/

2 1

[ruby-core:117733] [Ruby master Bug#20461] Unreadable pipe included in the readable IO of IO.select
by yamam (Masanari Yamamoto) 30 Apr '24

30 Apr '24

Issue #20461 has been reported by yamam (Masanari Yamamoto). ---------------------------------------- Bug #20461: Unreadable pipe included in the readable IO of IO.select https://bugs.ruby-lang.org/issues/20461 * Author: yamam (Masanari Yamamoto) * Status: Open * ruby -v: ruby 3.4.0dev (2024-04-27T12:55:28Z master c844968b72) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When executing the following script, pipe_r is not supposed to be readable because no writing is done to pipe_w, but pipe_r is included in the return value rs of IO.select. Since it is not possible to read from pipe_r, the IO::EAGAINWaitReadable exception is raised. ```ruby pipe_r, pipe_w = IO.pipe 1000.times do |i| File.popen(['seq', '1', '1000']) do |popen_r| j = 0 while true rs, ws, = IO.select([popen_r, pipe_r]) if rs.include?(popen_r) unless popen_r.gets break end end if rs.include?(pipe_r) puts "IO.select BUG pipe_r is not readable! i = #{i} j = #{j}" p pipe_r.read_nonblock(1) end j += 1 end end end ``` ``` $ ruby select.rb IO.select BUG pipe_r is not readable! i = 0 j = 20 <internal:io>:63:in `read_nonblock': Resource temporarily unavailable - read would block (IO::EAGAINWaitReadable) from select.rb:14:in `block (2 levels) in <main>' from select.rb:3:in `popen' from select.rb:3:in `block in <main>' from select.rb:2:in `times' from select.rb:2:in `<main>' [1] 73732 exit 1 ruby select.rb ``` -- https://bugs.ruby-lang.org/

1 1

[ruby-core:117738] [Ruby master Feature#15438] Threads can't switch faster than TIME_QUANTUM_(NSEC|USEC|MSEC)
by jhawthorn (John Hawthorn) 29 Apr '24

29 Apr '24

Issue #15438 has been updated by jhawthorn (John Hawthorn). Status changed from Open to Closed I think this is something we should improve more (I would like even faster switching times), but it does seem possible as of Ruby 3.3 to have threads switch faster than 100ms. ``` def test_switching(priority = 0) done = false started = false busy = Thread.new do Thread.current.priority = priority until done started = true end end Thread.pass until started times = [] while times.length < 10 before = Process.clock_gettime(Process::CLOCK_MONOTONIC) Thread.pass after = Process.clock_gettime(Process::CLOCK_MONOTONIC) times << (after - before) end done = true busy.join times end puts RUBY_VERSION (-3).upto(3) do |priority| print "Priority: #{priority}" times = test_switching(priority) times = times.sort[1..-2] # drop fastest and slowest average = (times.sum / times.length) puts " average: #{average}" end ``` ``` $ ruby test_measure_switching_times.rb 3.2.2 Priority: -3 average: 0.10010529025021242 Priority: -2 average: 0.10010091188087245 Priority: -1 average: 0.10010025675364886 Priority: 0 average: 0.10010364262052462 Priority: 1 average: 0.20017941037440323 Priority: 2 average: 0.4003785701279412 Priority: 3 average: 0.8007820281236491 ``` ``` $ ruby test_measure_switching_times.rb 3.3.1 Priority: -3 average: 0.02040818374371156 Priority: -2 average: 0.03023905074587674 Priority: -1 average: 0.05040335562443943 Priority: 0 average: 0.10088755612378009 Priority: 1 average: 0.20198591962616774 Priority: 2 average: 0.4033456226279668 Priority: 3 average: 0.8073137137507729 ``` ---------------------------------------- Feature #15438: Threads can't switch faster than TIME_QUANTUM_(NSEC|USEC|MSEC) https://bugs.ruby-lang.org/issues/15438#change-108145 * Author: sylvain.joyeux (Sylvain Joyeux) * Status: Closed ---------------------------------------- Thread#priority can be set to negative values, which when looking at the code is meant to reduce the time allocated to the thread. However, as far as I could understand in the codebase, the quantum of time is definitely hard-coded to 100ms (TIME_QUANTUM_...). This means that the "lower allocated time" would only work for threads that would often yield one way or the other (sleep, blocking calls, ...) My projects would definitely benefit from a faster switching period. I was wondering how best to implement this ability ? I thought of the following: 1. globally using an environment variable 2. globally using an API 3. trying to adapt dynamically, using the highest needed period 4. lowering the period when a priority lower than 0 is set, leaving it at the lower period. Obviously (3) would seem to be the best, but I'm not sure I would be able to get it right in a decent amount of time. (4) seem to be a good trade-off between simplicity and performance (nothing changes if you never use priorities lower than 0, and if you were you basically get what you wanted). What do you think ? ---Files-------------------------------- 0001-dynamically-modify-the-timer-thread-period-to-accoun.patch (3.12 KB) 0001-2.6-fix-handling-of-negative-priorities.patch (8.43 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:117735] [Ruby master Feature#18583] Pattern-matching: API for custom unpacking strategies?
by ntl (Nathan Ladd) 28 Apr '24

28 Apr '24

Issue #18583 has been updated by ntl (Nathan Ladd). Could the match operator, `=~`, could be used as a general complement to `===`? Example (following Victor's original sketch): ``` ruby class Matcher def initialize(regexp) @regexp = regexp end def ===(obj) @regexp.match?(obj) end def =~(obj) match_data = @regexp.match(obj) match_data end end case "some string" in ^(Matcher.new(/(?<some_named_capture>some) string/) => match_data some_named_capture = match_data[:some_named_capture] puts "Match: #{some_named_capture}" end ``` This would add `=~` to the pattern matching protocol that's currently comprised of `===`, `deconstruct` and `deconstruct_keys`. It would make `===` significantly more useful, and regular expressions provide a great example of why: when matching a string to a regular expression pattern, the string is already in lexical scope, but the match data is novel and only comes into existence upon a successful match: ``` subject = "some string" case subject in ^(Matcher.new(/(?<some_named_capture>some) string/) => match_data # Capturing the match data variable instead of the original string doesn't make the original string inaccessible: puts "Match subject: #{subject.inspect}" end ``` I also suspect this could be embedded into the pattern syntax itself, and could allow for some interesting possibilities. One example that leaps to mind is reifying primitive data parsed from JSON into a data structure: ``` ruby SomeStruct = Struct.new(:some_attr, :some_other_attr) do def self.===(data) data.is_a?(Hash) && data.key?(:some_attr) && data.key?(:some_other_attr) end def self.=~(data) new(**data) end end # Parse JSON into raw (primitive) data some_data = JSON.parse(<<JSON) { "some_attr": "some value", "some_other_attr": "some other value" } JSON # Reify data structure from raw data case some_data in SomeStruct => some_struct puts some_struct.inspect end ``` ---------------------------------------- Feature #18583: Pattern-matching: API for custom unpacking strategies? https://bugs.ruby-lang.org/issues/18583#change-108142 * Author: zverok (Victor Shepelev) * Status: Open ---------------------------------------- I started to think about it when discussing https://github.com/ruby/strscan/pull/30. The thing is, usage of StringScanner for many complicated parsers invokes some kind of branching. In pseudocode, the "ideal API" would allow to write something like this: ```ruby case <what next matches> in /regexp1/ => value_that_matched # use value_that_matched in /regexp2/ => value_that_matched # use value_that_matched # ... ``` This seems "intuitively" that there *should* be some way of implementing it, but we fall short. We can do some StringScanner-specific matcher object which defines its own `#===` and use it with pinning: ```ruby case scanner in ^(Matcher.new(/regexp1/)) => value_that_matched # ... ``` But there is no API to tell how the match result will be unpacked, just the whole `StringScanner` will be put into `value_that_matched`. So, I thought that maybe it would be possible to define some kind of API for pattern-like objects, the method with signature like `try_match_pattern(value)`, which by default is implemented like `return value if self === value`, but can be redefined to return something different, like part of the object, or object transformed somehow. This will open some interesting (if maybe uncanny) possibilities: not just slicing out the necessary part, but something like ```ruby value => ^(type_caster(Integer)) => int_value ``` So... Just a discussion topic! -- https://bugs.ruby-lang.org/

1 0