November 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:119738] [Ruby master Bug#20868] Method#hash changes after compaction
by peterzhu2118 (Peter Zhu) 15 Jan '25

15 Jan '25

Issue #20868 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #20868: Method#hash changes after compaction https://bugs.ruby-lang.org/issues/20868 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- Fix: https://github.com/ruby/ruby/pull/12004 The hash value of a Method may change after a compaction but must remain constant because otherwise it may not work as the key in a hash table. For example: ```ruby def a; end # Need this method here because otherwise the iseq may be on the C stack # which would get pinned and not move during compaction def get_hash method(:a).hash end puts get_hash # => 2993401401091578131 GC.verify_compaction_references(expand_heap: true, toward: :empty) puts get_hash # => -2162775864511574135 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:119470] [Ruby master Bug#20787] IO#readline does not check its arguments like IO#gets and will read more data than limit
by javanthropus (Jeremy Bopp) 15 Jan '25

15 Jan '25

Issue #20787 has been reported by javanthropus (Jeremy Bopp). ---------------------------------------- Bug #20787: IO#readline does not check its arguments like IO#gets and will read more data than limit https://bugs.ruby-lang.org/issues/20787 * Author: javanthropus (Jeremy Bopp) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- In revision d3574c117a637a4456aa3ee78e24d8df510b9355, the implementation of IO#readline was modified and consequently broke argument handling in a subtle way. It no longer checks that the encoding of the separator string is compatible with the internal encoding of the stream. Prior to version 3.3.0, the following script raises an ArgumentError when calling #readline: ```ruby require "tempfile" Tempfile.open(encoding: "utf-8:utf-32le") { |f| f.write("0123456789"); f.rewind; f.readline("\0", 1); } ``` After 3.3.0, the script will read all the data in the file, in this case 40 bytes, even though the limit argument is 1. Replacing #readline with #gets raises the ArgumentError in all versions. I'm fairly sure that the failure to check the separator string encoding leads to the incorrect handling of the limit argument. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:119621] [Ruby master Bug#20816] Potential regression in Ruby 3.3.x (compared with 3.1 and 3.2) regarding fast syscalls and multi-threading.
by adrienjarthon (Adrien Jarthon) 14 Jan '25

14 Jan '25

Issue #20816 has been reported by adrienjarthon (Adrien Jarthon). ---------------------------------------- Bug #20816: Potential regression in Ruby 3.3.x (compared with 3.1 and 3.2) regarding fast syscalls and multi-threading. https://bugs.ruby-lang.org/issues/20816 * Author: adrienjarthon (Adrien Jarthon) * Status: Open * ruby -v: 3.3.5 * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Ok I'm sorry in advance if this is not really a regression (maybe it's the cost to pay for other improvements) but it still showed as a 30% performance drop in my benchmark so I think it may be worth a look. For the full story about how I came down this rabbit hole I just wrote a blog post to avoid cluttering the issue: https://updown.io/blog/weird-results-comparing-ruby-3-1-3-2-3-3-with-jemall… (please read it first) Here I'm gonna put my last findings and the smallest reproducible example I was able to write. So basically I noticed that quick syscalls (at least with a good SSD) like `File.exists?` or `File.mtime`, when used in a multi-threaded environement (Sidekiq in my initial testing but I reduced it to a few Thread.new) are significantly slower in 3.3.x (tested with the same results on 3.3.0, 3.3.2 and 3.3.5 for the moment) in comparison with 3.1.2 or 3.2.5: about twice as slow. These syscalls are heavily used by the Rails Reloader which is how it had such a huge impact (30%) in my testing. Here is the small reproduction script I wrote: ``` ruby #!/usr/bin/env ruby GC.disable # just to make sure it doesn't skew results THREADS = (ARGV.first || 10).to_i # choose thread count from CLI N = 10_000_000 # to make the test longer or shorter puts "Ruby #{RUBY_VERSION} / YJIT: #{RubyVM::YJIT.enabled?} / #{THREADS} Threads" Array.new(THREADS).map do Thread.new do (N/THREADS).times { File.mtime(__FILE__) } end end.each(&:join) puts "#{N} calls completed" ``` The goal is just to run a lot of fast syscalls, in various number of threads, and here are the numbers on my machine: |Threads\Ruby|3.1.2 |3.3.5 | |--|--|--| |1 | 13.7s | ✅ 14.2s | |2 | 9.7s | ⚠️ 33.4s | |4 | 23.2s | ⚠️ 37.0s | |8 | 21.8s | ⚠️ 39.7s | |16 | 23.1s | ⚠️ 41.5s | (more numbers in the [blog post](https://updown.io/blog/weird-results-comparing-ruby-3-1-3-2-3-3-with-jemalloc-and-yjit)) I suspected the [new N:M thread scheduler introduced in 3.3](https://github.com/ruby/ruby/pull/8629), even though it's "disabled" by default on the main Ractor, thinking maybe these changes degraded a bit this (quite specific scenario). So I gave a quick try with the feature ENABLED, to see if it would at least change anything and it did ! (though not was I expected): |Threads\Ruby|3.1.2 |3.3.5 | 3.3.5 with RUBY_MN_THREADS=1 | |--|--|--|--| |1 | 13.7s | 14.2s | 14.5s ✅ | |2 | 9.7s | ⚠️ 33.4s | 12.3s 🎉 | |4 | 23.2s | ⚠️ 37.0s | 40.5s 🧐 | |8 | 21.8s | ⚠️ 39.7s | 37.9s 🧐 | |16 | 23.1s | ⚠️ 41.5s | 36.3s 🧐 | (I can make graphs for these numbers if it helps) Now I'm a bit stuck in my investigation, I'm not sure what change in 3.3 caused this (the new thread scheduler is a good lead IMO) or if it even is a regression. Maybe this downside is innevitable for other benefits? I am happy to contribute more time to this issue and perform more tests but I would need some guidance on what could be interesting to test, from people who understand a bit better than me how this works internally. Thanks ! (and sorry for the long read) -- https://bugs.ruby-lang.org/

4 5

[ruby-core:112005] [Ruby master Bug#19374] Issue with Ractor.make_shareable with curried procs
by luke-gru (Luke Gruber) 14 Jan '25

14 Jan '25

Issue #19374 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #19374: Issue with Ractor.make_shareable with curried procs https://bugs.ruby-lang.org/issues/19374 * Author: luke-gru (Luke Gruber) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- This works, but shouldn't: ```ruby class Worker def start(&blk) blk = blk.curry # bug in ruby allows sharing of non-shareable proc Ractor.make_shareable(blk) @ractor = Ractor.new(blk) do |b| main = b.call p "from ractor: #{main}" end end def work @ractor.take end end worker = Worker.new a = self # unshareable main object p "from main: #{a}" worker.start { a } worker.work ``` The curried proc has a reference to the original proc and it's not checked for shareability. -- https://bugs.ruby-lang.org/

3 4

[ruby-core:118284] [Ruby master Misc#20575] Syntax Moratorium til 3.4
by matz (Yukihiro Matsumoto) 14 Jan '25

14 Jan '25

Issue #20575 has been reported by matz (Yukihiro Matsumoto). ---------------------------------------- Misc #20575: Syntax Moratorium til 3.4 https://bugs.ruby-lang.org/issues/20575 * Author: matz (Yukihiro Matsumoto) * Status: Open ---------------------------------------- Hi, As some may remember from the closing keynote at RubyKaigi, we are introducing Syntax Moratorium to support fair development of parsers. During the Syntax Moratorium period, new syntactical changes will be suspended (changes not includes bug fixes, of course). The moratorium period is scheduled to end with the release of 3.4, since the respective parser developers told us they did not need a very long period of time. Matz. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:119944] [Ruby master Feature#20899] Reconsider adding `Array#find_map`
by toy (Ivan Kuchin) 12 Jan '25

12 Jan '25

Issue #20899 has been reported by toy (Ivan Kuchin). ---------------------------------------- Feature #20899: Reconsider adding `Array#find_map` https://bugs.ruby-lang.org/issues/20899 * Author: toy (Ivan Kuchin) * Status: Open ---------------------------------------- I would like to retry proposing method `Array#find_map` that was rejected in [8421](https://bugs.ruby-lang.org/issues/8421) which happened before introduction of `filter_map` in [15323](https://bugs.ruby-lang.org/issues/15323). It would make code nicer whenever there is a need to get the first truthy result of applying some code. Adapting examples from `filter_map` documentation, but if I need only the first value: ```rb (1..9).find_map {|i| i * 2 if i.even? } # => 4 {foo: 0, bar: 1, baz: 2}.find_map {|key, value| key if value.even? } # => :foo ``` Or an example of getting match group for first successful match: ```rb list = ['some 123', 'list 234', 'of 345', 'strings 456'] list.find_map{ |s| s[/\Aof (\d+)\z/, 1] } # => "345" ``` Currently I imagine either more code and/or inefficiency (extra calls and/or objects): ```rb # code called twice list.find{ |s| s[/\Aof (\d+)\z/, 1] }&.then{ |s| s[/\Aof (\d+)\z/, 1] } # => "345" # more logic result = nil list.each do |s| break if (result = s[/\Aof (\d+)\z/, 1]) end result # => "345" # or result = nil list.find do |s| result = s[/\Aof (\d+)\z/, 1] end result # => "345" # extra calls for items which come after item that we were looking for list.map{ |s| s[/\Aof (\d+)\z/, 1] }.find{ _1 } # => "345" # using lazy list.lazy.map{ |s| s[/\Aof (\d+)\z/, 1] }.find{ _1 } # => "345" # or as suggested by @alexbarret in https://bugs.ruby-lang.org/issues/8421?tab=history#note-7 list.lazy.filter_map{ |s| s[/\Aof (\d+)\z/, 1] }.first # => "345" # using tricks, as suggested by @zverok in https://bugs.ruby-lang.org/issues/8421?tab=history#note-4 list.find{ |s| result = s[/\Aof (\d+)\z/, 1] and break result } # => "345" ``` Implementation in ruby can be: ```rb Enumerable.class_eval do def find_map(&block) each do |element| block_result = block.call(element) return block_result if block_result end nil end end ``` An example from another language - scala method [`collect`](https://www.scala-lang.org/api/3.5.2/scala/collection/ArrayOps.html#collect-68e) works alike `filter_map` and [`collectFirst`](https://www.scala-lang.org/api/3.5.2/scala/collection/ArrayOps.html#collectFirst-25d) would be like `find_map`. -- https://bugs.ruby-lang.org/

5 5

[ruby-core:119272] [Ruby master Feature#20757] Make rb_tracearg_(parameters|eval_script|instruction_sequence) public C-API
by richardboehme 11 Jan '25

11 Jan '25

Issue #20757 has been reported by richardboehme (Richard Böhme). ---------------------------------------- Feature #20757: Make rb_tracearg_(parameters|eval_script|instruction_sequence) public C-API https://bugs.ruby-lang.org/issues/20757 * Author: richardboehme (Richard Böhme) * Status: Open ---------------------------------------- **Abstract** As a C-extension developer when using tracepoints I include "ruby/debug.h". This includes most of TracePoint's API but it seems like the C-equivalents for TracePoint#parameters, TracePoint#eval_script and TracePoint#instruction_sequence are missing/not being exported in the header. **Background** Most APIs like rb_tracearg_return_value are exported in "ruby/debug.h". If I understand correctly, the implementations for those methods are located in "ruby/vm_trace.c". The following methods implemented in "ruby/vm_trace.c" are missing in "ruby/debug.h": * rb_tracearg_parameters * rb_tracearg_eval_script * rb_tracearg_instruction_sequence **Proposal** I propose to add those methods to "ruby/debug.h". From my limiting understanding the change should be simple and not break backward compatibility, because we'd only need to add those function declarations to "ruby/debug.h". I'd be open to contribute this change if it was approved. **Use cases** I'm implementing a method call tracer for Ruby using the C-extension API. I wanted to get information about the parameters that the called method receives. When writing in Ruby this can be done using the TracePoint#parameters method, but I could not find the equivalent C-API. A workaround is to retrieve the method object (using the method_id) and check the method parameters. **See also** * Implementation of TracePoint#parameters in #14694 * Implementation of TracePoint#eval_script and TracePoint#instruction_sequence in #15287 -- https://bugs.ruby-lang.org/

3 4

[ruby-core:117826] [Ruby master Bug#20480] Complie failure for unable to configure NET_LUID using msys2 mingw64-gcc
by nekoyama32767 (Jinsong Yu) 10 Jan '25

10 Jan '25

Issue #20480 has been reported by nekoyama32767 (Jinsong Yu). ---------------------------------------- Bug #20480: Complie failure for unable to configure NET_LUID using msys2 mingw64-gcc https://bugs.ruby-lang.org/issues/20480 * Author: nekoyama32767 (Jinsong Yu) * Status: Open * ruby -v: ruby 3.4.0dev (2024-05-10T04:54:31Z master 2f915e729a) [x64-mingw32] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Master commit:`2f915e729ac8c66f4009f4b28a57773923d7e7d1` NET_LUID is not successfully checked in win32 configure. When try to compile in msys2 mingw64-gcc there will be a error message like follow. ``` win32/win32.c:4365:7: error: conflicting types for 'NET_LUID'; have 'struct <anonymous>' 4365 | } NET_LUID; | ^~~~~~~~ In file included from D:/msys64/mingw64/include/iptypes.h:17, from D:/msys64/mingw64/include/iphlpapi.h:17, from ./include/ruby/win32.h:40, from ./include/ruby/internal/dosish.h:38, from ./include/ruby/defines.h:78, from ./include/ruby/ruby.h:25, from win32/win32.c:24: D:/msys64/mingw64/include/ifdef.h:106:3: note: previous declaration of 'NET_LUID' with type 'NET_LUID' 106 | } NET_LUID, *PNET_LUID; | ^~~~~~~~ ``` Mingw64-gcc has NET_LUID, when comment out line 4357-line 4367 in `win32/win32.c` in follow, compile will be successed. ``` #ifndef HAVE_TYPE_NET_LUID typedef struct { uint64_t Value; struct { uint64_t Reserved :24; uint64_t NetLuidIndex :24; uint64_t IfType :16; } Info; } NET_LUID; #endif ``` mingw64-gcc version ``` gcc --version gcc.exe (Rev4, Built by MSYS2 project) 13.2.0 Copyright (C) 2023 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ``` -- https://bugs.ruby-lang.org/

2 3

[ruby-core:119801] [Ruby master Feature#20878] A new C API to create a String by adopting a pointer: `rb_enc_str_adopt(const char *ptr, long len, long capa, rb_encoding *enc)`
by byroot (Jean Boussier) 07 Jan '25

07 Jan '25

Issue #20878 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20878: A new C API to create a String by adopting a pointer: `rb_enc_str_adopt(const char *ptr, long len, long capa, rb_encoding *enc)` https://bugs.ruby-lang.org/issues/20878 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context A common use case when writing C extensions is to generate text or bytes into a buffer, and to return it back wrapped into a Ruby String. Examples are `JSON.generate(obj) -> String`, and all other format serializers, compression libraries such as `ZLib.deflate`, etc, but also methods such as `Time.strftime`, ### Current Solution #### Work in a buffer and copy the result The most often used solution is to work with a native buffer and to manage a native allocated buffer, and once the generation is done, call `rb_str_new*` to copy the result inside memory managed by Ruby. It works, but isn't very efficient because it cause an extra copy and an extra `free()`. On `ruby/json` macro-benchmarks, this represent around 5% of the time spent in `JSON.generate`. ```c static void fbuffer_free(FBuffer *fb) { if (fb->ptr && fb->type == FBUFFER_HEAP_ALLOCATED) { ruby_xfree(fb->ptr); } } static VALUE fbuffer_to_s(FBuffer *fb) { VALUE result = rb_utf8_str_new(FBUFFER_PTR(fb), FBUFFER_LEN(fb)); fbuffer_free(fb); return result; } ``` #### Work inside RString allocated memory Another way this is currently done, is to allocate an `RString` using `rb_str_buf_new`, and write into it with various functions such as `rb_str_catf`, or writing past `RString.len` through `RSTRING_PTR` and then resize it with `rb_str_set_len`. The downside with this approach is that it contains a lot of inefficiencies, as `rb_str_set_len` will perform numerous safety checks, compute coderange, and write the string terminator on every invocation. Another major inneficiency is that this API make it hard to be in control of the buffer growth, so it can result in a lot more `realloc()` calls than manually managing the buffer. This method is used by `Kernel#sprintf`, `Time#strftime` etc, and when I attempted to improve `Time#strftime` performance, this problem showed up as the biggest bottleneck: - https://github.com/ruby/ruby/pull/11547 - https://github.com/ruby/ruby/pull/11544 - https://github.com/ruby/ruby/pull/11542 ### Proposed API I think a more effcient way to do this would be to work with a native buffer, and then build a RString that "adopt" the memory region. Technically, you can currently do this by reaching directly into `RString` members, but I don't think it's clean, and a dedicated API would be preferable: ```c /** * Similar to rb_str_new(), but it adopts the pointer instead of copying. * * @param[in] ptr A memory region of `capa` bytes length. MUST have been allocated with `ruby_xmalloc` * @param[in] len Length of the string, in bytes, not including the * terminating NUL character, not including extra capacity. * @param[in] capa The usable length of `ptr`, in bytes, including the * terminating NUL character. * @param[in] enc Encoding of `ptr`. * @exception rb_eArgError `len` is negative. * @return An instance of ::rb_cString, of `len` bytes length, `capa - 1` bytes capacity, * and of `enc` encoding. * @pre At least `capa` bytes of continuous memory region shall be * accessible via `ptr`. * @pre `ptr` MUST have been allocated with `ruby_xmalloc`. * @pre `ptr` MUST not be manually freed after `rb_enc_str_adopt` has been called. * @note `enc` can be a null pointer. It can also be seen as a routine * identical to rb_usascii_str_new() then. */ rb_enc_str_adopt(const char *ptr, long len, long capa, rb_encoding *enc); ``` An alternative to the `adopt` term, could be `move`. -- https://bugs.ruby-lang.org/

8 31

[ruby-core:117929] [Ruby master Feature#20498] Negated method calls
by MaxLap (Maxime Lapointe) 05 Jan '25

05 Jan '25

Issue #20498 has been reported by MaxLap (Maxime Lapointe). ---------------------------------------- Feature #20498: Negated method calls https://bugs.ruby-lang.org/issues/20498 * Author: MaxLap (Maxime Lapointe) * Status: Open ---------------------------------------- I want to propose the following syntax: `foo.!bar`. I know it's already valid syntax, but please read on for details. When someone write a somewhat long line of code that is negated, the main way I've seen of doing it is: ``` must_create_user = !User.where(somelong: :condition, even_more: "thing").exists? ``` I personally highly dislike it, as I must keep the "not" in the back of my mind as I read the line. When quickly reading a line like this, it's super easy to misread and understand the opposite result. The current ways around this I can think of are: * rename the variable (can be annoying) * Use `unless` (only possible when in a condition; some people, like me, have a hard time grapsping a `unless`) * use a `.!`in the end (`foo.exists?.!`), I've never seen that and it looks ugly to me (this is subjective). * create a new method name with the negated meaning (not always possible) My proposal would look like this: ``` must_create_user = User.where(somelong: :condition, even_more: "thing").!exists? ``` You cannot forget the bang that you saw 15 words ago, it's right there. It also basically reads as English: "user where ... doesn't exists". The main argument against this I can think of is that it's technically already a valid syntax. I believe it's frowned upon to override the bang operator and I'm not aware of places where it is overridden to with also having a parameter. I made a prototype in RubyNext, which you can try here: https://ruby-next.github.io/#gist:0e133bf6f27f2437193dc034d58083dc -- https://bugs.ruby-lang.org/

8 9