July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114274] [Ruby master Feature#19783] Weak References in the GC
by peterzhu2118 (Peter Zhu) 23 Aug '23

23 Aug '23

Issue #19783 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19783: Weak References in the GC https://bugs.ruby-lang.org/issues/19783 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/8113 I'm proposing support for weak references in the Ruby garbage collector. This feature adds a new function called `void rb_gc_mark_weak(VALUE *ptr)` which marks `*ptr` as weak, meaning that if no other object strongly marks `*ptr` (using `rb_gc_mark` or `rb_gc_mark_movable`), then it will be overwritten with `*ptr = Qundef`. Weak references are implemented using a buffer in `objspace` that stores all the `ptr` in the latest marking phase. After marking has finished, we iterate over the buffer and check if the `*ptr` is a dead object. If it is, then we set `*ptr = Qundef`. Weak references are implemented on the callable method entry (CME) of callcaches, which fixes issue #19436. Weak references are also implemented on `ObjectSpace::WeakMap` and `ObjectSpace::WeakKeyMap`, which have: - Significantly simpler implementations because we no longer need to have multiple tables and do not need to define finalizers on the objects. - Support for compaction because finalizers pin objects and we no longer need to define finalizers on the objects. - Much faster performance (see [benchmarks](#microbenchmarks)). ## Benchmark results ### YJIT-bench We see largely no change in performance or memory usage after this feature. ``` -------------- --------- ---------- --------- ----------- ---------- --------- -------------- ----------- bench base (ms) stddev (%) RSS (MiB) branch (ms) stddev (%) RSS (MiB) branch 1st itr base/branch activerecord 72.3 2.2 51.9 72.9 2.2 51.9 0.99 0.99 chunky-png 889.2 0.3 43.9 874.5 0.3 42.5 1.02 1.02 erubi-rails 21.2 13.5 90.7 21.0 13.3 90.9 1.01 1.01 hexapdf 2557.0 0.8 157.1 2559.2 0.7 197.1 1.01 1.00 liquid-c 65.2 0.4 34.5 65.4 0.4 34.5 0.99 1.00 liquid-compile 62.5 0.4 30.9 62.2 0.4 31.0 1.00 1.01 liquid-render 164.6 0.4 33.1 162.6 0.3 33.1 1.01 1.01 mail 133.3 0.1 46.4 134.4 0.2 46.4 1.03 0.99 psych-load 2066.6 0.2 31.6 2083.6 0.1 31.6 0.99 0.99 railsbench 2027.0 0.5 88.8 2019.4 0.5 89.0 1.01 1.00 ruby-lsp 65.6 3.0 90.1 65.4 3.1 88.5 1.00 1.00 sequel 73.1 1.1 36.6 73.1 1.1 36.6 1.00 1.00 -------------- --------- ---------- --------- ----------- ---------- --------- -------------- ----------- ``` ### Microbenchmarks We can see signficantly improved performance in `ObjectSpace::WeakMap`, with `ObjectSpace::WeakMap#[]=` being nearly 3x faster. Base: ``` ObjectSpace::WeakMap#[]= 1.037M (± 0.5%) i/s - 5.262M in 5.072833s ObjectSpace::WeakMap#[] 12.367M (± 0.9%) i/s - 62.479M in 5.052365s ``` Branch: ``` ObjectSpace::WeakMap#[]= 3.054M (± 0.3%) i/s - 15.448M in 5.058783s ObjectSpace::WeakMap#[] 15.796M (± 4.8%) i/s - 79.245M in 5.028583s ``` Code: ```ruby require "bundler/inline" gemfile do source "https://rubygems.org" gem "benchmark-ips" end wmap = ObjectSpace::WeakMap.new key = Object.new val = Object.new wmap[key] = val Benchmark.ips do |x| x.report("ObjectSpace::WeakMap#[]=") do |times| i = 0 while i < times wmap[Object.new] = Object.new i += 1 end end x.report("ObjectSpace::WeakMap#[]") do |times| i = 0 while i < times wmap[key] wmap[val] # does not exist i += 1 end end end ``` -- https://bugs.ruby-lang.org/

4 5

[ruby-core:114168] [Ruby master Misc#19766] DevMeeting-2023-08-24
by mame (Yusuke Endoh) 22 Aug '23

22 Aug '23

Issue #19766 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19766: DevMeeting-2023-08-24 https://bugs.ruby-lang.org/issues/19766 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/08/24 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/08/21. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

8 8

[ruby-core:114113] [Ruby master Bug#19761] AddressSanitizer fails with Thread and Process
by nobu (Nobuyoshi Nakada) 20 Aug '23

20 Aug '23

Issue #19761 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19761: AddressSanitizer fails with Thread and Process https://bugs.ruby-lang.org/issues/19761 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- With AddressSanitizer on macOS, this code crashes. ```ruby system('') Thread.new {} 300_000.times do String.new(capacity: 1000) end ``` ``` ==62870==ERROR: AddressSanitizer failed to deallocate 0x8000 (32768) bytes at address 0x62d00000a400 AddressSanitizer: CHECK failed: sanitizer_posix.cpp:61 "(("unable to unmap" && 0)) != (0)" (0x0, 0x0) (tid=857549) <empty stack> Assertion Failed: ../debug/vm_core.h:1853:rb_current_execution_context:!expect_ec || ec != NULL ruby 3.3.0dev (2023-07-08T02:54:37Z tmp 0083edc3a0) [x86_64-darwin22] -- Crash Report log information -------------------------------------------- See Crash Report log file in one of the following locations: * ~/Library/Logs/DiagnosticReports * /Library/Logs/DiagnosticReports for more details. Don't forget to include the above Crash Report log file in bug reports. -- C level backtrace information ------------------------------------------- ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:114218] [Ruby master Bug#19770] TLS / Certificate Hostname Verification against IP fails with "address family must be specified"
by Schachi65 (Joachim Schachermayer) 17 Aug '23

17 Aug '23

Issue #19770 has been reported by Schachi65 (Joachim Schachermayer). ---------------------------------------- Bug #19770: TLS / Certificate Hostname Verification against IP fails with "address family must be specified" https://bugs.ruby-lang.org/issues/19770 * Author: Schachi65 (Joachim Schachermayer) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.4 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I dont't use Ruby directly but indirectly via Fluentd-1.16.1 (https://www.fluentd.org/) Fluentd uses Ruby 3.1.4. The problem is, that a TLS connection from a Fluentd Client to a Fluentd Server fails, if the connection is established using the IP address of the server and tls_verify_hostname is switched to on. In the servers certificate, in the SubjectAlternativeNames extention, there is the servers IP address set with the correct value. I get the error message " **address family must be specified** ". **Probable cause** : ext/openssl/lib/openssl/ **ssl.rb** line 288: Here the Methode IPAddr.new(hostname) is called with just one argument: return true if san.value == IPAddr.new( **hostname** ).hton lib/ **ipaddr.rb** line 594: Here the exception is raised with the observed error message: when Socket::AF_UNSPEC **raise** AddressFamilyError, " **address family must be specified** " The exception is raised, because the variable "family" has the value " **Socket::AF_UNSPEC** " because thats the default value of "family" if there is no additional parameter in the call to "initialize()". The documumentation of "initialize()" says in line 575/576/577: "Although the address family is determined automatically from a specified string, you can specify one explicitly by the optional second argument." But that automatic address determination is done too late. If no second parameter is given, it is not done at all because that exception is raised (assumption: the addr i always a string). **Probable solution** : If the address family is set to Socket::AF_UNSPEC with the call of "initialize()", dont't raise an exception now but do the in the comment announced automatic adress family detection here. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114323] [Ruby master Bug#19791] Debugger does not handle delete key correctly
by stuartb (Stuart Bury) 15 Aug '23

15 Aug '23

Issue #19791 has been reported by stuartb (Stuart Bury). ---------------------------------------- Bug #19791: Debugger does not handle delete key correctly https://bugs.ruby-lang.org/issues/19791 * Author: stuartb (Stuart Bury) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux-musl] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- When entering the debugger using `binding.debugger` or the `rdbg` command, pressing the delete key on the keyboard to delete characters at the REPL merely prints "^[[3~" to the console, and does not actually delete anything. This is **very** inconvenient. Can this be fixed somehow? I really like the new debugger, but my swearing like a sailor every time I hit "delete" is starting to really annoy my wife... It is worth noting that if I run the `pry` command and then, within pry, call `binding.debugger`, the delete key operates correctly. I believe the issue is related in some way to readline support, or the lack thereof – but my Ruby installation was compiled with readline support. I've tried everything I can think of, and spent some time on Google trying to solve it, but the problem remains. It happens in Alpine 3.18 and also in Debian 12. -- https://bugs.ruby-lang.org/

2 4

[ruby-core:114315] [Ruby master Bug#19789] YJIT + rustc 1.71.0 is broken with macOS
by hsbt (Hiroshi SHIBATA) 12 Aug '23

12 Aug '23

Issue #19789 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19789: YJIT + rustc 1.71.0 is broken with macOS https://bugs.ruby-lang.org/issues/19789 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * Priority: Normal * Assignee: yjit * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I got some segmentation fault on macOS. ``` 1) Failure: TestVMDump#test_darwin_invalid_call [/Users/hsbt/Documents/github.com/ruby/ruby/test/ruby/test_vm_dump.rb:11]: pid 35071 exit 1 | -e:1: [BUG] Bus Error at 0x0000000000000001 | ruby 3.3.0dev (2023-07-30T06:13:58Z master b5c74d5488) [arm64-darwin23] | | -- Crash Report log information -------------------------------------------- | See Crash Report log file in one of the following locations: | * ~/Library/Logs/DiagnosticReports | * /Library/Logs/DiagnosticReports | for more details. | Don't forget to include the above Crash Report log file in bug reports. | | -- Control frame information ----------------------------------------------- | c:0003 p:---- s:0010 e:000009 CFUNC :call | c:0002 p:0015 s:0006 e:000005 EVAL -e:1 [FINISH] | c:0001 p:0000 s:0003 E:000a30 DUMMY [FINISH] | | -- Ruby level backtrace information ---------------------------------------- | -e:1:in `<main>' | -e:1:in `call' | | -- Threading information --------------------------------------------------- | Total ractor count: 1 | Ruby thread count for this ractor: 1 | | -- Machine register context ------------------------------------------------ | x0: 0x0000000000000000 x1: 0x0000000000000000 x2: 0x0000000000000000 | x3: 0x0000000000000000 x4: 0x0000000000000000 x5: 0x0000000000000000 | x6: 0x0000000000000000 x7: 0x0000000000000000 x18: 0x0000000000000000 | x19: 0x0000000000000000 x20: 0x000000016faadf40 x21: 0x000000016faadf40 | x22: 0x000000016faadec0 x23: 0x0000600001256eb0 x24: 0x000000016faadc70 | x25: 0x0000000000000000 x26: 0x0000000000000000 x27: 0x0000000000000000 | x28: 0x0000000140e04f20 lr: 0x000000019b211050 fp: 0x000000016faadd30 | sp: 0x000000016faadd30 | | -- C level backtrace information ------------------------------------------- | 1360: Abbrev Number 707 not found .. ``` I filed it and other failures to https://gist.github.com/hsbt/ea3e52d9fb52aed58d25caf839eae5ff @katei found this issue caused by YJIT and rustc 1.71.0. It's not happend with ``` ../ruby/configure -C --disable-install-doc --with-openssl-dir=/opt/homebrew/opt/openssl(a)1.1 --with-readline-dir=/opt/homebrew/opt/readline --with-libyaml-dir=/opt/homebrew/opt/libyaml RUSTC='rustc +1.70.0' ``` But `RUSTC='rustc +1.71.0'` is failed with same error on macOS Sonoma(beta) and Ventura. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114221] [Ruby master Bug#19771] Backport 79a4484, abfac22
by krk (Kerem Kat) 08 Aug '23

08 Aug '23

Issue #19771 has been reported by krk (Kerem Kat). ---------------------------------------- Bug #19771: Backport 79a4484, abfac22 https://bugs.ruby-lang.org/issues/19771 * Author: krk (Kerem Kat) * Status: Open * Priority: Normal * ruby -v: 3.2 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- * https://github.com/ruby/ruby/commit/79a4484a072e9769b603e7b4fbdb15b1d7eccb15 * https://github.com/ruby/ruby/commit/abfac2222bd51a03ec11b64f4770179cb6d06515 Related backport PR: https://github.com/ruby/ruby/pull/8023 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:113096] [Ruby master Feature#19572] Proposal: New TracePoint event for rescued exceptions
by st0012 (Stan Lo) 02 Aug '23

02 Aug '23

Issue #19572 has been reported by st0012 (Stan Lo). ---------------------------------------- Feature #19572: Proposal: New TracePoint event for rescued exceptions https://bugs.ruby-lang.org/issues/19572 * Author: st0012 (Stan Lo) * Status: Open * Priority: Normal ---------------------------------------- **Summary** Support a new `rescue` event type in TracePoint. When the event is triggered, `TracePoint#rescued_exception` can be used to access the exception. **Reason** Currently, TracePoint supports `raise` events, which can be helpful for debugging by showing which exception occurs at which location. By adding a `rescue` event type, we can improve the developer's debugging experience by making it easier to check where an exception is rescued. Currently, the most effective way to check where an exception is rescued involves setting a breakpoint at the exception's raised location and stepping through the code to see whether the debugger stops inside a rescue block. However, this can be a tedious process, especially in large applications with deep call stacks. By using a TracePoint event for rescue, developers can easily track exceptions as they are rescued by adding a few lines of code: ``` TracePoint.trace(:rescue) do |tp| puts "Exception rescued: #{tp.rescued_exception} at #{tp.path}:#{tp.lineno}" end ``` This new TracePoint event will also improve the `ruby/debug`'s [`ExceptionTracer`](https://github.com/ruby/debug/blob/master/lib/debug/tracer.rb#L150-L166) and provide users with a better debugging experience. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:114295] [Ruby master Bug#19786] Data::define() does not work as documented
by thyresias (Thierry Lambert) 29 Jul '23

29 Jul '23

Issue #19786 has been reported by thyresias (Thierry Lambert). ---------------------------------------- Bug #19786: Data::define() does not work as documented https://bugs.ruby-lang.org/issues/19786 * Author: thyresias (Thierry Lambert) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Not sure if this is a bug or a feature. RDoc documentation for Data::define: > define(name, *symbols) → class > define(*symbols) → class > > Defines a new Data class. > If the first argument is a string, the class is stored in Data::<name> constant. ```ruby M1 = Data.define('Measure', :amount, :unit) p M1 #=> M1 p M1.members #=> [:Measure, :amount, :unit] S1 = Struct.new('Measure', :amount, :unit) p S1 #=> Struct::Measure p S1.members #=> [:amount, :unit] ``` Unlike `Struct.new`, `Data.define` does not accept a name as first argument: it converts it to a symbol, and therefore a member. There is no trace of `Data::Measure` in the example above, while we do have `Struct::Measure`. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:114298] [Ruby master Feature#19057] Hide implementation of `rb_io_t`.
by k0kubun (Takashi Kokubun) 26 Jul '23

26 Jul '23

Issue #19057 has been updated by k0kubun (Takashi Kokubun). `cool.io` is another gem broken by this feature. FYI, I filed a pull request here https://github.com/tarcieri/cool.io/pull/79. ---------------------------------------- Feature #19057: Hide implementation of `rb_io_t`. https://bugs.ruby-lang.org/issues/19057#change-103996 * Author: ioquatix (Samuel Williams) * Status: Assigned * Priority: Normal * Assignee: ioquatix (Samuel Williams) * Target version: 3.3 ---------------------------------------- In order to make improvements to the IO implementation like <https://bugs.ruby-lang.org/issues/18455>, we need to add new fields to `struct rb_io_t`. By the way, ending types in `_t` is not recommended by POSIX, so I'm also trying to rename the internal implementation to drop `_t` where possible during this conversion. Anyway, we should try to hide the implementation of `struct rb_io`. Ideally, we don't expose any of it, but the problem is backwards compatibility. So, in order to remain backwards compatibility, we should expose some fields of `struct rb_io`, the most commonly used one is `fd` and `mode`, but several others are commonly used. There are many fields which should not be exposed because they are implementation details. ## Current proposal The current proposed change <https://github.com/ruby/ruby/pull/6511> creates two structs: ```c // include/ruby/io.h #ifndef RB_IO_T struct rb_io { int fd; // ... public fields ... }; #else struct rb_io; #endif // internal/io.h #define RB_IO_T struct rb_io { int fd; // ... public fields ... // ... private fields ... }; ``` However, we are not 100% confident this is safe according to the C specification. My experience is not sufficiently wide to say this is safe in practice, but it does look okay to both myself, and @Eregon + @tenderlovemaking have both given some kind of approval. That being said, maybe it's not safe. There are two alternatives: ## Hide all details We can make public `struct rb_io` completely invisible. ```c // include/ruby/io.h #define RB_IO_HIDDEN struct rb_io; int rb_ioptr_descriptor(struct rb_io *ioptr); // accessor for previously visible state. // internal/io.h struct rb_io { // ... all fields ... }; ``` This would only be forwards compatible, and code would need to feature detect like this: ```c #ifdef RB_IO_HIDDEN #define RB_IOPTR_DESCRIPTOR rb_ioptr_descriptor #else #define RB_IOPTR_DESCRIPTOR(ioptr) rb_ioptr_descriptor(ioptr) #endif ``` ## Nested public interface Alternatively, we can nest the public fields into the private struct: ```c // include/ruby/io.h struct rb_io_public { int fd; // ... public fields ... }; // internal/io.h #define RB_IO_T struct rb_io { struct rb_io_public public; // ... private fields ... }; ``` ## Considerations I personally think the "Hide all details" implementation is the best, but it's also the lest compatible. This is also what we are ultimately aiming for, whether we decide to take an intermediate "compatibility step" is up to us. I think "Nested public interface" is messy and introduces more complexity, but it might be slightly better defined than the "Current proposal" which might create undefined behaviour. That being said, all the tests are passing. -- https://bugs.ruby-lang.org/

1 0