November 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:112301] [Ruby master Bug#19426] Endless `Range#step` of object with `#succ` method does not work
by nobu (Nobuyoshi Nakada) 13 Mar '25

13 Mar '25

Issue #19426 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19426: Endless `Range#step` of object with `#succ` method does not work https://bugs.ruby-lang.org/issues/19426 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Consider this `c` object which hash `#succ` method. ```ruby c = Struct.new(:i) do def succ; self.class.new(i+1); end def <=>(other) i <=> other.i;end end.new(0) ``` This `Range#step` works. ```ruby (c..c.succ).step(1) do |d| p d.i # 0, 1 end ``` But it fails on an endless range. ```ruby (c..).step(1) do |d| p d.i break if d.i > 3 end ``` ``` -:3:in `<=>': undefined method `i' for nil:NilClass (NoMethodError) from -:10:in `step' from -:10:in `<main>' ``` -- https://bugs.ruby-lang.org/

3 4

[ruby-core:116369] [Ruby master Feature#20202] Memoized endless method definitions
by matheusrich (Matheus Richard) 27 Feb '25

27 Feb '25

Issue #20202 has been reported by matheusrich (Matheus Richard). ---------------------------------------- Feature #20202: Memoized endless method definitions https://bugs.ruby-lang.org/issues/20202 * Author: matheusrich (Matheus Richard) * Status: Open * Priority: Normal ---------------------------------------- I propose introducing a shorthand for memoized endless method definitions: ```rb class Foo def bar ||= :memoized_value # It should behave like def bar = (@bar ||= :memoized_value) end ``` Not only is this shorter and (IMO) a natural follow-up for endless method definitions, but it's also a common pattern on Ruby codebases. It's very useful to decompose responsibilities into several objects: ```rb class User def notifications_enabled? = settings.notifications? def enable_notifications = (settings.notifications = true) def disable_notifications = (settings.notifications = false) private def settings = @settings ||= Settings.new(self) end class User::Settings attr_writer :notifications def initialize(user) @user = user @notifications = false end def notifications? = @notifications end u = User.new u.notifications_enabled? # => false u.enable_notifications u.notifications_enabled? # => true ``` In the example, the `settings` method could be rewritten as ```rb def settings ||= Settings.new(self) ``` -- https://bugs.ruby-lang.org/

4 5

[ruby-core:118682] [Ruby master Misc#20652] Memory allocation for gsub has increased from Ruby 2.7 to 3.3
by orisano (Nao Yonashiro) 24 Feb '25

24 Feb '25

Issue #20652 has been reported by orisano (Nao Yonashiro). ---------------------------------------- Misc #20652: Memory allocation for gsub has increased from Ruby 2.7 to 3.3 https://bugs.ruby-lang.org/issues/20652 * Author: orisano (Nao Yonashiro) * Status: Open ---------------------------------------- I recently upgraded from ruby 2.7.7 to 3.3.1 and noticed that the GC load increased. When I used the allocation profiler to investigate, I found that memory allocation from gsub had increased. The problem was code like this: ```ruby s = "foo " s.gsub(/ (\s+)/) { " #{' ' * Regexp.last_match(1).length}" } ``` When I compared the results of heap-profiler between 2.7.7 and 3.3.1, I found that MatchData was increasing. https://gist.github.com/orisano/98792dee260106e9b6fcb45bbabeb1e6 https://github.com/ruby/ruby/commit/abc0304cb28cb9dcc3476993bc487884c139fd11 I discovered that the cause is this commit, which stopped reusing backref to avoid race conditions. Is there a way to reuse backref while still avoiding race conditions? -- https://bugs.ruby-lang.org/

9 27

[ruby-core:119142] [Ruby master Bug#20727] YJIT segmentation fault inside of invalidate_block_version
by jhawthorn (John Hawthorn) 18 Feb '25

18 Feb '25

Issue #20727 has been reported by jhawthorn (John Hawthorn). ---------------------------------------- Bug #20727: YJIT segmentation fault inside of invalidate_block_version https://bugs.ruby-lang.org/issues/20727 * Author: jhawthorn (John Hawthorn) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- For a while we've been seeing an occasional segfault in production inside of YJIT's invalidation code. It seems to happen most often when a invalidation on a widely used CME. ``` [BUG] Segmentation fault at 0x0000000000000014" "ruby 3.3.5 (2024-09-04 revision 4f143c3038) +YJIT [x86_64-linux]" null "-- Control frame information -----------------------------------------------" "c:0174 p:---- s:1247 e:001246 CFUNC :extend_object" "c:0173 p:---- s:1244 e:001243 CFUNC :extend" "c:0172 p:0049 s:1239 e:001238 METHOD /build/vendor/gems/3.3.5/ruby/3.3.0/gems/activerecord-8.0.0.alpha.6395186/lib/active_record/relation/query_methods.rb:1467" "c:0171 p:0024 s:1233 e:001232 METHOD /build/vendor/gems/3.3.5/ruby/3.3.0/gems/activerecord-8.0.0.alpha.6395186/lib/active_record/relation/query_methods.rb:1456" "c:0170 p:0055 s:1227 e:001226 METHOD /build/vendor/gems/3.3.5/ruby/3.3.0/gems/will_paginate-4.0.1/lib/will_paginate/active_record.rb:170" "c:0169 p:0076 s:1219 e:001218 METHOD /build/vendor/gems/3.3.5/ruby/3.3.0/gems/will_paginate-4.0.1/lib/will_paginate/active_record.rb:154" -----8<----- null "-- Threading information ---------------------------------------------------" "Total ractor count: 1" "Ruby thread count for this ractor: 19" null "-- Machine register context ------------------------------------------------" " RIP: 0x0000563c0aaccef7 RBP: 0x00007f481524f5f0 RSP: 0x00007fffe8631d70" " RAX: 0x0000000000000014 RBX: 0x0000000000000010 RCX: 0x000000000000000e" " RDX: 0x0000000002ff9bab RDI: 0x00007f4839357520 RSI: 0x0000000000000008" " R8: 0x00007f480cc0f5e0 R9: 0x0000000000000800 R10: 0x00007f48af4008c0" " R11: 0x0000000000000060 R12: 0x0000563c0ad93108 R13: 0x00007f4816abe260" " R14: 0x000000000000000e R15: 0x0000000002f9d50f EFL: 0x0000000000010206" null "-- C level backtrace information -------------------------------------------" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(rb_print_backtrace+0x11) [0x563c0a9ff8df] vm_dump.c:820" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(rb_vm_bugreport) vm_dump.c:1151" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(rb_bug_for_fatal_signal+0xfc) [0x563c0abc2a1c] error.c:1065" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(sigsegv+0x4d) [0x563c0a94e02d] signal.c:926" "/lib/x86_64-linux-gnu/libpthread.so.0(__restore_rt+0x0) [0x7f48afba1420]" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(0x563c0aaccef7) [0x563c0aaccef7]" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby({closure#0}) yjit/src/core.rs:718" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(invalidate_block_version) yjit/src/core.rs:3237" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby({closure#0}+0xe6) [0x563c0aaf1586] yjit/src/invariants.rs:246" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(do_call<yjit::invariants::rb_yjit_cme_invalidate::{closure_env#0}, ()>) /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(try<(), yjit::invariants::rb_yjit_cme_invalidate::{closure_env#0}>) /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(catch_unwind<yjit::invariants::rb_yjit_cme_invalidate::{closure_env#0}, ()>+0x8) [0x563c0aae2d68] /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/s> "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(with_vm_lock<yjit::invariants::rb_yjit_cme_invalidate::{closure_env#0}, ()>) yjit/src/cruby.rs:646" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(rb_yjit_cme_invalidate+0x5c) [0x563c0aae6e0c] yjit/src/invariants.rs:243" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(vm_cme_invalidate+0x12) [0x563c0a9d78bd] vm_method.c:126" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(clear_method_cache_by_id_in_class) vm_method.c:240" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(clear_module_cache_i+0x13) [0x563c0ab2d263] class.c:1234" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(rb_id_table_foreach+0x84) [0x563c0a9891c4] id_table.c:278" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(do_include_modules_at+0x16a) [0x563c0ab31a0a] class.c:1314" "/build/vendor/ruby/4f143c30380724f14341665b622148b0646138fe/bin/ruby(include_modules_at+0x1a) [0x563c0ab31f61] class.c:1370" -----8<----- ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:120047] [Ruby master Bug#20920] When loading a file, __FILE__ gets relative paths expanded only when they start with "./"
by deivid 13 Feb '25

13 Feb '25

Issue #20920 has been reported by deivid (David Rodríguez). ---------------------------------------- Bug #20920: When loading a file, __FILE__ gets relative paths expanded only when they start with "./" https://bugs.ruby-lang.org/issues/20920 * Author: deivid (David Rodríguez) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- $ cat foo.rb puts __FILE__ $ ruby foo.rb foo.rb $ ruby ./foo.rb ./foo.rb $ ruby -e 'load "foo.rb"' foo.rb $ ruby -e 'load "./foo.rb"' /full/path/to/foo.rb ``` More than an issue, this is mainly a question. In principle, it seems more consistent to me to either expand or not expand, but this is not causing real issues for me. I just want to figure out what to do with one pending spec in Bundler (https://github.com/rubygems/rubygems/blob/cd65092deb3168759820c613ecbc54cd9…) -- https://bugs.ruby-lang.org/

5 5

[ruby-core:113033] [Ruby master Feature#19555] Allow passing default options to `Data.define`
by p8 (Petrik de Heus) 13 Feb '25

13 Feb '25

Issue #19555 has been reported by p8 (Petrik de Heus). ---------------------------------------- Feature #19555: Allow passing default options to `Data.define` https://bugs.ruby-lang.org/issues/19555 * Author: p8 (Petrik de Heus) * Status: Open * Priority: Normal ---------------------------------------- Defining a subclass of `Data` with default attributes can currently be implemented by overriding `intialize`: ```ruby class Point < Data.define(:x, :y, :z) def initialize(x:, y:, z: 0) = super end p Point.new(1, 2) #=> #<data Point x=1, y=2, z=0> ``` It would be nice if we could do it in `define` as well: ```ruby Point = Data.define(:x, :y, z: 0) ``` -- https://bugs.ruby-lang.org/

5 6

[ruby-core:116983] [Ruby master Feature#20309] Bundled gems for Ruby 3.5
by hsbt (Hiroshi SHIBATA) 06 Feb '25

06 Feb '25

Issue #20309 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Feature #20309: Bundled gems for Ruby 3.5 https://bugs.ruby-lang.org/issues/20309 * Author: hsbt (Hiroshi SHIBATA) * Status: Assigned * Assignee: hsbt (Hiroshi SHIBATA) ---------------------------------------- I propose migrate the following default gems to bundled gems at Ruby 3.5. So, It means users will get warnings if users try to load them. * ostruct * irb * reline * readline (wrapper file for readline-ext and reline) * io-console * logger * fiddle * pstore * open-uri * yaml (wrapper file for psych) * win32ole I have a plan to migrate the following default gems too. But I need to more feedback from other committers about them. * rdoc * We need to change build task like download rdoc gem before document generation. * or We make document generation is optional from Ruby 3.5 * We explicitly separate `make install` and `make install-doc` * un * `ruby -run` is one of cool feature of Ruby. Should we avoid uninstalling `un` gem? * singleton * This is famous design pattern. Should we enforce users add them to their Gemfile? * forwadable * `reline` needs to add forwardable their `runtime_dependency` after migration. * weakref * I'm not sure how impact after migrating bundled gems. * fcntl * Should we integrate these constants into ruby core? I would like to migrate `ipaddr` and `uri` too. But these are used by webrick that is mock server for our test suite. We need to rewrite `webrick` with `TCPSocker` or extract `ipaddr` and `uri` dependency from `webrick` Other default gems depend on our build process or other libraries deeply. I will update this proposal if I could extract them from default gems. -- https://bugs.ruby-lang.org/

9 27

[ruby-core:117765] [Ruby master Feature#20470] Extract Ruby's Garbage Collector
by peterzhu2118 (Peter Zhu) 29 Jan '25

29 Jan '25

Issue #20470 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #20470: Extract Ruby's Garbage Collector https://bugs.ruby-lang.org/issues/20470 * Author: peterzhu2118 (Peter Zhu) * Status: Open ---------------------------------------- # Extract Ruby's Garbage Collector ## Background As described in [[Feature #20351]](https://bugs.ruby-lang.org/issues/20351), we are working on the ability to plug alternative garbage collector implementations into Ruby. Our goal is to allow developers and researchers to create and experiment with new implementations of garbage collectors in Ruby in a simplified way. This will also allow experimentation with different GC implementations in production systems so users can choose the best GC implementation for their workloads. ## Implementation GitHub PR: [10721](https://github.com/ruby/ruby/pull/10721) In this patch, we have split the current `gc.c` file into two files: `gc.c` and `gc_impl.c`. `gc.c` now only contains code not specific to Ruby GC. This includes code to mark objects (which the GC implementation may choose not to use) and wrappers for internal APIs that the implementation may need to use (e.g. locking the VM). `gc_impl.c` now contains the implementation of Ruby's GC. This includes marking, sweeping, compaction, and statistics. Most importantly, `gc_impl.c` only uses public APIs in Ruby and a limited set of functions exposed in `gc.c`. This allows us to build `gc_impl.c` independently of Ruby and plug Ruby's GC into itself. ## Demonstration After [checking out the branch](https://github.com/ruby/ruby/pull/10721), we can first configure with `--with-shared-gc`: ```bash $ ./configure --with-shared-gc ... $ make -j ... ``` Let's now change the slot size of the GC to 64 bytes: ```bash $ sed -i 's/$#define BASE_SLOT_SIZE$.*/\1 64/' gc_impl.c ``` We can compile `gc_impl.c` independently using the following commands for clang or gcc (you may have to change the last `-I` to match your architecture and platform): ```bash $ clang -Iinclude -I. -I.ext/include/arm64-darwin23 -undefined dynamic_lookup -g -O3 -dynamiclib -o libgc.dylib gc_impl.c $ gcc -Iinclude -I. -I.ext/include/x86_64-linux -Wl,-undefined,dynamic_lookup -fPIC -g -O3 -shared -o libgc.so gc_impl.c ``` We can see that by default, the slot size is 40 bytes and objects are 40 bytes in size: ```bash $ ./ruby -e "puts GC.stat_heap(0, :slot_size)" 40 $ ./ruby -robjspace -e "puts ObjectSpace.dump(Object.new)" {"address":"0x1054a23f0", "type":"OBJECT", "shape_id":3, "slot_size":40, "class":"0x10528fd38", "embedded":true, "ivars":0, "memsize":40, "flags":{"wb_protected":true}} ``` We can now load our new GC using the `RUBY_GC_LIBRARY_PATH` environment variable (note that you may have to change the path to the DSO): ```bash $ RUBY_GC_LIBRARY_PATH=./libgc.dylib ./ruby -e "puts GC.stat_heap(0, :slot_size)" 64 $ RUBY_GC_LIBRARY_PATH=./libgc.dylib ./ruby -robjspace -e "puts ObjectSpace.dump(Object.new)" {"address":"0x1038de440", "type":"OBJECT", "shape_id":3, "slot_size":64, "class":"0x10355fc00", "embedded":true, "ivars":0, "memsize":64, "flags":{"wb_protected":true}} ``` ## Benchmark Benchmarks were ran on commit [c78cebb](https://github.com/ruby/ruby/commit/c78cebb469fe56b45ee5daad16ae97… on Ubuntu 22.04 using [yjit-bench](https://github.com/Shopify/yjit-bench/) on commit [cc5a76e](https://github.com/Shopify/yjit-bench/commit/cc5a76ef6240113650547…. Compiling gc_impl branch without `--with-shared-gc` (i.e. how the default Ruby is built), the benchmarks show little to no decrease in performance, with most of it being 0% to 1% slower: ``` -------------- ----------- ---------- ------------ ---------- --------------- -------------- bench master (ms) stddev (%) gc_impl (ms) stddev (%) gc_impl 1st itr master/gc_impl activerecord 73.9 0.3 74.6 0.3 1.00 0.99 chunky-png 911.1 0.2 937.2 0.2 0.97 0.97 erubi-rails 1582.4 0.1 1583.5 0.0 1.00 1.00 hexapdf 2716.2 1.1 2760.2 0.7 1.00 0.98 liquid-c 68.9 0.5 68.6 0.4 1.00 1.00 liquid-compile 67.9 0.1 68.2 0.2 0.99 1.00 liquid-render 172.8 0.1 174.9 0.1 0.99 0.99 lobsters 1033.9 0.4 1036.0 0.3 1.08 1.00 mail 135.1 0.2 136.5 0.2 0.99 0.99 psych-load 2250.8 0.1 2274.9 0.3 0.99 0.99 railsbench 2499.2 0.2 2502.9 0.1 1.00 1.00 rubocop 178.3 0.5 179.8 0.4 1.00 0.99 ruby-lsp 116.8 0.1 118.5 0.2 1.00 0.99 sequel 75.4 0.2 76.2 0.3 0.99 0.99 -------------- ----------- ---------- ------------ ---------- --------------- -------------- ``` Compiling gc_impl branch with `--with-shared-gc` and loading Ruby's current GC using `RUBY_GC_LIBRARY_PATH`, the benchmarks are still fairly good with performance decrease of only around 1% to 2%: ``` -------------- ----------- ---------- ------------ ---------- --------------- -------------- bench master (ms) stddev (%) gc_impl (ms) stddev (%) gc_impl 1st itr master/gc_impl activerecord 74.2 0.2 75.4 0.5 0.98 0.98 chunky-png 916.3 0.3 933.2 0.1 0.98 0.98 erubi-rails 1597.6 0.1 1586.3 0.2 1.01 1.01 hexapdf 2731.4 0.5 2776.8 0.7 1.00 0.98 liquid-c 68.5 0.1 68.9 0.4 0.97 0.99 liquid-compile 67.4 0.4 68.3 0.2 0.95 0.99 liquid-render 171.8 0.1 175.6 0.2 0.97 0.98 lobsters 1031.9 0.3 1041.4 0.3 0.94 0.99 mail 135.5 0.4 136.7 0.1 0.99 0.99 psych-load 2246.0 0.1 2281.3 0.1 0.99 0.98 railsbench 2490.9 0.0 2490.0 0.1 1.01 1.00 rubocop 179.8 2.3 180.0 0.4 0.94 1.00 ruby-lsp 117.3 0.1 118.5 0.1 0.99 0.99 sequel 75.8 0.5 76.3 0.2 0.99 0.99 -------------- ----------- ---------- ------------ ---------- --------------- -------------- ``` ## Limitations We recognize that our current implementation does not yet offer the flexibility required for a generic plug-in GC. Specifically, the set of APIs that the plug-in GC has to implement is relatively large, at around 70 functions. Additionally, some of these functions are specific to the current GC. We would like to emphasize that the API is NOT stable and is subject to change. We will be working on improving this API and reducing the surface area. This will be future work and we're not working on it in this phase. ## Future plans - Refactor and improve `gc_impl.c`. - Implement alternate GC implementations, such as the Epsilon GC and [MMTk](https://www.mmtk.io/) to prove that this API allows for alternate implementations of the GC. - Reduce and improve the API of the GC implementation. - Benchmark and improve performance of the DSO API. -- https://bugs.ruby-lang.org/

4 7

[ruby-core:119201] [Ruby master Bug#20743] ruby doesn't properly split RUBY_DEBUG_LOG_FILTER into items
by x-yuri (Yuri Kanivetsky) 28 Jan '25

28 Jan '25

Issue #20743 has been reported by x-yuri (Yuri Kanivetsky). ---------------------------------------- Bug #20743: ruby doesn't properly split RUBY_DEBUG_LOG_FILTER into items https://bugs.ruby-lang.org/issues/20743 * Author: x-yuri (Yuri Kanivetsky) * Status: Open * ruby -v: ruby 3.4.0dev (2024-09-15T01:06:11Z master 532af89e3b) +PRISM [x86_64-linux-musl] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ``` $ docker run --rm alpine:3.20.3 sh -euxc ' apk add git build-base autoconf ruby yaml-dev zlib-dev openssl-dev libffi-dev git clone https://github.com/ruby/ruby cd ruby git checkout 532af89e3b5b78dd3a6fe29c6cc64ad1b073afe2 ./autogen.sh ./configure cflags=-DUSE_RUBY_DEBUG_LOG --disable-install-doc make -j8 install ruby -v RUBY_DEBUG_LOG=stderr RUBY_DEBUG_LOG_FILTER=rb_thread_schedule_limits,rb_ractor_thread_switch ruby -e "" ' ... + RUBY_DEBUG_LOG=stderr RUBY_DEBUG_LOG_FILTER=rb_thread_schedule_limits,rb_ractor_thread_switch ruby -e RUBY_DEBUG_LOG=stderr [stderr] RUBY_DEBUG_LOG_FILTER[0]=rb_thread_schedule_limit (all) RUBY_DEBUG_LOG_FILTER[1]=rb_ractor_thread_switch (all) ... ``` As you can see `ruby` reports that `RUBY_DEBUG_LOG_FILTER[0]` equals `rb_thread_schedule_limit`, not `rb_thread_schedule_limits`. This happens because here: https://github.com/ruby/ruby/blob/532af89e3b5b78dd3a6fe29c6cc64ad1b073afe2/… `1` should not be subtracted. Also in the following lines `str` is increased, but `len` is not decreased. It makes sense to recheck if there are no further mistakes. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:118879] [Ruby master Bug#20682] Slave PTY output is lost after a child process exits in macOS
by ono-max (Naoto Ono) 27 Jan '25

27 Jan '25

Issue #20682 has been reported by ono-max (Naoto Ono). ---------------------------------------- Bug #20682: Slave PTY output is lost after a child process exits in macOS https://bugs.ruby-lang.org/issues/20682 * Author: ono-max (Naoto Ono) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- According to Launchable, the following PTY tests are flaky only on macOS. https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-… https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-… https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-… https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-… It's because the slave PTY output is lost after a child process exits in macOS. Here is the code to reproduce the problem. When I remove `sleep 3` from the code, "a" is returned. ``` require 'pty' r, w, pid = PTY.spawn('ruby', '-e', 'puts "a"') sleep 3 puts r.gets #=> Returns nil ``` Based on my investigation, this issue happens in the macOS side and it's almost same as https://github.com/pexpect/pexpect/issues/662. The cause is described as follows in the ticket: > // NOTE[macOS-S_CTTYREF]: On macOS, after a forkpty(), if the pty slave (child) // is closed before the pty master (parent) reads, the pty's buffer is cleared // thus the master (parent) reads nothing. This can happen if the child exits // before the parent has a chance to call master.read(). // // This issue has been reported to Apple, but has not been resolved: // https://developer.apple.com/forums/thread/663632 // // Work around this issue by opening /dev/tty then closing it. This ultimately // causes the child's exit() to flush the slave pty's output buffer in a // blocking way. This fixes the problem on macOS 13.2 in my testing. // // Here's how the workaround works in detail: // // If we open /dev/tty, it sets the S_CTTYREF flag on the process. This flag // remains set if we close the /dev/tty file descriptor. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5… // Additionally, opening /dev/tty retains a reference to the pty slave. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5… // // When the child process exits: // // 1. All open file descriptors (including stdin/stdout/stderr which are the pty // slave) are closed. This does *not* drain unread pty slave output. // * If S_CTTYREF was set, closing the file descriptors does not close the // last reference to the pty slave, so no cleanup happens yet. // * NOTE[macOS-pty-close-loss]: If S_CTTYREF was not set, closing the file // descriptors drops the last reference to the pty slave. Unread data is // dropped. // // 2. If the S_CTTYREF flag is set on the child process, the controlling // terminal (pty slave) is closed. XNU's ptsclose() ultimately calls // ttywait(). // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5… // * ttywait() is the same as ioctl(slave, TIOCDRAIN); it blocks waiting for // output to be received. // https://github.com/apple-oss-distributions/xnu/blob/aca3beaa3dfbd42498b42c5… // * NOTE[macOS-pty-waitpid-hang]: Because of the blocking ttywait(), the // process is in an exiting (but not zombie) state. waitpid() will hang. // // * NOTE[macOS-pty-close-loss]: If the S_CTTYREF flag is not set on the // child process, ttywait() is not called, thus the pty slave does not // block waiting for the output to be received, and the output is dropped. // A well-behaving parent will use a poll() loop anyway, so this isn't a // problem. (It does make quick tests annoying to write though.) // // Demonstration of NOTE[macOS-pty-close-loss] (S_CTTYREF is not set before // exit): // // // On macOS, this program should report 'data = ""', demonstrating that // // writes are lost. // // #include <stdlib.h> // #include <errno.h> // #include <stdio.h> // #include <string.h> // #include <unistd.h> // #include <util.h> // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // // Cause the child to write() then exit(). exit() will drop written // // data. // sleep(1); // // char buffer[10]; // ssize_t rc = read(tty_fd, buffer, sizeof(buffer)); // if (rc < 0) { perror("read"); abort(); } // fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // } // // return 0; // } // // Demonstration of NOTE[macOS-pty-waitpid-hang] (S_CTTYREF is set before exit): // // // On macOS, this program should hang, demonstrating that the child // // process doesn't finish exiting. // // // // During the hang, observe that the child is in an exiting state ("E"): // // // // $ ps -e -o pid,stat | grep 20125 // // 20125 ?Es // // #include <errno.h> // #include <fcntl.h> // #include <stdio.h> // #include <stdlib.h> // #include <string.h> // #include <unistd.h> // #include <util.h> // // int main() { // int tty_fd; // pid_t pid = forkpty(&tty_fd, /*name=*/NULL, /*termp=*/NULL, // /*winp=*/NULL); // if (pid == -1) { perror("forkpty"); abort(); } // // if (pid == 0) { // // Child. // close(open("/dev/tty", O_WRONLY)); // (void)write(STDOUT_FILENO, "y", 1); // exit(0); // } else { // // Parent. // // fprintf(stderr, "child PID: %d\n", pid); // // // This will hang because, despite the child being is an exiting // // state, the child is waiting for us to read(). // pid_t rc = waitpid(pid, NULL, 0); // if (rc < 0) { perror("waitpid"); abort(); } // } // // return 0; // } In Ruby, PTY is implemented with [fork()](https://github.com/ruby/ruby/blob/master/process.c#L1706) and [posix_openpt()](https://github.com/ruby/ruby/blob/master/ext/pty/pty.c#L329) in macOS. I could reproduce the problem in the following script. ``` #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <fcntl.h> #include <sys/ioctl.h> int main() { int master_fd, slave_fd; pid_t child_pid; char *slave_name; // Open a master pseudo-terminal master_fd = posix_openpt(O_RDWR | O_NOCTTY); if (master_fd == -1) { perror("posix_openpt"); exit(1); } // Grant access to the slave pseudo-terminal if (grantpt(master_fd) == -1) { perror("grantpt"); exit(1); } // Unlock the slave pseudo-terminal if (unlockpt(master_fd) == -1) { perror("unlockpt"); exit(1); } // Get the name of the slave pseudo-terminal slave_name = ptsname(master_fd); if (slave_name == NULL) { perror("ptsname"); exit(1); } // Fork a child process child_pid = fork(); if (child_pid == -1) { perror("fork"); exit(1); } else if (child_pid == 0) { // Child process // Open the slave pseudo-terminal slave_fd = open(slave_name, O_RDWR); if (slave_fd == -1) { perror("open"); exit(1); } // Create a new session and process group if (setsid() == -1) { perror("setsid"); exit(1); } // Set the controlling terminal for the child process if (ioctl(slave_fd, TIOCSCTTY, NULL) == -1) { perror("ioctl"); exit(1); } // Duplicate the slave file descriptor to stdin, stdout, and stderr if (dup2(slave_fd, STDIN_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDOUT_FILENO) == -1) { perror("dup2"); exit(1); } if (dup2(slave_fd, STDERR_FILENO) == -1) { perror("dup2"); exit(1); } // close(open("/dev/tty", O_WRONLY)); // Close the original slave file descriptor close(slave_fd); // Execute a shell or other program (void)write(STDOUT_FILENO, "y", 1); exit(1); } else { sleep(5); char buffer[10]; ssize_t rc = read(master_fd, buffer, sizeof(buffer)); if (rc < 0) { perror("read"); abort(); } fprintf(stderr, "data = \"%.*s\"\n", (int)rc, buffer); // Clean up close(master_fd); } return 0; } ``` -- https://bugs.ruby-lang.org/

3 8