July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114238] [Ruby master Feature#19775] Use the callback `main::const_added`
by UlyssesZhan (Ulysses Zhan) 19 Jul '23

19 Jul '23

Issue #19775 has been reported by UlyssesZhan (Ulysses Zhan). ---------------------------------------- Feature #19775: Use the callback `main::const_added` https://bugs.ruby-lang.org/issues/19775 * Author: UlyssesZhan (Ulysses Zhan) * Status: Open * Priority: Normal ---------------------------------------- In #17881, it was accepted that `Module#const_added` is called when there is a new constant added to the namespace of the module. Since `main` can also serve as a namespace, it should also be able to use this callback. ```ruby def const_added name p name end Foo = 1 # => :Foo ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:113517] [Ruby master Misc#19679] Migrate Wiki from bugs.ruby-lang.org to ruby/ruby GitHub repository
by jemmai (Jemma Issroff) 18 Jul '23

18 Jul '23

Issue #19679 has been reported by jemmai (Jemma Issroff). ---------------------------------------- Misc #19679: Migrate Wiki from bugs.ruby-lang.org to ruby/ruby GitHub repository https://bugs.ruby-lang.org/issues/19679 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- # Background There is currently a Wiki at https://bugs.ruby-lang.org/projects/ruby/wiki which contains documentation of processes, documentation of code, developers' meeting notes, and various other notes. There are three main problems with the current Wiki setup: * **Out of date:** Much of the documentation is out of date, or no longer relevant (for example: [How to Release Ruby 1.9](https://bugs.ruby-lang.org/projects/ruby/wiki/HowToReleaseRuby19)) * **Duplication:** Some of this documentation exists both in the Wiki and in the docs in ruby/ruby (for example: [DTrace Probes in the docs](https://docs.ruby-lang.org/en/master/dtrace_probes_rdoc.html) and [DTrace Probes in the Wiki](https://bugs.ruby-lang.org/projects/ruby/wiki/DTraceProbes) * **Editing docs is limited:** Only Core contributors can edit the Wiki, and version control is more limited (for example: I couldn't edit any mistakes I found in the Wiki while preparing this issue). # Proposal This proposal is to migrate any still relevant docs within the Wiki from the bugs.ruby-lang.org Wiki to a new GitHub Wiki. I have categorized all of the Wiki pages in [this google sheet](https://docs.google.com/spreadsheets/d/1Ld83ZKxknYgECXxNSh28fjFw82pS… and given my analysis of which should be migrated. If there are any pages that should be migrated which aren't currently included, please indicate so in the comments and we will migrate them. I also propose we migrate the old Developers' Meeting notes to the [developer's meeting repository](https://github.com/ruby/dev-meeting-log). I spoke about this idea with @hsbt at Ruby Kaigi, and he and @naruse suggested that I write this proposal. # Implementation 0. Before I begin the migration, someone who has the appropriate permissions on GitHub will need [to create the Wiki](https://docs.github.com/en/communities/documenting-your-project-with-… 1. I will migrate all pages listed in the Google sheet with "migrate?" checked to the new GitHub wiki 2. I will migrate all of the Developers' Meeting Notes to the developers' meeting repo 3. Someone who has the appropriate permissions will need to remove the Wiki from bugs.ruby-lang.org 4. I will update all GitHub wiki pages with "update?" checked in the Google Sheet to reflect the latest state -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113109] [Ruby master Bug#19577] Backport commit 537183cd2ac0163851277b46a2f21ea5914c11c0
by peterzhu2118 (Peter Zhu) 18 Jul '23

18 Jul '23

Issue #19577 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19577: Backport commit 537183cd2ac0163851277b46a2f21ea5914c11c0 https://bugs.ruby-lang.org/issues/19577 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- Please backport commit [537183cd2ac0163851277b46a2f21ea5914c11c0](https://github.com/ruby/ruby/comm…. We've seen this crash in our production systems. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:113129] [Ruby master Bug#19580] Ensure ruby_xfree won't segfault if called after ruby_vm_destruct
by mdalessio (Mike Dalessio) 18 Jul '23

18 Jul '23

Issue #19580 has been reported by mdalessio (Mike Dalessio). ---------------------------------------- Bug #19580: Ensure ruby_xfree won't segfault if called after ruby_vm_destruct https://bugs.ruby-lang.org/issues/19580 * Author: mdalessio (Mike Dalessio) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Github PR: https://github.com/ruby/ruby/pull/7663 The POSIX Threads API provides the ability to define a destructor to clean up thread-local storage at thread exit by using `pthread_key_create` to bind a storage key and an associated destructor function, and `pthread_setspecific` to associate a value with that key. If a C extension is using `ruby_xmalloc` and `ruby_xfree` to manage memory, then the destructor function will likely call `ruby_xfree`. There is a small window of time as a process is exiting -- after `ruby_vm_destruct` but before the process exits -- in which a thread may exit and the destructor function called, leading to a segfault. ### Real-world use case The real-world scenario motivating this change is libxml2's pthread code which uses `pthread_key_create` to set up a destructor that is called at thread exit to clean up thread-local storage associated with the thread by `pthread_setspecific`. Nokogiri has, since 2009, configured libxml2 to use `ruby_xmalloc` and `ruby_xfree` so that Ruby's memory management is aware of the allocated memory. As a result, `ruby_xfree` is being called by the pthread destructor, which may lead to a segfault that looks like: [BUG] Segmentation fault at 0x0000000000000420 ruby 3.3.0dev (2023-03-19T06:13:25Z master ca9355e173) [x86_64-linux] -- Machine register context ------------------------------------------------ RIP: 0x00007f9b6311124e RBP: 0x00007f9b58001530 RSP: 0x00007f9b5d8eed60 RAX: 0x0000000000000000 RBX: 0x00007f9b58001888 RCX: 0x00000000000003ff RDX: 0x00007f9b5dad62b0 RDI: 0x00007f9b58007090 RSI: 0x0000000000000000 R8: 0x00007f9b5d8eed64 R9: 0x00000000000000ca R10: 0x0000000000000000 R11: 0x0000000000000246 R12: 0x00007f9b58007090 R13: 0x00007f9b62e1bae8 R14: 0x0000000000000004 R15: 0x00007f9b5d8efb58 EFL: 0x0000000000010206 -- C level backtrace information ------------------------------------------- .../3.3.0-dev/lib/libruby.so.3.3(rb_print_backtrace+0xd) [0x7f9b632e6f1f] /home/flavorjones/code/oss/ruby/vm_dump.c:785 .../3.3.0-dev/lib/libruby.so.3.3(rb_vm_bugreport) /home/flavorjones/code/oss/ruby/vm_dump.c:1101 .../3.3.0-dev/lib/libruby.so.3.3(rb_bug_for_fatal_signal+0xf4) [0x7f9b630eead4] /home/flavorjones/code/oss/ruby/error.c:813 .../3.3.0-dev/lib/libruby.so.3.3(sigsegv+0x4d) [0x7f9b63233bbd] /home/flavorjones/code/oss/ruby/signal.c:964 /lib/x86_64-linux-gnu/libc.so.6(0x7f9b62c42520) [0x7f9b62c42520] .../3.3.0-dev/lib/libruby.so.3.3(ruby_sized_xfree+0x3) [0x7f9b6311124e] /home/flavorjones/code/oss/ruby/gc.c:12666 .../3.3.0-dev/lib/libruby.so.3.3(ruby_sized_xfree) /home/flavorjones/code/oss/ruby/gc.c:12663 .../lib/nokogiri/nokogiri.so(xmlResetError+0x0) [0x7f9b5da6ab4a] .../tmp/x86_64-linux/nokogiri/3.0.5/tmp/x86_64-pc-linux-gnu/ports/libxml2/2.10.3/libxml2-2.10.3/error.c:880 .../lib/nokogiri/nokogiri.so(xmlResetError) .../tmp/x86_64-linux/nokogiri/3.0.5/tmp/x86_64-pc-linux-gnu/ports/libxml2/2.10.3/libxml2-2.10.3/error.c:873 .../lib/nokogiri/nokogiri.so(xmlResetError) (null):0 .../lib/nokogiri/nokogiri.so(xmlFreeGlobalState+0x14) [0x7f9b5dad62c4] .../tmp/x86_64-linux/nokogiri/3.0.5/tmp/x86_64-pc-linux-gnu/ports/libxml2/2.10.3/libxml2-2.10.3/threads.c:548 /lib/x86_64-linux-gnu/libc.so.6(__nptl_deallocate_tsd+0x77) [0x7f9b62c91711] ./nptl/nptl_deallocate_tsd.c:73 /lib/x86_64-linux-gnu/libc.so.6(__nptl_deallocate_tsd) ./nptl/nptl_deallocate_tsd.c:22 /lib/x86_64-linux-gnu/libc.so.6(start_thread+0x17a) [0x7f9b62c949ca] ./nptl/pthread_create.c:453 [0x7f9b62d26a00] I've reproduced this in Ruby 3.0, 3.1, 3.2, and master by using the following ruby code: # must have an error in it to cause pthread_setspecific to be called html = "<div foo='asdf>asdf</div>" Thread.new { Nokogiri::HTML4::Document.parse(html) } sleep 3 # THREAD_CACHE_TIME exit 0 The timing is admittedly hard to reproduce, but [Gitlab has seen this in their CI pipelines a few dozen times](https://gitlab.com/gitlab-org/gitlab/-/issues/390313) and it's made easier if the process lifetime is extended by an `atexit`-registered function. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:113198] [Ruby master Bug#19593] Crash due to throw data set as cause
by peterzhu2118 (Peter Zhu) 18 Jul '23

18 Jul '23

Issue #19593 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19593: Crash due to throw data set as cause https://bugs.ruby-lang.org/issues/19593 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/7696 rb_ec_setup_exception did not check if errinfo is a throw_data. This can cause crashes in code since it is assumed that id_cause is an object. We saw a crash in show_cause due to id_cause of errinfo being a throw_data. It crashes on rb_obj_is_kind_of since it cannot be called on T_IMEMO objects. Unfortunately, we couldn't find a reproduction script, however we debugged the core dump and rb_ec_setup_exception is the only place where id_cause is assigned from errinfo without checking if it is a throw_data. ``` 0x0000556c5708e6dd in sigsegv (sig=11, info=0x7f301befa3f0, ctx=0x7f301befa2c0) at signal.c:964 0x00007f301d046420 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0 class_search_class_ancestor (c=139844586301760, cl=<optimized out>) at object.c:810 rb_obj_is_kind_of (obj=obj@entry=139839221734880, c=139844586301760) at object.c:861 0x0000556c56f2f00f in show_cause (errinfo=errinfo@entry=139838840645160, str=str@entry=139839221730520, opt=139839221730480, highlight=0, reverse=reverse@entry=0, backtrace_limit=backtrace_limit@entry=-1, shown_causes=0x7ffe9d1a2d68) at ./include/ruby/internal/special_consts.h:175 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114224] [Ruby master Bug#19773] Intermittent SIGSEGV during marking of Ripper parser
by eightbitraptor (Matthew Valentine-House) 18 Jul '23

18 Jul '23

Issue #19773 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19773: Intermittent SIGSEGV during marking of Ripper parser https://bugs.ruby-lang.org/issues/19773 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Intermittent SIGSEGV during marking of the Ripper parser. A failing build [can be seen on ci.rvm.jp (build 4643684)](http://ci.rvm.jp/results/trunk-random2@ruby-sp2-docker/4643684). Inside `ripper_s_allocate` first we use `TypedData_Make_Struct` to create a new `T_DATA` object in the Ruby heap. This allocates an empty struct (`struct ripper`). Further down the function we allocate another struct (`struct parser_params` during `rb_ruby_ripper_parser_allcoate`) and assign it to a pointer from the first struct, which is attached to the `T_DATA` object. `rb_ruby_ripper_parser_allocate` can trigger GC, as it uses `ruby_xcalloc`. But if GC is triggered at this point then the `parser_params` have not been allocated yet, and so the `struct ripper` contains a `NULL` reference. This causes a SEGV when we try and mark the `T_DATA` wrapper around the `struct ripper`. This can be replicated locally using the following test script ``` require "ripper" GC.stress = true Ripper.new ``` -- https://bugs.ruby-lang.org/

1 1

[ruby-core:114137] [Ruby master Feature#19764] Introduce defp keyword for defining overloadable, pattern matched methods
by zeke (Zeke Gabrielse) 17 Jul '23

17 Jul '23

Issue #19764 has been reported by zeke (Zeke Gabrielse). ---------------------------------------- Feature #19764: Introduce defp keyword for defining overloadable, pattern matched methods https://bugs.ruby-lang.org/issues/19764 * Author: zeke (Zeke Gabrielse) * Status: Open * Priority: Normal ---------------------------------------- Pattern matching has become one of my favorite features of Ruby, if not my favorite. It changed the way I write and express my thoughts through clean, maintainable code. And I'd like to use it *more*. I propose a new keyword, `defp`, for defining a method which applies pattern matching to its arguments. ```ruby defp call(String => s unless s in /^[a-z]/) puts "string: #{s.inspect} (capitalized)" end defp call(String => s) puts "string: #{s.inspect}" end defp call(Hash(foo:, bar:) => h) puts "hash: #{h.inspect}" end defp call(**nil) puts "no keyword args" end call("Example") # => string: "Example" (capitalized) call("test") # => string: "test" call(foo: 1, bar: 2) # => hash: { :foo => 1, :bar => 2 } ``` Internally, this could be represented as the following `case..in` pseudocode: ```ruby def call(...) case ... in String => s unless s in /foo/ puts "string: #{s.inspect} (not foo)" in String => s puts "string: #{s.inspect}" in Hash(foo:, bar:) => h puts "hash: #{h.inspect}" in **nil puts "no keyword args" else raise NoMatchingMethod end end ``` As you could imagine, this could be used to refactor a lot of code, making the developer's intent much clearer. From [complex methods that use `case` statements](https://github.com/rails/rails/blob/593893c901f87b4ed205751f72d… for taking varied arguments (I'm sure all our code bases contain such `case` statements), to defining smaller, simpler methods that handle particular argument patterns. In addition, not only can this improve code quality, but it brings in method overloads, and it also adds a way to define more typing to the language -- something that RBS has tried to do, to mixed reactions -- but in a more Ruby-like way that Rubyists are already learning *and loving*. Thoughts? Original idea by Victor Shepelev: https://zverok.space/blog/2023-05-05-ruby-types.html Further discussion: https://news.ycombinator.com/item?id=35834351 -- https://bugs.ruby-lang.org/

5 4

[ruby-core:114049] [Ruby master Bug#19750] Ruby hangs when parsing Regexp with unterminated unicode "/\u{"
by tompng (tomoya ishida) 17 Jul '23

17 Jul '23

Issue #19750 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #19750: Ruby hangs when parsing Regexp with unterminated unicode "/\u{" https://bugs.ruby-lang.org/issues/19750 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Ruby hangs up for a while and gets `failed to allocate memory (NoMemoryError)` ~~~shell $ ruby -e "/\\u{" ruby: failed to allocate memory (NoMemoryError) ~~~ ~~~ruby RubyVM::AbstractSyntaxTree.parse "/\\u{" # => SyntaxError RubyVM::AbstractSyntaxTree.parse "/\\u{\n" # => failed to allocate memory (NoMemoryError) ~~~ Found in https://github.com/ruby/irb/issues/620 -- https://bugs.ruby-lang.org/

4 4

[ruby-core:113260] [Ruby master Bug#19601] YJIT `try to mark T_NONE object` stemming from object shape transition on `self`
by alanwu (Alan Wu) 17 Jul '23

17 Jul '23

Issue #19601 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #19601: YJIT `try to mark T_NONE object` stemming from object shape transition on `self` https://bugs.ruby-lang.org/issues/19601 * Author: alanwu (Alan Wu) * Status: Closed * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) +YJIT [arm64-darwin22] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: REQUIRED ---------------------------------------- We've identified a false collection bug with YJIT. Symptoms can range from `[BUG] try to mark T_NONE object` to SEGVs. Due to the bug requiring specific transient heap state to reproduce, it may be hard to identify by looking at the crash-site stack trace. `ruby --yjit-call-threshold=1` reproducer: ```ruby class RegressionTest def initialize @a = @b = @fourth_ivar_does_shape_transition = nil end def extender @first_extended_ivar = [:ok] end end test = RegressionTest.new # Fill up the transient heap, so rb_ensure_iv_list_size() # listens to GC.stress and yields to the GC. fill = Array.new(0x400000) GC.stress = true # Used to crash due to GC run in rb_ensure_iv_list_size() # not marking the newly allocated [:ok]. test.extender GC.start ``` I will post a patch shortly. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:112533] [Ruby master Bug#19455] Ruby 3.2: wrong Regexp encoding with non-ASCII comments
by janosch-x 17 Jul '23

17 Jul '23

Issue #19455 has been reported by janosch-x (Janosch Müller). ---------------------------------------- Bug #19455: Ruby 3.2: wrong Regexp encoding with non-ASCII comments https://bugs.ruby-lang.org/issues/19455 * Author: janosch-x (Janosch Müller) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- comments and comment groups don't trigger the correct `Regexp#encoding` on Ruby 3.2 anymore: ```ruby # ruby 3.1 /#a/x.encoding # => #<Encoding:US-ASCII> # OK /(?#a)/.encoding # => #<Encoding:US-ASCII> # OK /#ü/x.encoding # => #<Encoding:UTF-8> # OK /(?#ü)/.encoding # => #<Encoding:UTF-8> # OK # ruby 3.2 /#a/x.encoding # => #<Encoding:US-ASCII> # OK /(?#a)/.encoding # => #<Encoding:US-ASCII> # OK /#ü/x.encoding # => #<Encoding:US-ASCII> # BUG /(?#ü)/.encoding # => #<Encoding:US-ASCII> # BUG /#ü/x.inspect # => "/#\\xC3\\xBC/x" /(?#ü)/.inspect # => "/(?#\\xC3\\xBC)/" # bug is hidden if there are non-ascii chars outside comments /ü#ü/x.encoding # => #<Encoding:UTF-8> /ü(?#ü)/.encoding # => #<Encoding:UTF-8> ``` i think these changes might be the cause: https://github.com/ruby/ruby/commit/ec3542229b29ec93062e9d90e877ea29d3c1947… @jeremyevans0 JFYI -- https://bugs.ruby-lang.org/

6 6