July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114241] [Ruby master Bug#19776] Warn bundled gems when it called from `require`
by hsbt (Hiroshi SHIBATA) 21 Jul '23

21 Jul '23

Issue #19776 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19776: Warn bundled gems when it called from `require` https://bugs.ruby-lang.org/issues/19776 * Author: hsbt (Hiroshi SHIBATA) * Status: Assigned * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- We should warn when users try to load default gems that will be promoted bundled gems at next version. We will warn them only under the bundler environment without Gemfile. * Ruby 3.3: * Warn for adding bundled gems to be addressed Ruby 3.4 to Gemfile when user load its gem without `gem 'foo'` in their Gemfile. * Target libraries are TBD * Also warn existing bundled gems was loaded without `gem 'foo'` of Gemfile. Ex. net-smtp, rexml etc. * Ruby 3.4: * Promote bundled gems. * Raise LoadError same as current behavior with warnings of Ruby 3.3. * Warn same as Ruby 3.3. I implemented PoC for this: https://github.com/ruby/ruby/pull/8096 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:114074] [Ruby master Feature#19755] Module#class_eval and Binding#eval use caller location by default
by byroot (Jean Boussier) 20 Jul '23

20 Jul '23

Issue #19755 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #19755: Module#class_eval and Binding#eval use caller location by default https://bugs.ruby-lang.org/issues/19755 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Background In Ruby we're very reliant on `Method#source_location` as well as `caller_locations` to locate source code. However, code generated via `Binding#eval`, `Module#class_eval` etc defeat this if called without a location: ```ruby Foo.class_eval <<~RUBY def bar end RUBY p Foo.instance_method(:bar).source_location # => ["(eval)", 1] ``` The overwhelming majority of open source code properly pass a `filename` and `lineno`, however a small minority doesn't and make locating the code much harder than it needs to be. Here's some example of anonymous eval uses I fixed in the past: - https://github.com/ruby/mutex_m/pull/11 - https://github.com/rails/execjs/pull/120 - https://github.com/jnunemaker/httparty/pull/776 - https://github.com/SiftScience/sift-ruby/pull/76 - https://github.com/activemerchant/active_merchant/pull/4675 - https://github.com/rails/thor/pull/807 - https://github.com/dry-rb/dry-initializer/pull/104 - https://github.com/rmosolgo/graphql-ruby/pull/4288 ### Proposal I suggest that instead of defaulting to `"(eval)"`, the optional `filename` argument of the eval family of methods should instead default to: `"(eval in #{caller_locations(1, 1).first.path})"` and `lineno` to `caller_locations(1, 1).first.lineno`. Which can pretty much be monkey patched as this: ```ruby module ModuleEval def class_eval(code, location = "(eval in #{caller_locations(1, 1).first.path})", lineno = caller_locations(1, 1).first.lineno) super end end Module.prepend(ModuleEval) module Foo class_eval <<~RUBY def foo end RUBY end p Foo.instance_method(:foo) ``` before: ``` #<UnboundMethod: Foo#foo() (eval):1> ``` after: ``` #<UnboundMethod: Foo#foo() (eval in /tmp/foo.rb):10> ``` Of course the `lineno` part is likely to not be fully correct, but the reasoning is that it's better than defaulting to 0 or 1. Another possiblity would be to include the caller `lineo` inside the `filename` part, and leave the actual lineo default to `1`: ``` #<UnboundMethod: Foo#foo() (eval at /tmp/foo.rb:10):1> ``` -- https://bugs.ruby-lang.org/

6 14

[ruby-core:111873] [Ruby master Bug#19351] Promote bundled gems at Ruby 3.3
by hsbt (Hiroshi SHIBATA) 19 Jul '23

19 Jul '23

Issue #19351 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19351: Promote bundled gems at Ruby 3.3 https://bugs.ruby-lang.org/issues/19351 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- In Ruby 3.2, the default gems and bundled gems are changed only adding `syntax_suggest`. I and some committers are considering promote default gems to bundled gems again for Ruby 3.3+. We hope to keep the current developer experience with dependency resolution and ignore the additional work like "Put gem "xxx" into your Gemfile" for developers. ### Proposal We propose the following libraries will promote default gems to bundled gems at Ruby 3.3. They are not the dependencies of Rails and RubyGems/Bundler. ``` abbrev getoptlong optparse observable resolv resolv-replace rinda un fcntl nkf syslog win32ole ``` ### Additional works I also propose to poromote rails dependencies: ``` ostruct base64 irb rdoc tsort singleton delegate ``` and gems maintained by @kou ``` csv strscan fiddle stringio ``` But if we promote them to bundled gems, many of users need to add `gem "csv"` into their Gemfile. I'm considering to avoid this situation. Can we the specific feature of bundled gems to RubyGems or Bundler? Example, bundler have allowed list for bundled gems. So, listed gems could be require without Gemfile under the bundle exec. -- https://bugs.ruby-lang.org/

6 25

[ruby-core:114238] [Ruby master Feature#19775] Use the callback `main::const_added`
by UlyssesZhan (Ulysses Zhan) 19 Jul '23

19 Jul '23

Issue #19775 has been reported by UlyssesZhan (Ulysses Zhan). ---------------------------------------- Feature #19775: Use the callback `main::const_added` https://bugs.ruby-lang.org/issues/19775 * Author: UlyssesZhan (Ulysses Zhan) * Status: Open * Priority: Normal ---------------------------------------- In #17881, it was accepted that `Module#const_added` is called when there is a new constant added to the namespace of the module. Since `main` can also serve as a namespace, it should also be able to use this callback. ```ruby def const_added name p name end Foo = 1 # => :Foo ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:113517] [Ruby master Misc#19679] Migrate Wiki from bugs.ruby-lang.org to ruby/ruby GitHub repository
by jemmai (Jemma Issroff) 19 Jul '23

19 Jul '23

Issue #19679 has been reported by jemmai (Jemma Issroff). ---------------------------------------- Misc #19679: Migrate Wiki from bugs.ruby-lang.org to ruby/ruby GitHub repository https://bugs.ruby-lang.org/issues/19679 * Author: jemmai (Jemma Issroff) * Status: Open * Priority: Normal ---------------------------------------- # Background There is currently a Wiki at https://bugs.ruby-lang.org/projects/ruby/wiki which contains documentation of processes, documentation of code, developers' meeting notes, and various other notes. There are three main problems with the current Wiki setup: * **Out of date:** Much of the documentation is out of date, or no longer relevant (for example: [How to Release Ruby 1.9](https://bugs.ruby-lang.org/projects/ruby/wiki/HowToReleaseRuby19)) * **Duplication:** Some of this documentation exists both in the Wiki and in the docs in ruby/ruby (for example: [DTrace Probes in the docs](https://docs.ruby-lang.org/en/master/dtrace_probes_rdoc.html) and [DTrace Probes in the Wiki](https://bugs.ruby-lang.org/projects/ruby/wiki/DTraceProbes) * **Editing docs is limited:** Only Core contributors can edit the Wiki, and version control is more limited (for example: I couldn't edit any mistakes I found in the Wiki while preparing this issue). # Proposal This proposal is to migrate any still relevant docs within the Wiki from the bugs.ruby-lang.org Wiki to a new GitHub Wiki. I have categorized all of the Wiki pages in [this google sheet](https://docs.google.com/spreadsheets/d/1Ld83ZKxknYgECXxNSh28fjFw82pS… and given my analysis of which should be migrated. If there are any pages that should be migrated which aren't currently included, please indicate so in the comments and we will migrate them. I also propose we migrate the old Developers' Meeting notes to the [developer's meeting repository](https://github.com/ruby/dev-meeting-log). I spoke about this idea with @hsbt at Ruby Kaigi, and he and @naruse suggested that I write this proposal. # Implementation 0. Before I begin the migration, someone who has the appropriate permissions on GitHub will need [to create the Wiki](https://docs.github.com/en/communities/documenting-your-project-with-… 1. I will migrate all pages listed in the Google sheet with "migrate?" checked to the new GitHub wiki 2. I will migrate all of the Developers' Meeting Notes to the developers' meeting repo 3. Someone who has the appropriate permissions will need to remove the Wiki from bugs.ruby-lang.org 4. I will update all GitHub wiki pages with "update?" checked in the Google Sheet to reflect the latest state -- https://bugs.ruby-lang.org/

5 10

[ruby-core:113109] [Ruby master Bug#19577] Backport commit 537183cd2ac0163851277b46a2f21ea5914c11c0
by peterzhu2118 (Peter Zhu) 18 Jul '23

18 Jul '23

Issue #19577 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19577: Backport commit 537183cd2ac0163851277b46a2f21ea5914c11c0 https://bugs.ruby-lang.org/issues/19577 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: DONTNEED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- Please backport commit [537183cd2ac0163851277b46a2f21ea5914c11c0](https://github.com/ruby/ruby/comm…. We've seen this crash in our production systems. -- https://bugs.ruby-lang.org/

3 2

[ruby-core:113129] [Ruby master Bug#19580] Ensure ruby_xfree won't segfault if called after ruby_vm_destruct
by mdalessio (Mike Dalessio) 18 Jul '23

18 Jul '23

Issue #19580 has been reported by mdalessio (Mike Dalessio). ---------------------------------------- Bug #19580: Ensure ruby_xfree won't segfault if called after ruby_vm_destruct https://bugs.ruby-lang.org/issues/19580 * Author: mdalessio (Mike Dalessio) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Github PR: https://github.com/ruby/ruby/pull/7663 The POSIX Threads API provides the ability to define a destructor to clean up thread-local storage at thread exit by using `pthread_key_create` to bind a storage key and an associated destructor function, and `pthread_setspecific` to associate a value with that key. If a C extension is using `ruby_xmalloc` and `ruby_xfree` to manage memory, then the destructor function will likely call `ruby_xfree`. There is a small window of time as a process is exiting -- after `ruby_vm_destruct` but before the process exits -- in which a thread may exit and the destructor function called, leading to a segfault. ### Real-world use case The real-world scenario motivating this change is libxml2's pthread code which uses `pthread_key_create` to set up a destructor that is called at thread exit to clean up thread-local storage associated with the thread by `pthread_setspecific`. Nokogiri has, since 2009, configured libxml2 to use `ruby_xmalloc` and `ruby_xfree` so that Ruby's memory management is aware of the allocated memory. As a result, `ruby_xfree` is being called by the pthread destructor, which may lead to a segfault that looks like: [BUG] Segmentation fault at 0x0000000000000420 ruby 3.3.0dev (2023-03-19T06:13:25Z master ca9355e173) [x86_64-linux] -- Machine register context ------------------------------------------------ RIP: 0x00007f9b6311124e RBP: 0x00007f9b58001530 RSP: 0x00007f9b5d8eed60 RAX: 0x0000000000000000 RBX: 0x00007f9b58001888 RCX: 0x00000000000003ff RDX: 0x00007f9b5dad62b0 RDI: 0x00007f9b58007090 RSI: 0x0000000000000000 R8: 0x00007f9b5d8eed64 R9: 0x00000000000000ca R10: 0x0000000000000000 R11: 0x0000000000000246 R12: 0x00007f9b58007090 R13: 0x00007f9b62e1bae8 R14: 0x0000000000000004 R15: 0x00007f9b5d8efb58 EFL: 0x0000000000010206 -- C level backtrace information ------------------------------------------- .../3.3.0-dev/lib/libruby.so.3.3(rb_print_backtrace+0xd) [0x7f9b632e6f1f] /home/flavorjones/code/oss/ruby/vm_dump.c:785 .../3.3.0-dev/lib/libruby.so.3.3(rb_vm_bugreport) /home/flavorjones/code/oss/ruby/vm_dump.c:1101 .../3.3.0-dev/lib/libruby.so.3.3(rb_bug_for_fatal_signal+0xf4) [0x7f9b630eead4] /home/flavorjones/code/oss/ruby/error.c:813 .../3.3.0-dev/lib/libruby.so.3.3(sigsegv+0x4d) [0x7f9b63233bbd] /home/flavorjones/code/oss/ruby/signal.c:964 /lib/x86_64-linux-gnu/libc.so.6(0x7f9b62c42520) [0x7f9b62c42520] .../3.3.0-dev/lib/libruby.so.3.3(ruby_sized_xfree+0x3) [0x7f9b6311124e] /home/flavorjones/code/oss/ruby/gc.c:12666 .../3.3.0-dev/lib/libruby.so.3.3(ruby_sized_xfree) /home/flavorjones/code/oss/ruby/gc.c:12663 .../lib/nokogiri/nokogiri.so(xmlResetError+0x0) [0x7f9b5da6ab4a] .../tmp/x86_64-linux/nokogiri/3.0.5/tmp/x86_64-pc-linux-gnu/ports/libxml2/2.10.3/libxml2-2.10.3/error.c:880 .../lib/nokogiri/nokogiri.so(xmlResetError) .../tmp/x86_64-linux/nokogiri/3.0.5/tmp/x86_64-pc-linux-gnu/ports/libxml2/2.10.3/libxml2-2.10.3/error.c:873 .../lib/nokogiri/nokogiri.so(xmlResetError) (null):0 .../lib/nokogiri/nokogiri.so(xmlFreeGlobalState+0x14) [0x7f9b5dad62c4] .../tmp/x86_64-linux/nokogiri/3.0.5/tmp/x86_64-pc-linux-gnu/ports/libxml2/2.10.3/libxml2-2.10.3/threads.c:548 /lib/x86_64-linux-gnu/libc.so.6(__nptl_deallocate_tsd+0x77) [0x7f9b62c91711] ./nptl/nptl_deallocate_tsd.c:73 /lib/x86_64-linux-gnu/libc.so.6(__nptl_deallocate_tsd) ./nptl/nptl_deallocate_tsd.c:22 /lib/x86_64-linux-gnu/libc.so.6(start_thread+0x17a) [0x7f9b62c949ca] ./nptl/pthread_create.c:453 [0x7f9b62d26a00] I've reproduced this in Ruby 3.0, 3.1, 3.2, and master by using the following ruby code: # must have an error in it to cause pthread_setspecific to be called html = "<div foo='asdf>asdf</div>" Thread.new { Nokogiri::HTML4::Document.parse(html) } sleep 3 # THREAD_CACHE_TIME exit 0 The timing is admittedly hard to reproduce, but [Gitlab has seen this in their CI pipelines a few dozen times](https://gitlab.com/gitlab-org/gitlab/-/issues/390313) and it's made easier if the process lifetime is extended by an `atexit`-registered function. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:113198] [Ruby master Bug#19593] Crash due to throw data set as cause
by peterzhu2118 (Peter Zhu) 18 Jul '23

18 Jul '23

Issue #19593 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Bug #19593: Crash due to throw data set as cause https://bugs.ruby-lang.org/issues/19593 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/7696 rb_ec_setup_exception did not check if errinfo is a throw_data. This can cause crashes in code since it is assumed that id_cause is an object. We saw a crash in show_cause due to id_cause of errinfo being a throw_data. It crashes on rb_obj_is_kind_of since it cannot be called on T_IMEMO objects. Unfortunately, we couldn't find a reproduction script, however we debugged the core dump and rb_ec_setup_exception is the only place where id_cause is assigned from errinfo without checking if it is a throw_data. ``` 0x0000556c5708e6dd in sigsegv (sig=11, info=0x7f301befa3f0, ctx=0x7f301befa2c0) at signal.c:964 0x00007f301d046420 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0 class_search_class_ancestor (c=139844586301760, cl=<optimized out>) at object.c:810 rb_obj_is_kind_of (obj=obj@entry=139839221734880, c=139844586301760) at object.c:861 0x0000556c56f2f00f in show_cause (errinfo=errinfo@entry=139838840645160, str=str@entry=139839221730520, opt=139839221730480, highlight=0, reverse=reverse@entry=0, backtrace_limit=backtrace_limit@entry=-1, shown_causes=0x7ffe9d1a2d68) at ./include/ruby/internal/special_consts.h:175 ``` -- https://bugs.ruby-lang.org/

3 2

[ruby-core:114224] [Ruby master Bug#19773] Intermittent SIGSEGV during marking of Ripper parser
by eightbitraptor (Matthew Valentine-House) 18 Jul '23

18 Jul '23

Issue #19773 has been reported by eightbitraptor (Matthew Valentine-House). ---------------------------------------- Bug #19773: Intermittent SIGSEGV during marking of Ripper parser https://bugs.ruby-lang.org/issues/19773 * Author: eightbitraptor (Matthew Valentine-House) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Intermittent SIGSEGV during marking of the Ripper parser. A failing build [can be seen on ci.rvm.jp (build 4643684)](http://ci.rvm.jp/results/trunk-random2@ruby-sp2-docker/4643684). Inside `ripper_s_allocate` first we use `TypedData_Make_Struct` to create a new `T_DATA` object in the Ruby heap. This allocates an empty struct (`struct ripper`). Further down the function we allocate another struct (`struct parser_params` during `rb_ruby_ripper_parser_allcoate`) and assign it to a pointer from the first struct, which is attached to the `T_DATA` object. `rb_ruby_ripper_parser_allocate` can trigger GC, as it uses `ruby_xcalloc`. But if GC is triggered at this point then the `parser_params` have not been allocated yet, and so the `struct ripper` contains a `NULL` reference. This causes a SEGV when we try and mark the `T_DATA` wrapper around the `struct ripper`. This can be replicated locally using the following test script ``` require "ripper" GC.stress = true Ripper.new ``` -- https://bugs.ruby-lang.org/

1 1

[ruby-core:114137] [Ruby master Feature#19764] Introduce defp keyword for defining overloadable, pattern matched methods
by zeke (Zeke Gabrielse) 18 Jul '23

18 Jul '23

Issue #19764 has been reported by zeke (Zeke Gabrielse). ---------------------------------------- Feature #19764: Introduce defp keyword for defining overloadable, pattern matched methods https://bugs.ruby-lang.org/issues/19764 * Author: zeke (Zeke Gabrielse) * Status: Open * Priority: Normal ---------------------------------------- Pattern matching has become one of my favorite features of Ruby, if not my favorite. It changed the way I write and express my thoughts through clean, maintainable code. And I'd like to use it *more*. I propose a new keyword, `defp`, for defining a method which applies pattern matching to its arguments. ```ruby defp call(String => s unless s in /^[a-z]/) puts "string: #{s.inspect} (capitalized)" end defp call(String => s) puts "string: #{s.inspect}" end defp call(Hash(foo:, bar:) => h) puts "hash: #{h.inspect}" end defp call(**nil) puts "no keyword args" end call("Example") # => string: "Example" (capitalized) call("test") # => string: "test" call(foo: 1, bar: 2) # => hash: { :foo => 1, :bar => 2 } ``` Internally, this could be represented as the following `case..in` pseudocode: ```ruby def call(...) case ... in String => s unless s in /foo/ puts "string: #{s.inspect} (not foo)" in String => s puts "string: #{s.inspect}" in Hash(foo:, bar:) => h puts "hash: #{h.inspect}" in **nil puts "no keyword args" else raise NoMatchingMethod end end ``` As you could imagine, this could be used to refactor a lot of code, making the developer's intent much clearer. From [complex methods that use `case` statements](https://github.com/rails/rails/blob/593893c901f87b4ed205751f72d… for taking varied arguments (I'm sure all our code bases contain such `case` statements), to defining smaller, simpler methods that handle particular argument patterns. In addition, not only can this improve code quality, but it brings in method overloads, and it also adds a way to define more typing to the language -- something that RBS has tried to do, to mixed reactions -- but in a more Ruby-like way that Rubyists are already learning *and loving*. Thoughts? Original idea by Victor Shepelev: https://zverok.space/blog/2023-05-05-ruby-types.html Further discussion: https://news.ycombinator.com/item?id=35834351 -- https://bugs.ruby-lang.org/

5 4