November 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115462] [Ruby master Feature#20018] Get the error codes as Socket::EAI_XXX when getaddrinfo and getnameinfo fail
by shioimm (Misaki Shioi) 07 Dec '23

07 Dec '23

Issue #20018 has been reported by shioimm (Misaki Shioi). ---------------------------------------- Feature #20018: Get the error codes as Socket::EAI_XXX when getaddrinfo and getnameinfo fail https://bugs.ruby-lang.org/issues/20018 * Author: shioimm (Misaki Shioi) * Status: Open * Priority: Normal ---------------------------------------- Feature for implementing Happy Eyeballs version 2 (RFC 8305) in `Socket.tcp`. ### Probrem I would like to implement Happy Eyeballs version 2 (RFC 8305) in `Socket.tcp`. In HEv2, `EAI_ADDRFAMILY` and `EAI_AGAIN` should be ignored when `getaddrinfoo` returns them in hostname resolution. However, there is currently no way to ask `SocketError` for errors returned by `getaddrinfo` ### Proposal I opened a PR to get errors when `getaddrinfo` and `getnameinfo` returns them. https://github.com/ruby/ruby/pull/9018 Todo this, - I added `Socket::ResolutionError` as a subclass of `SocketError` - I added an attribute of `Socket::ResolutionError` to get error code via `#error_code - I replaced `SocketError` (`rb_eSocket`) with `Socket::ResolutionError (`rb_eResolutionError`) in `rsock_raise_socket_error` because this function is only called when `getaddrinfo` and `getnameinfo` fail - I renamed `rsock_raise_socket_error` to `rock_raise_resolution_error` for the same reason In addition, `Socket::ResolutionError` is a subclass of `SocketError`, so it does not affect `rescue SocketError` in source codes. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:115428] [Ruby master Feature#20011] Reduce implicit array allocations on caller side of method calling
by jeremyevans0 (Jeremy Evans) 07 Dec '23

07 Dec '23

Issue #20011 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Feature #20011: Reduce implicit array allocations on caller side of method calling https://bugs.ruby-lang.org/issues/20011 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Priority: Normal ---------------------------------------- I would like to use the peephole optimizer to eliminate caller-side array allocations for the following cases, by switching `splatarray true` to `splatarray false`: ```ruby f(1, *a) f(1, *a, &lvar) f(1, *a, &@ivar) f(*a, **lvar) f(*a, **@ivar) f(*a, **lvar, &lvar) f(*a, **@ivar, &@ivar) f(*a, kw: 1) f(*a, kw:1, &lvar) f(*a, kw:1, &@ivar) ``` In terms of safety, currently, `f(*a, &lvar)` and `f(*a, &@ivar)` both avoid array allocations (`splatarray false`), and all of the above are as safe as those in terms of safety. Note that since at least Ruby 1.8, in pathlogical cases, `lvar.to_proc` can modify `a`, which results in behavior contrary to expected evaluation order: ```ruby ary = [1,2] kwd = Object.new kwd.define_singleton_method(:to_proc) {ary << 4; lambda{}} p(*ary, &kwd) # 4 included in output ``` I think that for both the current `f(*a, &lvar)` and `f(*a, &@ivar)` cases and all of the above cases, the benefit of avoiding the allocation is higher than the costs, considering that only pathologic cases fail. If we do not want the above optimization, for consistency, we should remove the optimization of `f(*a, &lvar)` and `f(*a, &@ivar)` (or update the optimization so that it is only used if `lvar` or `@ivar` is already a proc), which will slow Ruby down. I have submitted a pull request for these changes: https://github.com/ruby/ruby/pull/8853 To make sure these cases are worth optimizing, I did some analysis: For ruby -e '' (just loads rubygems): ``` 5 : f(1, *a) 2 : f(1, *a, &lvar) | f(1, *a, &@ivar) ``` Current stdlib: ``` 139 : f(1, *a) 3 : f(1, *a, &lvar) | f(1, *a, &@ivar) 4 : f(*a, **lvar) | f(*a, **@ivar) ``` Rails master: ``` 77 : f(1, *a) 5 : f(1, *a, &lvar) | f(1, *a, &@ivar) 26 : f(*a, **lvar) | f(*a, **@ivar) 5 : f(*a, kw: 1) ``` minitest 5.20 (bundled gem): ``` 4 : f(1, *a) 1 : f(1, *a, &lvar) | f(1, *a, &@ivar) 2 : f(*a, **lvar) | f(*a, **@ivar) 2 : f(*a, **lvar, &lvar) | f(*a, **@ivar, &@ivar) ``` capybara 3.39.2: ``` 15 : f(1, *a) 2 : f(1, *a, &lvar) | f(1, *a, &@ivar) 19 : f(*a, **lvar) | f(*a, **@ivar) 4 : f(*a, **lvar, &lvar) | f(*a, **@ivar, &@ivar) ``` This shows that all cases optimized except `f(*a, kw:1, &lvar)` and `f(*a, kw:1, &@ivar)` are used in common gems. Those cases could be removed if people think they are not worth optimizing. Code backing the above analysis can be found at https://github.com/ruby/ruby/pull/8853#issuecomment-1817656139 -- https://bugs.ruby-lang.org/

3 7

[ruby-core:115388] [Ruby master Feature#20005] Add C API to return symbols of native extensions resolved from features
by tagomoris (Satoshi Tagomori) 07 Dec '23

07 Dec '23

Issue #20005 has been reported by tagomoris (Satoshi Tagomori). ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/

4 13

[ruby-core:115304] [Ruby master Feature#19993] Optionally Free all memory at exit
by HParker (Adam Hess) 07 Dec '23

07 Dec '23

Issue #19993 has been reported by HParker (Adam Hess). ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/

4 7

[ruby-core:115333] [Ruby master Misc#19997] DevMeeting-2023-11-30
by mame (Yusuke Endoh) 06 Dec '23

06 Dec '23

Issue #19997 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19997: DevMeeting-2023-11-30 https://bugs.ruby-lang.org/issues/19997 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/11/30 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/11/27. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

10 14

[ruby-core:114785] [Ruby master Bug#19886] "default->bundled gem" warning is not shown under "bundle exec"
by mame (Yusuke Endoh) 06 Dec '23

06 Dec '23

Issue #19886 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #19886: "default->bundled gem" warning is not shown under "bundle exec" https://bugs.ruby-lang.org/issues/19886 * Author: mame (Yusuke Endoh) * Status: Assigned * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ``` $ cat Gemfile source "https://rubygems.org" $ cat test.rb require "base64" $ bundle exec ruby test.rb $ ``` In this situation, `bundle exec ruby test.rb` should print a warning: base64 which will be not part of the default gems since Ruby 3.4.0. Note that `ruby test.rb` shows the warning, which is not needed (see #19885). ``` $ ruby test.rb test.rb:1: warning: base64 which will be not part of the default gems since Ruby 3.4.0 ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115484] [Ruby master Bug#20022] GC.verify_compaction_references does not actually move alll objects
by kjtsanaktsidis (KJ Tsanaktsidis) 03 Dec '23

03 Dec '23

Issue #20022 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20022: GC.verify_compaction_references does not actually move alll objects https://bugs.ruby-lang.org/issues/20022 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- While debugging https://bugs.ruby-lang.org/issues/20021, I ran into a separate issue which I figured was worth resolving whilst I had it in my head. The intention of GC.verify_compaction_references is, I believe, to force every single movable object to be moved, so that it's possible to debug native extensions which not correctly updating their references to objects they mark as movable (if this is _not_ the case, you can stop reading now and accept my apologies!) To do this, it doubles the number of allocated pages for each size pool, and sorts the heap pages so that the free ones are swept first; thus, every object in an old page should be moved into a free slot in one of the new pages. This worked fine until movement of objects between size pools during compaction was implemented. That causes some problems for verify_compaction_references: - We were doubling the number of pages in each size pool, but actually if some objects need to move into a different pool, there's no guarantee that they'll be enough room in that one. - It's possible for the sweep & compact cursors to meet in one size pool before all the objects that want to move into that size pool from another are processed by the compaction. You can see these problems by changing some of the movement tests in test_gc_compact.rb to try and move e.g. 50,000 objects instead of 500 (by changing `HASH_COUNT`); the test is not able to actually move all of the objects in a single compaction run. What I implemented in this PR (https://github.com/ruby/ruby/pull/9041) is two things: - Firstly, it examines every object and determine where it wants to be compacted into; we use this information to calculate the desired number of pages to add to each size pool. - Secondly, it compacts twice; once in order of ascending size pool, and once descending. This means that we are guaranteed to be able to move everything we want to move into a size pool before we start advancing that pool's own compact cursor. With these fixes in place, I was able to make the compaction tests move any amount of objects (previously, `test_moving_hashes_down_size_pools` would begin failing on my machine if I set `HASH_COUNT` to above about 6,000. -- https://bugs.ruby-lang.org/

2 6

[ruby-core:115546] [Ruby master Feature#19117] Include the method owner in backtraces, not just the method name
by Eregon (Benoit Daloze) 30 Nov '23

30 Nov '23

Issue #19117 has been updated by Eregon (Benoit Daloze). mame (Yusuke Endoh) wrote in #note-21: > * How much impact does exception generation have on performance? (Each frame of the backtrace needs to maintain self) That's a good point. While the method name can be stored in the bytecode or so, not so much for the module name (especially since it can get a proper name later). In many cases it can be known statically or is the same when defining the method, but not always (e.g. `MyStruct = Struct.new(:x) { def foo; end; }`). Maybe we should only use this notation when the module name is known statically? ---------------------------------------- Feature #19117: Include the method owner in backtraces, not just the method name https://bugs.ruby-lang.org/issues/19117#change-105481 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.4 ---------------------------------------- ``` module Foo class Bar def inspect 1 + '1' end end end p Foo::Bar.new ``` This code produce the following backtrace: ``` /tmp/foo.rb:4:in `+': String can't be coerced into Integer (TypeError) from /tmp/foo.rb:4:in `inspect' from /tmp/foo.rb:9:in `p' from /tmp/foo.rb:9:in `<main>' ``` This works, but on large codebases and large backtraces the method name isn't always all that revealing, most of the time you need to open many of the locations listed in the backtrace to really understand what is going on. I propose that we also include the owner name: ``` /tmp/foo.rb:4:in `Integer#+': String can't be coerced into Integer (TypeError) from /tmp/foo.rb:4:in `Foo::Bar#inspect' from /tmp/foo.rb:9:in `Kernel#p' from /tmp/foo.rb:9:in `<main>' ``` I believe that in many case it would allow to much better understand the backtrace without having to jump back and forth between it and the source code. This is inspired by @ivoanjo 's `backtracie` gem: https://github.com/ivoanjo/backtracie -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115541] [Ruby master Feature#19117] Include the method owner in backtraces, not just the method name
by byroot (Jean Boussier) 30 Nov '23

30 Nov '23

Issue #19117 has been updated by byroot (Jean Boussier). Target version set to 3.4 Agreed. I took the liberty to create a 3.4 target version and add it. ---------------------------------------- Feature #19117: Include the method owner in backtraces, not just the method name https://bugs.ruby-lang.org/issues/19117#change-105476 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.4 ---------------------------------------- ``` module Foo class Bar def inspect 1 + '1' end end end p Foo::Bar.new ``` This code produce the following backtrace: ``` /tmp/foo.rb:4:in `+': String can't be coerced into Integer (TypeError) from /tmp/foo.rb:4:in `inspect' from /tmp/foo.rb:9:in `p' from /tmp/foo.rb:9:in `<main>' ``` This works, but on large codebases and large backtraces the method name isn't always all that revealing, most of the time you need to open many of the locations listed in the backtrace to really understand what is going on. I propose that we also include the owner name: ``` /tmp/foo.rb:4:in `Integer#+': String can't be coerced into Integer (TypeError) from /tmp/foo.rb:4:in `Foo::Bar#inspect' from /tmp/foo.rb:9:in `Kernel#p' from /tmp/foo.rb:9:in `<main>' ``` I believe that in many case it would allow to much better understand the backtrace without having to jump back and forth between it and the source code. This is inspired by @ivoanjo 's `backtracie` gem: https://github.com/ivoanjo/backtracie -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115539] [Ruby master Feature#19117] Include the method owner in backtraces, not just the method name
by mame (Yusuke Endoh) 30 Nov '23

30 Nov '23

Issue #19117 has been updated by mame (Yusuke Endoh). Let's discuss this towards Ruby 3.4 :cry: ---------------------------------------- Feature #19117: Include the method owner in backtraces, not just the method name https://bugs.ruby-lang.org/issues/19117#change-105474 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ``` module Foo class Bar def inspect 1 + '1' end end end p Foo::Bar.new ``` This code produce the following backtrace: ``` /tmp/foo.rb:4:in `+': String can't be coerced into Integer (TypeError) from /tmp/foo.rb:4:in `inspect' from /tmp/foo.rb:9:in `p' from /tmp/foo.rb:9:in `<main>' ``` This works, but on large codebases and large backtraces the method name isn't always all that revealing, most of the time you need to open many of the locations listed in the backtrace to really understand what is going on. I propose that we also include the owner name: ``` /tmp/foo.rb:4:in `Integer#+': String can't be coerced into Integer (TypeError) from /tmp/foo.rb:4:in `Foo::Bar#inspect' from /tmp/foo.rb:9:in `Kernel#p' from /tmp/foo.rb:9:in `<main>' ``` I believe that in many case it would allow to much better understand the backtrace without having to jump back and forth between it and the source code. This is inspired by @ivoanjo 's `backtracie` gem: https://github.com/ivoanjo/backtracie -- https://bugs.ruby-lang.org/

1 0