December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115428] [Ruby master Feature#20011] Reduce implicit array allocations on caller side of method calling
by jeremyevans0 (Jeremy Evans) 06 Dec '23

06 Dec '23

Issue #20011 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Feature #20011: Reduce implicit array allocations on caller side of method calling https://bugs.ruby-lang.org/issues/20011 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Priority: Normal ---------------------------------------- I would like to use the peephole optimizer to eliminate caller-side array allocations for the following cases, by switching `splatarray true` to `splatarray false`: ```ruby f(1, *a) f(1, *a, &lvar) f(1, *a, &@ivar) f(*a, **lvar) f(*a, **@ivar) f(*a, **lvar, &lvar) f(*a, **@ivar, &@ivar) f(*a, kw: 1) f(*a, kw:1, &lvar) f(*a, kw:1, &@ivar) ``` In terms of safety, currently, `f(*a, &lvar)` and `f(*a, &@ivar)` both avoid array allocations (`splatarray false`), and all of the above are as safe as those in terms of safety. Note that since at least Ruby 1.8, in pathlogical cases, `lvar.to_proc` can modify `a`, which results in behavior contrary to expected evaluation order: ```ruby ary = [1,2] kwd = Object.new kwd.define_singleton_method(:to_proc) {ary << 4; lambda{}} p(*ary, &kwd) # 4 included in output ``` I think that for both the current `f(*a, &lvar)` and `f(*a, &@ivar)` cases and all of the above cases, the benefit of avoiding the allocation is higher than the costs, considering that only pathologic cases fail. If we do not want the above optimization, for consistency, we should remove the optimization of `f(*a, &lvar)` and `f(*a, &@ivar)` (or update the optimization so that it is only used if `lvar` or `@ivar` is already a proc), which will slow Ruby down. I have submitted a pull request for these changes: https://github.com/ruby/ruby/pull/8853 To make sure these cases are worth optimizing, I did some analysis: For ruby -e '' (just loads rubygems): ``` 5 : f(1, *a) 2 : f(1, *a, &lvar) | f(1, *a, &@ivar) ``` Current stdlib: ``` 139 : f(1, *a) 3 : f(1, *a, &lvar) | f(1, *a, &@ivar) 4 : f(*a, **lvar) | f(*a, **@ivar) ``` Rails master: ``` 77 : f(1, *a) 5 : f(1, *a, &lvar) | f(1, *a, &@ivar) 26 : f(*a, **lvar) | f(*a, **@ivar) 5 : f(*a, kw: 1) ``` minitest 5.20 (bundled gem): ``` 4 : f(1, *a) 1 : f(1, *a, &lvar) | f(1, *a, &@ivar) 2 : f(*a, **lvar) | f(*a, **@ivar) 2 : f(*a, **lvar, &lvar) | f(*a, **@ivar, &@ivar) ``` capybara 3.39.2: ``` 15 : f(1, *a) 2 : f(1, *a, &lvar) | f(1, *a, &@ivar) 19 : f(*a, **lvar) | f(*a, **@ivar) 4 : f(*a, **lvar, &lvar) | f(*a, **@ivar, &@ivar) ``` This shows that all cases optimized except `f(*a, kw:1, &lvar)` and `f(*a, kw:1, &@ivar)` are used in common gems. Those cases could be removed if people think they are not worth optimizing. Code backing the above analysis can be found at https://github.com/ruby/ruby/pull/8853#issuecomment-1817656139 -- https://bugs.ruby-lang.org/

3 7

[ruby-core:115625] [Ruby master Bug#18886] Struct aref and aset don't trigger any tracepoints.
by ko1 (Koichi Sasada) 06 Dec '23

06 Dec '23

Issue #18886 has been updated by ko1 (Koichi Sasada). Please merge it. > I don't think this will affect performance unless TracePoint is used, because the same method caching is done in both cases. Showing benchmarking result will help IMO to confirm there is no penalty. ---------------------------------------- Bug #18886: Struct aref and aset don't trigger any tracepoints. https://bugs.ruby-lang.org/issues/18886#change-105557 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal * Assignee: ko1 (Koichi Sasada) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Given the following program, `thing.name` and `thing.shape` don't trigger `c_call` trace points (or any trace points actually). ```ruby pp RUBY_VERSION trace_point = TracePoint.new(:line, :call, :c_call, :a_call) do |trace| puts trace.event if trace.event == :call # Ruby doesn't always mark call-sites in sub-expressions, so we use this approach to compute a call site and mark it: if location = caller_locations(2, 1).first and path = location.path puts "> #{path}:#{location.lineno}:#{trace.event}" end end if path = trace.path puts "= #{path}:#{trace.lineno}:#{trace.event}" end end trace_point.enable # This will trigger call trace points class Thing def name :cat end def shape :square end end thing = Thing.new # Thing = Struct.new(:name, :shape) # thing = Thing.new(:cat, :rectangle) [ name: thing.name, shape: thing.shape, ] ``` ## Current HEAD ``` = ../test.rb:30:line: = ../test.rb:30:c_call:new = ../test.rb:30:c_call:inherited = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:const_added = ../test.rb:31:line: = ../test.rb:31:c_call:new = ../test.rb:31:c_call:initialize = ../test.rb:34:line: ``` ## Proposed PR ``` = ../test.rb:30:line: = ../test.rb:30:c_call:new = ../test.rb:30:c_call:inherited = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:singleton_method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:method_added = ../test.rb:30:c_call:const_added = ../test.rb:31:line: = ../test.rb:31:c_call:new = ../test.rb:31:c_call:initialize = ../test.rb:34:line: = ../test.rb:34:c_call:name = ../test.rb:35:c_call:shape ``` The reason is the internal implementation of struct doesn't have trace point instrumentation in `vm_call_opt_struct_aset` or `vm_call_opt_struct_aref`. Proposed fix: https://github.com/ruby/ruby/pull/6071 but this would need a review, maybe @jeremyevans0 and @ko1 can help. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115388] [Ruby master Feature#20005] Add C API to return symbols of native extensions resolved from features
by tagomoris (Satoshi Tagomori) 06 Dec '23

06 Dec '23

Issue #20005 has been reported by tagomoris (Satoshi Tagomori). ---------------------------------------- Feature #20005: Add C API to return symbols of native extensions resolved from features https://bugs.ruby-lang.org/issues/20005 * Author: tagomoris (Satoshi Tagomori) * Status: Open * Priority: Normal ---------------------------------------- I want an API to resolve symbols of other native extensions by a feature name and a symbol name (just like `dlsym`). (`rb_dln_resolve_symbol` is an example function name of this feature in the example below) ```c // "a_client_open" is a function defined in "/..../a_client.so", that should be loaded in Ruby beforehand. // in the extension to be "a.so" VALUE (*a_client_open)(VALUE); void Init_a(void) { a_client_open = (VALUE(*)(VALUE))rb_dln_resolve_symbol("a_client", "a_client_open"); // the return value may be NULL if the symbols is not found } ``` This API is to replace direct reference of external symbols. Currently, native extensions have to just call the function of other extensions directly on the assumption of: * The dependency native extension is already loaded * The symbol is defined correctly in the dependency (the version of dependency is correct&expected) Otherwise, it crashes. This API provides a way for developers to raise exceptions if the resolved symbol is `NULL`, and these exceptions can display meaningful messages and instructions to the users of extensions. -- https://bugs.ruby-lang.org/

4 13

[ruby-core:115619] [Ruby master Feature#16431] Optionally load did_you_mean (and RubyGems)
by hsbt (Hiroshi SHIBATA) 06 Dec '23

06 Dec '23

Issue #16431 has been updated by hsbt (Hiroshi SHIBATA). Status changed from Assigned to Closed https://github.com/ruby/ruby/pull/2764 has been merged. ---------------------------------------- Feature #16431: Optionally load did_you_mean (and RubyGems) https://bugs.ruby-lang.org/issues/16431#change-105551 * Author: vo.x (Vit Ondruch) * Status: Closed * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) ---------------------------------------- I just have opened PR [1], which allows Ruby to run when did_you_mean is not available. As I previously discussed in #16427, there are cases when speed, memory, disk or network bandwidth is a concern and did_you_mean is not useful for runtime. This is especially useful when Ruby is installed via packaging systems of (Linux) distributions. The PR is split into 4 commits, because since I was there, I prepared similar changes to RubyGems. [1]: https://github.com/ruby/ruby/pull/2764 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115304] [Ruby master Feature#19993] Optionally Free all memory at exit
by HParker (Adam Hess) 06 Dec '23

06 Dec '23

Issue #19993 has been reported by HParker (Adam Hess). ---------------------------------------- Feature #19993: Optionally Free all memory at exit https://bugs.ruby-lang.org/issues/19993 * Author: HParker (Adam Hess) * Status: Open * Priority: Normal ---------------------------------------- Add a runtime option allowing Ruby to optionally free all memory at shutdown. # why Today, memory sanitizers are difficult to use with Ruby, since not all memory is freed at shutdown. it is difficult to detect memory leaks or errors in Ruby or in Ruby C extensions when these tools are not available. While implementing this feature, we were able to identify and fix a number of memory leaks and errors. https://github.com/ruby/ruby/pull/8556 https://github.com/ruby/ruby/pull/8512 https://github.com/ruby/ruby/pull/8503 https://github.com/ruby/ruby/pull/8487 https://github.com/ruby/ruby/pull/8452 https://github.com/ruby/ruby/pull/8501 https://github.com/ruby/ruby/pull/8481 https://github.com/ruby/ruby/pull/8555 This shows that having access to tools like this can make finding and fixing memory bugs easier. # current progress Today we can allow ruby developers to enable freeing memory at shutdown via the `free-at-shutdown` runtime option. PR: https://github.com/ruby/ruby/pull/8868 running, without `--free-on-shutdown`: ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby basictest/test.rb ``` ``` ==573270== LEAK SUMMARY: ==573270== definitely lost: 658,324 bytes in 5,387 blocks ==573270== indirectly lost: 955,708 bytes in 11,957 blocks ==573270== possibly lost: 2,071,096 bytes in 12 blocks ==573270== still reachable: 161,881 bytes in 275 blocks ==573270== suppressed: 0 bytes in 0 blocks ==573270== Reachable blocks (those to which a pointer was found) are not shown. ==573270== To see them, rerun with: --leak-check=full --show-leak-kinds=all ``` with `--free-on-shutdown` ``` valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb ``` ``` ==573306== HEAP SUMMARY: ==573306== in use at exit: 0 bytes in 0 blocks ==573306== total heap usage: 43,643 allocs, 43,643 frees, 29,222,534 bytes allocated ==573306== ==573306== All heap blocks were freed -- no leaks are possible ``` # future plans * Continue improving memory safety * add `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes -- ./miniruby --free-on-shutdown basictest/test.rb` to CI * Allow C extensions to do a "optional destruct for memory safety" so they can leverage the same memory sanitizer tools. -- https://bugs.ruby-lang.org/

4 7

[ruby-core:115333] [Ruby master Misc#19997] DevMeeting-2023-11-30
by mame (Yusuke Endoh) 06 Dec '23

06 Dec '23

Issue #19997 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19997: DevMeeting-2023-11-30 https://bugs.ruby-lang.org/issues/19997 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/11/30 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/11/27. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

10 14

[ruby-core:115593] [Ruby master Bug#20041] Array destructuring and default values in parameters
by tenderlovemaking (Aaron Patterson) 06 Dec '23

06 Dec '23

Issue #20041 has been reported by tenderlovemaking (Aaron Patterson). ---------------------------------------- Bug #20041: Array destructuring and default values in parameters https://bugs.ruby-lang.org/issues/20041 * Author: tenderlovemaking (Aaron Patterson) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- It's possible to set the default value of a parameter to a previous parameter. For example: ```ruby def foo(a, b = a) b end foo([1, 2]) # => [1, 2] ``` However, if the parameters are destructured, the destructring happens _after_ default parameter assignment. For example: ```ruby def foo((x, y), b = x) [x, y, b] end foo([1, 2]) # => [1, 2, nil] ``` Is this expected behavior? I would have expected the parameters to be "evaluated" from left to right, and the array destructuring to happen _before_ the default parameter assignment. Thanks! -- https://bugs.ruby-lang.org/

2 3

[ruby-core:115604] [Ruby master Feature#18576] Rename `ASCII-8BIT` encoding to `BINARY`
by Eregon (Benoit Daloze) 06 Dec '23

06 Dec '23

Issue #18576 has been updated by Eregon (Benoit Daloze). Target version set to 3.4 @matz Could we try this again for 3.4, soon after the 3.3 release? Then there is plenty of time to discover any issue related to it (probably very few as gems have been patched, and applications using encoding names instead of encoding constants are likely very old and unlikely to use a recent Ruby). ---------------------------------------- Feature #18576: Rename `ASCII-8BIT` encoding to `BINARY` https://bugs.ruby-lang.org/issues/18576#change-105535 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.4 ---------------------------------------- ### Context I'm now used to it, but something that confused me for years was errors such as: ```ruby >> "fée" + "\xFF".b (irb):3:in `+': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError) ``` When you aren't that familiar with Ruby, it's really not evident that `ASCII-8BIT` basically means "no encoding" or "binary". And even when you know it, if you don't read carefully it's very easily confused with `US-ASCII`. The `Encoding::BINARY` alias is much more telling IMHO. ### Proposal Since `Encoding::ASCII_8BIT` has been aliased as `Encoding::BINARY` for years, I think renaming it to `BINARY` and then making asking `ASCII_8BIT` the alias would significantly improve usability without backward compatibility concerns. The only concern I could see would be the consistency with a handful of C API functions: - `rb_encoding *rb_ascii8bit_encoding(void)` - `int rb_ascii8bit_encindex(void)` - `VALUE rb_io_ascii8bit_binmode(VALUE io)` But that's for much more advanced users, so I don't think it's much of a concern. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114785] [Ruby master Bug#19886] "default->bundled gem" warning is not shown under "bundle exec"
by mame (Yusuke Endoh) 06 Dec '23

06 Dec '23

Issue #19886 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #19886: "default->bundled gem" warning is not shown under "bundle exec" https://bugs.ruby-lang.org/issues/19886 * Author: mame (Yusuke Endoh) * Status: Assigned * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ``` $ cat Gemfile source "https://rubygems.org" $ cat test.rb require "base64" $ bundle exec ruby test.rb $ ``` In this situation, `bundle exec ruby test.rb` should print a warning: base64 which will be not part of the default gems since Ruby 3.4.0. Note that `ruby test.rb` shows the warning, which is not needed (see #19885). ``` $ ruby test.rb test.rb:1: warning: base64 which will be not part of the default gems since Ruby 3.4.0 ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115602] [Ruby master Bug#19147] `TestFileExhaustive#test_expand_path_for_existent_username` and `TestDir#test_home` fails on i686
by vo.x (Vit Ondruch) 06 Dec '23

06 Dec '23

Issue #19147 has been updated by vo.x (Vit Ondruch). Testing further, there might be issue in some other places. While I can't reproduce the issue on Fedora Rawhide, it still fails on Fedora 38. I was not expecting this. Might the culprit be e.g. glibc? Or is it FS type related? ---------------------------------------- Bug #19147: `TestFileExhaustive#test_expand_path_for_existent_username` and `TestDir#test_home` fails on i686 https://bugs.ruby-lang.org/issues/19147#change-105533 * Author: vo.x (Vit Ondruch) * Status: Closed * Priority: Normal * ruby -v: ruby 3.2.0dev (2022-11-24 master 66e5200ba4) [i386-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Testing with commit:git|66e5200ba4 on Fedora Rawhide, I observe following error just on i686 (other platforms are passing just fine): ~~~ 1) Error: TestFileExhaustive#test_expand_path_for_existent_username: RuntimeError: can't set length of shared string /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_file_exhaustive.rb:1122:in `expand_path' /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_file_exhaustive.rb:1122:in `test_expand_path_for_existent_username' 2) Error: TestDir#test_home: RuntimeError: can't set length of shared string /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_dir.rb:537:in `expand_path' /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_dir.rb:537:in `block in test_home' ~~~ Previously testing with commit:git|4b1504ae0a, the tests were passing just fine. -- https://bugs.ruby-lang.org/

1 0