January 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:116483] [Ruby master Bug#20223] For simple objects each_cons appears to work as I expect, but for other objects it seems to fail to terminate iterations when I would expect.
by mjflynt (Jeffrey Flynt) 29 Jan '24

29 Jan '24

Issue #20223 has been reported by mjflynt (Jeffrey Flynt). ---------------------------------------- Bug #20223: For simple objects each_cons appears to work as I expect, but for other objects it seems to fail to terminate iterations when I would expect. https://bugs.ruby-lang.org/issues/20223 * Author: mjflynt (Jeffrey Flynt) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Reproduce process: ``` ruby p `ruby -v` class X include Enumerable attr_accessor :obj def initialize = @obj = (1..5).to_a def each(&block) = yield @obj.each(&block) end wtf = X.new (1..5).each_cons(3) { |g| p g } p '~' * 20 (1..5).to_a.each_cons(3) { |g| p g } p '~' * 20 ('a'..'e').each_cons(3) { |g| p g } p '~' * 20 wtf.each_cons(3) { |g| p g } ``` Result of reproduce process ``` "ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux]\n" [1, 2, 3] [2, 3, 4] [3, 4, 5] "~~~~~~~~~~~~~~~~~~~~" [1, 2, 3] [2, 3, 4] [3, 4, 5] "~~~~~~~~~~~~~~~~~~~~" ["a", "b", "c"] ["b", "c", "d"] ["c", "d", "e"] "~~~~~~~~~~~~~~~~~~~~" [1, 2, 3] [2, 3, 4] [3, 4, 5] [4, 5, [1, 2, 3, 4, 5]] [Done] exited with code=0 in 0.173 seconds ``` Expected result and the reason why you expect I expect the results to not include the last line of output "[4, 5, [1, 2, 3, 4, 5]]" because my understanding is each_cons ends its iteration over the collection before it runs out of elements. In this case, this fourth row does not have enough elements to do three consecutive elements, but it is doing so anyway, appearing to use the original array as the third element before it completes. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116473] [Ruby master Bug#20221] ASAN: make test-basic: un-prefixed symbol leakage
by leeN (David Klein) 29 Jan '24

29 Jan '24

Issue #20221 has been reported by leeN (David Klein). ---------------------------------------- Bug #20221: ASAN: make test-basic: un-prefixed symbol leakage https://bugs.ruby-lang.org/issues/20221 * Author: leeN (David Klein) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-01-29T08:16:49Z master 8bff7e996c) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When building and running the tests (here, test-basic) with ASAN enabled, it fails with the following message: ``` Checking leaked global symbols...leaked __odr_asan_gen_rb_cArray __odr_asan_gen_ruby_digitmap __odr_asan_gen_rb_mComparable ...snip... __odr_asan_gen_OnigEncodingASCII __odr_asan_gen_OnigEncodingUS_ASCII __odr_asan_gen_OnigEncodingUTF_8 232 un-prefixed symbols leaked ``` To reproduce: ``` shell git checkout master mkdir build && cd build CXX=clang++ CC=clang ../configure cppflags="-fsanitize=address -fno-omit-frame-pointer" optflags=-O0 LDFLAGS="-fsanitize=address -fno-omit-frame-pointer" ASAN_OPTIONS=use_sigaltstack=0:detect_leaks=0:abort_on_error=1 make ASAN_OPTIONS=use_sigaltstack=0:detect_leaks=0:abort_on_error=1 make test-basic ``` To fix: ``` diff --git a/tool/leaked-globals b/tool/leaked-globals index ee75f78d1d..9e78228274 100755 --- a/tool/leaked-globals +++ b/tool/leaked-globals @@ -79,6 +79,7 @@ next unless n.sub!(/^#{SYMBOL_PREFIX}/o, "") next if n.include?(".") next if !so and n.start_with?("___asan_") + next if n.start_with?("__odr_asan") case n when /\A(?:Init_|InitVM_|pm_|[Oo]nig|dln_|coroutine_)/ next ``` I'm not overly familiar with Ruby, so this might not be the preferred approach, but it made the issue go away for me. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116466] [Ruby master Bug#20220] M:N threading needs to tell ASAN about stack switches
by kjtsanaktsidis (KJ Tsanaktsidis) 28 Jan '24

28 Jan '24

Issue #20220 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20220: M:N threading needs to tell ASAN about stack switches https://bugs.ruby-lang.org/issues/20220 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When switching stacks via `coroutine_transfer` in the M:N threading information, we need to tell ASAN about it in the same way that we do in cont.c. This is a matter of calling `__sanitizer_start_switch_fiber` in the transferring M:N thread, and then calling `__sanitizer_finish_switch_fiber` in the transferred-to M:N thread. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116463] [Ruby master Bug#20219] Segfault with circular parameter
by kddnewton (Kevin Newton) 27 Jan '24

27 Jan '24

Issue #20219 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20219: Segfault with circular parameter https://bugs.ruby-lang.org/issues/20219 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby def foo(bar: (1 in ^bar)); end ``` Segfaults on master. (I don't think anyone would ever write this code.) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116458] [Ruby master Bug#20216] Circular parameter reference not checked for assignment
by kddnewton (Kevin Newton) 27 Jan '24

27 Jan '24

Issue #20216 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20216: Circular parameter reference not checked for assignment https://bugs.ruby-lang.org/issues/20216 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I wanted to check if this was a bug or desired. Usually when you reference a parameter within its own default value (optional positional or optional keyword) it raises a syntax error. This is unless you use it in an assignment, in which case it is not checked. Is this desired? Here's an example: ```ruby def foo(bar = bar = 1); end def foo(bar: bar = 1); end ``` I'm not sure what this code is supposed to be doing. I expected it to raise a syntax error. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116459] [Ruby master Bug#20217] void value not checked for begin/ensure/rescue
by kddnewton (Kevin Newton) 26 Jan '24

26 Jan '24

Issue #20217 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20217: void value not checked for begin/ensure/rescue https://bugs.ruby-lang.org/issues/20217 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby foo = begin return ensure end ``` I would expect this to raise a syntax error for a void value expression being assigned, but the `ensure` makes it so that this isn't checked. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116438] [Ruby master Feature#20211] Reconsider introducing 3.2-style support of Anonymous Args/Blocks
by johnnyshields (Johnny Shields) 25 Jan '24

25 Jan '24

Issue #20211 has been reported by johnnyshields (Johnny Shields). ---------------------------------------- Feature #20211: Reconsider introducing 3.2-style support of Anonymous Args/Blocks https://bugs.ruby-lang.org/issues/20211 * Author: johnnyshields (Johnny Shields) * Status: Open * Priority: Normal ---------------------------------------- (My sincere apologies if this issue has already been raised, I didn't see it in Ruby 3.4 backlog.) As per https://bugs.ruby-lang.org/issues/19370, Ruby 3.3 no longer allows the following: ```ruby def m(*) [1, 2, 3].each { |*| p(*) } end m('test', 'test', 'test') #=> Ruby 3.2: p(*) uses args from m(*) #=> test, test, test #=> Ruby 3.3: raises SyntaxError: anonymous rest parameter is also used within block ``` I am concerned because Rubocop auto-corrects Ruby 3.1+ code to syntax which is no longer supported in Ruby 3.3. There may be quite a bit of Ruby code in the wild which breaks on 3.3 (my own app broke in about 50 places). - [Ruby 3.1+: Naming/BlockForwarding (default true)](https://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Naming/BlockForwarding) - [Ruby 3.2+: Style/ArgumentsForwarding with UseAnonymousForwarding (default true)](https://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Style/ArgumentsForwarding) It would be nice if the Ruby 3.4 syntax could allow the old nested syntax, supporting arg shadowing with different behavior between implicit and explicit |*| usage. ```ruby def m(*) [1, 2, 3].each { p(*) } end m('test', 'test', 'test') #=> Ruby 3.2: p(*) uses args from m(*) #=> test, test, test #=> Ruby 3.4: p(*) uses args from m(*) #=> test, test, test def m(*) [1, 2, 3].each { |*| p(*) } end m('test', 'test', 'test') #=> Ruby 3.2: p(*) uses args from m(*) #=> test, test, test #=> Ruby 3.4: p(*) uses args from each #=> 1, 2, 3 ``` The same would also apply to anonymous blocks. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115749] [Ruby master Feature#20066] Reduce Implicit Array/Hash Allocations For Method Calls Involving Splats
by jeremyevans0 (Jeremy Evans) 25 Jan '24

25 Jan '24

Issue #20066 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Feature #20066: Reduce Implicit Array/Hash Allocations For Method Calls Involving Splats https://bugs.ruby-lang.org/issues/20066 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Priority: Normal ---------------------------------------- I have submitted a pull request (https://github.com/ruby/ruby/pull/9247) to reduce implicit array and hash allocations for method calls involving splats. The following optimizations are included: VM_CALL_ARGS_SPLAT_MUT callinfo flag This is similar to the VM_CALL_KW_SPLAT_MUT flag added in Ruby 3.0. This makes it so if the caller-side allocates an array for the method call, the flag is used to signal to the callee that it can reuse the allocated array and does not need to duplicate it. concattoarray VM instruction This instruction is similar to concatarray, but assumes the object being concatenated to is already a mutable array (such as those created by the splatarray VM instruction). This optimizes method calls with multiple splats such as `f(*a,*a,*a)` (which previously allocated 3 arrays), allocating a single array instead of an array per splatted array. pushtoarray VM instruction This is similar, but handles non-splat arguments after a splat. Previously, the VM would wrap those arguments in an array using newarray, and then call concatarray, such that `f(*a, a)` allocated 3 arrays caller-side. This instruction just appends to the mutable array, reducing the number of arrays allocated to 1. Allocationless Anonymous Splat Forwarding This allows `def f(*, **) end` to not allocate an array or hash callee side. This works because it is not possible to mutate the local variables, only pass them as splats to other methods. This can make the following call chain allocation less: ```ruby def f(a, b: 1) end def g(*, **) f(*, **) end ea a = [1] kw = {b: 2} g(*a, **kw) # No allocations in this call ``` Switch ... argument forwards to not use ruby2_keywords Using ruby2_keywords has probably been slower since Koichi's changes early in the Ruby 3.3 development cycle to not combine keyword splats into the positional splat array. This removes the FORWARD_ARGS_WITH_RUBY2_KEYWORDS define, so that `def f(...) end` operates similarly to `def f(*, **) end`, allowing allocationless splat forwarding Reduce array and hash allocations for nested argument forwarding calls This uses a combination of frame flags and callinfo flags to track mutability of anonymous splat variables. It can make it so the following call example only allocates a 1 array and 1 hash: ```ruby def m1(*args, **kw) end def m2(...) m1(...) end def m3(*, **) m2(*, **) end m3(1, a: 1) # 1 array and 1 hash allocated ``` In the above example, the call to `m3` allocates an array (`[1]`) and a hash (`{a: 1}`), but the call to `m2` passes them as mutable splats, `m2` treats them as mutable splats when calling `m1`, and `m1` reuses the array that `m3` allocated for `args` and the hash that `m3` allocated for `kw`. I created a benchmark for all of these changes. In the method calls optimized by these changes, it is significantly faster: ``` named_multi_arg_splat after: 5344097.6 i/s before: 3088134.0 i/s - 1.73x slower named_post_splat after: 5401882.3 i/s before: 2629321.8 i/s - 2.05x slower anon_arg_splat after: 12242780.9 i/s before: 6845413.2 i/s - 1.79x slower anon_arg_kw_splat after: 11277398.7 i/s before: 4329509.4 i/s - 2.60x slower anon_multi_arg_splat after: 5132699.5 i/s before: 3018103.7 i/s - 1.70x slower anon_post_splat after: 5602915.1 i/s before: 2645185.5 i/s - 2.12x slower anon_kw_splat after: 15403727.3 i/s before: 6249504.6 i/s - 2.46x slower anon_fw_to_named_splat after: 2985715.3 i/s before: 2049159.9 i/s - 1.46x slower anon_fw_to_named_no_splat after: 2941030.4 i/s before: 2100380.0 i/s - 1.40x slower fw_to_named_splat after: 2801008.7 i/s before: 2012416.4 i/s - 1.39x slower fw_to_named_no_splat after: 2742670.4 i/s before: 1957707.2 i/s - 1.40x slower fw_to_anon_to_named_splat after: 2309246.6 i/s before: 1375924.6 i/s - 1.68x slower fw_to_anon_to_named_no_splat after: 2193227.6 i/s before: 1351184.1 i/s - 1.62x slower ``` Only fallout from these changes: * Minor change to AST output for `...` not using `ruby2_keywords` * Prism and rbs need updating for `...` not using `ruby2_keywords` * typeprof need updating for new VM instructions (at least pushtoarray) VM_CALL_ARGS_SPLAT_MUT, concattoarray, and pushtoarray only affect uncommon callsites (multiple splats, argument after splat). Other commits only optimize calls to methods using anonymous splats or `...` argument forwarding. Previously, there was no performance reason to use anonymous splats or `...` argument forwarding, but with this change, using them can be faster, and can offer a new way for users to optimize their code. In my opinion, this is too late for consideration in Ruby 3.3, but it could be considered for Ruby 3.4. -- https://bugs.ruby-lang.org/

4 18

[ruby-core:116278] [Ruby master Misc#20191] Deprecate magic encoding comment
by kddnewton (Kevin Newton) 24 Jan '24

24 Jan '24

Issue #20191 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20191: Deprecate magic encoding comment https://bugs.ruby-lang.org/issues/20191 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- I would like to ask that we deprecate the magic encoding comment, and instead require all source files to be encoded in UTF-8. There would be many benefits to the performance of both the parser and compiler. It would also help to simplify both. For example, right now a string literal in a file encoded in US-ASCII can result in 3 different encodings, depending on its internal bytes. If the file is encoded in UTF-8, it can only be a UTF-8 string. The encoding comment itself is not very commonly used in gems. If you take the top 100 most downloaded gem versions from rubygems.org and look at the resolved encoding of all of the files, you get: - UTF-8: 11554 - ASCII-8BIT: 35 - US-ASCII: 10 For all of the most recent versions of gems on rubygems.org, you get: - UTF-8: 2967421 - US-ASCII: 20130 - ASCII-8BIT: 9237 - ISO-8859-1: 87 - Windows-1252: 45 - Shift_JIS: 32 - Windows-31J: 22 - Windows-1251: 15 - EUC-JP: 11 - GBK: 4 - KOI8-R: 3 - ISO-8859-15: 2 - UTF8-MAC: 1 - invalid: 33 Note that "invalid" here could have worked on some rubies < 3.2 if they used Encoding#replicate. If we were to change this, the main breaking change concern would be the encoding of strings and symbols that would leave the context of the file by virtue of a constant read/method call. That's why I think it should first be deprecated in a minor release, then removed in the next major. At the moment this would mean for the top 100 gems we would be worried about 0.39% of files, and on rubygems.org as a whole we would be worried about 0.99% of files. If deprecating the entire encoding comment is unacceptable from a compatibility point of view, I would suggest we try only allowing UTF-8, US-ASCII, and ASCII-8BIT. This would still have a lot of value/simplifications/performance opportunities, at the expense of still needing to be parsed and checked. On the top 100 gems this would mean no files would have to change, and on rubygems.org as a whole it would mean we would be worried about 0.009% of files. That being said, if we're going to deprecate this at all, we should probably just do it all the way to get the full benefit. (In case you want to check the math, the script used to calculate these is attached.) ---Files-------------------------------- gems.rb (4.33 KB) -- https://bugs.ruby-lang.org/

6 6

[ruby-core:116413] [Ruby master Feature#16049] optimization for frozen dynamic string literals "#{exp}".dup and +"#{exp}"
by Dan0042 (Daniel DeLorme) 24 Jan '24

24 Jan '24

Issue #16049 has been updated by Dan0042 (Daniel DeLorme). Dynamic string literals are not longer frozen, this can be closed. (Would it be possible for the author to close their own ticket?) ---------------------------------------- Feature #16049: optimization for frozen dynamic string literals "#{exp}".dup and +"#{exp}" https://bugs.ruby-lang.org/issues/16049#change-106435 * Author: Dan0042 (Daniel DeLorme) * Status: Open * Priority: Normal ---------------------------------------- When the decision was made that `frozen_string_literal: true` should also apply to dynamic string literals, it was mitigated with the following explanation: > "#{exp}".dup can be optimized so it won’t allocate extra objects like "...".freeze https://docs.google.com/document/u/1/d/1D0Eo5N7NE_unIySOKG9lVj_eyXf66BQPM4P… However that does not appear to be the case currently. Using this script that generates 100k String objects: ```ruby # frozen_string_literal: true def allocated GC.stat[:total_allocated_objects] end GC.disable c = ARGV.shift.to_sym x_eq_i = ARGV.shift=="i" x = "x" before = allocated 100_000.times do |i| x = i.to_s if x_eq_i case c when :normal then v = "#{x}" when :freeze then v = "#{x}".freeze when :minus then v = -"#{x}" when :dup then v = "#{x}".dup when :plus then v = +"#{x}" else raise end end after = allocated printf "%d\n", after-before ``` I get the following number of objects allocated ``` x= frozen_string_literal normal freeze minus dup plus 'x' false 100001 100001 100001 200001 100001 'x' true 100001 100001 100001 200001 200001 i false 200001 200001 299999 300001 200001 i true 200001 200001 200001 300001 300001 ``` We can see that `"#{x}".dup` and `+"#{x}"` allocate an extra object per iteration I also tested with `x = i.to_s` to see if deduplication of 100k identical strings was different from 100k different strings. In addition to the expected extra strings created by `i.to_s`, there's an additional 100k extra strings created for `-"#{i}"` when frozen_string_literal is false??? There may also be a memory leak here because while the number of objects increases by x3, memory usage increases by x4. Summary: I expected `"#{v}".dup` and `+"#{v}"` to behave the same regardless of frozen_string_literal (and optimize down to just one allocation) I expected `"#{v}".freeze` and `-"#{v}"` to behave the same regardless of frozen_string_literal (and optimize down to just one allocation) but they do not. I think they should. It would be nice. -- https://bugs.ruby-lang.org/

1 0