- ruby-core - ml.ruby-lang.org

[ruby-core:116345] [Ruby master Feature#20196] Proposal: Binary data literal
by ziggythehamster (Keith Gable) 29 Jan '24

29 Jan '24

Issue #20196 has been reported by ziggythehamster (Keith Gable). ---------------------------------------- Feature #20196: Proposal: Binary data literal https://bugs.ruby-lang.org/issues/20196 * Author: ziggythehamster (Keith Gable) * Status: Open * Priority: Normal ---------------------------------------- I sometimes find myself needing to write some bytes in a Ruby string literal, and this experience leaves a lot to be desired: * Bare strings don't work (potential for encoding corruption unless you remember to `.force_encoding` and never copy-paste just the literal into somewhere else) and are not particularly pleasant given all of the backslashes * Wrapping this in `String.new("\x89PNG\r\n\x1A\n\x00\x00\x00\rIHDR\x00\x00\x00`\x00\x00\x00`\b\x06\x00\x00\x00\xE2\x98w8\x00\x000%IDAT", encoding: 'BINARY')` is better, but many tools explode with this because they expect all strings to be valid UTF-8 even if they're an argument to String.new, and it still doesn't have the "beauty" one might expect from Ruby (also it's not frozen unless you also freeze it) * `["9805e474d0a0a1a0000000d094844425000000060000000680600000002e8977830000035294441445"].pack("h*")` parses in all tools and is less harsh to look at, but if you're writing binary data, you probably want to annotate it Here's my basic syntax proposal: ``` %b[ 89504e470d0a1a0a # PNG header 0000000d # Length = 13 bytes 49484452 # IHDR chunk 00000060 # Width = 96px 00000060 # Height = 96px 08 06 # 8bpp RGBA 00 00 00 # deflate / no filter / non-interlaced ] # => "\x89PNG\r\n\x1A\n\x00\x00\x00\rIHDR\x00\x00\x00`\x00\x00\x00`\b\x06\x00\x00\x00" ``` More formally: * To match the nibble ordering of a regular string escape, the hex characters are high nibble first (the same as the `H` unpack character). * It follows the same rules as other percent literals, and I am flexible on what character is used. I chose `b` because `h` could be confusing paired with the `h`/`H` unpack characters and the inverted meaning. * We could say that high-nibble-first is capitalized and the lower-case version is low-nibble-first, but I imagine most people will want high-nibble-first. We could also say that `%b[]` returns a `String` but `%B[]` returns an `IO::Buffer`, which has greater utility than having the capability of writing low-nibble-first literals * Whitespace is ignored * Comments are allowed * The encoding is always `Encoding::BINARY` * The resulting string is always frozen (and if `%B[]` means buffer then that is read-only as well) * a-f can also be written A-F Things to consider: * Interpolation could be allowed like normal strings * Embedding strings could be allowed (example below) * `?` literals (characters) should be handled identically to how other kinds of strings are embedded if that is allowed * If interpolation is allowed and you interpolate a number, this should either interpolate `.to_s` as you would expect in a string or raise an error, because there is no unsurprising way to take a number and convert it to one or more bytes * Strings encoded as `Encoding::BINARY` could have their `.inspect` output use this literal * When dealing with bitmasks, it's often convenient to write them out in binary instead of hex so the on bits are easier to identify, but there is no syntax for that here that I am fond of... but someone might have an idea. I thought about `.00001111` or `!00001111` with mandatory whitespace before resuming hex characters, but those didn't feel right to me Example with embedded strings: ``` %b[ 89 "PNG" 0d0a1a0a # PNG header 0000000d # Length = 13 bytes "IHDR" # IHDR chunk 00000060 # Width = 96px 00000060 # Height = 96px 08 06 # 8bpp RGBA 00 00 00 # deflate / no filter / non-interlaced ] ``` Example with interpolation: ``` %b[ #{png_header} #{ihdr = chunk(:ihdr, width: 96, height: 96, bpp: 8, format: :rgba)} #{png_crc(ihdr)} # I didn't include this in the other examples but I needed something to demonstrate here ] ``` Other possible alternatives: * A library (possibly standard library/gem) could have a function like `binary` take a string (potentially a heredoc) and parse it according to the same rules I wrote above. You would have to make the parser strip whitespace and comments, and only hex bytes could be interpolated. * A new pack/unpack symbol could be created that does the same thing as above, so you could `["hex #comments\netc"].pack("...")` * You could probably do a lot of this with an array of hex strings and `pack` but it doesn't allow for freeform whitespace and the way you do it is not obvious without reading the docs for `pack` ... and also you allocate a bunch of strings you don't need * A `Data`-like object more closely related to `IO::Buffer` could be defined that declares the size of things contained within a buffer and then a constant could be written to create an instance of the Data-subclass containing the actual data you want to write out ... but this is a lot of work Potential users: * People writing protocol code in Ruby * People who need to write out magic constants (in my case: the RDB encoding of a Redis value) * People using something like Metasploit Framework to reverse engineer something * Tools could e.g. disassemble x86 into a literal with comments showing the assembly mnemonics -- https://bugs.ruby-lang.org/

5 6

[ruby-core:116483] [Ruby master Bug#20223] For simple objects each_cons appears to work as I expect, but for other objects it seems to fail to terminate iterations when I would expect.
by mjflynt (Jeffrey Flynt) 29 Jan '24

29 Jan '24

Issue #20223 has been reported by mjflynt (Jeffrey Flynt). ---------------------------------------- Bug #20223: For simple objects each_cons appears to work as I expect, but for other objects it seems to fail to terminate iterations when I would expect. https://bugs.ruby-lang.org/issues/20223 * Author: mjflynt (Jeffrey Flynt) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Reproduce process: ``` ruby p `ruby -v` class X include Enumerable attr_accessor :obj def initialize = @obj = (1..5).to_a def each(&block) = yield @obj.each(&block) end wtf = X.new (1..5).each_cons(3) { |g| p g } p '~' * 20 (1..5).to_a.each_cons(3) { |g| p g } p '~' * 20 ('a'..'e').each_cons(3) { |g| p g } p '~' * 20 wtf.each_cons(3) { |g| p g } ``` Result of reproduce process ``` "ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [aarch64-linux]\n" [1, 2, 3] [2, 3, 4] [3, 4, 5] "~~~~~~~~~~~~~~~~~~~~" [1, 2, 3] [2, 3, 4] [3, 4, 5] "~~~~~~~~~~~~~~~~~~~~" ["a", "b", "c"] ["b", "c", "d"] ["c", "d", "e"] "~~~~~~~~~~~~~~~~~~~~" [1, 2, 3] [2, 3, 4] [3, 4, 5] [4, 5, [1, 2, 3, 4, 5]] [Done] exited with code=0 in 0.173 seconds ``` Expected result and the reason why you expect I expect the results to not include the last line of output "[4, 5, [1, 2, 3, 4, 5]]" because my understanding is each_cons ends its iteration over the collection before it runs out of elements. In this case, this fourth row does not have enough elements to do three consecutive elements, but it is doing so anyway, appearing to use the original array as the third element before it completes. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116473] [Ruby master Bug#20221] ASAN: make test-basic: un-prefixed symbol leakage
by leeN (David Klein) 29 Jan '24

29 Jan '24

Issue #20221 has been reported by leeN (David Klein). ---------------------------------------- Bug #20221: ASAN: make test-basic: un-prefixed symbol leakage https://bugs.ruby-lang.org/issues/20221 * Author: leeN (David Klein) * Status: Open * Priority: Normal * ruby -v: ruby 3.4.0dev (2024-01-29T08:16:49Z master 8bff7e996c) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When building and running the tests (here, test-basic) with ASAN enabled, it fails with the following message: ``` Checking leaked global symbols...leaked __odr_asan_gen_rb_cArray __odr_asan_gen_ruby_digitmap __odr_asan_gen_rb_mComparable ...snip... __odr_asan_gen_OnigEncodingASCII __odr_asan_gen_OnigEncodingUS_ASCII __odr_asan_gen_OnigEncodingUTF_8 232 un-prefixed symbols leaked ``` To reproduce: ``` shell git checkout master mkdir build && cd build CXX=clang++ CC=clang ../configure cppflags="-fsanitize=address -fno-omit-frame-pointer" optflags=-O0 LDFLAGS="-fsanitize=address -fno-omit-frame-pointer" ASAN_OPTIONS=use_sigaltstack=0:detect_leaks=0:abort_on_error=1 make ASAN_OPTIONS=use_sigaltstack=0:detect_leaks=0:abort_on_error=1 make test-basic ``` To fix: ``` diff --git a/tool/leaked-globals b/tool/leaked-globals index ee75f78d1d..9e78228274 100755 --- a/tool/leaked-globals +++ b/tool/leaked-globals @@ -79,6 +79,7 @@ next unless n.sub!(/^#{SYMBOL_PREFIX}/o, "") next if n.include?(".") next if !so and n.start_with?("___asan_") + next if n.start_with?("__odr_asan") case n when /\A(?:Init_|InitVM_|pm_|[Oo]nig|dln_|coroutine_)/ next ``` I'm not overly familiar with Ruby, so this might not be the preferred approach, but it made the issue go away for me. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116466] [Ruby master Bug#20220] M:N threading needs to tell ASAN about stack switches
by kjtsanaktsidis (KJ Tsanaktsidis) 28 Jan '24

28 Jan '24

Issue #20220 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20220: M:N threading needs to tell ASAN about stack switches https://bugs.ruby-lang.org/issues/20220 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Assignee: kjtsanaktsidis (KJ Tsanaktsidis) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- When switching stacks via `coroutine_transfer` in the M:N threading information, we need to tell ASAN about it in the same way that we do in cont.c. This is a matter of calling `__sanitizer_start_switch_fiber` in the transferring M:N thread, and then calling `__sanitizer_finish_switch_fiber` in the transferred-to M:N thread. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116463] [Ruby master Bug#20219] Segfault with circular parameter
by kddnewton (Kevin Newton) 27 Jan '24

27 Jan '24

Issue #20219 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20219: Segfault with circular parameter https://bugs.ruby-lang.org/issues/20219 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby def foo(bar: (1 in ^bar)); end ``` Segfaults on master. (I don't think anyone would ever write this code.) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116458] [Ruby master Bug#20216] Circular parameter reference not checked for assignment
by kddnewton (Kevin Newton) 27 Jan '24

27 Jan '24

Issue #20216 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20216: Circular parameter reference not checked for assignment https://bugs.ruby-lang.org/issues/20216 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I wanted to check if this was a bug or desired. Usually when you reference a parameter within its own default value (optional positional or optional keyword) it raises a syntax error. This is unless you use it in an assignment, in which case it is not checked. Is this desired? Here's an example: ```ruby def foo(bar = bar = 1); end def foo(bar: bar = 1); end ``` I'm not sure what this code is supposed to be doing. I expected it to raise a syntax error. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:116459] [Ruby master Bug#20217] void value not checked for begin/ensure/rescue
by kddnewton (Kevin Newton) 26 Jan '24

26 Jan '24

Issue #20217 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20217: void value not checked for begin/ensure/rescue https://bugs.ruby-lang.org/issues/20217 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby foo = begin return ensure end ``` I would expect this to raise a syntax error for a void value expression being assigned, but the `ensure` makes it so that this isn't checked. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:116438] [Ruby master Feature#20211] Reconsider introducing 3.2-style support of Anonymous Args/Blocks
by johnnyshields (Johnny Shields) 25 Jan '24

25 Jan '24

Issue #20211 has been reported by johnnyshields (Johnny Shields). ---------------------------------------- Feature #20211: Reconsider introducing 3.2-style support of Anonymous Args/Blocks https://bugs.ruby-lang.org/issues/20211 * Author: johnnyshields (Johnny Shields) * Status: Open * Priority: Normal ---------------------------------------- (My sincere apologies if this issue has already been raised, I didn't see it in Ruby 3.4 backlog.) As per https://bugs.ruby-lang.org/issues/19370, Ruby 3.3 no longer allows the following: ```ruby def m(*) [1, 2, 3].each { |*| p(*) } end m('test', 'test', 'test') #=> Ruby 3.2: p(*) uses args from m(*) #=> test, test, test #=> Ruby 3.3: raises SyntaxError: anonymous rest parameter is also used within block ``` I am concerned because Rubocop auto-corrects Ruby 3.1+ code to syntax which is no longer supported in Ruby 3.3. There may be quite a bit of Ruby code in the wild which breaks on 3.3 (my own app broke in about 50 places). - [Ruby 3.1+: Naming/BlockForwarding (default true)](https://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Naming/BlockForwarding) - [Ruby 3.2+: Style/ArgumentsForwarding with UseAnonymousForwarding (default true)](https://www.rubydoc.info/gems/rubocop/RuboCop/Cop/Style/ArgumentsForwarding) It would be nice if the Ruby 3.4 syntax could allow the old nested syntax, supporting arg shadowing with different behavior between implicit and explicit |*| usage. ```ruby def m(*) [1, 2, 3].each { p(*) } end m('test', 'test', 'test') #=> Ruby 3.2: p(*) uses args from m(*) #=> test, test, test #=> Ruby 3.4: p(*) uses args from m(*) #=> test, test, test def m(*) [1, 2, 3].each { |*| p(*) } end m('test', 'test', 'test') #=> Ruby 3.2: p(*) uses args from m(*) #=> test, test, test #=> Ruby 3.4: p(*) uses args from each #=> 1, 2, 3 ``` The same would also apply to anonymous blocks. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115749] [Ruby master Feature#20066] Reduce Implicit Array/Hash Allocations For Method Calls Involving Splats
by jeremyevans0 (Jeremy Evans) 25 Jan '24

25 Jan '24

Issue #20066 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Feature #20066: Reduce Implicit Array/Hash Allocations For Method Calls Involving Splats https://bugs.ruby-lang.org/issues/20066 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Priority: Normal ---------------------------------------- I have submitted a pull request (https://github.com/ruby/ruby/pull/9247) to reduce implicit array and hash allocations for method calls involving splats. The following optimizations are included: VM_CALL_ARGS_SPLAT_MUT callinfo flag This is similar to the VM_CALL_KW_SPLAT_MUT flag added in Ruby 3.0. This makes it so if the caller-side allocates an array for the method call, the flag is used to signal to the callee that it can reuse the allocated array and does not need to duplicate it. concattoarray VM instruction This instruction is similar to concatarray, but assumes the object being concatenated to is already a mutable array (such as those created by the splatarray VM instruction). This optimizes method calls with multiple splats such as `f(*a,*a,*a)` (which previously allocated 3 arrays), allocating a single array instead of an array per splatted array. pushtoarray VM instruction This is similar, but handles non-splat arguments after a splat. Previously, the VM would wrap those arguments in an array using newarray, and then call concatarray, such that `f(*a, a)` allocated 3 arrays caller-side. This instruction just appends to the mutable array, reducing the number of arrays allocated to 1. Allocationless Anonymous Splat Forwarding This allows `def f(*, **) end` to not allocate an array or hash callee side. This works because it is not possible to mutate the local variables, only pass them as splats to other methods. This can make the following call chain allocation less: ```ruby def f(a, b: 1) end def g(*, **) f(*, **) end ea a = [1] kw = {b: 2} g(*a, **kw) # No allocations in this call ``` Switch ... argument forwards to not use ruby2_keywords Using ruby2_keywords has probably been slower since Koichi's changes early in the Ruby 3.3 development cycle to not combine keyword splats into the positional splat array. This removes the FORWARD_ARGS_WITH_RUBY2_KEYWORDS define, so that `def f(...) end` operates similarly to `def f(*, **) end`, allowing allocationless splat forwarding Reduce array and hash allocations for nested argument forwarding calls This uses a combination of frame flags and callinfo flags to track mutability of anonymous splat variables. It can make it so the following call example only allocates a 1 array and 1 hash: ```ruby def m1(*args, **kw) end def m2(...) m1(...) end def m3(*, **) m2(*, **) end m3(1, a: 1) # 1 array and 1 hash allocated ``` In the above example, the call to `m3` allocates an array (`[1]`) and a hash (`{a: 1}`), but the call to `m2` passes them as mutable splats, `m2` treats them as mutable splats when calling `m1`, and `m1` reuses the array that `m3` allocated for `args` and the hash that `m3` allocated for `kw`. I created a benchmark for all of these changes. In the method calls optimized by these changes, it is significantly faster: ``` named_multi_arg_splat after: 5344097.6 i/s before: 3088134.0 i/s - 1.73x slower named_post_splat after: 5401882.3 i/s before: 2629321.8 i/s - 2.05x slower anon_arg_splat after: 12242780.9 i/s before: 6845413.2 i/s - 1.79x slower anon_arg_kw_splat after: 11277398.7 i/s before: 4329509.4 i/s - 2.60x slower anon_multi_arg_splat after: 5132699.5 i/s before: 3018103.7 i/s - 1.70x slower anon_post_splat after: 5602915.1 i/s before: 2645185.5 i/s - 2.12x slower anon_kw_splat after: 15403727.3 i/s before: 6249504.6 i/s - 2.46x slower anon_fw_to_named_splat after: 2985715.3 i/s before: 2049159.9 i/s - 1.46x slower anon_fw_to_named_no_splat after: 2941030.4 i/s before: 2100380.0 i/s - 1.40x slower fw_to_named_splat after: 2801008.7 i/s before: 2012416.4 i/s - 1.39x slower fw_to_named_no_splat after: 2742670.4 i/s before: 1957707.2 i/s - 1.40x slower fw_to_anon_to_named_splat after: 2309246.6 i/s before: 1375924.6 i/s - 1.68x slower fw_to_anon_to_named_no_splat after: 2193227.6 i/s before: 1351184.1 i/s - 1.62x slower ``` Only fallout from these changes: * Minor change to AST output for `...` not using `ruby2_keywords` * Prism and rbs need updating for `...` not using `ruby2_keywords` * typeprof need updating for new VM instructions (at least pushtoarray) VM_CALL_ARGS_SPLAT_MUT, concattoarray, and pushtoarray only affect uncommon callsites (multiple splats, argument after splat). Other commits only optimize calls to methods using anonymous splats or `...` argument forwarding. Previously, there was no performance reason to use anonymous splats or `...` argument forwarding, but with this change, using them can be faster, and can offer a new way for users to optimize their code. In my opinion, this is too late for consideration in Ruby 3.3, but it could be considered for Ruby 3.4. -- https://bugs.ruby-lang.org/

4 18

[ruby-core:116278] [Ruby master Misc#20191] Deprecate magic encoding comment
by kddnewton (Kevin Newton) 24 Jan '24

24 Jan '24

Issue #20191 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20191: Deprecate magic encoding comment https://bugs.ruby-lang.org/issues/20191 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- I would like to ask that we deprecate the magic encoding comment, and instead require all source files to be encoded in UTF-8. There would be many benefits to the performance of both the parser and compiler. It would also help to simplify both. For example, right now a string literal in a file encoded in US-ASCII can result in 3 different encodings, depending on its internal bytes. If the file is encoded in UTF-8, it can only be a UTF-8 string. The encoding comment itself is not very commonly used in gems. If you take the top 100 most downloaded gem versions from rubygems.org and look at the resolved encoding of all of the files, you get: - UTF-8: 11554 - ASCII-8BIT: 35 - US-ASCII: 10 For all of the most recent versions of gems on rubygems.org, you get: - UTF-8: 2967421 - US-ASCII: 20130 - ASCII-8BIT: 9237 - ISO-8859-1: 87 - Windows-1252: 45 - Shift_JIS: 32 - Windows-31J: 22 - Windows-1251: 15 - EUC-JP: 11 - GBK: 4 - KOI8-R: 3 - ISO-8859-15: 2 - UTF8-MAC: 1 - invalid: 33 Note that "invalid" here could have worked on some rubies < 3.2 if they used Encoding#replicate. If we were to change this, the main breaking change concern would be the encoding of strings and symbols that would leave the context of the file by virtue of a constant read/method call. That's why I think it should first be deprecated in a minor release, then removed in the next major. At the moment this would mean for the top 100 gems we would be worried about 0.39% of files, and on rubygems.org as a whole we would be worried about 0.99% of files. If deprecating the entire encoding comment is unacceptable from a compatibility point of view, I would suggest we try only allowing UTF-8, US-ASCII, and ASCII-8BIT. This would still have a lot of value/simplifications/performance opportunities, at the expense of still needing to be parsed and checked. On the top 100 gems this would mean no files would have to change, and on rubygems.org as a whole it would mean we would be worried about 0.009% of files. That being said, if we're going to deprecate this at all, we should probably just do it all the way to get the full benefit. (In case you want to check the math, the script used to calculate these is attached.) ---Files-------------------------------- gems.rb (4.33 KB) -- https://bugs.ruby-lang.org/

6 6