December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115604] [Ruby master Feature#18576] Rename `ASCII-8BIT` encoding to `BINARY`
by Eregon (Benoit Daloze) 06 Dec '23

06 Dec '23

Issue #18576 has been updated by Eregon (Benoit Daloze). Target version set to 3.4 @matz Could we try this again for 3.4, soon after the 3.3 release? Then there is plenty of time to discover any issue related to it (probably very few as gems have been patched, and applications using encoding names instead of encoding constants are likely very old and unlikely to use a recent Ruby). ---------------------------------------- Feature #18576: Rename `ASCII-8BIT` encoding to `BINARY` https://bugs.ruby-lang.org/issues/18576#change-105535 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Target version: 3.4 ---------------------------------------- ### Context I'm now used to it, but something that confused me for years was errors such as: ```ruby >> "fée" + "\xFF".b (irb):3:in `+': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError) ``` When you aren't that familiar with Ruby, it's really not evident that `ASCII-8BIT` basically means "no encoding" or "binary". And even when you know it, if you don't read carefully it's very easily confused with `US-ASCII`. The `Encoding::BINARY` alias is much more telling IMHO. ### Proposal Since `Encoding::ASCII_8BIT` has been aliased as `Encoding::BINARY` for years, I think renaming it to `BINARY` and then making asking `ASCII_8BIT` the alias would significantly improve usability without backward compatibility concerns. The only concern I could see would be the consistency with a handful of C API functions: - `rb_encoding *rb_ascii8bit_encoding(void)` - `int rb_ascii8bit_encindex(void)` - `VALUE rb_io_ascii8bit_binmode(VALUE io)` But that's for much more advanced users, so I don't think it's much of a concern. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:114785] [Ruby master Bug#19886] "default->bundled gem" warning is not shown under "bundle exec"
by mame (Yusuke Endoh) 06 Dec '23

06 Dec '23

Issue #19886 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #19886: "default->bundled gem" warning is not shown under "bundle exec" https://bugs.ruby-lang.org/issues/19886 * Author: mame (Yusuke Endoh) * Status: Assigned * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- ``` $ cat Gemfile source "https://rubygems.org" $ cat test.rb require "base64" $ bundle exec ruby test.rb $ ``` In this situation, `bundle exec ruby test.rb` should print a warning: base64 which will be not part of the default gems since Ruby 3.4.0. Note that `ruby test.rb` shows the warning, which is not needed (see #19885). ``` $ ruby test.rb test.rb:1: warning: base64 which will be not part of the default gems since Ruby 3.4.0 ``` -- https://bugs.ruby-lang.org/

2 2

[ruby-core:115602] [Ruby master Bug#19147] `TestFileExhaustive#test_expand_path_for_existent_username` and `TestDir#test_home` fails on i686
by vo.x (Vit Ondruch) 06 Dec '23

06 Dec '23

Issue #19147 has been updated by vo.x (Vit Ondruch). Testing further, there might be issue in some other places. While I can't reproduce the issue on Fedora Rawhide, it still fails on Fedora 38. I was not expecting this. Might the culprit be e.g. glibc? Or is it FS type related? ---------------------------------------- Bug #19147: `TestFileExhaustive#test_expand_path_for_existent_username` and `TestDir#test_home` fails on i686 https://bugs.ruby-lang.org/issues/19147#change-105533 * Author: vo.x (Vit Ondruch) * Status: Closed * Priority: Normal * ruby -v: ruby 3.2.0dev (2022-11-24 master 66e5200ba4) [i386-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Testing with commit:git|66e5200ba4 on Fedora Rawhide, I observe following error just on i686 (other platforms are passing just fine): ~~~ 1) Error: TestFileExhaustive#test_expand_path_for_existent_username: RuntimeError: can't set length of shared string /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_file_exhaustive.rb:1122:in `expand_path' /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_file_exhaustive.rb:1122:in `test_expand_path_for_existent_username' 2) Error: TestDir#test_home: RuntimeError: can't set length of shared string /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_dir.rb:537:in `expand_path' /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_dir.rb:537:in `block in test_home' ~~~ Previously testing with commit:git|4b1504ae0a, the tests were passing just fine. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115596] [Ruby master Feature#20040] Make Hash respond do deconstruct to allow matchting key/value pairs
by matz (Yukihiro Matsumoto) 05 Dec '23

05 Dec '23

Issue #20040 has been updated by matz (Yukihiro Matsumoto). Theoretically possible, but could you show us a specific use case, please? Matz. ---------------------------------------- Feature #20040: Make Hash respond do deconstruct to allow matchting key/value pairs https://bugs.ruby-lang.org/issues/20040#change-105527 * Author: Anonymous * Status: Open * Priority: Normal ---------------------------------------- It would be nice to allow pattern matching to work on key/value pairs of a Hash to e.g. find the first entry, or entries where the key is not known. Example: ``` class Hash; def deconstruct = to_a; end {a: 1, b: 2} => [[k, v], *] puts "k=#{k}" # :a puts "v=#{v}" # 1 {a: 1, b: 2} => [*, [k, 2], *] puts "k=#{k}" # :b ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115592] [Ruby master Feature#20040] Make Hash respond do deconstruct to allow matchting key/value pairs
by jochen＠seeber.me (Jochen Seeber) 05 Dec '23

05 Dec '23

Issue #20040 has been reported by jochen(a)seeber.me (Jochen Seeber). ---------------------------------------- Feature #20040: Make Hash respond do deconstruct to allow matchting key/value pairs https://bugs.ruby-lang.org/issues/20040 * Author: jochen(a)seeber.me (Jochen Seeber) * Status: Open * Priority: Normal ---------------------------------------- It would be nice to allow pattern matching to work on key/value pairs of a Hash to e.g. find the first entry, or entries where the key is not known. Example: ``` class Hash; def deconstruct = to_a; end {a: 1, b: 2} => [[k, v], *] puts "k=#{k}" # :a puts "v=#{v}" # 1 {a: 1, b: 2} => [*, [k, 2], *] puts "k=#{k}" # :b ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115591] [Ruby master Bug#19147] `TestFileExhaustive#test_expand_path_for_existent_username` and `TestDir#test_home` fails on i686
by vo.x (Vit Ondruch) 05 Dec '23

05 Dec '23

Issue #19147 has been updated by vo.x (Vit Ondruch). This was already fixed at 3.2.0 release it seems. Nevertheless, I am still unsure, what was cause and fix for this. ---------------------------------------- Bug #19147: `TestFileExhaustive#test_expand_path_for_existent_username` and `TestDir#test_home` fails on i686 https://bugs.ruby-lang.org/issues/19147#change-105524 * Author: vo.x (Vit Ondruch) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.0dev (2022-11-24 master 66e5200ba4) [i386-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Testing with commit:git|66e5200ba4 on Fedora Rawhide, I observe following error just on i686 (other platforms are passing just fine): ~~~ 1) Error: TestFileExhaustive#test_expand_path_for_existent_username: RuntimeError: can't set length of shared string /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_file_exhaustive.rb:1122:in `expand_path' /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_file_exhaustive.rb:1122:in `test_expand_path_for_existent_username' 2) Error: TestDir#test_home: RuntimeError: can't set length of shared string /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_dir.rb:537:in `expand_path' /builddir/build/BUILD/ruby-3.2.0-66e5200ba4/test/ruby/test_dir.rb:537:in `block in test_home' ~~~ Previously testing with commit:git|4b1504ae0a, the tests were passing just fine. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115588] [Ruby master Bug#20039] Matching US-ASCII string to copied UTF-8 Regexp causes invalid multibyte character error
by dbrown9＠gmail.com (Dustin Brown) 04 Dec '23

04 Dec '23

Issue #20039 has been reported by dbrown9(a)gmail.com (Dustin Brown). ---------------------------------------- Bug #20039: Matching US-ASCII string to copied UTF-8 Regexp causes invalid multibyte character error https://bugs.ruby-lang.org/issues/20039 * Author: dbrown9(a)gmail.com (Dustin Brown) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-12-03 master 85bc80a) * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Matching a US-ASCII string to a UTF-8 encoded regexp with multibyte characters works as expected. ```ruby re = Regexp.new("\u2018".encode("UTF-8")) "".encode("US-ASCII").match?(re) => false ``` However, if that regexp is used to initialize a new regexp, the comparison fails with a Invalid mutibyte character error. ```ruby re = Regexp.new("\u2018".encode("UTF-8")) "".encode("US-ASCII").match?(Regexp.new(re)) => ArgumentError: regexp preprocess failed: invalid multibyte character ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115581] [Ruby master Bug#16145] regexp match error if mixing /i, character classes, and utf8
by duerst 03 Dec '23

03 Dec '23

Issue #16145 has been updated by duerst (Martin Dürst). @zenspider I said that single-character character classes get reduced to just the actual character. So that would mean that your `"CAFÉ" =~ /[a]/i` gets reduced to `"CAFÉ" =~ /a/i`, and therefore works. That of course does not prove my guess, but it also doesn't disprove it. We'd need some other examples to test this further. ---------------------------------------- Bug #16145: regexp match error if mixing /i, character classes, and utf8 https://bugs.ruby-lang.org/issues/16145#change-105514 * Author: zenspider (Ryan Davis) * Status: Open * Priority: Normal * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- (reported on behalf of mage(a)mage.gold -- there appears to be an error in registration or login): See: ruby-talk @ X-Mail-Count: 440336 2.6.3 :049 > 'SHOP' =~ /[xo]/i => 2 2.6.3 :050 > 'CAFÉ' =~ /[é]/i => 3 2.6.3 :051 > 'CAFÉ' =~ /[xé]/i => nil 2.6.3 :052 > 'CAFÉ' =~ /[xÉ]/i => 3 Expected result: 2.6.3 :051 > 'CAFÉ' =~ /[xé]/i => 3 I tested it on random regex online pages. It does not match on https://regex101.com/ It matches on: https://regexr.com/ https://www.regextester.com/ https://www.freeformatter.com/regex-tester.html (Ignore case turned on). The reason I suppose it’s more like a bug than a feature is the fact that /[é]/i matches 'CAFÉ'. If the //i didn’t work for UTF-8 characters then the /[é]/i wouldn’t match it either. For example, [é] does not match 'CAFÉ' on https://regex101.com/ I could not find a page or a system that behaves the same way as Ruby does. For example, it matches in PostgreSQL 10 (under FreeBSD 12) too: # select 'CAFÉ'~ '[xé]'; ?column? ---------- f (1 row) # select 'CAFÉ' ~* '[xé]'; ?column? ---------- t (1 row) Tested it in IRB on macOS and FreeBSD. $ uname -a && ruby -v && locale Darwin xxx 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64 ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18] LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL="en_US.UTF-8" $ uname -a && ruby -v && locale FreeBSD xxx 12.0-RELEASE-p9 FreeBSD 12.0-RELEASE-p9 GENERIC amd64 ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-freebsd12.0] LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_ALL=en_US.UTF-8 I installed Ruby with RVM. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115484] [Ruby master Bug#20022] GC.verify_compaction_references does not actually move alll objects
by kjtsanaktsidis (KJ Tsanaktsidis) 02 Dec '23

02 Dec '23

Issue #20022 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20022: GC.verify_compaction_references does not actually move alll objects https://bugs.ruby-lang.org/issues/20022 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- While debugging https://bugs.ruby-lang.org/issues/20021, I ran into a separate issue which I figured was worth resolving whilst I had it in my head. The intention of GC.verify_compaction_references is, I believe, to force every single movable object to be moved, so that it's possible to debug native extensions which not correctly updating their references to objects they mark as movable (if this is _not_ the case, you can stop reading now and accept my apologies!) To do this, it doubles the number of allocated pages for each size pool, and sorts the heap pages so that the free ones are swept first; thus, every object in an old page should be moved into a free slot in one of the new pages. This worked fine until movement of objects between size pools during compaction was implemented. That causes some problems for verify_compaction_references: - We were doubling the number of pages in each size pool, but actually if some objects need to move into a different pool, there's no guarantee that they'll be enough room in that one. - It's possible for the sweep & compact cursors to meet in one size pool before all the objects that want to move into that size pool from another are processed by the compaction. You can see these problems by changing some of the movement tests in test_gc_compact.rb to try and move e.g. 50,000 objects instead of 500 (by changing `HASH_COUNT`); the test is not able to actually move all of the objects in a single compaction run. What I implemented in this PR (https://github.com/ruby/ruby/pull/9041) is two things: - Firstly, it examines every object and determine where it wants to be compacted into; we use this information to calculate the desired number of pages to add to each size pool. - Secondly, it compacts twice; once in order of ascending size pool, and once descending. This means that we are guaranteed to be able to move everything we want to move into a size pool before we start advancing that pool's own compact cursor. With these fixes in place, I was able to make the compaction tests move any amount of objects (previously, `test_moving_hashes_down_size_pools` would begin failing on my machine if I set `HASH_COUNT` to above about 6,000. -- https://bugs.ruby-lang.org/

2 6

[ruby-core:115579] [Ruby master Bug#19144] Ruby should set AI_V4MAPPED | AI_ADDRCONFIG getaddrinfo flags by default
by akr (Akira Tanaka) 02 Dec '23

02 Dec '23

Issue #19144 has been updated by akr (Akira Tanaka). kjtsanaktsidis (KJ Tsanaktsidis) wrote in #note-5: > @akr could you take a look at my PR when you get a chance? I think I addressed your feedback, please let me know if I have misunderstood! It seems fine. I agree that we remove the test because it is too complicated. ---------------------------------------- Bug #19144: Ruby should set AI_V4MAPPED | AI_ADDRCONFIG getaddrinfo flags by default https://bugs.ruby-lang.org/issues/19144#change-105512 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Currently, DNS lookups made with `getaddrinfo` from Ruby (i.e. not from the `Resolv` module) cause both A and AAAA DNS requests to be made, even on systems that don’t actually have an IPv6 address that could possibly make the AAAA response useful. I wouldn’t _really_ care about this, normally, but glibc has a bug (https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1961697) which can cause a 5-second delay in DNS lookups when both A and AAAA records are queried in parallel. This bug is fixed in glibc upstream but still present in some LTS linux distros (Ubuntu 18.04 and 20.04 at least), so I think it’s worthwhile to try and work around it in circumstances where the AAAA request is pointless anyway. The dual A/AAAA lookup happens because whenever Ruby calls getaddrinfo to perform DNS lookups, it always sets `hints`, and sets `hints->ai_flags` to zero by default unless flags are specified by the caller (e.g. `AI_PASSIVE` is set when binding a TCP server socket in `TCPServer.new`). This matches the default value of `ai_flags` specified by POSIX, which is zero. However, glibc behaves differently. When glibc’s `getaddrinfo` function is called with `NULL` for the `hints` parameter, it defaults the `ai_flags` value to `(AI_V4MAPPED | AI_ADDRCONFIG)`. The manpage (from the Linux man-pages project - https://man7.org/linux/man-pages/man3/getaddrinfo.3.html) claims “this is an improvement on the standard” (although I couldn’t find this mentioned in the glibc manual itself). Of course, we’re not _actually_ ever calling `getaddrinfo` with NULL `hints`; so, we never actually use these flags on glibc systems (unless they’re explicitly specified by the caller). My proposal is that we should change Ruby to set these two flags by default, when they’re available, in the following circumstances: * In all calls made internally to `rsock_getaddrinfo` as a result of socket functions like `TCPSocket.new`, `UDPSocket.new`, etc. * EXCEPT when `AI_PASSIVE` is also set (i.e. when we’re trying to get an address to bind for listener socket - see below) * In calls made to `rsock_getaddrinfo` as a direct result of calling `Addrinfo.getaddrinfo` from Ruby with nil flags * EXCEPT calls to `Addrinfo.getaddrinfo` where explicit flags are provided Both of these seem like something you would almost always want to be doing in any outgoing connection scenario: * `AI_V4MAPPED` ensures that, if AF_INET6 is explicitly specified as the desired protocol, and there is no AAAA record in DNS, that any A record that _is_ present gets converted to an IPv4-mapped IPv6 address so it can be used e.g. with NAT64. * `AI_ADDRCONFIG` ensures that, if a machine has no IPv6 address, it doesn’t bother making an AAAA lookup that will return IPv6 addresses that can’t actually be used for anything (and vice versa for IPv4). The reason why we wouldn’t want to set `AI_ADDRCONFIG` in circumstances where Ruby currently sets `AI_PASSIVE` is that loopback addresses are not considered in deciding if a system has an IPv4/IPv6 address. Conceivably, you might want to bind to a `::1` loopback address, and allow other processes on the same machine to connect to that. Does changing this default sound reasonable? If so I can prepare a patch. Another option I considered is doing this _only_ when Ruby is built against glibc (so that other system behaviour is most closely matched). -- https://bugs.ruby-lang.org/

1 0