December 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:115581] [Ruby master Bug#16145] regexp match error if mixing /i, character classes, and utf8
by duerst 03 Dec '23

03 Dec '23

Issue #16145 has been updated by duerst (Martin Dürst). @zenspider I said that single-character character classes get reduced to just the actual character. So that would mean that your `"CAFÉ" =~ /[a]/i` gets reduced to `"CAFÉ" =~ /a/i`, and therefore works. That of course does not prove my guess, but it also doesn't disprove it. We'd need some other examples to test this further. ---------------------------------------- Bug #16145: regexp match error if mixing /i, character classes, and utf8 https://bugs.ruby-lang.org/issues/16145#change-105514 * Author: zenspider (Ryan Davis) * Status: Open * Priority: Normal * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- (reported on behalf of mage(a)mage.gold -- there appears to be an error in registration or login): See: ruby-talk @ X-Mail-Count: 440336 2.6.3 :049 > 'SHOP' =~ /[xo]/i => 2 2.6.3 :050 > 'CAFÉ' =~ /[é]/i => 3 2.6.3 :051 > 'CAFÉ' =~ /[xé]/i => nil 2.6.3 :052 > 'CAFÉ' =~ /[xÉ]/i => 3 Expected result: 2.6.3 :051 > 'CAFÉ' =~ /[xé]/i => 3 I tested it on random regex online pages. It does not match on https://regex101.com/ It matches on: https://regexr.com/ https://www.regextester.com/ https://www.freeformatter.com/regex-tester.html (Ignore case turned on). The reason I suppose it’s more like a bug than a feature is the fact that /[é]/i matches 'CAFÉ'. If the //i didn’t work for UTF-8 characters then the /[é]/i wouldn’t match it either. For example, [é] does not match 'CAFÉ' on https://regex101.com/ I could not find a page or a system that behaves the same way as Ruby does. For example, it matches in PostgreSQL 10 (under FreeBSD 12) too: # select 'CAFÉ'~ '[xé]'; ?column? ---------- f (1 row) # select 'CAFÉ' ~* '[xé]'; ?column? ---------- t (1 row) Tested it in IRB on macOS and FreeBSD. $ uname -a && ruby -v && locale Darwin xxx 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64 ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin18] LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL="en_US.UTF-8" $ uname -a && ruby -v && locale FreeBSD xxx 12.0-RELEASE-p9 FreeBSD 12.0-RELEASE-p9 GENERIC amd64 ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-freebsd12.0] LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_ALL=en_US.UTF-8 I installed Ruby with RVM. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115484] [Ruby master Bug#20022] GC.verify_compaction_references does not actually move alll objects
by kjtsanaktsidis (KJ Tsanaktsidis) 03 Dec '23

03 Dec '23

Issue #20022 has been reported by kjtsanaktsidis (KJ Tsanaktsidis). ---------------------------------------- Bug #20022: GC.verify_compaction_references does not actually move alll objects https://bugs.ruby-lang.org/issues/20022 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- While debugging https://bugs.ruby-lang.org/issues/20021, I ran into a separate issue which I figured was worth resolving whilst I had it in my head. The intention of GC.verify_compaction_references is, I believe, to force every single movable object to be moved, so that it's possible to debug native extensions which not correctly updating their references to objects they mark as movable (if this is _not_ the case, you can stop reading now and accept my apologies!) To do this, it doubles the number of allocated pages for each size pool, and sorts the heap pages so that the free ones are swept first; thus, every object in an old page should be moved into a free slot in one of the new pages. This worked fine until movement of objects between size pools during compaction was implemented. That causes some problems for verify_compaction_references: - We were doubling the number of pages in each size pool, but actually if some objects need to move into a different pool, there's no guarantee that they'll be enough room in that one. - It's possible for the sweep & compact cursors to meet in one size pool before all the objects that want to move into that size pool from another are processed by the compaction. You can see these problems by changing some of the movement tests in test_gc_compact.rb to try and move e.g. 50,000 objects instead of 500 (by changing `HASH_COUNT`); the test is not able to actually move all of the objects in a single compaction run. What I implemented in this PR (https://github.com/ruby/ruby/pull/9041) is two things: - Firstly, it examines every object and determine where it wants to be compacted into; we use this information to calculate the desired number of pages to add to each size pool. - Secondly, it compacts twice; once in order of ascending size pool, and once descending. This means that we are guaranteed to be able to move everything we want to move into a size pool before we start advancing that pool's own compact cursor. With these fixes in place, I was able to make the compaction tests move any amount of objects (previously, `test_moving_hashes_down_size_pools` would begin failing on my machine if I set `HASH_COUNT` to above about 6,000. -- https://bugs.ruby-lang.org/

2 6

[ruby-core:115579] [Ruby master Bug#19144] Ruby should set AI_V4MAPPED | AI_ADDRCONFIG getaddrinfo flags by default
by akr (Akira Tanaka) 02 Dec '23

02 Dec '23

Issue #19144 has been updated by akr (Akira Tanaka). kjtsanaktsidis (KJ Tsanaktsidis) wrote in #note-5: > @akr could you take a look at my PR when you get a chance? I think I addressed your feedback, please let me know if I have misunderstood! It seems fine. I agree that we remove the test because it is too complicated. ---------------------------------------- Bug #19144: Ruby should set AI_V4MAPPED | AI_ADDRCONFIG getaddrinfo flags by default https://bugs.ruby-lang.org/issues/19144#change-105512 * Author: kjtsanaktsidis (KJ Tsanaktsidis) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Currently, DNS lookups made with `getaddrinfo` from Ruby (i.e. not from the `Resolv` module) cause both A and AAAA DNS requests to be made, even on systems that don’t actually have an IPv6 address that could possibly make the AAAA response useful. I wouldn’t _really_ care about this, normally, but glibc has a bug (https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1961697) which can cause a 5-second delay in DNS lookups when both A and AAAA records are queried in parallel. This bug is fixed in glibc upstream but still present in some LTS linux distros (Ubuntu 18.04 and 20.04 at least), so I think it’s worthwhile to try and work around it in circumstances where the AAAA request is pointless anyway. The dual A/AAAA lookup happens because whenever Ruby calls getaddrinfo to perform DNS lookups, it always sets `hints`, and sets `hints->ai_flags` to zero by default unless flags are specified by the caller (e.g. `AI_PASSIVE` is set when binding a TCP server socket in `TCPServer.new`). This matches the default value of `ai_flags` specified by POSIX, which is zero. However, glibc behaves differently. When glibc’s `getaddrinfo` function is called with `NULL` for the `hints` parameter, it defaults the `ai_flags` value to `(AI_V4MAPPED | AI_ADDRCONFIG)`. The manpage (from the Linux man-pages project - https://man7.org/linux/man-pages/man3/getaddrinfo.3.html) claims “this is an improvement on the standard” (although I couldn’t find this mentioned in the glibc manual itself). Of course, we’re not _actually_ ever calling `getaddrinfo` with NULL `hints`; so, we never actually use these flags on glibc systems (unless they’re explicitly specified by the caller). My proposal is that we should change Ruby to set these two flags by default, when they’re available, in the following circumstances: * In all calls made internally to `rsock_getaddrinfo` as a result of socket functions like `TCPSocket.new`, `UDPSocket.new`, etc. * EXCEPT when `AI_PASSIVE` is also set (i.e. when we’re trying to get an address to bind for listener socket - see below) * In calls made to `rsock_getaddrinfo` as a direct result of calling `Addrinfo.getaddrinfo` from Ruby with nil flags * EXCEPT calls to `Addrinfo.getaddrinfo` where explicit flags are provided Both of these seem like something you would almost always want to be doing in any outgoing connection scenario: * `AI_V4MAPPED` ensures that, if AF_INET6 is explicitly specified as the desired protocol, and there is no AAAA record in DNS, that any A record that _is_ present gets converted to an IPv4-mapped IPv6 address so it can be used e.g. with NAT64. * `AI_ADDRCONFIG` ensures that, if a machine has no IPv6 address, it doesn’t bother making an AAAA lookup that will return IPv6 addresses that can’t actually be used for anything (and vice versa for IPv4). The reason why we wouldn’t want to set `AI_ADDRCONFIG` in circumstances where Ruby currently sets `AI_PASSIVE` is that loopback addresses are not considered in deciding if a system has an IPv4/IPv6 address. Conceivably, you might want to bind to a `::1` loopback address, and allow other processes on the same machine to connect to that. Does changing this default sound reasonable? If so I can prepare a patch. Another option I considered is doing this _only_ when Ruby is built against glibc (so that other system behaviour is most closely matched). -- https://bugs.ruby-lang.org/

1 0

[ruby-core:115574] [Ruby master Bug#20036] Backport 6ebcf25de2859b5b6402b7e8b181066c32d0e0bf
by nagachika (Tomoyuki Chikanaga) 02 Dec '23

02 Dec '23

Issue #20036 has been reported by nagachika (Tomoyuki Chikanaga). ---------------------------------------- Bug #20036: Backport 6ebcf25de2859b5b6402b7e8b181066c32d0e0bf https://bugs.ruby-lang.org/issues/20036 * Author: nagachika (Tomoyuki Chikanaga) * Status: Closed * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- This is a backport management ticket. I think https://github.com/ruby/ruby/commit/6ebcf25de2859b5b6402b7e8b181066c32d0e0bf should be backported into ruby_{3_1,3_2}. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:115563] [Ruby master Bug#20033] `/#{''}(?<lvar>)/ =~ s` looks like a dynamic regexp match but assigns to local variable
by tompng (tomoya ishida) 01 Dec '23

01 Dec '23

Issue #20033 has been reported by tompng (tomoya ishida). ---------------------------------------- Bug #20033: `/#{''}(?<lvar>)/ =~ s` looks like a dynamic regexp match but assigns to local variable https://bugs.ruby-lang.org/issues/20033 * Author: tompng (tomoya ishida) * Status: Open * Priority: Normal * ruby -v: ruby 3.3.0dev (2023-11-30T16:23:25Z master d048bae96b) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I expect all of these code not to assign to lvar because all regexp looks like embedded dynamic regexp. ~~~ruby /#{''}(?<lvar>)/ =~ '' # assings to lvar /#{"#{''}"}(?<lvar>)/ =~ '' # does not assign to lvar /#{<<A}(?<lvar>)/ =~ '' # assigns to lvar string A /#{<<A}(?<lvar>)/ =~ '' # does not assign to lvar #{'string'} A ~~~ I think `/#{'a'}/` can be a DREGX because for string, `"#{'a'}#{'b'}"` is DSTR `(DSTR@1:0-1:14 "ab" nil nil)`. -- https://bugs.ruby-lang.org/

3 2