July 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:117431] [Ruby master Misc#20407] Question about applying encoding modifier to an interpolated Regexp
by andrykonchin (Andrew Konchin) 11 Jul '24

11 Jul '24

Issue #20407 has been reported by andrykonchin (Andrew Konchin). ---------------------------------------- Misc #20407: Question about applying encoding modifier to an interpolated Regexp https://bugs.ruby-lang.org/issues/20407 * Author: andrykonchin (Andrew Konchin) * Status: Open ---------------------------------------- I am wondering how Regexp encoding modifiers (u, s, e, n) interfere in encoding negotiation of interpolated Regexp literal. Examples #1 ```ruby # encoding: us-ascii # Unicode: Ф - U+0424 # windows-1251: Ф - 0xD4 # without encoding modifier puts /a #{ "\xd4".force_encoding("windows-1251") } c/.encoding # Windows-1251 puts /a #{ "b".encode("windows-1251") } c/.encoding # US-ASCII puts /a #{ "\u0424".force_encoding("UTF-8") } c/.encoding # UTF-8 puts /a #{ "\xc2\xa1".b } c/.encoding # ASCII-8BIT # with encoding modifier puts /a #{ "\xd4".force_encoding("windows-1251") } c/e.encoding # Windows-1251 puts /a #{ "b".encode("windows-1251") } c/e.encoding # EUC-JP puts /a #{ "\u0424".force_encoding("UTF-8") } c/e.encoding # UTF-8 puts /a #{ "\xc2\xa1".b } c/e.encoding # ASCII-8BIT # string literals concatenation puts ("a" + "\xd4".force_encoding("windows-1251") + "c").encoding # Windows-1251 puts ("a" + "b".encode("windows-1251") + "c").encoding # US-ASCII puts ("a" + "\u0424".force_encoding("UTF-8") + "c").encoding # UTF-8 puts ("a" + "\xc2\xa1".b + "c").encoding # ASCII-8BIT ``` Example #2 ```ruby # encoding: utf-8 # windows-1251: Ф - 0xD4 # unicode: Ф - U+0424 # without encoding modifier puts /a #{ "\xd4".force_encoding("windows-1251") } c/.encoding # Windows-1251 puts /a #{ "b".encode("windows-1251") } c/.encoding # US-ASCII puts /a #{ "\u0424".force_encoding("UTF-8") } c/.encoding # UTF-8 puts /a #{ "\xc2\xa1".b } c/.encoding # ASCII-8BIT # with encoding modifier puts /a #{ "\xd4".force_encoding("windows-1251") } c/e.encoding # Windows-1251 puts /a #{ "b".encode("windows-1251") } c/e.encoding # EUC-JP puts /a #{ "\u0424".force_encoding("UTF-8") } c/e.encoding # UTF-8 puts /a #{ "\xc2\xa1".b } c/e.encoding # ASCII-8BIT # string literals concatenation puts ("a" + "\xd4".force_encoding("windows-1251") + "c").encoding # Windows-1251 puts ("a" + "b".encode("windows-1251") + "c").encoding # UTF-8 puts ("a" + "\u0424".force_encoding("UTF-8") + "c").encoding # UTF-8 puts ("a" + "\xc2\xa1".b + "c").encoding # ASCII-8BIT ``` In the examples above the `e` modifier changes Regexp's encoding only in one case when Regexp's encoding would be `US-ASCII` without the modifier: ```ruby # encoding: us-ascii puts /a #{ "b".encode("windows-1251") } c/.encoding # US-ASCII puts /a #{ "b".encode("windows-1251") } c/e.encoding # EUC-JP ``` ```ruby # encoding: utf-8 puts /a #{ "b".encode("windows-1251") } c/.encoding # US-ASCII puts /a #{ "b".encode("windows-1251") } c/e.encoding # EUC-JP ``` And the `e` modifier doesn't change Regexp's final encoding in all the other cases either Regexp's encoding without modifier is a file source encoding or `ASCII-8BIT`. Looking at the following example: ```ruby # encoding: us-ascii # without modifier p /\xc2\xa1 #{ "a" }\xc2\xa1/.encoding # ASCII-8BIT p /a #{ "\xc2\xa1".force_encoding("EUC-JP") } b/.encoding # EUC-JP p /a #{ "\xc2\xa1".b } b/.encoding # ASCII-8BIT # with modifier p /\xc2\xa1 #{ "a" }\xc2\xa1/e.encoding # EUC-JP p /a #{ "\xc2\xa1".force_encoding("EUC-JP") } b/e.encoding # EUC-JP p /a #{ "\xc2\xa1".b } b/e.encoding # ASCII-8BIT ``` we can notice that the `e` modifier change `ASCII-8BIT` to `EUC-JP` in the first case but doesn't in the third one. So I assume that the `e` modifier could be applied to the Regexp fragments (`\xc2\xa1` and `\xc2\xa1`) before encoding negotiation and not to the whole result after negotiation. Could you please clarify how it works? -- https://bugs.ruby-lang.org/

4 5

[ruby-core:118383] [Ruby master Bug#20593] `Kernel#format` emits a `too many arguments for format string` warning when called with keyword arguments and none are used
by byroot (Jean Boussier) 11 Jul '24

11 Jul '24

Issue #20593 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #20593: `Kernel#format` emits a `too many arguments for format string` warning when called with keyword arguments and none are used https://bugs.ruby-lang.org/issues/20593 * Author: byroot (Jean Boussier) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- `Kernel#format` emits a warning when too many arguments are passed. It does so for postional arguments, which make sense, as it could be indicative of a bug: ```ruby $VERBOSE = true format("%s", 1, 2) # warning: too many arguments for format string ``` And doesn't for keyword arguments, as it's much less likely to be a bug: ```ruby $VERBOSE = true format("%{a}s", a: 1, unused: 2) # no warning ``` However it does emit a warning when passing keyword arguments, if none are used: ```ruby $VERBOSE = true format("test", unused: 2) # warning: too many arguments for format string ``` Which I believe is a bug, as it's common for `format` to take a user provided string to allow for safe interpolation, but with no intention to enforce any of the arguments to be interpolated. So I think that last case shouldn't emit a warning. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:118496] [Ruby master Bug#20618] ruby uses 1/2 (8 out of 16) cores for 16 ractors
by skorobogatydmitry (Dmitry Skorobogaty) 11 Jul '24

11 Jul '24

Issue #20618 has been reported by skorobogatydmitry (Dmitry Skorobogaty). ---------------------------------------- Bug #20618: ruby uses 1/2 (8 out of 16) cores for 16 ractors https://bugs.ruby-lang.org/issues/20618 * Author: skorobogatydmitry (Dmitry Skorobogaty) * Status: Open * ruby -v: ruby 3.3.1 (2024-04-23 revision c56cd86388) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- According to `top`, the below code uses 7/16 cores (700%) of my AMD Ryzen 7 5800HS: ``` #!/usr/bin/env ruby 16.times { Ractor.new { 10_000.downto(1) { |i| 100_000.downto(1) { |j| i * j } } } } sleep 30 ``` The same code occupies all 16 cores on `ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]` The problem reproduces on `ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x86_64-linux]` (ruby from the latest docker image). -- https://bugs.ruby-lang.org/

3 2

[ruby-core:118538] [Ruby master Feature#16482] net/http should support TLS connection to proxies
by hsbt (Hiroshi SHIBATA) 11 Jul '24

11 Jul '24

Issue #16482 has been updated by hsbt (Hiroshi SHIBATA). Assignee set to hsbt (Hiroshi SHIBATA) ---------------------------------------- Feature #16482: net/http should support TLS connection to proxies https://bugs.ruby-lang.org/issues/16482#change-109057 * Author: xformer (Frank Schwab) * Status: Closed * Assignee: hsbt (Hiroshi SHIBATA) ---------------------------------------- Right now net/http forces the user to use a clear text connection to a proxy. This massively reduces security as the user is forced to sent proxy authentication data in the clear. A proxy is specified in net/http like this: ``` proxy_addr = 'your.proxy.host' proxy_port = 8080 proxy_user = 'aProxyUser' proxy_pwd = 'aProxyPassword' Net::HTTP.new('example.com', nil, proxy_addr, proxy_port, proxy_user, proxy_pwd).start { |http| # always proxy via your.proxy.addr:8080, user 'aProxyUser', password 'aProxyPassword' } ``` There is no scheme present in the 'proxy_addr' variable. In the code of Net::HTTP::new the proxy connection is opened via a TCP socket, not via HTTP or HTTPS. As this considerably weakens security I would like to suggest that it should be made possible to specify that the connection to the proxy is done through a TLS connection. Maybe there could be a use_ssl parameter or the like. Note that this issue is not about the connection that is routed through the proxy but about the connection to the proxy itself. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118537] [Ruby master Feature#16482] net/http should support TLS connection to proxies
by hsbt (Hiroshi SHIBATA) 11 Jul '24

11 Jul '24

Issue #16482 has been updated by hsbt (Hiroshi SHIBATA). Status changed from Open to Closed I merged https://github.com/ruby/net-http/pull/55 now. ---------------------------------------- Feature #16482: net/http should support TLS connection to proxies https://bugs.ruby-lang.org/issues/16482#change-109056 * Author: xformer (Frank Schwab) * Status: Closed ---------------------------------------- Right now net/http forces the user to use a clear text connection to a proxy. This massively reduces security as the user is forced to sent proxy authentication data in the clear. A proxy is specified in net/http like this: ``` proxy_addr = 'your.proxy.host' proxy_port = 8080 proxy_user = 'aProxyUser' proxy_pwd = 'aProxyPassword' Net::HTTP.new('example.com', nil, proxy_addr, proxy_port, proxy_user, proxy_pwd).start { |http| # always proxy via your.proxy.addr:8080, user 'aProxyUser', password 'aProxyPassword' } ``` There is no scheme present in the 'proxy_addr' variable. In the code of Net::HTTP::new the proxy connection is opened via a TCP socket, not via HTTP or HTTPS. As this considerably weakens security I would like to suggest that it should be made possible to specify that the connection to the proxy is done through a TLS connection. Maybe there could be a use_ssl parameter or the like. Note that this issue is not about the connection that is routed through the proxy but about the connection to the proxy itself. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118281] [Ruby master Misc#20574] DevMeeting-2024-07-11
by mame (Yusuke Endoh) 10 Jul '24

10 Jul '24

Issue #20574 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #20574: DevMeeting-2024-07-11 https://bugs.ruby-lang.org/issues/20574 * Author: mame (Yusuke Endoh) * Status: Open ---------------------------------------- # The next dev meeting **Date: 2024/07/11 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2024/07/08. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

10 11

[ruby-core:118446] [Ruby master Bug#20608] Hash#find always allocates each iterated pair
by ivoanjo (Ivo Anjo) 10 Jul '24

10 Jul '24

Issue #20608 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #20608: Hash#find always allocates each iterated pair https://bugs.ruby-lang.org/issues/20608 * Author: ivoanjo (Ivo Anjo) * Status: Open * ruby -v: ruby 3.4.0preview1 (2024-05-16 master 9d69619623) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Hey there! Recently I ran into this sharp edge in `Hash#find`: ```ruby puts RUBY_DESCRIPTION def allocated_now = GC.stat(:total_allocated_objects) dummy_hash = 10_000.times.each_with_index.to_h before_allocs = allocated_now dummy_hash.any? { |k, v| } puts "Allocated #{allocated_now - before_allocs} for #{File.read(__FILE__).lines[__LINE__-2]}" before_allocs = allocated_now dummy_hash.each { |k, v| } puts "Allocated #{allocated_now - before_allocs} for #{File.read(__FILE__).lines[__LINE__-2]}" before_allocs = allocated_now dummy_hash.find { |k, v| } puts "Allocated #{allocated_now - before_allocs} for #{File.read(__FILE__).lines[__LINE__-2]}" before_allocs = allocated_now dummy_hash.any? { |k| } puts "Allocated #{allocated_now - before_allocs} for #{File.read(__FILE__).lines[__LINE__-2]}" ``` Result: ``` ruby 3.4.0preview1 (2024-05-16 master 9d69619623) [x86_64-linux] Allocated 0 for dummy_hash.any? { |k, v| } Allocated 0 for dummy_hash.each { |k, v| } Allocated 10002 for dummy_hash.find { |k, v| } Allocated 10000 for dummy_hash.any? { |k| } ``` That is, while `Hash#any?`, `Hash#each`, etc avoid doing any allocations during iteration, `Hash#find` does not hit the `rb_block_pair_yield_optimizable` => `each_pair_i_fast` fast path, and so is massively costly compared to the others. This is very surprising, as I'd expect a `find` to have a comparable cost to `any?` (and I ended up [redoing some code to avoid find](https://github.com/DataDog/dd-trace-rb/pull/3757)) Also while experimenting a bit, it was surprising to me that the allocation optimization only kicks in when `|k, v|` are declared, and thus `.any? { |k| }` is also more expensive. -- https://bugs.ruby-lang.org/

2 5

[ruby-core:117653] [Ruby master Bug#20446] OUtdated https://cache.ruby-lang.org/pub/ruby/index.txt
by vo.x (Vit Ondruch) 09 Jul '24

09 Jul '24

Issue #20446 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #20446: OUtdated https://cache.ruby-lang.org/pub/ruby/index.txt https://bugs.ruby-lang.org/issues/20446 * Author: vo.x (Vit Ondruch) * Status: Open * ruby -v: ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Is the https://cache.ruby-lang.org/pub/ruby/index.txt the place, where all Ruby releases can be found? If yes, then can this file be update with Ruby 3.3.1, 3.2.4 etc? If no, can this file be removed to avoid confusion? -- https://bugs.ruby-lang.org/

3 6

[ruby-core:118428] [Ruby master Bug#20605] PRs with backported fix for https://github.com/ruby/ruby/pull/11036 to Ruby 3.3 and 3.2
by ivoanjo (Ivo Anjo) 09 Jul '24

09 Jul '24

Issue #20605 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #20605: PRs with backported fix for https://github.com/ruby/ruby/pull/11036 to Ruby 3.3 and 3.2 https://bugs.ruby-lang.org/issues/20605 * Author: ivoanjo (Ivo Anjo) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- Hey! I work for Datadog on the Ruby profiler part of the [datadog (previously ddtrace)](https://github.com/datadog/dd-trace-rb) gem. Recently we merged a very small tweak to master to make ensure compilers did the right thing always: https://github.com/ruby/ruby/pull/11036 . It would be really cool to also get this fix on the Ruby versions still under active maintenance (3.2 and 3.3). I've gone ahead and opened PRs backporting this fix: * https://github.com/ruby/ruby/pull/11090 * https://github.com/ruby/ruby/pull/11091 Let me know if I can help any further :) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:118487] [Ruby master Bug#20617] /\pArabic/ character property doesn't match certain Arabic characters
by kytrinyx (Katrina Owen) 09 Jul '24

09 Jul '24

Issue #20617 has been reported by kytrinyx (Katrina Owen). ---------------------------------------- Bug #20617: /\pArabic/ character property doesn't match certain Arabic characters https://bugs.ruby-lang.org/issues/20617 * Author: kytrinyx (Katrina Owen) * Status: Open * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-darwin21] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I am not sure this is a bug. On some occasions I have Arabic text, but the Arabic character property rejects it as being Arabic. Example: ``` str = "شغل مرحلة أولى ، جداً؟" /^\p{Arabic}$/.match(str).inspect # => nil str.chars.reject {|char| /\p{Arabic}/.match(char)}.uniq # arabic space, arabic comma, arabic question mark, and arabic fatahan ``` This isn't a problem, since I defined my own regex to include the missing characters, but wanted to raise it in case it is, in fact, a bug. -- https://bugs.ruby-lang.org/

3 2