- ruby-core - ml.ruby-lang.org

[ruby-core:111300] [Ruby master Feature#18683] Allow to create hashes with a specific capacity.
by byroot (Jean Boussier) 15 Dec '22

15 Dec '22

Issue #18683 has been updated by byroot (Jean Boussier). Yeah the ruby side of the proposal wasn't merged, but I'll open an issue specifically for it. ---------------------------------------- Feature #18683: Allow to create hashes with a specific capacity. https://bugs.ruby-lang.org/issues/18683#change-100660 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal ---------------------------------------- Various protocol parsers such as Redis `RESP3` or `msgpack`, have to create hashes, and they know the size in advance. For efficiency, it would be preferable if they could directly allocate a Hash of the necessary size, so that large hashes wouldn't cause many re-alloccations. Example of code that would benefit: - [`hiredis` bindings](https://github.com/redis-rb/redis-client/blob/830d586b665bc956933… - [Ruby `redis RESP3` parser](https://github.com/redis-rb/redis-client/blob/830d586b665bc9569335d… - [magpack-ruby](https://github.com/msgpack/msgpack-ruby/blob/c46bb60f79312cab… `String` and `Array` both already offer similar APIs: ```ruby String.new(capacity: XXX) Array.new(XX) / rb_ary_new_capa(long) ``` However there's no such public API for Hashes, neither in Ruby land not in the C extension API. ### Proposal I think `Hash.new` should accept a `capacity:` named parameter: ```ruby hash = Hash.new(capacity: 1000) ``` Additionally I think the internal `rb_hash_new_with_size` function should be exposed to C extensions as `rb_hash_new_capa(long)`, for consistency with `rb_ary_new_capa(long)`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111299] [Ruby master Feature#18183] make SecureRandom.choose public
by mame (Yusuke Endoh) 15 Dec '22

15 Dec '22

Issue #18183 has been updated by mame (Yusuke Endoh). Status changed from Rejected to Assigned Assignee set to nobu (Nobuyoshi Nakada) ---------------------------------------- Feature #18183: make SecureRandom.choose public https://bugs.ruby-lang.org/issues/18183#change-100659 * Author: olleicua (Antha Auciello) * Status: Assigned * Priority: Normal * Assignee: nobu (Nobuyoshi Nakada) ---------------------------------------- This issue https://bugs.ruby-lang.org/issues/10849 added `SecureRandom.alphanumeric` and also the private method choose. `choose` was kept private because the method name wasn't the best name to represent the behavior. I think if it was called `random_string` it would be very clear what it does. I also think it should be aliased to choose as well to allow backwards compatibility for people bypassing the private method with `send` (e.g. https://www.thetopsites.net/article/58611103.shtml) I'm planning to put together a pull request for this. Please let me know if there are any complications I'm not considering. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111101] [Ruby master Bug#19162] [ANN] Restrict weak ssh public key for git.ruby-lang.org
by hsbt (Hiroshi SHIBATA) 15 Dec '22

15 Dec '22

Issue #19162 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #19162: [ANN] Restrict weak ssh public key for git.ruby-lang.org https://bugs.ruby-lang.org/issues/19162 * Author: hsbt (Hiroshi SHIBATA) * Status: Assigned * Priority: Normal * Assignee: hsbt (Hiroshi SHIBATA) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- We continuously migrate our ssh public keys on git.ruby-lang.org from svn and cvs. I found some of pub keys used weak algorithm. So, I will restrict them at 15 Dec 2022. I'll notify committers who are still use weak key later. -- https://bugs.ruby-lang.org/

1 2

[ruby-core:111297] [Ruby master Feature#18683] Allow to create hashes with a specific capacity.
by hsbt (Hiroshi SHIBATA) 15 Dec '22

15 Dec '22

Issue #18683 has been updated by hsbt (Hiroshi SHIBATA). Status changed from Open to Closed https://github.com/ruby/ruby/pull/5835 has been merged. I'll close this. @byroot If you have any concern or proposal, please re-open or file another issue. Thanks. ---------------------------------------- Feature #18683: Allow to create hashes with a specific capacity. https://bugs.ruby-lang.org/issues/18683#change-100657 * Author: byroot (Jean Boussier) * Status: Closed * Priority: Normal ---------------------------------------- Various protocol parsers such as Redis `RESP3` or `msgpack`, have to create hashes, and they know the size in advance. For efficiency, it would be preferable if they could directly allocate a Hash of the necessary size, so that large hashes wouldn't cause many re-alloccations. Example of code that would benefit: - [`hiredis` bindings](https://github.com/redis-rb/redis-client/blob/830d586b665bc956933… - [Ruby `redis RESP3` parser](https://github.com/redis-rb/redis-client/blob/830d586b665bc9569335d… - [magpack-ruby](https://github.com/msgpack/msgpack-ruby/blob/c46bb60f79312cab… `String` and `Array` both already offer similar APIs: ```ruby String.new(capacity: XXX) Array.new(XX) / rb_ary_new_capa(long) ``` However there's no such public API for Hashes, neither in Ruby land not in the C extension API. ### Proposal I think `Hash.new` should accept a `capacity:` named parameter: ```ruby hash = Hash.new(capacity: 1000) ``` Additionally I think the internal `rb_hash_new_with_size` function should be exposed to C extensions as `rb_hash_new_capa(long)`, for consistency with `rb_ary_new_capa(long)`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111296] [Ruby master Feature#18498] Introduce a public WeakKeysMap that compares by equality
by matz (Yukihiro Matsumoto) 15 Dec '22

15 Dec '22

Issue #18498 has been updated by matz (Yukihiro Matsumoto). LGTM. Probably too late for 3.2. For 3.3 maybe? Matz. ---------------------------------------- Feature #18498: Introduce a public WeakKeysMap that compares by equality https://bugs.ruby-lang.org/issues/18498#change-100656 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- This is a clean take on #16038 ### Spec #### Weak keys only After a chat with @eregon we believe that what would make the most sense and would be most useful would be a "WeakKeysMap". Meaning **the keys would be weak references, but the values would be strong references**. This behavior is also consistent with the terminology in other popular languages such as Javascript and Java were `WeakMap` only have weak keys. By default **it would use equality semantic, just like a regular `Hash`**. Having an option to compare by indentity instead could be useful, but not strictly required. #### Immediate objects support Many WeakMap implementation in other languages don't accept "immediate" objects (or their equivalent) in the weak slots. This is because since they are never collected, the weak reference will never possibly expire. `ObjectSpace::WeakMap` currently accept them, but since both keys and values are weak references, there is legitimate use. However in a `WeakKeysMap`, using an immediate object as key should likely raise a `TypeError`. What is less clear is wether `BIGNUM` (allocated `Integer`) and dynamic symbols (allocated `Symbol`) should be accepted. I believe they shouldn't for consistency, so: - No immediate objects - No `Integer` - No `Symbol` #### `member` method to lookup an existing key For some use case, notably deduplication sets, a `member` method that maps `st_get_key` would be useful. ```ruby def member(key) -> existing key or nil ``` Not sure if `member` is the best possible name, but `#key` is already used for looking up a key by value. That same method could be useful on `Hash` and `Set` as well, but that would be a distinct feature request. #### Naming Possible names: - `::WeakMap` - `::WeakKeysMap` - `::WeakRef::Map` - `::WeakRef::WeakMap` - `::WeakRef::WeakKeysMap` - `::WeakRef::WeakHash` - `::WeakRef::WeakKeysHash` My personal, ligthly held, preference goes toward `::WeakRef::WeakKeysMap`. ### Use cases #### WeakSet A `WeakKeysMap` would be a good enough primitive to implement a `WeakSet` in pure Ruby code, just like `Set` is implemented with a `Hash`. `WeakSet` are useful for use cases such as avoiding cycles in an object graph without holding strong references. #### Deduplication sets Assuming `WeakMap` have the `#member` method. A small variation on the "deduplicating constructors" use case, better suited for smaller but numerous objects. The difference here is that we first build the object and then lookup for a pre-existing one. This is the strategy used to [deduplicate Active Record schema metadata](https://github.com/rails/rails/blob/3be590edbedab8ddcacdf72790d50…. ```ruby class DeduplicationSet def initialize @set = WeakKeysMap.new end def dedup(object) if existing_object = @set.member(object) existing_object else @set[object] = true object end end end ``` #### Third party object extension When you need to record some associated data on third party objects for which you don't control the lifetime. A WeakMap can be used: ```ruby METADATA = WeakKeysMap.new def do_something(third_party_object) metadata = (METADATA[third_party_object] ||= Metadata.new) metadata.foo = "bar" end ``` -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111295] [Ruby master Feature#18033] Time.new to parse a string
by matz (Yukihiro Matsumoto) 15 Dec '22

15 Dec '22

Issue #18033 has been updated by matz (Yukihiro Matsumoto). Accepted. Performance improvement is a great benefit. Both `Time.iso8859` and `Time.parse` requires loading `date` gem, and providing those methods in the core may cause overwriting. Currently, `Time.new` only takes numeric arguments, so adding extra argument pattern add only a small complexity. Matz. ---------------------------------------- Feature #18033: Time.new to parse a string https://bugs.ruby-lang.org/issues/18033#change-100654 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal ---------------------------------------- Make `Time.new` parse `Time#inspect` and ISO-8601 like strings. * `Time.iso8601` and `Time.parse` need an extension library, `date`. * `Time.iso8601` can't parse `Time#inspect` string. * `Time.parse` often results in unintentional/surprising results. * `Time.new` also about 1.9 times faster than `Time.iso8601`. ``` $ ./ruby -rtime -rbenchmark -e ' n = 1000 s = Time.now.iso8601 Benchmark.bm(12) do |x| x.report("Time.iso8601") {n.times{Time.iso8601(s)}} x.report("Time.parse") {n.times{Time.parse(s)}} x.report("Time.new") {n.times{Time.new(s)}} end' user system total real Time.iso8601 0.006919 0.000185 0.007104 ( 0.007091) Time.parse 0.018338 0.000207 0.018545 ( 0.018590) Time.new 0.003671 0.000069 0.003740 ( 0.003741) ``` https://github.com/ruby/ruby/pull/4825 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111293] [Ruby master Feature#18033] Time.new to parse a string
by mame (Yusuke Endoh) 15 Dec '22

15 Dec '22

Issue #18033 has been updated by mame (Yusuke Endoh). https://github.com/ruby/ruby/pull/4825 ---------------------------------------- Feature #18033: Time.new to parse a string https://bugs.ruby-lang.org/issues/18033#change-100651 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal ---------------------------------------- Make `Time.new` parse `Time#inspect` and ISO-8601 like strings. * `Time.iso8601` and `Time.parse` need an extension library, `date`. * `Time.iso8601` can't parse `Time#inspect` string. * `Time.parse` often results in unintentional/surprising results. * `Time.new` also about 1.9 times faster than `Time.iso8601`. ``` $ ./ruby -rtime -rbenchmark -e ' n = 1000 s = Time.now.iso8601 Benchmark.bm(12) do |x| x.report("Time.iso8601") {n.times{Time.iso8601(s)}} x.report("Time.parse") {n.times{Time.parse(s)}} x.report("Time.new") {n.times{Time.new(s)}} end' user system total real Time.iso8601 0.006919 0.000185 0.007104 ( 0.007091) Time.parse 0.018338 0.000207 0.018545 ( 0.018590) Time.new 0.003671 0.000069 0.003740 ( 0.003741) ``` https://github.com/ruby/ruby/pull/4639 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111291] [Ruby master Bug#18605] Fails to run on (newer) 32bit Windows with ucrt
by larskanis (Lars Kanis) 14 Dec '22

14 Dec '22

Issue #18605 has been updated by larskanis (Lars Kanis). Thanks all for sharing the details! I started a branch to replace the UCRT/MSVCRT pioinfo here: https://github.com/ruby/ruby/compare/master...larskanis:ruby:pioinfo?expand… My rough plan is to first replace pioinfo with a ruby-internal mapping between FD and HANDLE/SOCKET. This makes it necessary to replace all uses of FD and change it to native WINAPI functions using HANDLE/SOCKET. Fortunately many functions are already based on the native WINAPI. Among others it currently fails at the many process spawn variations and the text file CRLF conversions are not yet implemented. Also the few uses of FILE based libc functions still have to be replaced. After pioinfo is replaced and the implementation is passing all tests again, I would like to replace FD completely by HANDLE/SOCKET including in `IO#fileno` and `IO.new`. This is because the FD is then just an internal number, with no external meaning. But HANDLE/SOCKET is the native Windows handle to access resources and to process events. Even if this sounds pretty simple, there are a lot of tricky challenges on the way and I'm sure @naruse had very good reasons to build the pioinfo hack. ---------------------------------------- Bug #18605: Fails to run on (newer) 32bit Windows with ucrt https://bugs.ruby-lang.org/issues/18605#change-100647 * Author: lazka (Christoph Reiter) * Status: Open * Priority: Normal * Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- 32bit ruby using ucrt has started to fail on newer Windows with "unexpected ucrtbase.dll" -> https://github.com/ruby/ruby/blob/3fb7d2cadc18472ec107b14234933b017a33c14d/… The problem is that ruby depends on ucrt internals and those have apparently changed with newer versions. See https://github.com/msys2/MINGW-packages/pull/10878 and https://github.com/msys2/MINGW-packages/issues/10896 for some background and a potential fix. But ideally ruby wouldn't depend on Windows internals like this. thanks! -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111281] [Ruby master Bug#19234] [3.2.0dev] YJIT code GC can lead to crashes
by byroot (Jean Boussier) 14 Dec '22

14 Dec '22

Issue #19234 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19234: [3.2.0dev] YJIT code GC can lead to crashes https://bugs.ruby-lang.org/issues/19234 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Assignee: k0kubun (Takashi Kokubun) * Target version: 3.2 * ruby -v: ruby 3.2.0dev (2022-12-13T16:07:29Z master a66a69865d) +YJIT [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Filing this bug here in case some people may have observed it too and may have more information, and also to keep track of it for the upcoming 3.2.0 release. After changing some settings on our CI to make sure YJIT's `code_gc` would trigger, we discovered that it sometimes cause crashes. The crash can take many different form (e.g. `[BUG] Segmentation fault at 0x00005604a8e78006` or `[BUG] Illegal instruction at 0x0000aaaacc0ce4c0`), and happens on both `x86` and `arm64`. It however happens very consistently on our CI, but only after running for 15 to 20 minutes and we haven't been able to reduce it to a local reproduction script. When it happens however the backtrace isn't really helpful: ``` -- C level backtrace information ------------------------------------------- /usr/local/ruby/bin/real-ruby(rb_print_backtrace+0x11) [0x5604a8a6df7d] vm_dump.c:770 /usr/local/ruby/bin/real-ruby(rb_vm_bugreport) vm_dump.c:1065 /usr/local/ruby/bin/real-ruby(rb_bug_for_fatal_signal+0xee) [0x5604a8ba927e] error.c:813 /usr/local/ruby/bin/real-ruby(sigsegv+0x4d) [0x5604a89c3ded] signal.c:964 /lib/x86_64-linux-gnu/libpthread.so.0(__restore_rt+0x0) [0x7fb5b4285420] [0x5604acd31079] ```` Like regular GC bugs, it is likely that the code GC need to trigger at a very specific place for the bug to happen. Our attempts at triggering it manually with `RubyVM::YJIT.code_gc` or to set the executable memory very low to trigger it more often didn't allow for a simpler reproduction. Both @k0kubun and @alanwu are investigating it right now. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:111278] [Ruby master Feature#19135] Support `UNIXSocket` on Windows
by ioquatix (Samuel Williams) 14 Dec '22

14 Dec '22

Issue #19135 has been updated by ioquatix (Samuel Williams). I would like to share this credit with @larskanis who was instrumental to getting this PR over the finish line. It was a true team effort with hard compatibility challenges. ---------------------------------------- Feature #19135: Support `UNIXSocket` on Windows https://bugs.ruby-lang.org/issues/19135#change-100629 * Author: ioquatix (Samuel Williams) * Status: Closed * Priority: Normal ---------------------------------------- In recent versions of Windows, the required parts are now supported: https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ This enables UNIXSocket, UNIXServer and UNIXSocket.pair on Windows. The anonymous socket is emulated on Windows using a temporary file, but it's good enough and the best we can do. The semantics is not completely identical. Because these constants are now available, more tests are run on Windows. -- https://bugs.ruby-lang.org/

1 0