July 2023 - ruby-core - ml.ruby-lang.org

[ruby-core:114276] [Ruby master Bug#19784] String#delete_prefix! problem
by inversion (Yura Babak) 01 Sep '23

01 Sep '23

Issue #19784 has been reported by inversion (Yura Babak). ---------------------------------------- Bug #19784: String#delete_prefix! problem https://bugs.ruby-lang.org/issues/19784 * Author: inversion (Yura Babak) * Status: Open * Priority: Normal * ruby -v: ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux] * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Here is the snipped and the question is in the comments: ``` ruby fp = 'with_BOM_16.txt' body = File.read(fp).force_encoding('UTF-8') p body # "\xFF\xFE1\u00001\u0000" p body.start_with?("\xFF\xFE") # true body.delete_prefix!("\xFF\xFE") # !!! why doesn't work? p body # "\xFF\xFE1\u00001\u0000" p body.start_with?("\xFF\xFE") # true body[0, 2] = '' p body # "1\u00001\u0000" p body.start_with?("\xFF\xFE") # false ``` Works same on Linux (ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]) and Windows (ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x64-mingw-ucrt]) -- https://bugs.ruby-lang.org/

6 9

[ruby-core:111269] [Ruby master Bug#19230] The openssl backend of securerandom is no longer needed
by mame (Yusuke Endoh) 31 Aug '23

31 Aug '23

Issue #19230 has been reported by mame (Yusuke Endoh). ---------------------------------------- Bug #19230: The openssl backend of securerandom is no longer needed https://bugs.ruby-lang.org/issues/19230 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- securerandom first checks if Random.urandom is available ([Line 77](https://github.com/ruby/securerandom/blob/5bfe7d6c163f7a8a45af8d2fc377f…) and if not available, it uses the openssl backend as a degeneration. However, the openssl backend does not work because it internally uses Random.urandom ([Line 55](https://github.com/ruby/securerandom/blob/5bfe7d6c163f7a8a45af8d2fc377f…) to create a seed. This issue is found by @hanachin. ``` $ ruby -ve 'def Random.urandom(*); raise; end; require "securerandom"; p SecureRandom.bytes(10)' ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux] -e:1: warning: method redefined; discarding old urandom -e:1:in `urandom': unhandled exception from /home/mame/local/lib/ruby/3.1.0/securerandom.rb:75:in `singleton class' from /home/mame/local/lib/ruby/3.1.0/securerandom.rb:42:in `<module:SecureRandom>' from /home/mame/local/lib/ruby/3.1.0/securerandom.rb:41:in `<top (required)>' from <internal:/home/mame/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from <internal:/home/mame/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from -e:1:in `<main>' ``` There has been this bug since commit:abae70d6ed63054d7d01bd6cd80c1b5b98b93ba3, which made the urandom backend as default and left the openssl backend just for degeneration. I think no one need the openssl anymore because no one has reported this bug for such a long time. How about removing it? ```diff diff --git a/lib/securerandom.rb b/lib/securerandom.rb index 07ae048634..32b76a2137 100644 --- a/lib/securerandom.rb +++ b/lib/securerandom.rb @@ -14,7 +14,6 @@ # # It supports the following secure random number generators: # -# * openssl # * /dev/urandom # * Win32 # @@ -46,21 +45,6 @@ def bytes(n) private - def gen_random_openssl(n) - @pid = 0 unless defined?(@pid) - pid = $$ - unless @pid == pid - now = Process.clock_gettime(Process::CLOCK_REALTIME, :nanosecond) - OpenSSL::Random.random_add([now, @pid, pid].join(""), 0.0) - seed = Random.urandom(16) - if (seed) - OpenSSL::Random.random_add(seed, 16) - end - @pid = pid - end - return OpenSSL::Random.random_bytes(n) - end - def gen_random_urandom(n) ret = Random.urandom(n) unless ret @@ -77,13 +61,7 @@ def gen_random_urandom(n) Random.urandom(1) alias gen_random gen_random_urandom rescue RuntimeError - begin - require 'openssl' - rescue NoMethodError - raise NotImplementedError, "No random device" - else - alias gen_random gen_random_openssl - end + raise NotImplementedError, "No random device" end public :gen_random ``` -- https://bugs.ruby-lang.org/

6 5

[ruby-core:114248] [Ruby master Feature#19777] Make `Kernel#lambda` raise when called without a literal block
by alanwu (Alan Wu) 24 Aug '23

24 Aug '23

Issue #19777 has been reported by alanwu (Alan Wu). ---------------------------------------- Feature #19777: Make `Kernel#lambda` raise when called without a literal block https://bugs.ruby-lang.org/issues/19777 * Author: alanwu (Alan Wu) * Status: Open * Priority: Normal ---------------------------------------- Since 3.0.0, released in 2020, calling `Kernel#lambda` without a literal block has been issuing a deprecation warning: ```ruby Warning[:deprecated] = true def foo(&b) lambda(&b) end foo {} # => test.rb:2: warning: lambda without a literal block is deprecated; use the proc without lambda instead ``` I think enough time has passed and we should make it raise in all situations where it currently issues a deprecation warning. The original decision to deprecate is here: https://bugs.ruby-lang.org/issues/15973#note-46 The new behavior allows one to predict whether `Kernel#lambda` will return by inspecting its direct caller, checking whether the call site has a literal block. It will remove some hard-to-predict cases where `Kernel#lambda` receives a non-literal block forwarded with `super` or `rb_funcall_passing_block`. The method will always return a lambda, if it returns. However, note that `send` will be a special exception in this new model: ```ruby Warning[:deprecated] = true singleton_class.send(:public, :lambda) p (send(:lambda) {}).lambda? # => true without warning p (public_send(:lambda) {}).lambda? # => true with warning, would raise instead ``` This newer model is friendlier to some optimization we're investigating for YJIT as it has fewer moving parts. -- https://bugs.ruby-lang.org/

4 3

[ruby-core:113954] [Ruby master Misc#19740] Block taking methods can't differentiate between a non-local return and a throw
by byroot (Jean Boussier) 24 Aug '23

24 Aug '23

Issue #19740 has been reported by byroot (Jean Boussier). ---------------------------------------- Misc #19740: Block taking methods can't differentiate between a non-local return and a throw https://bugs.ruby-lang.org/issues/19740 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- Opening this as Misc, as at this stage I don't have a fully formed feature request. Ref: https://github.com/ruby/ruby/commit/1a3bcf103c582b20e9ea70dfed0ee68b24243f55 Ref: https://github.com/ruby/timeout/pull/30 Ref: https://github.com/rails/rails/pull/29333 ### Context Rails has this problem in the Active Record transaction API. The way it works is that it yields to a block, and if no error was raised the SQL transaction is committed, otherwise it's rolled back: ```ruby User.transaction do do_thing end # COMMIT ``` or ```ruby User.transaction do raise SomeError end # ROLLBACK ``` The problem is that there are more ways a method can be exited: ```ruby User.transaction do return # non-local exit end ``` ```ruby User.transaction do throw :something end ``` In the case of a non-local return, we'd want to commit the transaction, but in the case of a throw, particularly since it's internally used by `Timeout.timeout` since Ruby 2.1, we'd rather consider that an error and rollback. But as far as I'm aware, there is not way to distinguish the two cases. ```ruby def transaction returned = false yield returned = true ensure if $! # error was raised elsif returned # no uniwnd else # non-local return or throw, don't know end end ``` I think it could be useful to have a way to access the currently thrown object, similar to `$!` for such cases, or some other way to tell what is going on. There is some discussion going on in https://github.com/ruby/timeout/pull/30 about whether `Timeout` should throw or raise, and that may solve part of the problem, but regardless of where this leads, I think being able to check if something is being thrown would be valuable. cc @matthewd FYI @jeremyevans0 @Eregon -- https://bugs.ruby-lang.org/

3 2

[ruby-core:112399] [Ruby master Bug#19436] Call Cache for singleton methods can lead to "memory leaks"
by byroot (Jean Boussier) 23 Aug '23

23 Aug '23

Issue #19436 has been reported by byroot (Jean Boussier). ---------------------------------------- Bug #19436: Call Cache for singleton methods can lead to "memory leaks" https://bugs.ruby-lang.org/issues/19436 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads. ### Minimal Reproduction ```ruby module Foo def bar end end def call_bar(obj) # Here the call cache we'll keep a ref on the method_entry # which then keep a ref on the singleton_class, making that # instance immortal until the method is called again with # another instance. # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT obj.bar end obj = Object.new obj.extend(Foo) call_bar(obj) id = obj.object_id obj = nil 4.times { GC.start } p ObjectSpace._id2ref(id) ``` ### Explanation Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class and in the cache of a singleton class, it keeps a strong reference onto the `attached_object` (instance). This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver. If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak. And if the `attached_object` is big, the wasted memory can be very substantial. ### Practical Implications Once relative common API impacted by this is [Rails' `extending` API](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#met…. This API allow to extend a "query result set" with a module. These query results set can sometimes be very big, especially since they keep references to the instantiated `ActiveRecord::Base` instances etc. ### Possible Solutions #### Only keep a weak reference to the CME The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work. However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is. Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class: ```c # vm_callinfo.h:275 struct rb_callcache { const VALUE flags; /* inline cache: key */ const VALUE klass; // should not mark it because klass can not be free'd // because of this marking. When klass is collected, // cc will be cleared (cc->klass = 0) at vm_ccs_free(). ``` So it appears that the class being also marked through the CME is some kind of oversight? #### Don't cache based on some heuristics If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a `Class` or `Module`. It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long. #### Make `Class#attached_object` a weak reference Alternatively we could make the `attached_object` a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario. The downside is that `Class#attached_object` was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit. cc @peterzhu2118 @ko1 -- https://bugs.ruby-lang.org/

8 24

[ruby-core:114290] [Ruby master Feature#19785] Deprecate `RUBY_GC_HEAP_INIT_SLOTS `
by peterzhu2118 (Peter Zhu) 23 Aug '23

23 Aug '23

Issue #19785 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19785: Deprecate `RUBY_GC_HEAP_INIT_SLOTS ` https://bugs.ruby-lang.org/issues/19785 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/8116 The `RUBY_GC_HEAP_INIT_SLOTS` environment variable is replaced by `RUBY_GC_HEAP_INIT_SIZE_%d_SLOTS` (introduced in commit [3ab3455](https://github.com/ruby/ruby/commit/3ab34551450c7a3a3e1ae0b24bf6b7…) which allows for more fine-grained tuning of size pools. So it doesn't make sense to keep the environment variable `RUBY_GC_HEAP_INIT_SLOTS`. I'm proposing to deprecate `RUBY_GC_HEAP_INIT_SLOTS` and eventually remove it. -- https://bugs.ruby-lang.org/

3 3

[ruby-core:114274] [Ruby master Feature#19783] Weak References in the GC
by peterzhu2118 (Peter Zhu) 23 Aug '23

23 Aug '23

Issue #19783 has been reported by peterzhu2118 (Peter Zhu). ---------------------------------------- Feature #19783: Weak References in the GC https://bugs.ruby-lang.org/issues/19783 * Author: peterzhu2118 (Peter Zhu) * Status: Open * Priority: Normal ---------------------------------------- GitHub PR: https://github.com/ruby/ruby/pull/8113 I'm proposing support for weak references in the Ruby garbage collector. This feature adds a new function called `void rb_gc_mark_weak(VALUE *ptr)` which marks `*ptr` as weak, meaning that if no other object strongly marks `*ptr` (using `rb_gc_mark` or `rb_gc_mark_movable`), then it will be overwritten with `*ptr = Qundef`. Weak references are implemented using a buffer in `objspace` that stores all the `ptr` in the latest marking phase. After marking has finished, we iterate over the buffer and check if the `*ptr` is a dead object. If it is, then we set `*ptr = Qundef`. Weak references are implemented on the callable method entry (CME) of callcaches, which fixes issue #19436. Weak references are also implemented on `ObjectSpace::WeakMap` and `ObjectSpace::WeakKeyMap`, which have: - Significantly simpler implementations because we no longer need to have multiple tables and do not need to define finalizers on the objects. - Support for compaction because finalizers pin objects and we no longer need to define finalizers on the objects. - Much faster performance (see [benchmarks](#microbenchmarks)). ## Benchmark results ### YJIT-bench We see largely no change in performance or memory usage after this feature. ``` -------------- --------- ---------- --------- ----------- ---------- --------- -------------- ----------- bench base (ms) stddev (%) RSS (MiB) branch (ms) stddev (%) RSS (MiB) branch 1st itr base/branch activerecord 72.3 2.2 51.9 72.9 2.2 51.9 0.99 0.99 chunky-png 889.2 0.3 43.9 874.5 0.3 42.5 1.02 1.02 erubi-rails 21.2 13.5 90.7 21.0 13.3 90.9 1.01 1.01 hexapdf 2557.0 0.8 157.1 2559.2 0.7 197.1 1.01 1.00 liquid-c 65.2 0.4 34.5 65.4 0.4 34.5 0.99 1.00 liquid-compile 62.5 0.4 30.9 62.2 0.4 31.0 1.00 1.01 liquid-render 164.6 0.4 33.1 162.6 0.3 33.1 1.01 1.01 mail 133.3 0.1 46.4 134.4 0.2 46.4 1.03 0.99 psych-load 2066.6 0.2 31.6 2083.6 0.1 31.6 0.99 0.99 railsbench 2027.0 0.5 88.8 2019.4 0.5 89.0 1.01 1.00 ruby-lsp 65.6 3.0 90.1 65.4 3.1 88.5 1.00 1.00 sequel 73.1 1.1 36.6 73.1 1.1 36.6 1.00 1.00 -------------- --------- ---------- --------- ----------- ---------- --------- -------------- ----------- ``` ### Microbenchmarks We can see signficantly improved performance in `ObjectSpace::WeakMap`, with `ObjectSpace::WeakMap#[]=` being nearly 3x faster. Base: ``` ObjectSpace::WeakMap#[]= 1.037M (± 0.5%) i/s - 5.262M in 5.072833s ObjectSpace::WeakMap#[] 12.367M (± 0.9%) i/s - 62.479M in 5.052365s ``` Branch: ``` ObjectSpace::WeakMap#[]= 3.054M (± 0.3%) i/s - 15.448M in 5.058783s ObjectSpace::WeakMap#[] 15.796M (± 4.8%) i/s - 79.245M in 5.028583s ``` Code: ```ruby require "bundler/inline" gemfile do source "https://rubygems.org" gem "benchmark-ips" end wmap = ObjectSpace::WeakMap.new key = Object.new val = Object.new wmap[key] = val Benchmark.ips do |x| x.report("ObjectSpace::WeakMap#[]=") do |times| i = 0 while i < times wmap[Object.new] = Object.new i += 1 end end x.report("ObjectSpace::WeakMap#[]") do |times| i = 0 while i < times wmap[key] wmap[val] # does not exist i += 1 end end end ``` -- https://bugs.ruby-lang.org/

4 5

[ruby-core:114168] [Ruby master Misc#19766] DevMeeting-2023-08-24
by mame (Yusuke Endoh) 22 Aug '23

22 Aug '23

Issue #19766 has been reported by mame (Yusuke Endoh). ---------------------------------------- Misc #19766: DevMeeting-2023-08-24 https://bugs.ruby-lang.org/issues/19766 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal ---------------------------------------- # The next dev meeting **Date: 2023/08/24 13:00-17:00** (JST) Log: *TBD* - Dev meeting *IS NOT* a decision-making place. All decisions should be done at the bug tracker. - Dev meeting is a place we can ask Matz, nobu, nurse and other developers directly. - Matz is a very busy person. Take this opportunity to ask him. If you can not attend, other attendees can ask instead of you (if attendees can understand your issue). - We will write a record of the discussion in the file or to each ticket in English. - All activities are best-effort (keep in mind that most of us are volunteer developers). - The date, time and place of the meeting are scheduled according to when/where we can reserve Matz's time. - *DO NOT* discuss then on this ticket, please. # Call for agenda items If you have a ticket that you want matz and committers to discuss, please post it into this ticket in the following format: ``` * [Ticket ref] Ticket title (your name) * Comment (A summary of the ticket, why you put this ticket here, what point should be discussed, etc.) ``` Example: ``` * [Feature #14609] `Kernel#p` without args shows the receiver (ko1) * I feel this feature is very useful and some people say :+1: so let discuss this feature. ``` - It is recommended to add a comment by 2023/08/21. We hold a preparatory meeting to create an agenda a few days before the dev-meeting. - The format is strict. We'll use [this script to automatically create an markdown-style agenda](https://gist.github.com/mame/b0390509ce1491b43610b9ebb665eb86). We may ignore a comment that does not follow the format. - Your comment is mandatory. We cannot read all discussion of the ticket in a limited time. We appreciate it if you could write a short summary and update from a previous discussion. -- https://bugs.ruby-lang.org/

8 8

[ruby-core:114113] [Ruby master Bug#19761] AddressSanitizer fails with Thread and Process
by nobu (Nobuyoshi Nakada) 20 Aug '23

20 Aug '23

Issue #19761 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19761: AddressSanitizer fails with Thread and Process https://bugs.ruby-lang.org/issues/19761 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- With AddressSanitizer on macOS, this code crashes. ```ruby system('') Thread.new {} 300_000.times do String.new(capacity: 1000) end ``` ``` ==62870==ERROR: AddressSanitizer failed to deallocate 0x8000 (32768) bytes at address 0x62d00000a400 AddressSanitizer: CHECK failed: sanitizer_posix.cpp:61 "(("unable to unmap" && 0)) != (0)" (0x0, 0x0) (tid=857549) <empty stack> Assertion Failed: ../debug/vm_core.h:1853:rb_current_execution_context:!expect_ec || ec != NULL ruby 3.3.0dev (2023-07-08T02:54:37Z tmp 0083edc3a0) [x86_64-darwin22] -- Crash Report log information -------------------------------------------- See Crash Report log file in one of the following locations: * ~/Library/Logs/DiagnosticReports * /Library/Logs/DiagnosticReports for more details. Don't forget to include the above Crash Report log file in bug reports. -- C level backtrace information ------------------------------------------- ``` -- https://bugs.ruby-lang.org/

2 1

[ruby-core:114218] [Ruby master Bug#19770] TLS / Certificate Hostname Verification against IP fails with "address family must be specified"
by Schachi65 (Joachim Schachermayer) 17 Aug '23

17 Aug '23

Issue #19770 has been reported by Schachi65 (Joachim Schachermayer). ---------------------------------------- Bug #19770: TLS / Certificate Hostname Verification against IP fails with "address family must be specified" https://bugs.ruby-lang.org/issues/19770 * Author: Schachi65 (Joachim Schachermayer) * Status: Open * Priority: Normal * ruby -v: ruby 3.1.4 * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN ---------------------------------------- I dont't use Ruby directly but indirectly via Fluentd-1.16.1 (https://www.fluentd.org/) Fluentd uses Ruby 3.1.4. The problem is, that a TLS connection from a Fluentd Client to a Fluentd Server fails, if the connection is established using the IP address of the server and tls_verify_hostname is switched to on. In the servers certificate, in the SubjectAlternativeNames extention, there is the servers IP address set with the correct value. I get the error message " **address family must be specified** ". **Probable cause** : ext/openssl/lib/openssl/ **ssl.rb** line 288: Here the Methode IPAddr.new(hostname) is called with just one argument: return true if san.value == IPAddr.new( **hostname** ).hton lib/ **ipaddr.rb** line 594: Here the exception is raised with the observed error message: when Socket::AF_UNSPEC **raise** AddressFamilyError, " **address family must be specified** " The exception is raised, because the variable "family" has the value " **Socket::AF_UNSPEC** " because thats the default value of "family" if there is no additional parameter in the call to "initialize()". The documumentation of "initialize()" says in line 575/576/577: "Although the address family is determined automatically from a specified string, you can specify one explicitly by the optional second argument." But that automatic address determination is done too late. If no second parameter is given, it is not done at all because that exception is raised (assumption: the addr i always a string). **Probable solution** : If the address family is set to Socket::AF_UNSPEC with the call of "initialize()", dont't raise an exception now but do the in the comment announced automatic adress family detection here. -- https://bugs.ruby-lang.org/

3 3