- ruby-core - ml.ruby-lang.org

[ruby-core:119135] CPU-bound load affects the timeout thread
by yuri.kanivetsky＠gmail.com 12 Sep '24

12 Sep '24

Hi, I'm not sure if it's a bug. Take a look at the following gist: https://gist.github.com/x-yuri/253f76df6287441f64b5eaee418813c0 It's supposedly a minimal reproducible example of what I ran into a couple of days ago. In my case it was basically a sort of a background jobs service. It received messages, processed them and sent the result to another service. Every message was processed and sent in a separate thread. And it made a number of http requests as part of processing messages (net/http which uses timeout()). It worked normally for a couple of hours (10-20 threads), but at some point the number of threads reached the thread pool's max_threads (320), http requests started to time out (later then they should, after 60-100 seconds instead of 3 seconds), and probably things started to get slow. Then I noticed the following line: patterns.any? { |pattern| quote.match(pattern['regex']) } ? 1 : 0 It was performed for every quote (10 times) in every thread, there were 300+ patterns and they (pattern['regex']) were strings. I moved converting strings to regexps to initialization (before starting to accept messages and creating threads) and it seems to work now with ~10 active threads. I'm trying to understand what exactly happened and how to avoid it or what awaits me in the future. Is there some critical load that breaks things? I guess in theory the number of threads should increase linearly with the load. But it looks like in this case there's some critical load that just makes things stop working. Or maybe for some time some negative effects get accumulated and then things break. My conjecture is that it has something to do with GIL. But what exactly happens? Or what can I do to further investigate the issue? I'm going to try to run ruby with RUBY_DEBUG_LOG and try to examine the debug output. I guess I need to figure out why the timeout thread doesn't always get enough time. Does something block the threads? How are they scheduled? When do they switch? Any pointers or links to the code are welcome. Regards, Yuri

2 2

[ruby-core:119129] [Ruby master Feature#20724] Update to Unicode 16.0
by duerst 12 Sep '24

12 Sep '24

Issue #20724 has been reported by duerst (Martin Dürst). ---------------------------------------- Feature #20724: Update to Unicode 16.0 https://bugs.ruby-lang.org/issues/20724 * Author: duerst (Martin Dürst) * Status: Open * Assignee: duerst (Martin Dürst) ---------------------------------------- Unicode 16.0 has been released (see https://www.unicode.org/versions/Unicode16.0.0/) We should update to 16.0 after updating to 15.1. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118388] [Ruby master Feature#20594] A new String method to append bytes while preserving encoding
by byroot (Jean Boussier) 11 Sep '24

11 Sep '24

Issue #20594 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20594: A new String method to append bytes while preserving encoding https://bugs.ruby-lang.org/issues/20594 * Author: byroot (Jean Boussier) * Status: Open ---------------------------------------- ### Context When working with binary protocols such as `protobuf` or `MessagePack`, you may often need to assemble multiple strings of different encoding: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title << 255 << body.bytesize << body end end Post.new("Hello", "World").serialize("somedata".b) # => "somedata\xFF\x05Hello\xFF\x05World" #<Encoding:ASCII-8BIT> ``` The problem in the above case, is that because `Encoding::ASCII_8BIT` is declared as ASCII compatible, if one of the appended string contains bytes outside the ASCII range, string is automatically promoted to another encoding, which then leads to encoding issues: ```ruby Post.new("H€llo", "Wôrld").serialize("somedata".b) # => incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError) ``` In many cases, you want to append to a String without changing the receiver's encoding. The issue isn't exclusive to binary protocols and formats, it also happen with ASCII protocols that accept arbitrary bytes inline, like Redis's RESP protocol or even HTTP/1.1. ### Previous discussion There was a similar feature request a while ago, but it was abandoned: https://bugs.ruby-lang.org/issues/14975 ### Existing solutions You can of course always cast the strings you append to avoid this problem: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize << title.b << 255 << body.bytesize << body.b end end ``` But this cause a lot of needless allocations. You'd think you could also use `bytesplice`, but it actually has the same issue: ```ruby Post = Struct.new(:title, :body) do def serialize(buf) buf << 255 << title.bytesize buf.bytesplice(buf.bytesize, title.bytesize, title) buf << 255 << body.bytesize buf.bytesplice(buf.bytesize, body.bytesize, title) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => 'String#bytesplice': incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8 (Encoding::CompatibilityError) ``` And even if it worked, it would be very unergonomic. ### Proposal: a `byteconcat` method A solution to this would be to add a new `byteconcat` method, that could be shimed as: ```ruby class String def byteconcat(*strings) strings.map! do |s| if s.is_a?(String) && s.encoding != encoding s.dup.force_encoding(encoding) else s end end concat(*strings) end end Post = Struct.new(:title, :body) do def serialize(buf) buf.byteconcat( 255, title.bytesize, title, 255, body.bytesize, body, ) end end Post.new("H€llo", "Wôrld").serialize("somedata".b) # => "somedata\xFF\aH\xE2\x82\xACllo\xFF\x06W\xC3\xB4rld" #<Encoding:ASCII-8BIT> ``` But of course a builtin implementation wouldn't need to dup the arguments. Like other `byte*` methods, it's the responsibility of the caller to ensure the resulting string has a valid encoding, or to deal with it if not. ### Method name and signature #### Name This proposal suggests `String#byteconcat`, to mirror `String#concat`, but other names are possible: - `byteappend` (like `Array#append`) - `bytepush` (like `Array#push`) #### Signature This proposal makes `byteconcat` accept either `String` or `Integer` (in char range) arguments like `concat`. I believe it makes sense for consistency and also because it's not uncommon for protocols to have some byte based segments, and Integers are more convenient there. The proposed method also accept variable arguments for consistency with `String#concat`, `Array#push`, `Array#append`. The proposed method returns self, like `concat` and others. ### YJIT consideration I consulted @maximecb about this proposal, and according to her, accepting variable arguments makes it harder for YJIT to optimize. I suspect consistency with other APIs trumps the performance consideration, but I think it's worth mentioning. -- https://bugs.ruby-lang.org/

12 31

[ruby-core:119112] [Ruby master Feature#18368] Range#step semantics for non-Numeric ranges
by knu (Akinori MUSHA) 10 Sep '24

10 Sep '24

Issue #18368 has been updated by knu (Akinori MUSHA). The compatibility with `(:a .. :z).step(3)` has also been restored. https://github.com/ruby/ruby/pull/11573 ---------------------------------------- Feature #18368: Range#step semantics for non-Numeric ranges https://bugs.ruby-lang.org/issues/18368#change-109705 * Author: zverok (Victor Shepelev) * Status: Open ---------------------------------------- I am sorry if the question had already been discussed, can't find the relevant topic. "Intuitively", this looks (for me) like a meaningful statement: ```ruby (Time.parse('2021-12-01')..Time.parse('2021-12-24')).step(1.day).to_a # ^^^^^ or just 24*60*60 ``` Unfortunately, it doesn't work with "TypeError (can't iterate from Time)". Initially it looked like a bug for me, but after digging a bit into code/docs, I understood that `Range#step` has an odd semantics of "advance the begin N times with `#succ`, and yield the result", with N being always integer: ```ruby ('a'..'z').step(3).first(5) # => ["a", "d", "g", "j", "m"] ``` The fact that semantic is "odd" is confirmed by the fact that for Float it is redefined to do what I "intuitively" expected: ```ruby (1.0..7.0).step(0.3).first(5) # => [1.0, 1.3, 1.6, 1.9, 2.2] ``` (Like with [`Range#===` some time ago](https://bugs.ruby-lang.org/issues/14575), I believe that to be a strong proof of the wrong generic semantics, if for numbers the semantics needed to be redefined completely.) Another thing to note is that "skip N elements" seem to be rather "generically Enumerable-related" yet it isn't defined on `Enumerable` (because nobody needs this semantics, typically!) Hence, two questions: * Can we redefine generic `Range#step` to new semantics (of using `begin + step` iteratively)? It is hard to imagine the amount of actual usage of the old behavior (with String?.. to what end?) in the wild * If the answer is "no", can we define a new method with new semantics, like, IDK, `Range#over(span)`? **UPD:** More examples of useful behavior (it is NOT only about core `Time` class): ```ruby require 'active_support/all' (1.minute..20.minutes).step(2.minutes).to_a #=> [1 minute, 3 minutes, 5 minutes, 7 minutes, 9 minutes, 11 minutes, 13 minutes, 15 minutes, 17 minutes, 19 minutes] require 'tod' (Tod::TimeOfDay.parse("8am")..Tod::TimeOfDay.parse("10am")).step(30.minutes).to_a #=> [#<Tod::TimeOfDay 08:00:00>, #<Tod::TimeOfDay 08:30:00>, #<Tod::TimeOfDay 09:00:00>, #<Tod::TimeOfDay 09:30:00>, #<Tod::TimeOfDay 10:00:00>] require 'matrix' (Vector[1, 2, 3]..).step(Vector[1, 1, 1]).take(3) #=> [Vector[1, 2, 3], Vector[2, 3, 4], Vector[3, 4, 5]] require 'unitwise' (Unitwise(0, 'km')..Unitwise(1, 'km')).step(Unitwise(100, 'm')).map(&:to_s) #=> ["0 km", "1/10 km", "1/5 km", "3/10 km", "2/5 km", "0.5 km", "3/5 km", "7/10 km", "4/5 km", "9/10 km", "1 km"] ``` **UPD:** Responding to discussion points: **Q:** Matz is concerned that the proposed simple definition will be confusing with the classes where `+` is redefined as concatenation. **A:** I believe that simplicity of semantics and ease of explaining ("it just uses `+` underneath, whatever `+` does, will be performed") will make the confusion minimal. **Q:** Why not introduce new API requirement (like "class of range's `begin` should implement `increment` method, and then it will be used in `step`) **A:** require *every* gem author to change *every* of their objects' behavior. For that, they should be aware of the change, consider it important enough to care, clearly understand the necessary semantics of implementation, have a resource to release a new version... Then all users of all such gems would be required to upgrade. The feature would be DOA (dead-on-arrival). The two alternative ways I am suggesting: change the behavior of `#step` or introduce a new method with desired behavior: 1. Easy to explain and announce 2. Require no other code changes to immediately become useful 3. With something like [backports](https://github.com/marcandre/backports) or [ruby-next](https://github.com/ruby-next/ruby-next) easy to start using even in older Ruby version, making the code more expressive even before it would be possible for some particular app/compny to upgrade to (say) 3.2 All examples of behavior from the code above are real `irb` output with monkey-patched `Range#step`, demonstrating how little change will be needed to code outside of the `Range`. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:118796] [Ruby master Bug#20667] Backport ReXML CVE fixes
by vo.x (Vit Ondruch) 10 Sep '24

10 Sep '24

Issue #20667 has been reported by vo.x (Vit Ondruch). ---------------------------------------- Bug #20667: Backport ReXML CVE fixes https://bugs.ruby-lang.org/issues/20667 * Author: vo.x (Vit Ondruch) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- It would be nice to have the recent ReXML CVE fixes backported everywhere. BTW it is surprising that ReXML was recently bumped in 3.1 / 3.2 branches, but 3.3 brach stays with older ReXML 3.2. -- https://bugs.ruby-lang.org/

4 4

[ruby-core:119106] [Ruby master Feature#15554] warn/error passing a block to a method which never use a block
by Dan0042 (Daniel DeLorme) 10 Sep '24

10 Sep '24

Issue #15554 has been updated by Dan0042 (Daniel DeLorme). This is a bit off-topic, but I've often thought it would be nice to have multiple "levels" of warnings. So we could have the fine-grained `Warning[:strict_unused_block]` which, if unset, defaults to the coarse-grained `Warning[:strict]` which, if unset, defaults to `$VERBOSE`. BTW "pedantic" might sound funny but it's both derogatory and undescriptive, and used here in a different sense to the -pedantic option of gcc. Maybe something like "potential_bug" would be better? ---------------------------------------- Feature #15554: warn/error passing a block to a method which never use a block https://bugs.ruby-lang.org/issues/15554#change-109695 * Author: ko1 (Koichi Sasada) * Status: Assigned * Assignee: matz (Yukihiro Matsumoto) ---------------------------------------- # Abstract Warn or raise an ArgumentError if block is passed to a method which does not use a block. In other words, detect "block user methods" implicitly and only "block user methods" can accept a block. # Background Sometimes, we pass a block to a method which ignores the passed block accidentally. ``` def my_open(name) open(name) end # user hopes it works as Kernel#open which invokes a block with opened file. my_open(name){|f| important_work_with f } # but simply ignored... ``` To solve this issue, this feature request propose showing warnings or raising an exception on such case. Last developer's meeting, matz proposed `&nil` which declares this method never receive a block. It is explicit, but it is tough to add this `&nil` parameter declaration to all of methods (do you want to add it to `def []=(i, e, &nil)`?). (I agree `&nil` is valuable on some situations) # Spec ## Define "use a block" methods We need to define which method accepts a block and which method does not. * (1) method has a block parameter (`&b`) * (2) method body has `yield' * (3) method body has `super` (ZSUPER in internal terminology) or `super(...)` * (4) method body has singleton method (optional) (1) and (2) is very clear. I need to explain about (3) and (4). (3). `super` (ZSUPER) passes all parameters as arguments. So there is no surprise that which can accept `block`. However `super(...)` also passes a block if no explicit block passing (like `super(){}` or `super(&b)`) are written. I'm not sure we need to continue this strange specification, but to keep compatibility depending this spec, I add this rule. (4). surprisingly, the following code invoke a block: ``` def foo class << Object.new yield end end foo{ p :ok } #=> :ok ``` I'm also not sure we need to keep this spec, but to allow this spec, I added (4) rule. Strictly speaking, it is not required, but we don't keep the link from singleton class ISeq to lexical parent iseq now, so I added it. ## Exceptional cases A method called by `super` doesn`t warn warning even if this method doesn't use a block. The rule (3) can pass blocks easily and there are many methods don`t use a block. So my patch ignores callings by `super`. ## corner cases There are several cases to use block without (1)-(4) rules. ### `Proc.new/proc/lambda` without a block Now it was deprecated in r66772 (commit:9f1fb0a17febc59356d58cef5e98db61a3c03550). Related discussion: [Bug #15539] ### `block_given?` `block_given?` expects block, but I believe we use it with `yield` or a block parameter. If you know the usecase without them, please tell us. ### `yield` in `eval` We can't know `yield` (or (3), (4) rule) in an `eval` evaluating string at calling time. ``` def foo eval('yield`) end foo{} # at calling time, # we can't know the method foo can accept a block or not. ``` So I added a warning to use `yield` in `eval` like that: `test.rb:4: warning: use yield in eval will not be supported in Ruby 3.` Workaround is use a block parameter explicitly. ``` def foo &b eval('b.call') end foo{ p :ok } ``` # Implementation Strategy is: * [compile time] introduce `iseq::has_yield` field and check it if the iseq (or child iseq) contains `yield` (or something) * [calling time] if block is given, check `iseq::has_yield` flag and show warning (or raise an exception) https://gist.github.com/ko1/c9148ad0224bf5befa3cc76ed2220c0b On this patch, now it raises an error to make it easy to detect. It is easy to switch to show the warning. # Evaluation and discussion I tried to avoid ruby's tests. https://gist.github.com/ko1/37483e7940cdc4390bf8eb0001883786 Here is a patch. There are several patterns to avoid warnings. ## tests for `block_given?`, `Proc.new` (and similar) without block Add a dummy block parameter. It is test-specific issue. ## empty `each` Some tests add `each` methods do not `yield`, like: `def each; end`. Maybe test-specific issue, and adding a dummy block parameter. ## Subtyping / duck typing https://github.com/ruby/ruby/blob/c01a5ee85e2d6a7128cccafb143bfa694284ca87/… This `parse` method doesn't use `yield`, but other sub-type's `parse` methods use. ## `super` with `new` method https://gist.github.com/ko1/37483e7940cdc4390bf8eb0001883786#file-tests-pat… This method override `Class#new` method and introduce a hook with block (yield a block in this hook code). https://github.com/ruby/ruby/blob/trunk/lib/rubygems/package/tar_writer.rb#… In this method, call `super` and it also passing a block. However, called `initialize` doesn't use a block. ## Change robustness This change reduce robustness for API change. `Delegator` requires to support `__getobj__` for client classes. Now `__getobj__` should accept block but most of `__getobj__` clients do not call given block. https://github.com/ruby/ruby/blob/trunk/lib/delegate.rb#L80 This is because of delegator.rb's API change. https://gist.github.com/ko1/37483e7940cdc4390bf8eb0001883786#file-tests-pat… Nobu says calling block is not required (ignoring a block is no problem) so it is not a bug for delegator client classes. ## Found issues. ``` [ 2945/20449] Rinda::TestRingServer#test_do_reply = 0.00 s 1) Error: Rinda::TestRingServer#test_do_reply: ArgumentError: passing block to the method "with_timeout" (defined at /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:787) is never used. /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:635:in `test_do_reply' [ 2946/20449] Rinda::TestRingServer#test_do_reply_local = 0.00 s 2) Error: Rinda::TestRingServer#test_do_reply_local: ArgumentError: passing block to the method "with_timeout" (defined at /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:787) is never used. /home/ko1/src/ruby/trunk/test/rinda/test_rinda.rb:657:in `test_do_reply_local' [10024/20449] TestGemRequestSetGemDependencyAPI#test_platform_mswin = 0.01 s 3) Error: TestGemRequestSetGemDependencyAPI#test_platform_mswin: ArgumentError: passing block to the method "util_set_arch" (defined at /home/ko1/src/ruby/trunk/lib/rubygems/test_case.rb:1053) is never used. /home/ko1/src/ruby/trunk/test/rubygems/test_gem_request_set_gem_dependency_api.rb:655:in `test_platform_mswin' [10025/20449] TestGemRequestSetGemDependencyAPI#test_platforms = 0.01 s 4) Error: TestGemRequestSetGemDependencyAPI#test_platforms: ArgumentError: passing block to the method "util_set_arch" (defined at /home/ko1/src/ruby/trunk/lib/rubygems/test_case.rb:1053) is never used. /home/ko1/src/ruby/trunk/test/rubygems/test_gem_request_set_gem_dependency_api.rb:711:in `test_platforms' ``` These 4 detection show the problem. `with_timeout` method (used in Rinda test) and `util_set_arch` method (used in Rubygems test) simply ignore the given block. So these tests are simply ignored. I reported them. (https://github.com/rubygems/rubygems/issues/2601) ## raise an error or show a warning? At least, Ruby 2.7 should show warning for this kind of violation with `-w`. How about for Ruby3? -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119105] to all devs! need help
by mikey 10 Sep '24

10 Sep '24

i have an interesting piece of androud hardware and i have a plan foe a fulk dep!oyment of its architechture but i want for it to be a controlled delloyment and for it to be released to only special people and also have ruby specifically embeded in its functionality. i am concerned over its release because its a build suplosedly from the underworld and its not a typical build . its not even sold on any market but its from another dimensional for of the same company. its really technical stuff and i want to keep it exclusive to ruby and matz because of his japan roots and the Yakuza background. i would like to establish a better relationship between Japan and the US. All devs will have access to this when a complete break down is completed. anyone specialized in android architecture i would exclusively be interested in your assistance. i also question if a debian type based build could be possible to integrate as its base build. i am not sure of its prob!em to associate witn ruby i would like to offer my designs as i said exclusively for ruby devs. i am in need of secure devices to further my work. i am afraid of insecure electronics and bad people trying to steal information. i never knew how difficult it was in the tech industry. god bless the beloved tech icons of our day and their struggles. please send support to my home address:Michae! Martinez (muerte) PO box 702 Aromas Ca 95004 i be!ieve there is a huge exploit, a huge bug bounty that is going to caus3 a disaster soon on behalf of apple if anyone is interested i collexting the reward i believe its in the AMDE files being deployed in the latest updates for mos telectronics so that apple users have backdoor exploits into everytning with spiritual assistance. their is no need for it and its a lawsuit really. i would like all the help i can get. i am sure kts the biggest bug bounty ever paid. file it and just sit back and enjoy. i am too unsecure to log in to my dev accounts. cant lose its just a small report. good luck ruby squad!!!

1 0

[ruby-core:119104] [Ruby master Feature#13820] Add a nil coalescing operator
by TylerRick (Tyler Rick) 10 Sep '24

10 Sep '24

Issue #13820 has been updated by TylerRick (Tyler Rick). Ruby already has the safe navigation operator `?.`, added back in Ruby 2.3, which was definitely a step in the right direction with regards to first-class `nil`-aware operations. Now we just need a null coalescing operator like most other popular languages have and our support for null-aware operators will be complete! I would jump for joy. Please add this. Adding this operator would make it elegantly concise and trivially simple to just write what you mean the *first* time: "if LHS is **nil** , then use (`??`) / initialize using (`??=`) this default value" — instead of being tempted to use the *almost*-correct but subtly different-meaning ("if LHS is **falsy** , then...") alternative that we've all ended up using instead, `something || default`. And then having to "upgrade"/fix the logic to something more complicated/verbose when you inevitably realize that `false` values were inadvertently *also* using the logic path you intended only for `nil` values. As akim (Akim Demaille) summed up perfectly in #note-10: > > In practice when people use `||`, they [really] mean `??` It should be easier to do the right thing here, and *not* so easy (by virtue of it being the *only* obvious/easy solution provided by the syntax) to shoot yourself in the foot by inadvertently allowing `false` values to use the same logic path that you only *wanted* to be used for `nil`... If we *had* the syntax to allow it, it would be a lot easier / more common to allow `false` values as *distinct* values from `nil` — and allow them (without any extra effort) to arrive all the way inside your method/etc. instead of getting mistakenly lumped together with `nil` values (and accidentally triggering the "default value" path). As Michael Harrington [put it](https://github.com/kibiz0r/coalesce)... :) > Ruby, as a community, unfairly discriminates against `false` [...] > `false` is a perfectly non-nil object, and you're excluding it out of convenience! --- I would prefer spelling it `??` since that seems the most consistent with the existing `?.` operator, and seems like the most popular spelling (used by C#, JavaScript, PHP, PowerShell, and Swift). (`?:` as in [Kotlin](https://kotlinlang.org/docs/null-safety.html#elvis-operator) would be a fun spelling, but IMHO it look too much like the ternary operator `? :`, which of course is different since it checks for *truthy* values rather than *nullish* values, so that similarity could be surprising.) Shoot, even ***JavaScript*** has [had this operator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference… since ECMAScript 2020. Are we really going to let JavaScript and PHP be more developer-friendly than Ruby on this point??? Say it ain't so. ---------------------------------------- Feature #13820: Add a nil coalescing operator https://bugs.ruby-lang.org/issues/13820#change-109694 * Author: williamn (William Newbery) * Status: Open ---------------------------------------- It would be nice if Ruby had an operator that only considered `nil` as false, like the null coalescing operators or "Logical Defined-Or operator" (Perl) found in some other languages. Ive seen things like `//` and `//=`m `??` and `??=`, or `?:` used for this. This would work like `||` and `||=` for short circuiting etc. except that only `nil` is considered a false condition. While Ruby considers only "false" and "nil" as false, with everything else true ("", [], {}, etc.) I still find occasionally people trip up when using logical or, `||` and `||=` when the value may be false. ```ruby a = 0 || 55 # = 0 Ruby already considers 0, "", etc. as true (oter languages do differ a lot here) a = 0 ?? 55 # = 0 So no change here a = nil || 55 # = 55, nil is false so right side is evaulated. a = nil ?? 55 # = 55, again no change a = false || 55 # = 55, however false is false for logical or a = false ?? 55 # = false, but its still a non-nil value ``` For example when doing things like: ```ruby def lazy @lazy ||= compute_this end def fetch(id, **opts) host = opts[:host] || default_host https = opts[:https] || true port = opts[:port] || (https ? 443 : 80) ... ``` Normally the intention is to use a default value or compute an action if no value is provided, which if the value may be false then requires special handling, or sometimes is missed and results in a bug. -- https://bugs.ruby-lang.org/

1 0

[ruby-core:119000] [Ruby master Bug#20710] Reducing Hash allocation introduces large performance degradation (probably related to VWA)
by pocke (Masataka Kuwabara) 09 Sep '24

09 Sep '24

Issue #20710 has been reported by pocke (Masataka Kuwabara). ---------------------------------------- Bug #20710: Reducing Hash allocation introduces large performance degradation (probably related to VWA) https://bugs.ruby-lang.org/issues/20710 * Author: pocke (Masataka Kuwabara) * Status: Open * ruby -v: ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin21] * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I found a surprising performance degradation while developing RBS. In short, I tried to remove unnecessary Hash allocations for RBS. Then, it made the execution time 2x slower. VWA for Hash probably causes this degradation. I'd be happy if we could mitigate the impact by updating the memory management strategy. ## Reproduce You can reproduce this problem on a PR in pocke/rbs repository. https://github.com/pocke/rbs/pull/2 This PR dedups empty Hash objects. 1. `git clone` and checkout 1. `bundle install` 1. `bundle exec rake compile` for C-ext 1. `bundle ruby benchmark/benchmark_new_env.rb` The "before" commit is https://github.com/pocke/rbs/commit/2c356c060286429cfdb034f88a74a6f94420fd21. The "after" commit is https://github.com/pocke/rbs/commit/bfb2c367c7d3b7f93720392252d3a3980d7bf335. The benchmark results are the following: ``` # Before $ bundle exec ruby benchmark/benchmark_new_env.rb (snip) new_env 6.426 (±15.6%) i/s - 64.000 in 10.125442s new_rails_env 0.968 (± 0.0%) i/s - 10.000 in 10.355738s # After $ bundle exec ruby benchmark/benchmark_new_env.rb (snip) new_env 4.371 (±22.9%) i/s - 43.000 in 10.150192s new_rails_env 0.360 (± 0.0%) i/s - 4.000 in 11.313158s ``` The IPS decreased 1.47x for `new_env` case (parsing small RBS env), and 2.69x for `new_rails_env` (parsing large RBS env). ## Investigation ### GC.stat `GC.stat` indicates the number of minor GCs increases. ```ruby # In the RBS repository require_relative './benchmark/utils' tmpdir = prepare_collection! new_rails_env(tmpdir) pp GC.stat ``` ``` # before {:count=>126, :time=>541, :marking_time=>496, :sweeping_time=>45, :heap_allocated_pages=>702, :heap_sorted_length=>984, :heap_allocatable_pages=>282, :heap_available_slots=>793270, :heap_live_slots=>787407, :heap_free_slots=>5863, :heap_final_slots=>0, :heap_marked_slots=>757744, :heap_eden_pages=>702, :heap_tomb_pages=>0, :total_allocated_pages=>702, :total_freed_pages=>0, :total_allocated_objects=>2220605, :total_freed_objects=>1433198, :malloc_increase_bytes=>5872, :malloc_increase_bytes_limit=>16777216, :minor_gc_count=>112, :major_gc_count=>14, :compact_count=>0, :read_barrier_faults=>0, :total_moved_objects=>0, :remembered_wb_unprotected_objects=>0, :remembered_wb_unprotected_objects_limit=>4779, :old_objects=>615704, :old_objects_limit=>955872, :oldmalloc_increase_bytes=>210912, :oldmalloc_increase_bytes_limit=>16777216} # after {:count=>255, :time=>1551, :marking_time=>1496, :sweeping_time=>55, :heap_allocated_pages=>570, :heap_sorted_length=>1038, :heap_allocatable_pages=>468, :heap_available_slots=>735520, :heap_live_slots=>731712, :heap_free_slots=>3808, :heap_final_slots=>0, :heap_marked_slots=>728727, :heap_eden_pages=>570, :heap_tomb_pages=>0, :total_allocated_pages=>570, :total_freed_pages=>0, :total_allocated_objects=>2183278, :total_freed_objects=>1451566, :malloc_increase_bytes=>1200, :malloc_increase_bytes_limit=>16777216, :minor_gc_count=>242, :major_gc_count=>13, :compact_count=>0, :read_barrier_faults=>0, :total_moved_objects=>0, :remembered_wb_unprotected_objects=>0, :remembered_wb_unprotected_objects_limit=>5915, :old_objects=>600594, :old_objects_limit=>1183070, :oldmalloc_increase_bytes=>8128, :oldmalloc_increase_bytes_limit=>16777216} ``` ### Warming up Hashes The following patch, which creates unnecessary Hash objects before the benchmark, improves the execution time. ```diff diff --git a/benchmark/benchmark_new_env.rb b/benchmark/benchmark_new_env.rb index 6dd2b73f..a8da61c6 100644 --- a/benchmark/benchmark_new_env.rb +++ b/benchmark/benchmark_new_env.rb @@ -4,6 +4,8 @@ require 'benchmark/ips' tmpdir = prepare_collection! +(0..30_000_000).map { {} } + Benchmark.ips do |x| x.time = 10 ``` The results are the following: ``` # Before Calculating ------------------------------------- new_env 10.354 (± 9.7%) i/s - 103.000 in 10.013834s new_rails_env 1.661 (± 0.0%) i/s - 17.000 in 10.282490s # After Calculating ------------------------------------- new_env 10.771 (± 9.3%) i/s - 107.000 in 10.010446s new_rails_env 1.584 (± 0.0%) i/s - 16.000 in 10.178984s ``` ### `RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO` The `RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO` env var also mitigates the performance impact. In this example, I set `RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO=0.6` (default: 0.20) ```console # Before Calculating ------------------------------------- new_env 10.271 (± 9.7%) i/s - 102.000 in 10.087191s new_rails_env 1.529 (± 0.0%) i/s - 16.000 in 10.538043s # After $ env RUBY_GC_HEAP_FREE_SLOTS_MIN_RATIO=0.6 bundle exec ruby benchmark/benchmark_new_env.rb Calculating ------------------------------------- new_env 11.003 (± 9.1%) i/s - 110.000 in 10.068428s new_rails_env 1.347 (± 0.0%) i/s - 14.000 in 11.117665s ``` ## Additional Information * I applied the same change to Array. But it does not cause this problem. * I guess the cause is the difference of the Size Pool. An empty Array uses 40 bytes like the ordinal Ruby object, but an empty Hash uses 160 bytes. * The Size Pool for 160 bytes objects has fewer objects than the 40 bytes one. So, reducing allocation affects the performance sensitively. * I tried it on Ruby 3.2. This change on Ruby 3.2 does not degrade the execution time. * VWA for Hash is introduced since Ruby 3.3. https://github.com/ruby/ruby/blob/73c39a5f93d3ad4514a06158e2bb7622496372b9/… ## Acknowledgement @mame, @ko1, and @soutaro helped the investigation. I would like to thank them. -- https://bugs.ruby-lang.org/

4 5

[ruby-core:119096] [Ruby master Bug#20720] prism incorrectly marks SuperNode as forwarding in some cases
by luke-gru (Luke Gruber) 07 Sep '24

07 Sep '24

Issue #20720 has been reported by luke-gru (Luke Gruber). ---------------------------------------- Bug #20720: prism incorrectly marks SuperNode as forwarding in some cases https://bugs.ruby-lang.org/issues/20720 * Author: luke-gru (Luke Gruber) * Status: Open * ruby -v: 3.4.0dev * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- ```ruby class A def foo(*b, &block) end end class B < A def foo(...) super() puts "2" end end ``` results in: ``` ./ruby --parser=prism -I../ruby/lib -I. -I.ext/x86_64-linux -I.ext/common -r./x86_64-linux-fake ../ruby/test.rb -- raw disasm-------- trace: 8 trace: 1 <L000> [sp: 0, unremovable: 0, refcnt: 0] 0000 putself ( 284) 0001 invokesuperforward <calldata:, 0>, nil ( 284) * 0004 pop ( 284) trace: 1 0005 putself ( 285) 0006 putchilledstring "2" ( 285) 0008 opt_send_without_block <calldata:puts, 1> ( 285) trace: 10 0010 leave ( 286) --------------------- ../ruby/test.rb: ../ruby/test.rb:284: argument stack underflow (-1) (SyntaxError) ``` It should differentiate between ZSUPER type nodes and super nodes in this case. -- https://bugs.ruby-lang.org/

1 1