October 2026 - ruby-core - ml.ruby-lang.org

[ruby-core:127102] [Ruby Feature#22437] Speed up creation and basic operations of Struct / Data
by shia (Sangyong Sim) 23 Oct '26

23 Oct '26

Issue #22437 has been reported by shia (Sangyong Sim). ---------------------------------------- Feature #22437: Speed up creation and basic operations of Struct / Data https://bugs.ruby-lang.org/issues/22437 * Author: shia (Sangyong Sim) * Status: Open ---------------------------------------- ## Summary Struct and Data are slower to create and to operate on than a hand-written class that does the same job. Struct / Data are usually chosen because people expect them to be lightweight value objects, yet once they sit on a hot path they turn out to be slow, and the code ends up being rewritten as a plain class. I think the standard value objects should be chosen for their original reasons, such as readability and immutability, rather than avoided for performance. This ticket proposes to narrow that gap. ## Approach The experimental branch below is the result of asking Claude Opus 5.5 to speed things up under the following constraints: - Do not change any interface - Do not add any fields - Do not change how stack traces look - Do not change any observable behavior So I do not intend to merge this branch as is. I will review the changes one by one, pick only the ones that are realistic and maintainable, and submit them as small PRs. ## Summary of the changes in the experimental branch - Hidden ivar lookups (`__members__` etc.) are cached in six file-static slots in `struct.c` (48 bytes per process), keyed by shape id. - `Struct#initialize` - Reads the member count from self. - Walks the keyword Hash without rb_ensure. - Instance creation of Data classes (`Point.new`, where `Point = Data.define(...)`) - Stores positional and keyword arguments directly. - Allocates the object already frozen. - `Data#with`: copies the existing values directly. - `Data#=`=, `#eql?` and Hash key lookup - Skip the recursion guard. A Data's members are fixed at creation, so the guard has nothing to protect. - Struct keeps the guard, because a member's method can modify and freeze the struct and then re-enter the same computation. - `to_h`, `deconstruct_keys`: fill the Hash directly. - `inspect`: reserves the buffer up front and stringifies built-in types directly. ## Benchmarks master d71692d49f (2026-10-12) vs. the experimental branch, on x86_64 Linux. | | No JIT | YJIT | ZJIT | |---|---|---|---| | Creation, Struct (geometric mean) | 1.48x | 1.60x | 1.32x | | Creation, Data (geometric mean) | 3.90x | 4.74x | 4.26x | | Basic operations (geometric mean) | 2.37x | 3.28x | 2.87x | "Basic operations" are five frequently used methods: `Data#with`, `Struct#==`, Hash lookup with a Struct key, `Struct#to_h` and `Struct#inspect`. Selected cases (no JIT, ns/op; `S = Struct.new(:a, :b, :c)`, `D = Data.define(:a, :b, :c)`): - `D.new(1, 2, 3)`: 214 → 39 - `S.new(1, 2, 3)`: 51 → 34 - `Data#with`: 334 → 86 - `inspect`: 449 → 166 - Hash lookup with a Data key: 250 → 85 ruby/ruby-bench (ms; no JIT / YJIT / ZJIT): - struct-new-no-escape: 72→47 / 36→24 / 17→12 - The same benchmark rewritten with Data: 180→60 / 150→29 / 139→24 ## Compatibility checks (on the experimental branch) - 47 checks comparing observable behavior against master, run with no JIT / YJIT / ZJIT: all identical. They cover backtraces, TracePoint, overridden methods, re-entry and frozen state. - All of test/ruby in 4 modes: the same set of failures as master. - ruby/spec core/struct, data, hash, marshal, plus language/pattern_matching and keyword_arguments in 3 modes: 0 failures. ## Links - Experimental branch: https://github.com/riseshia/ruby/compare/d71692d49f...experiment/struct-dat… - Benchmark and check scripts: https://github.com/riseshia/ruby/tree/experiment/struct-new-observable-probe ## Note: the ISEQ initialize idea If incompatibilities were acceptable, another idea is to have `Struct.new` generate an ISEQ initialize for each class it defines, so that it gets the same optimizations as a plain class's initialize. I have not tested how far this could go, but a rough PoC showed the following incompatibilities: - Reflection on initialize changes: - owner changes from Struct to the generated class. - arity is no longer `-1`, and parameters is no longer `[[:rest]]`. - source_location is no longer nil. - `:initialize` appears in `private_instance_methods(false)`. - The TracePoint event changes from `:c_call` to `:call`. - When an exception is raised inside initialize, the backtrace frame changes from `Struct#initialize` to the generated method. Whether these incompatibilities are acceptable, and whether the speedup would be large enough to justify them, is left for further discussion. -- https://bugs.ruby-lang.org/

5 10

[ruby-core:127219] [Ruby Feature#22466] Allow disabling the shape redblack cache via RUBY_SHAPE_REDBLACK_CACHE=disable
by jeremyevans0 (Jeremy Evans) 23 Oct '26

23 Oct '26

Issue #22466 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Feature #22466: Allow disabling the shape redblack cache via RUBY_SHAPE_REDBLACK_CACHE=disable https://bugs.ruby-lang.org/issues/22466 * Author: jeremyevans0 (Jeremy Evans) * Status: Open ---------------------------------------- The shape redblack cache mmaps 384MB of memory at startup. There is currently no way to configure the amount of memory used. The code already handles an mmap failure cleanly. I would like a way to skip the mmap attempt, and the simplest way I can think of is an environment variable. The reason behind the current behavior (I think) is that ruby currently assumes that the mmap is basically free, since the memory is not actually allocated in most cases. However, when operating system strict memory overcommit is used, this 384MB mmap allocation can result in the serious problems later with the operating system killing the process if it is low on memory. This provides a way to accept a performance hit and avoid the memory allocation. We are currently working around this issue in production by decreasing the amount of memory available to the process at startup to 256MB (causing the mmap allocation to fail), and then increasing it after startup using `Process.setrlimit`. However, that results in a lot of additional complexity that I would prefer to avoid. If it is preferable to specify the cache size via an environment variable, instead of just the ability to disable the cache, I can work on that approach instead. I submitted a pull request to implement this: https://github.com/ruby/ruby/pull/19499 -- https://bugs.ruby-lang.org/

1 0

[ruby-core:127202] [Ruby Bug#22462] Method#== can return false for methods that should be considered equal
by jeremyevans0 (Jeremy Evans) 23 Oct '26

23 Oct '26

Issue #22462 has been reported by jeremyevans0 (Jeremy Evans). ---------------------------------------- Bug #22462: Method#== can return false for methods that should be considered equal https://bugs.ruby-lang.org/issues/22462 * Author: jeremyevans0 (Jeremy Evans) * Status: Open * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- When a `Method` is created using `UnboundMethod#bind` and `#super_method`, the order in which the `#bind` and `#super_method` were applied could result in `Method` objects that were considered unequal. I believe we should have the invariant: for any `UnboundMethod` `um` where `super_method` is not `nil` and `o` is a valid object to bind to `um`, the following code should be true: ```ruby um.bind(o).super_method == um.super_method.bind(o) ``` However, that didn't work correctly in some cases: ```ruby module M; def m = [:M, *super]; end module N; include M; end class C; def m = :C; end class SC < C include N prepend M def m = [:SC, *super] end o = SC.new um = SC.instance_method(:m).super_method.super_method # M#m below SC (included via N) um.bind(o).super_method == um.super_method.bind(o) # => false # should be true ``` There is a similar invariant for Method: for any `Method` `m` where `super_method` is not `nil`, the following code should be true: ```ruby m.unbind.super_method == m.super_method.unbind ``` This appears to already hold, as the existing refinement tests already checked for this behavior. I've submitted a PR to fix this issue: https://github.com/ruby/ruby/pull/19474 -- https://bugs.ruby-lang.org/

2 2

[ruby-core:127217] [Ruby Bug#22465] Method lookup cache invalidation loop from BasicObject#extend
by alanwu (Alan Wu) 22 Oct '26

22 Oct '26

Issue #22465 has been reported by alanwu (Alan Wu). ---------------------------------------- Bug #22465: Method lookup cache invalidation loop from BasicObject#extend https://bugs.ruby-lang.org/issues/22465 * Author: alanwu (Alan Wu) * Status: Open * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- (More precisely, the invalidation happens with any method override in the range between `singleton_class.ancestors.first` and the first non-singleton class.) I'll use YJIT for the example because it has counters in the default build for observing method lookup cache (CME) invalidations, but CME invalidations also impact the interpreter. ```ruby # --yjit-call-threshold=1 def lookup(hash) hash.fetch(0, default: nil) # This call site only ever sees Hash end lookup({}) # ... but using BasicObject#extend on an unrelated # Hash instance it never sees invalidates the CME # for the site in `lookup`. unless ENV['DONT_EXTEND'] my_special_hash = {} my_special_hash.extend(Module.new { def fetch(*) = :unused }) # Note that `def my_special_hash.fetch = nil` doesn't suffer # from the same issue because the singleton class would be leaf. end p RubyVM::YJIT.runtime_stats(:invalidate_method_lookup) # => 1, or 0 when DONT_EXTEND=1. ``` ## Background The script hits the so called "unrelated invalidation" scenario from #16614 ("New method cache mechanism for Guild"), shown in invalidate_with_subclasses.png. It's also called the "tree invalidation" path in the implementation: any override in a sub-tree walks up to the root of the tree and invalidates the CME for the entire inheritance tree. Here's my own rendition of a tree invalidation scenario, with a diagram: ```ruby class Animal def speak = "poetically, everything can speak" end class Mammal < Animal; end class Cat < Mammal; end class Dog < Mammal; end class Raven < Animal; end def print_speech(object) = printf("a %5s: #{object.speak}\n", object.class) print_speech(Cat.new) print_speech(Raven.new) class Mammal; def speak = "mammals speak differently"; end print_speech(Raven.new) # IC miss inside print_speech(Cat.new) __END__ a Cat: poetically, everything can speak a Raven: poetically, everything can speak a Raven: poetically, everything can speak a Cat: mammals speak differently ``` ![](tree-invalidation.png) <details> ```mermaid flowchart BT Animal["Animal<br/>owns speak"] Dog Raven Cat Mammal["Mammal<br/>def speak ← newly added"] --> Animal Cat --> Mammal Dog --> Mammal Raven --> Animal classDef collateral fill:#ff8a8a,color:#000,stroke:#b33 classDef must fill:#ffd27a,color:#000,stroke:#b80 class Dog,Cat,Mammal must class Animal,Raven collateral subgraph legend direction TB req["necessary invalidation"] col["unrelated/collateral invalidation"] class req must class col collateral end ``` </details> ## Invalidation loop disproportionately slow under JITs This tree invalidation behavior happens every time on override through `extend`. In the interpreter, that means unrelated call sites take an inline cache miss on-demand. With YJIT and ZJIT, there's a large upfront cost on invalidation for patching all code that depend on the CME. After that, there is a heavy on-demand re-compile pause each time control reaches any of the invalidated call sites. Once re-compiled, this process repeats next time there is an override through `extend`. In a large application written by many teams of people, `extend` in one component ends up having an outsized global effect, especially when it overrides some methods on core classes such as `Hash`. It can also be hard to diagnose the issue back to `extend`, as much of the code never sees the singleton class. Here's a demo that simulates the worst case scenario from this issue for YJIT: ```ruby UNRELATED_PARTIES = 10_000 H = { 0 => 1 } call_sites = UNRELATED_PARTIES.times.map { RubyVM::InstructionSequence.compile('H.fetch(0)') } # The first wave of invalidations doesn't stop future ones 2.times do call_sites.each(&:eval) my_special_hash = {} # If the `extend` is in some specific route of a web app, # request traffic drives when the slowness hits. my_special_hash.extend(Module.new { def fetch(*) = :unused }) p RubyVM::YJIT.runtime_stats(:invalidate_method_lookup) end __END__ $ command time ruby test.rb nil nil 0.06 real 0.04 user 0.01 sys $ command time ruby --yjit-call-threshold=1 test.rb 10000 20000 2.21 real 0.74 user 1.46 sys ``` ## Solutions Going back to the first example, this is the ancestry tree on `extend`: ![](extend-invalidation.png) <details> ```mermaid flowchart BT Hash AnonMod["AnonMod<br/>def fetch ← newly added"] --> Hash Singleton["(singleton class of my_special_hash)"] --> AnonMod classDef collateral fill:#ff8a8a,color:#000,stroke:#b33 classDef must fill:#ffd27a,color:#000,stroke:#b80 class Singleton,AnonMod must class Hash collateral subgraph legend direction TB req["necessary invalidation"] col["unrelated/collateral invalidation"] class req must class col collateral end ``` </details> Tree invalidation handles branches in the inheritance tree, but the portion relevant to `extend` is always a straight line; e.g. `Class.new({}.singleton_class)` is a TypeError. We can use this property to limit unrelated invalidations. 1. A somewhat general fix involves walking towards the singleton class from the override point. If no CME for the method is found during the walk, then no invalidation is necessary. This adds a VM lock section proportional to the number of modules mixed into the singleton class. 2. A carve-out for `extend` calls that create a new singleton class. There's no CME cached on a freshly created singleton class and iclass, so no invalidation is necessary. This may end up a speed benefit for the `extend` call itself, at the cost of generality compared to (1): e.g. simulating `obj.extend(X)` with `class << obj; include(X); end` can still cause unrelated invalidations. Note that when there is a CME cached on any item in the singleton class chain, we have no choice but to invalidate the CME and accept the unrelated invalidations; finding all uses of that CME (e.g. interpreter CC, JIT code) is infeasible without significantly reworking the cache system. So, all the solutions focus on when it's possible to invalidate nothing. I've been working on implementing (1), but I'm now leaning towards (2) because it's easier to see correctness and hopefully a smaller patch. ---Files-------------------------------- extend-invalidation.png (64.3 KB) tree-invalidation.png (72.8 KB) -- https://bugs.ruby-lang.org/

1 0

[ruby-core:127208] [Ruby Bug#22464] Backport #16801 (overflowed repeat/null-check IDs in Onigmo) to 4.0 and 3.4; missing from 4.0.8 despite its release notes
by stephankaag (Stephan Kaag) 22 Oct '26

22 Oct '26

Issue #22464 has been reported by stephankaag (Stephan Kaag). ---------------------------------------- Bug #22464: Backport #16801 (overflowed repeat/null-check IDs in Onigmo) to 4.0 and 3.4; missing from 4.0.8 despite its release notes https://bugs.ruby-lang.org/issues/22464 * Author: stephankaag (Stephan Kaag) * Status: Open * ruby -v: ruby 4.0.8 (2026-10-22 revision 60c3d71bfb) +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- PR https://github.com/ruby/ruby/pull/16801 (commit 493d26dcde, merged 2026-04-29) rejects a regexp with more than 32767 counted repeats. Without it the repeat ID, a signed 16-bit value, wraps and the matcher writes out of bounds. GitLab shipped this fix on 2026-10-21 as part of CVE-2026-110264 (CVSS 10.0) for its bundled Ruby 3.3, 3.4 and 4.0. The fix is on master only. Ruby 4.0.8 (released 2026-10-22) does not contain it: * `include/ruby/onigmo.h` at v4.0.8 has `ONIGERR_TOO_BIG_COMPILED_PROGRAM` (-226) but not `ONIGERR_TOO_MANY_RANGE_REPEAT` (-224) or `ONIGERR_TOO_MANY_NULL_CHECK` (-225). * `regcomp.c` and `regerror.c` at v4.0.8 have no reference to these errors. * On a build of v4.0.8, `Regexp.new("(?:foobar){0,100}" * 40000)` compiles without error. With #16801 it raises RegexpError. The auto-generated GitHub release notes of v4.0.8 list the PR, which suggests the backport was intended. Please backport 493d26dcde to ruby_4_0 and ruby_3_4. The backport of the program size fixes (#19025) is already on both branches and in 4.0.8. -- https://bugs.ruby-lang.org/

2 1

[ruby-core:122947] [Ruby Feature#21539] Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process
by dalehamel (Dale Hamel) 22 Oct '26

22 Oct '26

Issue #21539 has been reported by dalehamel (Dale Hamel). ---------------------------------------- Feature #21539: Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process https://bugs.ruby-lang.org/issues/21539 * Author: dalehamel (Dale Hamel) * Status: Open ---------------------------------------- While ruby does have a great API for getting stack traces within the ruby processes, as used by profilers like vernier and stackprof, there are some projects which aim to profile ruby from outside of the process. Some examples include: - rbspy (https://github.com/rbspy/rbspy/tree/main/ruby-structs/src) - rbperf (https://github.com/javierhonduco/rbperf) - open telemetry (https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/sup…, https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/int…) These first two take the approach of embedding the ruby headers within the profiler to be able to walk the stack. otel's bpf exporter was relying on access to symbols (https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/202) which were removed some time ago https://github.com/ruby/ruby/pull/7459. As a result, it cannot profile newer rubies as it cannot unwind the stacks. All of these solutions kind of take the approach of trying to reverse engineer the ruby process and rely on somewhat hacky approaches that touch internal things that might frequently move around between ruby versions, and all feel a bit brittle. I think the root of this is that there is no public, stable api for this sort of external profiling. I'm not exactly sure what the solution should be, but it would be great for ruby to offer a recommended, supported, and stable way for external profilers (whether they are using perf api's or ptrace) to be able to: - Obtain the ruby native stack starting point, and walk it while being able to resolve the native symbols - Obtain a reference to the ruby execution context and walk the ruby interpreter stack - Have all of this work in a stable and reasonable way regardless of if / which ruby jit is enabled It would be awesome for some ruby maintainers / experts to weigh-in on what such an external API could look like. Perhaps a public header containing stable (or at least perhaps, versioned?) public symbols and perhaps structs as a starting point? This might necessitate that some fields be refactored out of existing structs, and could slow down work related to them (eg, members need to be added or removed). -- https://bugs.ruby-lang.org/

5 8

[ruby-core:111146] [Ruby master Bug#19172] `ruby_thread_has_gvl_p` is innacurate sometimes -- document or change?
by ivoanjo (Ivo Anjo) 22 Oct '26

22 Oct '26

Issue #19172 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #19172: `ruby_thread_has_gvl_p` is innacurate sometimes -- document or change? https://bugs.ruby-lang.org/issues/19172 * Author: ivoanjo (Ivo Anjo) * Status: Open * Priority: Normal * ruby -v: (All Ruby versions) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Howdy 👋! I work for Datadog [on the ddtrace gem](https://github.com/DataDog/dd-trace-rb) and I found a... sharp edge on the internal `ruby_thread_has_gvl_p` API. I am aware that `ruby_thread_has_gvl_p` is documented an experimental API that is exported as a symbol but not present on the VM include files. ### Background In the ddtrace profiling component, we setup a signal handler and then periodically send SIGPROF signals to try to interrupt the running Ruby thread (e.g. the thread that is holding the global VM lock or equivalent). In the signal handler, we need to perform some API calls which are not safe to do without the GVL. So we need to check if the signal handler got called in the thread that has the GVL. ### The issue ```ruby int ruby_thread_has_gvl_p(void) { rb_thread_t *th = ruby_thread_from_native(); if (th && th->blocking_region_buffer == 0) { return 1; } else { return 0; } } ``` In its current implementation, `ruby_thread_has_gvl_p` only checks if the thread has a `blocking_region_buffer` or not. Unfortunately, this means that when called from a thread that lost the GVL but not due to blocking (e.g. via `rb_thread_schedule()`), it can still claim that a thread is holding the GVL when that is not the case. I ran into this issue in https://github.com/DataDog/dd-trace-rb/pull/2415, and needed to find a workaround. ### Next steps Since this is an internal VM API, I'm not sure you'd want to change the current behavior, so I was thinking of perhaps two options: * Is it worth changing `ruby_thread_has_gvl_p` to be accurate in the case I've listed? * If not, would you accept a PR to document its current limitations, so that others don't run into the same issue I did? -- https://bugs.ruby-lang.org/

4 5

[ruby-core:127205] [Ruby Feature#22463] Improve String#index and line splitting performance on x64-mingw-ucrt
by watson1978 (Shizuo Fujita) 22 Oct '26

22 Oct '26

Issue #22463 has been reported by watson1978 (Shizuo Fujita). ---------------------------------------- Feature #22463: Improve String#index and line splitting performance on x64-mingw-ucrt https://bugs.ruby-lang.org/issues/22463 * Author: watson1978 (Shizuo Fujita) * Status: Open ---------------------------------------- ## Summary On x64-mingw-ucrt, `memchr` comes from `ucrtbase.dll`. On the tested system, it is a byte-by-byte loop and runs at about 2.5 GB/s, the same speed as a plain byte loop. Ruby calls `memchr` for single-byte searches, so `String#index`, `String#each_line` and `IO#gets` are slower than they need to be on this platform. When only `memchr` is replaced on the same machine, `String#index("\n")` over 1 KB lines becomes 3.1x (SWAR) to 4.2x (SSE2) faster. We found this while profiling Fluentd's in_tail plugin on Windows with rbspy. `String#index`, which splits lines there, took about 6% of the samples. ## Environment * ruby 4.0.7 (2026-09-15 revision 229531a6cf) +PRISM [x64-mingw-ucrt] (RubyInstaller 4.0.7-1) * Windows 10 (10.0.19045.6466), ucrtbase.dll 10.0.19041.3636 * AMD Ryzen 9 8945HS (virtual machine) * GCC 16.2.0 and mingw-w64-crt 14.0.0 from MSYS2 UCRT64 ## Where Ruby calls memchr * `rb_memsearch` in re.c calls `memchr` for a 1-byte pattern. For short patterns, `rb_memsearch_ss` also uses `memchr` to find the first byte. This is used by `String#index`, `String#byteindex` and `String#each_line`. * `rb_io_getline_fast` and `search_delim` in io.c call `memchr`. This is used by `IO#gets` and `IO#each_line`. ## memchr in ucrtbase.dll In ucrtbase.dll 10.0.19041.3636, `memchr` is a 28-byte function. It compares one byte per iteration and uses no vector registers (checked with `objdump -d`). We have not checked newer versions of ucrtbase.dll. Other string and memory functions that Ruby imports from the same DLL are much faster. These numbers are for 1040 bytes with the target at the end (crt_funcs_bench.c): | function | ucrtbase.dll | vcruntime140.dll | | -- | --: | --: | | `memchr` | 2.5 GB/s | 31.8 GB/s | | `strcmp` | 9.7 GB/s | not exported | | `strncmp` | 10.1 GB/s | not exported | | `strlen` | 18.1 GB/s | not exported | | `strstr` | 19.4 GB/s | 19.4 GB/s | | `strrchr` | 26.3 GB/s | 26.5 GB/s | | `strchr` | 26.4 GB/s | 26.4 GB/s | | `memcmp` | 37.1 GB/s | 36.9 GB/s | | `strnlen` | 38.1 GB/s | not exported | | `wcslen` | 62.9 GB/s | not exported | | `memmove` | 85.9 GB/s | 108.5 GB/s | | `memcpy` | 86.2 GB/s | 106.1 GB/s | | `memset` | 92.4 GB/s | 156.2 GB/s | A plain byte loop runs at about 2.5 GB/s on this machine. Only `memchr` is at that level, so we propose replacing only `memchr`. `__builtin_memchr` does not help. With non-constant arguments, GCC emits a jump to the library `memchr`. ## memchr implementations in C (same machine) The target byte is at the last position of the buffer. | implementation | 1040 bytes | 1 MB | | -- | --: | --: | | ucrtbase.dll `memchr` | 2.5 GB/s | 2.6 GB/s | | msvcrt.dll `memchr` | 2.5 GB/s | 2.6 GB/s | | plain byte loop (`gcc -O2` or `-O3`) | 2.5 GB/s | 2.6 GB/s | | SWAR, 8 bytes per step | 13.4 GB/s | 13.7 GB/s | | SSE2, 16 bytes per step | 26.8 GB/s | 27.3 GB/s | | SSE2, 64 bytes per step | 58.8 GB/s | 65.6 GB/s | | vcruntime140.dll `memchr` | 31.8 GB/s | 37.2 GB/s | Short inputs and early matches, in nanoseconds per call: | case | ucrtbase.dll | SWAR | SSE2, 16 bytes per step | | -- | --: | --: | --: | | n=1, match | 0.98 | 1.76 | 1.22 | | n=7, match at the end | 3.31 | 3.89 | 3.69 | | n=15, match at the end | 6.47 | 4.85 | 6.81 | | n=1040, match at offset 0 | 1.56 | 2.14 | 1.95 | | n=16, no match | 7.01 | 2.92 | 1.96 | | n=64, no match | 30.18 | 6.42 | 3.70 | | n=1040, no match | 410.09 | 77.81 | 39.40 | The replacements are up to 0.8 ns slower per call when the input is very short or the match is at the start. In the 16-byte-line benchmark below, each iteration takes about 80 ns, including the loop and argument handling. We observed no slowdown in that benchmark. ## Ruby-level effect (same machine, only memchr replaced) To change only `memchr`, we rewrote the `memchr` entry in the import address table of x64-ucrt-ruby400.dll at runtime with Fiddle. "SSE2" here is the 16-bytes-per-step version. Regexp search and `IO#read` do not use `memchr`, so they are shown as controls. | operation | ucrtbase.dll (as is) | SWAR | SSE2 | | -- | --: | --: | --: | | `String#index("\n")`, 64 MB of 1 KB lines | 2.11 GB/s | 6.61 GB/s | 8.93 GB/s | | `String#index("\n")`, 64 MB of 16 B lines | 0.20 GB/s | 0.22 GB/s | 0.22 GB/s | | `String#index("z")`, 64 MB, no match | 2.57 GB/s | 13.40 GB/s | 26.72 GB/s | | `String#each_line`, 64 MB of 1 KB lines | 1.36 GB/s | 2.44 GB/s | 2.60 GB/s | | `IO#each_line`, 256 MB file of 1 KB lines | 0.69 GB/s | 0.87 GB/s | 0.90 GB/s | | `String#index(/\n/)`, 64 MB of 1 KB lines (control) | 1.77 GB/s | 1.77 GB/s | 1.77 GB/s | | `IO#read` in 1 MB chunks, 256 MB file (control) | 8.12 GB/s | 7.91 GB/s | 8.13 GB/s | ## How to reproduce The attached files were built with GCC from MSYS2 UCRT64. C benchmarks: ``` gcc -O2 -Wall -o memchr_swar.exe memchr_swar.c memchr_swar.exe gcc -O2 -Wall -fno-builtin -o crt_funcs_bench.exe crt_funcs_bench.c crt_funcs_bench.exe ``` * memchr_swar.c first checks every implementation against a plain loop on 200,000 random inputs. Each throughput cell repeats calls on the same buffer until 512 MB is scanned. Each edge case runs 20,000,000 calls. Each value is a single run. * crt_funcs_bench.c loads each function from each DLL with `GetProcAddress`. Each cell scans 256 MB in total, and every result is checked. Ruby benchmark: ``` gcc -O2 -Wall -shared -o fastmemchr.dll fastmemchr.c set MEMCHR=ucrt ruby -r ./patch_memchr.rb ruby_level.rb set MEMCHR=swar ruby -r ./patch_memchr.rb ruby_level.rb set MEMCHR=sse2 ruby -r ./patch_memchr.rb ruby_level.rb ``` * patch_memchr.rb does nothing for `MEMCHR=ucrt`. Otherwise it points the `memchr` import of x64-ucrt-ruby400.dll to the function in fastmemchr.dll. It reads the import slot back to check that it now points to the new function, and it raises if no `memchr` import is found. It also checks that `String#index` and `String#each_line` still return correct results. * All strings are ASCII-8BIT. The 256 MB file is created in a new temporary directory (`Dir.mktmpdir`), which is removed at the end. The file is written and read in binary mode (`"rb"`), and it is read once before measuring to warm the page cache. * The `String#index("\n")` rows call `s.index("\n", pos + 1)` repeatedly until it returns nil, so every line is visited. The "no match" row is a single call over the whole string. * String operations use a 64 MB string. IO operations use a 256 MB file. * Each value is the best of 3 runs. ## Proposal Use our own `memchr` on x64-mingw-ucrt. We have only measured this platform, so we suggest limiting the change to it at first. * SSE2 is always available on x86-64, so an SSE2 version needs no runtime check. It is the fastest option we measured. * A SWAR version does not use SIMD intrinsics. It is slower than SSE2. The version below assumes a little-endian target and uses `__builtin_ctzll` from GCC and Clang. ```c #define ONES UINT64_C(0x0101010101010101) #define HIGHS UINT64_C(0x8080808080808080) #define HASZERO(x) (((x) - ONES) & ~(x) & HIGHS) /* little-endian only; __builtin_ctzll is a GCC/Clang builtin */ static void * swar_memchr(const void *s, int c, size_t n) { const unsigned char *p = s; uint64_t k = ONES * (unsigned char)c; while (n >= 8) { uint64_t v; memcpy(&v, p, 8); uint64_t m = HASZERO(v ^ k); if (m) return (void *)(p + (__builtin_ctzll(m) >> 3)); p += 8; n -= 8; } for (; n; p++, n--) { if (*p == (unsigned char)c) return (void *)p; } return NULL; } ``` include/ruby/win32.h already maps many CRT functions to `rb_w32_*` replacements, so this would follow an existing pattern. string.c also has a fallback `memrchr` (#20709), but that one is for platforms without the function. This proposal replaces an existing CRT function for performance, so the reason is different. Another option is to link `memchr` from vcruntime140.dll. MSYS2 ships `libvcruntime140.a`, and linking with `-lvcruntime140` worked in our test. However, vcruntime140.dll is not part of Windows, so it would have to be shipped with Ruby. We have not checked mswin builds. They link vcruntime140.dll, so they probably use its faster `memchr`. We have not measured aarch64-mingw-ucrt either. ## Attachments * memchr_swar.c: compares memchr implementations, including short inputs and early matches * crt_funcs_bench.c: compares CRT string and memory functions per DLL * fastmemchr.c: SWAR and SSE2 versions built as a DLL for patch_memchr.rb * patch_memchr.rb: replaces the `memchr` import of the Ruby DLL at runtime * ruby_level.rb: Ruby-level benchmark ---Files-------------------------------- fastmemchr.c (1.01 KB) memchr_swar.c (6.09 KB) patch_memchr.rb (2.5 KB) ruby_level.rb (1.31 KB) crt_funcs_bench.c (4.32 KB) -- https://bugs.ruby-lang.org/

2 2

[ruby-core:127131] [Ruby Bug#22447] parse.y `-> do ... end` COND state
by kddnewton (Kevin Newton) 22 Oct '26

22 Oct '26

Issue #22447 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #22447: parse.y `-> do ... end` COND state https://bugs.ruby-lang.org/issues/22447 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- With parse.y, `do`-blocks inside a `-> do ... end` lambda body are rejected when the lambda appears in a `while`/`until`/`for` condition: ```ruby while -> do foo do end end; end # => syntax error, unexpected 'do' for condition, expecting 'end' or dummy end ``` The same body is accepted outside a condition, and with brace delimiters inside one: ```ruby -> do foo do end end # OK while -> { foo do end }; end # OK while (foo do end); end # OK while def m; foo do end; end; end # OK while -> do foo do end end; end # SyntaxError until -> do foo do end end; end # SyntaxError for x in -> do foo do end end do end # SyntaxError ``` Nesting a `-> do` lambda in an unparenthesized lambda parameter default also triggers it: ```ruby while -> a = -> do foo do end end do end; end # SyntaxError ``` Pretty much everywhere this kind of thing happens there is a pair of `CMDARG_PUSH(0)` and `COND_PUSH(0)`, but lambda is missing the latter. I think that's really all that needs to change to make this work: ```diff --- a/parse.y +++ b/parse.y @@ lambda : tLAMBDA[lpar] f_larglist[args] { CMDARG_PUSH(0); + COND_PUSH(0); } lambda_body[body] { @@ restore_block_exit(p, $al + COND_POP(); CMDARG_POP(); ``` It looks like this was fixed in other places (test_methoddef_in_cond and test_classdef_in_cond both test this kind of thing). -- https://bugs.ruby-lang.org/

2 1

[ruby-core:127200] [Ruby Bug#22461] Use-after-free in `String` formatting with wide character encodings
by nobu (Nobuyoshi Nakada) 22 Oct '26

22 Oct '26

Issue #22461 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #22461: Use-after-free in `String` formatting with wide character encodings https://bugs.ruby-lang.org/issues/22461 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.3: REQUIRED, 3.4: REQUIRED, 4.0: REQUIRED ---------------------------------------- A heap use-after-free vulnerability exists in Ruby's string formatting implementation (`String#%` and `Kernel#sprintf`) when processing wide character encoded strings. -- https://bugs.ruby-lang.org/

1 1