October 2024 - ruby-core - ml.ruby-lang.org

[ruby-core:111678] [Ruby master Feature#19315] Lazy substrings in CRuby
by Eregon (Benoit Daloze) 11 Jun '26

11 Jun '26

Issue #19315 has been reported by Eregon (Benoit Daloze). ---------------------------------------- Feature #19315: Lazy substrings in CRuby https://bugs.ruby-lang.org/issues/19315 * Author: Eregon (Benoit Daloze) * Status: Open * Priority: Normal ---------------------------------------- CRuby should implement lazy substrings, i.e., "abcdef"[1..3] must not copy bytes. Currently CRuby only reuse the char* if the substring is until the end of the buffer. But it should also work wherever the substring starts and ends. Yes, it means RSTRING_PTR() might need to allocate to \0-terminate, so be it, it's worth it. There is already code for this (`SHARABLE_MIDDLE_SUBSTRING`), but it's disabled by default and `RSTRING_PTR()` needs to be changed to deal with this. It seems a good idea to introduce a variant of `RSTRING_PTR` which doesn't guarantee \0-termination, so such callers can then use the existing bytes always without copy. There are countless workarounds for this missing optimization, all not worth and all less readable: * https://bugs.ruby-lang.org/issues/19314 * https://github.com/ruby/net-protocol/pull/14 * Manual lazy substrings which track string + index + length * More but I don't remember all now, feel free to comment or link more urls/tickets. -- https://bugs.ruby-lang.org/

10 38

[ruby-core:113430] [Ruby master Feature#19633] Allow passing block to `Kernel#autoload` as alternative to second `filename` argument
by shioyama (Chris Salzberg) 10 Jun '26

10 Jun '26

Issue #19633 has been reported by shioyama (Chris Salzberg). ---------------------------------------- Feature #19633: Allow passing block to `Kernel#autoload` as alternative to second `filename` argument https://bugs.ruby-lang.org/issues/19633 * Author: shioyama (Chris Salzberg) * Status: Open * Priority: Normal ---------------------------------------- `Kernel#autoload` takes two arguments, a symbol `module` representing the constant to be autoloaded, and a `filepath` to load the constant from (with `require`). Currently, Zeitwerk has to [monkeypatch `Kernel#require`](https://github.com/fxn/zeitwerk/blob/a7c4a983df0f4e4058f32c610dac1e8b99f687da/lib/zeitwerk/kernel.rb) to fetch the loader for the file being loaded, then run the original (aliased) `require`, then run autoload callbacks. In addition to the monkeypatch, this also requires a registry (`Zeitwerk::Registry`) to map file paths to loaders, to know which loader should be used for a given autoload-triggered `require`. In fact, Zeitwerk has to _assume_ that the monkey-patched `require` call came from an autoload trigger; there is no way to really be sure of the source. If Ruby allowed passing a block as an alternative to the explicit filepath, then I think this could be improved and would eliminate the need for a monkeypatch. So something like this: ```ruby autoload(:B) do require "lib/b" # trigger callback, etc end ``` I am implementing a gem called [Im](https://github.com/shioyama/im) which is a fork of Zeitwerk, and in the case of this gem, such a feature would be even more useful. Im implements autoloads on anonymous modules by registering an autoload and then "catching" the require and converting it into a `load`, passing the module as the second argument (see [here](https://github.com/shioyama/im/blob/44ce348639a1aae563a5be7a40602761e….) This is currently quite tricky because, again, it's hard to know _where_ a `require` came from. In addition to removing the monkeypatch (inherited from Zeitwerk), Im would further benefit from the block argument because then it could simply access the module via a closure, rather than pulling it from a registry: ```ruby mod.autoload(:Foo) do load "lib/foo.rb", mod end ``` I don't know how hard or easy this would be to implement, but if there is interest in this as a feature I'd be happy to look into implementing it. -- https://bugs.ruby-lang.org/

4 16

[ruby-core:111146] [Ruby master Bug#19172] `ruby_thread_has_gvl_p` is innacurate sometimes -- document or change?
by ivoanjo (Ivo Anjo) 15 May '26

15 May '26

Issue #19172 has been reported by ivoanjo (Ivo Anjo). ---------------------------------------- Bug #19172: `ruby_thread_has_gvl_p` is innacurate sometimes -- document or change? https://bugs.ruby-lang.org/issues/19172 * Author: ivoanjo (Ivo Anjo) * Status: Open * Priority: Normal * ruby -v: (All Ruby versions) * Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN ---------------------------------------- Howdy 👋! I work for Datadog [on the ddtrace gem](https://github.com/DataDog/dd-trace-rb) and I found a... sharp edge on the internal `ruby_thread_has_gvl_p` API. I am aware that `ruby_thread_has_gvl_p` is documented an experimental API that is exported as a symbol but not present on the VM include files. ### Background In the ddtrace profiling component, we setup a signal handler and then periodically send SIGPROF signals to try to interrupt the running Ruby thread (e.g. the thread that is holding the global VM lock or equivalent). In the signal handler, we need to perform some API calls which are not safe to do without the GVL. So we need to check if the signal handler got called in the thread that has the GVL. ### The issue ```ruby int ruby_thread_has_gvl_p(void) { rb_thread_t *th = ruby_thread_from_native(); if (th && th->blocking_region_buffer == 0) { return 1; } else { return 0; } } ``` In its current implementation, `ruby_thread_has_gvl_p` only checks if the thread has a `blocking_region_buffer` or not. Unfortunately, this means that when called from a thread that lost the GVL but not due to blocking (e.g. via `rb_thread_schedule()`), it can still claim that a thread is holding the GVL when that is not the case. I ran into this issue in https://github.com/DataDog/dd-trace-rb/pull/2415, and needed to find a workaround. ### Next steps Since this is an internal VM API, I'm not sure you'd want to change the current behavior, so I was thinking of perhaps two options: * Is it worth changing `ruby_thread_has_gvl_p` to be accurate in the case I've listed? * If not, would you accept a PR to document its current limitations, so that others don't run into the same issue I did? -- https://bugs.ruby-lang.org/

3 3

[ruby-core:117434] [Ruby master Bug#20409] Missing reporting some invalid breaks
by kddnewton (Kevin Newton) 13 May '26

13 May '26

Issue #20409 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Bug #20409: Missing reporting some invalid breaks https://bugs.ruby-lang.org/issues/20409 * Author: kddnewton (Kevin Newton) * Status: Open * Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- -- https://bugs.ruby-lang.org/

4 12

[ruby-core:115241] [Ruby master Bug#19986] Win32: `HOME` is set to just `HOMEDRIVE` if `HOMEPATH` is unset
by nobu (Nobuyoshi Nakada) 11 May '26

11 May '26

Issue #19986 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #19986: Win32: `HOME` is set to just `HOMEDRIVE` if `HOMEPATH` is unset https://bugs.ruby-lang.org/issues/19986 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Priority: Normal * Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED ---------------------------------------- ``` $ env -u HOME -u USERPROFILE -u HOMEPATH HOMEDRIVE=no/such/home ./miniruby.exe -e 'p ENV["HOME"], Dir.home' "no/such/home" "no/such/home" ``` `HOMEDRIVE` should be ignored without `HOMEPATH`, and `HOME` should fallback to `CSIDL_PROFILE`. -- https://bugs.ruby-lang.org/

1 1

[ruby-core:118940] [Ruby master Bug#20696] `IO.read` ignores `err:` option
by nobu (Nobuyoshi Nakada) 11 May '26

11 May '26

Issue #20696 has been reported by nobu (Nobuyoshi Nakada). ---------------------------------------- Bug #20696: `IO.read` ignores `err:` option https://bugs.ruby-lang.org/issues/20696 * Author: nobu (Nobuyoshi Nakada) * Status: Open * Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN ---------------------------------------- I expected the following code captures the error message, but `err:` option is ignored. ```shell-session $ ruby -e 'p IO.read("|ruby -e raise", err: %i[child out])' -e:1:in '<main>': unhandled exception "" ``` -- https://bugs.ruby-lang.org/

1 1

[ruby-core:116382] [Ruby master Feature#20205] Enable `frozen_string_literal` by default
by byroot (Jean Boussier) 14 Apr '26

14 Apr '26

Issue #20205 has been reported by byroot (Jean Boussier). ---------------------------------------- Feature #20205: Enable `frozen_string_literal` by default https://bugs.ruby-lang.org/issues/20205 * Author: byroot (Jean Boussier) * Status: Open * Priority: Normal ---------------------------------------- ### Context The `frozen_string_literal: true` pragma was introduced in Ruby 2.3, and as far as I'm aware the plan was initially to make it the default for Ruby 3.0, but this plan was abandoned because it would be too much of a breaking change without any real further notice. According to Matz, he still wishes to enable `frozen_string_literal` by default in the future, but a reasonable migration plan is required. The main issue is backward compatibility, flipping the switch immediately would break a lot of code, so there must be some deprecation period. The usual the path forward for this kind of change is to emit deprecation warnings one of multiple versions in advance. One example of that was the Ruby 2.7 keyword argument deprecation. It was quite verbose, and some users were initially annoyed, but I think the community pulled through it and I don't seem to hear much about it anymore. So for frozen string literals, the first step would be to start warning when a string that would be frozen in the future is mutated. ### Deprecation Warning Implementation I implemented a quick proof of concept with @etienne in https://github.com/Shopify/ruby/pull/549 In short: - Files with `# frozen_string_literal: true` or `# frozen_string_literal: false` don't change in behavior at all. - Files with no `# frozen_string_literal` comment are compiled to use `putchilledstring` opcode instead of regular `putstring`. - This opcode mark the string with a user flag, when these strings are mutated, a warning is issued. Currently the proof of concept issue the warning at the mutation location, which in some case can make locating where the string was allocated a bit hard. But it is possible to improve it so the message also include the location at which the literal string was allocated, and learning from the keyword argument warning experience, we can record which warnings were already issued to avoid spamming users with duplicated warnings. As currently implemented, there is almost no overhead. If we modify the implementation to record the literal location, we'd incur a small memory overhead for each literal string in a file without an explicit `frozen_string_literal` pragma. But I believe we could do it in a way that has no overhead if `Warning[:deprecated] = false`. ### Timeline The migration would happen in 3 steps, each step can potentially last multiple releases. e.g. `R0` could be `3.4`, `R1` be `3.7` and `R2` be `4.0`. I don't have a strong opinion on the pace. - Release `R0`: introduce the deprecation warning (only if deprecation warnings enabled). - Release `R1`: make the deprecation warning show up regardless of verbosity level. - Release `R2`: make string literals frozen by default. ### Impact Given that `rubocop` is quite popular in the community and it has enforced the usage of `# frozen_string_literal: true` for years now, I suspect a large part of the actively maintained codebases in the wild wouldn't see any warnings. And with recent versions of `minitest` enabling deprecation warnings by default (and [potentially RSpec too](https://github.com/rspec/rspec-core/issues/2867)) the few that didn't migrate will likely be made compatible quickly. The real problem of course are the less actively developed libraries and applications. For such cases, any codebase can remain compatible by setting `RUBYOPT="--disable=frozen_string_literal"`, and so even after `R2` release. The flag would never be removed any legacy codebase can continue upgrading Ruby without changing a single line of cod by just flipping this flag. ### Workflow for library maintainers As a library maintainer, fixing the deprecation warnings can be as simple as prepending `# frozen_string_literal: false` at the top of all their source files, and this will keep working forever. Alternatively they can of course make their code compatible with frozen string literals. Code that is frozen string literal compatible doesn't need to explicitly declare it. Only code that need it turned of need to do so. ### Workflow for application owners For application owners, the workflow is the same than for libraries. However if they depend on a gem that hasn't updated, or that they can't upgrade it, they can run their application with `RUBYOPT="--disable=frozen_string_literal"` and it will keep working forever. Any user running into an incompatibility issue can set `RUBYOPT="--disable=frozen_string_literal"` forever, even in `4.x`, the only thing changing is the default value. And any application for which all dependencies have been made fully frozen string literal compatible can set `RUBYOPT="--enable=frozen_string_literal"` and start immediately removing magic comment from their codebase. -- https://bugs.ruby-lang.org/

23 82

[ruby-core:117658] [Ruby master Feature#20448] Make coverage event hooking C API public
by ms-tob (Matt S) 27 Mar '26

27 Mar '26

Issue #20448 has been reported by ms-tob (Matt S). ---------------------------------------- Feature #20448: Make coverage event hooking C API public https://bugs.ruby-lang.org/issues/20448 * Author: ms-tob (Matt S) * Status: Open ---------------------------------------- # Abstract Gathering code coverage information is a well-known goal within software engineering. It is most commonly used to assess code coverage during automated testing. A lesser known use-case is coverage-guided fuzz testing, which will be the primary use-case presented in this issue. This issue exists to request that Ruby coverage event hooking be made part of its official, public C API. # Background Ruby currently provides a number of avenues for hooking events *or* gathering coverage information: 1. The [Coverage](https://ruby-doc.org/3.3.0/exts/coverage/Coverage.html) module 2. The [TracePoint](https://ruby-doc.org/3.3.0/TracePoint.html) module 3. The [rb_add_event_hook](https://ruby-doc.org/3.3.0/extension_rdoc.html#label-Hoo… extension function Unfortunately, none of these pieces of functionality solve this issue's specific use-case. The `Coverage` module is not a great fit for real-time coverage analysis with an unknown start and stop point. Coverage-guided fuzz testing requires this. The `TracePoint` module and `rb_add_event_hook` are not able to hook branch and line coverage events. Coverage-guided fuzz testing typically tracks branch events. # Proposal The ultimate goal is to enable Ruby C extensions to process coverage events in real-time. I did some cursory investigation into the Ruby C internals to determine what it would take to achieve this, but I'm by no means an expert, so my list may be incomplete. The good news is that much of this functionality already exists, but it's part of the private, internal-only C API. 1. Make `RUBY_EVENT_COVERAGE_LINE` and `RUBY_EVENT_COVERAGE_BRANCH` public: https://github.com/ruby/ruby/blob/v3_3_0/vm_core.h#L2182-L2184 a. This would be an addition to the current public event types: https://github.com/ruby/ruby/blob/v3_3_0/include/ruby/internal/event.h#L32-… 2. Allow initializing global coverage state so that coverage tracking can be fully enabled a. Currently, if `Coverage.setup` or `Coverage.start` is not called, then coverage events cannot be hooked. I do not fully understand why this is, but I believe it has something to do with `rb_get_coverages` and `rb_set_coverages`. If calls to `rb_get_coverages` return `NULL` (https://github.com/ruby/ruby/blob/v3_3_0/iseq.c#L641-L647, https://github.com/ruby/ruby/blob/v3_3_0/iseq.c#L864-L868) then coverage hooking will not be enabled. I believe the `Coverage` module initializes that state via a `rb_set_coverages` call here: https://github.com/ruby/ruby/blob/v3_3_0/ext/coverage/coverage.c#L112-L120. b. So, to achieve this goal, a C extension would need to be able to call `rb_set_coverages` or somehow initialize the global coverage state. I've actually been able to achieve this functionality by calling undocumented features and defining `RUBY_EVENT_COVERAGE_BRANCH`: ```c #include <ruby.h> #include <ruby/debug.h> #define RUBY_EVENT_COVERAGE_BRANCH 0x020000 // ... rb_event_flag_t events = RUBY_EVENT_COVERAGE_BRANCH; rb_event_hook_flag_t flags = ( RUBY_EVENT_HOOK_FLAG_SAFE | RUBY_EVENT_HOOK_FLAG_RAW_ARG ); rb_add_event_hook2( (rb_event_hook_func_t) event_hook_branch, events, counter_hash, flags ); ``` If I call `Coverage.setup(branches: true)`, and add this event hook, then branch hooking works as expected. `rb_add_event_hook2` will still respect the `RUBY_EVENT_COVERAGE_BRANCH` value if its passed. But it would be better if I could rely on official functionality rather than undocumented features. The above two points would be requirements for this functionality, but there's an additional nice-to-have: 3. Extend the public `tracearg` functionality to include additional coverage information a. Currently, `tracearg` offers information like `rb_tracearg_lineno` and `rb_tracearg_path`. It would be helpful if it also provided additional coverage information like `coverage.c`'s column information and a unique identifier for each branch. Currently, I can only use `(path, lineno)` as a unique identifier for a branch because that's what's offered by the public API, but more information like column number would be helpful for uniquely identify branches. Since there can be multiple `if` statements on a single line, this can provide ambiguous identification for a branch event. # Use cases This use-case was born out of a new coverage-guided Ruby fuzzer: https://github.com/trailofbits/ruzzy. You can read more about its implementation details here: https://blog.trailofbits.com/2024/03/29/introducing-ruzzy-a-coverage-guided…. You can also find the Ruby C extension code behind its implementation here: https://github.com/trailofbits/ruzzy/blob/v0.7.0/ext/cruzzy/cruzzy.c#L220-L…. So, the primary use-case here is enabling real-time, coverage-guided fuzz testing of Ruby code. However, as mentioned in the abstract, gathering code coverage information is useful in many domains. For example, it could enable new workflows in standard unit/integration test coverage. It could also enable gathering coverage information in real-time as an application is running. I see this as the most generalized form of gathering code coverage information, and something like the `Coverage` module as a specialized implementation. Another example, https://bugs.ruby-lang.org/issues/20282 may be solved by this more generalized solution. We are tracking this request downstream here: https://github.com/trailofbits/ruzzy/issues/9 # Discussion Fuzz testing is another tool in a testers toolbelt. It is an increasingly common way to improve software's robustness. Go has it built in to the standard library, Python has Atheris, Java has Jazzer, JavaScript has Jazzer.js, etc. OSS-Fuzz has helped identify and fix over 10,000 vulnerabilities and 36,000 bugs [using fuzzing](https://google.github.io/oss-fuzz/#trophies). Ruby deserves a good fuzzer, and improving coverage gathering would help achieve that goal. The `Coverage` module, `TracePoint` module, and `rb_add_event_hook` function seem like they could fulfill this goal. However, after deeper investigation, none of them fit the exact requirements for this use-case. # See also - https://bugs.ruby-lang.org/issues/20282 - https://github.com/google/atheris - https://security.googleblog.com/2020/12/how-atheris-python-fuzzer-works.html - https://github.com/CodeIntelligenceTesting/jazzer/ - https://www.code-intelligence.com/blog/java-fuzzing-with-jazzer - https://go.dev/doc/security/fuzz/ -- https://bugs.ruby-lang.org/

2 9

[ruby-core:116456] [Ruby master Feature#20215] Introduce `IO#readable?`
by ioquatix (Samuel Williams) 19 Mar '26

19 Mar '26

Issue #20215 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #20215: Introduce `IO#readable?` https://bugs.ruby-lang.org/issues/20215 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- There are some cases where, as an optimisation, it's useful to know whether more data is potentially available. We already have `IO#eof?` but the problem with using `IO#eof?` is that it can block indefinitely for sockets. Therefore, code which uses `IO#eof?` to determine if there is potentially more data, may hang. ```ruby def make_request(path = "/") client = connect_remote_host # HTTP/1.0 request: client.write("GET #{path} HTTP/1.0\r\n\r\n") # Read response client.gets("\r\n") # => "HTTP/1.0 200 OK\r\n" # Assuming connection close, there are two things the server can do: # 1. peer.close # 2. peer.write(...); peer.close if client.eof? # <--- Can hang here! puts "Connection closed" # Avoid yielding as we know there definitely won't be any data. else puts "Connection open, data may be available..." # There might be data available, so yield. yield(client) end ensure client&.close end make_request do |client| puts client.read # <--- Prefer to wait here. end ``` The proposed `IO#readable?` is similar to `IO#eof?` but rather than blocking, would simply return false. The expectation is the user will subsequently call `read` which may then wait. The proposed implementation would look something like this: ```ruby class IO def readable? !!self.closed? end end class BasicSocket # Is it likely that the socket is still connected? # May return false positive, but won't return false negative. def readable? return false unless super # If we can wait for the socket to become readable, we know that the socket may still be open. result = self.recv_nonblock(1, MSG_PEEK, exception: false) # No data was available - newer Ruby can return nil instead of empty string: return false if result.nil? # Either there was some data available, or we can wait to see if there is data avaialble. return !result.empty? || result == :wait_readable rescue Errno::ECONNRESET # This might be thrown by recv_nonblock. return false end end ``` For `IO` itself, when there is buffered data, `readable?` would also return true immediately, similar to `eof?`. This is not shown in the above implementation as I'm not sure if there is any Ruby method which exposes "there is buffered data". -- https://bugs.ruby-lang.org/

8 18

[ruby-core:116589] [Ruby master Misc#20238] Use prism for mk_builtin_loader.rb
by kddnewton (Kevin Newton) 13 Mar '26

13 Mar '26

Issue #20238 has been reported by kddnewton (Kevin Newton). ---------------------------------------- Misc #20238: Use prism for mk_builtin_loader.rb https://bugs.ruby-lang.org/issues/20238 * Author: kddnewton (Kevin Newton) * Status: Open * Priority: Normal ---------------------------------------- I would like to propose that we use prism for mk_builtin_loader.rb. Right now the Ruby syntax that you can use in builtin classes is restricted to the base Ruby version (2.7). This means you can't use a lot of the nicer syntax that Ruby has shipped in the last couple of years. If we switch to using prism to parse the builtin files instead of using ripper, then we can always use the latest version of Ruby syntax. A pull request for this is here: https://github.com/kddnewton/ruby/pull/65. The approach for the PR is taken from how RJIT bindgen works. -- https://bugs.ruby-lang.org/

6 22