ruby-dev

Download

ruby-dev@ml.ruby-lang.org

August 2026

1 participants
2 discussions

[ruby-dev:52243] [Ruby Feature#22279] Region (Length / Range) Arguments for String Bit Operations
by hasumikin (hitoshi hasumi) 10 Sep '26

10 Sep '26

Issue #22279 has been reported by hasumikin (hitoshi hasumi). ---------------------------------------- Feature #22279: Region (Length / Range) Arguments for String Bit Operations https://bugs.ruby-lang.org/issues/22279 * Author: hasumikin (hitoshi hasumi) * Status: Open ---------------------------------------- PR URL: ## What this adds Region overloads for the methods introduced in #22118, so they act on a contiguous run of bits in a single call: ``` String#bit_set(offset, length, lsb_first: true) -> self String#bit_set(range, lsb_first: true) -> self String#bit_clear(offset, length, lsb_first: true) -> self String#bit_clear(range, lsb_first: true) -> self String#bit_flip(offset, length, lsb_first: true) -> self String#bit_flip(range, lsb_first: true) -> self String#bit_count(offset, length, lsb_first: true) -> Integer String#bit_count(range, lsb_first: true) -> Integer ``` Existing single-offset forms and the no-argument `bit_count` are unchanged. ## Use cases Apache Arrow stores validity masks and boolean columns as LSB-first packed bitmaps (the same layout as Ruby's default `lsb_first: true`) and touches them a range at a time. **Writing a buffer:** mark a contiguous run of elements valid or null in one call (the equivalent of Arrow C++'s `bit_util::SetBitsTo`), or invert a boolean column: ```ruby validity.bit_set(start, run) # mark [start, start+run) valid validity.bit_clear(start, run) # mark them null column.bit_flip(0...n_elements) # boolean NOT ``` **Reading a buffer:** `null_count` is `length - bit_count`, a rank query over the mask. An Arrow array is an O(1) slice with an arbitrary bit offset, so a slice's `null_count` counts a range that starts and ends mid-byte: ```ruby null_count = slice_length - validity.bit_count(slice_offset, slice_length) ``` ## API Contracts - `(offset, length)` and a Range are equivalent; passing both is an `ArgumentError`. The positional form follows `String#[](start, length)`. - Endless / beginless ranges work as usual (`0..` = to the last bit). An empty region (`length == 0`, or an empty Range) is a no-op for the mutations and `0` for `bit_count`. For the mutations, even an empty region must begin within `0..bit_size`: `"\x00".bit_set(9, 0)` and `"\x00".bit_set(9...9)` both raise `IndexError`, mirroring how `"abc"[4, 0]` is `nil` while `"abc"[3, 0]` is `""`. - Mutations require the whole region to be in range and raise `IndexError` on overrun without modifying any bits; `bit_count` clamps to the bits that exist: the same read/write split as #22118. - `bit_count` returns `0` (not `nil`) for a region that lies entirely beyond the end. It is an aggregate, not a slice or a single-position lookup: the "reads return nil" rule of #22118 exists because `bit_get` has no in-domain value for "no such bit", whereas the count of set bits in an empty intersection is legitimately `0`, and the result is meant to feed arithmetic (`length - bit_count(...)`). This also keeps one rule for clamping: whatever part of the region exists is counted, and an empty part counts as `0`. - Mutations raise `FrozenError` on a frozen receiver even for an empty region, following `[].freeze.push` and `"".freeze.concat`. An out-of-range region is still detected first (`IndexError`), as in `String#setbyte`. - Negative offsets and Range endpoints raise `IndexError`, with no count-from-end normalization, as in #22118. A negative `length` raises `ArgumentError`, following the decision in #22881. - An inverted Range (`5..2`) is treated as empty, like `"abcd"[2..0]` returning `""`. - Bit positions beyond `2**64 - 1` raise `ArgumentError` even for the clamping `bit_count` (the representable-range rule of #22118 applies before clamping). - A lone `offset` keeps its #22118 single-bit meaning, so it is valid only for the mutations (`bit_set(offset)` etc.). `bit_count` has no single-bit form: it takes no argument (whole string), `(offset, length)`, or a Range, and `bit_count(offset)` raises `ArgumentError`. A one-bit count would return the same 0/1 as `bit_get(offset)`, and reading it as "from `offset` to the end" would contradict `String#[](index)`; counting to the end is spelled `bit_count(offset..)`. - An explicit `nil` is an argument, not an omission: `bit_set(0, nil)` and `bit_count(0, nil)` raise `TypeError`, as `"abc"[0, nil]` does. - `lsb_first` only changes the bit-to-position mapping within a byte. ## Performance A region call lets the implementation use `memset` / byte-wide `popcount` instead of a Ruby-level per-bit loop. On an AMD Ryzen 5 5600X (x86_64-linux, gcc, `--zjit`), with `buf = "\x00".b * 2**20` (1 MiB), setting 8,388,600 bits at an unaligned offset: ```ruby buf.bit_set(4, 8388600) # 15 us 8388600.times { buf.bit_set(4 + it) } # 440 ms (~29,000x slower) ``` Most of that gap is the per-call overhead of 8 million Ruby method calls rather than the per-byte work; the point is that a region call removes the loop entirely, not that the single-bit form is slow. For `bit_count` the point is addressing rather than raw speed: with `data = Random.bytes(4096)`, a region such as `data.bit_count(12347, 1000)` cannot be expressed with `byteslice` at all, because it starts and ends mid-byte. Even for a byte-aligned region the region form avoids the intermediate String: `data.bit_count(12344, 1008)` 90 ns vs `data.byteslice(1543, 126).bit_count` 130 ns. -- https://bugs.ruby-lang.org/

2 1

[ruby-dev:52235] [Ruby Bug#22123] Ruby::Box + `BUNDLER_SETUP` can evaluate gemspecs before main-box RubyGems initialization
by se4weed.dev＠gmail.com (Yuto NORINAGA) 18 Aug '26

18 Aug '26

Issue #22123 has been reported by se4weed.dev(a)gmail.com (Yuto NORINAGA). ---------------------------------------- Bug #22123: Ruby::Box + `BUNDLER_SETUP` can evaluate gemspecs before main-box RubyGems initialization https://bugs.ruby-lang.org/issues/22123 * Author: se4weed.dev(a)gmail.com (Yuto NORINAGA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-06-13T03:25:52Z master 0e3b8918b3) +PRISM [arm64-darwin25] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- # Ruby::Box + `BUNDLER_SETUP` can evaluate gemspecs before main-box RubyGems initialization ## Subject Ruby::Box + BUNDLER_SETUP can evaluate gemspecs before main-box RubyGems initialization ## Description When Ruby is started with `RUBY_BOX=1` and Bundler's `BUNDLER_SETUP` environment variable is present, RubyGems can require `bundler/setup` while loading `gem_prelude` for the root box. That can make Bundler run code through `TOPLEVEL_BINDING` in the main box before the main box has finished loading RubyGems. In one common case, Bundler evaluates a path gem's `.gemspec`, and a normal gemspec using `Gem::Specification.new` then fails with: ```text uninitialized constant Gem::Specification ``` This happens during Ruby startup, before application code starts. ## Environment Observed with: ```text ruby 4.1.0dev (2026-06-13T03:25:52Z master 0e3b8918b3) +PRISM [arm64-darwin25] macOS 26.4.1 bundler 4.1.0.dev Ruby::Box enabled with RUBY_BOX=1 ``` ## Minimal Reproduction Create a tiny path gem: ```sh rm -rf /tmp/box-bundler-repro mkdir -p /tmp/box-bundler-repro/box_repro cd /tmp/box-bundler-repro ``` `Gemfile`: ```ruby source "https://rubygems.org" gem "box_repro", path: "./box_repro" ``` `box_repro/box_repro.gemspec`: ```ruby Gem::Specification.new do |spec| spec.name = "box_repro" spec.version = "0.1.0" spec.summary = "Ruby::Box BUNDLER_SETUP repro" spec.authors = ["repro"] spec.files = [] end ``` Choose the Ruby executable being tested: ```sh RUBY_UNDER_TEST="${RUBY_UNDER_TEST:-ruby}" ``` Generate the lockfile with the same Ruby/Bundler environment: ```sh "$RUBY_UNDER_TEST" -S bundle lock ``` Then use the same executable to find the bundled `bundler/setup` path: ```sh BUNDLER_SETUP_PATH="$("$RUBY_UNDER_TEST" -rrbconfig -e 'puts File.join(RbConfig::CONFIG["rubylibdir"], "bundler/setup")')" ``` Then run Ruby with `RUBY_BOX=1` and `BUNDLER_SETUP`: ```sh RUBY_BOX=1 \ BUNDLE_GEMFILE="$PWD/Gemfile" \ BUNDLER_SETUP="$BUNDLER_SETUP_PATH" \ "$RUBY_UNDER_TEST" -e 'puts :ok' ``` ## Actual Result ```text [!] There was an error while loading `box_repro.gemspec`: uninitialized constant Gem::Specification. Bundler cannot continue. # from /tmp/box-bundler-repro/box_repro/box_repro.gemspec:1 # ------------------------------------------- > Gem::Specification.new do |spec| # spec.name = "box_repro" # ------------------------------------------- ``` ## Expected Result ```text ok ``` ## Control Cases This succeeds without Ruby::Box: ```sh BUNDLE_GEMFILE="$PWD/Gemfile" \ BUNDLER_SETUP="$BUNDLER_SETUP_PATH" \ "$RUBY_UNDER_TEST" -e 'puts :ok' ``` This also succeeds with Ruby::Box if Bundler is loaded later through `-rbundler/setup` instead of RubyGems' `BUNDLER_SETUP` hook: ```sh RUBY_BOX=1 \ BUNDLE_GEMFILE="$PWD/Gemfile" \ "$RUBY_UNDER_TEST" -rbundler/setup -e 'puts :ok' ``` So this does not appear to be application-specific. The failure is triggered by the Bundler startup environment, especially `BUNDLER_SETUP`. ## Analysis Ruby currently loads `gem_prelude` for both the root and main boxes: ```c // builtin.c rb_load_gem_prelude((VALUE)rb_root_box()); rb_load_gem_prelude((VALUE)rb_main_box()); ``` `gem_prelude.rb` requires RubyGems: ```ruby require "rubygems" ``` At the end of RubyGems, Bundler may be loaded from `BUNDLER_SETUP`: ```ruby # lib/rubygems.rb require ENV["BUNDLER_SETUP"] if ENV["BUNDLER_SETUP"] && !defined?(Bundler) ``` This means the root box's RubyGems load can trigger `bundler/setup` before the main box's RubyGems state is ready. Bundler evaluates path gemspecs with `TOPLEVEL_BINDING`: ```ruby # lib/bundler.rb eval(contents, TOPLEVEL_BINDING.dup, path.expand_path.to_s) ``` To confirm which box evaluated the gemspec and which RubyGems constants were available, I temporarily added the following diagnostics at the beginning of the path gem's `.gemspec`, before `Gem::Specification.new`: ```ruby if defined?(Ruby::Box) warn "box=#{Ruby::Box.current.inspect}" warn "root=#{Ruby::Box.root.inspect}" warn "main=#{Ruby::Box.main.inspect}" else warn "box=nil" warn "root=nil" warn "main=nil" end warn "gem=#{!!defined?(Gem)}" warn "gem_spec=#{!!defined?(Gem::Specification)}" warn "gem_version=#{!!defined?(Gem::VERSION)}" ``` During the failure, that diagnostic output showed: ```text box=#<Ruby::Box:3,user,main> root=#<Ruby::Box:2,root> main=#<Ruby::Box:3,user,main> gem=true gem_spec=false gem_version=false ``` So the gemspec is evaluated in the main box, but before the main box's RubyGems state is ready. The diagnostics suggest that `Gem` is present, but RubyGems constants such as `Gem::Specification` and `Gem::VERSION` are not fully initialized in the main box at that point. ## Why this should not be fixed in gemspecs Normal gemspecs conventionally use: ```ruby Gem::Specification.new do |spec| # ... end ``` Adding `require "rubygems/specification"` to the gemspec does not seem like the right fix. In local testing it only moved the failure further into RubyGems initialization, with other missing pieces such as: ```text Gem::Deprecate Gem::Requirement Gem::VERSION Gem.platforms ``` The problem is that Bundler is evaluating the gemspec before RubyGems is ready in the current box. ## Possible Fix Direction The core issue seems to be that `BUNDLER_SETUP` is consumed from the root box's RubyGems prelude before the main box's RubyGems state is ready. One possible direction is to fix this in Ruby startup / Ruby::Box prelude sequencing. For example, `builtin.c` could prevent the root box `gem_prelude` from consuming `BUNDLER_SETUP`, while still allowing the main box `gem_prelude` to consume it normally. Another possible direction is to fix this on the Bundler/RubyGems side, so that the `BUNDLER_SETUP` hook does not run user/top-level code in the main box before the main box RubyGems state is ready. The invariant I think we want is: `BUNDLER_SETUP` should not be able to trigger main-box code execution before the main box's RubyGems initialization is complete. With a local prototype that temporarily hides `BUNDLER_SETUP` while loading the root box prelude in `builtin.c`, then restores it before loading the main box prelude, the minimal reproduction succeeds. ## Related PR A proposed fix with a regression test is available at: https://github.com/ruby/ruby/pull/17323 -- https://bugs.ruby-lang.org/

3 2