Issue #22275 has been updated by hsbt (Hiroshi SHIBATA). I merged the following four, since they fix plain bugs such as crashes and need no design decision. * https://github.com/ruby/ruby/pull/18704: `alias $new $old` in a box created a separate variable. * https://github.com/ruby/ruby/pull/18708: `trace_var` hooks did not run for assignments in a box. * https://github.com/ruby/ruby/pull/18710: `RUBY_FREE_AT_EXIT=1` hung or crashed at exit (#22339, 4.0 backport requested). * https://github.com/ruby/ruby/pull/18842: a GC while copying a class into a box crashed (#22380, 4.0 backport requested). ---------------------------------------- Misc #22275: Ruby::Box support plan for RubyGems and Bundler https://bugs.ruby-lang.org/issues/22275#change-119205 * Author: hsbt (Hiroshi SHIBATA) * Status: Open ---------------------------------------- This is a tracking issue for making RubyGems and Bundler work with Ruby::Box, so that reviewers can see the whole picture and what to look at next in one place. ## Goal The goal is to realize [Feature #13847](https://bugs.ruby-lang.org/issues/13847) with Ruby::Box, which means loading a specific version of a gem, including default gems and C extensions, isolated from the copy that RubyGems and Bundler themselves use. Vendoring with namespace rewriting has been our workaround for pure-Ruby gems for years. Box removes that limitation. As a further step, we want RubyGems and Bundler to activate different gem versions per box. That enables the use cases already proposed around Box, such as gradual dependency upgrades, plugin systems with conflicting dependencies, and multiple applications in one process. ## First milestone Before designing such APIs, Box needs to leave experimental status. Our first milestone is that Rails and RubyGems/Bundler themselves work under `RUBY_BOX=1`, and that the ruby/rubygems repository runs its test suites with Box enabled continuously so we do not regress. --- ## Current status ruby/rubygems now runs both suites against a ruby-core master build with and without `RUBY_BOX=1` on every pull request and push ([ruby/rubygems#9826](https://github.com/ruby/rubygems/pull/9826)), and all four jobs pass. That covers the continuous part of the first milestone for ruby/rubygems. With the stdio, `$?` and `defined?` fixes merged, [ruby/rubygems#9892](https://github.com/ruby/rubygems/pull/9892) removed 28 pending marks from the rubygems suite and 3 skips from the bundler suite. The rubygems suite now runs 3619 tests with 0 failures in both modes, with 10 pendings under Box and 8 without. The two box-only pendings wait on `$VERBOSE`, and the remaining bundler skip waits on `-r`, both below. ruby/ruby's own `make check` passes under `RUBY_BOX=1` on [ruby/ruby#18705](https://github.com/ruby/ruby/pull/18705), which collects every open fix below and adds a CI lane for it. The branch is for measuring the integrated state, not for merging as it is. Decisions 3 to 7 came out of that work. ## Critical problems * [ruby/ruby#18579](https://github.com/ruby/ruby/pull/18579) ([Bug #22282](https://bugs.ruby-lang.org/issues/22282)) is the last fix the rubygems suite waits on: under Box, `$VERBOSE = nil` does not reach the interpreter. @tagomoris asked on the pull request whether `$VERBOSE` can stay per box instead, which is decision 3. * [Bug #22295](https://bugs.ruby-lang.org/issues/22295) (decision 2) has no fix yet. The bundler install specs skip it, and the cargo builder test works around it by requiring inside `-e`. ## Decisions needed These block the rest and need a direction rather than more review. 1. Which box a builtin written in Ruby resolves its calls in. It is the master box by default, which matches not doing local rebinding ([Bug #21362#note-3](https://bugs.ruby-lang.org/issues/21362#note-3)). [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544) settled `Marshal.load` by opting it into the caller's box with `Primitive.attr! :caller_user_box`. What remains is which other builtins opt in the same way, such as `Ractor.new` ([ruby/ruby#18821](https://github.com/ruby/ruby/pull/18821)), `Kernel#clone`, `Kernel#warn` and prelude ([Bug #22332](https://bugs.ruby-lang.org/issues/22332)), and whether core-class stubs should reach builtins ([Misc #22296](https://bugs.ruby-lang.org/issues/22296)). Opting every builtin in by default brings the inline cache leak of [Bug #21362](https://bugs.ruby-lang.org/issues/21362) back, because all boxes share a builtin's call sites. On the [ruby/ruby#18705](https://github.com/ruby/ruby/pull/18705) branch, which tries that, redefining `Integer#succ` in one box changes `10.times.to_a` in the main box. 2. Whether command-line `-r` activates gems, and which boxes it reaches ([Bug #22295](https://bugs.ruby-lang.org/issues/22295)). Under Box, `-r` and `RUBYOPT=-r` go through `Ruby::Box#require` without the RubyGems override, so a regular gem cannot be loaded that way, and a test harness that injects a gem with `RUBYOPT=-r` fails. The feature also stays in the main box, so a box created later does not see it. Item 3 of [Bug #21760](https://bugs.ruby-lang.org/issues/21760) reports the same gap in `Ruby::Box#require` itself. 3. What C-backed global variables mean in an optional box ([Bug #22307](https://bugs.ruby-lang.org/issues/22307)). Assignments to `$/`, `$stdout` and the like stay in a box-local table that C never reads. [ruby/ruby#18711](https://github.com/ruby/ruby/pull/18711) makes the main box behave like Ruby without boxes and leaves optional boxes open. `$VERBOSE` in [ruby/ruby#18579](https://github.com/ruby/ruby/pull/18579) is the same question for one variable. 4. How prism ships ([Bug #22305](https://bugs.ruby-lang.org/issues/22305)). The statically linked prism loads in only one box, so a gem that requires it, such as error_highlight, fails in every box but the first. 5. How JIT hooks reach boxes ([Bug #22306](https://bugs.ruby-lang.org/issues/22306)). Enabling YJIT switches `Array#each` and a few other methods to Ruby only in the master box, so the root and main boxes never get them. 6. What a box's top self should be ([Bug #22331](https://bugs.ruby-lang.org/issues/22331)). Only the master box's top self has `include`, `using`, `private` and the other top-level definition methods, so `Ruby::Box#load` and `load(file, true)` fail. 7. Where prelude is evaluated ([Bug #22332](https://bugs.ruby-lang.org/issues/22332)). It runs once in the master box, so `pp` and `binding.irb` require into it. --- ## Open pull requests ### ruby/ruby * [ruby/ruby#18579](https://github.com/ruby/ruby/pull/18579): assignments to `$VERBOSE` and `$DEBUG` have no effect ([Bug #22282](https://bugs.ruby-lang.org/issues/22282)). Waiting on decision 3. * [ruby/ruby#18711](https://github.com/ruby/ruby/pull/18711): C-backed global variables in the main box ([Bug #22307](https://bugs.ruby-lang.org/issues/22307)). It depends on [ruby/ruby#18704](https://github.com/ruby/ruby/pull/18704), which fixes `alias $new $old` in a box. * [ruby/ruby#18708](https://github.com/ruby/ruby/pull/18708): `trace_var` hooks do not run for assignments in a box. * [ruby/ruby#18707](https://github.com/ruby/ruby/pull/18707): the top self of a box lacks the top-level definition methods ([Bug #22331](https://bugs.ruby-lang.org/issues/22331)). * [ruby/ruby#18713](https://github.com/ruby/ruby/pull/18713): `pp` and `binding.irb` require into the master box ([Bug #22332](https://bugs.ruby-lang.org/issues/22332)). [Feature #21881](https://bugs.ruby-lang.org/issues/21881) proposed loading `prelude.rb` per box, and this does it. * [ruby/ruby#18821](https://github.com/ruby/ruby/pull/18821): `Ractor.new` resolves the classes of its arguments in the master box, so passing an instance of a class required in the main box raises `undefined class/module`. It touches the Ractor and Box integration that [Feature #22226#note-6](https://bugs.ruby-lang.org/issues/22226#note-6) left to be decided with the Box author. * [ruby/ruby#18842](https://github.com/ruby/ruby/pull/18842): a box classext is attached to its class only after it is filled in, so a GC in between frees what it holds. * [ruby/ruby#18710](https://github.com/ruby/ruby/pull/18710): `RUBY_FREE_AT_EXIT=1` hangs at exit on macOS and crashes on Linux. * [ruby/ruby#18701](https://github.com/ruby/ruby/pull/18701): a frozen `$LOADED_FEATURES` raises `FrozenError` while the feature index is rebuilt. This is not specific to Box, but every box hits it on its first `require`. ## Fixed ### ruby/ruby * [ruby/ruby#18544](https://github.com/ruby/ruby/pull/18544): `Marshal.load` resolved classes outside the caller's box ([Bug #22090](https://bugs.ruby-lang.org/issues/22090)). This was the last item on the Bundler critical path. * [ruby/ruby#18574](https://github.com/ruby/ruby/pull/18574): reassigning `$stdout` and `$stderr` was invisible to builtin writers ([Bug #21867](https://bugs.ruby-lang.org/issues/21867)), which broke output-capturing test helpers everywhere. * [ruby/ruby#18577](https://github.com/ruby/ruby/pull/18577): `$?` was uninitialized after `Kernel#system` and `IO.popen` ([Bug #22280](https://bugs.ruby-lang.org/issues/22280)). * [ruby/ruby#18586](https://github.com/ruby/ruby/pull/18586): `defined?` did not see global variables assigned in a box ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)), so mkmf `have_devel?` recursed until the stack was exhausted. * [ruby/ruby#18218](https://github.com/ruby/ruby/pull/18218) / [ruby/ruby#18219](https://github.com/ruby/ruby/pull/18219): `BUNDLER_SETUP` was consumed outside the main box ([Bug #22123](https://bugs.ruby-lang.org/issues/22123)). Shipped in 4.0.7. * [ruby/ruby#18509](https://github.com/ruby/ruby/pull/18509): box-local extension DLLs on Windows were unloaded too early. They are now deferred. * [ruby/ruby#18534](https://github.com/ruby/ruby/pull/18534): autoload-triggered require bypassed the box's `Kernel#require` ([Bug #21830](https://bugs.ruby-lang.org/issues/21830)). This alone took Rails from 500 on every request to fully working. * [ruby/ruby#18535](https://github.com/ruby/ruby/pull/18535): with `--disable=gems`, modules prepended to `Kernel` in a user box ended up behind `Kernel` in the ancestry ([Bug #22270](https://bugs.ruby-lang.org/issues/22270)). * [ruby/ruby#18536](https://github.com/ruby/ruby/pull/18536): the box extension copy embedded the full path in the temporary filename, exceeding NAME_MAX on deep paths, and the name was predictable ([Bug #22110](https://bugs.ruby-lang.org/issues/22110), [Bug #22271](https://bugs.ruby-lang.org/issues/22271)). * [ruby/ruby#18578](https://github.com/ruby/ruby/pull/18578): the process-private directory added by the previous fix did not survive `fork`. A forked child removed it at exit ([Bug #22271](https://bugs.ruby-lang.org/issues/22271)). * [ruby/ruby#18575](https://github.com/ruby/ruby/pull/18575): box resolution crashed on an IFUNC frame ([Bug #21977](https://bugs.ruby-lang.org/issues/21977)). Without this the bundler suite died with `[BUG]` and the dead workers cascaded into unrelated failures. * [ruby/ruby#18546](https://github.com/ruby/ruby/pull/18546): `Symbol#to_proc` ignored box-local method definitions ([Bug #22015](https://bugs.ruby-lang.org/issues/22015), revived [ruby/ruby#16865](https://github.com/ruby/ruby/pull/16865)). Without this Bundler's spec harness died in rspec-core's `&:shellsplit` before running a single spec. * [ruby/ruby#18475](https://github.com/ruby/ruby/pull/18475): an isolated Proc made in a class or module body lost its box and crashed ([Bug #22260](https://bugs.ruby-lang.org/issues/22260)). * [ruby/ruby#18698](https://github.com/ruby/ruby/pull/18698), [ruby/ruby#18702](https://github.com/ruby/ruby/pull/18702), [ruby/ruby#18703](https://github.com/ruby/ruby/pull/18703), [ruby/ruby#18706](https://github.com/ruby/ruby/pull/18706) and [ruby/ruby#18712](https://github.com/ruby/ruby/pull/18712): test and harness fixes so that `make check` can run under `RUBY_BOX=1`. ### ruby/rubygems * [ruby/rubygems#9892](https://github.com/ruby/rubygems/pull/9892): re-enabled the stdio capture tests under Box once [Bug #21867](https://bugs.ruby-lang.org/issues/21867) was fixed. * [ruby/rubygems#9826](https://github.com/ruby/rubygems/pull/9826): made the RubyGems and Bundler suites green under `RUBY_BOX=1` and added the Box lanes to CI. * [ruby/rubygems#9809](https://github.com/ruby/rubygems/pull/9809): Bundler evaluated gemspecs through `TOPLEVEL_BINDING`, which always belongs to the main box. It now uses a binding in the box Bundler is loaded in, so gemspecs resolve the right `Gem::Specification`. * [ruby/rubygems#9810](https://github.com/ruby/rubygems/pull/9810): the `gem` CLI died under `RUBY_BOX=1`. `Marshal`-based deep copies could not resolve `Gem::` constants across boxes and were replaced with a plain deep dup, and `RUBY_BOX` is now stripped from extension build subprocesses, where mkmf `have_devel?` recurses until `SystemStackError` ([Bug #22283](https://bugs.ruby-lang.org/issues/22283)). It includes a CLI canary test that runs under `RUBY_BOX=1`. -- https://bugs.ruby-lang.org/