[ruby-core:126653] [Ruby Bug#22305] Ruby::Box: statically linked prism loads in only one box
Issue #22305 has been reported by hsbt (Hiroshi SHIBATA). ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. The first treats prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. The second treats it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. The third changes Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by hsbt (Hiroshi SHIBATA). I pended the `TestProcSyntaxTree` tests under `RUBY_BOX=1` until a design is chosen here. https://github.com/ruby/ruby/pull/19139 ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119264 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. The first treats prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. The second treats it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. The third changes Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by matz (Yukihiro Matsumoto). I prefer option 1. In Box, things defined by the interpreter are shared, and things loaded by `require` belong to each box. Option 1 keeps that model as it is, and it is also what #19900 originally described. Option 2 makes every process pay for 339 classes at boot and stops a prism gem from taking over, which I don't want. Option 3 adds a third kind of thing to Box, and I'd like to avoid that while we are still working out the Box semantics. @tagomoris, please tell us if you see it differently from the Box side. One question: how would option 1 work on platforms without dynamic loading, where every extension is statically linked? Matz. ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119352 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. 1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. 2. Treat it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. 3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by tagomoris (Satoshi Tagomori). I also prefer option 1. That makes box possible to load different Prism versions in boxes (if users want). ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119368 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. 1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. 2. Treat it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. 3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by mame (Yusuke Endoh). I'm also fine with option 1. My concern about option 1 was that `Proc#syntax_tree` loads the prism Ruby API via `require "prism"` and relies on it being the same prism that is built into the interpreter, which a per-box prism gem cannot guarantee. With `Ruby::Prism` proposed in #22379, `Proc#syntax_tree` no longer needs the prism gem, so `require "prism"` can simply load a separate prism in each box. What remains is `Ruby::Prism` itself. Since it is statically linked into the interpreter, it cannot be loaded separately in each box. I think it should be shared among all boxes, like other builtins. This needs a bit of hassle, since `Ruby::Prism` has to be autoloaded in the master box, but I have a prototype and will share it once #22379 is settled. ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119381 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. 1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. 2. Treat it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. 3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by kddnewton (Kevin Newton). I agree with @mame here, I think option 1 is best if we can get it to work, which it seems like after #22379 it will be. ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119382 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. 1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. 2. Treat it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. 3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by hsbt (Hiroshi SHIBATA). This is a note to sort out the discussion. The direction in #22379 is a hybrid of options 1 and 2. In mame's PoC (https://github.com/ruby/ruby/pull/19248), the prism gem becomes an ordinary bundled gem that each box dlopens on its own, and the parser built into the interpreter becomes `Ruby::Prism`, so the gem no longer collides with it. What remains here is sharing `Ruby::Prism` across boxes, which the PoC does not do yet. As mame noted, loading the autoload registered in the root box into the root box and sharing it with every box would solve it. I'll follow #22379 and keep this ticket for that part. ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119395 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. 1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. 2. Treat it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. 3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
Issue #22305 has been updated by mame (Yusuke Endoh). I've opened a pull request to share `Ruby::Prism` among boxes: https://github.com/ruby/ruby/pull/19296 `Ruby::Prism` is autoloaded in the master box, so that every box sees the same definition, like other builtins. This did not work because of two problems in autoload under Box: 1. When an autoload is triggered, the constant is defined only in the box that triggered it, not in the one that registered it. So after one box resolves the autoload, other boxes get NameError. 2. The autoload table is shared by all boxes. Once one box resolves the autoload, the entry is removed from the shared table, and other boxes can no longer see either the constant or the autoload. The PR has two commits: 1. Set the constants in the box that registered the autoload, and copy them to the triggering box. 2. Remove the autoload table and keep the autoload information in each constant entry instead. With 2, the autoload information is copied per box along with the constant table. I also tried copying the autoload table per box for 2, but the change was bigger and more complex because it needs a new mechanism to copy the autoload table. The PR is much simpler than that, and it even has fewer lines than the current master. Without Box, the behavior does not change. Note that `require "ruby/prism"` in a box loads it in that box, which leads to strange behavior. @tagomoris has decided that users must not explicitly `require "ruby/prism"`, and the PR also documents it. ---------------------------------------- Bug #22305: Ruby::Box: statically linked prism loads in only one box https://bugs.ruby-lang.org/issues/22305#change-119466 * Author: hsbt (Hiroshi SHIBATA) * Status: Open * ruby -v: ruby 4.1.0dev (2026-09-10T03:20:50Z master fe58143f12) +YJIT +MN +PRISM [arm64-darwin27] * Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN ---------------------------------------- Under `RUBY_BOX=1`, the statically linked prism can be required in only one box. The second box raises `LoadError`. ``` $ RUBY_BOX=1 ruby --disable-gems -e 'require "prism"; p Ruby::Box.new.eval(%q{require "prism"; Prism::VERSION})' lib/prism.rb:161:in 'Kernel#require': cannot load such file -- prism/prism (LoadError) ``` Swapping the order fails the same way. `--disable-gems` matters here. With a prism gem installed, each box dlopens its own copy of the gem's `prism.bundle` and the problem disappears, which makes it easy to miss. There is no `prism.so` file. `prism_init.c:8` registers the name `prism/prism.so` with `ruby_init_ext`, pointing at the `Init_prism` already linked into the interpreter, so `require "prism/prism"` behaves as if it had loaded an extension. That entry can be used only once (`load.c:1280`). The parser itself works in every box, but loading a gem that requires prism internally, such as error_highlight, is enough to break it in every box other than the one that loaded prism first. The obvious fixes do not hold. Running the init in every box leaves one set of C globals pointing at whichever box ran it last, so nodes parsed in the other box fail `is_a?(Prism::ProgramNode)`. That trades the `LoadError` for a silent type mismatch. Initializing prism before the boxes exist and marking the feature provided makes the `require "prism/prism"` inside an activated prism gem a no-op, so the gem's Ruby half binds to the interpreter's C half. The `ruby-lsp` benchmark fails on exactly that. I see three designs and would like one chosen. 1. Treat prism as an extension. The Ruby binding is built as its own shared object while the parser stays static, so each box dlopens its own copy the way it already does for other extensions. The same principle would also fix every extension under `--with-static-linked-ext`. It changes the build and needs symbol exports on Windows. 2. Treat it as part of the interpreter, like `String`. It is initialized once before the boxes and every box shares its classes. `defined?(Prism)` becomes true without a `require`, it adds 339 classes at boot, and a prism gem can no longer take over. 3. Change Box instead. One way would be to share a statically linked extension across boxes like a builtin class without marking it provided, so that a prism gem can still take over, though I am not sure that fits where Box is heading. ## How prism came to ship this way #19741 mirrored YARP into ruby/ruby together with its Ruby API and registered the native part with `ruby_init_ext`. #19772 then moved toward mirroring only the C implementation and leaving the Ruby API to a bundled prism gem (#19772#note-29), and #19900 stated that prism would be a bundled gem with a native extension. The static registration from #19741 survived the rename, so prism ships as a default gem linked into the interpreter, and that registration is what fails here. The native extension #19900 described works in every box, and the first design comes down to it. The second design has a history of its own. Matz first approved a built-in `Ruby::Parser` that no gem could replace (#19772#note-9), and the same idea came back as a gem-independent `Ruby::Node` for `Proc#syntax_tree` (#21795#note-1). There Matz required that the parser that ran the program and the parser that returns the tree be the same (#21795#note-19), and the feature settled on the default gem prism for that. That is exactly the prism a second box cannot load, which also blocks the way #21795#note-38 suggested Box could eventually expose the interpreter's own parser. -- https://bugs.ruby-lang.org/
participants (5)
-
hsbt (Hiroshi SHIBATA) -
kddnewton (Kevin Newton)
-
mame (Yusuke Endoh)
-
matz (Yukihiro Matsumoto)
-
tagomoris (Satoshi Tagomori)