[ruby-core:122615] [Ruby Misc#21458] Test 'make install'?

Issue #21458 has been reported by MSP-Greg (Greg L). ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by MSP-Greg (Greg L). I'm asking about this because https://github.com/ruby/ruby-dev-builder/actions has been failing for five days, and ruby-loco for a similar time. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-113863 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by Eregon (Benoit Daloze). MSP-Greg (Greg L) wrote:
Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)?
I agree, this is important to add, and means it's tested in a way end users use CRuby. Maybe there are already some tests ran on a `make install`-ed Ruby? ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-113868 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by Eregon (Benoit Daloze). MSP-Greg (Greg L) wrote in #note-1:
I'm asking about this because https://github.com/ruby/ruby-dev-builder/actions has been failing for five days, and ruby-loco for a similar time.
Looking at https://github.com/ruby/ruby-dev-builder/actions/runs/15981482361/job/450766... ``` Run bundle install Fetching gem metadata from https://rubygems.org/. Resolving dependencies... Fetching path 2.1.0 Installing path 2.1.0 Bundle complete! 3 Gemfile dependencies, 4 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. Run bundle exec rake --version bundler: failed to load command: rake (/home/runner/.rubies/ruby-head/bin/rake) /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/runtime.rb:317:in 'Bundler::Runtime#check_for_activated_spec!': You have already activated json 2.12.2, but your Gemfile requires json 2.2.0. Since json is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports json as a default gem. (Gem::LoadError) from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/runtime.rb:25:in 'block in Bundler::Runtime#setup' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/spec_set.rb:233:in 'Array#each' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/spec_set.rb:233:in 'Bundler::SpecSet#each' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/runtime.rb:24:in 'Enumerable#map' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/runtime.rb:24:in 'Bundler::Runtime#setup' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler.rb:166:in 'Bundler.setup' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/setup.rb:32:in 'block in <top (required)>' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/ui/shell.rb:173:in 'Bundler::UI::Shell#with_level' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/ui/shell.rb:119:in 'Bundler::UI::Shell#silence' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/setup.rb:32:in '<top (required)>' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/cli/exec.rb:57:in 'Kernel#require_relative' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/cli/exec.rb:57:in 'Bundler::CLI::Exec#kernel_load' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/cli/exec.rb:23:in 'Bundler::CLI::Exec#run' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/cli.rb:452:in 'Bundler::CLI#exec' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/vendor/thor/lib/thor/command.rb:28:in 'Bundler::Thor::Command#run' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/vendor/thor/lib/thor/invocation.rb:127:in 'Bundler::Thor::Invocation#invoke_command' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/vendor/thor/lib/thor.rb:538:in 'Bundler::Thor.dispatch' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/cli.rb:35:in 'Bundler::CLI.dispatch' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/vendor/thor/lib/thor/base.rb:584:in 'Bundler::Thor::Base::ClassMethods#start' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/cli.rb:29:in 'Bundler::CLI.start' from /home/runner/.rubies/ruby-head/lib/ruby/gems/3.5.0+2/gems/bundler-2.7.0.dev/exe/bundle:28:in 'block in <top (required)>' from /home/runner/.rubies/ruby-head/lib/ruby/3.5.0+2/bundler/friendly_errors.rb:118:in 'Bundler.with_friendly_errors' from /home/runner/.rubies/ruby-head/lib/ruby/gems/3.5.0+2/gems/bundler-2.7.0.dev/exe/bundle:20:in '<top (required)>' from /home/runner/.rubies/ruby-head/bin/bundle:25:in 'Kernel#load' from /home/runner/.rubies/ruby-head/bin/bundle:25:in '<main>' Error: Process completed with exit code 1. ``` This *looks* like a RubyGems/Bundler bug at first sight, could you report it there? I wonder what is loading `json` 2.12.2 before Bundler can load the Gemfile. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-113869 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by ntkme (Natsuki Natsume).
I wonder what is loading json before Bundler can load the Gemfile.
Nothing is loading `json`, it's a misreporting issue due to the gemspec of the `json` being broken: https://bugs.ruby-lang.org/issues/21462 The broken gemspec contains files from bundler - that whenever bundler is loaded, json is being considered as loaded despite that no one loaded it - this has been fixed on latest head branch. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-113914 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by luke-gru (Luke Gruber). I ran into this same bundler issue when trying to upgrade a Rails app to ruby 3.5.0-preview1. If it could have been caught by testing the installation, I think it should be done. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-114132 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by naruse (Yui NARUSE). Testing `make install` sounds reasonable. But complex multiple testing for installed files seems too complex.
Maybe there are already some tests ran on a make install-ed Ruby?
Many years ago, test-all was run for install-ed Ruby, but these days it runs with files in build directories. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-114301 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by hsbt (Hiroshi SHIBATA). FYI: We already tested `make install` HEAD version and all of stable versions at https://github.com/ruby/actions. I'm okay to add only `make install` test to GitHub Actions. But I'm against to add cli test like `rdoc --version` because we explicitly ignored that with development version. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-114318 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by MSP-Greg (Greg L). naruse (Yui NARUSE) wrote in #note-8:
Many years ago, test-all was run for install-ed Ruby, but these days it runs with files in build directories.
That would be helpful, but today bundled gems make things 'interesting'... hsbt (Hiroshi SHIBATA) wrote in #note-9:
I'm against to add cli test like `rdoc --version` because we explicitly ignored that with development version.
Many repos (both within the ruby org and external) are testing with Ruby head/master builds. I think that benefits all parties. You've used the term 'explicit', where is what it refers to defined? Many years ago (Travis & Appveyor) most maintainers had the opinion that Ruby master builds were not stable enough to use in CI. Today, a small number of people have worked to provide functionality, speed, etc within the GHA infrastructure such that the Ruby community is using GHA, and they are also using Ruby head/master builds. If Ruby CI is considered adequate, @Eregon and I will continue to post here when head/master breaks in the build repos. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-114341 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by Eregon (Benoit Daloze). hsbt (Hiroshi SHIBATA) wrote in #note-9:
But I'm against to add cli test like `rdoc --version` because we explicitly ignored that with development version.
I don't think such regressions were ever intentional. They are real bugs, and if ruby-dev-builder wouldn't test such things it would have broken gems CI testing against ruby-head, and maybe some gems would have dropped testing against ruby-head or started to just always ignore the result. So I think it's a clear win to test an installed Ruby in CRuby CI. Also it seems a good time to repeat that every production use of Ruby uses an installed Ruby. None of them uses a Ruby from just `make`. So it seems of utmost important to test something close to what people use in production (or simply what every CRuby user uses). There has also been numerous bugs related to `make install` and `tool/rbinstall.rb`. Some of those would have likely been prevented with tests running on an installed Ruby. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-114376 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/

Issue #21458 has been updated by naruse (Yui NARUSE).
There has also been numerous bugs related to make install and tool/rbinstall.rb. Some of those would have likely been prevented with tests running on an installed Ruby.
I agree to add tests for installed ruby as regression test which checks such numerous bugs related to make install and tool/rbinstall.rb. ---------------------------------------- Misc #21458: Test 'make install'? https://bugs.ruby-lang.org/issues/21458#change-114377 * Author: MSP-Greg (Greg L) * Status: Open ---------------------------------------- Can `make install` be tested? Two items come to mind: 1. Test that all gems can be required? 2. Check that all gem binstubs are installed and work. Some of the above are being done in ruby-loco and https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb. Both of these are cron jobs, they don't run on every commit. A long time ago, I think this was discussed, but it was back when CI was much more limited. It was also mixed in with the idea of running tests from the 'install' folder, rather than the 'build' folder. Maybe a few jobs could be added to check `make install` on the three main OS's (Ubuntu, macOS, Windows)? Lastly, with minimal changes, tests can be run from the install folder, which is what ruby-loco has done for years. Recently, two tests that I believe are skipped here started failing: ``` TestGemRequire#test_realworld_default_gem Line: 471 Expected "" to not be empty. TestGemRequire#test_realworld_upgraded_default_gem Line: 492 <"999.99.9"> expected but was <"/ruby/tmp/test_rubygems_20250628-8920-sifyoe/test_realworld_upgraded_default_gem.rb:3:in '<main>': undefined method 'version' for nil (NoMethodError)">. ``` -- https://bugs.ruby-lang.org/
participants (6)
-
Eregon (Benoit Daloze)
-
hsbt (Hiroshi SHIBATA)
-
luke-gru (Luke Gruber)
-
MSP-Greg (Greg L)
-
naruse (Yui NARUSE)
-
ntkme (Natsuki Natsume)