Issue #20254 has been updated by jaruga (Jun Aruga). Conflict of interest: I work at Red Hat, being paid by Red Hat. I am a committer in the Ruby project. In the project, one of my interests and caring is about maintaining the CI services and infra, because the downstream Ruby RPM pakcage in Fedora, CentOS Stream and Red Hat Enterprise Linux I am working for can have a benefit from it. But I am trying to make the best decision for the Ruby Project. @ono-max Thank you for opening this issue ticket, and thank you for your work to onboard the Launchable in the Ruby. I am positive for trying this CI service, Launchable.
Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, Launchable into ruby/ruby. This decision was made during the meeting.
Let me clarify the context. [An event that people (ono-max, yoshiori) at Launchable explained the service](https://chouseisan.com/s?h=4d6c803c3ea1492b9b61de0ea7668255) was shared on the Ruby committer Slack channel, and the event was held in January 2024, then I can see the maximum 6 people: ko1, mrkn, znz, mame, hsbt attended the meeting, and maybe they have granted, and they may have information about Lunchable to guide people seeing this ticket. I thought we should transparently share this information with other committers and contributors, who are working for the unit tests in the ruby/ruby. Therefore I asked @ono-max to open a new issue ticket. I thought we needed to show our intention to select the best tools and to be vendor-neutral for the Ruby project, while everyone has a cognitive bias, and some people may have conflict of interest. Below is my main response.
As the CI results are already public information, I don't see any problem in trying it out. If there is a problem by any chance, it is easy to remove it.
I don't see any problem in trying it out too. Below are the pull-requests about adding the Launchable. * https://github.com/ruby/ruby/pull/9757 * `tool/lib/test/unit.rb` * https://github.com/ruby/ruby/pull/9777 * `tool/lib/test/unit.rb` * `tool/lib/test/unit/parallel.rb` * `.github/workflows/macos.yml` * https://github.com/ruby/debug/pull/1070 * https://github.com/ruby/vscode-rdbg/pull/432 For the future improvements, I would love to see make the Launclable's code as coarse coupling, so that we can drop Launchable even easier when we want as a possibility. * For the `tool/lib/test/*`, we may be able to split the Launchable's logic in the `tool/lib/test/unit.rb` into the another file such as `tool/lib/test/unit/launchable.rb`. * For the GitHub Actoins YAML file, we may be able to manage the Launchable's YAML setting in a separate file by using `uses:` syntax such as https://github.com/ruby/setup-ruby . It looks useful when we use Launchable for other GitHub YAML files other than `macos.yml` ---------------------------------------- Misc #20254: FYI: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106694 * Author: ono-max (Naoto Ono) * Status: Open * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/