Issue #20254 has been updated by ono-max (Naoto Ono). Thank you for introducing the feature, @jaruga. Also, I'd like to introduce the page https://app.launchableinc.com/organizations/ruby/workspaces/ruby/insights/un.... This page displays Flaky Tests, Never Failing Tests, Longest Tests, and Most Failed Tests. For instance, the flakiness score is displayed in the Flaky Tests section. The flakiness score expresses how much the test exhibits flakiness. Let's say "test_nested_timeout" has a flakiness score of 0.2. This means that if "test_nested_timeout" fails against 10 different commits, the test failure is not considered a true failure in two of those commits. For more detailed information, you can visit the document page at https://www.launchableinc.com/docs/features/unhealthy-tests/. ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-107058 * Author: ono-max (Naoto Ono) * Status: Assigned * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/