Issue #20254 has been updated by mame (Yusuke Endoh). In short, Launchable is willing to provide us their service for free. I feel it convenient to see the test results formatted, instead of the log of GitHub Actions. As the CI results are already public information, I don't see any problem in trying it out. If there is a problem by any chance, it is easy to remove it. ---------------------------------------- Misc #20254: FYI: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106675 * Author: ono-max (Naoto Ono) * Status: Open * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- Ruby core team has granted me (Naoto Ono) the permission to integrate [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by ko1 (Koichi Sasada). Some more background: 1, I heard the introduction of Launchable service with some committers by my former colleague (as described in #5) in closed meeting. 2. On the meeting, we've learn the service and we found the task is enough small to introduce it. Although the impetus came from a referral from an acquaintance, by comparing the benefit from the service and the cost of introduction, I think trying it is reasonable and this does not constitute a conflict of interest (it is a normal selection of services for development). In this case, the cost of introduction is enough low. I agree there are room to improve the source code but we can improve after merging and trying it out, IMO. (and we can revert the code if we found another issues) ---------------------------------------- Misc #20254: FYI: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106704 * Author: ono-max (Naoto Ono) * Status: Open * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by byroot (Jean Boussier). So if I understand correctly, this isn't an alternative CI, just a side system that collect statistics and also better format test failures. I haven't looked at it in detail, but some form of flakiness analysis would indeed be very welcome. The one small bit of feedback I have about what I've seen, and that is mostly down to personal preference, is I tend to dislike bots that post GitHub comments, as I find them very noisy given my GitHub notification setup (email). So I wonder if it would be possible for it to register itself as a normal commit status instead? That said, GitHub Action already publish so many of these I understand one worry might be that it will be hard to find in the list? Also it would be nice to see a PR with it enabled on `ruby/ruby`. The result on `ruby/debug` look nice, but `ruby/ruby` CI output is way more complicated, so I'd be curious to see how well it adapts to that. ---------------------------------------- Misc #20254: FYI: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106785 * Author: ono-max (Naoto Ono) * Status: Open * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by hsbt (Hiroshi SHIBATA). Status changed from Open to Assigned In dev meeting at #20193, we discussed about Launchable. No one against Launchable integration for our GitHub Actions. In my opinion, Launchable helps to improve CI result of GitHub Actions and easily find flaky test. I as administrator of dev infrastructure am supporting @ono-max to integrate Launchable. ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106795 * Author: ono-max (Naoto Ono) * Status: Assigned * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by ono-max (Naoto Ono). I wonder if I should introduce the following options to customize the behavior of the GitHub comments feature in the Launchable Web Console: **Configurable Option 1:** Allow users to choose when Launchable creates a GitHub comment. * Option 1.1: Create a GitHub comment every time a commit is pushed. * Option 1.2: (Default) Launchable creates a single comment summarizing test results when a pull request is created. Subsequent test updates are reflected by updating this comment. **Configurable Option 2:** Allow users to choose the visibility of GitHub comments based on test results. * Option 2.1: Hide GitHub comments when all tests pass. * Option 2.2: (Default) Do not hide comments, regardless of test results. I welcome your thoughts on these proposed options. ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106812 * Author: ono-max (Naoto Ono) * Status: Assigned * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by jaruga (Jun Aruga). FYI: I added document about Launchable to ruby/ruby's wiki page. https://github.com/ruby/ruby/wiki/CI-Servers#launchable-ci ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106884 * Author: ono-max (Naoto Ono) * Status: Assigned * Priority: Normal * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by jaruga (Jun Aruga). jaruga (Jun Aruga) wrote in #note-5:

... Below are the pull-requests about adding the Launchable.

... * https://github.com/ruby/ruby/pull/9777 * `tool/lib/test/unit.rb` * `tool/lib/test/unit/parallel.rb` * `.github/workflows/macos.yml`

Note the https://github.com/ruby/ruby/pull/9777 mentioned above was merged and integrated into the GitHub Actions CI yesterday. Everyone can check how the test reports look like on the page below. https://app.launchableinc.com/organizations/ruby/workspaces/ruby/data/test-s... ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-106963 * Author: ono-max (Naoto Ono) * Status: Assigned * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by ono-max (Naoto Ono). FYI: I'm going to change the GitHub comment feature as follows tomorrow: If a test is executed multiple times, we'll treat it as a single test, and the status of the test will reflect the latest result. By implementing this behavior, retried tests will be marked as passed if the final result after retrying is passed. This feedback was given by Shibata-san and Kokubun-san. ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-107057 * Author: ono-max (Naoto Ono) * Status: Assigned * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/

Issue #20254 has been updated by hsbt (Hiroshi SHIBATA). Status changed from Assigned to Closed @ono-max and @jaruga Thank you for your works. Launchable has been added our repository. We discovered https://bugs.ruby-lang.org/issues/20314 with this integration. We should close this and create new ticket each concerns. ---------------------------------------- Misc #20254: Add Launchable into Ruby CI https://bugs.ruby-lang.org/issues/20254#change-107063 * Author: ono-max (Naoto Ono) * Status: Closed * Assignee: ono-max (Naoto Ono) ---------------------------------------- I’m a software engineer who works at Launchable, Inc. Some Ruby CI maintainers have granted me (Naoto Ono) the permission to integrate our service, [Launchable](https://www.launchableinc.com/) into ruby/ruby. This decision was made during the meeting. Thus, I have created the ticket to share the benefit and progress. ## Progress Currently, both ruby/debug and ruby/vscode-rdbg has started using Launchable. ## Benefits By introducing Launchable, we can enjoy the following benefits: ### Rich UI to see stderr and stdout in failed tests Some Ruby developers find it challenging to review test failure logs in Github Actions. Launchable provides a rich UI that makes it easier to examine stderr and stdout related to test failure. <img width="600" alt="Screenshot 2024-01-29 at 19 23 08" src="https://github.com/ruby/debug/assets/59436572/68f52a50-8e96-4f7f-861d-2dc098dec39f"> ### Test reports in Github comments Identifying which CI job failed in the Github UI can be cumbersome. Launchable addresses this by creating a comment that summarizes test results directly in Github pull request. This enhancement will significantly improve the development experiment. <img width="500" alt="Test reports in Github comments" src="https://github.com/ruby/debug/assets/59436572/283b27d6-7b43-40bf-b423-8bb94708228c"> ### Flakey test insight By developing Ruby, some flakey tests might be added. Launchable has [Unhealthy Tests](https://www.launchableinc.com/docs/features/unhealthy-tests/) page which analyze the flakey tests. For instance, the page shows the flakiness score which express how much the test is flakiness. Let's say `foo` test has a flakiness score of 0.5. This means that if `foo` test failed against 10 different commits, that failure was not a true failure in half of the commits. By seeing this graph, we can prioritize which tests that we have to deal with. In addition to that, we can understand the test is flakey or not. <img width="700" alt="Flakey test page" src="https://github.com/ruby/debug/assets/59436572/ca9f972e-9744-40b9-92a2-353b562d340e"> ### Predictive Test Selection Ruby has an extensive suite of tests, and the number of tests is expected to grow. However, waiting for all tests to pass can be time-consuming. To address this challenge, Launchable offers the [Predictive Test Selection](https://www.launchableinc.com/docs/features/predictive-test-selection/) feature. Predictive Test Selection leverages machine learning to identify the right tests to run for a specific code change. By analyzing data from past test runs and considering the changes being tested, Launchable determines which tests are most relevant. Here is a execution strategy with Launchable. 1. Initial Selection Launchable selects a subset of tests (let's say approximately 25% of all tests) based on the predictive analysis. These tests are executed, first. 2. Remaining Tests The remaining of 75% of tests are then executed. Since the initial selection focuses on potentially problematic areas, there's a high likelihood of identifying any failed tests sooner. -- https://bugs.ruby-lang.org/