[ruby-core:116853] [Ruby master Feature#20282] Enhancing Ruby's Coverage with Per-Test Coverage Reports
Issue #20282 has been reported by ioquatix (Samuel Williams). ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` -- https://bugs.ruby-lang.org/
Issue #20282 has been updated by ioquatix (Samuel Williams). It also occurred to me, that `Coverage.default_coverage(iseq)` might need to mutate the iseq. In the sense that we need to "turn on coverage trace points/callbacks for the given iseq". In that case, we might want to rename such a method, e.g. `Coverage.enable_coverage_for(iseq)` or something like that. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106885 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` -- https://bugs.ruby-lang.org/
Issue #20282 has been updated by mame (Yusuke Endoh). `Coverage.resume` and `Coverage.suspend` were introduced for per-test coverage. You can use them as follows. ```ruby # target.rb def test1 p 1 end def test2 p 2 end ``` ```ruby require "coverage" Coverage.setup load "target.rb" Coverage.resume test1 Coverage.suspend pp Coverage.result(stop: false, clear: true)["test.rb"] #=> [0, 1, nil, nil, 0, 0, nil] Coverage.resume test2 Coverage.suspend pp Coverage.result(stop: false, clear: true)["test.rb"] #=> [0, 0, nil, nil, 0, 1, nil] ``` BTW, in https://bugs.ruby-lang.org/issues/19857#change-106238, I think I clearly said that I do no longer intend to make Coverage library complicated for the time being. I was honestly even disappointed for you to create this proposal. Why don't you understand? Is this because of my bad English? Frankly, I am very tired of communicating with you. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106892 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location. # Similar structures for other coverage data. def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` By following this format, we will be compatible with a wide range of external tools. -- https://bugs.ruby-lang.org/
Issue #20282 has been updated by ioquatix (Samuel Williams). `Coverage.suspend` and `Coverage.resume` does not work for multi-thread test runner unfortunately, unless there is something I'm missing. In my integration with `vscode` and `sus`, this can be a problem, as the test host can run multiple tests at the same time in order to handle the requests from `vscode`.
Why don't you understand? Is this because of my bad English? Frankly, I am very tired of communicating with you.
I'm sorry to hear that. When we discussed it last time, you suggested introducing some kind of "coverage2" library:
I don't have time right now, but I would like to rebuild the library as a pure external gem in future, i.e. coverage2 or something, with dynamically generated code, etc. in mind from the beginning.
This proposal does not change the existing interface, but introduce new interface which conceivably be in a gem. However, such a `coverage2` cannot exist without some support from Ruby, e.g. instrumentation of instruction sequence, and so on. That is outlined in the proposal, e.g. `Coverage.enable_coverage(iseq)` etc. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106893 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location. # Similar structures for other coverage data. def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` By following this format, we will be compatible with a wide range of external tools. -- https://bugs.ruby-lang.org/
Issue #20282 has been updated by mame (Yusuke Endoh).
Coverage.suspend and Coverage.resume does not work for multi-thread test runner unfortunately
Yes, I mentioned the issue when I propose `Coverage.resume/suspend` in https://bugs.ruby-lang.org/issues/18176#note-1. Per-thread coverage is a far more difficult problem than per-test one. This would significantly conflict with the original API design and the current implementation. The format of Coverage.result will change again, and the internal data structure will need to be also changed. The coverage library has been repeatedly extended so that both the API and the implementation have become rather complicated. This was something I regretted for a long time, but "coverage for eval" has led me to believe that we should not repeat it any further. I hope you understand.
That is outlined in the proposal, e.g. Coverage.enable_coverage(iseq) etc. It might be possible to use RubyVM internals to achieve this, but it won't be compatible with TruffleRuby or JRuby etc.
Dealing with iseq directly is not compatible with other Ruby implementation, no? And a low-level method to instrument an existing iseq is clearly beyond the scope of responsibility of Coverage library. It clearly looks like a messed up proposal to me. I hope you understand that there is no way such a proposal is accepted. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106905 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location. # Similar structures for other coverage data. def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` By following this format, we will be compatible with a wide range of external tools. -- https://bugs.ruby-lang.org/
Issue #20282 has been updated by anmarchenko (Andrey Marchenko). ioquatix (Samuel Williams) wrote:
As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap.
Hi! I think this would be a great addition to Ruby, it would help me to solve an important business case at Datadog. I work as a library developer at Datadog and we are working on developer tools for test visibility and smart test execution in Ruby. One product that we are working on right now for Ruby is an intelligent test runner https://docs.datadoghq.com/intelligent_test_runner that saves time for devs by running only relevant tests for a feature branch. We use test impact analysis technique for that https://martinfowler.com/articles/rise-test-impact-analysis.html that requires us to collect per test code coverage for users of our library when they run their test suite using minitest/rspec/cucumber/etc. We need to create a solution that: - tracks code coverage per test while being invisible for users, do not interferes with existing simplecov/covered setup - has low performance overhead - works with parallel and sequential test runners I am evaluating several possible solutions for this case, one being using `Coverage.suspend/resume` with `Coverage.result(stop: false, clear: true)`. The most important problem with this solution for me is not only that it does not support threaded parallel executors, but it also skews existing code coverage that our users might have. All major code coverage libraries in Ruby (simplecov/covered/single_cov) use Coverage under the hood and Coverage has only one global state. So if we would use suspend/resume and clear results after each test, this would break all existing code coverage libraries. I would be delighted to see some API to spy on current code coverage collection and receive what was covered between specific points of time (start/stop of each test) and I think it would benefit many developer tools out there. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106915 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location. # Similar structures for other coverage data. def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` By following this format, we will be compatible with a wide range of external tools. -- https://bugs.ruby-lang.org/
Issue #20282 has been updated by mame (Yusuke Endoh). @anmarchenko I think you want to look into a C API called `rb_thread_add_event_hook`. You can use it to register your C hook function to be called for each specified event, e.g., line execution (RUBY_EVENT_LINE), method call (RUBY_EVENT_CALL), etc. You can register a hook per thread. It also allows multiple hooks registered simultaneously. It is orthogonal with Coverage, i.e., does not interfere with an existing Coverage setup. You need to write C, but you can create something specific to your use case, which could be more efficient than general-purpose Coverage library. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106923 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location. # Similar structures for other coverage data. def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` By following this format, we will be compatible with a wide range of external tools. -- https://bugs.ruby-lang.org/
participants (3)
-
anmarchenko (Andrey Marchenko) -
ioquatix (Samuel Williams)
-
mame (Yusuke Endoh)