Issue #20282 has been updated by mame (Yusuke Endoh).
Coverage.suspend and Coverage.resume does not work for multi-thread test runner unfortunately
Yes, I mentioned the issue when I propose `Coverage.resume/suspend` in https://bugs.ruby-lang.org/issues/18176#note-1. Per-thread coverage is a far more difficult problem than per-test one. This would significantly conflict with the original API design and the current implementation. The format of Coverage.result will change again, and the internal data structure will need to be also changed. The coverage library has been repeatedly extended so that both the API and the implementation have become rather complicated. This was something I regretted for a long time, but "coverage for eval" has led me to believe that we should not repeat it any further. I hope you understand.
That is outlined in the proposal, e.g. Coverage.enable_coverage(iseq) etc. It might be possible to use RubyVM internals to achieve this, but it won't be compatible with TruffleRuby or JRuby etc.
Dealing with iseq directly is not compatible with other Ruby implementation, no? And a low-level method to instrument an existing iseq is clearly beyond the scope of responsibility of Coverage library. It clearly looks like a messed up proposal to me. I hope you understand that there is no way such a proposal is accepted. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106905 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location. # Similar structures for other coverage data. def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` By following this format, we will be compatible with a wide range of external tools. -- https://bugs.ruby-lang.org/