Issue #20282 has been updated by ioquatix (Samuel Williams). It also occurred to me, that `Coverage.default_coverage(iseq)` might need to mutate the iseq. In the sense that we need to "turn on coverage trace points/callbacks for the given iseq". In that case, we might want to rename such a method, e.g. `Coverage.enable_coverage_for(iseq)` or something like that. ---------------------------------------- Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports https://bugs.ruby-lang.org/issues/20282#change-106885 * Author: ioquatix (Samuel Williams) * Status: Open * Priority: Normal ---------------------------------------- As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap. ## Objectives The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas: 1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process. 2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable. 3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience. ## Proposed Solutions The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism: ```ruby class Coverage::Capture def self.start self.new.tap(&:start) end # Start receiving coverage callbacks. def start end # Stop receiving coverage callbacks. def stop end # User-overridable statement coverage callback. def statement(iseq, location) fetch(iseq)&.statement_coverage.increment(location) end # Additional methods for branch/declaration coverage would follow a similar pattern. end class MyCoverageCapture < Coverage::Capture # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc. def fetch(iseq) @coverage[iseq] ||= Coverage.default_coverage(iseq) end end # Usage example: my_coverage_capture = MyCoverageCapture.start # Execute test suite or specific tests my_coverage_capture.stop # Access detailed coverage data puts my_coverage_capture.coverage.statement_coverage ``` In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc701021... - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality. ```ruby # Hypothetical interface (mostly copied from vscode's proposed interface): module Coverage # Contains coverage metadata for a file class Target attr_reader :instruction_sequence attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil) @instruction_sequence = instruction_sequence @statement_coverage = statement_coverage @branch_coverage = branch_coverage @declaration_coverage = declaration_coverage end end # Coverage information for a single statement or line. class StatementCoverage # The number of times this statement was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. attr_accessor :executed # Statement location (line number? or range? or position? AST?) attr_accessor :location # Coverage from branches of this line or statement. If it's not a # conditional, this will be empty. attr_accessor :branches # Initializes a new instance of the StatementCoverage class. # # @parameter executed [Number, Boolean] The number of times this statement was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the statement will be marked as un-covered. # # @parameter location [Position, Range] The statement position. # # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line. # If it's not a conditional, this should be omitted. def initialize(executed, location, branches=[]) @executed = executed @location = location @branches = branches end end # Coverage information for a branch class BranchCoverage # The number of times this branch was executed, or a boolean indicating # whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. attr_accessor :executed # Branch location. attr_accessor :location # Label for the branch, used in the context of "the ${label} branch was # not taken," for example. attr_accessor :label # Initializes a new instance of the BranchCoverage class. # # @param executed [Number, Boolean] The number of times this branch was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the branch will be marked as un-covered. # # @param location [Position, Range] (optional) The branch position. # # @param label [String] (optional) Label for the branch, used in the context of # "the ${label} branch was not taken," for example. def initialize(executed, location=nil, label=nil) @executed = executed @location = location @label = label end end # Coverage information for a declaration class DeclarationCoverage # Name of the declaration. Depending on the reporter and language, this # may be types such as functions, methods, or namespaces. attr_accessor :name # The number of times this declaration was executed, or a boolean # indicating whether it was executed if the exact count is unknown. If # zero or false, the declaration will be marked as un-covered. attr_accessor :executed # Declaration location. attr_accessor :location # Initializes a new instance of the DeclarationCoverage class. # # @param name [String] Name of the declaration. # # @param executed [Number, Boolean] The number of times this declaration was executed, or a # boolean indicating whether it was executed if the exact count is unknown. If zero or false, # the declaration will be marked as un-covered. # # @param location [Position, Range] The declaration position. def initialize(name, executed, location) @name = name @executed = executed @location = location end end end ``` -- https://bugs.ruby-lang.org/