Add RFC 019: Code coverage messages & consumer model for MTP#9906
Merged
Conversation
Proposes a cross-language-informed coverage contract for Microsoft.Testing.Platform: counts-based measurements, a seeded append-only metric enum, scope-aware threshold results, a typed report-artifact reference, and an ITestCoverageResult consumer read model. Provides design context for PR #9896. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cec9403-3bea-4115-bd84-b02d37197b94
Contributor
There was a problem hiding this comment.
Pull request overview
Adds RFC 019, proposing a first-class code coverage contract for Microsoft.Testing.Platform.
Changes:
- Defines count-based, scope-aware coverage messages and thresholds.
- Proposes a correlated consumer read model and artifact references.
- Documents platform wiring, exit-code behavior, and open design questions.
Show a summary per file
| File | Description |
|---|---|
docs/RFCs/019-Code-Coverage-Messages.md |
Introduces the code coverage messages and consumer-model RFC. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 6
- Review effort level: Medium
Evangelink
changed the title
- Scope levels are overlapping dimensions, not a strict tree (type<->file is many-to-many); replace ParentPath with a non-authoritative ContainerHint and point exact containment at the report artifact. - Threshold aggregation now names its population via AggregatedOver so (scope, metric, aggregation) is unambiguous; add CoverageAggregation.None. - Add ProducerId to measurements/thresholds/results so multiple collectors are not silently merged. - Validate threshold percentages (reject NaN/inf/out-of-range) and enforce the Custom metric-name invariant. - Fix the Custom-metric indexer ambiguity with a GetCustom(name) lookup. - Make the consumer example compile against the real ITestSessionLifetimeHandler / IDataConsumer contracts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cec9403-3bea-4115-bd84-b02d37197b94
- CoverageScope: enforce name invariant (only Overall is unnamed); exclude the non-authoritative ContainerHint from equality/hashing so it never splits scope identity. - Require a non-empty ProducerId on measurements, thresholds, reports and CoverageMetricResult so multi-collector data is never silently merged. - Validate coverableCount and coveredCount separately for correct parameter attribution. - Threshold: carry HasCoverableData + TreatNoDataAsFailure so no-data is distinct from genuine 0% and Passed applies an explicit policy. - Report message: enforce the Custom<->CustomFormatName discriminated-pair invariant both ways; require a report path. - Read-model classes: add explicit (internal) constructors so they compile under nullable/warnings-as-errors and aren't publicly constructible in an invalid state. - Consumer example: use a stable literal Uid instead of nameof(...). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cec9403-3bea-4115-bd84-b02d37197b94
- Enforce the Custom<->CustomMetricName discriminated pair both ways on TestCoverageMessage and TestCoverageThresholdMessage (reject a custom name when metric is not Custom). - Normalize ActualPercentage to a stable 0 in the no-data case so NaN/Infinity can never leak into consumers that read it without checking HasCoverableData. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cec9403-3bea-4115-bd84-b02d37197b94
0101
approved these changes
Jul 13, 2026
- Reframe the raw layer as a deliberately reduced line/branch common denominator, not full-fidelity primitives (statement/instruction/block/region can't be reconstructed from line+branch); authoritative detail lives in the artifact. - Derive all three messages from DataWithSessionUid instead of PropertyBagData and add SessionUid to the correlation key, so multi-session hosts attribute coverage correctly. - Give CoverageScopeSummary compilable indexer/GetCustom implementations backed by Metrics (incl. the Custom-rejection and ordinal name match). - Make the read-model DTO constructors public (validated, get-only props) so third-party consumers can construct snapshots and fakes in tests without reflection. - Define the full correlation key (SessionUid, ProducerId, Scope, Metric, CustomMetricName-when-Custom) and specify last-write-wins on duplicate keys. - Update the property-bag open question to reflect the new base. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 5cec9403-3bea-4115-bd84-b02d37197b94
|
|
||
| public override bool Equals(object? obj) => obj is CoverageScope s && Equals(s); | ||
|
|
||
| public override int GetHashCode() => HashCode.Combine(Level, Name); |
Comment on lines
+596
to
+603
| // Public so third-party report generators can construct snapshots in their own unit tests | ||
| // and build fakes of ITestCoverageResult without reflection. Get-only properties still | ||
| // prevent mutation after construction. | ||
| public CoverageScopeSummary(CoverageScope scope, IReadOnlyList<CoverageMetricResult> metrics) | ||
| { | ||
| Scope = scope; | ||
| Metrics = metrics ?? throw new ArgumentNullException(nameof(metrics)); | ||
| } |
Comment on lines
+663
to
+667
| Metric = metric; | ||
| CoveredCount = coveredCount; | ||
| CoverableCount = coverableCount; | ||
| ProducerId = producerId; | ||
| CustomMetricName = customMetricName; |
Comment on lines
+694
to
+697
| Path = path; | ||
| Format = format; | ||
| ProducerId = producerId; | ||
| CustomFormatName = customFormatName; |
Comment on lines
+761
to
+766
| The read model correlates messages into `CoverageScopeSummary` / threshold / report entries | ||
| using the **full** key, not just `(scope, metric)`: | ||
|
|
||
| ```text | ||
| (SessionUid, ProducerId, Scope, Metric, CustomMetricName when Metric == Custom) | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds RFC 019 (
docs/RFCs/019-Code-Coverage-Messages.md), a design proposal for a first-class code coverage contract in Microsoft.Testing.Platform.This RFC provides the design context for #9896, which introduces permanent public coverage API while the producer side does not yet exist. Since public API is effectively forever, the shape is captured here for team sign-off before implementation continues.
What it proposes
covered/coverableand derive%, matching every surveyed ecosystem (lcov, Cobertura, JaCoCo, Istanbul, coverage.py, llvm-cov, SonarQube generic, Codecov) and enabling honest cross-scope aggregation.CoverageMetricenum (Line/Statement/Branch/Method/Function/Block/Instruction/Region/Class/Condition/Complexity) with a singleCustomescape hatch — instead of the prototype's 3-value Line/Branch/Method enum (which can't even nameblock, the primary metric of Microsoft's owndotnet-coverage).TestCoverageReportMessage/CoverageReportReference) — per-line fidelity stays in the standardized report file rather than being duplicated over IPC.ITestCoverageResultconsumer read model that correlates data once, so third-party extensions (HTML/UI report generators, PR-comment bots, dashboards, badges) — and the terminal itself — consume a single source of truth without re-aggregating.Evidence
The design is grounded in a cross-language survey of how real coverage tools model their data (metric vocabularies and raw file formats), plus the precedent of purpose-built language-agnostic contracts (SonarQube generic coverage, Codecov), which both chose a small closed model and derive the rest.
Notes
docs/RFCs/house style and status checkboxes; marked Under discussion.This RFC was drafted with AI assistance during a review of #9896.