Skip to content

Add Copilot-driven test coverage agent#4853

Open
Copilot wants to merge 7 commits intomainfrom
copilot/add-copilot-test-coverage-agent
Open

Add Copilot-driven test coverage agent#4853
Copilot wants to merge 7 commits intomainfrom
copilot/add-copilot-test-coverage-agent

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 31, 2026 •
edited by hanniavalera
Loading

This pull request introduces a comprehensive automated test coverage tracking system for the repository. It adds a scheduled GitHub Actions workflow that measures backend test coverage, creates or updates issues when coverage falls below a configurable threshold, and standardizes the process for contributors to improve test coverage. The changes include new workflow automation, scripts, contributor instructions, templates, and a new dev dependency.

Coverage automation and workflow:

  • Added a new GitHub Actions workflow (.github/workflows/coverage-agent.yml) that runs weekly or on demand, measures backend test coverage using c8, uploads coverage reports, and triggers a script to open or update an issue if coverage is below a threshold.
  • Introduced a Node.js script (.github/scripts/check-coverage-and-open-issue.mjs) that parses coverage results, identifies files below the threshold, ensures the appropriate GitHub label exists, and creates or updates a detailed coverage issue with actionable instructions for contributors.

Contributor guidance and templates:

  • Added detailed instructions for Copilot and contributors on how to improve test coverage, including a mandatory self-audit protocol, quality rules, and PR requirements in .github/instructions/copilot-test-coverage.instructions.md.
  • Created a standardized pull request template (.github/PULL_REQUEST_TEMPLATE/coverage.md) specifically for coverage improvement PRs, including a table for coverage deltas and a checklist to enforce quality and process.
  • Updated the general contributor instructions (.github/copilot-instructions.md) to reference the new coverage protocol and instructions for test-coverage issues.

Tooling:

  • Added c8 as a development dependency in package.json to enable V8-native code coverage collection.

Copilot AI changed the title [WIP] Add Copilot-driven test coverage agent Add Copilot-driven test coverage agent Mar 31, 2026
Copilot AI requested a review from hanniavalera March 31, 2026 16:23
@hanniavalera hanniavalera force-pushed the copilot/add-copilot-test-coverage-agent branch 4 times, most recently from cf2596c to e29f705 Compare April 6, 2026 19:59
gcampbell-msft
gcampbell-msft reviewed Apr 7, 2026
View reviewed changes
@hanniavalera hanniavalera force-pushed the copilot/add-copilot-test-coverage-agent branch from e29f705 to c1c0662 Compare April 7, 2026 14:28
@hanniavalera hanniavalera marked this pull request as ready for review April 7, 2026 17:11
@hanniavalera hanniavalera requested a review from snehara99 April 7, 2026 18:24
@hanniavalera hanniavalera force-pushed the copilot/add-copilot-test-coverage-agent branch from e4b7b7a to 603f8e2 Compare April 7, 2026 20:25
@hanniavalera hanniavalera force-pushed the copilot/add-copilot-test-coverage-agent branch from 603f8e2 to 94b8a3e Compare April 8, 2026 16:31
gcampbell-msft
gcampbell-msft reviewed Apr 9, 2026
View reviewed changes
.github/PULL_REQUEST_TEMPLATE/coverage.md
@@ -0,0 +1,33 @@
---
Copy link
Copy Markdown
Collaborator

@gcampbell-msft gcampbell-msft Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any specific thing that is supposed to read this? Copilot will grab it when it searches for files, but from what I can tell, it doesn't match any filename format that copilot cli inherently reads, so I'm not sure it will get reliably run.

Is there something I'm missing here?

Copy link
Copy Markdown
Contributor

@hanniavalera hanniavalera Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see! Thanks for taking a look at this :) The template isn't auto-applied by GitHub or Copilot, it has to be explicitly referenced. The coding agent is instructed to use it in .github/instructions/copilot-test-coverage.instructions.md, but relying on the agent consistently reading and following that is fragile. I'll either rename it to PULL_REQUEST_TEMPLATE.md so it becomes the default single template, or add an explicit instruction in the issue body itself directing the agent to use the ?template=coverage.md URL when opening the PR. Leaning toward the latter since the first would affect all PRs.

gcampbell-msft
gcampbell-msft reviewed Apr 9, 2026
View reviewed changes
.github/scripts/check-coverage-and-open-issue.mjs
const belowThreshold = [];
const repoRoot = resolve('.');

for (const [file, metrics] of Object.entries(summary)) {
Copy link
Copy Markdown
Collaborator

@gcampbell-msft gcampbell-msft Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we tested this locally to validate that it works?

@hanniavalera hanniavalera force-pushed the copilot/add-copilot-test-coverage-agent branch from f38c7f1 to a402a11 Compare April 9, 2026 18:05
@hanniavalera hanniavalera self-requested a review April 9, 2026 18:09
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 9, 2026

@hanniavalera Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: a3342342-f741-4090-a919-a31754359f5c

Sorry for the inconvenience!

2 similar comments
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 9, 2026

@hanniavalera Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: a3342342-f741-4090-a919-a31754359f5c

Sorry for the inconvenience!

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 9, 2026

@hanniavalera Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: a3342342-f741-4090-a919-a31754359f5c

Sorry for the inconvenience!

Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 9, 2026

@hanniavalera Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: a3342342-f741-4090-a919-a31754359f5c

Sorry for the inconvenience!

1 similar comment
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 9, 2026

@hanniavalera Unfortunately I hit an unexpected error while processing your comment. I've automatically reported this to GitHub.

You can ask me to try again later by mentioning me in a new comment.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: a3342342-f741-4090-a919-a31754359f5c

Sorry for the inconvenience!

Copilot AI and others added 7 commits April 9, 2026 16:24
@hanniavalera
Add Copilot-driven test coverage agent (4-file system)
a4f7acf 
Agent-Logs-Url: https://github.com/microsoft/vscode-cmake-tools/sessions/2ef2460f-de0e-4cf1-bca3-cc67940ee648

Co-authored-by: hanniavalera <90047725+hanniavalera@users.noreply.github.com>
@hanniavalera
Update coverage tooling to use c8 instead of nyc and adjust related c…
2322c5b 
…onfigurations
@hanniavalera
updated lock file
924bc03
@hanniavalera
enhance test coverage protocols and scripts for Copilot integration
bbbf730
@hanniavalera
switch to use spawnSync for gh CLI commands and improve error handling
0d5dfa7
@hanniavalera
Move copilot-test-coverage to .github/instructions/ and update refere…
cd1dc0a 
…nces

Rename .github/copilot-test-coverage.md to
.github/instructions/copilot-test-coverage.instructions.md so Copilot
picks it up as an instructions file. Update the two references in
copilot-instructions.md and check-coverage-and-open-issue.mjs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hanniavalera
Add explicit PR template instructions in issue body and agent protocol
9f84254 
Address review feedback:
- Add '### PR template' section in generated issue body directing the
  agent to use ?template=coverage.md or copy template contents
- Update copilot-test-coverage.instructions.md PR requirements with
  explicit template usage instructions
- Validated script locally with mock data: below-threshold, above-
  threshold, and absolute-path normalization all pass correctly

Agent-Logs-Url: https://github.com/microsoft/vscode-cmake-tools/sessions/cb96c55e-0fc9-4e59-b0a8-a1fc704e2182

Co-authored-by: hanniavalera <90047725+hanniavalera@users.noreply.github.com>
@hanniavalera hanniavalera force-pushed the copilot/add-copilot-test-coverage-agent branch from 8e0fbe3 to 9f84254 Compare April 9, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gcampbell-msft gcampbell-msft gcampbell-msft left review comments

@itodirel itodirel Awaiting requested review from itodirel

@snehara99 snehara99 Awaiting requested review from snehara99

@hanniavalera hanniavalera Awaiting requested review from hanniavalera

At least 1 approving review is required to merge this pull request.

Assignees

@hanniavalera hanniavalera

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gcampbell-msft @hanniavalera