CI: Report coverage and test results per library on Codecov - #1360
Open
ACSimon33 wants to merge 19 commits into
Open
CI: Report coverage and test results per library on Codecov#1360ACSimon33 wants to merge 19 commits into
ACSimon33 wants to merge 19 commits into
Conversation
ACSimon33
changed the title
…BLAS/CBLAS/LAPACK/LAPACKE)
ACSimon33
marked this pull request as draft
August 8, 2026 00:23
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1360 +/- ##
===========================================
+ Coverage 0.00% 69.01% +69.01%
===========================================
Files 1937 6122 +4185
Lines 190566 486111 +295545
Branches 0 23286 +23286
===========================================
+ Hits 0 335510 +335510
+ Misses 190566 150412 -40154
- Partials 0 189 +189
... and 6059 files with indirect coverage changes Continue to review full report in Codecov by Harness.
|
ACSimon33
marked this pull request as ready for review
August 8, 2026 16:00
The coverage build measured only the libraries, so nothing showed which parts of the testers themselves never run. The test executables now get the same instrumentation. Their link calls move to the keyword signature of target_link_libraries, which lapack_add_coverage already uses; CMake refuses to mix the two on one target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix generators were the last uninstrumented sources: 74 files that every LAPACK test calls to build its inputs, with no coverage of their own. They report 56% of lines. Their reports and those of the LIN and EIG testers build under TESTING/ rather than SRC/, which no upload covered, so they need one of their own. It carries the existing lapack flag; the flags are unchanged. codecov.yml gains a component for the generators and one per test framework, so that a test written and never reached shows up as such instead of being averaged into the library it exercises. The LAPACKE entry matches nothing until its test suite lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The single badge averaged everything together, so a drop in one library was invisible unless it moved the whole. The badges select components rather than flags. A flag covers whatever its upload contained, which since the test programs became instrumented includes them; the components are scoped to the library sources alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ACSimon33
changed the title
The pin back to v5.5.5 was made while the test analytics reports were not being displayed, on the suspicion that the action was at fault. It was not: the reports carried no JUnit timing fields at all, which is what the parser was tripping over, and the run that first showed them correctly had already been through CI on v5.5.5 without them. So there is nothing to be gained from staying a major version behind. v7.0.0 is the current release, and the workflow was written against it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov's test results parser documents the fields it expects a report to carry: name, tests, failures, errors and time on <testsuites>, those plus skipped and timestamp on <testsuite>, and classname, name and time on <testcase>. Of those, timestamp was the only one the report did not have. A suite is stamped with the earliest start among its cases: the moment this script launched the first driver under --run, and otherwise the modification time of the oldest of their output files, which is when the driver that wrote it finished. That is a close enough stand-in for when the suite ran, and it keeps the stamp of a report built from output files honest rather than claiming the time of the analysis. The synthetic suite that reports unrecognized .out files has no driver behind it, so it is stamped with the report time and given a run time of zero -- a true zero, unlike the unknown time of a case whose output is missing, which continues to carry no time at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Codecov report was a single number for the whole repository, built from a
coverage job that measured almost nothing, and the test results were not
reported at all.
codecov.yml). The test programs and tmglib are components too: a routine that never runs there is a test that was written and never reached. The README badge becomes one badge per library.lapack_add_coverage()helper instead of ad-hoc per-directory code, gcov runs with-b -cso branch coverage exists, and the job prints a per-component table and fails when a component measured nothing — an empty flag would otherwise disappear from the report silently.lapack_testing.pyis uploaded as a test-results report from every test job, so failures show up in the pull request comment (this is a preview feature on codecov.io and doesn't work reliably yet). Each<testsuite>gainedassertionsandtime, and the BLAS, CBLAS and DMD drivers gained theTotal time usedfooter the LAPACK drivers already printed, so that every test case is timed.CODECOV_TOKENhas to be set as a secret of thecodecovenvironment for the uploads to be attributed.