ci: reintroduce code coverage reporting with cargo-llvm-cov by buraksenn · Pull Request #23336 · apache/datafusion

buraksenn · 2026-07-06T09:01:21Z

Which issue does this PR close?

Closes Coverage jobs are failing on master: Failed to compile tests! Error: sql_integration: linking with cc failed: exit status: 1" #3678.

Rationale for this change

Code coverage has been disabled since #3678. The reason was it failed because of sql_integration test binary. This PR reintroduces it

What changes are included in this PR?

Reintroduce code coverage

Are these changes tested?

I've been successfully make it work in my fork: buraksenn#345

I needed to add CODECOV_TOKEN as a secret variable to the fork.

You can check #23336 (comment) comment of codecov for successful run

Are there any user-facing changes?

no

github-actions · 2026-07-06T09:55:59Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 105.545s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [ 105.374s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.002s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 214.007s] datafusion
    Building datafusion-ffi v54.0.0 (current)
       Built [  58.698s] (current)
     Parsing datafusion-ffi v54.0.0 (current)
      Parsed [   0.065s] (current)
    Building datafusion-ffi v54.0.0 (baseline)
       Built [  58.303s] (baseline)
     Parsing datafusion-ffi v54.0.0 (baseline)
      Parsed [   0.066s] (baseline)
    Checking datafusion-ffi v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.356s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure feature_missing: package feature removed or renamed ---

Description:
A feature has been removed from this package's Cargo.toml. This will break downstream crates which enable that feature.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#cargo-feature-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/feature_missing.ron

Failed in:
  feature tarpaulin_include in the package's Cargo.toml

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 119.158s] datafusion-ffi

codecov-commenter · 2026-07-09T13:04:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@2177864). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23336   +/-   ##
=======================================
  Coverage        ?   80.65%           
=======================================
  Files           ?     1086           
  Lines           ?   366097           
  Branches        ?   366097           
=======================================
  Hits            ?   295265           
  Misses          ?    53237           
  Partials        ?    17595

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

buraksenn · 2026-07-09T13:09:54Z

As expected it could not compare since there is no code coverage uploaded for main branch. After this PR merges it will be ready

Jefffrey

👀

alamb · 2026-07-11T00:21:51Z

+  coverage:
+    name: coverage
+    needs: linux-build-lib
+    runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}


will this consume more of our ASF runner time?

maybe we can try it for a while and make sure it is useful before we spend too much xtra time.

this is correct can you check #23336 (comment)

alamb · 2026-07-11T00:25:06Z

Looks like the coverage run takes 19m and the main tests take 20m, so this adds a lot of resource usage, but it won't increase testing latancy

Jefffrey · 2026-07-11T00:55:28Z

would it also be an option to have it run occasionally on main only, to at least give us a general idea on the health of the codebase in terms of coverage, without consuming too many resources? 🤔

buraksenn · 2026-07-11T09:59:44Z

I've first introduced it as uncommenting the coverage step but cargo-llvm-cov internally runs cargo test and calculates coverage with it so instead of this I've changed cargo test (amd64) to use llvm-cov. With this it actually does not cause any resource usage:

cargo test amd64 with llvm-cov:

unchanged cargo test amd64:

Another approach would be adding it as nightly/weekly. @alamb @Jefffrey

2010YOUY01

LGTM overall—thank you!

Andrew mentioned the resource-usage concern, so here is some background: our CI runner resources are provided by the ASF, and they have noted that DataFusion consumes a significant amount of compute. We are still within the hard limit, IIRC, but it would be good to continue reducing our usage as they suggested.

(I think the biggest contributor is PRs with dozens of commits, but we could address that separately.)

I would prefer to run this check on every PR. It is true that 100% line coverage does not necessarily imply good test quality; however, low coverage strongly suggests that we are missing test cases. I'd want to check the coverage metrics regularly when reviewing.

2010YOUY01 · 2026-07-13T06:35:40Z

 const KB: usize = 1 << 10;
 #[tokio::test]
-#[cfg_attr(tarpaulin, ignore)]
+#[cfg_attr(coverage, ignore)]


I think they're not needed:

datafusion/datafusion/core/tests/fuzz.rs

Line 21 in f87e817

#[cfg(feature = "extended_tests")]

Those tests are under extended_tests attribute, and now we run llvm-cov in the regular test suite, so those slow tests won't be included.

In the future if there are other tests that are too expensive to run for llvm-cov, I think we should also put them in extended_tests feature, then we don't have to manage this additional coverage attribute to keep things simpler.

Thanks for the heads up, I did not think if they were necessary and blindly applied. Now they are removed

…rage-with-llvm-cov

2010YOUY01

Thanks again!

I plan to merge it tomorrow, if the extra CI resource consumption is not acceptable, we could later let it run less frequently.

Jefffrey

this is pretty good if it just substitutes in for cargo test as is 👍

buraksenn · 2026-07-15T11:59:04Z

Thanks again!

I plan to merge it tomorrow, if the extra CI resource consumption is not acceptable, we could later let it run less frequently.

With the current state it does not cause extra CI consumption actually. I've replaced cargo test with llvm-cov which adds trivial overhead as far as I can see from the last 2-3 runs from the CI

2010YOUY01 · 2026-07-16T09:26:37Z

Probably we could add the generated files to .gitignore, to make local runs easier.

However let's merge first, and do it as a follow up PR.

alamb · 2026-07-16T10:29:24Z

Epic!

Jefffrey · 2026-07-16T11:13:19Z

when i open https://app.codecov.io/gh/apache/datafusion it seems to show master as default

the latest commit shown there seems to be pretty old a9f7cac

i cant seem to find main in the dropdown

is this an issue we need to fix? i see a comment on my PR how base (main) doesnt have a coverage report uploaded, is this related?

Various ScalarValue numeric method fixes & refactors (especially decimal) #23631 (comment)

edit: or maybe its still processing on codecov's end

Jefffrey · 2026-07-16T11:22:10Z

oh im seeing this log on main:

info - 2026-07-16 10:36:05,200 -- Found 1 coverage files to report
info - 2026-07-16 10:36:05,200 -- > /__w/datafusion/datafusion/target/codecov.json
info - 2026-07-16 10:36:05,651 -- Upload queued for processing complete
error - 2026-07-16 10:36:05,651 -- Upload queued for processing failed: {"message":"Token required because branch is protected"}

https://github.com/apache/datafusion/actions/runs/29490778583/job/87596785411

alamb · 2026-07-16T17:41:34Z

I filed a ticket to track the fact the reports aren't showing any meaningful coverage reports yet

Code coverage report not working #23647

initial coverage impl

f8630b5

github-actions Bot added development-process Related to development process of DataFusion core Core DataFusion crate ffi Changes to the ffi crate labels Jul 6, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jul 6, 2026

buraksenn added 2 commits July 8, 2026 14:56

utils fix draft

fb33a17

Merge branch 'main' into reintroduce-code-coverage-with-llvm-cov

34ac89c

buraksenn marked this pull request as ready for review July 9, 2026 13:09

buraksenn mentioned this pull request Jul 9, 2026

Coverage jobs are failing on master: Failed to compile tests! Error: sql_integration: linking with cc failed: exit status: 1" #3678

Closed

Jefffrey approved these changes Jul 9, 2026

View reviewed changes

alamb reviewed Jul 11, 2026

View reviewed changes

buraksenn added 2 commits July 11, 2026 11:09

try coverage with test

606e901

upload at the end because of clean directory failure

c896ab7

delete coverage step

ea5f57c

2010YOUY01 reviewed Jul 13, 2026

View reviewed changes

buraksenn added 3 commits July 14, 2026 07:33

Merge branch 'main' into reintroduce-code-coverage-with-llvm-cov

12f7abf

remove unnecessary ignores

69a6cf8

Merge remote-tracking branch 'apache/main' into reintroduce-code-cove…

3006d94

…rage-with-llvm-cov

2010YOUY01 approved these changes Jul 15, 2026

View reviewed changes

Jefffrey approved these changes Jul 15, 2026

View reviewed changes

2010YOUY01 and others added 2 commits July 16, 2026 15:41

Merge branch 'main' into reintroduce-code-coverage-with-llvm-cov

a9b1bb9

move codecov after directory check

8f2b1fb

2010YOUY01 added this pull request to the merge queue Jul 16, 2026

Merged via the queue into apache:main with commit 80bc64f Jul 16, 2026
38 checks passed

buraksenn deleted the reintroduce-code-coverage-with-llvm-cov branch July 16, 2026 11:09

alamb mentioned this pull request Jul 16, 2026

Code coverage report not working #23647

Closed

Uh oh!

Conversation

buraksenn commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

buraksenn commented Jul 9, 2026

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

buraksenn Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 11, 2026

Uh oh!

Jefffrey commented Jul 11, 2026

Uh oh!

buraksenn commented Jul 11, 2026

Uh oh!

2010YOUY01 left a comment

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

buraksenn Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 left a comment

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

buraksenn commented Jul 15, 2026

Uh oh!

2010YOUY01 commented Jul 16, 2026

Uh oh!

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Jefffrey commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jefffrey commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

buraksenn commented Jul 6, 2026 •

edited

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

codecov-commenter commented Jul 9, 2026 •

edited

Loading

Jefffrey commented Jul 16, 2026 •

edited

Loading