Skip to content

Check requires-mark coverage across CI runs#11293

Open
officialasishkumar wants to merge 3 commits intopydata:mainfrom
officialasishkumar:ci-11273-requires-mark-coverage
Open

Check requires-mark coverage across CI runs#11293
officialasishkumar wants to merge 3 commits intopydata:mainfrom
officialasishkumar:ci-11273-requires-mark-coverage

Conversation

@officialasishkumar
Copy link
Copy Markdown

Description

Add a CI safeguard that uploads pytest-reportlog output from each test-matrix job and checks that tests skipped via requires_* markers are exercised somewhere in CI, unless the dependency gate is explicitly allowlisted as unsupported in GitHub Actions.

This also adds focused tests for the coverage checker script.

Checklist

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
      Tools: Codex

Test plan

  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -o addopts= /tmp/xarray-wt-11273/ci/test_check_requires_coverage.py -q

@github-actions github-actions bot added CI Continuous Integration tools Automation Github bots, testing workflows, release automation labels Apr 9, 2026
@VeckoTheGecko
Copy link
Copy Markdown
Contributor

Hi @officialasishkumar , thanks for looking into this!

Hmmm. I was wondering whether in the original issue whether this can be explored using custom marks (whether that's something that Pytest supports). Have you looked into this @officialasishkumar ? Maybe that would be a way forward?

Solely grepping through error messages seems quite error prone/flaky and I don't know if that's the way we should be handling something as fundamental as determining if our test suite passes

Also, can you elaborate on how you tested this @officialasishkumar ? The test plan doesn't really make sense to me and the job was failing with many errors

@VeckoTheGecko
Copy link
Copy Markdown
Contributor

cc @keewis since you were thinking along with me in the original issue

@VeckoTheGecko
Copy link
Copy Markdown
Contributor

@officialasishkumar

I see that you've posted #11290, #11291, #11292, and this PR in very quick succession. Have you read through our AI policy? https://docs.xarray.dev/en/latest/contribute/ai-policy.html And can you confirm whether all of these contributions are in line with this policy?

This is a lot of PRs to go through and I want to make sure that maintainers time is being respected while reviewing these

@officialasishkumar
Copy link
Copy Markdown
Author

hi @VeckoTheGecko , yes, i have gone through the AI policy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Automation Github bots, testing workflows, release automation CI Continuous Integration tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CI to ensure all requires_ tests are run at least once

2 participants