Add pytest main CI smoke test

[Herrtian]

Fixes #1132.

This adds a separate non-blocking CI job that installs pytest from the pytest main branch and runs the existing test target against it. The regular tox matrix stays unchanged.

[Herrtian]

The failing timeline check is for the missing changelog fragment. Since this is only adding a CI smoke test and doesn't change runtime behavior or docs, I think this should probably use the skip-changelog label rather than adding a release note.

[seifertm]

Thanks for the initiative @Herrtian !

It's desirable to have parity between the CI/CD workflow and the local development environment. That means, it should be possible to run the tests against pytest dev in the local development environment.

There was another Pull Request which seems like it had this figured out. Can you take a look and extend the local development environment so that there's a tox environment that runs all tests against the development version of pytest?

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.50%. Comparing base (31772d2) to head (d151e90).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1431   +/-   ##
=======================================
  Coverage   94.50%   94.50%           
=======================================
  Files           2        2           
  Lines         510      510           
  Branches       62       62           
=======================================
  Hits          482      482           
  Misses         22       22           
  Partials        6        6           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Herrtian]

Thanks, updated this in 09a6a37. The job name now uses a literal latest, and the CI job runs a new local tox env, pytest-dev, so the same pytest-main check can be run outside CI too. I checked the tox config locally and ran the env against pytest main. It currently surfaces a few strict-mode behavior failures from pytest main, so I kept the CI job non-blocking as originally intended.

[seifertm]

Test errors in the pytest-dev environment are related to pytest's stricter handling of async fixtures, which now gives an error, instead of a warning.

[seifertm]

Thanks @Herrtian !

[seifertm]

Test errors in the pytest-dev environment are related to pytest's stricter handling of async fixtures, which now gives an error, instead of a warning.

Actually, no. This was introduced as early as v8.4.0. Something else is off here.

[seifertm]

pytest-dev/pytest#14015 is the responsible change. We need to adjust the tests accordingly.

[seifertm]

I decided to skip the offending tests on Pytest 9.1 onwards. They will no longer be needed in pytest-asyncio once the minimum pytest version is bumped to v9.1, because the logic is already handled in pytest.