Add manually-triggered workflow for expensive (live-LLM) tests#433
Open
rajeee wants to merge 1 commit into
Open
Add manually-triggered workflow for expensive (live-LLM) tests#433rajeee wants to merge 1 commit into
rajeee wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds infrastructure to safely run opt-in “expensive” pytest tests (those that make live, billable LLM calls) via a manually triggered GitHub Actions workflow, while ensuring they are deselected by default in normal test runs.
Changes:
- Registers an
expensivepytest marker and deselects it by default via globaladdopts. - Adds a
tests-expensivePixi task to run onlyexpensive-marked integration tests. - Introduces a
workflow_dispatch-only GitHub Actions workflow to run these tests on demand with optional filtering.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
pyproject.toml |
Adds expensive marker registration and deselects expensive tests by default via pytest addopts. |
pixi.toml |
Adds a tests-expensive task to run integration tests marked expensive. |
.github/workflows/expensive-tests.yml |
Adds a manual-only workflow to run expensive (live-LLM) tests with configurable model and optional test filtering. |
Comment on lines
+36
to
+42
| - uses: prefix-dev/setup-pixi@1b2de7f3351f171c8b4dfeb558c639cb58ed4ec0 # v0.9.5 | ||
| with: | ||
| pixi-version: v0.62.2 | ||
| locked: true | ||
| cache: true | ||
| cache-write: false | ||
| environments: pdev |
Comment on lines
+57
to
+60
| if [ -n "${{ github.event.inputs.test_filter }}" ]; then | ||
| pixi run -e pdev pytest -rapP -vv -s --log-cli-level=INFO \ | ||
| -m expensive -k "${{ github.event.inputs.test_filter }}" \ | ||
| tests/python/integration |
Comment on lines
+45
to
+55
| env: | ||
| AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }} | ||
| AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }} | ||
| AZURE_OPENAI_VERSION: ${{ secrets.AZURE_OPENAI_VERSION }} | ||
| COMPASS_DATE_TEST_MODEL: ${{ github.event.inputs.model }} | ||
| run: | | ||
| if [ -z "${AZURE_OPENAI_API_KEY}" ] || [ -z "${AZURE_OPENAI_ENDPOINT}" ]; then | ||
| echo "::error::AZURE_OPENAI_API_KEY / AZURE_OPENAI_ENDPOINT secrets are not set." | ||
| echo "Add them in repo Settings -> Secrets and variables -> Actions." | ||
| exit 1 | ||
| fi |
Comment on lines
+56
to
+62
| pixi reinstall -e pdev INFRA-COMPASS | ||
| if [ -n "${{ github.event.inputs.test_filter }}" ]; then | ||
| pixi run -e pdev pytest -rapP -vv -s --log-cli-level=INFO \ | ||
| -m expensive -k "${{ github.event.inputs.test_filter }}" \ | ||
| tests/python/integration | ||
| else | ||
| pixi run -e pdev tests-expensive |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #433 +/- ##
=======================================
Coverage 56.11% 56.11%
=======================================
Files 63 63
Lines 6080 6080
Branches 591 591
=======================================
Hits 3412 3412
Misses 2598 2598
Partials 70 70
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
workflow_dispatch-only workflow to run tests markedexpensive(live, billable LLM calls) on demand from the Actions tab. Registers theexpensivepytest marker, deselects it by default via addopts, and adds atests-expensivepixi task.Split out from #432 so the workflow lands on main first —
workflow_dispatchworkflows only become triggerable once the YAML is on the default branch. The tests it runs come with #432.Requires
AZURE_OPENAI_API_KEYandAZURE_OPENAI_ENDPOINTrepo secrets to run (fails fast with a clear message otherwise).