Ci/trustworthy system tests

.agents/skills/run-system-tests/SKILL.md

@@ -232,7 +232,7 @@ The workflow:
 2. Opens an in-progress GitHub Check Run on the PR's head SHA so the run shows up in the **Checks** tab (issue_comment events otherwise associate runs with the default branch)
 3. Runs pytest on a freshly-spawned ephemeral OSMO GPU pod (`runs-on: [self-hosted, airstack-ephemeral]`)
 4. Uploads `tests/results/` as artifact `test-results-<sha>-<run_id>` (90-day retention)
-5. The downstream `report` job runs `parse_metrics.py`, compares only a matching complete simulation baseline, posts the result, and finalizes the PR-head Check Run
+5. The downstream `report` job selects the newest matching complete simulation baseline, runs `parse_metrics.py`, posts the advisory comparison, and finalizes the PR-head Check Run
 6. Closes the Check Run with the final conclusion
 
 ### Why fork PRs are blocked
@@ -284,7 +284,7 @@ Keys follow `test_node_id → metric_key → {value, unit, direction, ...}`. Tim
 # Single-run report — markdown table, exits 0 always
 python tests/parse_metrics.py --current tests/results/2025-04-21_14-30-00/
 
-# Diff mode — side-by-side, exits 1 on regression
+# Comparison mode — side-by-side; numeric deltas are advisory
 python tests/parse_metrics.py \
   --current  tests/results/2025-04-21_14-30-00/ \
   --baseline tests/results/2025-04-20_09-00-00/ \
@@ -298,7 +298,11 @@ The report has three sections per test module:
 - **Sim publishing rates** — pivoted Hz aggregates per topic (`mean`, `start_mean`, `end_mean`, `min`, `max`) from the `sensors` mark (sim + robot streams)
 - **Compute usage** — pivoted CPU/mem/GPU per container
 
-Regressions exceeding `--threshold` (default 20%) are flagged `:red_circle:`; improvements beyond threshold get `:green_circle:`. CI fails only when both artifacts are complete and have the same simulation campaign fingerprint.
+Changes exceeding `--threshold` (default 20%) are flagged `:red_circle:` or
+`:green_circle:` for review. Numeric deltas never fail CI. Pytest assertions,
+infrastructure/prerequisite failures, missing artifacts, and report-parser
+errors remain blocking. The fingerprint includes normalized tests and all
+behavior-changing campaign options.
 
 When local-debugging a CI regression, download both artifacts (`test-results-<sha>-<run_id>` from the PR run and from the base branch's most recent run), unzip them under `tests/results/`, and run `parse_metrics.py` locally to see the same table the bot posted.
 
@@ -453,7 +457,7 @@ python tests/parse_metrics.py \
 - `tests/harness/` — helpers split by concern: `session`, `discovery`, `commands`, `containers`, `metrics` (`MetricsRecorder`), `run_meta`, `test_ids`, `sim`, `collection` (ordering)
 - `tests/meta/` — fast contract tests (`unit` mark) pinning CLI/docs/stack contracts
 - `tests/pytest.ini` — mark registration, log format
-- `tests/parse_metrics.py` — markdown reporter, regression diff
+- `tests/parse_metrics.py` — markdown reporter and advisory comparison
 - `tests/README.md` — user-facing docs (CLI options, output layout, CI/CD orchestrator)
 - `.github/workflows/system-tests.yml` — CI workflow with `/pytest` comment trigger
 - `.github/orchestrator/README.md` — ephemeral OSMO runner setup and worker-debug procedure

.env

@@ -12,7 +12,7 @@ PROJECT_NAME="airstack"
 # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made 
 # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version.
 # auto-generated from git commit hash
-VERSION="0.20.0-alpha.19"
+VERSION="0.20.0-alpha.20"
 # Image-tag discriminator ONLY (appears in the image tag suffix, e.g. ..._robot-x86-64_dev).
 # No Dockerfile consumes it: "prebuilt" does NOT bake the built ros_ws into the image today —
 # a real prebuilt (workspace-baked) stage is future work. Keep "dev" (mounted code, built live).

.github/workflows/module-system-tests.yml

@@ -32,9 +32,10 @@ name: Module System Tests
 #     -f airstack_ref=develop -f marks="build_packages or liveliness"
 #
 # GPU runner note: with the default runs_on, the job queues for the ephemeral
-# OSMO-backed runners (.github/orchestrator/). The orchestrator polls one repo
-# per instance, so a module repo must be added to the poll list before its
-# calls can be picked up — see .github/orchestrator/README.md "Module repos".
+# OSMO-backed runners (.github/orchestrator/). The orchestrator polls a
+# `repos:` list, so a module repo must be added to that list (and covered by
+# the orchestrator PAT) before its calls can be picked up — see
+# .github/orchestrator/README.md "Module repos".
 #
 # Docs: docs/development/module_ci.md
 

.github/workflows/system-tests.yml

@@ -30,6 +30,14 @@ on:
         description: "Seconds for test_stable polling window"
         default: "120"
         required: false
+      trajectory_types:
+        description: "Fixed trajectories, comma-separated (e.g. Circle or Circle,Figure8)"
+        default: "Circle,Figure8,Racetrack,Line"
+        required: false
+      takeoff_velocities:
+        description: "Takeoff velocities, comma-separated (e.g. 0.5 or 0.5,1)"
+        default: "0.5"
+        required: false
       baseline_run_id:
         description: "Run ID to use as baseline for metric comparison (blank = latest successful run on main)"
         default: ""
@@ -58,8 +66,8 @@ jobs:
        startsWith(github.event.comment.body, '/pytest') &&
        contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association))
     concurrency:
-      group: system-tests-${{ github.event_name == 'pull_request' && github.event.pull_request.number || github.run_id }}
-      cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+      group: system-tests-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }}
+      cancel-in-progress: true
     timeout-minutes: 120
     # Adding any `permissions:` entry disables GITHUB_TOKEN's defaults, so
     # every scope used here has to be re-granted explicitly:
@@ -111,7 +119,6 @@ jobs:
             core.setOutput('base_ref', pr.data.base.ref);
 
       - name: Resolve tested revision identity
-        if: always()
         id: identity
         env:
           EVENT_NAME: ${{ github.event_name }}
@@ -121,7 +128,11 @@ jobs:
           EVENT_PR_NUMBER: ${{ github.event.pull_request.number }}
         run: |
           if [[ "$EVENT_NAME" == "issue_comment" ]]; then
-            echo "tested_sha=${COMMENT_HEAD_SHA:-$EVENT_SHA}" >> "$GITHUB_OUTPUT"
+            if [[ -z "$COMMENT_HEAD_SHA" ]]; then
+              echo "::error::Refusing /pytest run: PR head SHA was not resolved."
+              exit 1
+            fi
+            echo "tested_sha=$COMMENT_HEAD_SHA" >> "$GITHUB_OUTPUT"
             echo "pr_number=$COMMENT_PR_NUMBER" >> "$GITHUB_OUTPUT"
           else
             echo "tested_sha=$EVENT_SHA" >> "$GITHUB_OUTPUT"
@@ -141,6 +152,8 @@ jobs:
           INPUT_NUM_ROBOTS: ${{ inputs.num_robots }}
           INPUT_ITERATIONS: ${{ inputs.stress_iterations }}
           INPUT_STABLE: ${{ inputs.stable_duration }}
+          INPUT_TRAJECTORIES: ${{ inputs.trajectory_types }}
+          INPUT_TAKEOFF_VELOCITIES: ${{ inputs.takeoff_velocities }}
         run: |
           python3 <<'PYEOF'
           import os, shlex, sys
@@ -158,6 +171,10 @@ jobs:
                   args.extend(['--stress-iterations', it])
               if (st := os.environ.get('INPUT_STABLE', '').strip()):
                   args.extend(['--stable-duration', st])
+              if (trajectories := os.environ.get('INPUT_TRAJECTORIES', '').strip()):
+                  args.extend(['--trajectory-types', trajectories])
+              if (velocities := os.environ.get('INPUT_TAKEOFF_VELOCITIES', '').strip()):
+                  args.extend(['--takeoff-velocities', velocities])
           elif event == 'pull_request':
               # Automatic PR validation is deliberately build-scoped. Fast
               # Python unit tests run in unit-tests.yml; GPU simulation remains
@@ -424,6 +441,8 @@ jobs:
       - name: Run tests
         env:
           AIRSTACK_ROOT: ${{ github.workspace }}
+          AIRSTACK_TESTED_SHA: ${{ steps.identity.outputs.tested_sha }}
+          AIRSTACK_PR_NUMBER: ${{ steps.identity.outputs.pr_number }}
           DISPLAY: ""
           PYTEST_ARGS: ${{ steps.parse.outputs.pytest_args }}
         run: |
@@ -486,6 +505,7 @@ jobs:
     if: >
       always() &&
       needs.run-tests.result != 'skipped' &&
+      needs.run-tests.outputs.tested_sha != '' &&
       (needs.run-tests.result != 'cancelled' || github.event_name != 'pull_request')
     permissions:
       actions: read
@@ -533,15 +553,19 @@ jobs:
       # the PR's base branch (e.g. develop or main).
       - name: Download baseline results (PR)
         if: github.event_name == 'issue_comment' || github.event_name == 'pull_request'
-        uses: dawidd6/action-download-artifact@v6
-        continue-on-error: true
-        with:
-          workflow: system-tests.yml
-          branch: ${{ steps.pr_ctx.outputs.base_ref }}
-          name_is_regexp: true
-          name: "test-results-.*"
-          path: baseline-results/
-          if_no_artifact_found: warn
+        env:
+          GH_TOKEN: ${{ github.token }}
+          BASE_REF: ${{ steps.pr_ctx.outputs.base_ref }}
+        run: |
+          mkdir -p baseline-results
+          gh api --method GET \
+            "repos/${{ github.repository }}/actions/workflows/system-tests.yml/runs" \
+            -f branch="$BASE_REF" -f status=success -f per_page=20 \
+            --jq '.workflow_runs[].id' |
+          while read -r run_id; do
+            gh run download "$run_id" --repo "${{ github.repository }}" \
+              --pattern "test-results-*" --dir "baseline-results/$run_id" || true
+          done
 
       # Manual dispatch with explicit baseline run ID
       - name: Download baseline results (manual, explicit run ID)
@@ -562,29 +586,38 @@ jobs:
         if: >
           github.event_name == 'workflow_dispatch' &&
           inputs.baseline_run_id == ''
-        uses: dawidd6/action-download-artifact@v6
-        continue-on-error: true
-        with:
-          workflow: system-tests.yml
-          branch: main
-          name_is_regexp: true
-          name: "test-results-.*"
-          path: baseline-results/
-          if_no_artifact_found: warn
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          mkdir -p baseline-results
+          gh api --method GET \
+            "repos/${{ github.repository }}/actions/workflows/system-tests.yml/runs" \
+            -f branch=main -f status=success -f per_page=20 \
+            --jq '.workflow_runs[].id' |
+          while read -r run_id; do
+            gh run download "$run_id" --repo "${{ github.repository }}" \
+              --pattern "test-results-*" --dir "baseline-results/$run_id" || true
+          done
 
       - name: Locate result directories
         id: dirs
-        # Find the dir holding results.xml. Nesting depth differs by downloader:
-        # actions/download-artifact@v4 (single name) extracts straight into the
-        # path, while dawidd6/action-download-artifact@v6 with name_is_regexp
-        # wraps each artifact in a subdir named after it. `find` handles both.
+        # Find the current dir, then choose a baseline from the recent-run
+        # candidate tree by completed campaign fingerprint.
         run: |
           CURRENT_XML=$(find current-results/ -name results.xml 2>/dev/null | sort -r | head -1)
           [ -n "$CURRENT_XML" ] && echo "current=$(dirname "$CURRENT_XML")" >> "$GITHUB_OUTPUT"
 
-          BASELINE_XML=$(find baseline-results/ -name results.xml 2>/dev/null | sort -r | head -1)
-          if [ -n "$BASELINE_XML" ]; then
-            echo "baseline=$(dirname "$BASELINE_XML")" >> "$GITHUB_OUTPUT"
+          if [ -n "$CURRENT_XML" ] && [ -d baseline-results ]; then
+            CURRENT_DIR="$(dirname "$CURRENT_XML")"
+            BASELINE=$(PYTHONPATH=tests python3 - "$CURRENT_DIR" <<'PYEOF'
+          import sys
+          from pathlib import Path
+          from harness.baseline import select_baseline_path
+          selected = select_baseline_path(Path(sys.argv[1]), Path("baseline-results"))
+          print(selected or "")
+          PYEOF
+          )
+            echo "baseline=$BASELINE" >> "$GITHUB_OUTPUT"
           else
             echo "baseline=" >> "$GITHUB_OUTPUT"
           fi
@@ -604,8 +637,8 @@ jobs:
 
           Pass-rate and regression tables are suppressed because no completed test campaign is available.
           EOF
-            echo "parser_exit=0" >> "$GITHUB_OUTPUT"
-            exit 0
+            echo "parser_exit=2" >> "$GITHUB_OUTPUT"
+            exit 2
           fi
 
           set +e
@@ -655,17 +688,13 @@ jobs:
             echo "_No metrics report generated._" >> "$GITHUB_STEP_SUMMARY"
           fi
 
-      - name: Fail on regression
+      - name: Fail on report integrity error
         if: steps.report.outcome == 'failure'
-        # parse_metrics.py writes report.md before exiting 1 on a regression;
-        # an uncaught crash also exits 1 but never writes the report. Require
-        # both so a parser crash cannot masquerade as a metric regression.
+        # Numeric metric deltas are advisory (parse_metrics.py exits 0 on
+        # them); this step fires only on parser/integrity failures (exit 2)
+        # or an uncaught crash, and never labels either a metric regression.
         run: |
-          if [ "${{ steps.report.outputs.parser_exit }}" = "1" ] && [ -f report.md ]; then
-            echo "::error::Metric regression detected — see the report above for details."
-          else
-            echo "::error::Metrics report generation failed — see the report step log."
-          fi
+          echo "::error::Metrics report generation failed — see the report step log."
           exit 1
 
       - name: Finalize check on PR head

.github/workflows/unit-tests.yml

@@ -55,6 +55,8 @@ jobs:
       - name: Run unit tests
         env:
           AIRSTACK_ROOT: ${{ github.workspace }}
+          AIRSTACK_TESTED_SHA: ${{ github.sha }}
+          AIRSTACK_PR_NUMBER: ${{ github.event.pull_request.number }}
         run: pytest tests/ -m unit
 
       - name: Upload unit-test results

AGENTS.md

@@ -176,7 +176,7 @@ airstack install        # Install Docker and dependencies
 
 # Container management
 airstack up [service]    # Start services (robot, isaac-sim, gcs)
-airstack up --sim isaac|airsim --robots N   # Intent flags: derive profiles/URDF/sim script (add --headless, --play/--no-play, --no-autolaunch, --wait, --dry-run)
+airstack up --sim isaac|airsim --robots N   # Intent flags: derive profiles/URDF/sim script (add --headless, --play/--no-play, --no-autolaunch, --wait, --dry-run, --config-only)
 airstack up --stack <name>[:<entry>] --sim isaac   # Stack launch (RFC #379): stacks/<name>/launch/<entry>.launch.xml — the ONLY dispatch (no --stack = full_default; legacy AUTONOMY_ROLE was removed); see docs/development/stacks.md
 airstack up --fleet <name> --sim isaac      # Fleet launch (RFC #380): config/fleets/<name>.yaml drives identity/placement/spawns; see docs/development/fleets.md
 airstack fleet list|generate <name>         # Fleet files: table / per-robot compose for heterogeneous fleets
@@ -271,7 +271,7 @@ Pytest-based system tests live under [`tests/system/`](tests/system/). They brin
 | [`tests/system/test_fixed_trajectory.py`](tests/system/test_fixed_trajectory.py) | `autonomy` | 4-phase flight chain (PX4 ready → takeoff → execute Circle/Figure8/Racetrack/Line trajectory → land) per (sim, num_robots, iter, trajectory_type); records cross-track error and path RMSE | Docker, GPU, sim license |
 | [`tests/system/test_waypoint_flight.py`](tests/system/test_waypoint_flight.py) | `waypoint_flight` | 4-phase flight chain (PX4 ready → takeoff → NavigateTask waypoint route → land) per (sim, num_robots, iter); pass/fail judged on the odometry track by the standalone [`tests/waypoint_checker.py`](tests/waypoint_checker.py) (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`) | Docker, GPU, sim license |
 
-The pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures live in [`tests/conftest.py`](tests/conftest.py); the shared helpers are split by concern into the [`tests/harness/`](tests/harness/) package (`session`, `discovery`, `commands`, `containers`, `metrics` (with `MetricsRecorder`), `run_meta`, `test_ids`, `sim`, `collection`) and re-exported through `conftest`, so `from conftest import <name>` still resolves. Each run produces a timestamped directory under `tests/results/<timestamp>/` with `summary.txt`, `results.xml`, `run_meta.json`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) compares only matching, complete simulation campaigns and exits 1 on a genuine metric regression.
+The pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures live in [`tests/conftest.py`](tests/conftest.py); the shared helpers are split by concern into the [`tests/harness/`](tests/harness/) package (`session`, `discovery`, `commands`, `containers`, `metrics` (with `MetricsRecorder`), `run_meta`, `test_ids`, `sim`, `collection`) and re-exported through `conftest`, so `from conftest import <name>` still resolves. Each run produces a timestamped directory under `tests/results/<timestamp>/` with `summary.txt`, `results.xml`, `run_meta.json` (schema v2: completion state, `failure_class`, campaign fingerprint over tests + behavior-changing CLI config), `metrics.json`, and — on bring-up/readiness failures — a bounded `diagnostics/` bundle (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) compares only fingerprint-identical, complete simulation campaigns; numeric metric deltas are **advisory** (exit 0) and never fail CI — only real test failures, infrastructure/prerequisite failures, and report-integrity errors (exit 2) do.
 
 **Run via the CLI** (containerized runner — no local Python needed):