Skip to content
Merged
19 changes: 16 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -590,11 +590,24 @@ jobs:
# every OS, so $VAR resolves the same on the Windows legs (Git Bash).
FAULT_TIMEOUT: ${{ matrix.fault_timeout }}
PYTEST_TIMEOUT: ${{ matrix.pytest_timeout }}
run: pytest -q -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT"
# `--ignore-glob` subtracts the web console package, which runs as its OWN step below. BACKLOG
# #1027 added that package to the root `testpaths` so a bare local `pytest` stops silently
# excluding it; without this subtraction the SAME 356 tests would then run TWICE on every leg.
# Measured 2026-08-08 on this branch: bare collection 11,956, console-only 356, with this flag
# 11,600.
# NOT `--ignore=<path>` -- measured, it does NOT prune a directory that `testpaths` names as a
# collection root, and still collected all 11,956. The obvious spelling is the one that fails.
# NOT `pytest tests` either: hardcoding the engine path would make CI silently miss any future
# third entry in `testpaths`, which is the exact drift #1027 exists to prevent. Subtracting FROM
# `testpaths` keeps it the single source of truth for what the suite is.
# `tests/test_ci_engine_step_excludes_webconsole.py` pins all three of those decisions.
run: pytest -q --ignore-glob='*messagefoundry-webconsole*' -o faulthandler_timeout="$FAULT_TIMEOUT" --timeout="$PYTEST_TIMEOUT"

# The web console's OWN suite (Option B, ADR 0065): the moved /ui tests live in the package's
# tests/ (the engine `pytest` above no longer collects them — engine testpaths = ["tests"]), so
# run them as a SECOND step on the SAME leg. The package was installed editable in the install
# tests/. They ARE in the root `testpaths` since BACKLOG #1027, so a bare local `pytest` collects
# them -- the engine step above subtracts them explicitly via `--ignore-glob` rather than relying
# on `testpaths` to exclude them, which it no longer does. They still run as a SECOND step on the
# SAME leg, under the package's OWN pytest configuration. The package was installed editable in the install
# step above (`-e packaging/messagefoundry-webconsole`), so both suites exercise the same
# engine build. Its pyproject sets asyncio_mode="auto" + session loop scopes; the timeout flags
# mirror the engine step (a hung ASGI test fails fast + named, not at the silent job cap).
Expand Down
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,7 @@ diverge enough to warrant it; keep this root file general.

```
# tests (PySide6 harness/Qt tests need the offscreen platform)
# testpaths now also collects packaging/messagefoundry-webconsole/tests, so this covers the web console suite too.
QT_QPA_PLATFORM=offscreen pytest -q # PowerShell: $env:QT_QPA_PLATFORM="offscreen"; pytest -q

# format / lint / types
Expand Down
2 changes: 1 addition & 1 deletion docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4995,7 +4995,7 @@ The comment immediately above says *"Scope is deliberately the posture the requi

## 1027. The documented `pytest` command silently excludes the webconsole package, so a local green is not evidence about ~344 tests

> 🔢 **Filed 2026-08-05not started.** Value **5/10** · Difficulty **3/10** · _fill-in_. `testpaths = ["tests"]` means the command CLAUDE.md documents as the verification gate never collects `packaging/messagefoundry-webconsole/tests`. A **failing** webconsole test sat on `main` through a full day of lanes because every quartet used the documented single path.
> **SHIPPED 2026-08-06the root `testpaths` now also collects `packaging/messagefoundry-webconsole/tests`, so a bare `pytest -q` from the repo root stops silently excluding the web console suite; the one webauthn-extra-dependent console test that lacked a guard (`test_webauthn_rp_fail_closed_legible`) now skips-with-reason when the optional `[webauthn]` extra is absent, so an extra-less local venv stays green.** Value **5/10** · Difficulty **3/10** · _fill-in_. Local developer-signal fix only — CI already covered the console via its dedicated `Web console tests (pytest)` step; the gap was that the documented local gate collected less than it appeared to.

**Cluster:** Testing / verification integrity. **Priority:** P3. **Verdict:** build (small). **Severity:** no product effect; the defect is that the project's own verification instruction produces a green that is not evidence about roughly 344 tests, and CLAUDE.md §5 states a task is not done until it passes.

Expand Down
1 change: 1 addition & 0 deletions packaging/messagefoundry-webconsole/tests/test_webui.py
Original file line number Diff line number Diff line change
Expand Up @@ -3674,6 +3674,7 @@ async def test_webauthn_cross_site_posts_rejected(engine: Engine) -> None:
async def test_webauthn_rp_fail_closed_legible(engine: Engine) -> None:
# AC-7: public_origin unset + request-derivation disallowed (the declared-proxy topology) —
# ceremonies fail closed with the shared notice on every surface, never a redirect loop.
pytest.importorskip("webauthn")
service = await _service(engine)
await _add(service, "boss", Role.ADMINISTRATOR)
transport = httpx.ASGITransport(
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ asyncio_mode = "auto"
# the fixture key needs >=0.24); the dev floor below is >=0.26 to match, and the lock pins 1.4.0.
asyncio_default_test_loop_scope = "session"
asyncio_default_fixture_loop_scope = "session"
testpaths = ["tests"]
testpaths = ["tests", "packaging/messagefoundry-webconsole/tests"]
# Per-test watchdog: a single hung await/socket is killed at 60s with a full thread-stack dump naming
# the culprit, instead of burning the CI job's wall-clock. `thread` method works cross-platform (no
# SIGALRM) and fires even when the main thread is blocked in a C call. Individual tests run well under
Expand Down
75 changes: 75 additions & 0 deletions tests/test_ci_engine_step_excludes_webconsole.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 MessageFoundry Organization and contributors
"""CI's engine pytest step must subtract the web console package that ``testpaths`` now includes.

BACKLOG #1027 added ``packaging/messagefoundry-webconsole/tests`` to the root ``testpaths`` so a
bare local ``pytest`` stops silently excluding roughly 350 tests. That fix has a CI-side
consequence the original change did not carry: ``ci.yml`` runs a bare ``pytest`` for the engine
step AND a second explicit step for the console package, so once ``testpaths`` includes the
console the SAME tests run TWICE on every leg.

Measured 2026-08-08: bare collection 11,956; console-only 356; with the subtraction 11,600.

**The obvious spelling does not work, which is why this guard pins the working one.**
``--ignore=packaging/messagefoundry-webconsole/tests`` was measured and does NOT prune a directory
that ``testpaths`` names as a collection root -- it still collected all 11,956. Only the glob form
subtracts. A future edit "simplifying" the flag back to ``--ignore`` would restore the double-run
silently, with every check still green, which is the same shape as the defect #1027 fixed.

``pytest tests`` is also rejected, and deliberately: hardcoding the engine path would make CI miss
any future third entry in ``testpaths``. Subtracting FROM ``testpaths`` keeps it the one source of
truth for what the suite is.
"""

from __future__ import annotations

import re
import tomllib
from pathlib import Path

import pytest

_ROOT = Path(__file__).resolve().parents[1]
_CI = _ROOT / ".github" / "workflows" / "ci.yml"
_PYPROJECT = _ROOT / "pyproject.toml"
_CONSOLE = "packaging/messagefoundry-webconsole/tests"


def _engine_step_run_line() -> str:
"""The `run:` line of the engine `Tests (pytest)` step."""
text = _CI.read_text(encoding="utf-8")
# The engine step is the bare `pytest -q ...` invocation; the console step names its path.
for line in text.splitlines():
stripped = line.strip()
if stripped.startswith("run: pytest -q"):
return stripped
pytest.fail(f"no engine `run: pytest -q` line found in {_CI}")


def test_console_is_in_testpaths() -> None:
"""Precondition: this guard is only meaningful while #1027's change is in place."""
cfg = tomllib.loads(_PYPROJECT.read_text(encoding="utf-8"))
testpaths = cfg["tool"]["pytest"]["ini_options"]["testpaths"]
assert _CONSOLE in testpaths, (
f"expected {_CONSOLE!r} in testpaths (BACKLOG #1027); got {testpaths!r}. "
"If that was reverted deliberately, this guard and the ci.yml subtraction go with it."
)


def test_engine_step_subtracts_the_console_package() -> None:
"""Without this, the console's ~356 tests run twice per leg."""
run = _engine_step_run_line()
assert "--ignore-glob" in run and "messagefoundry-webconsole" in run, (
"ci.yml's engine `Tests (pytest)` step must subtract the web console package, which "
f"`testpaths` now includes and which runs as its own step. Found:\n {run}"
)


def test_engine_step_does_not_use_plain_ignore() -> None:
"""`--ignore=<path>` was measured NOT to prune a `testpaths` collection root."""
run = _engine_step_run_line()
plain = re.search(r"--ignore(?!-glob)[= ]", run)
assert plain is None, (
"`--ignore=<path>` does not prune a directory named by `testpaths` -- measured 2026-08-08, "
"it still collected all 11,956 tests. Use `--ignore-glob`. Found:\n " + run
)
35 changes: 35 additions & 0 deletions tests/test_testpaths_webconsole_coverage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright (C) 2026 MessageFoundry Organization and contributors
"""The default pytest gate must collect the web console package, not silently exclude it.

BACKLOG #1027: `testpaths = ["tests"]` made a bare `pytest -q` from the repo root skip the
~344 tests under ``packaging/messagefoundry-webconsole/tests`` while the green summary line
looked complete. This guard reads the root ``pyproject.toml`` and fails if the console tree is
ever dropped from ``testpaths`` again — it lives under ``tests/`` on purpose, so the very
narrowing it guards against can never exclude the guard itself.
"""

from __future__ import annotations

import tomllib
from pathlib import Path

_REPO_ROOT = Path(__file__).resolve().parents[1]
_WEBCONSOLE_TESTS = "packaging/messagefoundry-webconsole/tests"


def test_default_testpaths_collect_the_webconsole_package() -> None:
pyproject = _REPO_ROOT / "pyproject.toml"
data = tomllib.loads(pyproject.read_text(encoding="utf-8"))
testpaths = data["tool"]["pytest"]["ini_options"]["testpaths"]

assert testpaths[0] == "tests", (
f"the engine suite must stay first in testpaths, got {testpaths!r}"
)
assert _WEBCONSOLE_TESTS in testpaths, (
f"the web console package tests must be in testpaths so a default pytest run "
f"collects them (BACKLOG #1027), got {testpaths!r}"
)
assert (_REPO_ROOT / _WEBCONSOLE_TESTS).is_dir(), (
f"testpaths names {_WEBCONSOLE_TESTS!r} but that directory does not exist"
)
Loading