Symptom
mix test reports a clean suite. It is scoped to 83% of it.
$ mix test → 1433 tests, 0 failures, 242 excluded
$ mix test --only verisim_data → 1433 tests, 129 failures, 1191 excluded
242 tests (16.9% of the suite) are excluded from every run, are never run anywhere, and fail 129/242 when run.
The comment asserting otherwise
test/test_helper.exs excludes :verisim_data unconditionally, under this comment:
"These tests pass in CI (where verisim-data is available) but are excluded from local development runs"
Both halves are false:
- They never run in CI.
--include verisim_data appears nowhere in .github/workflows/ or the Justfile. The only invocations are just test-elixir (mix test) and tests.yml (mix test --include integration).
- They do not pass. 129 failures, concentrated in
VCL.FileExecutorTest (27), RecipeAdditionalRecipesTest (17), RecipeNewRecipesTest (11), E2E.PipelineTest (11), RecipeMatcherTest (10), PatternRegistryTest (8), ZigFFI.SmokeTest (6), VerisimConnectorTest (6).
Sample failure: test/vcl_query_test.exs:871: assert health.repos_scanned >= 3.
Why this matters more than the raw number
Every quality claim this repo makes — including its own release gate (PLAYBOOK.a2ml: "hard-pass-gate = mix test 0 failures") — inherits the exclusion silently. A reader sees "0 failures" and reasonably concludes the suite passes. The dark set is not merely untested; it is known red and documented as green.
Options
- Provision verisim-data in CI and run them. Highest value: these cover the VCL executor, the recipe corpus and the E2E pipeline — arguably the most important paths in the project.
- Retag honestly. If the fixture data cannot be provisioned, they are not "excluded from local runs", they are unrunnable — say so, and fix or delete the 129 failures.
Either way the comment in test/test_helper.exs must stop asserting something untrue.
Evidence: docs/DEBT-REGISTER.md (T-1).
Symptom
mix testreports a clean suite. It is scoped to 83% of it.242 tests (16.9% of the suite) are excluded from every run, are never run anywhere, and fail 129/242 when run.
The comment asserting otherwise
test/test_helper.exsexcludes:verisim_dataunconditionally, under this comment:Both halves are false:
--include verisim_dataappears nowhere in.github/workflows/or theJustfile. The only invocations arejust test-elixir(mix test) andtests.yml(mix test --include integration).VCL.FileExecutorTest(27),RecipeAdditionalRecipesTest(17),RecipeNewRecipesTest(11),E2E.PipelineTest(11),RecipeMatcherTest(10),PatternRegistryTest(8),ZigFFI.SmokeTest(6),VerisimConnectorTest(6).Sample failure:
test/vcl_query_test.exs:871: assert health.repos_scanned >= 3.Why this matters more than the raw number
Every quality claim this repo makes — including its own release gate (
PLAYBOOK.a2ml: "hard-pass-gate = mix test 0 failures") — inherits the exclusion silently. A reader sees "0 failures" and reasonably concludes the suite passes. The dark set is not merely untested; it is known red and documented as green.Options
Either way the comment in
test/test_helper.exsmust stop asserting something untrue.Evidence:
docs/DEBT-REGISTER.md(T-1).