Skip to content

test: 242 :verisim_data tests never run anywhere and fail 129/242 — documented as passing #692

Description

@hyperpolymath

Symptom

mix test reports a clean suite. It is scoped to 83% of it.

$ mix test                      → 1433 tests, 0 failures, 242 excluded
$ mix test --only verisim_data  → 1433 tests, 129 failures, 1191 excluded

242 tests (16.9% of the suite) are excluded from every run, are never run anywhere, and fail 129/242 when run.

The comment asserting otherwise

test/test_helper.exs excludes :verisim_data unconditionally, under this comment:

"These tests pass in CI (where verisim-data is available) but are excluded from local development runs"

Both halves are false:

  • They never run in CI. --include verisim_data appears nowhere in .github/workflows/ or the Justfile. The only invocations are just test-elixir (mix test) and tests.yml (mix test --include integration).
  • They do not pass. 129 failures, concentrated in VCL.FileExecutorTest (27), RecipeAdditionalRecipesTest (17), RecipeNewRecipesTest (11), E2E.PipelineTest (11), RecipeMatcherTest (10), PatternRegistryTest (8), ZigFFI.SmokeTest (6), VerisimConnectorTest (6).

Sample failure: test/vcl_query_test.exs:871: assert health.repos_scanned >= 3.

Why this matters more than the raw number

Every quality claim this repo makes — including its own release gate (PLAYBOOK.a2ml: "hard-pass-gate = mix test 0 failures") — inherits the exclusion silently. A reader sees "0 failures" and reasonably concludes the suite passes. The dark set is not merely untested; it is known red and documented as green.

Options

  1. Provision verisim-data in CI and run them. Highest value: these cover the VCL executor, the recipe corpus and the E2E pipeline — arguably the most important paths in the project.
  2. Retag honestly. If the fixture data cannot be provisioned, they are not "excluded from local runs", they are unrunnable — say so, and fix or delete the 129 failures.

Either way the comment in test/test_helper.exs must stop asserting something untrue.

Evidence: docs/DEBT-REGISTER.md (T-1).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcx/L2Scoped — one repo, needs code reading + tests, no new designvfy/testCompletion provable by a test

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions