Multi-checkpoint step (multiple ANALYZE + combining ASSERT) authors green but fails deterministically on replay with null assert values

[komal-lt]

Summary

A testmd step that emits multiple ANALYZE checkpoints plus a combining ASSERT (i.e. the step's prose asks the agent to verify more than one condition) authors green but fails deterministically on replay. On the replay run, every ANALYZE sub-checkpoint passes, but the final assert step fails with all of its checkpoint values (expected, extracted_value, actual) equal to null. The page is fine — nothing actually regressed.

This breaks the core value of saved tests: "author once, replay cheap and deterministic." Any multi-condition step is a latent replay failure.

Environment

• @testmuai/kane-cli 0.4.1 (npm global, macOS / node v20.18.0)
• mode: testing, prod, profile-authenticated browser

Repro

A single step whose prose asks for two things + a fail condition, e.g.:

## Open the app and verify the signed-in identity
Open {{TM_BASE_URL}}/projects. Wait for the projects view to load.
Verify the top header shows a signed-in user's profile avatar / account menu,
AND that no email/password login form is present. The test MUST FAIL if no
profile avatar is present, or if a login form is shown instead.

kane-cli authors this as: analyze (avatar present) → analyze (login form not present) → combining assert.

1. kane-cli testmd run <file> --author → PASSES (author_decisions:1).
2. kane-cli testmd run <file> → FAILS (replay_decisions:1), every run, deterministically.

Expected

Replay reproduces the authored result. If the sub-analyze checkpoints pass, the combining assert should pass.

Actual

run_summary.json (replay run):

final_status: failed | reason: assertion_failed: @ step 5
 step 1 navigate => passed
 step 2 wait     => passed
 step 3 analyze  => passed   (avatar present = true)
 step 4 analyze  => passed   (login form not shown = true)
 step 5 assert   => failed   <-- combining assert

The step-5 assert's checkpoints carry no evaluation data on replay:

[
  { "operator": "visual", "expected": null, "extracted_value": null, "actual": null },
  { "operator": "visual", "expected": null, "extracted_value": null, "actual": null }
]

So the assert isn't re-evaluated against the (passing) analyze results on replay — it defaults to failed.

Workaround (verified)

Collapse each step to a single positive assertion (one ANALYZE/check, no combining ASSERT over multiple conditions). Reworded that way, the same test replays green consistently (verified 2×). Steps that were already single-checkpoint replay fine.

Impact

• Multi-condition steps silently pass in authoring/CI-first-run and then fail on every cheap replay, so a green author run is not a reliable signal that the saved test will replay.
• Forces authors to hand-split every assertion into its own step to get deterministic replays.

[komal-lt]

Impact follow-up — this is a bigger deal than a null value; it's a structural limit on test expressiveness.

Because the combining assert nulls on replay, the only reliable authoring pattern is one assertion per step. That has a real consequence for anyone building a suite on testmd replay:

• A natural "verify X and Y and Z" check (e.g. "the dashboard shows the Summary tab, the Code tab, and the Runs tab") authors green but flakes on replay, so it must be decomposed into N single-landmark steps — or abandoned.
• This effectively caps a replay-based suite at breadth smoke ("does this view load / is this one landmark present"). Rich multi-assertion journeys — the kind that catch real regressions — can't be expressed as natural steps without losing deterministic replay.

So the severity isn't "a checkpoint shows null" — it's "you can't author depth-y regression tests against the replay model." Fixing this (re-evaluate the combining assert against the already-passing sub-ANALYZE results on replay) directly unlocks multi-condition steps and complex-journey tests.

Concrete example from a suite we built: a test-case dashboard step that verified Summary + Code + Runs + Issues + Version-History tabs authored green and failed on every replay; reducing it to a single "Summary tab visible" check made it deterministic — at the cost of no longer asserting the other four tabs.

[komal-lt]

Adding CI evidence that sharpens the root cause here.

Replay re-evaluates the analyzer — it does not just play back recorded actions. On testmd run replays (no --author), the verify step's checkpoint re-invokes the model against the live page each time. So a visual checkpoint is non-deterministic on replay:

• A test whose verify step asserts a visual landmark ("a list/grid of projects is visible" / "the Create Project button is visible") flakes ~1 in 3 replays, with no app change between runs.
• Concretely: across 3 plain replays of the same committed cache we saw passed / failed / passed for the identical assertion.

Re-authoring does not stabilize it. Re-authoring the test with a single tight landmark still produced a multi-analyzer verify (an explicit analyze + a terminal __cp_final analyze), and it still flaked. Switching the assertion to a URL check was worse — the re-recorded optional login step itself then authenticated only intermittently on the fresh isolated browser.

Why it matters for CI: combined with the false-green masking (filed separately), a non-deterministic checkpoint means the only safe posture is an advisory gate. A deterministic replay assertion mode — checkpoint resolved from the DOM/text/URL rather than re-running the visual model — would make testmd replays usable as a blocking gate.

Env: kane-cli 0.4.1, testmd run --agent --headless.