Summary
A test that stores a value in step N (vision extraction → {{var}}) and asserts against it in step N+1 passes every authoring run but deterministically fails on replay — the step-N+1 checkpoint compares against a stale/wrong value instead of what step N's replay just extracted. The 0.6.4 release notes say "Step-produced variables cross testmd step boundaries — variables set in one step are available to later steps in the same run," but replay behaves as if the propagation doesn't happen (or the extraction re-runs against the wrong target).
Repro matrix (kane-cli 0.6.4, 2026-07-22)
Two tests in our suite have this shape (UI figures extracted in step 1/2, reconciled against API JSON in the next step). Both were freshly re-authored green the same hour. Results:
| Mode |
Result |
Authoring (testmd run, prose changed) |
✅ pass — repeatedly |
testrun run replay (with --retry) |
❌ fail at the cross-step assertion |
testrun run replay (no --retry, fresh recordings) |
❌ same |
Standalone testmd run replay |
❌ same (2/3 steps, replay_decisions:3) |
The 7 sibling tests whose assertions stay within the extraction step replay green every time — the failure isolates cleanly to cross-step variable use.
Evidence (from the sealed evidence pack, execution 0948d0d4-4a14-4be1-b855-ea5feea7ce27)
failure.yaml: "First-payment reconciliation assertion uses stale stored value" — the replay compared the API schedule against 870.91 (the pre-advance baseline payment) when the stored receipt value at that point in the flow is ~921.34 (post-advance). The stale number is exactly the value from a different state, i.e., the variable wasn't repopulated by the replayed extraction.- Both failures typed
replay_failure, both at the first assertion referencing a step-produced {{var}}.
Secondary defect: the RCA blames the product
For the second test, the automated triage concluded application_issue/ui_data_defect (severity: major) — "the app is likely showing two different numbers for the same loan." The app is not: the same reconciliation passes under authoring and under TMS cloud execution minutes earlier, and the field is computed server-side from the same API the check reads. A replay-runtime artifact being triaged as a major product defect is a trust problem — please make the RCA consider "stale replay variable" before accusing the AUT.
Ask
- On replay, re-populate step-produced variables from the replayed extraction (or carry the recorded resolution forward consistently) before evaluating downstream checkpoints.
- Teach the replay-failure triage to distinguish "checkpoint input variable was stale/unset" from both automation misconfiguration and application defects.
Env: kane-cli 0.6.4 · Windows 11 · Node 24 · account toddl · AUT: public demo app on Render
Summary
A test that stores a value in step N (vision extraction →
{{var}}) and asserts against it in step N+1 passes every authoring run but deterministically fails on replay — the step-N+1 checkpoint compares against a stale/wrong value instead of what step N's replay just extracted. The 0.6.4 release notes say "Step-produced variables crosstestmdstep boundaries — variables set in one step are available to later steps in the same run," but replay behaves as if the propagation doesn't happen (or the extraction re-runs against the wrong target).Repro matrix (kane-cli 0.6.4, 2026-07-22)
Two tests in our suite have this shape (UI figures extracted in step 1/2, reconciled against API JSON in the next step). Both were freshly re-authored green the same hour. Results:
testmd run, prose changed)testrun runreplay (with--retry)testrun runreplay (no--retry, fresh recordings)testmd runreplayreplay_decisions:3)The 7 sibling tests whose assertions stay within the extraction step replay green every time — the failure isolates cleanly to cross-step variable use.
Evidence (from the sealed evidence pack, execution
0948d0d4-4a14-4be1-b855-ea5feea7ce27)failure.yaml: "First-payment reconciliation assertion uses stale stored value" — the replay compared the API schedule against870.91(the pre-advance baseline payment) when the stored receipt value at that point in the flow is~921.34(post-advance). The stale number is exactly the value from a different state, i.e., the variable wasn't repopulated by the replayed extraction.replay_failure, both at the first assertion referencing a step-produced{{var}}.Secondary defect: the RCA blames the product
For the second test, the automated triage concluded
application_issue/ui_data_defect (severity: major)— "the app is likely showing two different numbers for the same loan." The app is not: the same reconciliation passes under authoring and under TMS cloud execution minutes earlier, and the field is computed server-side from the same API the check reads. A replay-runtime artifact being triaged as a major product defect is a trust problem — please make the RCA consider "stale replay variable" before accusing the AUT.Ask
Env: kane-cli 0.6.4 · Windows 11 · Node 24 · account toddl · AUT: public demo app on Render