Budget the trade CLI tests for the engine import their CLI pays - #784
Conversation
`test_cli_fails_when_margins_missing_from_window` timed out on main (run 32910539480) after passing on #766's final run, which tested a merge ref with the same content — nothing landed between that run and the merge. Same code, same group, 380 passed there against 379 plus one timeout here, so the break is marginal timing rather than logic. The margin is thin because the CLI pays for machinery it never uses. Importing `us_runtime.us_trade.*` executes `us_runtime/__init__`, which pulls in spine_agreement and through it the whole policyengine-us system. Measured on the failing test's own fixture — a CLI that reads a two-row parquet and exits 1 — that is 35.9 s with the engine installed against 5.4 s without, and `build_us_import_entry_margins.py` carries the identical 33.0 s chain. The import is parameter-tree file I/O, so it stretches with runner disk contention; 300 s left under 10x headroom on a call whose useful work is milliseconds. Name the budget and raise it to 900 s at the six full-CLI spawn sites across both trade test files, with the measurements recorded where the number lives. The lightweight `-c` spawns keep their 60 s: they do not import the chain. This buys headroom; it does not remove the cost. The cost disappears by deferring the engine import in `us_runtime/__init__`, which would also cut ~30 s from each of these spawns — US source with unclear load-bearing behavior, so it is filed for its owners rather than attempted in a hotfix. Verified: both files pass against the change, 91 passed in 10m15s in an engine environment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Non-determinism confirmed — main is green againI re-ran the failed job on main with no code change. It passed:
Run 32910539480 is now Note the re-run took 1415 s against 1263 s for the failing run, a 12% swing on the same commit. That is the runner variance the 300 s budget was sitting inside: when the group's work stretches, so does the 33 s engine import each CLI spawn pays, and the spawn nearest the edge is the one that tips. This also rules out the alternative explanations I was holding open — it is not something that landed in #743 or #747 (both merged before #766's final green run), and not a deterministic break from the regrouping, since the identical grouping passes on re-run. One consequence worth stating plainly: the same fragility exists on |
Summary
Fixes the red main run after #766 merged (run 32910539480). One test failed —
test_us_trade_entries_cli.py::test_cli_fails_when_margins_missing_from_window— withsubprocess.TimeoutExpiredafter 300 s.The test spawns the CLI over a two-row parquet and expects an immediate exit 1. It passed on #766's final run, which tested a merge ref with the same content: nothing landed between that run (21:36) and the merge (23:23). Same code, same group,
380 passedthere against379 passed, 1 failedhere, and near-identical job wall clock (1281 s vs 1263 s).Why the margin was thin
The CLI pays for machinery the trade code never touches.
tools/build_us_import_entries.pyimportsus_runtime.us_trade.cbp_entry_stats, which executesus_runtime/__init__, which pulls inspine_agreementand through it the wholepolicyengine_ussystem.Measured on the failing test's own fixture:
engine-uslane)-X importtimeattributes it:us_runtime32.9 s →spine_agreement30.9 s →policyengine_us8.7 s.build_us_import_entry_margins.py(used bytest_us_trade_imdb_bulk.py) carries the identical 33.0 s chain — checked rather than assumed, which is why its four spawn sites are in this change too.That import is parameter-tree file I/O, so it stretches with runner disk contention. A 300 s budget left under 10× headroom on a call whose useful work is milliseconds.
The change
CLI_TIMEOUT_SECONDS = 900at the six full-CLI spawn sites across both trade test files, with the measurements recorded next to the number so the next reader knows what the budget is for. The lightweight-cspawns keep their 60 s — they do not import the chain.What this does not do
It buys headroom; it does not remove the cost. The cost disappears by deferring the engine import in
us_runtime/__init__, which would cut ~30 s from every one of these spawns and speed up any other tool importing aus_runtimesubmodule. That is US source with unclear load-bearing behavior, so it belongs to its owners as a follow-up rather than a hotfix — flagging it here rather than filing separately, per the current preference not to open US-side issues.Verification
ruff checkclean on both files.🤖 Generated with Claude Code