Skip to content

test(turn-driver): pin the host-failure retry ceiling and unmapped policies - #4951

Merged
huangruiteng merged 2 commits into
loopx-project:mainfrom
yuedai-pbc:codex/host-failure-retry-ceiling
Sep 23, 2026
Merged

huangruiteng merged 2 commits into
loopx-project:mainfrom
yuedai-pbc:codex/host-failure-retry-ceiling

Conversation

@yuedai-pbc

Copy link
Copy Markdown
Contributor

Goal And Delivered Outcome

  • Goal/source and gap: build_host_failure_record() publishes the bounded retry hint that
    the outer Turn controller reads to decide when to wake the same Turn again. The suite
    reached at most attempt=3, and only for three of the five retryable kinds, so the
    min(..., 300) ceiling in loopx/control_plane/turn_driver/host_failure.py:84-87 never
    executed — the largest value any existing case could produce is 60 * 2**2 = 240.
    executor_timeout (the only two-attempt policy) and transport_lost were absent from
    coverage entirely, and the attempt type guard, the rebuilt-record boundary and both
    retryability-mismatch exits had no assertion anywhere in tests/.
  • Observable before → after, with the validation row that proves it: before, dropping the
    min() call, widening the ceiling to 3000, turning executor_timeout into (3, 5),
    removing the isinstance(attempt, bool) exclusion, returning the caller's mapping from
    normalize_host_failure_record(), shifting the exponent, or deleting either
    retryability exit all passed the suite. After, each of those eight edits fails at least
    one test — see the regression_parity row.
  • Issue/task and intended base: no open issue tracks this gap; the request and the
    regression are self-contained. Base is current main (eb16c5443). No production file
    is modified.

Scope And Continuation

  • Completed scope and remaining work: the retry hint is now pinned for the capped ladder,
    both previously unmapped policies, the producer path the executor journal actually uses
    (record_host_failure() at loopx/control_plane/turn_driver/executor.py:793), the empty
    projection, the attempt-type guard and the two retryability-mismatch exits. Not claimed
    here: the mirrored TypeScript journal builder and the outer controller's wake timing —
    both consume these hints rather than compute them, so their contracts are unchanged by
    this PR and were left alone deliberately.
  • Slice boundary / successor: N/A — complete within this scope; a successor would need a
    new unpinned caller-visible behavior rather than a larger version of this file.

Validation

  • Tested revision: 187df5a6c
  • Run state: finished
  • Input classes: synthetic
Check kind Result Public-safe evidence / limitation
unit passed python -m pytest -q tests/test_loopx_turn_host_failure.py → 20 passed: the 7 pre-existing cases plus 6 new functions (one parametrized over 5 rejected attempt shapes). Covers the ceiling, both missing policies, the journal producer, the empty projection and the three validation guards.
unit passed python -m pytest -q tests/test_loopx_turn_host_failure.py tests/test_loopx_turn_executor.py tests/test_loop_turn_loop_controller.py tests/test_loop_turn_controller_contract.py tests/test_loopx_turn_journal_inspection.py tests/test_dsh_goal_mode.py tests/capabilities/test_codex_subagent_host_capacity.py → 437 passed / 0 failed. These are every Python module that classifies a host failure into one of these kinds or reads the retry hint back.
static passed python -m ruff check clean; ruff format --check clean on the touched file; python -m mypy (same no-argument invocation as CI) → success in 22 source files; loopx canary premerge --from-git-diff → public boundary ok: true, 0 failures; python -m pytest -q tests/canary → 25 passed with the reviewed module findings unchanged.
regression_parity passed Mutation matrix run against the baseline copy of host_failure.py, one edit at a time, on this revision: drop min(...) → 2 failed; ceiling 3003000 → 2 failed; executor_timeout (2, 5)(3, 5) → 1 failed; exponent attempt-1attempt → 6 failed; remove the bool guard → 1 failed; normalize returns the caller's mapping → 1 failed; delete the "non-retryable must not declare retry policy" raise → 1 failed; delete the retryability/kind consistency raise → 1 failed. Reverting the module → 20 passed. Failing-before holds by construction: grep -n "300|executor_timeout|transport_lost|record_host_failure" tests/test_loopx_turn_host_failure.py returned 0 matches on the base revision.
real_entrypoint not_run No CLI or host entrypoint is added or changed; the executor producer is pinned at the function the executor calls, not through loopx turn.
integration running Full python -m pytest -q tests/ on this revision is running locally; the aggregate will be posted as a follow-up comment on this PR rather than claimed here.
  • Coverage and gaps: the only changed path is one test file, and the checks above cover
    every reader of the retry hint it pins. Not covered and not changed: scheduler wake
    timing and the TypeScript journal mirror (they consume the number). No known-failing
    local baseline is attributed to this PR; if the full run surfaces one, it will be posted
    with its file and name rather than smoothed over.

Frontend / Visual Evidence

  • UI impact: none
  • Before: N/A
  • After: N/A
  • States and viewports shown: N/A
  • Source data: none
  • Attention review: N/A — no user-visible surface changed.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Refactoring (no functional changes)
  • Documentation update
  • Test update

LoopX Area

  • Control plane (goals, todos, quota, scheduler, registry, runtime)
  • Benchmark boundary (adapters, runners, verifiers, scoring, evidence)
  • Capability or extension (providers, adapters, skills)
  • Public docs or presentation surface (README, protocols, dashboard)
  • Build, packaging, installer, or CI
  • Host or runtime integration

Technical Direction

  • Direction / acceptance reference, when applicable: Core control-plane hardening
    (coverage of an already-shipped bounded-retry contract). No roadmap id claimed.

Shared-authority RFC fixture impact

  • Production-scale fixture schema: N/A
  • Semantic dimensions changed, or reviewed no-impact rationale: N/A
  • Provider conformance arms run: N/A
  • Read-only legacy/file/PostgreSQL three-arm rehearsal: N/A — this PR changes no
    promotion, runtime-routing or compatibility projection.

Boundary Checklist

  • Neither the diff nor this PR body/comments/attachments disclose private state, credentials, raw traces or verifier output, internal links, or local machine paths (including .loopx/, .codex/goals/, and live ACTIVE_GOAL_STATE.md).
  • I did not duplicate maintainer-owned benchmark work unless a maintainer split out a public issue for it.
  • I kept the change scoped to the linked issue/task.
  • I completed the visual evidence section for UI changes, or marked UI impact none.
  • Every commit includes a DCO Signed-off-by trailer (git commit -s).

…pt range

The suite reached at most attempt=3 over three of the five retry policies, so the 300-second ceiling never executed, executor_timeout (the only two-attempt policy) and transport_lost were absent entirely, and the attempt-type guard and rebuilt-record isolation had no assertion. Pin the ladder, the cap, both missing policies, and the two fail-closed boundaries.

Signed-off-by: Yue Dai <54579099+yuedai-pbc@users.noreply.github.com>
Signed-off-by: Yue Dai <54579099+yuedai-pbc@users.noreply.github.com>
@yuedai-pbc

yuedai-pbc commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: the full local suite on this revision (187df5a6c), which the PR body left as running.

python -m pytest -q tests/92 failed, 11497 passed, 41 skipped in 47m36s.

None of the 92 is in the file this PR touches, or in any module it reads. Composition and the two independent facts I checked instead of asserting a parity number:

90 of 92 — tests/architecture/**, missing TypeScript parser dependency in this worktree

file failures
test_semantic_field_use.py 54
test_semantic_vocabulary_drift.py 16
test_semantic_production.py 9
test_semantic_producer_binding.py 8
test_project_registry_io_census.py 2
test_semantic_incident_retrodiction.py 1
test_goal_instance_binding_inventory.py 1

Each raises ValueError: TypeScript production parser failed; run npm ci --ignore-scripts and check the Node runtime. This checkout has no node_modules, so the production parser these census tests drive cannot start — an environment prerequisite, not a repository defect and not this diff. The same root cause takes down examples/semantic-vocabulary-drift-smoke.py locally.

1 of 92 — tests/test_external_scheduler_worker.py::test_default_invocation_persists_backoff_state

Reproduces when that file is run on its own, i.e. with this PR's test file absent from the selection (1 failed, 5 passed), and passes when the single test is run by name. So it is order-dependent in this local environment and cannot be attributed to this diff, which adds no module the scheduler worker imports. Its log line is a quota should-run failed: timeout tick error, consistent with a timing-sensitive path.

What I did not measure, stated plainly: I did not run the same 92-failure set against an unmodified checkout of this base revision, so I am not quoting a "identical failure set before and after" number — the two controls above (selection independence, and an explicit missing-dependency error) are what I actually have. CI runs the suite on the merge ref, which has the Node dependencies installed.

Everything else passed: 11497 tests, including the 20 in tests/test_loopx_turn_host_failure.py and the 437-test turn-driver neighborhood reported in the body.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

既有 host_failure.py 已发布带上限的重试提示,但原测试最多走到第 3 次尝试,只覆盖 5 种可重试策略中的 3 种,也没有把非法 attempt、伪造 retryability 与标准化后的对象隔离固定下来。本 PR 是补现有行为的回归测试,不改生产策略。

改动思路

沿用现有 tests/test_loopx_turn_host_failure.py,直接断言 ceiling、executor_timeout 两次上限、transport_lost 梯度、executor journal 的 producer、空投影、返回对象重建和两条拒绝路径。测试位置贴近策略所有者,没有另造 smoke 或协议层。

具体改动

唯一改动是测试文件新增 95 行;没有生产、配置或文档更改。exact head 187df5a6cf2394913935338da1f22f4c61cfce4a 的单文件测试 20/20、相邻 Turn/host/controller 七模块 438/438、Ruff 和风险分层 premerge 均通过;基线单文件是 8 个测试。现有 executor/controller 测试覆盖首次提示,但没有重复断言本次补充的上限和拒绝分支。作者同期的 #4955 检查另一项 public-safety 契约、#4954 处理仓库页面卫生,未见同形态重复拆 PR。

对主干的风险

没有发现阻断项。第 4/10 次 rate_limited 的 300 秒断言保护的是超出自动重试上限时仍可被记录/回读的有界提示,不代表控制器会安排第 4 次自动重试;测试同时明确断言 retry_available=false。新增测试直接调用策略和 journal producer,没有声称验证真实 host 的唤醒时序或 TypeScript journal 镜像;那两处本 PR 没有修改。未跑远端 CI:Goal 的当前评审策略为 wait_for_ci=false,本地所需验证已通过。

我的整体评价

这是对已有、正在使用的重试契约的聚焦持久化测试,覆盖空白明确、相邻路径通过,规模与收益相称。我未找到需要本 PR 修复的问题,批准当前 head。对相邻 owner 做未来导向检查后,保持测试留在现有模块比抽象新 helper 更合适;批准不代表合并授权。

English verdict: APPROVE - exact head 187df5a; focused retry-policy coverage is durable and non-duplicative. 20 focused and 438 adjacent tests, Ruff, and premerge validation passed; no production behavior changed.

@huangruiteng
huangruiteng merged commit 050483d into loopx-project:main Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants