Skip to content

fix(authority): paginate canonical snapshots without mixing revisions - #4922

Open
huangruiteng wants to merge 5 commits into
mainfrom
codex/canonical-snapshot-pages-0923
Open

huangruiteng wants to merge 5 commits into
mainfrom
codex/canonical-snapshot-pages-0923

Conversation

@huangruiteng

@huangruiteng huangruiteng commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

Problem and result

A valid long-lived Goal could exceed the unchanged 2 MiB Python/TypeScript RPC response ceiling when a canonical read returned every Todo, archived dependency, lease and acceptance guard at once. The public list/status call then failed despite a healthy authority store. This PR assembles complete, byte-bounded pages tied to one provider incarnation, revision, cursor and query. An overlapping commit rejects the whole read; no partial collection or legacy display fallback escapes.

This is the TS T3 / shared-authority L5 read-transport slice for #4574. It does not change the default provider or promote an active Goal.

Implementation and semantics

  • TypeScript owns collection admission, page selection and acceptance guard membership. Both the existing one-shot endpoint and new page endpoint share the same Todo/read-model validation owner; Python only validates and assembles the transport.
  • Pages contain at most 4,096 full records and 1,792 KiB of UTF-8 JSON beneath the existing 2 MiB RPC ceiling. Ordinary collections stay on one RPC. A record or metadata object too large for a page fails explicitly.
  • Continuations bind Goal, store identity, provider revision, cursor, query and population. A changed head requires restarting the complete read, even if record content happens to look the same.
  • A fenced but absent File authority remains missing with the existing recovery response; reading never creates a replacement store. An existing head with an unreadable identity remains an identity failure. The public result schema also stays present on failure.
  • The complex synthetic production population now crosses the real byte ceiling while preserving native and imported shapes, archives, leases and acceptance guards. File, SQLite, NoKV transport and real PostgreSQL use the shared conformance contract.
  • The latest main introduced a bilingual RFC without the repository-required semantic-mirror declaration. A separate two-line docs commit repairs that pre-existing governance-smoke failure; it does not change this RFC's normative contract.

The bilingual RFC checkpoint lists seven principal PR boundaries including this one; after merge, six remain, or up to eight when SQLite qualification and migration each need a second reviewable batch. In-flight #4920 covers shadow-drain planning only; merged #4910 adds SQLite capacity axes but does not close the D2 soak/recovery gate. PostgreSQL service, tenant, restore and capacity qualification remains a separate medium-term lane.

Validation on 373ae0fa4add247abd316f9e4be25de21527156b

Check Result Evidence / limit
Python caller/recovery regression Passed 58 focused cases after the latest-main rebase, including split-root, missing authority and shadow fence.
Native page conformance Passed 26 File/qualified SQLite cases; the public-minimum Node 22.18 run passes 17 File cases and marks SQLite conformance skipped on its unqualified embedded driver.
Real PostgreSQL 16 Passed 288 provider integration cases against a separate disposable server/database, both removed after testing. The first local attempt used an old child Python and failed four import cases; the source-checkout interpreter rerun passed 288/288.
Real source and packaged entrypoint Passed Owner-authorized read-only Goal source copied to disposable File/SQLite authorities; complete readback and CLI with absent display passed, while source registry/state/leases/fences remained byte-identical. A rebuilt wheel matches all nine changed product files; extracted-wheel File/SQLite RPC and CLI readback passed on an over-budget synthetic collection. No active Goal was promoted.
Static and docs Passed TypeScript typecheck, configured mypy (23 modules), focused Ruff, generated-contract check, docs governance, public/private scan and diff hygiene.
Change-quality Passed Final 28-file scope fingerprint 33d86fc50fc36bf261c802933eaa4c40cbdb6b872dedd3c49ee5e6fdd2332e68; receipt cqr_33d86fc50fc36bf261c8 verified valid. No blockers or warnings; one efficiency advisory.
Risk-based premerge canary Passed 18 selected and 5 direct checks; no failures, warnings or manual holds.
GitHub CI Running Exact-head checks are queued/running; maintainer review and merge remain required.

Each page currently reloads the full head, and Python ultimately holds the complete collection. The RPC is bounded; total provider I/O and client memory are not. Busy writers may force an explicit full-read restart. This does not qualify SQLite D2 durability/soak, persistent projection freshness, whole-Goal migration or PostgreSQL deployment.

The CLI is the affected entrypoint. Dashboard, Chat and Lark continue to consume the same assembled public list/status contracts; there is no new setting or frontend asset. This control-plane PR stays open for maintainer review and merge.

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
@huangruiteng
huangruiteng force-pushed the codex/canonical-snapshot-pages-0923 branch from a07903f to a5acb89 Compare September 23, 2026 05:58
… check

Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>
Signed-off-by: huangruiteng <14976749+huangruiteng@users.noreply.github.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
Exact head: 373ae0f
English verdict: APPROVE — no blocking finding on this head.

动机:已验证旧的 one-shot canonical Todo/lease RPC 在完整集合超过 2 MiB 时失效;这会让已提升的长期 Goal 无法读取 Todo。此次交付是有用且独立可回滚的有界读传输增量,不宣称解决总体内存/IO 或后续 D1–D3。

改动思路:TypeScript 继续拥有 canonical Todo 选择和排序,复用同一 collection owner;新 page RPC 把 store identity、revision、cursor、query digest、数量与偏移绑定到 continuation,并限制每页字节和条数。Python 只校验、组装完整结果;任一页变更/异常就丢弃此前页面,不混合 revision、不回退 legacy。原 one-shot RPC 保留给既有直接调用者。

具体改动:检查了完整 base→head 28 文件差异、schema/handler/provider、Python promoted 与 pre-fence 分支、文档/RFC和相关调用者。源工作树验证:Python canonical snapshot 34 passed;TypeScript File/SQLite page 26 passed;旧路径/写入栅栏 72 passed;独立临时 PostgreSQL 16 数据库的完整 authority-store suite 288/288 passed(无 skip);control-plane typecheck 与 diff check 均通过。首次 PostgreSQL 运行的 4 个失败来自子进程误用系统旧 Python,改为工作树 uv 解释器后全套通过;不归因于此 PR。私有路径/凭据扫描无命中。

对主干的风险:最强反例是两页之间发生同内容或不同内容的 provider commit,导致表面合法的页面被拼成混合快照;负例覆盖了 revision/identity/offset 拒绝以及 Python 不返回部分结果。每页仍重读完整 head,最终 Python 结果仍占用总量内存,繁忙写入时可能反复重启;文档明确承认,这是后续按实测决定是否引入 pinned revision/streaming 的范围。没有对活动 Goal 做 promotion、迁移或原始数据改写,PostgreSQL 验证只用隔离的合成数据库。远端 CI 按本轮 review 契约不作为等待条件。

我的整体评价:未发现阻塞项。范围与已复现的 2 MiB 故障相称,规则仍归属于 TS 的 canonical owner,off/default legacy 分支有回归覆盖;可由 maintainer 合并。相关小型未来重构已通过共享 canonicalTodoCollection 完成,暂不扩展为无依据的流式框架。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant