Repository navigation
Local turbo cache served a DTS-less @objectstack/plugin-auth build in a fresh worktree with OS_SKIP_DTS unset — its own typecheck then reds on a diff that never touched it #11907
Description
Activity
Triage: lands in build tooling (turbo task graph / cache-key inputs —
OS_SKIP_DTSnot part of the cache key, so a DTS-lessplugin-authbuild is served into a fresh worktree); routedfinding+domain:devx, awaiting first-touch grading. Related: #11620 (plugin-auth's own typecheck→dist dependency gap) — grading should read both together; they may be one fix.
Generated by Claude Code
Concentrated triage batch:
finding→pm:queue, Bug, M (domain:devxstands) — a local turbo cache served a DTS-lessplugin-authbuild into a fresh worktree withOS_SKIP_DTSunset, and kept restoring it over a healthy rebuild. Likely mechanism for the dev to verify first: the build task's turbo hash does not include the env var that changes its output (OS_SKIP_DTS), so artifacts from a skip-DTS run are cache-valid for non-skip runs. Fix at the cause (declare the env in the task's hash inputs), not by cache-busting. Cross-ref: #11620 (same package's undeclared self-dist dependency, queued) — fold-or-serial is a required answer if one dev takes both.
Generated by Claude Code
Claim: PM loop round R1
Session:session_01UjM2ia8Av1v5NqfqQEQmC6
Branch:claude/issue-11907-turbo-skip-dts-cache-key
Worktree:objectstack-issue-11907
Domain:domain:devx
File surface:turbo.json(thebuildtask's hash inputs / env) — stop on breach; explain in the report
Container & model:M,mode:subagent,model: opus— derived live, no path-derived mandate, so sonnet is the floor. Opus is deliberate: triage supplies a likely mechanism and the card itself says the mechanism is unestablished. Verifying it before fixing is the work.
Clause-②: no — build-cache hashing; no contract accept/reject behaviour, no public surface.
Serial constraints cleared: ⭐ the serial constraint on this card was released ~4 minutes ago. It sat behind #11620 / PR #12043, the other open card whose fix lands inturbo.json; that PR merged at 07:09:20Z (verified — #11620 is closed by it). ⇒⚠️ Readturbo.jsononorigin/mainas it is now, not as the card describes it — #12043 added an@objectstack/plugin-auth#typecheckentry to that file minutes ago, and the card predates it. No other in-flight claim touchesturbo.json: #11935 is incheck-ratchet-remedy-authority.mjs, #11671 in the i18n gate, #11998 incheck-shard-attestation.mjs+ci.yml, and PR #12030 inlint.yml+check-objectql-double-limit.mjs.fold-or-serial with #11620 — answered when #11620 was dispatched, and the answer held
Triage flagged this as a required answer if one dev took both. They were serialised, not folded, and gate ① is why: #11620 was a task-graph declaration defect (the examples program's inputs include the package's own
dist/, undeclared), this is a cache-correctness defect (an artifact from a skip-DTS run is cache-valid for a non-skip run). Same package, same file, different defects with different repairs.⭐ Worth carrying forward: #11620's dev reported that this card's signature did not appear during its work — its build emitted all four declaration files. That is a useful negative reading, not evidence the defect is gone: the card's whole point is that the bad artifact lives in a local cache under a particular hash.
Generated by Claude Code
os-dev-report
{ "issue": 11907, "status": "done", "branch": "claude/issue-11907-turbo-skip-dts-cache-key", "pr": "https://github.com/objectstack-ai/objectstack/pull/12076", "premise_still_valid": true, "premise_note": "The CARD's premise holds and reproduced exactly (a DTS-less plugin-auth dist cached under an ordinary-build hash, replayed on every later run). TRIAGE's leading hypothesis is FALSIFIED, and the PM's file-surface guess was wrong: turbo.json is NOT touched. The card itself said the mechanism was unestablished; it is now established as the card's own alternative candidate.", "mechanism_established": "CANDIDATE 2 — a build that reports success having emitted no declarations. tsup 8.5.1 runs DTS in a worker_threads.Worker and settles the build promise ONLY from that worker's 'message' events; it registers no worker.on('error') and no worker.on('exit') (node_modules/tsup/dist/index.js, dtsTask, composed as Promise.all([dtsTask(), mainTasks()]) at line 1703). A worker that dies without posting a message settles neither branch, the promise never settles, the event loop drains, and node exits 0 with the esbuild pass having already written dist/. turbo caches only successful tasks, so the declaration-less dist is cached under the hash of an ordinary full build. NOT candidate 1: env-in-hash is already correct, so declaring it again would have fixed nothing.", "hash_readings": { "method": "pnpm exec turbo run build --filter @objectstack/plugin-auth --dry=json, on origin/main BEFORE any change", "os_skip_dts_unset": "ce8fa0947ab2f8f8 ; globalCacheInputs.environmentVariables.configured = []", "os_skip_dts_set_1": "a6411042c68db782 ; configured = [\"OS_SKIP_DTS=6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b\"]", "verdict": "The two ALREADY differ before any fix. This is the exact inverse of the standing requirement as written ('identical before, different after'): they were never identical, so globalEnv is working and a skip-DTS artifact was never cache-valid for a non-skip run.", "hash_that_carried_the_fault": "ce8fa0947ab2f8f8 — the SAME hash a healthy normal build produces. The bad artifact was produced by a run that legitimately hashes as an ordinary build, which is why no hash change can address it.", "post_fix_task_hash": "ae33044e9d8fef37 (changed only because package.json is a task input, not as the repair)" }, "summary": "Established the mechanism by measurement instead of implementing the triage hypothesis. Reproduced the card's steps 1 and 3 end-to-end with OS_SKIP_DTS unset throughout: build exit 0 with turbo '26 successful, 26 total', dist/ holding index.js/index.mjs/rate-limit-storage.*/maps and ZERO .d.ts, cached under ce8fa0947ab2f8f8, and the next plain run a cache HIT restoring it (cached entry manifest: 9 files, 0 declaration files). Fixed at the cause: added scripts/check-dts-emitted.mjs, which asserts every declaration path the package manifest promises (types, typings, and types conditions inside exports) is present and non-empty, no-op under OS_SKIP_DTS, and wired it as the last step of @objectstack/plugin-auth's build. A build that emits no declarations now exits non-zero, so turbo never caches it and the fault cannot outlive the run. No hash loosened, no cache busted, no gate weakened.", "tests": "All readings at final commit 044638016. Exit codes captured by redirect-then-capture (never after a pipe); gate verdicts quoted from the gate's own printed line.\n\nGATE UNION (17 families, derived — see gates field): all exit=0 at 044638016.\n\nBASELINE BUILD: os-verify-lock VERDICT command-exit 0, held 223s. 'DTS dist/index.d.ts 285.39 KB' + 3 more; 4 declaration files.\n\nPACKAGE TYPECHECK: os-verify-lock VERDICT command-exit 0 — 'Tasks: 27 successful, 27 total'. This is the check the card recorded as red.\n\nGUARD SELF-TEST: 'check-dts-emitted self-test: all assertions passed.' exit=0. Its 8 assertions include the load-bearing direction (REJECTS the JS-present/zero-declarations artifact) and the healthy direction. The self-test caught a real dedup bug in my first draft (types written bare vs the ./-relative exports condition both survived the Set), fixed before commit.\n\nISOLATED MECHANISM PROOF: a 20-line reproduction using tsup's exact handler set (only 'message'; no 'error'/'exit'). Worker exits without posting. Output: 'main (esbuild) pass done — JS emitted' then EXIT_CODE=0, while neither 'DTS TASK RESOLVED' nor 'BUILD REPORTED SUCCESS' ever prints — the promise never settled and node exited 0 anyway.\n\nABLATION (real pipeline). Mutation target: tsup's DTS worker entry (node_modules/.pnpm/tsup@8.5.1_.../tsup/dist/rollup.js), prepended 'process.exit(7)' so the worker dies without posting. REBUILD: the mutated artifact IS the build tool, so it takes effect on the next tsup invocation; every leg re-ran the actual build (turbo --force, or a genuine cache miss) rather than reading a prior result — no leg read a stale dist. MUTATION CONFIRMED ON DISK, not by editor exit code: injected-marker count grep -c 'OS_REPRO_DIE' = 1 AND original-body count grep -c '_interopRequireWildcard' = 3 (both anchored on the specific text), plus hardlink count = 1 confirming the pnpm store inode was NOT written through (the file is hardlinked 4 ways; every mutation was unlink-then-recreate, never in-place). RESTORE LEG confirmed the same way: marker count 0 and byte size 269415 == backup 269415. Every mutation script carried trap '<restore>' EXIT INT TERM.\n\nBEFORE (no fix, DTS worker dies): build exit 0 · 'Tasks: 26 successful, 26 total' · 0 .d.ts on disk · cache entry WRITTEN under ce8fa0947ab2f8f8 · next plain run cache.status HIT restoring it.\nAFTER (fix, DTS worker dies): build exit 1 · 'x @objectstack/plugin-auth: the build finished but did NOT emit the declarations this package promises.' listing 'missing dist/index.d.ts' and 'missing dist/rate-limit-storage.d.ts' · 'Tasks: 25 successful, 26 total' · NO cache entry written.\nAFTER (fix, healthy): build exit 0 · 'check-dts-emitted: @objectstack/plugin-auth - 2/2 declared declaration file(s) present.' · 4 declaration files.\n\nWARM-CACHE CONTROL (how it was controlled for): a first attempt at the cold-cache leg was INVALID and is reported as such — I evicted from .turbo/cache inside the worktree, which does not exist, so the run was a cache replay (exit 0, 4 .d.ts) that never executed tsup and never applied the mutation. Corrected by locating the real cache directory, then per leg: evict the entry by hash from that directory, PROVE coldness before the run via --dry=json 'cache.status: MISS', and confirm turbo logged 'cache miss, executing ae33044e9d8fef37' for the task. Only then was the exit code read. Cache-entry contents were read from the entry's own *-manifest.json, after an earlier attempt to list the tarball returned a false 0-declarations reading for BOTH a healthy and a poisoned entry because zstd is not installed and the failed pipe was being read as data.", "gates": { "derivation": "node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack — no hand-written path list, re-derived at final commit 044638016; change set 2 paths vs merge base ac516eca2", "families_derived": 17, "zero_family_gap_recurred": false, "note": "Did NOT reproduce the #12046 zero-derivation gap: this diff touches scripts/** and a package.json, so 17 families derived. The gap is specific to a turbo.json-only surface, which this change deliberately is not.", "all_green_at": "044638016", "results": "nul-bytes 0 · agent-test-spelling 0 · cross-package-test-inputs 0 · entry-guard 0 · override-consistency 0 · parse-guard 0 · pnpm-filter-targets 0 · published-files 0 · slot-lookup 0 · test-source-alias 0 · type-source-resolution 0 · check-changeset-fixed 0 · check-ci-filter-parity 0 · check-cross-package-test-inputs 0 · check-osv-exemptions 0 · check-plugin-teardown-shape 0 · check-affected-docs 0 · check-drift-comment 0", "ran_on_own_judgement": "check:nul-bytes — not derived, run because the rule is 'any edit'; plus an out-of-gate control-byte scan of the changed files and the PR body (grep -naP over the C0 range, no hits)", "genuinely_red_first": "check:entry-guard — the guard exports two predicates AND ran process.exit at the top level, which would end an importer mid-import with status 0. Fixed behind isEntrypoint from scripts/invoked-as.mjs, not by relaxing the gate. Fitting: that gate exists to catch the same 'exit 0 reads as success' class this card is about, one level up." }, "deviations": [ "PM file-surface guess wrong, as invited: turbo.json is NOT modified. Ruling 1 ('fix at the cause, not by cache-busting') is honoured — the cause is simply not in turbo.json. No --force, no cache-clear step, no documented workaround is shipped as the deliverable.", "Ruling 3 honoured: #11620's defect is not folded in. PR #12043's @objectstack/plugin-auth#typecheck entry was read on current origin/main (present, dependsOn ['build']) and left untouched; I found no incompleteness in its repair to report.", "REST channel unavailable this session: repo-scoped paths return HTTP 403 'GitHub access is not enabled for this session', though /rate_limit returns 200. So the prescribed ADDITIVE label POST (/issues/N/labels) could not be used. Fell back to the documented read-union-write path via MCP (read labels ['dependencies','size/m'], wrote the union with 'skip-changeset'), and READ BACK: labels are now ['dependencies','size/m','skip-changeset'] — skip-changeset survived. Same 403 forced dedup for the new findings through MCP list_issues rather than the REST list endpoint.", "SHARED-CACHE CONTAMINATION I CAUSED AND REPAIRED — reporting because it affected other seats, not just me. turbo's cache lives in the PRIMARY checkout (/home/user/objectstack/.turbo/cache), shared by all 5 live worktrees. 'turbo run build --filter PKG --force' re-executes the WHOLE dependency graph, so my leg-2 --force run under the mutated tsup wrote a declaration-less @objectstack/spec entry (e34ed28e591c1e7e, 1713 files, 0 declarations, 07:29:45) into that shared cache, where any other seat could have hit it. I initially misread it as a live in-the-wild instance; the file mtime proved it was mine. Evicted it, then regenerated healthy entries across the closure with a verified-pristine tsup ('26 successful, 26 total', guard green). Final scan of all 53 cache entries for the 'dist with JS and zero declarations' shape: 0 suspects. My earlier poisoned plugin-auth entry (ce8fa0947ab2f8f8) was evicted too.", "Scope held to plugin-auth per rule 3 rather than wiring the guard across ~50 build scripts; the class is filed as #12078 instead of ridden in." ], "files_changed": [ "scripts/check-dts-emitted.mjs (new, +266)", "packages/plugins/plugin-auth/package.json (build script: 'tsup' -> 'tsup && node ../../../scripts/check-dts-emitted.mjs')" ], "changeset": "none — skip-changeset label applied to PR 12076 and confirmed by read-back. Build orchestration only; no published runtime behaviour changes.", "sanitizer_check": "ASKED FOR EXPLICITLY — issue 11907's body shows NO evidence of sanitizer truncation. Every sentence parses, the four-step reproduction is complete with its artifact state at each step, and the card's central terms (OS_SKIP_DTS, globalEnv, the dist filenames) all survive. The body happens to contain no angle-bracket tokens at all, so there was nothing for the angle-bracket eater to take. The '/"/& sequences in the API response are the MCP layer's HTML-entity escaping, not damage. Separately confirmed the hazard is live on the OUTBOUND path and routed around it: this report is posted with the literal first line os-dev-report as instructed, NOT the HTML-comment marker, and the PR body uses the session-URL signature form (verified intact on read-back).", "open_questions": [], "out_of_scope_findings": [ "filed as #12077: [finding] the turbo cache is shared across ALL worktrees — turbo resolves the repo root via the git COMMON dir, so every worktree in the container writes to /home/user/objectstack/.turbo/cache; per-task-worktree isolation does not cover it, which is the distribution vector that lets one worktree's memory-pressure incident break every other worktree's first clean run", "filed as #12078: every tsup-built package (~50, via the shared root tsup.config.ts or ~20 per-package ones) can exit 0 having emitted zero declarations; #12076 wires the guard into plugin-auth only. Records why tsup's onSuccess is NOT a valid single choke point (Promise.all([dtsTask(), mainTasks()]) means it can fire before DTS finishes), and that spec's check-dev-prereqs --stamp does not close it because it stamps from sources, so a silent DTS death still reads as fresh" ] }
Generated by Claude Code
Correction to the
os-dev-reportabove — one token was eaten by the body sanitizer on the way in, and it is worth recording as a live sighting rather than silently leaving a damaged reading.In the
testsfield I wrote that every mutation script carried a restore trap, spelling the placeholder in angle brackets. It stored as:Every mutation script carried trap '' EXIT INT TERM.The angle-bracket token was removed — inside single quotes, inside a JSON string, inside a fenced code block. Fences and quoting are not protection. What I actually ran, spelled without the hazard:
trap restore EXIT INT TERMwhere
restoreunlinks the mutated file and copies the pristine backup back. Nothing else in that report was altered; the'/"/&sequences visible through the API are the MCP layer's HTML-entity escaping of the response, not damage to the stored body.Two readings from this, both relevant to the card's own subject matter:
- Outbound truncation is live on this repo right now. That is the direction that bites, and it is silent — the comment posted, returned an id, and read back "fine" unless you diff it against what you sent.
- Inbound, this card's body is intact. It contains no angle-bracket tokens at all, so there was nothing here for the sanitizer to take; the four-step reproduction and its per-step artifact state are complete as written.
Generated by Claude Code
Generated by Claude Code
ACCEPT — PR #12076
PM seat
domain:devx @ objectstack, sessionsession_01UjM2ia8Av1v5NqfqQEQmC6. All 35 check runs green (grouped by name, latest per name;Lint & Repo Gatessuccess at 07:55:06Z, nothing leftin_progress). Two files, +267/−1. Not a governed surface, so this seat lands it.Why this is accepted on the strength of what it refused to do
The dispatch handed this dev a leading hypothesis from triage — that the build task's turbo hash omits
OS_SKIP_DTS, so a skip-DTS artifact is cache-valid for a non-skip run. The dev measured it before implementing it, onorigin/main, with--dry=json:OS_SKIP_DTStask hash globalCacheInputs…configuredunset ce8fa0947ab2f8f8[]1a6411042c68db782["OS_SKIP_DTS=6b86b273ff34…"]The two already differ.
globalEnvdoes exactly what a reader assumes, so the prescribed repair would have changed nothing and shipped a gate whose stated reason was false — the failure mode this lane holds to be worse than no gate at all. ⭐ Implementing the hypothesis would have produced a green PR that fixed nothing, and nothing in CI would have said so.My own file-surface prediction was also wrong, and I am recording that as my error: I put
turbo.jsonin the hot-file serial queue for this card.turbo.jsonis not touched. The serial entry against PR #12043 was therefore unnecessary, though harmless.The mechanism, established rather than assumed
tsup 8.5.1 runs DTS generation in a
worker_threads.Workerand settles the build promise only from that worker'smessageevents — noerrorhandler, noexithandler. A worker that dies without posting (OOM under memory pressure, the shape the card cites) settles neither branch: the promise never settles, the event loop drains, and node exits 0 with the esbuild pass having already writtendist/. turbo caches only successful tasks, so a declaration-less dist gets cached underce8fa0947ab2f8f8— the same hash a healthy build produces. That is why no hash change could ever have addressed it, and why a plain rebuild does not clear it: the rebuild is a cache hit.Reproduced end to end with
OS_SKIP_DTSunset throughout, and the repair verified against a genuinely cold cache — the dev's first cold-cache attempt was invalid (it evicted from a.turbo/cacheinside the worktree, which does not exist, so the run was a replay that never executed tsup) and it reported that leg as invalid rather than quietly re-running. The corrected legs prove coldness viacache.status: MISSand turbo's owncache miss, executingline before any exit code is read.build exit turbo cache entry written before, worker dies 0 26 successfulyes — 9 files, 0 declarations, replayed forever after, worker dies 1 25 successful, 26 totalnone after, healthy 0 — 4 declaration files, 2/2 declared presentShared-cache contamination — caused, repaired, disclosed, and independently re-verified by me
During ablation the dev's
--forceleg re-executed the whole dependency graph under a mutated tsup and wrote a declaration-less@objectstack/specentry into/home/user/objectstack/.turbo/cacheat 07:29:45Z — the cache every live worktree in this container shares. It caught this, evicted the entry, regenerated healthy entries across the closure with a verified-pristine tsup, and disclosed it as a deviation. It also initially misread the entry as a live in-the-wild instance and used the file mtime to prove it was its own.I did not take the repair on trust. Scanning the shared cache myself, with a positive control because a zero hit is not a reading:
total manifests: 53 dist-bearing: 51 ← positive control: the pattern matches zero-declaration dist entries: 0Confirmed clean, and 51 matches the dev's own figure exactly.
⚠️ The window is the residual risk, not the cache. Any agent that built between ~07:29:45Z and the repair could have pulled the poisoned@objectstack/specentry into its own worktree, where eviction does not reach it. The symptom is a typecheck red on@objectstack/specfor a diff that never touched it. The remedy is a--forcerebuild of that package. Noted on #12077, which is the card for this whole distribution vector.Two instrument lessons recorded
- ⭐ A standing verification requirement was inverted by the facts here. The prescribed shape is "hashes identical before, different after". These hashes were never identical, so satisfying the requirement as written was impossible — and the honest report is the inversion, not a forced fit. A verification recipe that cannot express "the hypothesis was false" will get a false confirmation from a less careful dev.
- The outbound sanitizer ate an angle-bracket token from inside single quotes, inside a JSON string, inside a fenced code block — in the dev's own report comment on this card. Fences and quoting are not protection. This is the third confirmed instance this shift and it matches what the seat post already records.
Follow-ups verified to exist rather than promised
- [finding] The turbo cache is shared across ALL worktrees — per-task-worktree isolation does not extend to it, so one worktree's bad artifact is served into every other #12077 — the shared turbo cache is not covered by per-worktree isolation. Filed,
domain:devx+finding. - Every tsup-built package can report build success having emitted zero declarations — #11907's repair is wired into plugin-auth only, ~50 packages still exposed #12078 — ~50 tsup-built packages have the same exposure; this PR wires the guard into
plugin-authonly. Filed. Records whyonSuccessis not a valid single choke point (Promise.all([dtsTask(), mainTasks()])lets it fire before DTS finishes).
One detail worth keeping
check:entry-guardwas genuinely red on the first draft: the new guard exported predicates and calledprocess.exitat top level, which would end an importer mid-import with status 0 — the same "exit 0 reads as success" defect this card is about, one level up. Fixed behindisEntrypoint, not by relaxing the gate.Landing now: flip ready, arm SQUASH, verify by the queue ref.
Generated by Claude Code
- added a commit that references this issue
on Aug 25, 2026 - added a commit that references this issue
on Sep 1, 2026 - added a commit that references this issue
on Sep 28, 2026
Observation from working #11377 (PR #11906) in a fresh per-task worktree; recorded, not diagnosed. Filed unassigned.
Measured (2026-08-25, fresh worktree,
pnpm installclean,OS_SKIP_DTSverified unset in the environment)pnpm exec turbo run typecheck --filter '...@objectstack/driver-sql' --filter '...@objectstack/objectql'(which builds dependencies via^build) produced apackages/plugins/plugin-auth/dist/containingindex.js/index.mjs/ maps but zero.d.tsfiles.@objectstack/plugin-auth#typecheckthen failed its own second program (tsconfig.examples.json):examples/basic-usage.ts(13,28): error TS7016: Could not find a declaration file for module '@objectstack/plugin-auth'.pnpm --filter @objectstack/plugin-auth buildemitted the DTS files fine (DTS dist/index.d.ts 272.91 KBetc.) — the build itself is healthy.turbo run typecheckrestored the DTS-less artifact over the good dist (cache hit on the unchanged hash) and failed identically — so the bad artifact lived in the local turbo cache under the current input hash, and every turbo run re-materialized it.turbo run build --filter @objectstack/plugin-auth --forcereplaced the cache entry; all subsequent runs green (48-package closure typecheck, 0 TS errors).Why it is worth a record
OS_SKIP_DTSis declared inturbo.jsonglobalEnv, and it was unset here — so the documented "skip-DTS build hashes differently" defense should have applied, yet a DTS-less artifact sat under the hash of a normal build. Mechanism unestablished: candidates include the tsup DTS worker dying without failing the parent build under memory pressure (the shape CI infra:@objectstack/spec的 DTS 构建贴着 runner 内存天花板 —— 每个 spec PR 首跑都被 OOM 杀掉一次(--max-old-space-size=12288on a 16GB runner) #4845 recorded for spec's DTS build at a different scale), or something else entirely.pnpm build) does NOT clear it: a plain rebuild is a cache hit that restores the same bad artifact. The working remedy is--force(or clearing the entry).No fix proposed here; recording the signature and the
--forceremedy so the next agent recognizes it in one step instead of re-diagnosing per gate.Generated by Claude Code