Skip to content

feat(telemetry): give every failure an error code and report where a run stopped - #441

Draft
djgould wants to merge 1 commit into
mainfrom
devin/init-telemetry-stages
Draft

feat(telemetry): give every failure an error code and report where a run stopped#441
djgould wants to merge 1 commit into
mainfrom
devin/init-telemetry-stages

Conversation

@djgould

@djgould djgould commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

clerk init fails on 21.4% of runs (1,596 runs / 30 days), and the telemetry can't say why or where. This makes both legible.

  • Error codes on all 87 CliError sites. 19 were uncoded and collapsing into a generic cli_error — 125 of 342 init errors (36.5%). Adds 14 ERROR_CODE entries across init, update, deploy, env, users, and switch-env. Agent-mode JSON gates on error.code, so these failures now emit structured JSON where they previously fell through to plain text.
  • A stage dimension recording how far a multi-step command got, emitted on success, error, and abort — a drop-off funnel, not an error-only field. clerk init is instrumented across ten markers: flags → detect → strategy → link → bootstrap → install → scaffold → keys → skills → done, plus already_set_up for the clean-rerun short-circuit.
  • Registry failures split by cause. clerk update --channel typo reaches npm fine but reported registry_unreachable, so agents retried a permanent input error. Now usage_error for a missing dist-tag, update_failed for a bad response, registry_unreachable only for transport/timeout.

Disclosure notice and README both updated — this collects a new field.

What it does not fix

The 107 unexpected_error runs (31.3% of init errors) are plain Error throws that never reach CliError; stage will say where they die but the error stays anonymous. Capturing error.constructor.name would decompose them — deliberately left out of this PR.

Review

Carries a prior multi-model adversarial review (Claude + Codex, 19 agents, 10 findings, 0 refuted). All 8 distinct findings are applied, with one deviation: finding 7 proposed deleting the dead users/shared.ts; this takes the leaner option the reviewer offered — dropping the NOT_IMPLEMENTED code so the PR doesn't mint a public error code for dead code, leaving the out-of-scope file alone.

The three new stage tests were mutation-checked: each fails when its marker is removed.

Verification

typecheck, lint, format:check clean; 2673 pass / 0 fail. E2E not run (needs 1Password-injected secrets).

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Aug 22, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6e9f1d3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
clerk Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…run stopped

Errors were collapsing into one bucket: 19 of 87 `CliError` sites carried no
code, so 36.5% of init failures reported a generic `cli_error`. Adds 14
`ERROR_CODE` entries and applies them across init, update, deploy, env, users,
and switch-env, so agent-mode JSON also gets a code where it previously fell
through to plain text.

Adds a `stage` dimension recording how far a multi-step command got, emitted on
success, error, and abort — a drop-off funnel rather than an error-only field.
`clerk init` is instrumented across ten markers. Splits npm registry failures so
a bad `--channel` reports a usage error instead of an unreachable registry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant