feat(telemetry): missed-opportunity signal at checkpoint condensation

[jdx]

https://entire.io/gh/entireio/cli/trails/1071

Raw search-command counts can't say whether adoption is low — most sessions have nothing worth searching for. The denominator that makes the rate meaningful is commits that landed files already carrying AI checkpoint history without the session ever consulting search. Entire is the only tool that can compute it, and nothing emitted it.

On each successful post-commit condensation, emit cli_commit_condensed.

Properties

Property	Meaning
used_search	Did the session invoke entire search? Omitted when not measurable — see below.
used_search_source	How used_search was determined: command, subagent, none, or unsupported. Always present.
prior_ai_history	Did any committed file appear in a recent commit carrying an Entire-Checkpoint trailer?
files_committed	Count only.

The event is named for the commit, not the checkpoint, because every property is commit-scoped: files_committed counts that commit's files, and prior_ai_history asks whether commits before this one touched them — which is what the probe's --skip=1 is for. The commit-less condensation paths (doctor repair, session-end leftovers) deliberately emit nothing; rows without a commit would be indistinguishable from genuine misses and would inflate the denominator rather than complete it. Covering them would need a trigger discriminator plus nullable commit-scoped fields — a change to what the metric means, not a bug fix. The rationale is recorded at both call sites and on newCommitCondensedSignal.

Measuring used_search

The first version was a bytes.Contains probe. Review measured it at 18 false positives to 1 true positive across 328 real transcripts, and the false positives are structural — Entire installs the artifacts that trip its own probe (setup_search_skill.go embeds entire search --json in the search skill's body; investigate/prompt.go injects it into every investigate prompt). The direction is what made it unusable: inflating used_search deflates the missed-opportunity rate, so the metric reads "adoption is fine" for the wrong reason.

It now matches a recorded tool invocation through a new built-in-only ToolInvocationScanner capability whose dispatcher returns (found, supported):

• command — a shell tool ran the command, matched in command position (start of command or after a shell separator, tolerating env assignments and a path prefix), so cd sub && entire search x matches while grep -rn "entire search" cmd/ does not.
• subagent — the entire-search subagent was dispatched. This is the primary path, not a nicety: setup_search_skill.go installs search as a subagent, so the intended usage records an Agent/Task tool_use and runs the actual command in a separate subagent transcript that condensation never reads. A shell-only matcher would report "did not search" for exactly the sessions that adopted the feature.
• unsupported — no walker exists for this agent's transcript shape. used_search is then omitted from the payload rather than sent as false, so a consumer filtering used_search = false excludes unknowns instead of absorbing them; a missing PostHog property is not false. Claude Code is the only implementer today.

Consumers should compute the rate over the measurable population:

SELECT countIf(properties.prior_ai_history AND NOT properties.used_search) / count()
FROM events
WHERE event = 'cli_commit_condensed'
  AND properties.used_search_source != 'unsupported'

Privacy and cost

Content-free by construction: booleans, a count, and the agent registry key — never file paths, prompts, or transcript content. Gated on ENTIRE_TELEMETRY_OPTOUT and then the opt-in telemetry setting before any probe work, so an opted-out user never pays for the git-log subprocess. The PostHog call is the existing detached-child path and never blocks the hook.

Both commit-scoped costs are memoized once per commit rather than per session: the settings load and the single git log, whose 13.8ms is almost entirely process spawn (8.7ms for git --version alone, and flat across a 60x range of output size). Both resolve on first use, so a commit where no session condenses pays nothing.

The git log record framing is NUL-anchored (--format=%x00%H%n%B). A commit message cannot contain NUL, so the record marker rests on nothing about message content — the previous %x1e/%x1f framing took its delimiters from the raw body, and a body containing either control character split a record early and dropped a real trailer, making prior_ai_history read false and suppressing a genuine miss.

Tests

Regression coverage for each documented false-positive class (grep, commit message, tool output, assistant prose, the scaffolded skill body being written, the injected investigate prompt); unprobeable agents required to report unsupported when fed a transcript that does contain a real invocation; the hint/matcher case-sensitivity relationship pinned in both directions; the zero-value probe required not to present itself as a measurement; a commit body containing literal \x1e/\x1f still attributed; empty-message and merge commits pinned; and the probe and settings load asserted to run once across several sessions.

Verified green: mise run lint 0 issues, all 64 packages, plus both canary suites (56 Vogon, 4 roger-roger).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds a new, content-free telemetry signal emitted on successful post-commit checkpoint condensation to better measure “missed opportunity” adoption: sessions that committed files with prior AI checkpoint history without using entire search.

Changes:

• Introduces a cli_checkpoint_condensed telemetry event payload and detached sender (CheckpointCondensedSignal, builder, tracker).
• Adds strategy-side signal computation: used_search (substring probe over in-memory transcript) and prior_ai_history (bounded git log scan for recent commits with an Entire-Checkpoint trailer touching committed files).
• Wires emission into the post-commit condensation flow and adds unit tests for transcript probing and the git-log-based prior-history detector.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
cmd/entire/cli/telemetry/detached.go	Adds the new condensed-checkpoint telemetry signal type, payload builder, and detached tracking function.
cmd/entire/cli/telemetry/detached_test.go	Adds payload-building tests for the new event and centralizes the agent test constant.
cmd/entire/cli/strategy/telemetry_signals.go	Implements used_search detection + prior AI-history probing and emits the condensed-checkpoint telemetry signal (telemetry opt-in gated).
cmd/entire/cli/strategy/telemetry_signals_test.go	Adds unit tests for transcript search detection and prior-checkpoint-touch detection via a temp git repo.
cmd/entire/cli/strategy/manual_commit_types.go	Extends CondenseResult with UsedSearch for downstream telemetry emission.
cmd/entire/cli/strategy/manual_commit_hooks.go	Hooks telemetry emission into the successful condensation path.
cmd/entire/cli/strategy/manual_commit_condensation.go	Populates CondenseResult.UsedSearch from the extracted raw transcript.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jdx]

Both findings addressed in 6f6eda2:

• Agent registry key: emitCheckpointCondensedTelemetry now resolves state.AgentType (display name, "Claude Code") to the registry key ("claude-code") via agent.GetByAgentType, so the agent property lines up with skill and command events. Unknown agent types fall back to the stored string rather than dropping the signal.
• Doc wording: CheckpointCondensedSignal now states the actual invariant — content-free metadata (no file paths, prompts, or transcript content) — and documents Agent as the registry key. Also applied the shared "auto" defaulting to this payload builder for consistency.

Rebased on the updated #2023.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (3)

cmd/entire/cli/strategy/telemetry_signals.go:93

• The environment opt-out is not checked until TrackCheckpointCondensedDetached, after this function has already run the git log probe. A user who opted in previously and then sets ENTIRE_TELEMETRY_OPTOUT still incurs the repository scan, contrary to the PR's guarantee that no work/probe occurs for opted-out users. Include the environment opt-out in this early gate.

	s, err := settings.Load(ctx)
	if err != nil || s.Telemetry == nil || !*s.Telemetry {
		return

cmd/entire/cli/strategy/manual_commit_hooks.go:1411

• This call executes inside the MutateSessionState callback at lines 1004-1010, so settings I/O, a git log subprocess, machine-ID lookup, and detached-process spawn all extend the per-session gate hold. The established telemetry pattern explicitly requires emission after mutation returns (cmd/entire/cli/lifecycle.go:1579-1581) to avoid blocking concurrent hooks. Capture the signal while locked, then run this telemetry work after MutateSessionState completes.

	// Content-free adoption signal (opt-in telemetry gated inside).
	emitCheckpointCondensedTelemetry(ctx, state, result)

cmd/entire/cli/strategy/telemetry_signals.go:40

• The acknowledged quoting behavior makes prior_ai_history incorrect for ordinary non-ASCII paths (for example, Git outputs "caf\303\251.go" while FilesTouched contains café.go), and line-based parsing also cannot represent filenames containing newlines. This systematically creates false negatives in the metric. Request NUL-delimited names (-z) and parse those names without trimming or Git quote decoding.

// Paths git quotes in --name-only output (e.g. non-ASCII names) won't match
// their unquoted FilesTouched form; that false-negative is acceptable for a
// telemetry boolean.

[jdx]

Addressed the three remaining review findings in 5a79a16, and rebased onto the updated #2023:

• Env opt-out checked before any probe work: emitCheckpointCondensedTelemetry now gates on the new telemetry.IsEnvOptedOut() before settings load and the git-log scan, so a user who opted in previously and then set ENTIRE_TELEMETRY_OPTOUT never pays for the repository probe. The other trackers share the helper.
• Emission moved outside the session gate: condensation now only snapshots a cheap, I/O-free condensedTelemetrySignal while locked; the PostCommit loop emits it after the session's MutateSessionState saves, alongside the skill-event emission — same pattern the previous round established on feat(telemetry): emit skill invocations as cli_skill_invoked events #2023.
• prior_ai_history no longer false-negatives on non-ASCII paths: the probe passes -z, so names arrive unquoted and NUL-terminated (café.go matches its FilesTouched form, and names containing newlines survive parsing). Pinned by a new test committing a non-ASCII path under a checkpoint trailer.

Full suite re-run on the stack: 9,324 unit / 514 integration / canary green.

[jdx]

Rebased onto current main (was 33 behind, including #2107 which touches the same in-gate region) and addressed all six findings from the two review runs that landed after the approval. Three review: commits.

827a20066 — measure used_search from tool invocations, not substrings (HIGH + the in-gate MEDIUM)

The substring probe measured 18 false positives to 1 true positive across 328 real transcripts, and Entire installs the artifacts that trip it (setup_search_skill.go embeds entire search --json in the skill body; investigate/prompt.go:84 injects it into every investigate prompt).

I did not implement the suggested fix as written — matching a Bash command field would have been worse, in the opposite direction:

1. It would fabricate a negative for most agents. agent.ExtractSkillEvents dispatches through SkillEventExtractor, which has one implementer; the dispatcher returns nil for codex, cursor, copilotcli, geminicli, opencode, factoryaidroid, pi and external without touching the transcript. used_search was an unconditional payload property, so all of them would have reported false. Cursor is unprobeable rather than unimplemented — its transcripts contain no tool_use blocks at all — so per-line shape sniffing doesn't rescue it either.
2. The primary Claude adoption path isn't a Bash call in the transcript being scanned. setup_search_skill.go:83 installs search as a subagent (.claude/agents/entire-search.md), so the intended path records an Agent/Task tool_use with subagent_type: entire-search and runs the command in a separate subagent transcript that condensation never reads. Bash-only reports "did not search" for exactly the sessions that adopted the feature.

So: new built-in-only ToolInvocationScanner capability whose dispatcher returns (found, supported), Claude Code the sole implementer, and both matchers (command-position regex + subagent dispatch).

⚠️ Payload shape changed — worth a second opinion. used_search is now *bool and is omitted when unmeasurable, alongside a new always-present used_search_source (unsupported | none | command | subagent). A consumer filtering used_search = false now excludes unknowns instead of absorbing them, since a missing PostHog property is not false. Free now for the same reason the rename in a433d60e9 was — the event has never shipped and nothing keys on it.

Query shape becomes:

SELECT countIf(properties.prior_ai_history AND NOT properties.used_search) / count()
FROM events
WHERE event = 'cli_commit_condensed'
  AND properties.used_search_source != 'unsupported'

59c5febc3 — probe prior AI history once per commit, and anchor its parse on NUL (the remaining MEDIUM + two LOWs)

priorAICommitTouchedFiles (predicate) → priorAICommitFiles (set), with a commitCondensedEmitter memoizing both the git-log scan and the settings load for one commit. Both halves resolve on first use, so a commit where nothing condenses runs neither settings.Load nor git log, and the gate stays in front of the probe — both pinned by tests rather than left to the reading. s.IsTelemetryEnabled() replaces the fourth hand-rolled copy of the telemetry gate (#2023's helper).

Format is now --format=%x00%H%n%B. -z NUL-terminates the format output itself and a commit message cannot contain NUL, so an empty field is an unambiguous record marker — one that rests on nothing about message content, which was the actual defect. %H guarantees the post-marker field is non-empty so an empty-message commit can't mis-frame the next record.

I verified the old format's failure against git 2.53.0: on a body containing literal \x1e/\x1f, it yields a record whose message is " and ", whose trailer is absent, and whose "file list" is the rest of the message. TestPriorAICommitFiles_ControlCharsInBody pins it.

Rejected %(trailers:key=…) despite it keeping the body out of the pipeline: it needs git ≥ 2.15 and on older git the placeholder expands literally rather than erroring, so prior_ai_history reads false for that whole population with nothing in the data to say why — silent, version-correlated and directional, which is disqualifying under the same standard the HIGH was held to.

Merge commits: deliberately unchanged. --diff-merges=first-parent would make a "Merge origin/main into X" commit attribute every file it merged in, inflating prior_ai_history and therefore the miss rate. A merge's content is already attributed to the individual trailer-carrying commits it brings in, and those are in the 50-commit window on their own account — 51 of the last 300 merges on main carry the trailer, so this is a frequent path, not a rare one. TestPriorAICommitFiles_MergeCommitContributesNothing pins the decision. Push back if you disagree; this half is a judgment call.

e31193697 — document the skill-event ledger's size envelope (LOW)

Comment-only, no cap. Moved to the field that actually costs something — SessionState.SkillEvents in session/state.go — with the measured envelope and why trimming isn't available (the ledger is what makes skill telemetry exactly-once, so a cap re-enables double-reporting for exactly the long sessions it would target). The finding pointed at the transient per-condensation copy, which now carries a pointer to the durable one.

One correction to a finding's premise

The in-gate MEDIUM said to fold the probe into "the pass that already walks tool_use blocks" to remove a traversal. There was no second traversal to remove — extractSessionData already runs CalculateTokenUsage and ExtractSkillEvents in the same gate-held block, each a full ParseFromBytes plus a per-assistant-line unmarshal, beside which the 9.1ms/10MB bytes.Contains scan was a rounding error. What this change buys is precision at comparable in-gate cost, and it does not move work out of MutateSessionState. The commit says so rather than claiming a win that isn't there.

---

mise run check green (lint 0 issues, 56 Vogon + 4 roger-roger canary tests). #2100 rebased on top.

[jdx]

Update — rebased again, and #2101 needed a follow-up

Rebased onto current main (89ebc1ba4), which now includes #2101 and #2034.

#2101 was a semantic conflict, not a textual one. It converted the four machineid.ProtectedID call sites to the memoized telemetryMachineID() and dropped the machineid import from detached.go. This PR adds a fifth builder, BuildCommitCondensedPayload, in a different region of the same file — so git rebased cleanly and the result didn't compile. Converted, and folded into the commit that introduced the call rather than tacked on at the tip, so every commit on the branch still builds (verified by building each of the 8 in turn). Net effect: cli_commit_condensed shares the one-per-process lookup instead of paying its own ~11.6ms ioreg.

Corrected an overstated rejection rationale. The review: probe prior AI history once per commit, and anchor its parse on NUL commit rejected %(trailers:key=Entire-Checkpoint,valueonly) and led with "it needs git ≥ 2.15." That's 2017 and not a real constraint — the version argument was the weak half. The message now leads with the reason that actually decides it and is version-independent: %(trailers:key=…) hands trailer detection to git's parser, which only recognises a trailer block in the message's final paragraph, while trailers.ParseCheckpoint matches anywhere. Reworded commits, hand-edited squash bodies, and the multi-trailer squash-merge case ParseAllCheckpoints exists for would newly be missed, invisibly. The code choice is unchanged; only the argument for it is.

Gained a commit from #2100. docs(telemetry): say why the two commit-less condensation paths emit no signal now lives here. It documents why CondenseSessionByID (doctor repair) and CondenseAndMarkFullyCondensed (session-end leftovers) deliberately emit no commit-condensed signal — the two absences that originally read as data loss to reviewers. It was written on #2100 because that PR moved the surviving emission onto an onSaved callback; #2100 is now based on main and no longer contains this signal at all, so the comments follow the code they describe.

mise run lint 0 issues, 56 Vogon + 4 roger-roger canary tests green.

[jdx]

Rebased onto 5a157342a — one decision worth a look

#2013 (session-identity-linking) landed a guest-linked commit early-return in condenseAndUpdateState, at exactly the spot this PR's signal snapshot occupied, so the rebase needed a real choice rather than a mechanical resolution.

I put the snapshot before the guest branch, so a guest-linked commit does emit. The reasoning: a guest-linked commit is still a commit — the transcript is condensed and the trailer stamped — so files_committed and prior_ai_history describe something real, and --skip=1 still excludes the right HEAD. Skipping it would silently under-count exactly the cross-worktree sessions, which is the failure direction this PR's review pass has been trying to eliminate. The rationale is recorded at the call site, and the guest path's early return carries the signal.

If you'd rather guest-linked commits were excluded, that's a one-line move — but it should be a deliberate metric decision, not a rebase artifact, which is why it's called out here.

Also picked up #2013's condenseAndUpdateState signature change (variadic opts ...condenseOpts → opts condenseOpts).

Base is now main, conflicts resolved, mergeable: MERGEABLE. Lint 0 issues, 56 Vogon + 4 roger-roger canary tests green.

[jdx]

Two more findings — and a correction to something I asserted above

Both fixed in 87e2d6211.

LOW — case-folded matcher behind a case-sensitive prefilter. detectSearchUsage compared SubagentType with strings.EqualFold, so it accepted Entire-Search — but the hint list is applied through case-sensitive bytes.Contains, which discards that line before the matcher ever sees it. A silent false negative, and exactly the hazard ToolInvocationScanner's own doc warns callers about. TestSearchHintsCoverPattern missed it because it pinned only the canonical lowercase constant, which can never catch a case-folding mismatch.

Dropped EqualFold rather than widening the hints: the subagent name is a literal we scaffold ourselves, so matcher and prefilter agreeing on exactness is the property worth having. The test now asserts both halves — that the hint list does not cover the case variants, and that the matcher rejects them — so they can't drift apart again silently.

MEDIUM — I was wrong that Cursor is unprobeable, and I stated it as fact in this PR.

My earlier comment said Cursor "is genuinely unprobeable, not just unimplemented." I took that from cursor/transcript.go's "Cursor transcripts do not contain tool_use blocks" and repeated it in ToolInvocationScanner's doc and the cursor test's justification. I should have checked its provenance instead of its plausibility:

• It dates to 366e4eeb6 (2026-03-03, "copilot feedback") — nearly six months old.
• It's restated in four comments plus AGENT.md, and pinned by no test. cursor_test.go:306 asserts the consequence, not the premise, so nothing would catch it becoming false.
• Meanwhile cursor/transcript.go's own GetTranscriptPosition says Cursor "uses the same JSONL format as Claude Code," and the shared transcript.ContentBlock carries tool_use.

The finding measured six real Cursor transcripts carrying tool_use blocks with name and input (Glob ×4, Write ×3, Read ×1). I could not reproduce that — this machine has no Cursor transcripts and the repo has no Cursor transcript fixtures — so I'm taking it on its specificity plus the stale, untested provenance of the claim it contradicts. Flagging the non-reproduction rather than implying I confirmed it.

Both comments now say "no walker yet" and explicitly warn against reading absence as impossibility.

Cursor still doesn't implement the scanner, but for the narrow true reason rather than the false one: ToolInvocation.Command assumes Claude's command input key, the finding's own sample contains no shell-tool invocation, so Cursor's key is unconfirmed. Implementing against a guessed key would manufacture precisely the silent false negative the (found, supported) split exists to prevent — trading a visible "cannot tell" for an invisible "did not run". The test now names that blocker and asks to be deleted once the mapping is known.

Filed separately, and bigger than this PR: the same stale premise makes cursor.ExtractModifiedFilesFromOffset return nothing for every Cursor session, so file detection falls back to git status while real Write/Read blocks sit unread. That's a file-attribution bug, not a telemetry one, and wants its own change with real fixtures.

Lint 0 issues, 56 Vogon + 4 roger-roger canary tests green. Both findings resolved on the trail.