Skip to content

fix: require grep -n evidence for work-on-task-assistant tracking claim - #104

Merged
bborbe merged 1 commit into
masterfrom
fix/already-tracked-grep-evidence
Aug 22, 2026
Merged

fix: require grep -n evidence for work-on-task-assistant tracking claim#104
bborbe merged 1 commit into
masterfrom
fix/already-tracked-grep-evidence

Conversation

@bborbe

@bborbe bborbe commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Problem

v0.114.5 added a guard to Phase 4 after work-on-task-assistant reported a daily-note tracking claim for a task that was not in the note:

Never report Already tracked without a matching line in hand — quote it verbatim.

The same day, with that guard in place, it happened again. During Plan Week - 2026W34-sat the agent reported:

Daily Note: already tracked as [/] [[Plan Week - 2026W34-sat]] on 60 Periodic Notes/Daily/2026-08-22.md — no change needed.

grep -in "plan week" 2026-08-22.md returned nothing. The task had no line in the note at all; the entry had to be hand-written later by /vault-cli:sync-progress.

Why the guard failed: "quote it verbatim" is satisfiable by fabrication. A well-formed, plausible, nonexistent line reads identically to a real match, so the instruction asks for exactly the artifact a model can produce without looking.

Two misses, same day, same daily note, same shape — the second one after the fix. That is a measured recurrence, not a projected one.

Change

Phase 4's tracking claim now requires evidence that does not survive fabrication:

  • The search MUST be a grep -n that was actually run
  • Report as ℹ️ Already tracked: <file>:<line-no>: <matched line>the line number is the evidence, spot-checkable by the caller with one sed -n
  • No grep hit → ℹ️ Not tracked, and add the line

Also records both 2026-08-22 occurrences, including that the second happened with the weaker guard already active — that is the argument for the stronger form, and it should survive in the file so the rule is not softened back later.

Verification

  • make precommit → PASS (18.4s) — tests, coverage, linting, security scans, license headers
  • Docs-only change to one agent file; no code paths touched

Note

Same class as the incident that produced the v0.114.5 guard, found by /coding:self-improve on the session that hit it.

@ben-s-pull-request-reviewer ben-s-pull-request-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The review is complete. The diff touches only documentation files (CHANGELOG.md and agents/work-on-task-assistant.md). The changes strengthen the tracking claim validation by requiring grep -n evidence with line numbers rather than just prose quotes — a sound fix that prevents fabricated evidence.

{
  "verdict": "approve",
  "summary": "Documentation-only change to the work-on-task-assistant agent instructions and CHANGELOG. The fix correctly strengthens Phase 4's daily-note tracking claim by demanding grep -n evidence (file:line-no format) instead of a prose quote — line numbers are spot-checkable and resist fabrication in ways a verbatim quote does not. No Go code was modified.",
  "comments": [],
  "concerns_addressed": [
    "correctness: agents/work-on-task-assistant.md now requires grep -n evidence — line number + matched line — instead of a prose quote that could be fabricated. Verified by reading the diff and the actual file at lines 148-151."
  ]
}

@bborbe
bborbe merged commit 052a537 into master Aug 22, 2026
1 check passed
@bborbe
bborbe deleted the fix/already-tracked-grep-evidence branch August 22, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant