fix: require grep -n evidence for work-on-task-assistant tracking claim - #104
Merged
Conversation
There was a problem hiding this comment.
The review is complete. The diff touches only documentation files (CHANGELOG.md and agents/work-on-task-assistant.md). The changes strengthen the tracking claim validation by requiring grep -n evidence with line numbers rather than just prose quotes — a sound fix that prevents fabricated evidence.
{
"verdict": "approve",
"summary": "Documentation-only change to the work-on-task-assistant agent instructions and CHANGELOG. The fix correctly strengthens Phase 4's daily-note tracking claim by demanding grep -n evidence (file:line-no format) instead of a prose quote — line numbers are spot-checkable and resist fabrication in ways a verbatim quote does not. No Go code was modified.",
"comments": [],
"concerns_addressed": [
"correctness: agents/work-on-task-assistant.md now requires grep -n evidence — line number + matched line — instead of a prose quote that could be fabricated. Verified by reading the diff and the actual file at lines 148-151."
]
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
v0.114.5 added a guard to Phase 4 after
work-on-task-assistantreported a daily-note tracking claim for a task that was not in the note:The same day, with that guard in place, it happened again. During
Plan Week - 2026W34-satthe agent reported:grep -in "plan week" 2026-08-22.mdreturned nothing. The task had no line in the note at all; the entry had to be hand-written later by/vault-cli:sync-progress.Why the guard failed: "quote it verbatim" is satisfiable by fabrication. A well-formed, plausible, nonexistent line reads identically to a real match, so the instruction asks for exactly the artifact a model can produce without looking.
Two misses, same day, same daily note, same shape — the second one after the fix. That is a measured recurrence, not a projected one.
Change
Phase 4's tracking claim now requires evidence that does not survive fabrication:
grep -nthat was actually runℹ️ Already tracked: <file>:<line-no>: <matched line>— the line number is the evidence, spot-checkable by the caller with onesed -nℹ️ Not tracked, and add the lineAlso records both 2026-08-22 occurrences, including that the second happened with the weaker guard already active — that is the argument for the stronger form, and it should survive in the file so the rule is not softened back later.
Verification
make precommit→ PASS (18.4s) — tests, coverage, linting, security scans, license headersNote
Same class as the incident that produced the v0.114.5 guard, found by
/coding:self-improveon the session that hit it.