From 1114f44fe63cbdf87c564f0f4b1feacaa64eb6b0 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 19 Jul 2026 03:11:44 +0000 Subject: [PATCH 1/5] pstack: strengthen blinded eval bakeoffs Co-authored-by: lauren --- pstack/.cursor-plugin/plugin.json | 2 +- pstack/skills/poteto-mode/playbooks/eval.md | 26 ++++++++++++--------- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/pstack/.cursor-plugin/plugin.json b/pstack/.cursor-plugin/plugin.json index 2d9c4a53..b1544741 100644 --- a/pstack/.cursor-plugin/plugin.json +++ b/pstack/.cursor-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "pstack", "displayName": "pstack", - "version": "0.11.3", + "version": "0.11.4", "description": "if you want to go fast, go deep first. pstack helps you write less, but higher quality code. rigorous agent workflows you can parallelize with confidence.", "author": { "name": "Lauren Tan" diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md index fe18dd58..d583417f 100644 --- a/pstack/skills/poteto-mode/playbooks/eval.md +++ b/pstack/skills/poteto-mode/playbooks/eval.md @@ -2,26 +2,30 @@ **You own the experiment design. Plan, blind, run, synthesize.** -Evals test how a change affects agent behavior before promoting it: a new skill variant, a structural change, a prompt tweak. The failure mode is the observer effect. An agent that knows it's being evaluated behaves differently, so candidates must run blind. +Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a change: a new skill variant, a structural change, a prompt tweak. Each trial gets one clean attempt with no feedback or repair. This is not a standing regression suite or a CI merge gate. -**Non-negotiables for blinding:** +**Non-negotiables for blinding and isolation:** - No `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, `candidate`, or `arena` in any directory, file, or prompt the candidate sees. - The candidate prompt looks like an organic user request. State the goal, not the meta. "build me a small todo cli" not "show me how you follow the principles chain". - No chain-eliciting cues. Don't ask the candidate to list which skills, principles, or files they applied; that meta-prompt inflates citation behavior. Ask for design notes generally and grade chain-following from code shape, not self-report. -- Sanitize directory and slug names. Use project-shaped names a user might pick, not labels like `candidate-1` or `agent-a`. +- Sanitize directory, slug, and arm names. Use project-shaped names a user might pick, not labels like `candidate-1`, `agent-a`, `control`, or `skill-off`. - Don't tell the candidate other candidates exist. - The judge can know it's judging but sees outputs by sanitized label only, never by model name. - Comparing two variants: one judge scores both sets in a single pass on one scale, blind to which set each came from. Two judge runs with different prompts don't compare, the calibration drifts. +- Start each trial in a fresh workspace and preferably a new session. Clear prior chat and give it no sibling memory. Never plant prior transcripts, judge notes, or sibling outputs. **Steps:** -1. **Frame.** State what variant is under test and what behavior counts as success. Write the rubric (3-6 concrete criteria) for the judge only. Hold it back from candidates. -2. **Set up sanitized environments.** Per-candidate working dir with the variant in place. Plant any context an organic task would have: a project skeleton, the skills the candidate would naturally read. -3. **Author one organic prompt.** What a user would type. No leakage of what's being measured. -4. **Spawn N parallel candidates** on different models per the **arena** skill's Phase B. Each works in its own sanitized dir; same prompt to each. -5. **Spawn one blinded judge** on a different model family per the **arena** skill's Phase C. Judge sees outputs by sanitized label and the rubric, never a model name. -6. **Verify the chain from transcripts, not self-report.** Read each candidate's local transcript under the active workspace's `agent-transcripts/` directory (the system prompt names this path). Do not glob across `~/.cursor/projects/*/`; that crosses workspace boundaries and reads private chats from unrelated projects. Look at which files each candidate actually opened. Citing a principle is not reading its leaf skill, and reading it is not applying it. Grade chain-following from the files it really read plus the shape of the code, never from the candidate's own claims. -7. **Read every candidate output yourself** end to end. Compare to the judge's verdict. Disagreement means a model is biased or the rubric is ambiguous. Synthesize. +1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition. +2. **Build the ablation arms.** When adding, rewriting, or supplying a skill, compare the variant with that skill absent. Use the prior version when absence would make the task artificial. Give both arms the same organic prompt and project skeleton. The control is just another sanitized label. Promote only if the variant-on arm beats the variant-off arm on the rubric. +3. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. +4. **Set up isolated trials.** Create a fresh per-trial workspace with only the arm's variant and context an organic task would have. Keep the project skeleton identical across arms. A cheap deterministic preflight may confirm setup facts, but it only aids synthesis. Regex or `should_trigger` checks do not judge task success or decide promotion. +5. **Run 2-3 one-shot trials per prompt and arm** through the **arena** skill's Phase B. Match the model mix across arms and run candidates in parallel. No retries, coaching, or repair. When budget binds, prefer 2 trials across fewer models over 1 trial across more models. +6. **Spawn one blinded judge** on a different model family per the **arena** skill's Phase C. In one pass, it scores every output by randomized sanitized label against the same rubric and marks each criterion and output pass or fail. +7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read each candidate's local transcript under the active workspace's `agent-transcripts/` directory (the system prompt names this path). Do not glob across `~/.cursor/projects/*/`; that crosses workspace boundaries and reads private chats from unrelated projects. Use the transcript to verify isolation, see what the candidate read, and explain the output. It is synthesis evidence, not a pass gate. +8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare your read with the judge's verdict. Promote only when the variant-on arm beats the variant-off arm overall without adding false positives. Otherwise reject it. Explain disagreements as judge bias, contamination, or rubric ambiguity. -**Reply:** variant under test, rubric, per-candidate notes, judge's verdict, your synthesis, and a recommendation for whether to promote the variant. +**Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff. + +**Reply:** variant and control, prompt set, rubric, trial pass rates, per-candidate notes, judge's verdict, your synthesis, and the promote-or-reject decision. From b0bb156221bba5fb2f0d52a0fbd50de41b0de8cf Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 18 Jul 2026 21:53:26 -0700 Subject: [PATCH 2/5] pstack: trim eval bakeoff autofix Adopt bugbot autofix with leaner prose: drop arena pick/graft, record per-trial transcript paths, scope skill-absent controls, and author prompts before building arms. --- pstack/skills/poteto-mode/playbooks/eval.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md index d583417f..6044d5d1 100644 --- a/pstack/skills/poteto-mode/playbooks/eval.md +++ b/pstack/skills/poteto-mode/playbooks/eval.md @@ -18,13 +18,13 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a **Steps:** 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition. -2. **Build the ablation arms.** When adding, rewriting, or supplying a skill, compare the variant with that skill absent. Use the prior version when absence would make the task artificial. Give both arms the same organic prompt and project skeleton. The control is just another sanitized label. Promote only if the variant-on arm beats the variant-off arm on the rubric. -3. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. -4. **Set up isolated trials.** Create a fresh per-trial workspace with only the arm's variant and context an organic task would have. Keep the project skeleton identical across arms. A cheap deterministic preflight may confirm setup facts, but it only aids synthesis. Regex or `should_trigger` checks do not judge task success or decide promotion. -5. **Run 2-3 one-shot trials per prompt and arm** through the **arena** skill's Phase B. Match the model mix across arms and run candidates in parallel. No retries, coaching, or repair. When budget binds, prefer 2 trials across fewer models over 1 trial across more models. -6. **Spawn one blinded judge** on a different model family per the **arena** skill's Phase C. In one pass, it scores every output by randomized sanitized label against the same rubric and marks each criterion and output pass or fail. -7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read each candidate's local transcript under the active workspace's `agent-transcripts/` directory (the system prompt names this path). Do not glob across `~/.cursor/projects/*/`; that crosses workspace boundaries and reads private chats from unrelated projects. Use the transcript to verify isolation, see what the candidate read, and explain the output. It is synthesis evidence, not a pass gate. -8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare your read with the judge's verdict. Promote only when the variant-on arm beats the variant-off arm overall without adding false positives. Otherwise reject it. Explain disagreements as judge bias, contamination, or rubric ambiguity. +2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt. +3. **Build comparison arms.** Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the prior skill. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric. +4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, isolate or disable every candidate-visible copy, preflight resolved sources, and fail setup if the skill remains visible. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric. +5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many. +6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring. +7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects//agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate. +8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity. **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff. From 8d53b5c38ba99f54272c8fbbb5b11a53a10953dc Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 18 Jul 2026 22:00:52 -0700 Subject: [PATCH 3/5] pstack: keep skill-absent disables arm-local Snapshot prior skill contents before editing, never plant the ablation target into a skill-absent control, and forbid shared user/plugin disables that would strip the skill from the variant arm. --- pstack/skills/poteto-mode/playbooks/eval.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md index 6044d5d1..87e821ac 100644 --- a/pstack/skills/poteto-mode/playbooks/eval.md +++ b/pstack/skills/poteto-mode/playbooks/eval.md @@ -19,8 +19,8 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition. 2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt. -3. **Build comparison arms.** Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the prior skill. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric. -4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, isolate or disable every candidate-visible copy, preflight resolved sources, and fail setup if the skill remains visible. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric. +3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the snapshotted prior skill. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric. +4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric. 5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many. 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring. 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects//agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate. From 98957c306a008823d85d83dffbf21e4cd9a8df6a Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 18 Jul 2026 22:23:28 -0700 Subject: [PATCH 4/5] pstack: encode smoke lessons in eval bakeoff Prompt hygiene (do not name the rubric tell), harness-fidelity preflight for sticky/mode/description triggers, require two model families when shipping cross-model, keep programmatic checks from deciding promote, and scale the human-read rule for large N. --- pstack/skills/poteto-mode/playbooks/eval.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md index 87e821ac..5c6aafff 100644 --- a/pstack/skills/poteto-mode/playbooks/eval.md +++ b/pstack/skills/poteto-mode/playbooks/eval.md @@ -18,13 +18,13 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a **Steps:** 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition. -2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type, with no leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt. +2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt. 3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the snapshotted prior skill. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric. -4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric. -5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many. -6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring. +4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that the variant reaches the candidate the way production does (reminder in context, description always loaded, and so on). If the harness cannot inject it that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric. +5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many. +6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Programmatic checks may filter obvious fails before the judge; they do not decide promote or reject. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring. 7. **Inspect transcripts after scoring to explain how, not to decide pass or fail.** Read only the recorded transcript for each trial from that workspace's transcript directory (normally `~/.cursor/projects//agent-transcripts/`), using the session or transcript ID from setup. Derive the slug from the recorded workspace path. Do not glob across `~/.cursor/projects/*/` or open unregistered workspaces. Transcripts verify isolation and explain the output. They are not a pass gate. -8. **Read every output yourself** end to end. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity. +8. **Read the outputs yourself.** At small N, read every output end to end. At large N, read every fail plus a stated random sample of passes; silent skim is not enough. Report pass rates by arm and prompt, then compare with the judge. Promote only when the variant beats the control overall without adding false positives. Otherwise reject. Explain disagreements as judge bias, contamination, or rubric ambiguity. **Related:** Shipped skills may keep a separate standing regression pack of 5-20 cases. It is distinct from this bakeoff. From e330ed58f2969a7cc051d6f441e97af481906baa Mon Sep 17 00:00:00 2001 From: lauren Date: Sat, 18 Jul 2026 22:31:51 -0700 Subject: [PATCH 5/5] pstack: forbid skill-forcing prompts; require absent arm Candidate prompts must not name or path a skill. Skill-body bakeoffs run prior and skill-absent controls, not prior alone. --- pstack/skills/poteto-mode/playbooks/eval.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pstack/skills/poteto-mode/playbooks/eval.md b/pstack/skills/poteto-mode/playbooks/eval.md index 5c6aafff..aa0fa2ea 100644 --- a/pstack/skills/poteto-mode/playbooks/eval.md +++ b/pstack/skills/poteto-mode/playbooks/eval.md @@ -8,6 +8,7 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a - No `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, `candidate`, or `arena` in any directory, file, or prompt the candidate sees. - The candidate prompt looks like an organic user request. State the goal, not the meta. "build me a small todo cli" not "show me how you follow the principles chain". +- Candidate prompts must not name a skill, give a skill path, or say to apply/use/follow a skill. Plant skills in the workspace; the candidate discovers them. - No chain-eliciting cues. Don't ask the candidate to list which skills, principles, or files they applied; that meta-prompt inflates citation behavior. Ask for design notes generally and grade chain-following from code shape, not self-report. - Sanitize directory, slug, and arm names. Use project-shaped names a user might pick, not labels like `candidate-1`, `agent-a`, `control`, or `skill-off`. - Don't tell the candidate other candidates exist. @@ -18,8 +19,8 @@ Evals are blinded, one-shot bakeoffs for deciding whether to promote or reject a **Steps:** 1. **Frame.** State the variant and the promote-or-reject claim. Write a judge-only rubric with 3-6 concrete criteria. Grade task success and the intended behavioral shape. Never make a turn-1 skill load, a particular file read, a citation, or "did the skill trigger?" a pass condition. -2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt. -3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, the control may be skill-absent when absence is the realistic baseline, otherwise the snapshotted prior skill. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. The control is another sanitized label. Promote only when the variant beats the control on the rubric. +2. **Author an organic prompt set.** Include at least one task where the behavior should apply. If the variant changes a description, routing, sticky behavior, or when-to-apply rule, include at least one task where it should not engage and add false-positive cost to the rubric. Write what a user would type. Never name the behavioral tell the rubric grades, and never name or path a skill (if you measure dated headings, do not say "dated note" in the prompt). No other leakage of what is measured. If the task prompt itself is the target, write matched current and proposed versions here; otherwise every arm gets the same prompt. +3. **Build comparison arms.** Before editing, snapshot any prior skill contents the control will need. Variant gets the proposed skill, structure, or prompt. Control gets the current version. For skill presence or content changes, run both a prior-version control and a skill-absent arm unless absence is impossible. Never plant the ablation-target skill into a skill-absent control. Hold the project skeleton, model mix, and every non-target input constant. Controls are other sanitized labels. Promote only when the variant beats the prior control on the rubric without looking worse than absent on false positives. 4. **Set up isolated trials.** Fresh per-trial workspace with only that arm's variant and organic-task context. Identical project skeleton across arms. A fresh workspace does not clear skills from workspace `.cursor/skills/`, user `~/.cursor/skills/`, or plugin installs: for a skill-absent arm, use a workspace-local isolation (or disable that is restored before any other arm runs). Never apply a shared user/plugin disable that also strips the skill from the variant arm. Preflight resolved sources and fail setup if the skill remains visible on a skill-absent arm or missing on a variant arm. For sticky, mode, description, or other always-on triggers, preflight that the variant reaches the candidate the way production does (reminder in context, description always loaded, and so on). If the harness cannot inject it that way, stop: the bakeoff is invalid for that variant class. Record each trial's workspace path and transcript ID as orchestrator-only metadata. Cheap deterministic preflights aid synthesis only; they never replace the blinded rubric. 5. **Run 2-3 one-shot trials per prompt and arm.** Launch each runner directly in its recorded workspace with that arm's isolated context. Fan out in parallel with no shared grounding and no candidate-visible files across workspaces. Match model and trial pairings across arms. If the skill ships across models, use at least two model families; matched pairings on one family are not enough. Ask only for the organic task output, not a graft rationale. Missing output fails the trial. No retries, coaching, or repair. When budget binds, prefer 2 trials on fewer models over 1 on many. 6. **Spawn one blinded judge** on a different model family after every trial finishes. In one pass, score every output by randomized sanitized label against the same rubric. Mark each criterion and output pass or fail. Programmatic checks may filter obvious fails before the judge; they do not decide promote or reject. Do not run the arena pick/graft workflow. This bakeoff ends at arm-level scoring.