Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/README.skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,16 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to
| [playwright-explore-website](../skills/playwright-explore-website/SKILL.md)<br />`gh skills install github/awesome-copilot playwright-explore-website` | Website exploration for testing using Playwright MCP | None |
| [playwright-generate-test](../skills/playwright-generate-test/SKILL.md)<br />`gh skills install github/awesome-copilot playwright-generate-test` | Generate a Playwright test based on a scenario using Playwright MCP | None |
| [poka-yoke](../skills/poka-yoke/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke` | Mistake-proof code so misuse cannot be expressed, rather than warning against it. Use when designing an interface, schema, or state machine and the user wants it hard to get wrong ("make invalid states unrepresentable", "so callers cannot screw it up", "type-safe API", "pit of success"); when auditing existing code for footguns ("what could bite us here", "what is easy to misuse", "poka-yoke this repo", "review this diff for ways to get it wrong"); or when a bug has recurred and the fix must close the class rather than the case ("make sure this never happens again", "this is the third time"). Especially for money, auth, permissions, deletion, migrations, and pipelines where failure is silent. Classifies every finding by what happens when the mistake occurs and how the device notices, which is what keeps it from collapsing into generic code review. | `references/hazard-catalog.md`<br />`references/lang-python.md`<br />`references/lang-rust-go.md`<br />`references/lang-typescript.md`<br />`scripts/detect_hazards.py` |
| [poka-yoke-agent-guardrails](../skills/poka-yoke-agent-guardrails/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-agent-guardrails` | Stop an AI agent damaging your repo: PreToolUse hooks, permission deny rules, protected paths, verification gates. Use when "claude keeps force pushing", "CLAUDE.md says X but it still does Y", "stop the agent touching prod or .env", or making a repo safe for unattended agent work. For AI features you ship to users use llm. | `assets/devices/claude-hooks/README.md`<br />`assets/devices/claude-hooks/guard_dangerous_commands.py`<br />`assets/devices/claude-hooks/suggest_poka_yoke.py` |
| [poka-yoke-audit](../skills/poka-yoke-audit/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-audit` | Find footguns in code that already exists: swappable arguments, silent fallbacks, unguarded deletes, signatures that are easy to misuse. Use when someone asks "what could bite us here", "what is easy to misuse", "poka-yoke this repo", or wants a diff or PR reviewed for ways to get it wrong. Ranks by blast radius. For code not yet written use design; for something that already broke use retro. | `references/hazard-catalog.md`<br />`references/lang-python.md`<br />`references/lang-rust-go.md`<br />`references/lang-typescript.md`<br />`scripts/detect_hazards.py`<br />`scripts/device_registry.py` |
| [poka-yoke-authz](../skills/poka-yoke-authz/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-authz` | Multi-tenant isolation, IDOR and row-level security. Use to find every path where one tenant could read or write another tenant data: "we forgot to filter by org_id", "can users see each other data", "audit these endpoints for cross-tenant leaks", "make an unscoped query impossible". Covers scoped repositories, RLS, default-deny routing and the two-tenant test. For what the UI shows use ux. | None |
| [poka-yoke-data](../skills/poka-yoke-data/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-data` | Pipelines, warehouses, dbt models and metrics, where failure is silently wrong numbers rather than a crash. Use when "the dashboard is wrong", "the numbers do not match", "add data quality checks", "safe backfill", or an upstream schema change broke a join. Covers freshness, row-count and null-rate assertions, data contracts, reconciliation. For a crash rather than wrong numbers use audit. | None |
| [poka-yoke-design](../skills/poka-yoke-design/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-design` | Design APIs, schemas, types and state machines so misuse cannot be expressed. Use when writing a new interface and someone asks "what should the types look like", "make invalid states unrepresentable", "so callers cannot screw it up", or wants illegal state transitions rejected. Covers branded types, discriminated unions, typestate, parse-don't-validate. For code that already exists use audit. | `references/hazard-catalog.md` |
| [poka-yoke-guardrails](../skills/poka-yoke-guardrails/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-guardrails` | Pre-commit hooks, CI gates, lint rules, database constraints and branch protection. Use when a rule needs enforcing rather than documenting: "set up enforcement", "unformatted or untyped code must not get merged", "gate this in CI", "we agreed to X and people still do not", "stop secrets getting committed". Covers baselining and ratcheting so existing violations do not block anyone. For constraining an AI agent use agent-guardrails. | `assets/devices/claude-hooks/README.md`<br />`assets/devices/claude-hooks/guard_dangerous_commands.py`<br />`assets/devices/claude-hooks/suggest_poka_yoke.py`<br />`assets/devices/github-actions/poka-yoke-gates.yml`<br />`assets/devices/lint/README.md`<br />`assets/devices/pre-commit/.pre-commit-config.yaml` |
| [poka-yoke-llm](../skills/poka-yoke-llm/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-llm` | AI features you ship to users: structured output, tool schemas, prompt injection, evals. Use when "the model returns bad JSON", "it hallucinates", "stop it calling the wrong tool", "add evals", or an LLM feature can trigger refunds, emails or writes. Covers schema-constrained output, idempotent tool calls, confirmation gates. For agents editing your repo use agent-guardrails. | None |
| [poka-yoke-ops](../skills/poka-yoke-ops/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-ops` | Deploys, schema migrations, rollback and infrastructure. Use when "can I ship this on Friday", "this migration is scary", "what is the blast radius", "prevent accidental deletion of the database", or a change drops a column. Covers expand/contract, canary rollout, kill switches, prevent_destroy, tested backups. For an incident that already happened use retro. | None |
| [poka-yoke-retro](../skills/poka-yoke-retro/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-retro` | Turn a bug, outage or repeated mistake into a device that makes the whole class impossible. Use when something already broke: "make sure this never happens again", "this is the third time", "postmortem", "how did this get through". Root-causes to the missing constraint, then sweeps every other site where the mistake is still available. For a pipeline use data, a deploy use ops, cross-tenant use authz, an AI feature use llm. | `scripts/detect_hazards.py` |
| [poka-yoke-ux](../skills/poka-yoke-ux/SKILL.md)<br />`gh skills install github/awesome-copilot poka-yoke-ux` | Forms, destructive actions and flows users get wrong. Use when "users keep deleting the wrong thing", "add a confirmation dialog", "this flow is error-prone", or building a delete, bulk action, checkout or settings page. Covers undo over confirmation, type-to-confirm, safe defaults, input constraints, double-submit. For the server-side rules behind the screen use authz. | `references/hazard-catalog.md`<br />`references/ux-patterns.md` |
| [postgresql-code-review](../skills/postgresql-code-review/SKILL.md)<br />`gh skills install github/awesome-copilot postgresql-code-review` | PostgreSQL-specific code review assistant focusing on PostgreSQL best practices, anti-patterns, and unique quality standards. Covers JSONB operations, array usage, custom types, schema design, function optimization, and PostgreSQL-exclusive security features like Row Level Security (RLS). | None |
| [postgresql-optimization](../skills/postgresql-optimization/SKILL.md)<br />`gh skills install github/awesome-copilot postgresql-optimization` | PostgreSQL-specific development assistant focusing on unique PostgreSQL features, advanced data types, and PostgreSQL-exclusive capabilities. Covers JSONB operations, array types, custom types, range/geometric types, full-text search, window functions, and PostgreSQL extensions ecosystem. | None |
| [power-apps-code-app-scaffold](../skills/power-apps-code-app-scaffold/SKILL.md)<br />`gh skills install github/awesome-copilot power-apps-code-app-scaffold` | Scaffold a complete Power Apps Code App project with PAC CLI setup, SDK integration, and connector configuration | None |
Expand Down
171 changes: 171 additions & 0 deletions skills/poka-yoke-agent-guardrails/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
---
name: poka-yoke-agent-guardrails
description: >-
Stop an AI agent damaging your repo: PreToolUse hooks, permission deny rules, protected paths, verification gates. Use when "claude keeps force pushing", "CLAUDE.md says X but it still does Y", "stop the agent touching prod or .env", or making a repo safe for unattended agent work. For AI features you ship to users use llm.
license: MIT
---

# Poka-Yoke for AI-Written Code

An agent is a fast, tireless operator with no memory of yesterday and a strong prior toward
appearing successful. That is the exact profile Shingo designed poka-yoke for, except an
agent makes mistakes faster than any human, and never learns from the ones you correct in
conversation.

The governing insight: **instructions to an agent are rung zero.** A line in CLAUDE.md saying
"never commit to main" is training, and training degrades, under long contexts, compaction,
and subagents that never read the file. A PreToolUse hook that denies the push is a device. If
you have been repeating the same correction to an agent, that is the signal to stop writing
instructions and install a device.

## A complete answer covers all five

**The diagnosis is not the answer.** "Instructions are not enforcement" is the right insight,
and it is satisfying to write, but someone asking *"what am I doing wrong?"* has a repo they
need to fix: not a question about their prose. Explaining why the rules fail and stopping
there leaves them exactly where they started. State the insight in a sentence, then spend the
rest of the answer on the replacement.

Replacing an instruction with a device is not one step, it is five, and stopping after the
first leaves the person with a rule that looks enforced and is not. Naming the deny rule is
the easy part and the least of it. Cover every one of these, briefly, before adding depth:

1. **The deny rule, with real syntax.** Show the actual `permissions.deny` entry for their
case, `"Bash(git push --force:*)"`: not a description of one. A pattern they have to
invent themselves is a step where this fails.
2. **A hook where a pattern is not enough.** Deny rules match strings. Anything conditional: a `DELETE` without a `WHERE`, an edit allowed in one directory but not another, a
production hostname, needs a `PreToolUse` hook that inspects the call and returns a deny.
Say which of their two rules needs which.
3. **What the deny message says.** The agent reads it and acts on it, so a bare refusal
produces a workaround, often a worse one. The message must name what was blocked, why, and
what to do instead. This is the one place prose belongs in a device.
4. **Where the config lives, so it applies to everyone.** `.claude/settings.json`, committed.
A rule in `settings.local.json` protects one machine, which is the same failure as
documenting it: the protection exists only where someone remembered to set it up.
5. **Proof that it fires.** Run the blocked action and confirm the denial *and* its message,
then run the legitimate neighbouring action and confirm it still works. Untested hooks fail
open more often than people expect: a regex that does not match the real command string is
a hook that does nothing while looking like protection. **An unverified device is worse
than no device, because it creates confidence without protection.**

Steps 3 and 5 are the ones most often dropped, and they are what separate a device that works
from one that merely exists.

## The three failure modes, and the device for each

**1. The agent does something destructive.** Force-push, `rm -rf`, dropping a table, editing
`.env`, running against production, `git checkout .` over uncommitted work, `--no-verify`.
These are irreversible and fast. Device: **deny at the tool boundary**: a hook or permission
rule that refuses the call before it executes. This is Control and it is the only rung that
matters for irreversible actions.

**2. The agent writes code that looks right and isn't.** Plausible-but-wrong is an agent's
characteristic defect: correct-looking imports of things that don't exist, tests that assert
nothing, error handling that swallows, a stub that returns a hardcoded value. Device: **the
type checker and the test suite as required gates**, plus lint rules against silent failure.
Everything in `guardrails` applies here with extra force, because the volume of
generated code is higher and human review attention per line is lower.

**3. The agent reports success it didn't achieve.** "All tests pass" when the suite wasn't
run; "done" with the build broken. Device: **verification the agent cannot fake**: a Stop
hook that actually runs the tests, or a CI gate. Never accept a claim of completion that only
exists as text.

## Devices, strongest first

### Deny rules in settings.json

The cheapest device and the first thing to install. Permission denies are evaluated before the
tool runs and need no scripting:

```jsonc
{
"permissions": {
"deny": [
"Bash(git push --force:*)",
"Bash(git push -f:*)",
"Bash(git commit --no-verify:*)",
"Read(./.env)",
"Read(./.env.*)",
"Edit(./.env)",
"Edit(./migrations/**)",
"Bash(terraform apply:*)"
]
}
}
```

Reading `.env` matters as much as writing it: an agent that reads a secret can echo it into a
log, a commit, or a message to a third-party service. Deny the read.

A deny entry matches the **start** of the command, so it only holds where the dangerous form
is the prefix. That is why `rm -rf` is not on this list: `"Bash(rm -rf /:*)"` would leave
`rm -fr /`, `rm -Rf /` and `cd / && rm -rf *` untouched while looking like coverage.
Recursive delete needs the hook below, see the `rm` pattern in
`assets/devices/claude-hooks/guard_dangerous_commands.py`.

Put team-wide rules in `.claude/settings.json` (committed) and personal ones in
`.claude/settings.local.json` (gitignored), otherwise the rules exist only on the machine of
whoever set them up, which is the same failure as documenting them.

### PreToolUse hooks for anything conditional

When the rule needs logic, "block `DELETE` without a `WHERE`", "block edits to
`schema.prisma` unless a migration exists", "block production hostnames in a connection
string": a hook script inspects the call and returns a deny with a reason.

Templates in `assets/devices/claude-hooks/`. The critical detail:
**the deny message is read by the agent and is your only chance to redirect it.** A bare
"denied" produces a workaround attempt, often a creative and worse one. A message that says
what was blocked, why, and what to do instead produces the right action. Write it as you would
write an error message for a colleague:

> Blocked: `DELETE` without a `WHERE` clause on `users`. Unbounded deletes are irreversible
> here. Add a `WHERE` clause, or if a full truncate is genuinely intended, ask the user to
> confirm and run it themselves.

### Stop hooks that verify completion

Run the type check and the test suite when the agent tries to finish. This converts "tests
pass" from a claim into a fact, and it is the single highest-value hook in most repos.

### Machine-checkable CLAUDE.md

Anything in CLAUDE.md that *can* be a check should be one; what remains should be facts the
agent needs rather than rules you hope it follows.

- "Always run `make fmt` before committing" → a pre-commit hook.
- "Never use `any`" → a lint rule with a required check.
- "Don't edit generated files" → a deny rule, plus a header in the generated files.
- "Use `pnpm`, not `npm`" → a deny on `Bash(npm install:*)` with a message naming `pnpm`.

What legitimately stays as prose: architecture, domain vocabulary, where things live, why
past decisions were made. Facts, not commands.

### Make the safe path the easy path

Agents follow the shortest route to a working answer. If `make test` runs the right thing with
the right env, it gets used; if the correct invocation is a fifteen-flag command documented in
a wiki, it does not. Every ergonomic improvement here is a poka-yoke: a `make check` that
bundles fmt + lint + types + tests, a `.env.example` with every key present, a devcontainer or
a single setup script. Ambiguity is where agents improvise, and improvisation is where damage
comes from.

## A caution about over-restriction

Deny rules that block ordinary work produce an agent that spends its turns fighting the
harness, and a user who turns the rules off. Aim the strong devices at **irreversible and
outward-facing** actions, force-push, prod, secrets, destructive SQL, deletion, publishing, and leave ordinary editing and reading alone. Reversibility is the right axis: git makes most
code changes cheap to undo, so they do not need a gate. A rotated credential and a dropped
table do not.

## Verify each device

Same discipline as any other guardrail, and easy to check here: try the blocked action and
confirm the denial and its message, then confirm the legitimate neighbouring action still
works. Untested hooks fail open surprisingly often: a regex that doesn't match the real
command string is a hook that does nothing while looking like protection.

Leave a `poka-yoke:` marker comment on each rule naming what it prevents, and show the user
each config before writing it. Hooks execute code on their machine on every tool call; that is not a change to
make on someone's behalf unseen.
Loading
Loading