Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .agents/logs/afdocs_audit_runs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# AFDocs audit run log

Written by the `afdocs-audit` skill on every scheduled run — clean, regressed, or blocked.

This log serves two purposes:

1. **Regression baseline.** The skill compares each run's score and failing-check set against the most recent entry marked `valid`. Entries marked `blocked` are skipped for comparison: a Vercel Firewall challenge makes every check a false positive, so its score is an artifact rather than a measurement.
2. **Proof a quiet run happened.** The skill only posts to Slack on a regression or a blocked audit, so a clean run is silent. This log is what distinguishes "ran, nothing to report" from "did not run."

Newest entries first. Prepend, do not append.

Entry format:

```markdown
## YYYY-MM-DD — [valid | blocked]
- **Score**: N/100 (grade)
- **Checks**: N total — N pass, N fail, N warn
- **Failing check ids**: comma-separated list, or "none"
- **Allowlisted**: N
- **Oz run**: [URL]
- **Notes**: [anything unusual]
```

For a `blocked` run, omit the score line rather than recording the meaningless value.

---
62 changes: 56 additions & 6 deletions .agents/references/skill-authoring-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,31 @@ Reviewers should merge the log PR periodically so entries reach `main` and becom

**Keep the log branch separate from content PRs.** Never write log updates and skill/content edits in the same commit or branch.

### One standing PR per automation

**The second biggest failure mode: PR stacking.** A skill that mints a new date-suffixed branch on every run accumulates one open PR per run. Because recurring skills tend to edit the same small set of files, those PRs conflict with each other and none of them can be merged cleanly. The `improve-drafting-skills` agent produced four mutually-conflicting open PRs in six days this way, every one of them editing `draft_docs/SKILL.md`.

The log-branch pattern above already solves this for logs. Generalize it to content PRs: **an automation has at most one open PR at any time.**

**Required pattern for all PR-opening skills:**

1. **Use a stable branch name with no date suffix** — `docs/<skill-name>`, not `docs/<skill-name>-2026-08-06`.
2. **Use a stable PR title with no date.** The run date belongs in a dated section of the PR body, not the title. A date in the title defeats title-based lookup and guarantees a new PR on every run.
3. **Look before creating:**
```bash
gh pr list --repo warpdotdev/docs --state open \
--search '<stable title> in:title' --json number,headRefName
```
4. **If an open PR exists**, add to it rather than opening another:
- Check out its branch and rebase on the latest `origin/main`.
- Apply this run's edits, commit, and push.
- Append a new dated section to the PR body. Fetch the current body and make a minimal additive edit — never regenerate it wholesale (see "Outer loop PR body integrity").
- Re-run `check_pr_body.py` after the edit.
5. **If no open PR exists**, create the stable branch from the latest `origin/main` and open a draft PR.
6. **Never leave two open PRs for the same automation.** If a stale or superseded one is found, close it with an explanatory comment before opening a replacement.

This keeps a run's work reviewable without letting unreviewed work pile up, and it means a missed review cycle costs one stale PR rather than one per run.

### Verifying log writes explicitly

Agents often proceed past a failed file write without noticing. For any log update step, verify explicitly:
Expand Down Expand Up @@ -117,7 +142,28 @@ Always read Slack tokens and other secrets from environment variables — never

### Slack notifications

Post a Slack notification on every run, including no-action runs and runs that exited early because a source signal was unavailable. A missing notification on a no-action run is indistinguishable from a run that silently failed. Use a simple text message (not Block Kit) that can be scanned in under 30 seconds.
**Post only when the run produced something a human needs to act on.** Recurring agents that post unconditionally train the channel to ignore them, which costs more than a missed notification does.

Actionable means one of:

- A PR was created or received new commits.
- A threshold was crossed (broken links found, significant 404 gaps, a score regression).
- The agent hit a failure or an early exit that stopped it from completing its job — including a missing or expired credential, an unavailable source signal, a stale-snapshot exit, or a blocked audit. These are not no-ops, and they always post.

Everything else is silent. A no-change or no-op run records its outcome and posts nothing.

**Silence means "ran, nothing to do."** An earlier version of this guidance required posting on every run, reasoning that a silent no-action run is indistinguishable from a run that silently failed. That concern is real, but Slack is the wrong place to solve it: Oz lifecycle events surface failed and errored runs directly, and every scheduled run leaves an inspectable run record on the Runs page.

The corollary is a requirement, not a nicety: **a skill may only be silent when the run leaves a durable record of its outcome.** One of these must be true:

- The skill writes a run log entry on every run, including no-ops. Prefer this for any skill whose history is read by an outer loop — without it, the outer loop cannot tell a quiet period from a broken one.
- Or the run writes an explicit outcome line to run output stating what it checked and why it took no action. This is sufficient for a short-circuit exit that happens before the skill does any work, such as a schedule guard.

A skill that can exit without producing either must post instead.

**Never post twice for one run.** If a skill has multiple phases, fold the later phase's results into the single message rather than posting a follow-up.

Use a simple text message (not Block Kit) that can be scanned in under 30 seconds.

---

Expand All @@ -133,15 +179,19 @@ This minimum must be stated explicitly in the skill's `## Schedule` section so t

### Log availability

The outer loop reads the inner loop's log from `main`. For entries to be available, the inner loop's standing log PR must be merged into `main` before the outer loop runs. Document this as a prerequisite:
**Read the log from the log branch, not from `main`.** The inner loop writes every entry to `chore/<inner-loop>-log` and only reaches `main` when a human merges the standing PR. An outer loop that reads `main` therefore sees a truncated history whose staleness depends on review cadence — and silently analyzes fewer entries than it thinks it has.

```markdown
## Prerequisites
Read from the branch, which always holds the complete history:

- The standing log PR (`chore: <inner-loop> run log`) merged into `main` so the entries are present there.
If it is unmerged, merge it first (or read the log from the `chore/<inner-loop>-log` branch) before analyzing.
```bash
git fetch origin chore/<inner-loop>-log
git checkout origin/chore/<inner-loop>-log -- .agents/logs/<log-file>.md
```

Treat `main` as the convenience case only — if the PR happens to have been merged, the branch and `main` agree, and the branch read is still correct.

**Do not make merging the standing log PR a step in the outer loop.** Merging is a human housekeeping task, not a precondition for analysis. An outer loop that tries to merge its own input couples the run to a repo write it may not have permission to perform, and turns an unmerged PR into a hard failure instead of a non-event.

### Security boundary for signal logs

Outer loops read logs that contain untrusted content: human review comments, PR descriptions, run output from external contributors. Apply these rules before using any log content to propose skill edits:
Expand Down
21 changes: 11 additions & 10 deletions .agents/skills/aeo_crosslink_audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ If Google Search Console data is unavailable, say what could not be verified and
4. **Make only safe edits.** Add links with minimal surrounding copy changes. Preserve the existing page structure and voice. Follow the link quality rules below when choosing anchor text and surrounding context.
5. **Run self-review.** Apply the quality gates in this skill before opening a PR or writing a no-change report.
6. **Deduplicate, re-validate, then open a PR or report no changes.**
- **Deduplicate first.** Check for an existing open AEO cross-link PR before opening one: `gh pr list --repo warpdotdev/docs --search 'docs: add AEO cross-links in:title' --state open`. Never leave two open AEO cross-link PRs. If one already exists, either skip this run (note it in the run output) or, if the existing PR is stale or superseded, close it with an explanatory comment before opening the new one.
- **Deduplicate first.** This skill follows the "One standing PR per automation" contract in `.agents/references/skill-authoring-guidelines.md`. Check for an existing open AEO cross-link PR before opening one: `gh pr list --repo warpdotdev/docs --search 'docs: add AEO cross-links in:title' --state open`. Never leave two open AEO cross-link PRs. If one already exists, prefer adding this run's links to it — check out its branch, rebase on the latest `origin/main`, apply the new links, push, and append them to the existing PR body under its existing headings. Skip the run only when the new links duplicate what the open PR already proposes.
- **Re-validate against the latest `main`.** Fetch `origin/main` and confirm every edited file still exists at its path and every link target resolves to a current page (see "Self-review before opening a PR"). If a restructure moved your targets, rebase onto the latest `main` and fix paths before opening.
- **Open a PR** only when there are at least 2 high-confidence link additions (at least 3 for low-signal runs; see "Source data"). Otherwise, write a no-change report in the Oz run output.

Expand All @@ -91,7 +91,9 @@ If Google Search Console data is unavailable, say what could not be verified and

This produces one perpetual, low-noise PR that accumulates every run's entry regardless of outcome. Reviewers merge it periodically (at minimum before each monthly `improve-aeo-crosslink-skill` run) so the log reaches `main`. If any git step fails, write the log entry to the run output instead and continue to step 8.

8. **Post Slack notification.** After writing the log entry, post the formatted message to `#growth-docs` using the Python snippet below. Python is preferred over curl because it reads `BUZZ_SLACK_TOKEN` from the environment (keeping the token out of process argv) and JSON-encodes the payload correctly regardless of newlines or special characters. If either secret is unavailable, write the notification body to the run output instead.
8. **Post Slack notification — only if there is something to act on.** Follow the actionable-only rule in `.agents/references/skill-authoring-guidelines.md`: post **only** when a cross-link PR was opened or updated this run, or when the run failed or exited early in a way that stopped it from completing (including an unavailable Peec credential). A no-change run is silent — the run log entry from step 7 is its record.

When the run is actionable, post the formatted message to `#growth-docs` using the Python snippet below. Python is preferred over curl because it reads `BUZZ_SLACK_TOKEN` from the environment (keeping the token out of process argv) and JSON-encodes the payload correctly regardless of newlines or special characters. If either secret is unavailable, write the notification body to the run output instead.

```bash
python3 - <<'SLACK_EOF'
Expand Down Expand Up @@ -229,7 +231,7 @@ Use this format:
- [One specific improvement for the next run.]
```

No-change reports stay in the Oz run output. The Oz run link is posted automatically to `#growth-docs` as part of step 8.
No-change reports stay in the Oz run output and are recorded in the run log from step 7. They are **not** posted to Slack — see the notification rules in step 8 and the actionable-only rule in `.agents/references/skill-authoring-guidelines.md`.

## Human review expectations

Expand Down Expand Up @@ -261,27 +263,26 @@ Keep each entry to 7 fields and under 10 lines. Do not add narrative prose.

Use a simple text message (not Block Kit). The message should be scannable in under 30 seconds.

**PR opened:**
**PR opened or updated:**

```
✅ AEO crosslink audit · YYYY-MM-DD
PR opened: [PR URL]
PR [opened | updated]: [PR URL]
Links added: [N links] across [N pages]: [page names]
Signals: [Peec | GSC | Peec + GSC]
Oz run: [run URL]
```

**No change:**
**Run blocked by a failure:**

```
ℹ️ AEO crosslink audit · YYYY-MM-DD — No changes
Checked: agents, cloud agents, and orchestration docs
No PR: [brief reason — e.g., "fewer than 2 high-confidence opportunities"]
⚠️ AEO crosslink audit · YYYY-MM-DD — run blocked
What failed: [brief reason — e.g., "docs repo checkout unavailable"]
Oz run: [run URL]
```

Rules:
- Post on every run, including no-change runs.
- Post only when a PR was opened or updated, or when the run was blocked by a failure. A no-change run posts nothing — its record is the run log entry.
- Never include raw secret values, personal access tokens, or credential file paths in the Slack message.
- Build the `Oz run` link at runtime — never hard-code the Oz host (for example `app.warp.dev` or `oz.warp.dev`). This agent may run on staging or production, and a hard-coded host resolves to the wrong environment (or a generic Runs page). Resolve the environment-correct link from your current run with `oz run get "<your run ID>" --output-format json | jq -r '.session_link'`, substituting the run ID this agent is executing as. Cloud sandboxes ship the `oz` CLI; `oz-dev` is a local development build and is not present, so do not call it.
- If the Oz run URL is unavailable, omit that line rather than posting a broken link.
Expand Down
74 changes: 66 additions & 8 deletions .agents/skills/afdocs-audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,29 +110,87 @@ AFDocs audit complete: 23 checks run, score 82/100 (B).

After reporting, ask the user which issues they want to address.

## Slack notification (optional)
## Run log

Write a run log entry on **every** scheduled run — clean, regressed, or blocked. The log is what makes the regression comparison possible and what makes a silent run distinguishable from a broken one.

Use the standing log-branch pattern from `.agents/references/skill-authoring-guidelines.md`:

1. Fetch and check out `chore/afdocs-audit-log`. Create it from the latest `origin/main` if it does not exist.
2. Prepend the entry to `.agents/logs/afdocs_audit_runs.md`.
3. Verify the write with `head -10 .agents/logs/afdocs_audit_runs.md` before committing.
4. Stage only the log file and commit:
```text
chore: log afdocs audit run YYYY-MM-DD
```
5. Push and verify with `git log --oneline -1 origin/chore/afdocs-audit-log`.
6. Ensure exactly one open PR exists from `chore/afdocs-audit-log` into `main`, titled `chore: afdocs audit run log`.

If any git step fails, write the entry to the run output and continue.

### Run log format

```markdown
## YYYY-MM-DD — [valid | blocked]
- **Score**: N/100 (grade)
- **Checks**: N total — N pass, N fail, N warn
- **Failing check ids**: comma-separated list, or "none"
- **Allowlisted**: N
- **Oz run**: [URL]
- **Notes**: [anything unusual]
```

For a firewall-blocked run, record `blocked`, omit the score entirely rather than logging the meaningless one, and note the mitigation status.

## Regression detection

Compare this run against the most recent **valid** entry in the run log — never against a `blocked` entry, whose score is an artifact of the firewall challenge rather than a real measurement. If there is no prior valid entry, this run establishes the baseline: log it and post nothing.

A run is a regression when either:
- The score dropped versus the last valid entry.
- A check id appears in this run's failing set that was not in the last valid entry's failing set.

## Slack notification

Post **only** when the run is actionable, per the actionable-only rule in `.agents/references/skill-authoring-guidelines.md`:

If instructed to send a report to Slack, post a summary after the audit completes.
- **Regression detected** — post the summary below.
- **Audit blocked** by the Vercel Firewall challenge — post the blocked notice. This is a failure, so it always posts. Never post a score for a blocked run; every check is a false positive.
- **Clean run with no regression** — post nothing. The run log entry is the record.

A first-ever run with no baseline posts nothing.

1. Check if `BUZZ_SLACK_TOKEN` environment variable exists.
2. If the token exists, send a summary to the channel the user specified (or the channel configured in the agent's instructions).
2. If the token exists, send the summary to the channel the user specified (or the channel configured in the agent's instructions).

**Format:**
**Format — regression:**

```
*AFDocs Audit — <date>*
Score: <score>/100 (<grade>) | <total_checks> checks | <pass> pass, <fail> fail, <warn> warn
*AFDocs Audit — <date>* — regression
Score: <score>/100 (<grade>), down from <previous_score>/100 on <previous_date>
<total_checks> checks | <pass> pass, <fail> fail, <warn> warn

*Failures (<count>):*
*New failures since last valid run (<count>):*
• <check_id>: <message>

*Warnings (<count>):*
*Pre-existing failures (<count>):*
• <check_id>: <message>

*Allowlisted (<count>):*
• <check_id>: <reason>
```

Lead with what changed. Pre-existing failures are context, not news — keep that list short or omit it when long.

**Format — audit blocked:**

```
*AFDocs Audit — <date>* — audit blocked, no score
The crawler was blocked by the Vercel Firewall bot challenge, so no checks could run.
Fix: disable Attack Mode, switch Bot Protection to log mode, or add a WAF bypass for the runner.
Details: references/vercel-firewall-challenge.md
```

Send using:

```bash
Expand Down
27 changes: 21 additions & 6 deletions .agents/skills/afdocs-fix/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,14 +151,29 @@ These checks require infrastructure or design changes that can't be automated:

## Applying fixes

1. Create a branch: `git checkout -b afdocs-fixes origin/main`
2. Apply the fixes for each failing check (skip allowlisted checks).
3. Validate: `npm run build` (the build must succeed).
4. Commit with the prefix: `AFDocs fixes: <summary of what was fixed>`
5. Open a PR: `gh pr create`
This skill maintains **one** long-lived fixes PR rather than one per run — see "One standing PR per automation" in `.agents/references/skill-authoring-guidelines.md`.

1. Look for an existing open PR before creating a branch:
```bash
gh pr list --repo warpdotdev/docs --state open \
--search 'AFDocs fixes in:title' --json number,headRefName
```
2. Check out the standing branch. If the PR exists, continue on its branch and rebase; otherwise create it from `main`:
```bash
git fetch origin
git checkout afdocs-fixes 2>/dev/null || git checkout -b afdocs-fixes origin/main
git rebase origin/main
```
3. Apply the fixes for each failing check (skip allowlisted checks). If a fix on the existing branch already addresses a check that is still failing, do not duplicate it — the audit may have run before the PR merged.
4. Validate: `npm run build` (the build must succeed).
5. Commit with the prefix: `AFDocs fixes: <summary of what was fixed>`
6. Push. If the PR already exists the push updates it; otherwise open one with `gh pr create`.

Never leave two open AFDocs PRs. If you find more than one, consolidate onto `afdocs-fixes` and close the extras with a comment pointing at the survivor.

## PR conventions

- Title must be prefixed with `AFDocs fixes:` (e.g., `AFDocs fixes: add llms.txt directive and content negotiation middleware`)
- Title must be prefixed with `AFDocs fixes:` and must not contain a date — a dated title defeats the title search in step 1 and produces a new PR every run
- Include the audit score (before/after if known) in the PR description
- When updating an existing PR, append the new run's score and fixes under the existing headings rather than adding duplicate headings, which `check_pr_body.py` rejects
- Include the co-author line: `Co-Authored-By: Oz <oz-agent@warp.dev>`
Loading
Loading