Skip to content

docs(skills): reduce automation noise and consolidate drafting improvements - #487

Open
rachaelrenk wants to merge 3 commits into
mainfrom
rrenk/automation-tuning
Open

docs(skills): reduce automation noise and consolidate drafting improvements#487
rachaelrenk wants to merge 3 commits into
mainfrom
rrenk/automation-tuning

Conversation

@rachaelrenk

@rachaelrenk rachaelrenk commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Recurring docs agents were generating more PRs and Slack messages than the team could absorb. This fixes the three systemic causes, repairs a batch of GitBook-era migration artifacts found along the way, and consolidates four conflicting stacked PRs into this one.

The trigger was improve-drafting-skills opening a new PR and posting to #growth-docs two days running. The root cause turned out not to be the skill's logic.

Root cause: the cron was wrong

The schedule used 0 17 1-7 * 1, documented as "first Monday of each month." That is not what cron does. When an expression restricts both day-of-month and day-of-week, the two fields are ORed, so it fired on every day of the 1st–7th and every Monday — roughly 11 times a month. Confirmed by the schedule's own data: last_ran Thursday 2026-08-06, next_run Friday 2026-08-07.

Standard cron cannot express "first Monday," so the fix is 0 17 * * 1 plus an in-skill first-week guard. The same broken expression was also present, undeployed, in improve-aeo-crosslink-skill and improve-404-monitor-skill; both are fixed here before they could reproduce it.

Changes

Shared conventions — .agents/references/skill-authoring-guidelines.md

Two existing rules in this file were the direct cause of the problems, not drift. Skills were following them correctly.

  • New: "One standing PR per automation" - Stable branch and title, look before creating, add to the existing PR instead of opening another. This generalizes the log-branch pattern the repo already used successfully.
  • Inverted: "Slack notifications" - Previously required posting on every run including no-ops. Now actionable-only. The old rule's rationale (a silent no-op is indistinguishable from a silent failure) is real and preserved: silence is only permitted for skills that write a run log, and genuine failures still post.
  • Rewritten: "Log availability" - Previously told outer loops to read the log from main, which meant analyzing a truncated history whose staleness depended on review cadence. Now they read the log branch, and no agent merges the standing log PR as a workflow step.

PR reuse

Applied to improve-drafting-skills, weekly-404-monitor, afdocs-fix, sync-error-docs, sync_terminology, sync-openapi-spec, improve-aeo-crosslink-skill, and improve-404-monitor-skill. aeo_crosslink_audit already had the guard and now points at the shared reference.

update-changelog is deliberately excluded from the standing-PR rule — one PR per release is correct there. It instead gained stacking detection, since consecutive release PRs all insert at the top of the same changelog file and would otherwise conflict or merge out of order.

Slack volume

  • aeo_crosslink_audit - no longer posts on no-change runs.
  • weekly-404-monitor - gates on the reporting threshold, and folds its Phase 2 redirect results into the single Phase 1 message rather than posting twice per run.
  • afdocs-audit - posts only on a score regression or a firewall-blocked audit. This needed a baseline, so it gets a new run log (.agents/logs/afdocs_audit_runs.md) that compares against the most recent valid entry, so a blocked run never registers as a regression.

GitBook migration artifacts

A find-and-replace during the migration substituted descriptions into file paths, leaving instructions pointing at files that do not exist.

  • sync-error-docs was the worst: it treated astro.config.mjs (sidebar config) and vercel.json (redirects) as literal paths in eight places, contained a grep that would be a shell syntax error, and still called the GitBook API via GITBOOK_TOKEN. It could not have succeeded as written. Rewritten against verified reality — sidebar is src/sidebar.ts with bare slug strings, redirects are JSON in vercel.json, and the /errors/:code catch-all already covers the prefix case so no per-code GitBook call is needed.
  • .warp/references/terminology.md is an empty leftover directory; the real file is .agents/references/terminology.md. Four skills pointed at the dead path, which meant sync_terminology would git add a nonexistent file. Fixed in sync_terminology, answer_question, triage-issue-local, and create_pr.
  • Smaller path corrections in draft_docs, check_for_broken_links, create_pr, and review-docs-pr.

Consolidated stacked PRs

Supersedes #450, #454, #468, and #484. All four edited draft_docs/SKILL.md and conflicted with each other. Overlapping patterns were merged into single coherent rules rather than appended as four blocks — "Related pages required" appeared in both #454 and #468, and #450's screenshot guidance was folded into the existing screenshot rule instead of competing with it. #468's frontmatter-description edits were dropped entirely: main has since moved past them.

Verification

  • style_lint.py --changed — 0 issues (no content files touched).
  • check_links.py --internal-only — 363 files, 3449 internal links, 0 broken.
  • YAML frontmatter parses on all 17 changed SKILL.md files.
  • Balanced code fences across all 22 changed markdown files.
  • No unintended non-ASCII on any added line.
  • git diff --check clean.
  • First-week guard boundary-tested (days 01/07 run, 08/09/15/31 skip), including the leading-zero case.
  • The accumulating standing-PR body format was tested against check_pr_body.py across a simulated two-run body.

Notes for reviewers

Two things worth a second opinion:

  • check_pr_body.py compatibility. It asserts each required heading appears exactly once and rejects duplicate headings. A standing PR whose runs each append their own ## Patterns addressed would fail on the second run and silently block every future update. The prescribed body format therefore keeps headings fixed and appends dated bullets underneath. Worth confirming that reads well after several months of accumulation.
  • update-changelog exemption. I judged one-PR-per-release correct there rather than forcing the standing-PR rule. If you would rather it also use a standing PR, that is a small change.

Unverified claims

None — the sidebar location, redirect format and catch-all, platformerrors.go path, error page directory, and archived state of warpdotdev/gitbook were each verified directly against the repos before being written down.

Follow-ups (Oz-side, not in this PR)

Schedule changes cannot be made through a repo change. Full list is in the plan; the load-bearing ones:

  • Update schedule LziMazxr8KQWQGL4lz14NC cron from 0 17 1-7 * 1 to 0 17 * * 1.
  • Delete the six schedules still targeting the archived warpdotdev/gitbook repo (~17 runs/week against a dead repo).
  • Create a weekly check_for_broken_links schedule on warpdotdev/docs to replace the three deleted overlapping ones.
  • Delete obsolete Warp Docs / gitbook environment
  • Create a weekly sync-error-docs schedule — only after this merges and a dry run passes.

Update

Merged latest main (conflicts resolved) and addressed the review:

  • Error redirects now add both slash variants. All 17 codes in vercel.json carry both forms, so documenting one would have left new codes half-covered.
  • Refined the silence rule. Outer loops have no run log, and improve-drafting-skills wrongly claimed its signal log covered guard-skipped runs. Silence now requires a durable record of the outcome — a run log entry (still required where an outer loop reads the history) or an explicit outcome line for a pre-work short-circuit.
  • Declined one comment: the catch-all grep is correct, since both /errors/:code and /errors/:code/ exist. Explained on the thread.
  • Aligned the three outer loops to BUZZ_SLACK_TOKEN, matching every other #growth-docs poster.

Co-Authored-By: Warp Agent agent@warp.dev

…ements

Recurring docs agents were producing more PRs and Slack messages than the
team could absorb. Three systemic causes, plus a batch of GitBook-era
migration artifacts that left several skills unable to run as written.

Shared conventions (skill-authoring-guidelines.md):
- Add "One standing PR per automation": stable branch and title, look
  before creating, add to the existing PR rather than opening another.
- Invert "Slack notifications" to actionable-only. The old rule required
  posting on every run and was the direct cause of the channel noise. Its
  silent-failure rationale is preserved by requiring a run log instead.
- Rewrite "Log availability" so outer loops read the log branch rather
  than main, and never merge the standing log PR as a workflow step.

Cron correctness:
- `0 17 1-7 * 1` is not "first Monday". Cron ORs day-of-month with
  day-of-week, so it fired ~11 times a month and produced four
  conflicting PRs in six days. Replace with `0 17 * * 1` plus an
  in-skill first-week guard in improve-drafting-skills,
  improve-aeo-crosslink-skill, and improve-404-monitor-skill.

PR reuse applied to: improve-drafting-skills, weekly-404-monitor,
afdocs-fix, sync-error-docs, sync_terminology, sync-openapi-spec,
improve-aeo-crosslink-skill, improve-404-monitor-skill. update-changelog
keeps one PR per release (correct) but now detects stacked release PRs.

Slack volume: aeo_crosslink_audit no longer posts on no-change runs;
weekly-404-monitor gates on threshold and folds its Phase 2 results into
a single message instead of two; afdocs-audit posts only on regression or
a blocked audit, backed by a new run log for the baseline.

Migration artifacts: a find-and-replace during the GitBook-to-Astro move
substituted descriptions into file paths. sync-error-docs referenced
`astro.config.mjs (sidebar config)` and `vercel.json (redirects)` as real
paths, had an invalid grep, and still called the GitBook API - it could
not have succeeded. Also corrected the sidebar location to src/sidebar.ts,
dropped the GITBOOK_TOKEN dependency, and fixed dead
`.warp/references/terminology.md` paths in four skills.

Consolidates PRs #450, #454, #468, and #484, which all edited
draft_docs/SKILL.md and conflicted with each other. Overlapping patterns
were merged rather than stacked, and PR #468's frontmatter-description
edits were dropped as already superseded on main.

Co-Authored-By: Warp Agent <agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Aug 6, 2026
@vercel

vercel Bot commented Aug 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 6, 2026 7:36pm

Request Review

@oz-for-oss

oz-for-oss Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@rachaelrenk

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

@oz-for-oss oz-for-oss Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This PR consolidates recurring docs automation guidance, adds standing-PR conventions, reduces Slack notifications, and updates GitBook-era path and redirect instructions across docs skills.

Concerns

  • sync-error-docs now documents error redirect handling that does not match the current vercel.json shape: existing error redirects are maintained in both slash and no-slash source variants, and the new catch-all verification checks for the wrong source string.
  • The actionable-only Slack policy is applied to outer-loop skipped/no-action runs that do not write their own run log, which contradicts the new hard requirement that quiet runs must have a per-run log record.

Verdict

Found: 0 critical, 3 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

Comment thread .agents/skills/sync-error-docs/SKILL.md
Comment thread .agents/skills/sync-error-docs/SKILL.md
Comment thread .agents/skills/improve-404-monitor-skill/SKILL.md
@rachaelrenk rachaelrenk self-assigned this Aug 6, 2026
Two conflicts, both in areas this PR deliberately rewrites.

`skill-authoring-guidelines.md` — Slack notifications section. Main reworded
the old post-on-every-run rule; this PR inverts it. Kept the inversion, and
folded main's added case ("runs that exited early because a source signal was
unavailable") into the actionable list, where it belongs as a failure rather
than a no-op.

`aeo_crosslink_audit/SKILL.md` — step 8. Main renamed `SLACK_BOT_TOKEN` to
`BUZZ_SLACK_TOKEN`; this PR rewrote the same paragraph for actionable-only
posting. Kept both: the new posting rule with main's token name. Taking either
side wholesale would have lost one of the two changes.

Also preserved from main across the auto-merged files: the `oz-dev` to `oz`
rename. Cloud sandboxes only ship `oz`, so a skill calling `oz-dev` silently
loses its run link. Verified no `oz-dev` invocations survive; the three
remaining mentions are main's prose warning against it.

Review feedback (oz-for-oss):

- Error redirects must be added in both slash forms. Verified: all 17 error
  codes in vercel.json carry both a trailing-slash and a no-trailing-slash
  source (34 entries, none with only one). The skill documented one form, so a
  new code would have gotten half the coverage of every existing one. Fixed in
  sync-error-docs and its redirect-patterns reference, including the matching
  catch-all pair.

- Silent no-action runs contradicted this PR's own rule that silence requires a
  per-run log. Correct catch: the outer loops have no run log, and
  improve-drafting-skills claimed its signal log covers guard-skipped runs,
  which is false since the guard exits before the collector. Rather than adding
  three more standing log PRs to monthly agents — which fights this PR's goal —
  the rule now accepts either a run log entry or an explicit outcome line in run
  output, with the run log required for any skill an outer loop reads. Corrected
  the false claim in improve-drafting-skills.

- The third comment claimed the catch-all source is `/errors/:code/` so the
  documented grep would report it missing. Not accurate: both `/errors/:code`
  and `/errors/:code/` exist, so the grep matches. The underlying observation
  about both variants was still worth acting on and is covered above.

Aligned the three outer-loop skills to `BUZZ_SLACK_TOKEN` for consistency with
main's new guidance that the token must match the destination channel's bot.
Both secrets exist in the Oz store, so the old name would authenticate but can
fail with `channel_not_found`.

Co-Authored-By: Warp Agent <agent@warp.dev>
…-skills

The schedule was deployed as `0 15 1 * *` (the 1st of each month) rather
than the `0 17 * * 1` + first-week-guard combination the skill documented.
Both are correct and both fire exactly once a month, but the docs and the
deployed schedule disagreed.

Documented the deployed expression. Restricting only day-of-month is
unambiguous because day-of-week stays `*`, so there is no ORing hazard. The
tradeoff is noted: the 1st can land on a weekend, delaying review.

Kept the first-week guard as a safety net and explained why, since it no
longer trips on its own: it is what would narrow a day-of-week expression
back to the first Monday, and it contains the blast radius if the
day-of-month/day-of-week ORing mistake is ever reintroduced. Reworded the
guard's skip message, which still referenced 'first Monday'.

Co-Authored-By: Warp Agent <agent@warp.dev>
@rachaelrenk

Copy link
Copy Markdown
Contributor Author

/oz-review

@oz-for-oss

oz-for-oss Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@rachaelrenk

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

@oz-for-oss oz-for-oss Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This PR consolidates recurring docs automation guidance, reduces no-op Slack noise, fixes stale migration-era paths, and standardizes standing PR behavior across several skills. The changes are broadly coherent, but one outer-loop fallback still allows analysis to proceed from an incomplete log source after the branch fetch fails, which undercuts the core log-branch guarantee.

Concerns

  • .agents/skills/improve-aeo-crosslink-skill/SKILL.md should not silently fall back to the current checkout when the log branch is unavailable; that can analyze stale data and then skip notification.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz


**Do not merge the standing log PR.** An earlier version of this skill attempted the merge as its first step. That coupled the analysis to a repo write the agent may not have permission to perform, and turned an unmerged PR into a hard failure rather than a non-event. Merging is human housekeeping; see "Log availability" in `.agents/references/skill-authoring-guidelines.md`.

If the branch does not exist or the fetch fails, fall back to reading `.agents/logs/aeo_crosslink_audit_runs.md` from the current checkout and note in the run output that the history may be incomplete. Do not abort the run.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [IMPORTANT] Falling back to the current checkout after the log branch is unavailable reintroduces the stale-history problem this PR is trying to eliminate; treat this as a blocked/actionable run (or stop before analysis) so a fetch failure cannot produce silent decisions from incomplete data.

@rachaelrenk
rachaelrenk requested a review from hongyi-chen August 6, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants