Add dry-run doc validation skill by ChrisJBurns · Pull Request #724 · stacklok/docs-website

ChrisJBurns · 2026-04-15T20:17:42Z

Summary

New /test-docs-dryrun skill for fast CRD schema validation of YAML
blocks in K8s and vMCP documentation. Complements the existing
/test-docs skill (full execution) with a lightweight, fast alternative.

/test-docs-dryrun extracts all YAML blocks containing
toolhive.stacklok.dev resources from docs, runs
kubectl apply --dry-run=server on each, and reports pass/fail per doc
in a table. No resources are created. Completes in under 5 minutes for
all 109 YAML blocks across 26 docs.

Files

SKILL.md - skill definition with scope selection, validation rules,
and per-doc table output format
scripts/extract-yaml.py - Python script to extract YAML blocks from
.mdx files, split multi-document blocks, strip Docusaurus highlight
comments
scripts/check-prereqs.sh - lightweight prereqs checker (only needs
kubectl, cluster with CRDs, and python3 - no operator required)

When to use each skill

Skill	Speed	What it catches	Needs operator?
`/test-docs-dryrun`	~5 min	Field names, types, schema drift	No (CRDs only)
`/test-docs`	~30-45 min	Runtime: pods, health, outputs	Yes

Test plan

Validated against all 26 K8s + vMCP docs (109 blocks)
Result: 99 PASS, 1 known CRD bug, 7 expected, 2 skip

🤖 Generated with Claude Code

vercel · 2026-04-15T20:17:47Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs-website	Ready	Preview, Comment	Apr 15, 2026 9:22pm

Copilot

Pull request overview

Adds a new Claude skill (/test-docs-dryrun) to quickly validate ToolHive CRD YAML snippets embedded in docs by extracting YAML blocks from .mdx files and running kubectl apply --dry-run=server against a cluster that has the ToolHive CRDs installed.

Changes:

Added a new skill definition describing scope selection, validation rules, and result reporting.
Added a Python extractor to pull relevant YAML documents from MDX fenced code blocks (including multi-doc splits and Docusaurus highlight comment stripping).
Added a lightweight prerequisites checker for kubectl, cluster connectivity, CRDs, and python3.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
`.claude/skills/test-docs-dryrun/SKILL.md`	Defines the `/test-docs-dryrun` workflow, classification rules, and reporting format.
`.claude/skills/test-docs-dryrun/scripts/extract-yaml.py`	Extracts ToolHive-related YAML docs from MDX fenced blocks into per-document `.yaml` files.
`.claude/skills/test-docs-dryrun/scripts/check-prereqs.sh`	Verifies required tooling and cluster/CRD availability before running dry-run validation.

Copilot · 2026-04-15T20:21:00Z

+2. Extract 108 YAML blocks from 26 docs
+3. Dry-run validate all blocks (~2 minutes)
+4. Report: 97 PASS, 4 EXPECTED FAIL, 7 REAL FAIL


The example session hard-codes specific block counts ("Extract 108 YAML blocks" and a PASS/FAIL breakdown). Since these counts will drift as docs change (and differ from the PR description), consider using placeholders like "" or removing the concrete numbers to avoid future confusion.

Suggested change

2. Extract 108 YAML blocks from 26 docs

3. Dry-run validate all blocks (~2 minutes)

4. Report: 97 PASS, 4 EXPECTED FAIL, 7 REAL FAIL

2. Extract YAML blocks from the relevant docs

3. Dry-run validate all blocks

4. Report: `<PASS>` PASS, `<EXPECTED_FAIL>` EXPECTED FAIL, `<REAL_FAIL>` REAL FAIL

Not applicable - this references line 144 but SKILL.md is only 57 lines. The hard-coded block counts from the earlier version were removed when the skill was refactored into modular procedures.

Copilot · 2026-04-15T20:21:00Z

+    with open(filepath) as f:
+        content = f.read()


Read uses the platform default encoding; several docs include non-ASCII characters (e.g., em dashes), so this can raise UnicodeDecodeError on systems with a non-UTF-8 locale. Open the MDX file with an explicit UTF-8 encoding (and consider a clear failure message if decoding fails).

Suggested change

with open(filepath) as f:

content = f.read()

try:

with open(filepath, encoding="utf-8") as f:

content = f.read()

except UnicodeDecodeError as exc:

print(

f"Error: failed to decode {filepath} as UTF-8: {exc}",

file=sys.stderr,

)

sys.exit(1)

Good catch - fixed in 6d71bc2. Added explicit encoding="utf-8" to the file read.

Copilot · 2026-04-15T20:21:01Z

+    output_dir = sys.argv[2]
+    basename = os.path.basename(filepath).replace(".mdx", "")
+
+    blocks = extract_yaml_blocks(filepath)
+
+    for i, block in enumerate(blocks):
+        outfile = os.path.join(output_dir, f"{basename}_{i}.yaml")
+        with open(outfile, "w") as f:
+            f.write(block + "\n")


The script assumes the output directory already exists; if it doesn't, opening the output files will fail. Create the output directory (e.g., mkdir -p equivalent) before writing extracted YAML documents.

Not needed - the calling procedure (procedures/extract.md) creates the output directory with mkdir -p before invoking the script. The script doesn't need to duplicate that responsibility.

Copilot · 2026-04-15T20:21:01Z

+
+    for i, block in enumerate(blocks):
+        outfile = os.path.join(output_dir, f"{basename}_{i}.yaml")
+        with open(outfile, "w") as f:


Writing YAML uses the platform default encoding; if extracted blocks contain non-ASCII characters, this can fail or produce inconsistent output across environments. Write files with an explicit UTF-8 encoding for predictable behavior.

Suggested change

with open(outfile, "w") as f:

with open(outfile, "w", encoding="utf-8") as f:

Good catch - fixed in 6d71bc2. Added explicit encoding="utf-8" to the file write.

Copilot · 2026-04-15T20:21:01Z

+
+    filepath = sys.argv[1]
+    output_dir = sys.argv[2]
+    basename = os.path.basename(filepath).replace(".mdx", "")


Using string replacement to derive the basename can produce surprising results if the filename contains ".mdx" elsewhere (or has a different extension). Prefer using os.path.splitext (and optionally validate the extension) to derive the output prefix reliably.

Suggested change

basename = os.path.basename(filepath).replace(".mdx", "")

filename = os.path.basename(filepath)

basename, _ = os.path.splitext(filename)

Fair point - fixed in 6d71bc2. Switched to os.path.splitext which is the idiomatic approach.

Copilot · 2026-04-15T20:21:01Z

+if kubectl cluster-info &>/dev/null; then
+    CONTEXT=$(kubectl config current-context 2>/dev/null || echo "unknown")
+    echo -e "${GREEN}[OK]${NC} Cluster reachable (context: $CONTEXT)"
+else
+    echo -e "${RED}[MISSING]${NC} No Kubernetes cluster reachable"
+    ERRORS=$((ERRORS + 1))
+fi
+
+# Check ToolHive CRDs installed
+echo ""
+echo "--- ToolHive CRDs ---"
+CRD_COUNT=$(kubectl get crd 2>/dev/null | grep -c toolhive.stacklok.dev || true)
+if [ "$CRD_COUNT" -gt 0 ]; then
+    echo -e "${GREEN}[OK]${NC} $CRD_COUNT ToolHive CRDs installed"
+else
+    echo -e "${RED}[MISSING]${NC} No ToolHive CRDs found"
+    echo "       Install with: helm upgrade --install toolhive-operator-crds oci://ghcr.io/stacklok/toolhive/toolhive-operator-crds -n toolhive-system --create-namespace"
+    ERRORS=$((ERRORS + 1))
+fi


CRD detection is run even when the cluster is unreachable; in that case CRD_COUNT becomes 0 and the script reports "No ToolHive CRDs found" with a Helm install hint, which is misleading because the real issue is cluster connectivity. Gate the CRD check on successful cluster connectivity (or detect and report connectivity errors separately) so the output points to the correct fix.

Good catch - fixed in 6d71bc2. The CRD check is now gated on a CLUSTER_REACHABLE flag. When the cluster is unreachable, it shows [SKIPPED] Cannot check CRDs (cluster unreachable) instead of the misleading Helm install hint.

New /test-docs-dryrun skill for fast CRD schema validation of all YAML blocks in K8s and vMCP documentation. Extracts YAML from docs, runs kubectl apply --dry-run=server, and reports pass/fail per doc. No resources created. Completes in under 5 minutes for all docs. Includes: - SKILL.md with scope selection (single page, section, all) - Python extraction script for YAML blocks from .mdx files - Lightweight prereqs checker (only needs CRDs, not operator) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Split skill into SKILL.md (workflow) + 4 procedure files: cluster-setup.md, extract.md, validate.md, report.md - Fix duplicate filename bug: extraction script now takes --prefix flag to namespace output files by section (k8s-, vmcp-) - Fix cluster lifecycle: check for existing cluster first, default to keeping it after the run - Fix result tracking: use CSV file instead of bash associative arrays for zsh compatibility - Require per-doc breakdown table as primary output every run Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Long-running for loops with nested if/elif and subshells get backgrounded or time out when run as inline Bash tool calls. The procedure now instructs writing the loop to a .sh file first, then executing with bash. Also uses || true after kubectl and checks output string instead of exit code for reliability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 15, 2026 20:17

vercel bot deployed to Preview April 15, 2026 20:18 View deployment

Copilot started reviewing on behalf of ChrisJBurns April 15, 2026 20:18 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

danbarr previously approved these changes Apr 15, 2026

View reviewed changes

ChrisJBurns dismissed danbarr’s stale review via d2ee264 April 15, 2026 20:28

vercel bot deployed to Preview April 15, 2026 20:29 View deployment

danbarr previously approved these changes Apr 15, 2026

View reviewed changes

ChrisJBurns and others added 2 commits April 15, 2026 21:59

ChrisJBurns force-pushed the add-dryrun-skill branch from d2ee264 to 4febafb Compare April 15, 2026 20:59

vercel bot deployed to Preview April 15, 2026 21:00 View deployment

ChrisJBurns dismissed danbarr’s stale review via daea023 April 15, 2026 21:02

vercel bot deployed to Preview April 15, 2026 21:03 View deployment

danbarr previously approved these changes Apr 15, 2026

View reviewed changes

Address PR review feedback for dry-run skill

6d71bc2

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ChrisJBurns dismissed danbarr’s stale review via 6d71bc2 April 15, 2026 21:21

vercel bot deployed to Preview April 15, 2026 21:22 View deployment

danbarr approved these changes Apr 15, 2026

View reviewed changes

ChrisJBurns merged commit 81c5e7d into main Apr 15, 2026
16 checks passed

ChrisJBurns deleted the add-dryrun-skill branch April 15, 2026 21:22

-    with open(filepath) as f:
-        content = f.read()
+    try:
+        with open(filepath, encoding="utf-8") as f:
+            content = f.read()
+    except UnicodeDecodeError as exc:
+        print(
+            f"Error: failed to decode {filepath} as UTF-8: {exc}",
+            file=sys.stderr,
+        )
+        sys.exit(1)

	with open(outfile, "w") as f:
	with open(outfile, "w", encoding="utf-8") as f:

	basename = os.path.basename(filepath).replace(".mdx", "")
	filename = os.path.basename(filepath)
	basename, _ = os.path.splitext(filename)

Conversation

ChrisJBurns commented Apr 15, 2026

Summary

Files

When to use each skill

Test plan

Uh oh!

vercel bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisJBurns Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisJBurns Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisJBurns Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisJBurns Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisJBurns Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ChrisJBurns Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vercel bot commented Apr 15, 2026 •

edited

Loading