Skip to content

chore: automate AI model catalog sync#32057

Open
adriandlam wants to merge 1 commit into
cloudflare:productionfrom
adriandlam:ci/automate-ai-model-catalog-sync
Open

chore: automate AI model catalog sync#32057
adriandlam wants to merge 1 commit into
cloudflare:productionfrom
adriandlam:ci/automate-ai-model-catalog-sync

Conversation

@adriandlam

@adriandlam adriandlam commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Automates daily synchronization of AI Gateway catalog model files from the Unified Catalog API. The workflow uses cloudflare-docs-bot to open or refresh a pull request when catalog changes are detected, and closes an obsolete catalog pull request when the source matches production again.

The catalog fetcher rejects malformed detail responses and empty catalogs before replacing the existing catalog, omits schema when the API returns null, and reports model-specific failures with a 30-second request timeout.

Required configuration

Configure the AI_MODEL_CATALOG_ACCOUNT_ID and AI_MODEL_CATALOG_API_TOKEN Actions secrets. The workflow reuses the existing cloudflare-docs-bot GitHub App credentials.

Validation

  • pnpm --config.verify-deps-before-run=false run test (137 tests)
  • pnpm --config.verify-deps-before-run=false run sync
  • pnpm --config.verify-deps-before-run=false exec eslint bin/fetch-catalog-models.ts
  • pnpm --config.verify-deps-before-run=false run format:core:check
  • git diff --check
  • CLI mock API controls: malformed model data is rejected without replacement, and a checked-in valid model is accepted

The live API fetch was not run locally because the catalog API token is only available through Actions secrets. Full type-check and lint remain blocked by existing generated skills/turnstile-spin errors unrelated to this change.

@adriandlam adriandlam force-pushed the ci/automate-ai-model-catalog-sync branch 6 times, most recently from 143b631 to 07ff037 Compare July 14, 2026 16:59
@adriandlam adriandlam marked this pull request as ready for review July 14, 2026 17:03
@cloudflare-docs-bot

cloudflare-docs-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Review

🚨 1 critical, ⚠️ 1 warning, 💡 1 suggestion found in commit 8bf0781.

Fix in your agent
Fix the following review findings in PR #32057 (https://github.com/cloudflare/cloudflare-docs/pull/32057).

Before making changes, review each finding and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g. the fix requires disproportionate effort for minimal benefit,
  or the finding is factually incorrect), explain why
- If you need clarification before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

After triaging, fix all legitimate findings. For any you decide to skip,
post a comment on this PR with the finding ID and your reasoning.

---

## Code Review

### Critical (1)

#### CR-fb410b562741 · PR creation silently skipped on first run
- **File:** `.github/workflows/sync-ai-model-catalog.yml` line 124
- **Issue:** When there is no existing open PR, `gh pr list ... --jq '.[0].number'` returns `null`, making `existing` non-empty. The step then prints "Updated existing PR #null" and exits 0 without creating a PR, so catalog changes are pushed to an orphan branch.
- **Fix:** Change the jq filter to `--jq '.[0].number // empty'` and keep the `[[ -n "$existing" ]]` guard.

### Warnings (1)

#### CR-35e0ff3561ef · Incorrect empty-result handling from gh/jq
- **File:** `.github/workflows/sync-ai-model-catalog.yml` line 93
- **Issue:** When no open PR exists, `gh pr list ... --jq '.[0].number'` outputs the literal string `null`, so `existing` is non-empty and `gh pr close "$existing"` runs `gh pr close null`.
- **Fix:** Use `--jq '.[0].number // empty'` so the variable is truly empty when no PR exists.

### Suggestions (1)

#### CR-dff00a670dfc · Opaque validation failure
- **File:** `bin/fetch-catalog-models.ts` line 402
- **Issue:** When `data.result` fails `isCatalogModel`, `fetchModelDetail` returns the generic summary "invalid detail response" with no indication of which field was invalid or why the model was rejected.
- **Fix:** Make `isCatalogModel` return or accept a diagnostic callback that reports the path/reason of the first validation failure, and include that detail in the returned summary so the sync log explains why a model was rejected.

Code Review

This code review is in beta and may not always be helpful — use your judgment.

Critical (1)
File Issue
.github/workflows/sync-ai-model-catalog.yml line 124 PR creation silently skipped on first run — When there is no existing open PR, gh pr list ... --jq '.[0].number' returns null, making existing non-empty. The step then prints "Updated existing PR #null" and exits 0 without creating a PR, so catalog changes are pushed to an orphan branch. Fix: Change the jq filter to --jq '.[0].number // empty' and keep the [[ -n "$existing" ]] guard.
Warnings (1)
File Issue
.github/workflows/sync-ai-model-catalog.yml line 93 Incorrect empty-result handling from gh/jq — When no open PR exists, gh pr list ... --jq '.[0].number' outputs the literal string null, so existing is non-empty and gh pr close "$existing" runs gh pr close null. Fix: Use --jq '.[0].number // empty' so the variable is truly empty when no PR exists.
Suggestions (1)
File Issue
bin/fetch-catalog-models.ts line 402 Opaque validation failure — When data.result fails isCatalogModel, fetchModelDetail returns the generic summary "invalid detail response" with no indication of which field was invalid or why the model was rejected. Fix: Make isCatalogModel return or accept a diagnostic callback that reports the path/reason of the first validation failure, and include that detail in the returned summary so the sync log explains why a model was rejected.

Conventions

No convention issues found.

Style Guide Review

No style-guide issues found.

Commands

Only codeowners can run commands. Post a comment with the command to trigger it.

Command Description
/review Runs a review now. Incremental if a prior review exists, full if not.
/full-review Re-reviews the entire PR diff from scratch, ignoring incremental history. Useful after a rebase, when you want a fresh review, or if the bot gets out of sync and reports issues that no longer exist.
/ignore-review-limit Permanently lifts the 2-review automatic limit for this PR. Future pushes will trigger reviews as normal.
/disable-auto-review Stops automatic reviews from triggering on future pushes to this PR. Codeowners can still run /review or /full-review manually.

@adriandlam adriandlam changed the title [AI Gateway] Automate model catalog sync chore: automate AI model catalog sync Jul 14, 2026
@adriandlam adriandlam marked this pull request as draft July 14, 2026 21:44
@adriandlam adriandlam force-pushed the ci/automate-ai-model-catalog-sync branch from 07ff037 to 8bf0781 Compare July 15, 2026 16:09
@adriandlam adriandlam marked this pull request as ready for review July 15, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.