Skip to content

Guard bill summarization against unknown DB state#34

Merged
mikaalnaik merged 1 commit into
mainfrom
mikaal/builder-mp-rerun-guard
Jul 15, 2026
Merged

Guard bill summarization against unknown DB state#34
mikaalnaik merged 1 commit into
mainfrom
mikaal/builder-mp-rerun-guard

Conversation

@mikaalnaik

Copy link
Copy Markdown
Contributor

Why

On July 14 the deploy env lost MONGO_URI (and NEXTAUTH_SECRET). getBillByIdFromDB silently returns null when the URI is missing, which is indistinguishable from "bill not in DB" — so every bill page view fell through to the Civics API + GPT-5 (effort: high) summarization path. Saves were also skipped without a valid URI, so nothing persisted and every request re-summarized again. Sitemap crawler traffic swept every bill and it surfaced as an OpenAI cost spike.

The PR #33 auth fix was the accidental trigger: before it, getServerSession throwing on the missing secret 500'd the page before the LLM path — an accidental circuit breaker. Fixing auth unblocked rendering straight into the loop.

What

  • lookupBillInDB returns found / not-found / unavailable instead of an ambiguous null, with a loud console.error when the DB can't be checked. getBillByIdFromDB stays a compatible wrapper, so callers are untouched.
  • fromCivicsProjectApiBill only summarizes when the DB is confirmed reachable AND the bill is genuinely new or its source/billTextsCount actually changed. (The old "source unchanged" branch never actually compared source — regen keyed off a missing stored rationale, and any DB error deliberately fell through to regeneration.)
  • Fallback analyses (no OpenAI key, parse failure, API error) are tagged isFallback and never persisted or Slack-notified. The admin reprocess route returns 502 rather than overwriting a good stored analysis with a fallback.
  • Backstop: a grep-able [LLM_SUMMARIZE] log line on every OpenAI call, plus a 25-calls/hour/process soft cap that returns a fallback instead of calling OpenAI. The reprocess route and evals runner pass bypassCap: true.
  • Placeholder analysis judgment changed from "no" to "abstain" so unanalyzed bills no longer render a real "Vote No" determination.
  • MONGO_URI and NEXTAUTH_SECRET documented in .env.local.example (they weren't listed at all).

Behavior changes to note

  • Dev without Mongo no longer silently summarizes on page views — set MONGO_URI locally, or use the reprocess route under BILLS_DEV_OPEN_ACCESS.
  • A bill stored with an empty rationale no longer re-summarizes on every fallback-path request; use the reprocess route to regenerate it explicitly.

Verification

  • Reproduced the incident scenario locally (MONGO_URI unset, OpenAI key present): bill pages render 200 with placeholder analysis, zero LLM/classifier calls, [bills] DB unavailable logged per request.
  • Connection-error path (unreachable Atlas): pages degrade gracefully in ~360ms, no LLM calls, self-heals on next ISR revalidation.
  • Healthy path: DB-served bills render their stored analysis unchanged.
  • tsc --noEmit clean, pnpm lint clean (pre-existing warnings only), pnpm eval:bills runs end-to-end (all cache hits, no API spend).
  • Not exercised: summarize-once-and-persist for a genuinely new bill (would mutate the shared Atlas DB / spend a real GPT-5 call). Worth one manual reprocess check after deploy.

🤖 Generated with Claude Code

On July 14 the deploy env lost MONGO_URI, and getBillByIdFromDB
silently treated that as "bill not in DB" — every bill page view fell
through to the Civics API + GPT-5 summarization path, unpersisted, on
loop, until it surfaced as an OpenAI cost spike.

Make LLM summarization fail-safe:

- lookupBillInDB returns found / not-found / unavailable instead of an
  ambiguous null, logging loudly when the DB can't be checked.
- fromCivicsProjectApiBill only summarizes when the DB is confirmed
  reachable AND the bill is new or its source/billTextsCount actually
  changed (the old "source unchanged" branch never compared source).
- Fallback analyses are tagged isFallback and are never persisted or
  Slack-notified; the admin reprocess route returns 502 instead of
  overwriting stored data with a fallback.
- Backstop: [LLM_SUMMARIZE] log line on every OpenAI call plus a
  25-calls/hour/process soft cap; the reprocess route and evals runner
  bypass the cap explicitly.
- Placeholder analysis judgment changed from "no" to "abstain" so
  unanalyzed bills don't render a "Vote No" determination.
- Document MONGO_URI and NEXTAUTH_SECRET in .env.local.example.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mikaalnaik mikaalnaik self-assigned this Jul 15, 2026
@mikaalnaik mikaalnaik requested a review from xrendan July 15, 2026 16:17
@mikaalnaik mikaalnaik merged commit c3504c1 into main Jul 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants