Guard bill summarization against unknown DB state#34
Merged
Conversation
On July 14 the deploy env lost MONGO_URI, and getBillByIdFromDB silently treated that as "bill not in DB" — every bill page view fell through to the Civics API + GPT-5 summarization path, unpersisted, on loop, until it surfaced as an OpenAI cost spike. Make LLM summarization fail-safe: - lookupBillInDB returns found / not-found / unavailable instead of an ambiguous null, logging loudly when the DB can't be checked. - fromCivicsProjectApiBill only summarizes when the DB is confirmed reachable AND the bill is new or its source/billTextsCount actually changed (the old "source unchanged" branch never compared source). - Fallback analyses are tagged isFallback and are never persisted or Slack-notified; the admin reprocess route returns 502 instead of overwriting stored data with a fallback. - Backstop: [LLM_SUMMARIZE] log line on every OpenAI call plus a 25-calls/hour/process soft cap; the reprocess route and evals runner bypass the cap explicitly. - Placeholder analysis judgment changed from "no" to "abstain" so unanalyzed bills don't render a "Vote No" determination. - Document MONGO_URI and NEXTAUTH_SECRET in .env.local.example. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
xrendan
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On July 14 the deploy env lost
MONGO_URI(andNEXTAUTH_SECRET).getBillByIdFromDBsilently returnsnullwhen the URI is missing, which is indistinguishable from "bill not in DB" — so every bill page view fell through to the Civics API + GPT-5 (effort: high) summarization path. Saves were also skipped without a valid URI, so nothing persisted and every request re-summarized again. Sitemap crawler traffic swept every bill and it surfaced as an OpenAI cost spike.The PR #33 auth fix was the accidental trigger: before it,
getServerSessionthrowing on the missing secret 500'd the page before the LLM path — an accidental circuit breaker. Fixing auth unblocked rendering straight into the loop.What
lookupBillInDBreturnsfound/not-found/unavailableinstead of an ambiguousnull, with a loudconsole.errorwhen the DB can't be checked.getBillByIdFromDBstays a compatible wrapper, so callers are untouched.fromCivicsProjectApiBillonly summarizes when the DB is confirmed reachable AND the bill is genuinely new or itssource/billTextsCountactually changed. (The old "source unchanged" branch never actually comparedsource— regen keyed off a missing storedrationale, and any DB error deliberately fell through to regeneration.)isFallbackand never persisted or Slack-notified. The admin reprocess route returns 502 rather than overwriting a good stored analysis with a fallback.[LLM_SUMMARIZE]log line on every OpenAI call, plus a 25-calls/hour/process soft cap that returns a fallback instead of calling OpenAI. The reprocess route and evals runner passbypassCap: true."no"to"abstain"so unanalyzed bills no longer render a real "Vote No" determination.MONGO_URIandNEXTAUTH_SECRETdocumented in.env.local.example(they weren't listed at all).Behavior changes to note
MONGO_URIlocally, or use the reprocess route underBILLS_DEV_OPEN_ACCESS.rationaleno longer re-summarizes on every fallback-path request; use the reprocess route to regenerate it explicitly.Verification
MONGO_URIunset, OpenAI key present): bill pages render 200 with placeholder analysis, zero LLM/classifier calls,[bills] DB unavailablelogged per request.tsc --noEmitclean,pnpm lintclean (pre-existing warnings only),pnpm eval:billsruns end-to-end (all cache hits, no API spend).🤖 Generated with Claude Code