Skip to content

Campaign finance scrapper for legislators#2195

Open
J-C-L wants to merge 7 commits into
codeforboston:mainfrom
J-C-L:finance-scrapper
Open

Campaign finance scrapper for legislators#2195
J-C-L wants to merge 7 commits into
codeforboston:mainfrom
J-C-L:finance-scrapper

Conversation

@J-C-L

@J-C-L J-C-L commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

This populates the finance tab on the legislator profile pages. See Issue #2118
It includes:

  • Fetches campaign finance data from the OCPF website [currently this is a run on demand function. Once data is validated, we'll switch to nightly scheduled run
  • Stores campaign finance data per legislator in Firestore under new collection: generalCourts > 194 > membersFinance
    <Member_ID>
  • Displays campaign finance summary on finance tab of legislator page for each legislator

Checklist

  • [Y] On the frontend, I've made my strings translate-able.
  • [N/A ] If I've added shared components, I've added a storybook story.
  • [Y ] I've made pages responsive and look good on mobile.
  • [ N/A] If I've added new Firestore queries, I've added any new required indexes to firestore.indexes.json (Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)

Screenshots

finance-tab-screenshot

Known issues

  • scrapeOcpfFinance function must be run after matchOcpfMembers, so court member IDs are already associated in firestore with OCPF IDs
  • Code is safe to push while legislator pages are behind a flag, and hidden in prod.
  • Still need to consult with campaign finance expert on some data choices made here.
  • "small donors" box is not yet populated. Need to decide what we want to use there (<$50 or <$200)

Steps to test/reproduce

For reproducing locally until firebase functions deployed:

  1. Load members in firestore (run a few times with delay)
    curl "http://localhost:5001/demo-dtp/us-central1/triggerPubsubFunction?scheduled=startMemberBatches"

  2. Match member IDs:
    curl -X POST http://localhost:5001/demo-dtp/us-central1/matchOcpfMembers

  3. Run finance scraper:
    curl -X POST http://localhost:5001/demo-dtp/us-central1/scrapeOcpfFinance

  4. View Legislator Page:
    Ex: http://localhost:3000/legislators/194/RLR0

@vercel

vercel Bot commented Jul 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
maple-dev Ready Ready Preview, Comment Jul 14, 2026 11:40pm

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant