Skip to content

USFEMA_FloodInsuranceClaims download-fix#2070

Open
kartik-s21 wants to merge 18 commits into
datacommonsorg:masterfrom
kartik-s21:fema-download-retries
Open

USFEMA_FloodInsuranceClaims download-fix#2070
kartik-s21 wants to merge 18 commits into
datacommonsorg:masterfrom
kartik-s21:fema-download-retries

Conversation

@kartik-s21

@kartik-s21 kartik-s21 commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

This PR resolves download failures in the FEMA NFIP claims data pipeline caused by transient HTTP 503 Service Unavailable errors from the FEMA API.

During heavy load or rapid sequential paging, the FEMA endpoint frequently returns 503 errors, causing the script to abort prematurely with Failed to download chunk or
file not found. Exiting..

Changes Made

  • Updated fema_download.py:
    • Increased tries from 3 to 10 in download_file.
    • Increased delay between retries from 5 to 10 seconds in download_file.

PR Checklist -
https://docs.google.com/spreadsheets/d/1SQJmVsUVwyI9P4P7uea2TOQHEOkFUQWZ-ubwKCFzuP8/edit?resourcekey=0-7h8GibY2hKZgbCHAxy_cOQ&gid=1086229089#gid=1086229089

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the retry attempts and delay for downloading FEMA flood insurance claims data. The reviewer noted that the new configuration (10 tries with a delay of 10 and backoff of 2) results in an excessively long cumulative wait time of over 85 minutes, which could trigger execution timeouts. They suggested reducing the number of retries to 6 to keep the total duration within a reasonable limit.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread statvar_imports/fema/flood_insurance_claims/fema_download.py
@kartik-s21 kartik-s21 changed the title Fema download retries USFEMA_FloodInsuranceClaims download-fix Jun 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant