Skip to content

Faster 3DEP DEM download#2

Open
supathdhitalGEO wants to merge 1 commit into
mainfrom
dev-sd
Open

Faster 3DEP DEM download#2
supathdhitalGEO wants to merge 1 commit into
mainfrom
dev-sd

Conversation

@supathdhitalGEO

Copy link
Copy Markdown
Collaborator

Summary

core/dem.py previously downloaded entire 1° USGS 3DEP tiles (~150–450 MB
each) with urllib.request.urlretrieve before clipping to the AOI, so even a
small AOI paid for a full tile- and a one-tile AOI was limited to a single
S3 connection (~2–3 MB/s).

This PR replaces the transfer layer with windowed reads against the same
tiles as Cloud-Optimized GeoTIFFs
(/vsicurl + HTTP range requests): only
the AOI's pixels ever cross the network, and each tile's window is split into
row-strips read on parallel connections, so even a single-tile AOI uses the
full connection budget.

Nothing else in the workflow changes — _download_3dep_tiles keeps its
signature and still returns a list of local tile paths, so
_clip_and_reproject, the corrupt-tile retry loop in prepare_dem, the HAND
path, the user-DEM path, and the LISFLOOD/TRITON ASCII exports are all
untouched. The output grid still exactly matches the AOI bounding box.

What changed in core/dem.py

Piece Before After
Transfer full 1° tiles via urlretrieve AOI window only, via /vsicurl range reads
Parallelism fixed 6 threads, one per tile device-scaled connections (_max_connections(), bounded 4–16 from CPU count), tiles × row-strips
GDAL tuning none _GDAL_ENV: HTTP/2 multiplexing, no S3 dir listing, VSI cache, RAM-proportional block cache (5%), GDAL_NUM_THREADS=ALL_CPUS, built-in HTTP retries — applied with setdefault so user overrides win
Corruption handling 64×64 spot-check + delete/re-download range reads fail fast; one retry per strip; failure of the whole windowed path falls back automatically to the legacy full-tile download (_download_full_tiles)
DEM_raw_<aoi>/ folder full tiles kept forever as a download cache small *_aoi.tif windows used during the run, then deleted once the clip succeeds (folder removed if empty); only the final DEM outputs remain. Full tiles fetched by the legacy fallback stay cached, since re-downloading them is expensive

New/refactored internals (all private, no API change):

  • _tile_url(), _tile_names_for_bounds() — tile naming/URL logic factored out (unchanged behaviour).
  • _fetch_tile_windows() — parallel probe of each tile's AOI window, then
    parallel row-strip reads; windows padded by 3 source cells so bilinear
    resampling keeps its kernel context at AOI edges.
  • _download_full_tiles() — the previous downloader, kept verbatim as the
    automatic fallback (with the spot-check validation it needs).
  • _apply_gdal_env(), _max_connections() — environment tuning and
    device-adaptive concurrency; no hard-coded machine-specific numbers.

Time Comparision on new vs old (Benchmark)

Square bboxes centred on Tuscaloosa, AL (EPSG:4326), 1/3 arc-second 3DEP,
end-to-end = download + clip/reproject to 10 m NAD83 UTM 16N. Fresh cache for
every run (the old path re-validates/downloads full tiles; the new path always
re-reads its AOI window).

AOI Tiles Old total New total Speed-up Data transferred (old → new)
500 km² 1 75.5 s 11.7 s 6.5× 514 MB → 24 MB (22× less)
1000 km² 1 98.2 s 12.8 s 7.7× 514 MB → 47 MB (11× less)
2500 km² 2 129.2 s 28.8 s 4.5× 1022 MB → 119 MB (9× less)
5000 km² 2 134.8 s 39.6 s 3.4× 1022 MB → 239 MB (4× less)

Download stage alone: 6.7× / 8.3× / 4.7× / 3.8× faster respectively; the
clip/reproject stage is unchanged (and slightly faster, since it now merges
small windows instead of full tiles).

Numbers are single runs on one machine/network and will vary with bandwidth,
but the mechanism is structural: the old path's cost scales with tile count ×
full-tile size
, the new path's with AOI area, the smaller the AOI relative
to its tiles, the larger the gain.

@supathdhitalGEO supathdhitalGEO added the enhancement New feature or request label Jul 3, 2026
pnikrou pushed a commit that referenced this pull request Jul 6, 2026
Port the faster download method from PR #2 into the shared DEM engine.
Instead of downloading entire 1-degree 3DEP tiles (often hundreds of MB),
_download_3dep_tiles now reads ONLY the AOI window from each Cloud-Optimized
GeoTIFF over /vsicurl, split into row-strips read in parallel, with GDAL
HTTP/2 + vsicurl caching tuned via env vars.  Falls back to the old
full-tile urllib download if the windowed read fails, so behaviour is never
worse.  AOI-window intermediates are cleaned up after the clip.

Because prepare_dem() is the single shared entry point, this speeds up DEM
downloads everywhere: DEM standalone mode, LISFLOOD-FP, TRITON, and
ARC-Curve2Flood.

Composes cleanly with the earlier fixes — the exact-cellsize grid
(_clip_and_reproject) and the shapefile-CRS selection are untouched; the
faster download just replaces how tiles are fetched.

Verified: download block is byte-identical to PR #2; cellsize fix intact;
core.dem + all orchestrators + app import cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant