Faster 3DEP DEM download#2
Open
supathdhitalGEO wants to merge 1 commit into
Open
Conversation
pnikrou
pushed a commit
that referenced
this pull request
Jul 6, 2026
Port the faster download method from PR #2 into the shared DEM engine. Instead of downloading entire 1-degree 3DEP tiles (often hundreds of MB), _download_3dep_tiles now reads ONLY the AOI window from each Cloud-Optimized GeoTIFF over /vsicurl, split into row-strips read in parallel, with GDAL HTTP/2 + vsicurl caching tuned via env vars. Falls back to the old full-tile urllib download if the windowed read fails, so behaviour is never worse. AOI-window intermediates are cleaned up after the clip. Because prepare_dem() is the single shared entry point, this speeds up DEM downloads everywhere: DEM standalone mode, LISFLOOD-FP, TRITON, and ARC-Curve2Flood. Composes cleanly with the earlier fixes — the exact-cellsize grid (_clip_and_reproject) and the shapefile-CRS selection are untouched; the faster download just replaces how tiles are fetched. Verified: download block is byte-identical to PR #2; cellsize fix intact; core.dem + all orchestrators + app import cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
core/dem.pypreviously downloaded entire 1° USGS 3DEP tiles (~150–450 MBeach) with
urllib.request.urlretrievebefore clipping to the AOI, so even asmall AOI paid for a full tile- and a one-tile AOI was limited to a single
S3 connection (~2–3 MB/s).
This PR replaces the transfer layer with windowed reads against the same
tiles as Cloud-Optimized GeoTIFFs (
/vsicurl+ HTTP range requests): onlythe AOI's pixels ever cross the network, and each tile's window is split into
row-strips read on parallel connections, so even a single-tile AOI uses the
full connection budget.
Nothing else in the workflow changes —
_download_3dep_tileskeeps itssignature and still returns a list of local tile paths, so
_clip_and_reproject, the corrupt-tile retry loop inprepare_dem, the HANDpath, the user-DEM path, and the LISFLOOD/TRITON ASCII exports are all
untouched. The output grid still exactly matches the AOI bounding box.
What changed in
core/dem.pyurlretrieve/vsicurlrange reads_max_connections(), bounded 4–16 from CPU count), tiles × row-strips_GDAL_ENV: HTTP/2 multiplexing, no S3 dir listing, VSI cache, RAM-proportional block cache (5%),GDAL_NUM_THREADS=ALL_CPUS, built-in HTTP retries — applied withsetdefaultso user overrides win_download_full_tiles)DEM_raw_<aoi>/folder*_aoi.tifwindows used during the run, then deleted once the clip succeeds (folder removed if empty); only the final DEM outputs remain. Full tiles fetched by the legacy fallback stay cached, since re-downloading them is expensiveNew/refactored internals (all private, no API change):
_tile_url(),_tile_names_for_bounds()— tile naming/URL logic factored out (unchanged behaviour)._fetch_tile_windows()— parallel probe of each tile's AOI window, thenparallel row-strip reads; windows padded by 3 source cells so bilinear
resampling keeps its kernel context at AOI edges.
_download_full_tiles()— the previous downloader, kept verbatim as theautomatic fallback (with the spot-check validation it needs).
_apply_gdal_env(),_max_connections()— environment tuning anddevice-adaptive concurrency; no hard-coded machine-specific numbers.
Time Comparision on new vs old (Benchmark)
Square bboxes centred on Tuscaloosa, AL (EPSG:4326), 1/3 arc-second 3DEP,
end-to-end = download + clip/reproject to 10 m NAD83 UTM 16N. Fresh cache for
every run (the old path re-validates/downloads full tiles; the new path always
re-reads its AOI window).
Download stage alone: 6.7× / 8.3× / 4.7× / 3.8× faster respectively; the
clip/reproject stage is unchanged (and slightly faster, since it now merges
small windows instead of full tiles).
Numbers are single runs on one machine/network and will vary with bandwidth,
but the mechanism is structural: the old path's cost scales with tile count ×
full-tile size, the new path's with AOI area, the smaller the AOI relative
to its tiles, the larger the gain.