Skip to content

Update gcsfs requirement from <=2026.7.0 to <=2026.8.0 - #347

Open
dependabot[bot] wants to merge 1 commit into
mainfrom
dependabot/pip/gcsfs-lte-2026.8.0
Open

Update gcsfs requirement from <=2026.7.0 to <=2026.8.0#347
dependabot[bot] wants to merge 1 commit into
mainfrom
dependabot/pip/gcsfs-lte-2026.8.0

Conversation

@dependabot

@dependabot dependabot Bot commented on behalf of github Aug 19, 2026

Copy link
Copy Markdown
Contributor

Updates the requirements on gcsfs to permit the latest version.

Release notes

Sourced from gcsfs's releases.

2026.8.0

What's Changed

Adaptive Concurrent Prefetching is now the default read path

Enhanced read path via adaptive concurrent prefetching is now the default in GCSFS. Starting with this version, GCSFS predicts the next byte range an application will read, fetches it in the background across several concurrent HTTP requests, and keeps the bytes in memory before next read() is called. Network round-trips overlap with application compute instead of blocking calls where compute has to wait for data to be fetched. We are also enabling read concurrency, so a single reader is no longer limited by the bandwidth of a single HTTP connection.

GCSFS prefetcher adapts to workload read IO patterns. It tracks the rolling average of recent read sizes and scales the prefetch window linearly with the detected sequential streak, rather than using a fixed block size or exponential doubling. This is inline to what modern Linux kernels will do to balance prefetch and memory footprint. When the pattern turns random read i.e. we are not able to leverage the prefetched buffer to answer the next read() call, it drains the buffer to zero, so that random-access workloads pay no bandwidth or memory penalty.

Why this matters for AI/ML workloads

  • Model loading and checkpoint restore are typically large sequential reads - and prefetcher shines there. In our benchmarking, a single-stream sequential throughput improved from 23.69 MB/s to 658.71 MB/s for 1 MB I/O, and from 156 MB/s to 736 MB/s for 16 MB I/O.
  • Training data pipelines stay fed. Parquet and sharded dataset reads issue small-to-medium sequential ranges that previously suffered from low throughput, but now achieve significantly more; at 16 MB I/O throughput rises from 150 MB/s to 730 MB/s. Reducing the wait time for data loading improves accelerator goodput(amount of time accelerator is utilised for training than waiting).
  • Multi-worker dataloader scaling. The prefetcher manufactures its own parallelism per worker instead of relying on process count alone.
  • Accelerate the throughput even further with Rapid Buckets. With Rapid Buckets single node throughput reaches 21 GiB/s with 16-process sequentially reading at 16 MiB I/O compared to standard buckets with 48processes.

Adaptive prefetcher is enabled by default when cache_type is not explicitly set and concurrency value is set at 4(DEFAULT_GCSFS_CONCURRENCY=4) for both Standard and Rapid buckets. You can disable adaptive prefetcher by setting an explicit cache_type, or by setting USE_EXPERIMENTAL_ADAPTIVE_PREFETCHING='false', or by passing use_experimental_adaptive_prefetching=False to open() call.

(Warning) Impact on memory: Prefetching trades memory for throughput. Peak memory rises from ~170 MB to 600 MB on single-stream reads for 16 MB IO size and varies with requested IO sizes, and would be materially more under high process counts. Please ensure that application memory limits accordingly to use prefetcher without any Out of Memory(OOM) issues. To put hard limit, you can also use user_max_prefetch_size

For details on architecture, tuning, full benchmark tables, along with known limitations please refer to : https://github.com/fsspec/gcsfs/blob/main/docs/source/prefetcher.rst

(#795, #805, #818, #877)

Bug Fixes & Improvements

  • Zero-cost local backward seeks in PrefetchConsumer - Parquet footer and ZIP directory reads are served from the existing buffer instead of re-issuing a network request. (#930)
  • Concurrent downloads cap task count against a minimum chunk size, removing per-task overhead on small ranges. (#926)
  • Generation consistency across parallel fetches, guaranteeing every chunk comes from the same object version. (#921)
  • Fixed silent truncation on short reads in zonal bucket downloads. (#920)
  • Zero-copy read and write paths via memoryview, cutting CPU and transient memory in the hot path. (#840, #907, #928)
  • Graceful fallback where ctypes.pythonapi is unavailable. (#938)

New Contributors

Full Changelog: fsspec/gcsfs@2026.7.0...2026.8.0

Commits
  • ea2a1ff chore: release 2026.8.0 (#995)
  • 158badb feat(subsystembenchmarks): add webdataset image dataloading read benchmark (#...
  • 2089c10 feat(subsystembenchmarks): generalize dataloading seam for multiple loaders (...
  • c99137d subsystembenchmarks: support model id substitution in checkpointing save (#1003)
  • 9d38875 Track whether cache_type is explicitly set or defaulting in User-Agent header...
  • 8accb61 fix(benchmarks): force reinstall requirement overrides and wait for JobSet we...
  • 807e628 feat(subsystembenchmarks): add PyTorch Lightning checkpointing save benchmark...
  • 38aaf37 fix(macrobenchmarks): enable autoupgrade on GKE clusters and node pools (#999)
  • afcaa35 Update benchmarks to remove custom env for prefetcher (#996)
  • 8880f63 Makes prefetcher none cache default (#992)
  • Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Updates the requirements on [gcsfs](https://github.com/fsspec/gcsfs) to permit the latest version.
- [Release notes](https://github.com/fsspec/gcsfs/releases)
- [Commits](fsspec/gcsfs@0.0.1...2026.8.0)

---
updated-dependencies:
- dependency-name: gcsfs
  dependency-version: 2026.8.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
@dependabot dependabot Bot added dependencies Pull requests that update a dependency file python Pull requests that update Python code labels Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file python Pull requests that update Python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants