Skip to content

Commit 3906f0e

Browse files
committed
gvfs-helper: parallelize POST requests (#980)
Improve full-clone performance by allowing `gvfs-helper post` to download object batches concurrently. The new `gvfs.postThreads` configuration defaults to 1, preserving the existing sequential behavior unless explicitly enabled. The series is organized into six reviewable commits: 1. Add the configuration surface. 2. Factor reusable curl-handle preparation. 3. Introduce the parallel POST success path. 4. Complete retry, authentication, fallback, throttling, and pack installation behavior. 5. Test sequential and parallel success paths. 6. Test failure handling and request-header parity. Each worker owns an independently prepared curl handle and streams responses into a dedicated `index-pack --stdin` child. Work is distributed through a mutex-protected queue, with the worker count capped by the number of object batches. Requests with configured cookies retain the established sequential path because libcurl cookie state cannot safely be shared by concurrently performing handles. The parallel path preserves the sequential HTTP policy for authentication refresh, transient errors, Retry-After, cache and backup-cache fallback, `--no-fallback`, configured headers, `X-Session-Id`, and `X-VSS-E2EID`. Sequential and parallel requests share response-header parsing, while soft-throttle state remains local to each worker. Child pipes are created and marked close-on-exec before spawning, and child setup and completion are serialized around process-global run-command state. Each `index-pack` attempt uses unique pack and index paths, validates its reported pack hash, retries corrupt or truncated responses, and tolerates another process winning installation of the same final pack. The focused `t5798-gvfs-helper-post-threads.sh` suite contains 22 tests covering configuration boundaries, sequential and parallel requests, multi-worker participation, singleton remainders, duplicate downloads, deadlock prevention, authentication, throttling, cache fallback, `--no-fallback`, corrupt packs, cookies, and configured headers. Parallel tests use pthread prerequisites and Trace2 assertions to prove the intended execution path. Neil Kainga diagnosed and tested the pipe-inheritance fix on a 1JS full clone with `gvfs.postThreads=8` and is credited throughout the series.
2 parents 92bfd44 + eefe24c commit 3906f0e

7 files changed

Lines changed: 1732 additions & 50 deletions

File tree

‎Documentation/config/gvfs.adoc‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,14 @@ gvfs.prefetchThreads::
5555
index-pack execution, which can significantly speed up the
5656
installation of multiple prefetch packs. Values less than `1` are
5757
treated as `1`.
58+
59+
gvfs.postThreads::
60+
Set the number of parallel workers used when fetching objects
61+
via HTTP POST requests. Each worker creates its own HTTP
62+
connection and streams the response directly into an
63+
`index-pack --stdin` child process. The default value is `1`,
64+
which processes POST requests sequentially using the existing
65+
code path. Setting this to a higher value (for example `4`)
66+
downloads multiple batches of objects concurrently when the POST
67+
block size is at least `100`. Values less than `1` are treated as
68+
`1`.

0 commit comments

Comments
 (0)