Commit 3906f0e
committed
gvfs-helper: parallelize POST requests (#980)
Improve full-clone performance by allowing `gvfs-helper post` to
download
object batches concurrently. The new `gvfs.postThreads` configuration
defaults to 1, preserving the existing sequential behavior unless
explicitly
enabled.
The series is organized into six reviewable commits:
1. Add the configuration surface.
2. Factor reusable curl-handle preparation.
3. Introduce the parallel POST success path.
4. Complete retry, authentication, fallback, throttling, and pack
installation behavior.
5. Test sequential and parallel success paths.
6. Test failure handling and request-header parity.
Each worker owns an independently prepared curl handle and streams
responses
into a dedicated `index-pack --stdin` child. Work is distributed through
a
mutex-protected queue, with the worker count capped by the number of
object
batches. Requests with configured cookies retain the established
sequential
path because libcurl cookie state cannot safely be shared by
concurrently
performing handles.
The parallel path preserves the sequential HTTP policy for
authentication
refresh, transient errors, Retry-After, cache and backup-cache fallback,
`--no-fallback`, configured headers, `X-Session-Id`, and `X-VSS-E2EID`.
Sequential and parallel requests share response-header parsing, while
soft-throttle state remains local to each worker.
Child pipes are created and marked close-on-exec before spawning, and
child
setup and completion are serialized around process-global run-command
state.
Each `index-pack` attempt uses unique pack and index paths, validates
its
reported pack hash, retries corrupt or truncated responses, and
tolerates
another process winning installation of the same final pack.
The focused `t5798-gvfs-helper-post-threads.sh` suite contains 22 tests
covering configuration boundaries, sequential and parallel requests,
multi-worker participation, singleton remainders, duplicate downloads,
deadlock prevention, authentication, throttling, cache fallback,
`--no-fallback`, corrupt packs, cookies, and configured headers.
Parallel
tests use pthread prerequisites and Trace2 assertions to prove the
intended
execution path.
Neil Kainga diagnosed and tested the pipe-inheritance fix on a 1JS full
clone
with `gvfs.postThreads=8` and is credited throughout the series.7 files changed
Lines changed: 1732 additions & 50 deletions
File tree
- Documentation/config
- t
- helper
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
0 commit comments