fix(git-remote): drive multi-round fetch as a v0 stateless-RPC client loop (#117)#192
fix(git-remote): drive multi-round fetch as a v0 stateless-RPC client loop (#117)#192beardthelion wants to merge 14 commits into
Conversation
… loop (#117) git-remote-gitlawb advertised `connect`, so git spoke the stateful native protocol, but handle_connect collapsed the exchange into one GET plus one POST. A multi-round fetch (more than ~32 overlapping commits) deadlocked: git sent a flush-terminated have-batch and blocked for an ACK/NAK the helper never sent, while the helper blocked reading for a `done` git never sent. Turn Phase 2 into a per-round stateless-RPC client loop. read_upload_pack_round returns one round at a time (at a flush, done, or EOF) instead of buffering past flushes waiting for `done`; negotiate_upload_pack captures the wants once and POSTs a self-contained request per flush-terminated batch: wants, one flush, every have accumulated so far, and exactly one terminator. Each ACK/NAK is streamed back to git so it advances, until the done round returns the pack. The node's `git upload-pack --stateless-rpc` keeps no state between POSTs, so wants and all prior haves are re-sent every round and no intermediate flush survives into a body (which would truncate the negotiation server-side). Signing is preserved per round: every POST carries the Phase-1 decision (signed after the 404 escalation for a private repo, anonymous for a public one), and a mid-negotiation denial surfaces through the sanitized error path rather than reading as an empty or successful fetch. The receive-pack path is unchanged. Covers the deadlock repro, multi-round accumulation on the wire, per-round signing for private and public fetches, mid-negotiation denial surfacing, the withheld-shaped forwarding path, and single-round non-regression.
… loop (#117) A committed integration test drives a real `git fetch` through the built helper against a real `git upload-pack --stateless-rpc`, so the stateful-to-stateless ACK bridging is proven by execution rather than reasoned. The main.rs mock tests cannot falsify it: a pre-scripted Cursor never reacts to the server's ACKs. An in-test shim replicates the node's v0 serving; the fetch is forced to at least two negotiation rounds (a fixture that resolved in one round fails the test) and the resulting object graph is verified. The second scenario drives the withheld-blob shape (a full pack on the first POST, as upload_pack_excluding does) through real git and records the outcome: real git rejects a pack where it expected an ACK continuation ("expected ACK/NAK, got ..."), so a multi-round fetch of a withheld repo does not complete. The helper forwards and terminates cleanly rather than hanging, so this is a node-side concern, not a helper defect, left as a follow-up per the plan's Withheld-Path Decision. The test guards the helper's non-hang behavior and surfaces the break if the assumption ever changes.
A code-review pass found the negotiation loop would POST once per bare `0000` flush that carried no haves, so a malformed `wants + N*0000 + EOF` stream amplified into N signed POSTs. Real git never sends a content-free mid-negotiation flush and the peer is upstream of the local git process, so this was not reachable in practice, but it left the loop unbounded per input. Skip a round that carries no new haves (an empty flush) and only POST rounds with have content or a terminator, which also makes the loop provably bounded by git's finite have set. The fresh-clone done-with-no-haves round still POSTs.
Adds executed coverage for the negotiation branches the first pass left unrun: an invalid pkt-line length (rejected, not underflowed), a non-EOF read error (propagated, not swallowed as end-of-stream), an immediately empty upload-pack request (skips the POST), a nonempty have-batch terminated by EOF (POSTed as a final done round), and an empty receive-pack body (skips the POST). Each was mutation-checked to fail without its guard, so the coverage is load-bearing rather than vacuous.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe helper now supports multi-round ChangesUpload-pack negotiation
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Git
participant Helper
participant HTTPShim
participant UploadPack
Git->>Helper: Send upload-pack negotiation round
Helper->>HTTPShim: POST round
HTTPShim->>UploadPack: Forward stateless request
UploadPack-->>HTTPShim: Return ACK, NAK, or pack response
HTTPShim-->>Helper: Return HTTP response
Helper-->>Git: Stream response
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
beardthelion
added
crate:git-remote
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
crates/git-remote-gitlawb/src/main.rs (1)
301-336: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicates
post_pack_round's POST/status-check/forward logic.Lines 313-334 re-implement almost exactly what
post_pack_round(lines 502-533) does: build the signed POST, send, check status viahttp_error_message, read the response body, and forward it to stdout. Sincepost_pack_roundalready takes(client, post_url, service, signing_key, body, stdout)matching this call site's variables, the receive-pack path can just call it directly instead of duplicating the block.♻️ Suggested refactor
let mut request_body = Vec::new(); stdin .read_to_end(&mut request_body) .context("reading receive-pack request")?; tracing::debug!("pack request: {} bytes from git", request_body.len()); if request_body.is_empty() { // e.g., already up-to-date — nothing to send tracing::debug!("empty request body — skipping POST"); return Ok(()); } - tracing::debug!("POST {post_url} ({} bytes)", request_body.len()); - let req = build_pack_post_request(&client, &post_url, service, &request_body, signing_key); - - // Attach the body after signing so the pack bytes are moved, not cloned — - // packs can be large and the clone doubled peak memory on push. - let pack_resp = req - .body(request_body) - .send() - .with_context(|| format!("POST {post_url}"))?; - - if !pack_resp.status().is_success() { - let status = pack_resp.status(); - let body = read_error_body(pack_resp); - let path = format!("/{service}"); - bail!("{}", http_error_message("POST", &path, status, &body, None)); - } - - let pack_bytes = pack_resp.bytes().context("reading pack response")?; - tracing::debug!("pack response: {} bytes from node", pack_bytes.len()); - - stdout.write_all(&pack_bytes)?; - stdout.flush()?; - - Ok(()) + post_pack_round(&client, &post_url, service, signing_key, request_body, &mut stdout) }This keeps behavior identical (same log lines, same error path) while removing a divergence risk between the two copies.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/git-remote-gitlawb/src/main.rs` around lines 301 - 336, Replace the duplicated POST, status-check, response-reading, and stdout-forwarding block after the empty-body guard with a direct call to post_pack_round, passing client, post_url, service, signing_key, request_body, and stdout. Preserve the existing request-body logging and empty-body early return while reusing post_pack_round for the shared behavior.crates/git-remote-gitlawb/tests/real_git_fetch.rs (1)
354-436: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTemp repos leak on assertion failure.
cleanup(&[server, clone])(lines 387, 429) only runs if every priorassert!/assert_eq!in the test succeeds. A failed assertion (e.g.completed,fsck, or theFETCH_HEADequality checks) panics before cleanup, leaking theunique_dir()-created server/clone repos (each with a real object DB) under the OS temp dir.Shimalready avoids this class of leak for its own thread/listener viaDrop(lines 111-126); the repo directories don't get the same RAII treatment.♻️ Wrap temp dirs in an RAII guard so they're removed even on panic
struct TempDir(PathBuf); impl std::ops::Deref for TempDir { type Target = Path; fn deref(&self) -> &Path { &self.0 } } impl Drop for TempDir { fn drop(&mut self) { let _ = std::fs::remove_dir_all(&self.0); } }Then have
unique_dir/build_divergent_reposreturnTempDirand drop the explicitcleanup(...)calls at the end of each test.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/git-remote-gitlawb/tests/real_git_fetch.rs` around lines 354 - 436, Introduce an RAII temporary-directory guard for paths created by unique_dir and returned through build_divergent_repos, implementing path access and Drop to remove each directory. Update both real_git_multi_round_fetch_completes and real_git_withheld_shaped_first_post to use the guarded values and remove the explicit cleanup calls, ensuring directories are deleted during assertion panics as well as normal completion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/git-remote-gitlawb/tests/real_git_fetch.rs`:
- Around line 244-284: Update fetch_with_helper so the child’s stdout and stderr
are drained concurrently while the parent polls for completion, preventing
pipe-buffer backpressure from blocking git fetch. Preserve the existing
30-second timeout, kill-and-collect behavior, and boolean result semantics.
---
Nitpick comments:
In `@crates/git-remote-gitlawb/src/main.rs`:
- Around line 301-336: Replace the duplicated POST, status-check,
response-reading, and stdout-forwarding block after the empty-body guard with a
direct call to post_pack_round, passing client, post_url, service, signing_key,
request_body, and stdout. Preserve the existing request-body logging and
empty-body early return while reusing post_pack_round for the shared behavior.
In `@crates/git-remote-gitlawb/tests/real_git_fetch.rs`:
- Around line 354-436: Introduce an RAII temporary-directory guard for paths
created by unique_dir and returned through build_divergent_repos, implementing
path access and Drop to remove each directory. Update both
real_git_multi_round_fetch_completes and real_git_withheld_shaped_first_post to
use the guarded values and remove the explicit cleanup calls, ensuring
directories are deleted during assertion panics as well as normal completion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 38d3b6d3-d10b-460c-b569-9017a993168a
📒 Files selected for processing (2)
crates/git-remote-gitlawb/src/main.rscrates/git-remote-gitlawb/tests/real_git_fetch.rs
jatmn
left a comment
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Findings
-
[P2] Do not convert an aborted negotiation into a
donerequest
crates/git-remote-gitlawb/src/main.rs:610
A nonempty EOF is explicitly documented here as the case where Git aborted mid-negotiation, but it is combined withRoundEnd::Done, receives a synthetic0009done\n, and is sent to the node. Cancelling a fetch after writing a partial have batch therefore still issues an authorized (and, for a private fetch, signed) upload-pack request, making the node generate and stream a pack to a client that has gone away. EOF should terminate the helper without a POST; only the actualdonepkt should finalize the stateless request. The newnonempty_eof_batch_posts_as_final_roundtest currently locks in the incorrect behavior. -
[P2] Drain the real Git child output while waiting for it
crates/git-remote-gitlawb/tests/real_git_fetch.rs:266
fetch_with_helperpipes both stdout and stderr, then only pollstry_waituntil the process exits. A fetch that emits more progress or error output than an OS pipe can hold blocks in Git before it exits, so this harness reaches its 30-second timeout and reports the protocol deadlock even when the helper is making progress. Drain both streams concurrently while enforcing the deadline (or do not pipe them) so the new load-bearing integration test can distinguish a real negotiation hang from pipe backpressure. -
[P3] Make the integration-test repositories cleanup-safe
crates/git-remote-gitlawb/tests/real_git_fetch.rs:356
The server and clone directories are deleted only after every assertion succeeds. Any timeout, failed fetch,fsckfailure, or panic exits beforecleanup, leaving real Git repositories in the shared temporary directory; the predictable pid/counter names can also poison a later run if a pid is reused. Return RAII-owned temporary directories (or usetempfile) so cleanup runs while unwinding as well.
…git test harness (#192) F1: a nonempty have-batch terminated by EOF means git aborted mid-negotiation, but the fetch loop folded RoundEnd::Eof into the Done arm, appended a synthetic 0009done, and POSTed it — so a cancelled private fetch still issued an authorized, signed upload-pack request and made the node generate and stream a full pack to a client that had gone away. Split the arm: only a real done pkt finalizes; a nonempty EOF terminates without a POST. The nonempty_eof_batch_posts_as_final_round test that locked in the wrong behavior is flipped to assert no POST. F2: fetch_with_helper piped stdout+stderr but only polled try_wait, so a fetch emitting more than an OS pipe buffer (~64 KiB) blocked git before it exited and the harness reported a false 30s deadlock. Drain both streams on reader threads while enforcing the deadline, so the deadline measures a real protocol hang only. F3: the integration repos were removed only on the success path (predictable pid/counter names), so a timeout/panic leaked real git repos and could poison a later run. Root them in a RAII tempfile::TempDir that drops on unwind. RED->GREEN: nonempty_eof_batch_aborts_without_posting (pre-fix POSTs a synthetic done → the aborted-negotiation POST fails against the single-response server; fixed makes one request, no POST). repos_are_cleaned_up_on_unwind (temp dir gone after a panic unwinds past the guard). Full git-remote-gitlawb suite 42 unit + 3 integration green, fmt + clippy clean.
|
All three addressed on Aborted EOF no longer POSTs (F1). The fetch loop folded Concurrent drain (F2). Factored the spawn/deadline logic into RAII cleanup (F3). The server and clone now live under a single Each fix is RED->GREEN with the guard reverted to confirm it is load-bearing. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/git-remote-gitlawb/tests/real_git_fetch.rs`:
- Around line 272-325: Update run_bounded so a timeout terminates the spawned
git fetch process and all descendant remote-helper processes, using
process-group termination or an equivalent descendant-reaping mechanism before
joining out_reader and err_reader. Preserve the existing completed result and
output collection behavior while ensuring inherited pipe handles are closed and
reader threads cannot hang.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 58807c87-026a-40d6-9bcf-6e2e1dfc5edd
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (3)
crates/git-remote-gitlawb/Cargo.tomlcrates/git-remote-gitlawb/src/main.rscrates/git-remote-gitlawb/tests/real_git_fetch.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/git-remote-gitlawb/src/main.rs
jatmn
left a comment
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Findings
-
[P2] Kill the entire fetch process tree before joining the drainers
crates/git-remote-gitlawb/tests/real_git_fetch.rs:305
On the timeout path this kills only the top-levelgit fetchprocess and then joins theread_to_endthreads. Itsgit-remote-gitlawbdescendant inherits those stdout/stderr pipe FDs, so a helper stuck in its HTTP request keeps the pipes open after Git dies; the joins then wait for that helper's 300-second request timeout instead of returning the promised 30-second deadlock result. Start the fetch in a process group and terminate/reap that group (or otherwise reap descendants) before joining the readers. This is also the unresolved current-head CodeRabbit request. -
[P2] Make the withheld-path integration test prove it reached the withheld response
crates/git-remote-gitlawb/tests/real_git_fetch.rs:518
The non-success branch accepts every promptgit fetchfailure and only prints the observed POST count. A regression that fails before the helper sends the first request to the shim—for example a broken advertisement, helper lookup, or connection—therefore passes as the expected rejected filtered pack, so this new test no longer verifies the path it claims to record. Require at least one POST (and preferably assert the expected protocol failure when a failure remains acceptable) before accepting the nonzero exit.
…hheld guard load-bearing (#192) Address jatmn's two P2 review findings on real_git_fetch.rs. F1 (INV-22): run_bounded killed only the top-level `git fetch` on the timeout path, so the git-remote-gitlawb helper descendant kept the inherited stdout/stderr pipe write-ends open and the reader joins blocked on the helper's ~300s HTTP timeout instead of the 30s deadline. Spawn the child with process_group(0) and, on timeout, tear the whole group down (SIGTERM, ESRCH poll, SIGKILL escalation, bounded cap, then reap) before joining, mirroring gitlawb-node/src/git/smart_http.rs. libc is added as a cfg(unix) dev-dep; non-unix keeps the child.kill() fallback so the always-on test still compiles. New regression run_bounded_reaps_descendants_holding_the_pipe: a backgrounded sleep inherits the pipe and outlives a 1s-deadline leader; reverting to a leader-only kill turns it RED (elapsed ~10s). F2 (INV-21): the non-success branch of real_git_withheld_shaped_first_post accepted any `git fetch` failure and only printed the POST count, so a failure before the first POST (broken advertisement, helper lookup, connection) passed as the expected withheld rejection. Assert posts >= 1 before accepting the nonzero exit. Forcing a pre-POST failure (0 POSTs) turns the assertion RED.
|
@jatmn both P2s addressed in 4a3f59c, each verified RED/GREEN by execution. F1 (kill the whole process tree before joining): F2 (prove the withheld test reached the withheld response): the non-success branch now asserts Full crate green (42 unit + 5 integration), fmt and workspace clippy clean. |
jatmn
left a comment
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Findings
-
[P2] Keep the large-output regression test portable to the shipped Windows target
crates/git-remote-gitlawb/tests/real_git_fetch.rs:558
This test unconditionally spawnssh -c 'seq …', but the release workflow shipsgit-remote-gitlawbforx86_64-pc-windows-msvcand neithershnorseqis a guaranteed Windows dependency. Consequentlycargo test -p git-remote-gitlawbfails atspawn childon a normal Windows checkout before it tests the drainer. Gate the shell-based fixture to Unix or replace it with a platform-native/Rust output generator. -
[P2] Terminate the remote-helper tree in the non-Unix timeout path
crates/git-remote-gitlawb/tests/real_git_fetch.rs:379
The Unix path creates and kills a process group because thegit-remote-gitlawbdescendant inherits Git's stdout/stderr pipe handles; the non-Unix fallback kills only the top-levelgitprocess and then joinsread_to_enddrainers. When a timed-out fetch leaves the helper blocked in its 300-second HTTP request, those inherited handles remain open and the joins wait for that timeout rather than returning after the promised 30 seconds. Use a Windows process-tree mechanism (for example a Job Object or equivalent) before joining the drainers, and cover that path on its supported platform.
…d windows target (#192) The large-output and descendants tests re-invoke the test binary as their fixtures instead of spawning sh/seq, so they run on x86_64-pc-windows-msvc; the non-unix timeout path tears down the whole fetch tree with taskkill /T before joining the drainers instead of killing only the leader. Also folds the receive-pack POST block into post_pack_round (review nitpick), removing the duplicated status-check/forward logic.
|
@jatmn both P2s addressed in 34040be, each verified by execution. F1 (portable large-output fixture): the test binary now re-invokes itself as the output generator (an F2 (non-unix tree teardown): the timeout branch now calls One honest limit: CI runs tests on ubuntu only, so the windows branch is compile-verified (the test file type-checks verbatim against Also folded the receive-pack POST block into |
jatmn
left a comment
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Findings
-
[P2] Count a multi-round negotiation as one fetch completion
crates/git-remote-gitlawb/src/main.rs:604
Each flush-terminated have batch now produces a successfulgit-upload-packPOST, but the server incrementsgitlawb_fetches_totalafter every successful POST even though that metric is documented as completed fetches. An incremental fetch with N negotiation rounds is therefore reported as N completed fetches (and the repeatedly accumulated request bodies are observed as separate pack sizes). Update the protocol/metrics boundary so intermediate stateless-RPC rounds are not counted as completed fetches, or redefine and separately expose request-level metrics. -
[P2] Keep the timeout in force after the fetch leader exits
crates/git-remote-gitlawb/tests/real_git_fetch.rs:393
try_wait()marks the run complete as soon as the top-levelgitprocess exits, then the code joins the stdout/stderr drainers without any deadline. A remote-helper descendant can still hold those inherited pipe write ends after its parent exits, making either join wait for the helper's 300-second HTTP timeout (or indefinitely) despite the requested bound. The existing process-group cleanup only runs when the leader is still alive at the deadline, so add coverage for a leader-exits-first descendant and keep/reap the process group until the pipe readers have completed within the deadline.
…RPC POST git_upload_pack recorded a fetch and observed a pack size on every successful POST. In HTTP stateless-RPC the plain serve path streams a pack only on the finalizing `done` round, so an N-round incremental fetch was reported as N completed fetches. The filtered path (upload_pack_excluding) instead serves a self-contained full pack on the single POST that reaches it, with no `done`, so gating purely on `done` would drop those private (path-scoped) fetches to zero. Count once on the POST that actually completes a fetch: the `done` round on the plain path (request_finalizes_fetch parses pkt-lines and fails closed on a malformed body), or the one filtered-serve POST (served_filtered_pack). should_count_fetch captures the decision; a completed fetch is exactly one such POST, so a multi-round negotiation is never double-counted. The observe_pack_size call still measures the request body, not the served pack; that mislabel predates this change and is left as a follow-up. Verified by execution: a three-round plain fetch and a no-`done` filtered serve each record exactly one completion; reverting the gate to count per POST makes the plain fetch record 3, and breaking the decision to AND drops both to 0.
…before a pipe-holder run_bounded only tore down the process group on the deadline path. When the git leader exits on its own but a descendant (the git-remote-gitlawb helper mid-HTTP request) outlives it holding the inherited stdout/stderr write-ends, the reader joins blocked on the helper's ~300s timeout, past the deadline run_bounded promises. Close the group on the clean-exit path too, guarded by a kill(-pgid, 0) liveness probe so a recycled pid is never signalled (the process-group ID stays reserved while any member is alive). Adds a leader-exits-first fixture and a regression test. Verified by execution: with the clean-path reap removed the new test blocks ~10s on the descendant (RED); with the reap it returns promptly.
|
Both addressed on [P2] Count a multi-round negotiation as one fetch completion. [P2] Keep the timeout in force after the fetch leader exits. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/gitlawb-node/src/api/repos.rs`:
- Around line 737-761: Update request_finalizes_fetch to recognize only 0000,
0001, and 0002 as zero-payload control packets; reject other lengths below 4 as
malformed. Track a saw_done flag instead of returning immediately when the
payload is done, continue validating every packet and trailing bytes, and return
saw_done only after the entire body has been parsed successfully.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 2ddc9d14-29e2-4d11-9ff1-977d95633f20
📒 Files selected for processing (3)
crates/git-remote-gitlawb/tests/real_git_fetch.rscrates/gitlawb-node/src/api/repos.rscrates/gitlawb-node/src/metrics.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/git-remote-gitlawb/tests/real_git_fetch.rs
jatmn
left a comment
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Findings
-
[P2] Count successful
no-donefetches
crates/gitlawb-node/src/api/repos.rs:709
The node advertisesmulti_ack_detailed no-done, so a normal smart-HTTP client may finish a flush-terminated have round by receivingACK <oid> readyfollowed immediately by the pack, without ever sending adonepkt-line. I reproduced that response with this checkout's exactgit upload-pack --stateless-rpcpath. In that successful case bothfinalizes_fetchandserved_filtered_packare false, so this gate records neithergitlawb_fetches_totalnor the pack observation. Please derive completion from the response/protocol outcome rather than assuming every plain fetch ends withdone, and add a no-doneregression. -
[P2] Do not count rejected filtered responses as completed fetches
crates/gitlawb-node/src/api/repos.rs:686
Settingserved_filtered_packfor everyupload_pack_excludingresponse labels the request as a completed fetch even when the client rejects the response. The committed real-Git scenario currently sends two POSTs and exits nonzero withexpected ACK/NAK; against this handler both filtered POSTs would set this flag and increment the completion counter, reporting one failed fetch as two successful completions. Until #191 makes this negotiation valid, gate the metric on a genuinely final request/outcome (fresh filtered clones carrydone) or use a request-level metric instead. -
[P2] Keep the clean-exit timeout bounded on Windows
crates/git-remote-gitlawb/tests/real_git_fetch.rs:431
The new leader-exits-first cleanup runs only undercfg(unix). On Windows, if the top-level Git process exits while its remote-helper descendant remains blocked in the 300-second HTTP request,completedbecomes true and execution goes straight to the reader-thread joins while that descendant still owns the pipe handles. The 30-second runner can therefore block for minutes; the matching regression is also Unix-only, andtaskkill /Tcannot recover the tree after the leader has exited. Use a Windows Job Object or another ownership mechanism established at spawn time, and exercise this ordering on Windows. -
[P2] Avoid racing the process-wide metrics initializer in tests
crates/gitlawb-node/src/api/repos.rs:1937
This adds a second test that callsmetrics::init, while the existing metrics test does the same andinituses an unsynchronizedREGISTRY.get()check followed by severalOnceLock::set(...).expect(...)calls. If both tests observe an empty registry, the loser panics atset INFO once(or a later lock); selecting these two tests with two test threads reproduced that panic immediately. Use an atomic one-time initializer such asget_or_init/Once, or make this test assert the pure decision without invoking the global initializer. -
[P2] Reject malformed pkt-line headers instead of treating them as flushes
crates/git-remote-gitlawb/src/main.rs:450
Both UTF-8 and hex parse failures are converted to zero, so a header such aszzzzis handled exactly like the valid0000flush. An invalid opener followed by EOF now returns success without a POST, and an invalid terminator after valid have lines can make those lines go out as a synthesized, potentially signed flush round. This masks/desynchronizes corrupted protocol input; propagate either parse error and add non-hex/non-UTF-8 cases alongside the existing0001test. -
[P2] Bind the withheld-path test to the expected rejection
crates/git-remote-gitlawb/tests/real_git_fetch.rs:736
Once one POST has reached the shim, this branch accepts every nonzero Git exit as the intended withheld-pack rejection. A truncated/corrupt response, later HTTP failure, or helper protocol regression after the first POST therefore leaves the test green even though the comments claim it proved the helper forwarded and terminated correctly. The current runtime error is specificallyexpected ACK/NAK; assert that failure signature (and the expected request/response shape) so unrelated post-POST failures cannot masquerade as the recorded #191 limitation.
…flushing A non-UTF-8 or non-hex 4-byte length prefix was collapsed to 0000 via unwrap_or, so a garbled header was handled exactly like a flush -- masking corrupted/desynchronized protocol input and letting accumulated have lines go out as a synthesized flush round. Propagate the parse error instead; a real 0000 still parses as a valid flush. (#192)
init() did an unsynchronized REGISTRY.get() check then OnceLock::set(...). expect(...) for each metric, so two concurrent callers both passed the check and the loser panicked at 'set INFO once'. Two tests call init(), so the parallel test runner could hit this. Guard the body with std::sync::Once so it runs exactly once. Verified: a 32-thread barrier stress-test panics before the fix and passes after (run in isolation; not committed since the process-global registry makes the race non-deterministic in the shared-binary suite). (#192)
…request The gitlawb_fetches_total gate keyed on whether the request carried `done`, which both undercounts a no-done completion (a client can finish a flush- terminated have round with `ACK <oid> ready` + pack and no `done` -> counted 0) and, on the filtered path, counts a rejected response per POST (the pre-#191 two-POST negotiation the client refuses was counted twice). Derive completion from the response instead. upload_pack / upload_pack_excluding now return whether they served a pack; response_served_pack detects the PACK magic in both side-band-64k and raw framing. The plain path counts when the response delivered a pack (catches no-done, skips negotiation-only rounds so an N-round fetch counts once); the filtered path stays gated on the finalizing `done` round until #191 makes that negotiation valid. Also tighten request_finalizes_fetch to reject 0003 framing and trailing bytes after the `done` pkt-line. (#192 F1/F2 + CodeRabbit repos.rs:761)
… rejection The test accepted any nonzero git exit (with posts>=1) as the recorded #191 withheld rejection, so a truncated response or a later HTTP failure after the first POST could pass green. Assert the specific `expected ACK/NAK` signature so only that break passes, and pin LC_ALL=C so the diagnostic stays the untranslated English string on a git build with l10n installed. (#192 F6)
|
Pushed
On the Windows clean-exit timeout: I'm leaving the Windows ownership mechanism for a follow-up rather than shipping it here. CI is Linux-only (every Re-requesting your review. |
jatmn
left a comment
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Findings
- [P2] Keep the clean-exit harness bounded on Windows
crates/git-remote-gitlawb/tests/real_git_fetch.rs:435
The clean-exit cleanup that prevents a remote-helper descendant from retaining the reader pipes is compiled only on Unix. On Windows,try_waitcan observe thegit fetchleader exiting whilegit-remote-gitlawbremains blocked in its 300-second HTTP request with inherited stdout/stderr handles; the following reader joins then wait for that descendant, despite the runner's 30-second bound.taskkill /Tis only usable while the leader remains alive, and the corresponding regression is Unix-only. Establish tree ownership at spawn time (for example, with a Job Object) and tear it down before joining, with a Windows regression.
…ws (#192) run_bounded's clean-exit descendant reap was Unix-only: on Windows a git fetch leader that exits cleanly leaves the git-remote-gitlawb helper blocked in its ~300s HTTP request holding the inherited pipes, and taskkill /T can't reach a descendant once the leader is gone, so the reader joins blew past the promised bound. The child is now assigned to a Job Object at spawn (the analog of the Unix process_group(0)), and the job is terminated on both the timeout and clean-exit paths, which kills the whole tree regardless of leader liveness. KILL_ON_JOB_CLOSE is the safety net. The clean-exit regression is now cross-platform so it exercises this on a future Windows CI lane (#228). Unix tests unchanged and green; the Windows path is compile-verified for x86_64-pc-windows-msvc (runtime coverage lands with #228's CI lane).
|
Implemented, reversing my earlier deferral on this. Pushed as Last round I punted the Windows clean-exit case as a follow-up on the grounds that CI is Linux-only. That was the wrong call, and you were right to keep pushing:
On verification, honestly: the Unix path is unchanged and green (44 + 6 tests, clippy/fmt clean). The Windows path is compile-verified for |
2 participants
Fixes #117.
git-remote-gitlawbadvertised theconnectcapability, so git spoke the stateful native protocol, buthandle_connectcollapsed the exchange into a single GET + single POST. A multi-round fetch (more than ~32 commits of overlapping history, the common "pull into an existing clone") deadlocked: git sent a flush-terminatedhavebatch and blocked for an ACK/NAK the helper never sent, while the helper blocked reading for adonegit never sent.Phase 2 for
git-upload-packis now a per-round v0 smart-HTTP stateless-RPC client loop, the role git's ownremote-curlplays.read_upload_pack_roundreturns one negotiation round at a time (at a flush,done, or EOF) instead of buffering past flushes;negotiate_upload_packcaptures the wants once and POSTs a self-contained request per flush-terminated batch (wants + one flush + every have accumulated so far + exactly one terminator), streams each ACK/NAK back to git, and stops on thedoneround. The node'sgit upload-pack --stateless-rpckeeps no state between POSTs, so wants and all prior haves are re-sent every round and no intermediate flush survives into a body.Signing is preserved per round: every POST carries the Phase-1 decision (signed after the 404 escalation for a private repo, anonymous for a public one), and a mid-negotiation denial surfaces through the sanitized error path rather than reading as an empty or successful fetch. The
git-receive-pack(push) path is unchanged, and no node code is touched.Verification
tests/real_git_fetch.rs) drives a realgit fetch(forced to >=2 rounds) through the built helper against a realgit upload-pack --stateless-rpc, so the stateful-to-stateless bridging is executed, not reasoned.Follow-up
The one case this does not make work is an incremental fetch of a repo with a path-scoped withheld subtree: the node's
upload_pack_excludingsends its pack on the first POST, which real git rejects mid-negotiation. The helper forwards and terminates cleanly, so it is a node-side issue, filed as #191. A committed test here (real_git_withheld_shaped_first_post) records that rejection so a node-side fix can verify against it.STRATEGY track: stabilize.
Summary by CodeRabbit
git-upload-pack --stateless-rpcnegotiation with correct per-round termination, streamed forwarding, and accumulated “haves/wants”.git fetchcoverage (multi-round boundaries, withheld-first behavior, negotiation denial, signing verification, and protocol edge cases).