serve: integration harness + fixes for --trackDeps over-reporting and JGit fetch failures#413
Merged
Merged
Conversation
… hits The serve query service returned inflated /impacted_targets sets whenever it was started with --trackDeps and served a `from` revision from the per-SHA hash cache. TargetHash is a data class whose generated equals() includes the `deps` field, but TargetHash.fromJson (used to read cache/JSON entries) only parses type#hash~directHash and always leaves deps=null. A cache-miss `to` (freshly generated, deps populated under --trackDeps) compared against a cache-hit `from` (deserialized, deps=null) therefore reported every dep-carrying target as changed even when its hash was identical. C1->C2 (both freshly generated) was correct; C2->C3 / C2->C4 (with `from` served from cache) were not. The CLI never hit this because both sides are deserialized, hence symmetric. Fix: compare targets by their hash identity (type#hash~directHash) rather than the full TargetHash in computeSimpleImpactedTargets and computeAllDistances -- `deps` is auxiliary data used only to derive distance metrics, not part of a target's impactedness. Adds regression tests for both the impacted and distance paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JGit 5.13 cannot fetch some clone shapes native git handles fine -- notably shallow
(--depth) and partial (--filter=blob:none) clones, whose thin packs are
delta-compressed against objects absent from the clone ("Missing delta base <sha>").
With the on-demand refetch added in #411, such a fetch now surfaces to callers as a
400.
JGitClient.fetch() now retries a failed in-process fetch with the native `git` binary
(at --gitPath) before giving up, and logs a one-time warning at startup if the
workspace is a shallow or partial clone. JGit stays the default for resolve/checkout.
If native git also fails, both causes are combined so neither is hidden.
--gitEngine=subprocess still skips the in-process attempt entirely.
Adds JGitClientTest coverage (fallback-disabled surfaces the JGit error; both-fail
names the native fallback; a shallow clone can still fetch a just-landed commit) and
documents the behavior in the serve README section.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds tools/serve_harness.py, a standalone integration harness that exercises `bazel-diff serve` end to end against a live git:// remote (git daemon) across both git engines x full/shallow/partial clones and the on-demand refetch path -- coverage the in-process E2ETest cannot reach. It builds //cli:bazel-diff, drives the real binary over HTTP, and asserts on the deterministic rule+source impacted subset (generated-file targets are filtered as noisy). This harness is what surfaced the --trackDeps over-reporting bug fixed earlier in this branch. Wired into CI as .github/workflows/serve-harness.yml, cron-only for now (12h schedule + workflow_dispatch) until it proves stable on runners; promote to push/PR gating afterward. The harness gained a --bazel flag so CI can point it at bazelisk. It is intentionally not a py_binary (a nested `bazel run` would deadlock on the output-base lock). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the
bazel-diff servequery service. Adds a standalone integration harness that exercisesserveend to end (which surfaced a real correctness bug), plus two fixes:--trackDepsover-reporting —serve --trackDepsreturned inflated/impacted_targetssets on cache hits.Missing delta base; it now falls back to native git.The
--trackDepsbug (commit 1)TargetHashis adata classwhose generatedequals()includes thedepsfield, butTargetHash.fromJson(used to read cache entries) only parsestype#hash~directHashand always leavesdeps = null. So a cache-missto(freshly generated,depspopulated under--trackDeps) compared against a cache-hitfrom(deserialized,deps = null) reported every dep-carrying target as changed even when its hash was identical:{core, core.txt, mid}fromcached){core, core.txt, mid}+ otherfromcached){other, other.txt}+ core, midAnyone running
serve --trackDeps(required for/impacted_targets_with_distances) got over-broad impacted sets on cache hits. The CLI never hit it (both sides deserialized → symmetric). Fixed by comparing hash identity (type#hash~directHash), not the fullTargetHash, incomputeSimpleImpactedTargetsandcomputeAllDistances.JGit fetch fallback (commit 2)
JGit 5.13 can't fetch shallow (
--depth) or partial (--filter=blob:none) clones whose thin packs are delta-compressed against absent objects (Missing delta base <sha>) — the class of the reported production error.JGitClient.fetch()now retries with the nativegitbinary (at--gitPath) on any in-process fetch failure, and warns once at startup if the clone is shallow/partial. JGit stays the default for resolve/checkout;--gitEngine=subprocessstill skips the in-process attempt.Harness + CI (commit 3)
tools/serve_harness.pydrives the real binary over HTTP against a livegit://remote (git daemon) across both git engines × full/shallow/partial clones + the on-demand refetch path — coverageE2ETest#testServeEndToEnd(a single local--no-initial-fetchrepo) can't reach. Runpython3 tools/serve_harness.py(--only,--skip-build,--keep-artifacts,-v,--bazel).Wired into CI as
.github/workflows/serve-harness.yml, cron-only for now (12hschedule+workflow_dispatch) while it proves stable on runners; promote to push/PR gating later (one-line change, documented at the top of the workflow). Intentionally not apy_binary— a nestedbazel runwould deadlock on the output-base lock.Testing
--bazel).CalculateImpactedTargetsInteractorTest(deps-asymmetry, both endpoints),JGitClientTest(fallback-disabled surfaces the JGit error; both-fail names the native fallback; shallow-clone fetch).//clisuite green except 4E2ETestsub-tests that need a system JDK for their nestedlocal_jdktoolchain (environment-only, unrelated to this change) —testServeEndToEndand the coreget-impacted-targetstests pass.🤖 Generated with Claude Code