test(node): invariant deny-prober (driven registry + completeness cross-check)

[beardthelion]

Stacks on #194 (base feat/real-node-deny-harness), reusing its real-node spawn rig and RFC-9421 signing client. Review after #194.

This turns the hand-written deny cases into a driven registry: one declarative table of every route that carries a runtime deny, and a sweep that boots a real node and drives each one hostile-then-authorized.

What's here

• Registry (tests/support/routes.rs): every deny-bearing route as a row (owner-gate 403, read-gate 404, signature 401), each classified by reading its handler, not the route name. 12 owner-gate + 1 signature + 14 repo-scoped read rows.
• Probe generator (tests/support/probe.rs): a two-repo fixture (public owner-gate substrate, private read-gate substrate, plus a seeded PR and issue so the author-or-owner close gates are reached rather than 404'd on an absent entity) and probes_for, which expands each row into the hostile request plus a positive twin so a deny for the wrong reason cannot false-pass.
• Runtime sweep (deny_bearing_registry_...): walks the registry against a real node. Hostile probe must return the exact deny and leak nothing; owner-gate twins must reach the handler (not 403); read twins must return a non-empty 2xx (an empty 2xx is a denial rendered as success). A terminal invariant asserts one hostile probe per row was actually driven, so the sweep can't pass by testing nothing.
• Completeness cross-check (completeness::..., no DB): a source scrape of server.rs (multiline .route parser, floor 90) that keeps the registry honest: no row points at a route that no longer mounts, and no handler carrying an owner-gate marker escapes the registry. Complements the in-crate authz_guard egress guard (which proves handlers are gated) by proving the deny-bearing ones are actually driven, and reaches the non-API git mounts it never sees.

Verification

Every deny class was mutation-verified load-bearing, RED-then-GREEN by execution: breaking did_matches drove the owner-gate hostile to 500 and, inverted, drove the owner twin to 403; neutering authorize_repo_read drove the read hostile to 200 and, as always-deny, the read twin to 404; the signature assertion was isolated at git-receive-pack; both completeness floors and both drift guards were tripped and named their target. All reverted. 22 harness tests green.

CI already runs this file on every PR (the --features test-harness --test deny_harness step added in #194), so U3 and U4 run there with no config change.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1c9c9840-eb36-4a0c-bce9-a786c436d57d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/invariant-deny-prober

---

_{Comment @coderabbitai help to get the list of available commands.}

[jatmn]

I found issues that need to be addressed before this is ready.

Findings

• [P2] Cover the deferred read-gate routes in the completeness invariant
  crates/gitlawb-node/tests/support/routes.rs:351
  The registry explicitly leaves out get_issue, get_pr, PR diff/review/comment reads, get_cert, get_bounty, get_encrypted_blob, and path-scoped get_tree, although these handlers return the same 404 authorization denial before looking up their entities. U3 only drives registry rows, and U4 checks mounted rows plus owner-gate markers only, so removing or bypassing authorize_repo_read on any of these endpoints leaves the new real-node sweep green. Seed the required entities and include the routes (with positive twins), or make the completeness check derive and require every mounted read-denying handler.

• [P2] Assert that hostile registry responses do not expose the private fixture data
  crates/gitlawb-node/tests/deny_harness.rs:507
  Every hostile probe calls assert_denied with an empty withheld-token list. That verifies only the status: a 403/404 body containing the private blob content, object ID, or repository metadata would pass even though this sweep claims the denial "leak[s] nothing." Carry the relevant private fixture tokens on the probes/rows and pass them to assert_denied for the affected read routes.

• [P2] Do not let inline owner gates bypass the registry drift check
  crates/gitlawb-node/tests/deny_harness.rs:788
  The orphan scan deliberately looks only for require_repo_owner(/require_owner( and excludes did_matches, but existing owner-only handlers such as protect_branch and unprotect_branch use the latter idiom. A future mounted owner-only handler using that established pattern can be omitted from deny_bearing_routes() while U4 still passes, so the advertised completeness guard does not keep the owner-gate registry complete. Extend the classifier to recognize the owner-specific inline pattern (or otherwise tie mounted owner routes to the registry) rather than excluding that gate class.

[beardthelion]

All three fixed; each guard now goes red when the protection it checks is removed.

Read-gate orphans — added a read-gate orphan guard symmetric to the owner one: it scans src/api for every authorize_repo_read / visibility_check handler and requires each to be either a driven ReadGate row or an enumerated READ_GATE_NOT_DRIVEN entry with a reason, replacing the free-text DEFERRED/EXCLUDED prose as the single source of truth. Deleting a driven row (get_repo) or an allowlist entry (get_cert) both fail it, and a staleness check rejects a dead exemption. The scan surfaced three repo-root reads the prose had missed (get_star_status, get_icaptcha_proof, replicate_encrypted_blobs), now allowlisted with a reason rather than silently dropped.

Sweep leak assertion — the private fixture now seeds a distinctive secret and captures its blob OID; the read-gate hostile probes carry those tokens into assert_denied (was &[]), so a 404/403 body echoing withheld content or the OID fails. Owner-gate (403) and signature (401) probes fire on NO_ENTITY repos before any entity lookup, so they keep an honest empty list.

Owner did_matches idiom — the owner-orphan scan now folds in has_owner_did_matches, matching the inline did_matches(caller, &record.owner_did) form protect_branch / unprotect_branch use, without matching the signer-self (register_replica) or author (close_pr) forms — pinned by unit tests on the exact bodies. Removing the protect_branch row now fails where it passed before.

[jatmn]

I found issues that need to be addressed before this is ready.

Findings

• [P2] Exercise read denial for a signed non-reader
  crates/gitlawb-node/tests/support/probe.rs:185
  Every ReadGate hostile probe is anonymous, although the fixture also creates a signed stranger. This leaves the signed-non-reader branch of every private read untested: a regression that keeps anonymous callers at 404 but treats any valid HTTP signature as authorized exposes private repositories while the registry sweep remains green. Generate a Signer::Stranger 404/no-leak hostile probe for each read row (and adjust the hostile-count invariant accordingly).

• [P2] Do not exempt probeable private root reads from the runtime sweep
  crates/gitlawb-node/tests/deny_harness.rs:1082
get_star_status, get_icaptcha_proof, and replicate_encrypted_blobs are allowlisted despite needing no additional fixture entity. The new guard only proves that their functions retain an authorize_repo_read(/visibility_check( marker; it never drives a request. Consequently an early return or conditional bypass that leaves the marker in the body can leak private-repo metadata while every new test passes. Add these as ReadGate rows with hostile and authorized probes rather than treating them as a source-only exemption.

• [P2] Keep the protected-push owner gate under a real probe
  crates/gitlawb-node/tests/deny_harness.rs:929
  The owner-marker scan recognizes the did_matches(&auth.0, &record.owner_did) check in git_receive_pack, then removes that handler because its registry row only tests the unsigned 401 path. A signed stranger pushing to a protected branch therefore never reaches a test; deleting or inverting the 403 at src/api/repos.rs:897-923 leaves both the sweep and completeness checks green. Add a signed-stranger protected-branch push probe with an owner control, or keep this gate in the completeness invariant until it is driven.

• [P2] Withhold private repository metadata in the denial-body assertion
  crates/gitlawb-node/tests/support/probe.rs:199
  The leak assertion carries only the private blob content and its object ID. Fixture::seed discards the private repository record/ID, so a 404 that serializes the private record's internal UUID or visibility metadata still passes assert_denied. That leaves the repository-metadata portion of the earlier no-leak request unverified. Preserve distinctive private repository metadata in the fixture, include it in the hostile probes' withheld tokens, and add the corresponding synthetic leaking-body check.

[beardthelion]

All four addressed on a5fa334, each guard proven load-bearing by reverting the exact production auth-line it protects.

Signed non-reader (F1). Every ReadGate row now drives a Signer::Stranger 404/no-leak probe alongside the anon one, and a dedicated readgate_stranger_driven counter asserts one per read row so a dropped probe fails loudly rather than being absorbed into the total. Reverting authorize_repo_read to authorize any signed caller makes the stranger probe leak (200) while the anon probe stays 404; dropping the stranger probe fails the counter.

Driven, not source-only (F2). get_star_status, get_icaptcha_proof, and replicate_encrypted_blobs are now real ReadGate rows and out of READ_GATE_NOT_DRIVEN. For each, a runtime bypass that keeps the authorize_repo_read/visibility_check marker but ignores the denial makes the driven probe leak (200) while the source scan stays green — the exact bypass a marker check cant see.

Protected-push 403 under a real probe (F3). Added signed_stranger_protected_branch_push_is_forbidden: a signed non-owner push to a protected branch expects 403, with an owner control that must not be 403 (plus a seed_protected_branch harness helper). Inverting the gate drops the stranger 403; denying all on a protected branch makes the owner control fire. git_receive_pack stays out of the owner-orphan scan because that gate is now driven here, not because it is unchecked.

Repo metadata withheld (F4). The fixture preserves the private repo id and the read-gate no-leak assertion withholds it; a synthetic 404 body carrying the UUID fails check_denied (it passed before the token was added). No live denial path leaks it today, so this is regression-hardening.

No production behavior ships — the auth-line reverts were RED-proof scaffolding, all restored.

[jatmn]

I found issues that need to be addressed before this is ready.

Findings

• [P1] Restore the configured shutdown-grace deadline
  crates/gitlawb-node/src/lib.rs:539
  This replaces the base branch's drive_serve_with_grace race with a plain with_graceful_shutdown(...).await, then explicitly discards grace at line 571. Axum will now wait indefinitely for a stalled in-flight request after SIGTERM, so teardown and a deployment restart can hang forever despite GITLAWB_SHUTDOWN_GRACE_SECS documenting that the server exits after that duration. Rebase onto the target (or restore the deadline race and its tests) rather than rolling this behavior back.

• [P1] Do not revert the atomic identity-key creation path
  crates/gitlawb-node/src/lib.rs:1007
  The target branch's create_new(...).mode(0o600)/AlreadyExists handling is replaced by exists() followed by fs::write() and a later chmod. Concurrent first starts can both generate and overwrite the same PEM, leaving running nodes with divergent DIDs; on Unix the PEM is also created with umask-derived permissions before line 1027 narrows it, exposing the private signing key during that window. Keep the atomic create, winner-load retry, and existing-key permission tightening from the base branch.

• [P2] Exercise both sides of the owner-or-author close rule
  crates/gitlawb-node/tests/support/probe.rs:65
  The fixture seeds each close target with author_did == owner_did, and the generated positive probe only signs as that same key. Consequently the new sweep passes if either close handler loses its is_author branch: a real contributor can no longer close their own PR/issue, but the owner-as-author control still reaches the handler and the stranger still receives 403. Seed these resources with a distinct author and add an author-signed positive probe (with the owner control retained) so the declared owner-or-author contract is actually load-bearing.

• [P2] Drive the remaining deferred private read endpoints
  crates/gitlawb-node/tests/deny_harness.rs:1073
get_issue, get_pr, the PR/comment/review reads, get_cert, get_bounty, get_encrypted_blob, and path-scoped get_tree are still source-only allowlist entries. The new guard only checks that their bodies contain an authorization-marker call, so an early return or ignored authorization result that leaves that call present keeps the guard green while exposing private data. This leaves the earlier request to cover the deferred 404-denying reads unresolved; seed the necessary entities and run their anonymous/signed-stranger denial and authorized positive probes.

[beardthelion]

Both P2s are fixed, and the same pass turned up a third instance of the class, so I generalized the invariant rather than patch the two.

Owner-or-author (probe.rs). The close rows were driven with only an owner twin while the fixture seeded author == owner, so the author arm was dead. Reworked into a general MultiPrincipalGate that carries its authorizing-principal set and drives a Not403 twin per arm plus the stranger 403. close_pr/close_issue now drive owner+author; reverting either is_owner or is_author on either handler turns that arm's twin red. The same rework surfaced dispute_bounty (creator-or-claimant) as the third untested-arm gate, so it's driven too. registry_internal_consistency now derives the required twin set from each row's declared arms, so a future multi-arm handler can't register with an arm untested.

Deferred reads (deny_harness). The ten READ_GATE_NOT_DRIVEN entries were source-scanned for a marker, never driven. Nine are now real ReadGate rows (anon/signed-stranger 404, owner 2xx), each carrying its own seeded marker in the withheld set, so a 404 that leaks that read's own content fails, not just a status check. get_encrypted_blob stays deferred under a closed structural reason (its owner twin needs a live IPFS backend the harness doesn't have), and the free-text reason list is now a closed enum. Added a completeness gate that derives the required read-gate set from the mounted route table, so a read that gates via an unrecognized helper can't slip past.

Every guard proven load-bearing by reverting the exact production line and observing red.

The two lib.rs findings (shutdown-grace, atomic-key) aren't in this PR's own diff, which is test-only. They're inherited from the base branch at 30672cf, a point before #194 landed its own grace and atomic-key fixes (both present on #194's head). They clear when this branch rebases onto #194's current tip.

Separately, I hit a pre-existing gap while generalizing the invariant: submit_bounty's claimant gate is vacuous under the existing every_in_scope_mutation_has_its_gate substring check. Out of scope here; filed as #FOLLOWUP.

[jatmn]

I found issues that need to be addressed before this is ready.

Findings

• [P2] Pin the object format in the temporary Git repositories
  crates/gitlawb-node/tests/support/probe.rs:506
  The new fixture initializes its worktree without an object-format argument, then compares its blob ID with the SHA-1 ID produced by seed_bare_repo(..., "sha1"). On a machine with init.defaultObjectFormat=sha256, this produces 64-character IDs and the assertion at line 517 fails before the registry sweep runs. The analogous helper in tests/deny_harness.rs:407 has the same assumption and will construct incompatible ref updates. Initialize both worktrees explicitly with --object-format=sha1, matching their served fixture repositories.

• [P2] Include the remaining identity-gated bounty mutations in the invariant sweep
  crates/gitlawb-node/tests/support/routes.rs:172
submit_bounty, approve_bounty, and cancel_bounty each return a runtime 403 for a non-claimant or non-creator, but none is a registry row or has a real-node hostile/control probe. The source guard in api/mod.rs only checks that their function text still contains did_matches(, so an ignored result or conditional bypass can expose these mutations while every new registry and completeness check stays green. Seed status-appropriate bounty fixtures and drive stranger denials plus claimant/creator controls for these routes, or make an explicit, enforced exclusion that does not claim the registry covers every runtime deny.

[beardthelion]

Both addressed on 07abcc6.

Object format (P2).

On a machine with init.defaultObjectFormat=sha256, this produces 64-character IDs and the assertion at line 517 fails before the registry sweep runs.

Both worktree inits now pass --object-format=sha1, with a comment tying the pin to the served sha1 fixture repos. One precision on the repro: git here is 2.43.0, which predates the init.defaultObjectFormat key (git 2.47), so that config alone is silently ignored; the same hazard reproduces via GIT_DEFAULT_HASH=sha256 (git 2.29+), and the pin closes both spellings. Observed before the fix: the registry sweep panics at probe.rs:518 with a 64-character blob OID against the fixture's sha1 OID, and with only the probe site pinned, get_pr_diff_withheld_path_is_denied independently fails from the second site's incompatible ref update (owner push 400 vs 200). After the fix, both tests pass under GIT_DEFAULT_HASH=sha256 and under the default config.

Bounty mutation gates (P2).

submit_bounty, approve_bounty, and cancel_bounty each return a runtime 403 for a non-claimant or non-creator, but none is a registry row or has a real-node hostile/control probe.

All three are now registry rows driven by execution. Each is a MultiPrincipalGate row (claimant arm for submit, creator arm for approve and cancel) with a signed-stranger hostile asserting exactly 403 and a Not403 twin per declared arm; the gate-class doc now covers one-or-more named principals, since the prober only ever required non-empty arms. These handlers check bounty status before the auth gate, so each row seeds its own bounty through the real API in the gate-reaching status (seed_bounty_at_stage: create for open, +claim for claimed, +submit for submitted); the twins then consume that status, which is why the rows cannot share a bounty. The hostile-count invariant recomputed from 66 to 69 and the multi-principal structural floor went from 3 to 6.

Load-bearing, by execution rather than inspection: disabling each handler's enforcement while keeping its did_matches( text makes the sweep fail with "must deny with 403, got 200 OK" naming exactly that row, for all three handlers. Under the submit mutation the api/mod.rs text guard stayed green while the sweep was red, which is precisely the #203 scenario; these gates no longer depend on that guard. Two more executed negatives: pointing the submit row at the wrong-status (open) bounty fails the sweep with "got 400", proving the status wiring is load-bearing and not decorative, and all three arm twins were observed reaching 200 on first execution. This covers the enforcement half of #203 for these three routes; the guard-hardening half of #203 stays open.

Full run: 488 unit plus 33 integration tests green, fmt and clippy clean.

[jatmn]

I found issues that need to be addressed before this is ready.

Findings

• [P2] Keep every real-node probe on a bounded client
  crates/gitlawb-node/tests/deny_harness.rs:145
  The new protected-branch probe (and the new PR-diff probe at line 488) uses reqwest::Client::new() instead of the suite's bounded client, so a wedged router, Git subprocess, or stalled response makes .send().await wait indefinitely until the CI job kills it. That hides precisely the route failure this harness is meant to report; use the shared timeout-bearing client for these real-node requests.

• [P2] Actually withhold the pushed PR-diff object's ID
  crates/gitlawb-node/tests/deny_harness.rs:541
oids is captured before secret/x.txt is created, so this lookup is always empty and the conditional never supplies an object ID to assert_denied. The test therefore checks the secret text but not the full or short SHA it promises to protect; a 404 path that exposes the withheld blob ID still passes. Have build_branch_push_body retain the new blob ID and assert that neither representation is present in the denial body.

• [P2] Do not describe the registry as covering every runtime denial while excluding signer-self gates
  crates/gitlawb-node/tests/support/routes.rs:1
  The PR describes a driven table of every route with a runtime deny, but this file deliberately omits signer-self 403 routes and the completeness checks only cover owner/read gates. For example, tasks::create_task rejects a signer who supplies somebody else's delegator_did; bypassing that did_matches result would let callers impersonate a delegator while the new sweep and completeness checks stay green. The same class exists in the task claim/complete/fail paths. Drive those gates in the real-node invariant, or narrow the registry and PR claims so they do not promise coverage they do not provide.