feat(node,git): cap concurrent served git ops with a 503 load-shed (#62)

.env.example

@@ -24,8 +24,11 @@ DATABASE_URL=postgresql://gitlawb:changeme@localhost:5432/gitlawb
 # ── Database pool & startup resilience ────────────────────────────────────
 # Maximum connections in the PostgreSQL pool. A cap, not a floor —
 # connections open lazily. Size against the DB server's max_connections,
-# remembering admin tooling opens its own pool.
-GITLAWB_DB_MAX_CONNECTIONS=20
+# remembering admin tooling opens its own pool. Each concurrent write pins one
+# connection for its whole duration (the connection-affine advisory lock), so the
+# node REJECTS at boot any value below GITLAWB_MAX_CONCURRENT_GIT_PUSHES + 8
+# headroom — keep this comfortably above that (default 48 for pushes 32).
+GITLAWB_DB_MAX_CONNECTIONS=48
 # Seconds a request waits for a pool connection before failing with 503.
 GITLAWB_DB_ACQUIRE_TIMEOUT_SECS=5
 # Upper bound on each startup connect+migrate attempt, in seconds. Keep it
@@ -109,15 +112,96 @@ GITLAWB_MAX_PACK_BYTES=2147483648
 
 # Max seconds a served git upload-pack / receive-pack (clone / push) may run
 # before it is aborted with a 504. Bounds a hung git that would otherwise pin a
-# worker and, on push, the repo write lock. Does NOT cover the info/refs
-# advertisement or the withheld-blob path, which remain unbounded. Default 600.
+# worker and, on push, the repo write lock. Also bounds both info/refs
+# advertisements, the withheld-blob pack build, and the push-side candidate
+# discovery (rev-list / cat-file), all reaped via process-group teardown (#174).
+# Default 600.
 GITLAWB_GIT_SERVICE_TIMEOUT_SECS=600
 
+# Max seconds the storage-ACQUISITION phase of a served git op may run before the
+# request is shed with a 503, separate from the git-run timeout above. A
+# concurrency permit is taken before this phase and GITLAWB_GIT_SERVICE_TIMEOUT_SECS
+# only starts once git spawns, so without this a stalled backend (a hung Tigris
+# HEAD/GET, or a hung pg advisory-lock iteration on push) pins the permit and drains
+# the pool until every later request 503s. On expiry the permit is released
+# (fail-closed). Kept separate because acquisition and git execution are distinct
+# cost centers. Must be positive; set very large to effectively disable. Default 30.
+GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS=30
+
+# Max concurrent git READ ops (upload-pack + the upload-pack info/refs
+# advertisement) served at once, a global pool separate from the push pool below.
+# The anon receive-pack info/refs advertisement has its OWN pool (see below), not
+# this one. Over-cap sheds a clean 503 + Retry-After. Anonymous reads draw from
+# here, so pair it with GITLAWB_MAX_CONCURRENT_READS_PER_CALLER (below) so one
+# caller cannot monopolize the pool. Default 128.
+GITLAWB_MAX_CONCURRENT_GIT_OPS=128
+
+# Max concurrent git-receive-pack (push) POST operations, in a pool separate from
+# the read pool (GITLAWB_MAX_CONCURRENT_GIT_OPS) so anonymous reads cannot shed an
+# authenticated push at admission. The anon receive-pack info/refs advertisement
+# runs in a SEPARATE pool of the same size (disjoint from this one), so an
+# advertisement flood cannot shed a push either. Over-cap sheds a 503 +
+# Retry-After. Default 32.
+GITLAWB_MAX_CONCURRENT_GIT_PUSHES=32
+
+# Max concurrent post-push pin loops (IPFS + Pinata pin_new_objects) across all
+# repos. Each loop holds a full per-push object-id list while pinning, so this
+# bounds that MB-scale memory even though the per-repo encrypt-task set already
+# caps the task COUNT. A loop DEFERS (waits) when the pool is full, never drops a
+# pin. Default 8.
+GITLAWB_MAX_CONCURRENT_PIN_TASKS=8
+
+# Max concurrent read ops (upload-pack + the upload-pack info/refs advertisement)
+# a single caller may hold, so one caller cannot monopolize the read pool. Keyed
+# on the resolved SOURCE IP, never the DID: a signature does not move a caller off
+# this cap. The source-IP key is only as granular as GITLAWB_TRUSTED_PROXY below:
+# left unset, a node behind an edge/NAT keys all callers on the edge IP and this
+# collapses to one global cap. Set GITLAWB_TRUSTED_PROXY for per-client keying; a
+# high-fanout caller (CI behind one NAT) then needs the operator to raise this.
+# Default 16.
+GITLAWB_MAX_CONCURRENT_READS_PER_CALLER=16
+
 # ── Push rate limiting (git-receive-pack flood brake) ─────────────────────
 # Max receive-pack requests (info/refs advertisement + push POST) per client
 # IP per hour. 0 disables. Default 600.
 GITLAWB_PUSH_RATE_LIMIT=600
 
+# ── /ipfs/{cid} visibility-walk admission (#174) ──────────────────────────
+# GET /ipfs/{cid} runs a per-repo full-history git walk in a blocking thread to
+# decide whether the caller may read a path-scoped blob. It is publicly reachable,
+# so it is bounded to keep a permissionless caller from fanning out unbounded
+# concurrent walks and exhausting blocking-pool threads + PIDs.
+# Max concurrent /ipfs walks across all callers (a pool of its own, disjoint from
+# the served-git pools). Over-cap sheds a 503. Default 32.
+GITLAWB_MAX_CONCURRENT_IPFS_WALKS=32
+# Max concurrent /ipfs walks a single SOURCE IP may hold (keyed like the git
+# per-caller caps via GITLAWB_TRUSTED_PROXY; reject-before-insert bounded map).
+# Default 4.
+GITLAWB_IPFS_WALK_PER_SOURCE=4
+# Max EXPENSIVE path-scope visibility walks per single /ipfs request (only a
+# blob in a path-scoped repo costs a full-history walk). Over-cap repos are
+# skipped without a verdict and the scan continues; if the object is then found
+# nowhere the request sheds a retryable 503 instead of a false 404. Default 64.
+GITLAWB_IPFS_MAX_REPOS_WALKED=64
+# Ceiling on repos one /ipfs request may VISIT past the visibility gate. Each
+# visit costs a repo acquire — on a Tigris cache miss a full archive download,
+# so this is also the worst-case object-store fetch count per request — plus a
+# cat-file probe. On exhaustion the scan stops and sheds a retryable 503.
+# Default 1024.
+GITLAWB_IPFS_MAX_REPO_VISITS=1024
+# Absolute wall-clock budget for one admitted /ipfs request's acquire+walk
+# lifetime (all stages of the whole scan). Per-stage clamps bound the acquire
+# and walk stages to the remaining budget, and no stage starts once it is
+# exhausted (the scan then sheds a retryable 503). The object-type probe and
+# content-read cat-file subprocesses are budget-checked before starting but
+# have NO duration bound of their own: a hung git probe (corrupt pack, stuck
+# filesystem) holds the request's walk slot for the full duration of the hang.
+# Default 600.
+GITLAWB_IPFS_REQUEST_BUDGET_SECS=600
+# Max /ipfs/{cid} requests per client IP per hour (route flood brake, distinct
+# from the concurrency caps above). 0 disables. Default 600.
+GITLAWB_IPFS_RATE_LIMIT=600
+
 # ── Creation rate limiting (repo/agent/issue/PR flood brake) ──────────────
 # Max creation requests (POST /api/v1/repos, /api/register, fork, issues,
 # pulls) per client IP per hour, in addition to the per-DID limit. The per-DID

README.md

@@ -343,7 +343,14 @@ Important node settings:
 | `GITLAWB_REQUIRE_SIGNED_PEER_WRITES` | Require signed peer announce/sync writes. |
 | `GITLAWB_AUTO_SYNC` | Enable automatic sync from known peers. |
 | `GITLAWB_MAX_PACK_BYTES` | Max git pack body size for smart-HTTP routes. |
-| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack/receive-pack may run before it is aborted (504). Default 600. Does not bound `info/refs` or the withheld-blob path. |
+| `GITLAWB_GIT_SERVICE_TIMEOUT_SECS` | Max seconds a served git upload-pack, receive-pack, or `info/refs` advertisement may run before it is aborted (504). Default 600. Also bounds the withheld-blob classification walk (on both the upload-pack serve and receive-pack replication paths) and the push-side pin-candidate discovery (`rev-list` / `cat-file`), each reaped via process-group teardown at the deadline. |
+| `GITLAWB_GIT_ACQUIRE_TIMEOUT_SECS` | Max seconds the storage-acquisition phase (Tigris HEAD/GET, push advisory-lock) of a served git op may run before the request is shed with a 503, separate from the git-run timeout. The concurrency permit is released on expiry so a stalled backend cannot pin the pool. Default 30. |
+| `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. |
+| `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. |
+| `GITLAWB_IPFS_MAX_REPOS_WALKED` | Max expensive path-scope visibility walks per `/ipfs/{cid}` request; over-cap repos are skipped and the scan continues, shedding a retryable 503 (not a false 404) if the object is then found nowhere. Default 64. |
+| `GITLAWB_IPFS_MAX_REPO_VISITS` | Ceiling on repos one `/ipfs/{cid}` request may visit (acquire + probe) past the visibility gate — also the worst-case per-request Tigris fetch count. On exhaustion the scan stops with a retryable 503. Default 1024. |
+| `GITLAWB_IPFS_REQUEST_BUDGET_SECS` | Absolute wall-clock budget for one admitted `/ipfs/{cid}` request's acquire+walk lifetime. Per-stage clamps bound the acquire and walk stages to the remaining budget, and no stage starts once it is exhausted; the scan then stops with a retryable 503. The object-type probe and content-read `cat-file` subprocesses are budget-checked before starting but have no duration bound of their own, so a hung git probe (corrupt pack, stuck filesystem) holds the request's walk slot for the full duration of the hang. Default 600. |
+| `GITLAWB_IPFS_RATE_LIMIT` | Max `/ipfs/{cid}` requests per client IP per hour (route flood brake). 0 disables. Default 600. |
 | `GITLAWB_TIGRIS_BUCKET` | Optional S3/Tigris shared repo storage bucket. |
 | `GITLAWB_PINATA_JWT` | Optional Pinata/IPFS warm-storage pinning. |
 | `GITLAWB_IRYS_URL` | Optional Irys/Arweave permanent anchoring. |