PROXY protocol v2 + downstream JA3/JA4 fingerprint forwarding#23
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for PROXY protocol v2 and downstream TLS fingerprint forwarding (JA3/JA4). Fingerprints are forwarded either via a PROXY v2 TLV or as injected HTTP headers (X-JA3/X-JA4) for plaintext HTTP/1 upstreams, with client-supplied headers stripped to prevent spoofing. Comprehensive integration tests and documentation updates are also included. Feedback is provided regarding a potential silent truncation issue in write_proxy_v2_header when casting the TLV length to u16, where using try_into() with proper error handling is recommended to ensure safety.
|
CI status: all 5 platform builds + every unit-test job (ubuntu Node 20/22, macOS-14 Node 20) are green. Two checks are red/cancelled for reasons unrelated to this change:
Nothing to action on those two. — KrAIs (Claude Opus 4.8) |
73c04c0 to
18bea07
Compare
DavidCockerill
left a comment
There was a problem hiding this comment.
Nice work on this — the v2 encoder is bounds-safe (u16 lengths guarded with try_into → io::Error, no truncation/panic/unwrap), header injection is correctly gated to plaintext HTTP/1, and it quietly fixes a pre-existing XFF-in-passthrough corruption along the way. Since this is PROXY v2 emission (not inbound parsing) and the emitted IP comes from the kernel peer_addr, the classic "client forges a PROXY header to spoof their source IP" vector doesn't apply here — good.
One significant gap worth closing, though, because it undercuts the PR's own "authoritative, can't be spoofed" claim for the fingerprint.
In plain terms: the backend is meant to trust X-JA3/X-JA4 as "symphony measured this," so symphony has to guarantee a client can't set that header themselves. It does strip client-supplied copies — but only within the first 8 KB it reads. If a client makes its request head bigger than 8 KB (trivial: long cookies, extra headers, padding) and puts its own X-JA3: line past byte 8192, the strip never sees it and it passes through verbatim. The backend then gets two fingerprint headers — symphony's real one and the client's forged one. On a last-header-wins backend that's a full spoof; on a comma-joining backend (Node's http) it's a corrupted value that can slip past a JA3/JA4 blocklist match. It's a detection/fraud signal rather than an auth boundary, and forwarding is opt-in, so I'm flagging rather than blocking — but it does defeat the guarantee the PR advertises.
Details / where: inject_request_headers (upstream.rs:220, strip loop :248) reads a single 8192-byte chunk; when the head exceeds it, the "No terminating CRLF in this chunk" branch forwards the remainder verbatim through the bidirectional copy.
Fix: strip X-JA3/X-JA4 (and X-Forwarded-For) across the entire header block, not just the first buffered read — or reject/normalize a request whose header block exceeds the scan window rather than letting the unscanned tail through.
Smaller, non-blocking:
- No runtime guard against
proxyProtocolV2being configured on a Harper-core UDS route (core's UDS reader is v1-only). The docs warn about it, but a hard error (or the existingeprintln!warning) would beat doc-only. - Perf: the rewritten head is emitted as several small writes — coalescing into one buffer + a single
write_allwould be cleaner and tighter underTCP_NODELAY. - v2 destination family-mismatch falls back to loopback:0 — benign (backends consume the source), noting for completeness.
— DAIvid (Claude Opus 4.8)
High — HTTP fingerprint headers remain client-spoofableInline on src/upstream.rs, around line 220:
Medium — Missing fingerprint permits direct spoofed-header pass-throughInline on src/proxy_conn.rs, around line 304:
Medium — Ineffective forwarding configurations deploy successfullyInline on src/proxy.rs, around line 542:
Medium — Private PROXY v2 TLVs lack an end-to-end consumer contractInline on src/upstream.rs, around PP2_TYPE_JA3 and PP2_TYPE_JA4:
Review summary
|
|
Comment 1: Un-timeouted Client Read leads to Connection Exhaustion (Slowloris DoS)
1 ### 🔴 Critical Security/Reliability: Un-timeouted Client Read leads to Connection Exhaustion (Slowloris DoS) 1 let mut buf = [0u8; 4096]; Comment 2: Header Spoofing / Strip Bypass via TCP Fragmentation
1 ### 1 Comment 3: Header Spoofing Bypass via Oversized Headers
1 ### 1 Any request headers that lie beyond the first 4096 bytes are not captured during this phase and are instead transferred verbatim by the subsequent bidirectional copy loop. |
Fixes two trust-boundary findings on the client→upstream HTTP/1 injection path (hdbjeff, PR #23): - Critical (Slowloris): the first-chunk client read in the old inject_request_headers ran outside the idle-timeout copy, so a client could complete the TLS handshake then stall forever. The header rewrite is now the client→upstream half of the bidirectional copy, which is wrapped in the same idle timeout — a post-TLS stall is dropped. - High (spoofable / partial-read corruption): the old code rewrote only a single ≤8 KiB read, so a spoofed X-JA3/X-JA4/X-Forwarded-For could survive by fragmenting across TCP segments, crossing the read boundary, or riding a later keep-alive request; a partial first read also prepended headers before a complete request line. Replaced with rewrite_request_stream, which frames a complete, bounded (64 KB) header block for *every* request — fragmented, pipelined, and keep-alive — stripping client copies and injecting the trusted values on each. Bodies are framed (Content-Length / chunked) only to locate the next request; TE+CL, obs-fold, and malformed framing are refused. Also always strips the configured fingerprint header even when symphony has no authoritative value to substitute (value: None), closing the empty-fingerprint pass-through gap. Reuses the existing http_proxy.rs framing (read_header_block factored out of read_http_headers) rather than the hand-rolled single-read parser. Tests: Rust unit tests for fragmentation, keep-alive, oversized-bound, partial-request-line, TE+CL/obs-fold rejection, strip-without-value; JS integration tests for post-TLS stall-drop, end-to-end spoof-strip, and per-request keep-alive strip. Co-Authored-By: Claude Opus <noreply@anthropic.com>
|
Addressed the two trust-boundary findings on the HTTP/1 injection path in Root cause: the original 🔴 Critical — un-timeouted read (Slowloris): the header rewrite is now the client→upstream half of the bidirectional copy, wrapped in the same idle timeout as the rest of the connection. A client that completes the TLS handshake and then stalls without sending a request is dropped when the idle timeout elapses. New integration test: drops a client that stalls post-TLS without sending a request.
Related (empty-fingerprint pass-through): a configured fingerprint header is now stripped from the client's request even when symphony has no authoritative value to substitute, so a client can't smuggle its own value through precisely when we have nothing to replace it with. Tests: Rust unit tests cover cross-read fragmentation, keep-alive, the 64 KB bound, partial-request-line, TE+CL / obs-fold rejection, and strip-without-value; JS integration tests cover post-TLS stall-drop, end-to-end spoof-strip, and per-request keep-alive strip. Full Rust + JS unit suites pass locally. The remaining Medium findings (config-time validation of ineffective carrier combinations; PROXY v2 private-TLV consumer contract/docs) are separate from these two and not included here — flagging for triage. — KrAIs (Claude Opus 4.8) |
heskew
left a comment
There was a problem hiding this comment.
The two flagged findings are genuinely closed here — the 8192-byte strip window is gone (full header-block parse, all requests rewritten) and the client read is now inside the idle timeout. Verified. But a focused second pass on the new parser surfaced a residual spoof path in the tunnel handling that undercuts the same anti-spoof invariant, plus a compat regression — both inline. (This second pass was Codex standing in for the Gemini leg, which failed on this PR.)
Also, one thing the new tunnel classification now leans on: is_upgrade only inspects the first Connection header, so a client that splits Connection: keep-alive / Connection: Upgrade across two fields is read as no-tunnel — a valid WebSocket/h2c client breaks after the 101. Worth folding a multi-field Connection parse in while this code is open.
🤖 Draft prepared with Claude (cross-model review pipeline; Codex second-model pass); reviewed and posted by @heskew.
…d on main) Rebase of kris/proxy-v2-fingerprint-9 over current main (h2 upstream dispatch #24, JA4/GREASE #20, http2 flag fix #26), squashed. Includes the review fixes: tunnel passthrough gated on the upstream's accept verdict, byte-level header parsing (obs-text safe), multi-field Connection upgrade detection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eebc088 to
2181578
Compare
|
@heskew both your findings (plus the multi-field 🤖 Claude Fable 5, on Kris's behalf |
…23 review) A route that sets forwardFingerprint in passthrough mode (terminateTls: false) without sourceAddressHeader='proxyProtocolV2' has no carrier — there's no HTTP request to inject an X-JA3/X-JA4 header into, and no v2 TLV. Previously the requested signal was silently discarded at runtime; build_route now logs a startup warning (fingerprint_has_no_carrier predicate, unit-tested). Other terminated modes can still inject the header for HTTP/1 connections, so they aren't flagged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
I reviewed the final implementation and test coverage. The previously reported HTTP-header spoofing paths are genuinely closed:
I found two new reliability issues: Medium — Request metadata queue is unbounded
Please use a bounded channel or cap outstanding pipelined requests so request forwarding receives backpressure. A test should demonstrate that the request pump stops advancing when the response side falls behind. Medium — Response framing only examines the first TE/CL fieldThe response parser uses
Please combine all Transfer-Encoding fields in wire order, require the final coding to be Existing findings still unresolvedThe two findings explicitly deferred in the follow-up remain open:
Test accuracyThe documented 64 KiB header limit is not exact: |
|
Follow-up issues from the post-merge review are now tracked:
The original fragmented/oversized/keep-alive header-spoofing fixes, empty-fingerprint stripping, timeout coverage, rejected Upgrade/CONNECT handling, multi-field Connection parsing, and obs-text handling were verified as closed in the final #23 code. The four issues above capture the remaining deferred work and new follow-up findings. |
Closes #9. Adds the two edge-proxy capabilities that issue tracks, bundled because PROXY v2's TLV is the natural carrier for the forwarded fingerprint.
Stacked on #20 (
kris/ja4-12) so this PR's diff is just my two commits — it carries JA3 and JA4 (per direction to ship both together). GitHub will auto-retarget tomainwhen #20 merges.What it does
write_proxy_v2_header) beside the v1 text one, selected per route viasourceAddressHeader: 'proxyProtocolV2'. v2 adds the TLV section used below. Stays opt-in; UDS default remains v1 (Harper core's own UDS reader parses v1 only).forwardFingerprint: 'ja3' | 'ja4' | 'none'hands symphony's computed ClientHello fingerprint to the upstream. Carrier follows the mode: a PROXY v2 TLV (0xE0=JA3 /0xE1=JA4, HAProxy private range) underproxyProtocolV2— works in passthrough since it prefixes the raw bytes — otherwise an injectedX-JA3/X-JA4header for terminated HTTP/1 upstreams. Motivating use case: Automattic wants the edge fingerprint passed through to their own app servers behind us.Where to look
src/upstream.rs—write_proxy_v2_header(byte layout: 12-byte sig +0x21+ family/proto + BE length + address block + TLV). Cross-model review verified it against the HAProxy 2.8 spec; unit tests cover IPv4+TLV, empty-TLV skip, IPv4-mapped-IPv6 collapse, IPv6.src/proxy_conn.rs—apply_source_headergates all HTTP-header injection onl7_http1(terminated TLS and non-h2 ALPN), so text is never spliced into passthrough TLS ciphertext or an HTTP/2 frame stream — a no-op otherwise.inject_request_headersstrips any client-supplied copy of an injected header (anti-spoof, so a client can't pre-seedX-JA3).Notes for the reviewer
toJsRoute/resolveConnectionints/proxy.tsnever forwarded thehttp2field — it was declared in types and parsed in Rust but silently dropped by the JS wrapper, sohttp2: truewas a no-op on that path. Fixed here (it's adjacent, and it's what makes the h2 carrier path reachable to test). Note the effect: any existing route carryinghttp2: truewill now genuinely negotiate h2. Confirmed with Kris that it stays bundled here rather than splitting out.agyleg hung both attempts, so there's no second-model perf/maintainability pass this run.Tests
Rust unit tests for the v2 encoder + header strip; integration tests for v2-TLV forwarding, no-TLV v2,
X-JA3injection, passthrough no-op, and h2 negotiation + no-injection. Full suite green (Rust 22, Node 39).Generated by an LLM (Claude Opus 4.8).