From 6723f41e6525192e3d26751c2161183538b51d3a Mon Sep 17 00:00:00 2001 From: tamirms Date: Mon, 17 Aug 2026 15:50:14 +0200 Subject: [PATCH 1/3] =?UTF-8?q?docs:=20correct=20=C2=A77.7=20adversarial-k?= =?UTF-8?q?eys=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Block routing is unkeyed, so a random globalSeed does not mitigate adversarial keys as the spec claimed, and PreHash (unkeyed xxHash3-128) is not a defense either. Document the actual availability risk and prescribe a secret-keyed upstream transform (SipHash/HMAC) for adversary-influenced keys. Note the same in the README pre-hashing section. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Egmh6b54gbmrVTZ6CXxTUa --- README.md | 2 ++ streamhash-spec.md | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index cd676c5..938b751 100644 --- a/README.md +++ b/README.md @@ -156,6 +156,8 @@ if err := builder.Finish(); err != nil { rank, err := idx.QueryRank(streamhash.PreHash(originalKey)) ``` +> **Security note:** pre-hashing defends against *accidental* skew, not a deliberate adversary. `PreHash` is unkeyed, so an attacker can compute it too, and `WithGlobalSeed` does not affect block routing. If keys may be adversary-influenced, apply a secret-keyed transform (e.g. SipHash/HMAC) upstream and index that. See [`streamhash-spec.md`](streamhash-spec.md) §7.7. + ## Build Options | Option | Description | Default | diff --git a/streamhash-spec.md b/streamhash-spec.md index 434f290..07aebe7 100644 --- a/streamhash-spec.md +++ b/streamhash-spec.md @@ -883,8 +883,10 @@ For end-to-end integrity, the footer's two unseeded-xxHash64 region hashes (§3. StreamHash assumes uniformly random input. Consequences of violating that: -- **Non-uniform keys** cluster into blocks, overflowing temp-file regions (unsorted mode) — the region margin (§6.2) is calibrated for uniform keys. Pre-hash structured or correlated keys with xxHash3-128 (required, not optional). -- **Adversarial keys:** an attacker who knows `globalSeed` can craft keys that collide into one block. Mitigation: use a random `globalSeed` that untrusted sources cannot learn. +- **Non-uniform keys** cluster into blocks, overflowing a block's per-block cap (aborting the build) or the temp-file regions (unsorted mode) — both margins are calibrated for uniform keys. Pre-hash structured or correlated keys with xxHash3-128 (`PreHash`) to restore uniformity. This defends against *accidental* skew only, not a deliberate adversary (see below). +- **Adversarial keys.** Block routing is *unkeyed*: a key's block is `FastRange32(ReverseBytes64(key[0:8]), numBlocks)`, a public function of the key prefix and the (public) key count. `globalSeed` affects only within-block hashing and fingerprints; it does **not** participate in routing. A random `globalSeed` therefore does **not** mitigate adversarial keys, and neither does `PreHash` (unkeyed xxHash3-128, which the attacker can also compute). An attacker who can choose or grind keys can concentrate enough of them into one block to exceed its per-block cap, causing `ErrBlockOverflow` and aborting the build. This is a denial-of-service on index construction; it fails closed (no partial index is written, see §7.8) and does not affect lookup integrity. + + If keys may be adversary-influenced, apply a **secret-keyed** transform upstream — e.g. `key' = SipHash(secret, key)` or `HMAC(secret, key)` — and index `key'`. Keep `secret` unknown to the untrusted key source (and, per §7.6, outside the index file if that file is itself exposed to the source). Without the secret an attacker cannot predict a key's block and so cannot target one. - **Fingerprints** are a probabilistic filter (false-positive rate `2^(−8 × FingerprintSize)`), not a cryptographic authenticator. ### 7.8. Durability and compatibility From 89e775045a57f57a4f87db54ba0da2a3c378c8b0 Mon Sep 17 00:00:00 2001 From: tamirms Date: Mon, 17 Aug 2026 18:33:02 +0200 Subject: [PATCH 2/3] feat: export ErrBlockOverflow sentinel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-export sherr.ErrBlockOverflow so consumers can errors.Is on it — e.g. to rebuild a skewed/adversarial key set with a different routing transform instead of treating the overflow as fatal. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Egmh6b54gbmrVTZ6CXxTUa --- sentinels.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sentinels.go b/sentinels.go index 996f4ca..6a1733d 100644 --- a/sentinels.go +++ b/sentinels.go @@ -20,6 +20,13 @@ var ( ErrFingerprintTooLarge = sherr.ErrFingerprintTooLarge ErrSplitBucketSeedSearchFailed = sherr.ErrSplitBucketSeedSearchFailed ErrIndistinguishableHashes = sherr.ErrIndistinguishableHashes + + // ErrBlockOverflow signals that a block exceeded its per-block key cap + // during construction — non-uniform or adversarial keys concentrating into + // one block (see the security notes in streamhash-spec.md §7.7). Exported so + // consumers can errors.Is on it, e.g. to rebuild with a different routing + // transform instead of treating the failure as fatal. + ErrBlockOverflow = sherr.ErrBlockOverflow ) // Index errors. From ec99ea8ac0cac13dab93d2d30617a8f8c67d2052 Mon Sep 17 00:00:00 2001 From: tamirms Date: Wed, 26 Aug 2026 07:19:02 +0100 Subject: [PATCH 3/3] =?UTF-8?q?docs:=20rewrite=20=C2=A77.7=20adversarial-k?= =?UTF-8?q?eys=20guidance=20for=20readability=20and=20accuracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shorter, one-idea sentences per review on #14; remove jargon (not keyed / unkeyed / grind / fails closed / probabilistic filter); drop the inaccurate unsorted-mode temp-file overflow wording (the per-block cap raises ErrBlockOverflow in both modes); present the SipHash/HMAC mitigation as clear alternatives. Co-Authored-By: Claude Opus 4.8 --- streamhash-spec.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/streamhash-spec.md b/streamhash-spec.md index 07aebe7..22c3a3e 100644 --- a/streamhash-spec.md +++ b/streamhash-spec.md @@ -883,11 +883,18 @@ For end-to-end integrity, the footer's two unseeded-xxHash64 region hashes (§3. StreamHash assumes uniformly random input. Consequences of violating that: -- **Non-uniform keys** cluster into blocks, overflowing a block's per-block cap (aborting the build) or the temp-file regions (unsorted mode) — both margins are calibrated for uniform keys. Pre-hash structured or correlated keys with xxHash3-128 (`PreHash`) to restore uniformity. This defends against *accidental* skew only, not a deliberate adversary (see below). -- **Adversarial keys.** Block routing is *unkeyed*: a key's block is `FastRange32(ReverseBytes64(key[0:8]), numBlocks)`, a public function of the key prefix and the (public) key count. `globalSeed` affects only within-block hashing and fingerprints; it does **not** participate in routing. A random `globalSeed` therefore does **not** mitigate adversarial keys, and neither does `PreHash` (unkeyed xxHash3-128, which the attacker can also compute). An attacker who can choose or grind keys can concentrate enough of them into one block to exceed its per-block cap, causing `ErrBlockOverflow` and aborting the build. This is a denial-of-service on index construction; it fails closed (no partial index is written, see §7.8) and does not affect lookup integrity. +- **Non-uniform keys** collect in a small number of blocks. Each block has a maximum number of keys, sized for uniform input. Exceed it and the build stops with `ErrBlockOverflow`. If your keys are structured or correlated, hash them first with xxHash3-128 (`PreHash`); this restores uniformity. `PreHash` protects against *accidental* skew. It does not protect against an attacker — see the next item. +- **Adversarial keys.** A key's block is a fixed, public function of the key — no secret is involved: - If keys may be adversary-influenced, apply a **secret-keyed** transform upstream — e.g. `key' = SipHash(secret, key)` or `HMAC(secret, key)` — and index `key'`. Keep `secret` unknown to the untrusted key source (and, per §7.6, outside the index file if that file is itself exposed to the source). Without the secret an attacker cannot predict a key's block and so cannot target one. -- **Fingerprints** are a probabilistic filter (false-positive rate `2^(−8 × FingerprintSize)`), not a cryptographic authenticator. + ``` + blockIdx = FastRange32(ReverseBytes64(key[0:8]), numBlocks) + ``` + + The only inputs are the first 8 bytes of the key and the block count, both public, so anyone can compute which block any key lands in. `globalSeed` does not change this: it applies only to hashing inside a block and to fingerprints, so a random `globalSeed` gives no protection here. `PreHash` gives none either, because xxHash3-128 uses no secret and the attacker can compute it too. + + An attacker can therefore make many keys land in one block — either by choosing keys directly, or, when the keys are values they cannot choose (like transaction hashes), by generating many candidates and keeping only those that land in the target block. Enough of them exceeds the block's maximum, and the build stops with `ErrBlockOverflow`. This is a denial of service against index construction, and nothing worse: the build simply aborts and writes no index file (see §7.8), so the attack can only prevent an index from being built — it cannot corrupt one or cause wrong lookups. + + **Mitigation.** If an untrusted source can influence your keys, transform each key with a secret before indexing it — for example `key' = SipHash(secret, key)`, or any equivalent such as `HMAC(secret, key)`. Build the index over `key'` instead of `key`. Keep `secret` unknown to the untrusted source; if that source can read the index file, keep `secret` outside it (see §7.6). Without the secret an attacker cannot predict a key's block, and so cannot target one. ### 7.8. Durability and compatibility