feat(rules): publish the bot policy you enforce, and see attack payloads - #30
Merged
Conversation
cport1
force-pushed
the
feat/rate-limit-store
branch
2 times, most recently
from
August 22, 2026 02:14
014e46a to
fba9c9a
Compare
cport1
force-pushed
the
feat/bot-policy-and-signatures
branch
from
August 22, 2026 02:14
6e9c1fc to
5519de7
Compare
botPolicy() closes a gap that was ours to close. BOT_REGISTRY is generated from the Go registry and already carries the customer-facing categories, and bots() already enforces against it -- but nothing published from it, so every site that wanted to control AI crawlers hand-wrote a robots.txt that drifted from whatever the code actually did. Drift there is not cosmetic. A robots.txt disallowing GPTBot while the middleware lets it through is a policy the operator believes is in force and is not; the reverse is how a site quietly leaves a search index. Both outputs now come from one resolved set, and a test walks all 169 registry agents asserting the published file and the rule agree on every one. The generated file also names the agents in the deny set whose operator does not document honouring robots.txt, so it says which of its own lines are merely a request and which are backed by the rule. Nobody else ships that. attackSignatures() is the checkbox we were missing, built to stay a checkbox. It is not a WAF and should not grow into one: a WAF's value is breadth, and breadth is bought with the false positives every incumbent's customers complain about. This is a curated set where each signature has no innocent reading in a path or query, and it earns its place by composing with the deterministic signals -- an injection payload plus a tripwire hit is much stronger than either alone. Bodies and headers are opt-in because a CMS saving an article and a URL in a query parameter both legitimately look like attacks. Cookie is never inspected: session tokens are opaque, and one that trips a signature logs a user out for a reason nobody can explain. Patterns are anchored or literal with no nested quantifiers and input is truncated, so a crafted body cannot make the rule the denial of service it exists to catch. Adapters now populate RequestMetadata.query -- Express's req.path excludes it, which is where the payloads are. Closes WebDecoy/app#728 Closes WebDecoy/app#732
cport1
force-pushed
the
feat/bot-policy-and-signatures
branch
from
August 22, 2026 02:16
5519de7 to
0dd49a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #728 and #732. Stacked on #29 → #28.
#728 —
botPolicy(): publish the policy you enforceThis is the one on the list that's ours rather than catch-up.
BOT_REGISTRYis generated from the Go registry and already carries the customer-facing categories;bots()already enforces against it. Nothing ever published from it — so every site that wanted to control AI crawlers hand-wrote arobots.txtthat drifted from whatever the code did.Drift there isn't cosmetic. A
robots.txtdisallowing GPTBot while the middleware lets it through is a policy the operator believes is in force and is not. The reverse — enforcing against a crawler the published file invites — is how a site quietly leaves a search index.Both come from one resolved set, and the headline test walks all 169 registry agents asserting the published file and the rule agree on every one.
The generated file also names the agents in your deny set whose operator does not document honouring robots.txt:
So the file itself says which of its own lines are voluntary.
policy.unenforceableis the same list in code. I haven't found anyone else shipping that.#732 —
attackSignatures(): the checkbox, built to stay a checkboxTripwires catch scanners by the path they ask for. Nothing looked at what they send.
This is not a WAF and should not grow into one. A WAF's value is breadth, and breadth is bought with the false positives every incumbent's customers complain about — which is the opposite of the "deterministic evidence, you decide" position. So: a small curated set (SQLi, XSS, traversal, command injection,
${jndi:) where each signature has no innocent reading in a path or query. It earns its place by composing — an injection payload plus a tripwire hit is much stronger evidence than either alone, and both are facts rather than probabilities.Three deliberate constraints:
Cookieis never inspected, even with headers on. Session tokens are opaque and application-defined; one that trips a signature logs a user out for a reason nobody can explain.maxBytes. A regex that backtracks catastrophically on a crafted body turns a detection rule into the denial of service it exists to catch. There's a test that hammers it with 6KB of'or'/<a/../repeats.Signatures are stricter than the obvious version:
sqli_tautologyrequires the quote (so?q=coffee or teais clean), traversal requires two segments (so?next=../dashboardis clean), and event handlers require a tag context (so?onerror=1is clean). All of that is pinned by a 17-case false-positive corpus of ordinary traffic — URLs as parameters, JWTs, JSON filters,price=10..50, a stray%.Also
RequestMetadata.queryis now populated by all three adapters. Express'sreq.pathexcludes the query string, which is exactly where injection payloads live — without this the rule would have been blind by construction.bodyis never populated automatically; buffering a body the app hasn't already parsed would change its streaming behaviour.Verification
53 new tests, 394 total. Build, lint,
check:edgegreen.