Skip to content

feat(rules): publish the bot policy you enforce, and see attack payloads - #30

Merged
cport1 merged 1 commit into
mainfrom
feat/bot-policy-and-signatures
Aug 22, 2026
Merged

feat(rules): publish the bot policy you enforce, and see attack payloads#30
cport1 merged 1 commit into
mainfrom
feat/bot-policy-and-signatures

Conversation

@cport1

@cport1 cport1 commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Closes #728 and #732. Stacked on #29#28.

#728 — botPolicy(): publish the policy you enforce

This is the one on the list that's ours rather than catch-up. BOT_REGISTRY is generated from the Go registry and already carries the customer-facing categories; bots() already enforces against it. Nothing ever published from it — so every site that wanted to control AI crawlers hand-wrote a robots.txt that drifted from whatever the code did.

Drift there isn't cosmetic. A robots.txt disallowing GPTBot while the middleware lets it through is a policy the operator believes is in force and is not. The reverse — enforcing against a crawler the published file invites — is how a site quietly leaves a search index.

const policy = botPolicy({ deny: ['training_crawler'], allow: ['perplexitybot'] });

app.get('/robots.txt', (_req, res) => res.type('text/plain').send(policy.robotsTxt()));
const wd = new WebDecoy({ rules: [policy.rule()] });

Both come from one resolved set, and the headline test walks all 169 registry agents asserting the published file and the rule agree on every one.

The generated file also names the agents in your deny set whose operator does not document honouring robots.txt:

# These do not document honouring robots.txt, so the lines below are a
# request only. The bots() rule is what actually stops them:
#   ByteSpider (ByteDance)
#   Webz.io (Webz.io)

So the file itself says which of its own lines are voluntary. policy.unenforceable is the same list in code. I haven't found anyone else shipping that.

#732 — attackSignatures(): the checkbox, built to stay a checkbox

Tripwires catch scanners by the path they ask for. Nothing looked at what they send.

This is not a WAF and should not grow into one. A WAF's value is breadth, and breadth is bought with the false positives every incumbent's customers complain about — which is the opposite of the "deterministic evidence, you decide" position. So: a small curated set (SQLi, XSS, traversal, command injection, ${jndi:) where each signature has no innocent reading in a path or query. It earns its place by composing — an injection payload plus a tripwire hit is much stronger evidence than either alone, and both are facts rather than probabilities.

Three deliberate constraints:

  • Bodies and headers are opt-in. A CMS saving an article and a URL passed as a query parameter both legitimately contain things that look like attacks. Default is path + query only.
  • Cookie is never inspected, even with headers on. Session tokens are opaque and application-defined; one that trips a signature logs a user out for a reason nobody can explain.
  • No nested quantifiers, and input truncated at maxBytes. A regex that backtracks catastrophically on a crafted body turns a detection rule into the denial of service it exists to catch. There's a test that hammers it with 6KB of 'or'/<a /../ repeats.

Signatures are stricter than the obvious version: sqli_tautology requires the quote (so ?q=coffee or tea is clean), traversal requires two segments (so ?next=../dashboard is clean), and event handlers require a tag context (so ?onerror=1 is clean). All of that is pinned by a 17-case false-positive corpus of ordinary traffic — URLs as parameters, JWTs, JSON filters, price=10..50, a stray %.

Also

RequestMetadata.query is now populated by all three adapters. Express's req.path excludes the query string, which is exactly where injection payloads live — without this the rule would have been blind by construction. body is never populated automatically; buffering a body the app hasn't already parsed would change its streaming behaviour.

Verification

53 new tests, 394 total. Build, lint, check:edge green.

@cport1
cport1 force-pushed the feat/rate-limit-store branch 2 times, most recently from 014e46a to fba9c9a Compare August 22, 2026 02:14
@cport1
cport1 force-pushed the feat/bot-policy-and-signatures branch from 6e9c1fc to 5519de7 Compare August 22, 2026 02:14
@cport1
cport1 changed the base branch from feat/rate-limit-store to main August 22, 2026 02:16
botPolicy() closes a gap that was ours to close. BOT_REGISTRY is generated
from the Go registry and already carries the customer-facing categories,
and bots() already enforces against it -- but nothing published from it,
so every site that wanted to control AI crawlers hand-wrote a robots.txt
that drifted from whatever the code actually did.

Drift there is not cosmetic. A robots.txt disallowing GPTBot while the
middleware lets it through is a policy the operator believes is in force
and is not; the reverse is how a site quietly leaves a search index. Both
outputs now come from one resolved set, and a test walks all 169 registry
agents asserting the published file and the rule agree on every one.

The generated file also names the agents in the deny set whose operator
does not document honouring robots.txt, so it says which of its own lines
are merely a request and which are backed by the rule. Nobody else ships
that.

attackSignatures() is the checkbox we were missing, built to stay a
checkbox. It is not a WAF and should not grow into one: a WAF's value is
breadth, and breadth is bought with the false positives every incumbent's
customers complain about. This is a curated set where each signature has
no innocent reading in a path or query, and it earns its place by
composing with the deterministic signals -- an injection payload plus a
tripwire hit is much stronger than either alone.

Bodies and headers are opt-in because a CMS saving an article and a URL in
a query parameter both legitimately look like attacks. Cookie is never
inspected: session tokens are opaque, and one that trips a signature logs
a user out for a reason nobody can explain. Patterns are anchored or
literal with no nested quantifiers and input is truncated, so a crafted
body cannot make the rule the denial of service it exists to catch.

Adapters now populate RequestMetadata.query -- Express's req.path excludes
it, which is where the payloads are.

Closes WebDecoy/app#728
Closes WebDecoy/app#732
@cport1
cport1 force-pushed the feat/bot-policy-and-signatures branch from 5519de7 to 0dd49a0 Compare August 22, 2026 02:16
@cport1
cport1 merged commit 60b03a5 into main Aug 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant