From 2da3449aef81ea420652d7a5a896c543efee518f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 17:10:01 +0000 Subject: [PATCH] Fix CIDR whitelist entries being discarded on hot reload A CIDR range added to /logs/whitelist.txt worked until the next reload, then silently stopped: Ignoring invalid whitelisted IP entry in /logs/whitelist.txt: 10.42.0.0/16 Ignoring invalid whitelisted IP entry in /logs/whitelist.txt: 129.215.0.0/16 The list compiler had two copies. docker-entrypoint.sh learned to route ranges into a `geo` map (273c827, 1ae044e), but health-monitor.sh kept a private pre-CIDR copy and recompiled the whitelist with it on every file change. So ranges were dropped on reload, and because the monitor never wrote whitelisted-cidrs.map at all, a range added after boot never took effect without a container restart. Both scripts now source ip-maps.sh, so they cannot drift again. The test suite includes static wiring assertions that fail the build if they do -- unit tests alone would not have caught the original bug, since each copy of the compiler was individually correct. Also fixes a second defect exposed by the same asymmetry: the auto-block exemption used an exact string match, so an address inside a whitelisted CIDR could trip a probe pattern and be auto-blocked despite its whole network being whitelisted, leaving the two map files disagreeing about the same address. It is now range-aware. Range membership is computed by division rather than 2^(32-bits): BusyBox can be built without libm, and there `^` aborts with "Math support is not compiled in", which made every in-range address read as out-of-range -- silently, and only on the alpine runtime image. 46 checks, run at image build time so a regression fails `docker build` rather than surfacing later as a cache bypass that quietly doesn't happen. Verified under BusyBox ash with a libm-less awk, dash, and bash in POSIX mode. README documents range support, the fact that rejected lines are logged, and X-Force-Refresh, which until now existed only as a code comment. Co-Authored-By: Claude Claude-Session: https://claude.ai/code/session_01Y8ZAhRM7gh8arDx9SuSWcT --- Dockerfile | 9 +- README.md | 21 ++++- docker-entrypoint.sh | 87 ++---------------- health-monitor.sh | 67 ++++---------- ip-maps.sh | 203 ++++++++++++++++++++++++++++++++++++++++++ test/ip-maps-test.sh | 207 +++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 459 insertions(+), 135 deletions(-) create mode 100755 ip-maps.sh create mode 100755 test/ip-maps-test.sh diff --git a/Dockerfile b/Dockerfile index 49a234e..3803518 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,16 +13,23 @@ ENV WORKER_RLIMIT_NOFILE=65535 ARG NGINX_CONF=nginx.conf.template COPY $NGINX_CONF /etc/nginx/nginx.conf.template +COPY ip-maps.sh /usr/local/bin/ip-maps.sh COPY health-monitor.sh /usr/local/bin/health-monitor.sh +COPY test/ip-maps-test.sh /usr/local/bin/ip-maps-test.sh COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh COPY purge-cached-404s.sh /usr/local/bin/purge-cached-404s.sh RUN mkdir -p /var/cache/nginx/owlery /logs/hacks && \ touch /logs/blocked.txt /logs/whitelist.txt /etc/nginx/blocked-ips.map /etc/nginx/whitelisted-ips.map /etc/nginx/whitelisted-cidrs.map && \ chown -R nginx:nginx /var/cache/nginx /logs && \ - chmod +x /usr/local/bin/health-monitor.sh /usr/local/bin/docker-entrypoint.sh /usr/local/bin/purge-cached-404s.sh && \ + chmod +x /usr/local/bin/health-monitor.sh /usr/local/bin/docker-entrypoint.sh /usr/local/bin/purge-cached-404s.sh /usr/local/bin/ip-maps-test.sh && \ apk add --no-cache gettext +# Fail the build rather than the deployment: a whitelist entry that the map +# compiler quietly discards is invisible until someone notices a cache bypass +# not happening, so the list-compilation logic is unit tested here. +RUN IP_MAPS_LIB=/usr/local/bin/ip-maps.sh /usr/local/bin/ip-maps-test.sh + EXPOSE 80 8080 VOLUME ["/var/cache/nginx", "/logs"] diff --git a/README.md b/README.md index 3b7396a..cddf4c2 100644 --- a/README.md +++ b/README.md @@ -109,7 +109,7 @@ Example response: - **Probe log output**: Refused probe requests are logged to `/logs/hacks/probes.log`, including both raw `X-Forwarded-For` and the extracted left-most client IP. - **Automatic scanner blocking**: When `AUTO_BLOCK_SCANNERS=true`, newly detected `client_ip` values in `/logs/hacks/probes.log` are appended to `/logs/blocked.txt` (unless already present or whitelisted), and NGINX is reloaded so the block takes effect without container restart. - **Manual IP blocklist**: Add one IPv4/IPv6 address per line in `/logs/blocked.txt` (comments allowed with `#`). -- **Manual IP whitelist**: Add one IPv4/IPv6 address per line in `/logs/whitelist.txt` (comments allowed with `#`). +- **Manual IP whitelist**: Add one IPv4/IPv6 address, or one CIDR range, per line in `/logs/whitelist.txt` (comments allowed with `#`). Ranges are matched as subnets, so a VPN or pod network whose addresses are reassigned per session can be whitelisted once instead of being re-added every time it changes. Example `/logs/blocked.txt`: @@ -125,13 +125,30 @@ Example `/logs/whitelist.txt`: 203.0.113.50 # trusted monitoring source 2001:db8::beef +# whole networks, matched as subnets +10.42.0.0/16 +2001:db8::/32 ``` +The blocklist takes single addresses only. A range there would be far more damaging to get wrong than an over-broad whitelist, so a CIDR line in `/logs/blocked.txt` is refused and logged rather than compiled. + Blocked IP requests return HTTP `403` and are logged to `/logs/hacks/blocked.log`. Whitelist entries take precedence over both the blocklist and probe filter. -Blocklist/whitelist entries are watched continuously by the runtime monitor. Updates to `/logs/blocked.txt` or `/logs/whitelist.txt` are converted into map files and applied via `nginx -s reload` within a few seconds. +Blocklist/whitelist entries are watched continuously by the runtime monitor. Updates to `/logs/blocked.txt` or `/logs/whitelist.txt` are converted into map files and applied via `nginx -s reload` within a few seconds. A rejected line is reported on the container log, so check there if an entry does not seem to take effect. + +### Per-request cache refresh + +A whitelisted caller may send `X-Force-Refresh: true` (`1`, `yes` and `on` also work) to bypass the cache for that one request. The upstream response is written into the same cache slot the request would otherwise have read, so the next ordinary caller gets the refreshed copy — this is `proxy_cache_bypass` without `proxy_no_cache`, and it is how the post-release VFBquery warmup tool refreshes entries without flushing the cache. + +The header is honoured only for addresses in `/logs/whitelist.txt`; from anywhere else it falls back to `FORCE_CACHE_REFRESH_ON_REQUEST` and is otherwise ignored. Nothing in the response says the header was refused, so confirm it took by reading `X-Cache-Status`: + +```bash +curl -sS -o /dev/null -D - -H 'X-Force-Refresh: true' 'https://v3-cached.virtualflybrain.org/some/path' | grep -i '^x-cache-status' +``` + +`BYPASS` means the request went upstream and rewrote the cache slot. `HIT` means the header was ignored and the caller is not whitelisted. ### Cache Headers diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index d2b9082..a8eeb61 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -10,6 +10,12 @@ WHITELIST_MAP="/etc/nginx/whitelisted-ips.map" # Plain `map` keys are literal strings and can't match a range. WHITELIST_CIDR_MAP="/etc/nginx/whitelisted-cidrs.map" +# List compilation is shared with health-monitor.sh, which recompiles the same +# two files on every change. See ip-maps.sh for why it is not inlined here. +IP_MAPS_LIB="${IP_MAPS_LIB:-/usr/local/bin/ip-maps.sh}" +# shellcheck source=ip-maps.sh +. "$IP_MAPS_LIB" + prepare_log_paths() { # Ensure required runtime directories exist, including fresh bind mounts/volumes. mkdir -p \ @@ -27,87 +33,6 @@ prepare_log_paths() { touch "$WHITELIST_CIDR_MAP" } -# Loopback and private IPs must never end up in the blocked map -- they only -# get there via spoofed X-Forwarded-For and would lock out the local health -# monitor and any other internal caller. -is_safe_to_block() { - case "$1" in - 127.*|::1) return 1 ;; - 10.*|192.168.*) return 1 ;; - 172.1[6-9].*|172.2[0-9].*|172.3[01].*) return 1 ;; - fc*|fd*) return 1 ;; - fe8*|fe9*|fea*|feb*) return 1 ;; - esac - return 0 -} - -generate_ip_map() { - source_file="$1" - target_map="$2" - label="$3" - tmp_map="$(mktemp /tmp/${label}-ips.XXXXXX)" - : > "$tmp_map" - - { - while IFS= read -r raw_line || [ -n "$raw_line" ]; do - line="$(printf '%s' "$raw_line" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//')" - [ -z "$line" ] && continue - - if printf '%s' "$line" | grep -Eq '^[0-9a-f:.]+$'; then - if [ "$label" = "blocked" ] && ! is_safe_to_block "$line"; then - printf 'Refusing to compile loopback/private IP into blocked map: %s\n' "$line" >&2 - continue - fi - printf '%s\n' "$line" - else - printf 'Ignoring invalid %s IP entry in %s: %s\n' "$label" "$source_file" "$raw_line" >&2 - fi - done < "$source_file" - } | sort -u | while IFS= read -r line; do - printf '%s 1;\n' "$line" >> "$tmp_map" - done - - mv "$tmp_map" "$target_map" -} - -# Compile whitelist.txt into TWO outputs: -# - plain IPs go to $ip_map (consumed by the existing nginx `map` block) -# - CIDR ranges go to $cidr_map (consumed by an nginx `geo` block) -# Routing happens by line shape; everything else is flagged invalid. -# Rancher pod-network ranges (10.42.0.0/16 by default for Canal/Flannel) are -# the canonical use case -- without CIDR support the warmup tool running from -# a pod can't be whitelisted for X-Force-Refresh. -generate_whitelist_maps() { - source_file="$1" - ip_map="$2" - cidr_map="$3" - tmp_ip="$(mktemp /tmp/whitelisted-ips.XXXXXX)" - tmp_cidr="$(mktemp /tmp/whitelisted-cidrs.XXXXXX)" - : > "$tmp_ip" - : > "$tmp_cidr" - - while IFS= read -r raw_line || [ -n "$raw_line" ]; do - line="$(printf '%s' "$raw_line" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//')" - [ -z "$line" ] && continue - - if printf '%s' "$line" | grep -Eq '^[0-9a-f:.]+/[0-9]+$'; then - # CIDR -- emitted in nginx `geo` syntax. - printf '%s 1;\n' "$line" >> "$tmp_cidr" - elif printf '%s' "$line" | grep -Eq '^[0-9a-f:.]+$'; then - # Plain IP -- emitted in nginx `map` syntax. - printf '%s 1;\n' "$line" >> "$tmp_ip" - else - printf 'Ignoring invalid whitelist entry in %s: %s\n' "$source_file" "$raw_line" >&2 - fi - done < "$source_file" - - # Dedupe each output independently; atomic publish via mv. - sort -u -o "$tmp_ip" "$tmp_ip" - sort -u -o "$tmp_cidr" "$tmp_cidr" - mv "$tmp_ip" "$ip_map" - mv "$tmp_cidr" "$cidr_map" -} - export UPSTREAM_SERVER="${UPSTREAM_SERVER:-owl.virtualflybrain.org:80}" export CACHE_MAX_SIZE="${CACHE_MAX_SIZE:-20g}" export CACHE_STALE_TIME="${CACHE_STALE_TIME:-6M}" diff --git a/health-monitor.sh b/health-monitor.sh index 7a471d8..8a01a9d 100644 --- a/health-monitor.sh +++ b/health-monitor.sh @@ -13,8 +13,15 @@ BLOCKLIST_SOURCE=${BLOCKLIST_SOURCE:-/logs/blocked.txt} BLOCKLIST_MAP=${BLOCKLIST_MAP:-/etc/nginx/blocked-ips.map} WHITELIST_SOURCE=${WHITELIST_SOURCE:-/logs/whitelist.txt} WHITELIST_MAP=${WHITELIST_MAP:-/etc/nginx/whitelisted-ips.map} +WHITELIST_CIDR_MAP=${WHITELIST_CIDR_MAP:-/etc/nginx/whitelisted-cidrs.map} AUTO_BLOCK_SCANNERS=${AUTO_BLOCK_SCANNERS:-true} +# List compilation is shared with docker-entrypoint.sh, which compiles the same +# two files once at startup. See ip-maps.sh for why it is not inlined here. +IP_MAPS_LIB=${IP_MAPS_LIB:-/usr/local/bin/ip-maps.sh} +# shellcheck source=ip-maps.sh +. "$IP_MAPS_LIB" + UPSTREAM_HOST=$(printf '%s' "$UPSTREAM_SERVER" | cut -d: -f1) UPSTREAM_PORT=$(printf '%s' "$UPSTREAM_SERVER" | cut -d: -f2) @@ -103,25 +110,9 @@ is_truthy() { esac } -is_valid_ip() { - printf '%s' "$1" | grep -Eq '^[0-9A-Fa-f:.]+$' -} - -# Loopback and private IPs cannot legitimately reach nginx as $remote_addr -# from the outside; if one shows up in the probe log it is because a scanner -# spoofed X-Forwarded-For. Auto-blocking such an IP locks out the local -# health monitor (and anything else on the same bridge network), so refuse. -is_safe_to_block() { - case "$1" in - 127.*|::1) return 1 ;; - 10.*|192.168.*) return 1 ;; - 172.1[6-9].*|172.2[0-9].*|172.3[01].*) return 1 ;; - fc*|fd*) return 1 ;; - fe8*|fe9*|fea*|feb*) return 1 ;; - esac - return 0 -} - +# Exact-match membership test, used to avoid appending a duplicate to +# blocked.txt. The whitelist side needs range awareness and uses +# is_ip_whitelisted from ip-maps.sh instead. is_ip_listed() { source_file="$1" ip="$2" @@ -139,42 +130,15 @@ prepare_security_paths() { "$(dirname "$BLOCKLIST_SOURCE")" \ "$(dirname "$BLOCKLIST_MAP")" \ "$(dirname "$WHITELIST_SOURCE")" \ - "$(dirname "$WHITELIST_MAP")" + "$(dirname "$WHITELIST_MAP")" \ + "$(dirname "$WHITELIST_CIDR_MAP")" touch "$PROBE_LOG" touch "$BLOCKLIST_SOURCE" touch "$BLOCKLIST_MAP" touch "$WHITELIST_SOURCE" touch "$WHITELIST_MAP" -} - -generate_ip_map() { - source_file="$1" - target_map="$2" - label="$3" - tmp_map="$(mktemp /tmp/${label}-ips.XXXXXX)" - : > "$tmp_map" - - { - while IFS= read -r raw_line || [ -n "$raw_line" ]; do - line="$(printf '%s' "$raw_line" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//')" - [ -z "$line" ] && continue - - if is_valid_ip "$line"; then - if [ "$label" = "blocked" ] && ! is_safe_to_block "$line"; then - printf 'Refusing to compile loopback/private IP into blocked map: %s\n' "$line" >&2 - continue - fi - printf '%s\n' "$line" - else - printf 'Ignoring invalid %s IP entry in %s: %s\n' "$label" "$source_file" "$raw_line" >&2 - fi - done < "$source_file" - } | sort -u | while IFS= read -r line; do - printf '%s 1;\n' "$line" >> "$tmp_map" - done - - mv "$tmp_map" "$target_map" + touch "$WHITELIST_CIDR_MAP" } reload_nginx() { @@ -194,7 +158,7 @@ sync_ip_maps_if_needed() { fi generate_ip_map "$BLOCKLIST_SOURCE" "$BLOCKLIST_MAP" "blocked" - generate_ip_map "$WHITELIST_SOURCE" "$WHITELIST_MAP" "whitelisted" + generate_whitelist_maps "$WHITELIST_SOURCE" "$WHITELIST_MAP" "$WHITELIST_CIDR_MAP" last_blocklist_signature="$blocklist_signature" last_whitelist_signature="$whitelist_signature" reload_nginx @@ -241,7 +205,8 @@ update_auto_blocklist_from_probe_log() { fi # Whitelisted IPs remain exempt even if they trigger probe patterns. - if is_ip_listed "$WHITELIST_SOURCE" "$ip"; then + # Range-aware: an address inside a whitelisted CIDR is exempt too. + if is_ip_whitelisted "$WHITELIST_SOURCE" "$ip"; then continue fi diff --git a/ip-maps.sh b/ip-maps.sh new file mode 100755 index 0000000..3181138 --- /dev/null +++ b/ip-maps.sh @@ -0,0 +1,203 @@ +#!/bin/sh +# shellcheck shell=sh +# +# Shared compilation of /logs/blocked.txt and /logs/whitelist.txt into the +# nginx map/geo files. +# +# This file exists because the logic had two copies. docker-entrypoint.sh +# compiles the lists once at container start; health-monitor.sh recompiles them +# every time one of the files changes. When the entrypoint learned to route +# CIDR ranges into a `geo` map, the monitor's private copy did not -- so a CIDR +# added to whitelist.txt worked until the next hot reload, then silently +# stopped, logging only "Ignoring invalid whitelisted IP entry". Both scripts +# now source this file, so they cannot drift again. +# +# POSIX sh only: this runs under BusyBox ash in nginx:*-alpine. + +# A bare address, v4 or v6. Deliberately loose -- nginx does the real parsing +# and rejects a malformed key at reload; this only has to separate "address" +# from "range" from "junk". +is_valid_ip() { + printf '%s' "$1" | grep -Eq '^[0-9a-f:.]+$' +} + +# An address with a prefix length: 10.42.0.0/16, 2001:db8::/32. +is_valid_cidr() { + printf '%s' "$1" | grep -Eq '^[0-9a-f:.]+/[0-9]+$' +} + +# Strip CR, lowercase hex, drop `#` comments, trim surrounding space. +normalise_list_line() { + printf '%s' "$1" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//' +} + +# Loopback and private IPs cannot legitimately reach nginx as $remote_addr from +# the outside; if one shows up in the probe log it is because a scanner spoofed +# X-Forwarded-For. Blocking such an address would lock out the local health +# monitor and anything else on the same bridge network, so refuse. +is_safe_to_block() { + case "$1" in + 127.*|::1) return 1 ;; + 10.*|192.168.*) return 1 ;; + 172.1[6-9].*|172.2[0-9].*|172.3[01].*) return 1 ;; + fc*|fd*) return 1 ;; + fe8*|fe9*|fea*|feb*) return 1 ;; + esac + return 0 +} + +# Compile a list of bare addresses into nginx `map` syntax. Used for the +# blocklist, which has no range support by design -- an over-broad block is far +# more damaging than an over-broad whitelist. +generate_ip_map() { + source_file="$1" + target_map="$2" + label="$3" + tmp_map="$(mktemp "/tmp/${label}-ips.XXXXXX")" + : > "$tmp_map" + + { + while IFS= read -r raw_line || [ -n "$raw_line" ]; do + line="$(normalise_list_line "$raw_line")" + [ -z "$line" ] && continue + + if is_valid_ip "$line"; then + if [ "$label" = "blocked" ] && ! is_safe_to_block "$line"; then + printf 'Refusing to compile loopback/private IP into blocked map: %s\n' "$line" >&2 + continue + fi + printf '%s\n' "$line" + elif is_valid_cidr "$line"; then + printf 'Ignoring %s CIDR range in %s (ranges are not supported here): %s\n' \ + "$label" "$source_file" "$raw_line" >&2 + else + printf 'Ignoring invalid %s IP entry in %s: %s\n' "$label" "$source_file" "$raw_line" >&2 + fi + done < "$source_file" + } | sort -u | while IFS= read -r line; do + printf '%s 1;\n' "$line" >> "$tmp_map" + done + + mv "$tmp_map" "$target_map" +} + +# Compile whitelist.txt into TWO outputs: +# - plain IPs go to $ip_map (consumed by the existing nginx `map` block) +# - CIDR ranges go to $cidr_map (consumed by an nginx `geo` block) +# Routing happens by line shape; everything else is flagged invalid. +# Rancher pod-network ranges (10.42.0.0/16 by default for Canal/Flannel) are +# the canonical use case -- without CIDR support the warmup tool running from +# a pod can't be whitelisted for X-Force-Refresh. +generate_whitelist_maps() { + source_file="$1" + ip_map="$2" + cidr_map="$3" + tmp_ip="$(mktemp /tmp/whitelisted-ips.XXXXXX)" + tmp_cidr="$(mktemp /tmp/whitelisted-cidrs.XXXXXX)" + : > "$tmp_ip" + : > "$tmp_cidr" + + while IFS= read -r raw_line || [ -n "$raw_line" ]; do + line="$(normalise_list_line "$raw_line")" + [ -z "$line" ] && continue + + if is_valid_cidr "$line"; then + # CIDR -- emitted in nginx `geo` syntax. + printf '%s 1;\n' "$line" >> "$tmp_cidr" + elif is_valid_ip "$line"; then + # Plain IP -- emitted in nginx `map` syntax. + printf '%s 1;\n' "$line" >> "$tmp_ip" + else + printf 'Ignoring invalid whitelist entry in %s: %s\n' "$source_file" "$raw_line" >&2 + fi + done < "$source_file" + + # Dedupe each output independently; atomic publish via mv. + sort -u -o "$tmp_ip" "$tmp_ip" + sort -u -o "$tmp_cidr" "$tmp_cidr" + mv "$tmp_ip" "$ip_map" + mv "$tmp_cidr" "$cidr_map" +} + +# Is an IPv4 address inside an IPv4 CIDR? Compared by network index rather than +# a bitwise AND: BusyBox awk has no and(), but integer division by the block +# size gives the same answer and stays inside awk's double precision (a /0 +# block is 2^32, well under 2^53). +# +# The block size is doubled in a loop rather than written as 2^(32-bits): +# BusyBox can be built without libm, and there the `^` operator dies with +# "Math support is not compiled in" -- which would have made every in-range +# address look out-of-range, silently, only on the runtime image. +ipv4_in_cidr() { + awk -v ip="$1" -v cidr="$2" ' + function tonum(a, p, n, i) { + n = split(a, p, ".") + if (n != 4) return -1 + for (i = 1; i <= 4; i++) { + if (p[i] !~ /^[0-9]+$/ || p[i] + 0 > 255) return -1 + } + return ((p[1] * 256 + p[2]) * 256 + p[3]) * 256 + p[4] + } + BEGIN { + slash = index(cidr, "/") + if (slash == 0) exit 1 + base = tonum(substr(cidr, 1, slash - 1)) + addr = tonum(ip) + bits = substr(cidr, slash + 1) + if (bits !~ /^[0-9]+$/) exit 1 + bits += 0 + if (base < 0 || addr < 0 || bits > 32) exit 1 + block = 1 + for (i = bits; i < 32; i++) block *= 2 + exit (int(base / block) == int(addr / block)) ? 0 : 1 + }' +} + +# Is this address exempt from auto-blocking? Checks exact entries and IPv4 +# ranges. Without the range check, an address inside a whitelisted CIDR could +# trip a probe pattern and get auto-blocked despite the operator having +# whitelisted its whole network -- the map files would then disagree about the +# same address. +# +# IPv6 ranges are not matched arithmetically here (no bitwise ops in BusyBox +# awk, and `::` compression makes textual prefix matching unsafe). Instead, if +# any IPv6 range is whitelisted at all, no IPv6 address is auto-blocked. That +# errs towards leaving a scanner unblocked rather than locking out a trusted +# host, which is the right way round: the probe filter still returns 403 either +# way, auto-blocking only saves the work of matching it. +is_ip_whitelisted() { + source_file="$1" + ip="$2" + + [ -f "$source_file" ] || return 1 + + candidate="$(normalise_list_line "$ip")" + [ -n "$candidate" ] || return 1 + + case "$candidate" in + *:*) candidate_is_v6=1 ;; + *) candidate_is_v6=0 ;; + esac + + while IFS= read -r raw_line || [ -n "$raw_line" ]; do + line="$(normalise_list_line "$raw_line")" + [ -z "$line" ] && continue + + [ "$line" = "$candidate" ] && return 0 + + is_valid_cidr "$line" || continue + + case "$line" in + *:*) + [ "$candidate_is_v6" -eq 1 ] && return 0 + ;; + *) + if [ "$candidate_is_v6" -eq 0 ] && ipv4_in_cidr "$candidate" "$line"; then + return 0 + fi + ;; + esac + done < "$source_file" + + return 1 +} diff --git a/test/ip-maps-test.sh b/test/ip-maps-test.sh new file mode 100755 index 0000000..55e3448 --- /dev/null +++ b/test/ip-maps-test.sh @@ -0,0 +1,207 @@ +#!/bin/sh +# Unit tests for ip-maps.sh. +# +# Run directly (`sh test/ip-maps-test.sh`) or let the Docker build run it -- +# the Dockerfile executes this before the image is finalised, so a regression +# here fails `docker build` rather than surfacing as a silently-ignored +# whitelist entry hours later in production. +# +# POSIX sh only, to match BusyBox ash in the runtime image. + +set -eu + +# Defaults to the copy next to this checkout; the image build points it at the +# installed copy instead. +IP_MAPS_LIB="${IP_MAPS_LIB:-$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd)/ip-maps.sh}" +# shellcheck source=ip-maps.sh disable=SC1090 +. "$IP_MAPS_LIB" + +WORK="$(mktemp -d /tmp/ip-maps-test.XXXXXX)" +trap 'rm -rf "$WORK"' EXIT + +failures=0 +checks=0 + +ok() { + checks=$((checks + 1)) + printf ' ok %s\n' "$1" +} + +fail() { + checks=$((checks + 1)) + failures=$((failures + 1)) + printf ' FAIL %s\n' "$1" +} + +assert_status() { + # assert_status + description="$1" + expected="$2" + shift 2 + if "$@" >/dev/null 2>&1; then + actual=0 + else + actual=1 + fi + if [ "$actual" = "$expected" ]; then + ok "$description" + else + fail "$description (expected status $expected, got $actual)" + fi +} + +assert_file_is() { + # assert_file_is + if [ "$(cat "$2")" = "$3" ]; then + ok "$1" + else + fail "$1" + printf ' expected: %s\n' "$3" + printf ' actual: %s\n' "$(cat "$2")" + fi +} + +echo 'line classification' +assert_status 'plain IPv4 is an IP' 0 is_valid_ip '129.215.105.177' +assert_status 'plain IPv6 is an IP' 0 is_valid_ip '2001:db8::beef' +assert_status 'CIDR is not a plain IP' 1 is_valid_ip '129.215.0.0/16' +assert_status 'IPv4 CIDR is a CIDR' 0 is_valid_cidr '10.42.0.0/16' +assert_status 'IPv6 CIDR is a CIDR' 0 is_valid_cidr '2001:db8::/32' +assert_status 'plain IP is not a CIDR' 1 is_valid_cidr '203.0.113.50' +assert_status 'hostname is neither' 1 is_valid_ip 'example.org' +assert_status 'missing prefix is not CIDR' 1 is_valid_cidr '10.42.0.0/' + +echo +echo 'whitelist compilation splits IPs from ranges' +# The regression this file exists for: 129.215.0.0/16 must land in the geo map, +# not be discarded as an invalid IP. +printf '%s\n' \ + '203.0.113.50' \ + '# the VPN pool' \ + '129.215.0.0/16' \ + '10.42.0.0/16' \ + '2001:DB8::BEEF' \ + '2001:db8::/32' \ + '' \ + ' 198.51.100.7 ' \ + 'not-an-address' \ + '203.0.113.50' \ + > "$WORK/whitelist.txt" +printf '192.0.2.1\r\n' >> "$WORK/whitelist.txt" + +generate_whitelist_maps "$WORK/whitelist.txt" "$WORK/ips.map" "$WORK/cidrs.map" 2>"$WORK/warnings" + +assert_file_is 'plain IPs compiled, deduped, sorted, CRLF and case normalised' \ + "$WORK/ips.map" \ +'192.0.2.1 1; +198.51.100.7 1; +2001:db8::beef 1; +203.0.113.50 1;' + +assert_file_is 'CIDR ranges compiled into the geo map' \ + "$WORK/cidrs.map" \ +'10.42.0.0/16 1; +129.215.0.0/16 1; +2001:db8::/32 1;' + +if grep -q 'not-an-address' "$WORK/warnings"; then + ok 'junk entry reported' +else + fail 'junk entry reported' +fi +if grep -q '129.215.0.0/16' "$WORK/warnings"; then + fail 'CIDR must not be reported as invalid' +else + ok 'CIDR not reported as invalid' +fi + +echo +echo 'blocklist compilation' +printf '%s\n' '203.0.113.10' '127.0.0.1' '10.0.0.5' '198.51.100.0/24' > "$WORK/blocked.txt" +generate_ip_map "$WORK/blocked.txt" "$WORK/blocked.map" blocked 2>"$WORK/blocked-warnings" +assert_file_is 'public IPs blocked, loopback and RFC1918 refused, range refused' \ + "$WORK/blocked.map" '203.0.113.10 1;' +if grep -q 'ranges are not supported' "$WORK/blocked-warnings"; then + ok 'blocklist range rejected with a specific message' +else + fail 'blocklist range rejected with a specific message' +fi + +echo +echo 'IPv4 range membership' +assert_status 'inside a /16' 0 ipv4_in_cidr '129.215.105.177' '129.215.0.0/16' +assert_status 'outside a /16' 1 ipv4_in_cidr '129.216.105.177' '129.215.0.0/16' +assert_status 'inside a /24' 0 ipv4_in_cidr '10.42.0.99' '10.42.0.0/24' +assert_status 'outside a /24' 1 ipv4_in_cidr '10.42.1.99' '10.42.0.0/24' +assert_status 'first address of range' 0 ipv4_in_cidr '10.42.0.0' '10.42.0.0/16' +assert_status 'last address of range' 0 ipv4_in_cidr '10.42.255.255' '10.42.0.0/16' +assert_status 'one past the range' 1 ipv4_in_cidr '10.43.0.0' '10.42.0.0/16' +assert_status 'one before the range' 1 ipv4_in_cidr '10.41.255.255' '10.42.0.0/16' +assert_status '/32 matches exactly' 0 ipv4_in_cidr '203.0.113.50' '203.0.113.50/32' +assert_status '/32 rejects neighbour' 1 ipv4_in_cidr '203.0.113.51' '203.0.113.50/32' +assert_status '/0 matches anything' 0 ipv4_in_cidr '8.8.8.8' '0.0.0.0/0' +assert_status 'top of the address space' 0 ipv4_in_cidr '255.255.255.255' '255.255.255.0/24' +assert_status 'octet over 255 rejected' 1 ipv4_in_cidr '10.42.0.256' '10.42.0.0/16' +assert_status 'non-numeric rejected' 1 ipv4_in_cidr 'example.org' '10.42.0.0/16' +assert_status 'IPv6 candidate rejected' 1 ipv4_in_cidr '2001:db8::1' '10.42.0.0/16' + +echo +echo 'auto-block exemption honours ranges' +printf '%s\n' '203.0.113.50' '129.215.0.0/16' > "$WORK/wl-v4.txt" +assert_status 'exact entry exempt' 0 is_ip_whitelisted "$WORK/wl-v4.txt" '203.0.113.50' +assert_status 'address inside range exempt' 0 is_ip_whitelisted "$WORK/wl-v4.txt" '129.215.105.177' +assert_status 'address outside range not exempt' 1 is_ip_whitelisted "$WORK/wl-v4.txt" '198.51.100.9' +assert_status 'trailing comment tolerated' 0 is_ip_whitelisted "$WORK/wl-v4.txt" '203.0.113.50 # office' +assert_status 'IPv6 not exempt via IPv4 range' 1 is_ip_whitelisted "$WORK/wl-v4.txt" '2001:db8::1' +assert_status 'missing file is not exempt' 1 is_ip_whitelisted "$WORK/nope.txt" '203.0.113.50' + +printf '%s\n' '2001:db8::/32' > "$WORK/wl-v6.txt" +assert_status 'IPv6 range present means no IPv6 auto-block' 0 \ + is_ip_whitelisted "$WORK/wl-v6.txt" '2001:db8::1' +assert_status 'IPv6 range does not exempt IPv4' 1 \ + is_ip_whitelisted "$WORK/wl-v6.txt" '198.51.100.9' + +: > "$WORK/wl-empty.txt" +assert_status 'empty whitelist exempts nothing' 1 is_ip_whitelisted "$WORK/wl-empty.txt" '203.0.113.50' + +echo +echo 'callers are wired to the shared library' +# The original bug was not in the compiler but in who called it: the monitor +# had its own copy and recompiled the whitelist with the IP-only function, so +# every hot reload dropped the ranges the entrypoint had accepted at boot. +# These checks fail the build if the two ever diverge again. +SCRIPT_DIR="$(CDPATH='' cd -- "$(dirname -- "$IP_MAPS_LIB")" && pwd)" +for script in health-monitor.sh docker-entrypoint.sh; do + path="$SCRIPT_DIR/$script" + if [ ! -f "$path" ]; then + fail "$script found next to ip-maps.sh" + continue + fi + ok "$script found next to ip-maps.sh" + + if grep -q '^\. "\$IP_MAPS_LIB"' "$path"; then + ok "$script sources the shared library" + else + fail "$script sources the shared library" + fi + + if grep -qE '^(is_valid_ip|is_safe_to_block|generate_ip_map|generate_whitelist_maps)\(\)' "$path"; then + fail "$script keeps no private copy of the compiler" + else + ok "$script keeps no private copy of the compiler" + fi + + if grep -q 'generate_whitelist_maps "\$WHITELIST_SOURCE" "\$WHITELIST_MAP" "\$WHITELIST_CIDR_MAP"' "$path"; then + ok "$script compiles the whitelist with CIDR routing" + else + fail "$script compiles the whitelist with CIDR routing" + fi +done + +echo +if [ "$failures" -eq 0 ]; then + printf '%s checks passed\n' "$checks" +else + printf '%s of %s checks FAILED\n' "$failures" "$checks" + exit 1 +fi