Skip to content

Fix CIDR whitelist entries being discarded on hot reload #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Robbie1977 merged 1 commit into from
Aug 1, 2026
Merged

Fix CIDR whitelist entries being discarded on hot reload #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,23 @@ ENV WORKER_RLIMIT_NOFILE=65535

ARG NGINX_CONF=nginx.conf.template
COPY $NGINX_CONF /etc/nginx/nginx.conf.template
COPY ip-maps.sh /usr/local/bin/ip-maps.sh
COPY health-monitor.sh /usr/local/bin/health-monitor.sh
COPY test/ip-maps-test.sh /usr/local/bin/ip-maps-test.sh
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
COPY purge-cached-404s.sh /usr/local/bin/purge-cached-404s.sh

RUN mkdir -p /var/cache/nginx/owlery /logs/hacks && \
touch /logs/blocked.txt /logs/whitelist.txt /etc/nginx/blocked-ips.map /etc/nginx/whitelisted-ips.map /etc/nginx/whitelisted-cidrs.map && \
chown -R nginx:nginx /var/cache/nginx /logs && \
chmod +x /usr/local/bin/health-monitor.sh /usr/local/bin/docker-entrypoint.sh /usr/local/bin/purge-cached-404s.sh && \
chmod +x /usr/local/bin/health-monitor.sh /usr/local/bin/docker-entrypoint.sh /usr/local/bin/purge-cached-404s.sh /usr/local/bin/ip-maps-test.sh && \
apk add --no-cache gettext

# Fail the build rather than the deployment: a whitelist entry that the map
# compiler quietly discards is invisible until someone notices a cache bypass
# not happening, so the list-compilation logic is unit tested here.
RUN IP_MAPS_LIB=/usr/local/bin/ip-maps.sh /usr/local/bin/ip-maps-test.sh

EXPOSE 80 8080

VOLUME ["/var/cache/nginx", "/logs"]
Expand Down
21 changes: 19 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Example response:
- **Probe log output**: Refused probe requests are logged to `/logs/hacks/probes.log`, including both raw `X-Forwarded-For` and the extracted left-most client IP.
- **Automatic scanner blocking**: When `AUTO_BLOCK_SCANNERS=true`, newly detected `client_ip` values in `/logs/hacks/probes.log` are appended to `/logs/blocked.txt` (unless already present or whitelisted), and NGINX is reloaded so the block takes effect without container restart.
- **Manual IP blocklist**: Add one IPv4/IPv6 address per line in `/logs/blocked.txt` (comments allowed with `#`).
- **Manual IP whitelist**: Add one IPv4/IPv6 address per line in `/logs/whitelist.txt` (comments allowed with `#`).
- **Manual IP whitelist**: Add one IPv4/IPv6 address, or one CIDR range, per line in `/logs/whitelist.txt` (comments allowed with `#`). Ranges are matched as subnets, so a VPN or pod network whose addresses are reassigned per session can be whitelisted once instead of being re-added every time it changes.

Example `/logs/blocked.txt`:

Expand All @@ -125,13 +125,30 @@ Example `/logs/whitelist.txt`:
203.0.113.50
# trusted monitoring source
2001:db8::beef
# whole networks, matched as subnets
10.42.0.0/16
2001:db8::/32
```

The blocklist takes single addresses only. A range there would be far more damaging to get wrong than an over-broad whitelist, so a CIDR line in `/logs/blocked.txt` is refused and logged rather than compiled.

Blocked IP requests return HTTP `403` and are logged to `/logs/hacks/blocked.log`.

Whitelist entries take precedence over both the blocklist and probe filter.

Blocklist/whitelist entries are watched continuously by the runtime monitor. Updates to `/logs/blocked.txt` or `/logs/whitelist.txt` are converted into map files and applied via `nginx -s reload` within a few seconds.
Blocklist/whitelist entries are watched continuously by the runtime monitor. Updates to `/logs/blocked.txt` or `/logs/whitelist.txt` are converted into map files and applied via `nginx -s reload` within a few seconds. A rejected line is reported on the container log, so check there if an entry does not seem to take effect.

### Per-request cache refresh

A whitelisted caller may send `X-Force-Refresh: true` (`1`, `yes` and `on` also work) to bypass the cache for that one request. The upstream response is written into the same cache slot the request would otherwise have read, so the next ordinary caller gets the refreshed copy — this is `proxy_cache_bypass` without `proxy_no_cache`, and it is how the post-release VFBquery warmup tool refreshes entries without flushing the cache.

The header is honoured only for addresses in `/logs/whitelist.txt`; from anywhere else it falls back to `FORCE_CACHE_REFRESH_ON_REQUEST` and is otherwise ignored. Nothing in the response says the header was refused, so confirm it took by reading `X-Cache-Status`:

```bash
curl -sS -o /dev/null -D - -H 'X-Force-Refresh: true' 'https://v3-cached.virtualflybrain.org/some/path' | grep -i '^x-cache-status'
```

`BYPASS` means the request went upstream and rewrote the cache slot. `HIT` means the header was ignored and the caller is not whitelisted.

### Cache Headers

Expand Down
87 changes: 6 additions & 81 deletions docker-entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ WHITELIST_MAP="/etc/nginx/whitelisted-ips.map"
# Plain `map` keys are literal strings and can't match a range.
WHITELIST_CIDR_MAP="/etc/nginx/whitelisted-cidrs.map"

# List compilation is shared with health-monitor.sh, which recompiles the same
# two files on every change. See ip-maps.sh for why it is not inlined here.
IP_MAPS_LIB="${IP_MAPS_LIB:-/usr/local/bin/ip-maps.sh}"
# shellcheck source=ip-maps.sh
. "$IP_MAPS_LIB"

prepare_log_paths() {
# Ensure required runtime directories exist, including fresh bind mounts/volumes.
mkdir -p \
Expand All @@ -27,87 +33,6 @@ prepare_log_paths() {
touch "$WHITELIST_CIDR_MAP"
}

# Loopback and private IPs must never end up in the blocked map -- they only
# get there via spoofed X-Forwarded-For and would lock out the local health
# monitor and any other internal caller.
is_safe_to_block() {
case "$1" in
127.*|::1) return 1 ;;
10.*|192.168.*) return 1 ;;
172.1[6-9].*|172.2[0-9].*|172.3[01].*) return 1 ;;
fc*|fd*) return 1 ;;
fe8*|fe9*|fea*|feb*) return 1 ;;
esac
return 0
}

generate_ip_map() {
source_file="$1"
target_map="$2"
label="$3"
tmp_map="$(mktemp /tmp/${label}-ips.XXXXXX)"
: > "$tmp_map"

{
while IFS= read -r raw_line || [ -n "$raw_line" ]; do
line="$(printf '%s' "$raw_line" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//')"
[ -z "$line" ] && continue

if printf '%s' "$line" | grep -Eq '^[0-9a-f:.]+$'; then
if [ "$label" = "blocked" ] && ! is_safe_to_block "$line"; then
printf 'Refusing to compile loopback/private IP into blocked map: %s\n' "$line" >&2
continue
fi
printf '%s\n' "$line"
else
printf 'Ignoring invalid %s IP entry in %s: %s\n' "$label" "$source_file" "$raw_line" >&2
fi
done < "$source_file"
} | sort -u | while IFS= read -r line; do
printf '%s 1;\n' "$line" >> "$tmp_map"
done

mv "$tmp_map" "$target_map"
}

# Compile whitelist.txt into TWO outputs:
# - plain IPs go to $ip_map (consumed by the existing nginx `map` block)
# - CIDR ranges go to $cidr_map (consumed by an nginx `geo` block)
# Routing happens by line shape; everything else is flagged invalid.
# Rancher pod-network ranges (10.42.0.0/16 by default for Canal/Flannel) are
# the canonical use case -- without CIDR support the warmup tool running from
# a pod can't be whitelisted for X-Force-Refresh.
generate_whitelist_maps() {
source_file="$1"
ip_map="$2"
cidr_map="$3"
tmp_ip="$(mktemp /tmp/whitelisted-ips.XXXXXX)"
tmp_cidr="$(mktemp /tmp/whitelisted-cidrs.XXXXXX)"
: > "$tmp_ip"
: > "$tmp_cidr"

while IFS= read -r raw_line || [ -n "$raw_line" ]; do
line="$(printf '%s' "$raw_line" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//')"
[ -z "$line" ] && continue

if printf '%s' "$line" | grep -Eq '^[0-9a-f:.]+/[0-9]+$'; then
# CIDR -- emitted in nginx `geo` syntax.
printf '%s 1;\n' "$line" >> "$tmp_cidr"
elif printf '%s' "$line" | grep -Eq '^[0-9a-f:.]+$'; then
# Plain IP -- emitted in nginx `map` syntax.
printf '%s 1;\n' "$line" >> "$tmp_ip"
else
printf 'Ignoring invalid whitelist entry in %s: %s\n' "$source_file" "$raw_line" >&2
fi
done < "$source_file"

# Dedupe each output independently; atomic publish via mv.
sort -u -o "$tmp_ip" "$tmp_ip"
sort -u -o "$tmp_cidr" "$tmp_cidr"
mv "$tmp_ip" "$ip_map"
mv "$tmp_cidr" "$cidr_map"
}

export UPSTREAM_SERVER="${UPSTREAM_SERVER:-owl.virtualflybrain.org:80}"
export CACHE_MAX_SIZE="${CACHE_MAX_SIZE:-20g}"
export CACHE_STALE_TIME="${CACHE_STALE_TIME:-6M}"
Expand Down
67 changes: 16 additions & 51 deletions health-monitor.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,15 @@ BLOCKLIST_SOURCE=${BLOCKLIST_SOURCE:-/logs/blocked.txt}
BLOCKLIST_MAP=${BLOCKLIST_MAP:-/etc/nginx/blocked-ips.map}
WHITELIST_SOURCE=${WHITELIST_SOURCE:-/logs/whitelist.txt}
WHITELIST_MAP=${WHITELIST_MAP:-/etc/nginx/whitelisted-ips.map}
WHITELIST_CIDR_MAP=${WHITELIST_CIDR_MAP:-/etc/nginx/whitelisted-cidrs.map}
AUTO_BLOCK_SCANNERS=${AUTO_BLOCK_SCANNERS:-true}

# List compilation is shared with docker-entrypoint.sh, which compiles the same
# two files once at startup. See ip-maps.sh for why it is not inlined here.
IP_MAPS_LIB=${IP_MAPS_LIB:-/usr/local/bin/ip-maps.sh}
# shellcheck source=ip-maps.sh
. "$IP_MAPS_LIB"

UPSTREAM_HOST=$(printf '%s' "$UPSTREAM_SERVER" | cut -d: -f1)
UPSTREAM_PORT=$(printf '%s' "$UPSTREAM_SERVER" | cut -d: -f2)

Expand Down Expand Up @@ -103,25 +110,9 @@ is_truthy() {
esac
}

is_valid_ip() {
printf '%s' "$1" | grep -Eq '^[0-9A-Fa-f:.]+$'
}

# Loopback and private IPs cannot legitimately reach nginx as $remote_addr
# from the outside; if one shows up in the probe log it is because a scanner
# spoofed X-Forwarded-For. Auto-blocking such an IP locks out the local
# health monitor (and anything else on the same bridge network), so refuse.
is_safe_to_block() {
case "$1" in
127.*|::1) return 1 ;;
10.*|192.168.*) return 1 ;;
172.1[6-9].*|172.2[0-9].*|172.3[01].*) return 1 ;;
fc*|fd*) return 1 ;;
fe8*|fe9*|fea*|feb*) return 1 ;;
esac
return 0
}

# Exact-match membership test, used to avoid appending a duplicate to
# blocked.txt. The whitelist side needs range awareness and uses
# is_ip_whitelisted from ip-maps.sh instead.
is_ip_listed() {
source_file="$1"
ip="$2"
Expand All @@ -139,42 +130,15 @@ prepare_security_paths() {
"$(dirname "$BLOCKLIST_SOURCE")" \
"$(dirname "$BLOCKLIST_MAP")" \
"$(dirname "$WHITELIST_SOURCE")" \
"$(dirname "$WHITELIST_MAP")"
"$(dirname "$WHITELIST_MAP")" \
"$(dirname "$WHITELIST_CIDR_MAP")"

touch "$PROBE_LOG"
touch "$BLOCKLIST_SOURCE"
touch "$BLOCKLIST_MAP"
touch "$WHITELIST_SOURCE"
touch "$WHITELIST_MAP"
}

generate_ip_map() {
source_file="$1"
target_map="$2"
label="$3"
tmp_map="$(mktemp /tmp/${label}-ips.XXXXXX)"
: > "$tmp_map"

{
while IFS= read -r raw_line || [ -n "$raw_line" ]; do
line="$(printf '%s' "$raw_line" | tr -d '\r' | tr 'A-F' 'a-f' | sed 's/#.*//;s/^[[:space:]]*//;s/[[:space:]]*$//')"
[ -z "$line" ] && continue

if is_valid_ip "$line"; then
if [ "$label" = "blocked" ] && ! is_safe_to_block "$line"; then
printf 'Refusing to compile loopback/private IP into blocked map: %s\n' "$line" >&2
continue
fi
printf '%s\n' "$line"
else
printf 'Ignoring invalid %s IP entry in %s: %s\n' "$label" "$source_file" "$raw_line" >&2
fi
done < "$source_file"
} | sort -u | while IFS= read -r line; do
printf '%s 1;\n' "$line" >> "$tmp_map"
done

mv "$tmp_map" "$target_map"
touch "$WHITELIST_CIDR_MAP"
}

reload_nginx() {
Expand All @@ -194,7 +158,7 @@ sync_ip_maps_if_needed() {
fi

generate_ip_map "$BLOCKLIST_SOURCE" "$BLOCKLIST_MAP" "blocked"
generate_ip_map "$WHITELIST_SOURCE" "$WHITELIST_MAP" "whitelisted"
generate_whitelist_maps "$WHITELIST_SOURCE" "$WHITELIST_MAP" "$WHITELIST_CIDR_MAP"
last_blocklist_signature="$blocklist_signature"
last_whitelist_signature="$whitelist_signature"
reload_nginx
Expand Down Expand Up @@ -241,7 +205,8 @@ update_auto_blocklist_from_probe_log() {
fi

# Whitelisted IPs remain exempt even if they trigger probe patterns.
if is_ip_listed "$WHITELIST_SOURCE" "$ip"; then
# Range-aware: an address inside a whitelisted CIDR is exempt too.
if is_ip_whitelisted "$WHITELIST_SOURCE" "$ip"; then
continue
fi

Expand Down
Loading
Loading