Skip to content

Backend triggers: resolve dependencies by their real container name - #124

Open
antoncxx wants to merge 1 commit into
NullNet-ai:mainfrom
antoncxx:feature/backend-trigger-container-names
Open

Backend triggers: resolve dependencies by their real container name#124
antoncxx wants to merge 1 commit into
NullNet-ai:mainfrom
antoncxx:feature/backend-trigger-container-names

Conversation

@antoncxx

@antoncxx antoncxx commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Lets a backend-trigger dependency chain declare a literal Docker container
name (chain = ["redis"]) instead of a purpose-built DNS alias
(chain = ["redis.nullnet.com"]). Also adds support for two dependencies
that happen to share a real port (e.g. two plain-HTTPS deps, both 443),
which the first version of this change left as a known gap and later
closed out.

Why

Backend-trigger dependency chains only build the first time the initiator
actually opens a connection on the watched port — an NFQUEUE listener holds
that first packet, reports it to the server, the server builds the tunnel,
then the packet is released once DNAT is in place. That only works if a
real packet leaves the container in the first place. A bare name like
redis fails before it gets that far: with nothing to resolve it to, the
app's connect() never succeeds, no packet is ever sent, and the trigger
never fires. Today's workaround is a purpose-built alias
(redis.nullnet.com) backed by a pre-provisioned *.nullnet.com DNS
wildcard, whose only job is giving the name something to resolve to.

This PR removes that workaround: the client pre-seeds a placeholder
/etc/hosts entry for the dependency's literal name before any packet is
observed, so the bare name resolves, a real first packet gets sent, and the
existing NFQUEUE → backend_trigger → tunnel-setup → DNAT flow runs
unchanged after that (it never inspected the destination address to begin
with).

What changed

Proto (nullnet_grpc.proto)

  • ServiceTrigger.trigger_ports: repeated TriggerPort { port, target_name }
    replaces the old flat ports list, so the server tells the client not
    just which port to watch but which literal name (chain[0]) it resolves.
  • ServiceTrigger.initiator_container: the real container name the client
    should seed, resolved server-side from data it already had.
  • BackendTriggerRequest.target_name: the client reports back which name
    its placeholder's destination address belonged to, so the server can pick
    the right chain when a port has more than one.

Server

  • ServiceInfo.triggers moves from HashMap<u16, Vec<String>> (one chain
    per port) to HashMap<u16, Vec<Vec<String>>> — more than one chain can
    share a port now. New chain_for(port, target_name) selector: a single
    chain on a port is used regardless of target_name (unchanged behavior
    for every existing config); multiple chains require an exact match — no
    guessing when ambiguous.
  • Config validation rejects two chains on the same port with the same
    chain[0] outright (nothing could ever tell them apart).
  • Admin HTTP API (http_server/services.rs) and UI
    (Services.tsx/types.ts) updated to the new per-port chain list.

Client

  • placeholder.rs (new): deterministic name → IP in 203.0.113.0/24
    (RFC 5737 TEST-NET-3 — reserved, never a real host, never on-link for a
    container's own subnet, so it always falls through to the container's
    default route). Range is env-overridable via TRIGGER_PLACEHOLDER_CIDR.
  • Placeholder seeding is wired into the existing declare-services reconcile
    loop (same ~10s / docker events fast-path cadence already used for the
    watched-port ipset) — idempotent, self-heals across container restarts
    (Docker wipes /etc/hosts on every start).
  • TriggersState and DNAT (commands/dnat.rs) both widened to key on
    destination address as well as source + port, so two chains sharing a
    port get independent state and independent -d-scoped DNAT rules instead
    of clobbering each other.
  • NFQUEUE listener carries the destination IP end to end (ipv4_flow,
    already existed for the egress listener) and, when a port has more than
    one candidate target, disambiguates by matching the packet's observed
    destination against each candidate's own deterministic placeholder
    address — passing through unaltered rather than guessing if nothing
    matches.
  • Teardown re-seeds the placeholder instead of deleting it for
    backend-trigger entries, so an idle-torn-down chain can re-trigger on the
    next connection attempt instead of dead-ending exactly like an unseeded
    bare name would. Proxy-dependency mappings are unaffected — a fresh proxy
    request rebuilds the chain and writes the real mapping before forwarding,
    so there's no gap to cover there.

### Demo (demo/name-resolution/)
Three-container stack (portaldep-a, dep-b) exercising the whole
thing end to end, reachable through nullnet-proxy by Host header.
dep-a and dep-b deliberately share the same real port (80) — two
trigger chains on one port, told apart by chain[0] — so the demo actually
runs the disambiguation path, not just the base case. Each container sits
on its own isolated Docker network: no shared network, no path between them
except the one nullnet builds on demand, which is what proves the
placeholder-seeded trigger is doing the work rather than Docker's own
embedded DNS resolving things directly.

Out of scope

  • Hardcoded-IP initiators (an app that dials a literal IP instead of
    resolving any name) — the backend-trigger path already handles this for
    free (it never inspected the destination address), but proxy-dependency
    edges would need the same DNAT fallback backend triggers have; not
    addressed here.
  • Egress/forward-proxy paths are untouched.

@antoncxx
antoncxx force-pushed the feature/backend-trigger-container-names branch from 59f0a21 to 2c271dc Compare July 30, 2026 01:20
@antoncxx antoncxx changed the title Backend trigger container names Backend triggers: resolve dependencies by their real container name Jul 30, 2026
@antoncxx
antoncxx marked this pull request as ready for review July 30, 2026 01:53
@antoncxx
antoncxx force-pushed the feature/backend-trigger-container-names branch from f868bd5 to a5deaa6 Compare July 31, 2026 20:22
@GyulyVGC

GyulyVGC commented Aug 3, 2026

Copy link
Copy Markdown
Member

Thanks @antoncxx looks good, just two things:

  • I think we can avoid pushing the demo material if you already tested it working fine
  • the IPs from the TEST_NET you're using are only 255, so this means that a service can in theory have no more than 255 backend triggers or else there will be a /etc/hosts conflict right? It's a pretty high number so it's reasonable, I just want to make sure that the hashing function is able to provide 255 different IPs for 255 different names, is this the case?

@antoncxx

antoncxx commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @antoncxx looks good, just two things:

  • I think we can avoid pushing the demo material if you already tested it working fine
  • the IPs from the TEST_NET you're using are only 255, so this means that a service can in theory have no more than 255 backend triggers or else there will be a /etc/hosts conflict right? It's a pretty high number so it's reasonable, I just want to make sure that the hashing function is able to provide 255 different IPs for 255 different names, is this the case?
  1. Demo materials have been removed.
  2. The address space isn't a per-node or per-service budget, since /etc/hosts is per-container and
    disambiguation keys on (container, dst_ip, port), so the same IP can be reused freely across containers or ports.
    The only place it's shared is among dependency names on the same container hitting the same port, and since
    FNV-1a mod 254 isn't collision-free, even that's not a guaranteed 1:1 mapping — but in practice that count is
    always tiny, so collisions aren't a real concern day to day. That said, there's no config-time check today that
    would catch one if it happened, so I'll add validation to the PR: at declare-time, compute ip_for() for every
    chain[0] name sharing a (container, port) pair and reject the config if two collide — cheap to do since it only
    needs to check the names actually configured, not the theoretical 254-value space, and it directly catches the
    one real failure mode (resolve_target silently picking the wrong dependency).

@antoncxx

antoncxx commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@GyulyVGC

Added config validation

services_map() builds a per-service TriggerMap (port → list of chains), then calls placeholder_collision(&triggers), which for each port compares every pair of chains' chain[0] names via nullnet_grpc_lib::last_octet_for(). If two names hash to the same octet, it returns Some((port, name_a, name_b)) and services_map() errors out with that info baked into the message.

This runs in two places that both call services_map(): the startup TOML loader, and validate_stack_toml() (called from the /api/config/:stack save handler). On error, the handler returns 422 with {ok: false, error: "<message>"}; the UI reads that straight into the badge/message shown on the Config page.

@GyulyVGC

GyulyVGC commented Aug 5, 2026

Copy link
Copy Markdown
Member

Let me know when this is ready.

Also test in a setup with mixed proxy deps and backend triggers please, and verify there are no collisions or weird stuff happening with egress edges.

Also at every new feature we should really be careful of avoiding regressions and that at network teardowns and nullnet-client restart, the state is properly cleaned up, I'm saying it's especially now because we're going in production

Lets a service dial a backend dependency by its literal Docker container
name (e.g. "dep-a") instead of a "*.nullnet.com" alias — nullnet-client
pre-seeds a deterministic placeholder /etc/hosts entry for the name so a
bare-name lookup produces a real first packet, NFQUEUE catches it on the
declared trigger port, and the server brings up the real tunnel on demand.

Supports multiple dependencies sharing one real port (TriggerMap is a list
of chains per port, not one), disambiguated at trigger time by matching the
observed destination against each candidate's own placeholder address
(nullnet-grpc-lib::last_octet_for, shared by client and server so they can't
disagree on the mapping). Config validation rejects both a literal
chain[0] duplicate on a shared port and the rarer case where two distinct
names hash to the same placeholder address, with a message identifying
both names so the reason surfaces in the config UI, not just a generic
parse error.

Also includes three nullnet-client reliability fixes found while building
and validating this against an end-to-end demo stack (since removed):
- Stale TriggersState entries now self-heal on container restart/recreate
  (previously required a manual client restart) by hooking the existing
  docker-events watcher to purge a recreated container's trigger state.
- The backend-trigger placeholder block is now exempted from egress/
  country-policy classification, closing a startup race where a trigger
  dial landing before its port was watched got misrouted into the egress
  path and stalled past its own caller's timeout instead of failing fast.
- nullnet-client now warns loudly, once per bad spell, when a managed
  container has no default route — the precondition this whole mechanism
  depends on — instead of the only symptom being a bare ENETUNREACH three
  layers away in the initiator's own app.
@antoncxx
antoncxx force-pushed the feature/backend-trigger-container-names branch from 1fe4ccb to be37274 Compare August 6, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Simplify mesh configuration by allowing Docker container names in backend-triggers

2 participants