fix(stack): two k3s blockers — loopback Endpoints, and the storefront preview host missing from /etc/hosts

[bussyjd]

The failure

obol stack up fails on the k3s backend, blocking the whole defaults helmfile:

UPGRADE FAILED: cannot patch "ollama" with kind Endpoints:
Endpoints "ollama" is invalid: subsets[0].addresses[0].ip:
Invalid value: "127.0.0.1": may not be in the loopback range (127.0.0.0/8, ::1/128)

Reproduced on a k3s host (server v1.35.5+k3s1) while validating v0.14.0-rc2. It is not a regression from that RC — the same box's helm history shows it failing identically under 0.13.0-rc3 four weeks earlier:

rev 3  Thu Jul  9 18:07  failed    ← identical error
rev 4  Thu Jul  9 18:12  failed    ← identical error
rev 5  Thu Jul  9 18:16  deployed  ← worked around, cause never fixed
rev 6  Wed Aug  5 16:45  failed    ← today

Why it happens

OllamaHostForBackend returns 127.0.0.1 for k3s — correct in spirit, since k3s runs directly on the host. OllamaHostIPForBackend sees a value that already parses as an IP and returns it unchanged, and it lands verbatim in the ollama Endpoints in base/templates/llm.yaml.

Kubernetes has rejected loopback addresses in Endpoints since v1.33, and it is right to: inside a pod's network namespace 127.0.0.1 is the pod, not the host. The endpoint could never have routed to host Ollama — the API server now simply refuses to pretend otherwise. Any k3s user on Kubernetes ≥1.33 hits this.

The fix

One guard in the shared resolver: when resolution lands on loopback, substitute the host's primary routable IPv4 address. Deliberately not a k3s branch — it also covers Docker runtimes that map host.docker.internal to loopback.

hostPrimaryIP() asks the kernel which route it would take (a UDP "dial" to TEST-NET-1, no packets sent), and falls back to scanning for the first up, non-loopback IPv4 on hosts with no default route.

Sibling caller found and collapsed

Grepping the callers turned up resolveHostIP in cmd/obol/sell.go — a second copy of the same strategy that had drifted with the identical bug: it returned 127.0.0.1 for k3s and handed it to createHostService, which also builds an Endpoints object. So obol sell inference on k3s would fail the same way. It now delegates to the shared resolver instead of duplicating it (net −27 lines there).

The stale createHostService doc comment already admitted the duplication: "using the same strategy as ollamaHostIPForBackend in internal/stack".

Tests

Three existing tests asserted the old loopback values — they encoded the bug. They now assert the contract that matters: the resolved address is a valid, non-loopback IP.

• TestOllamaHostIPForBackend_K3s — was == "127.0.0.1", now "valid and not loopback"
• TestOllamaHostIPForBackend_AlreadyIP — still exercises the numeric short-circuit path, without pinning the loopback value
• TestCopyInfrastructureRendersStackPlaceholders — was a literal ip: "127.0.0.1" substring match, now extracts the rendered IP and asserts it is routable

Plus two new regression tests: TestOllamaHostIPForBackendNeverReturnsLoopback and TestHostPrimaryIPIsRoutable.

Verification: go build ./... OK · go vet ./... clean · go test ./... exit 0, 40 packages · gofmt clean on all touched files.

Notes

• Based on integration/v0.14.0-rc2 so it can go into an rc3 and unblock end-to-end validation of the storefront live preview. It should also land on main — it affects every k3s user, independent of this release train.
• Not covered here: base on any cluster that already has a failed release revision may need helm rollback before a clean stack up. That is remediation on existing boxes, not a code fix.

[bussyjd]

Second commit added — and the honest status of the first

I was asked whether the loopback fix was actually validated. It wasn't: it was unit-tested on macOS only. Validating it properly on a real k3s box surfaced a second, independent blocker, so this PR now carries two commits.

Validation trap worth recording

My first attempt to validate on a k3s host appeared to fail — stack up still errored with 127.0.0.1. The fix was fine; the test was wrong. CopyInfrastructure only re-renders when the stamp changes:

digest=<embedded template bytes> backend=<name> stackID=<id> image=<StampIdentity()>

The loopback fix changes Go code, not template bytes, and I had deliberately built with GitCommit=a5bd91e so component images resolved to the published rc2 set — which made StampIdentity() identical too. Same stamp → no re-render → the stale llm.yaml with 127.0.0.1 was reused. A real rc3 has a different commit and re-renders normally; only the hand-built test binary hit this. Forcing it with rm ~/.config/obol/defaults/.obol-defaults-stamp let the fix take effect.

The second bug

With base finally deploying, /storefront served 200 and the preview HTTPRoute was Accepted ResolvedRefs — but storefront-preview.obol.stack did not resolve, so the editor's iframe had nowhere to load from. The route and backend were healthy; only DNS was missing.

Cause: EnsureHostsEntries replaces the managed block wholesale, and #805 added the preview hostname at exactly one of five call sites.

stack.go:254     EnsureHostsEntries(CollectHostnames + StorefrontPreviewHostname)   ← has it
hermes.go:135    EnsureHostsEntries(CollectHostnames(...))                          ← omits
hermes.go:169    EnsureHostsEntries(CollectHostnames(...))                          ← omits
hermes.go:241    EnsureHostsEntries(CollectHostnames(...))                          ← omits
openclaw.go:275  EnsureHostsEntries(CollectHostnames(...))                          ← omits

stack up resumes agents after syncing defaults, so hermes runs last and rewrites the block without it. Last writer wins.

Fixed by emitting the origin unconditionally, like the base domain, instead of relying on call-site discipline. The constant moves to internal/dns — a leaf package (stdlib imports only) that owns the managed block — and internal/tunnel points at it. Block rendering is extracted into buildHostsBlock so the guarantee is testable without root.

End-to-end on a k3s host — server v1.35.5+k3s1

managed block   127.0.0.1 obol.stack
                127.0.0.1 storefront-preview.obol.stack   ← was missing
                127.0.0.1 hermes-obol-agent.obol.stack
                127.0.0.1 obol-agent.obol.stack

getent hosts storefront-preview.obol.stack   127.0.0.1    ← was NXDOMAIN
ollama endpoint                              192.168.50.167  ← was 127.0.0.1
helm base                                    deployed     ← was failed x2

/api/version  {"obol":{"version":"0.14.0-rc2-fixes2"},
               "frontend":{"tag":"v0.1.28-rc5",...}}
/storefront                                  200          ← 404 on rc3
http://storefront-preview.obol.stack/        200          ← via real DNS, not a Host header

Ollama on that host listens on *:11434 (OLLAMA_HOST=0.0.0.0), so the LAN-IP endpoint genuinely reaches it — a non-loopback Endpoints would otherwise be valid but refused.

The /api/version line also confirms the footer feature composes across the two release trains: OBOL_STACK_VERSION from #802 on this branch, read by front-end v0.1.28-rc5. Neither RC shows it alone.

go build ./... OK · go vet ./... clean · go test ./... exit 0, 40 packages.

Still not covered

• Neither fix is on main — both affect every k3s user independently of this train.
• Boxes with an existing failed helm revision for base recover on the next stack up once the render is corrected; no manual helm rollback was needed in testing.
• The browser-side check (editing name/tagline/theme/accent and watching the iframe update over postMessage) is still unrun — that needs a human at a browser.