docs: masquerade-off HA is achievable, not impossible - #893
docs: masquerade-off HA is achievable, not impossible#893SunsetDrifter wants to merge 4 commits into
Conversation
Both the routing-peers page and the masquerade page stated that high availability "stops working" with masquerade off. That reads as a flat impossibility and loses a legitimate use case (source IP preservation for audit, per-user database rules, or firewall policy keyed on the real source). What is actually true is narrower: NetBird's half of the failover still works, clients move to the standby peer on their own. What is missing is the return route inside the destination network following that move. State the condition instead of the impossibility, and add a short section on what meeting it takes: distinct metrics so the active peer is the same for every client, plus a virtual IP or a router-moved route so the next hop follows. Both live outside NetBird, so the section names the pattern rather than shipping a third-party config we would then own. Also: - Soften "failover is automatic and immediate". Clients re-select within seconds, which is fast but not instant, and "immediate" invites a support ticket from anyone who measures it. Phrased so it does not assert whether the client or the management side detects the loss, which this change did not verify. - Warn against pointing the return route at both peers, as two static routes or an equal-cost pair. It is the intuitive fix and it fails silently. - Note that the second interruption on recovery cannot be avoided by pinning the virtual IP to the standby, since the client's switch back is automatic. Pinning the route while the client moves back strands the return path. - The return route can live on the destination subnet's default gateway even when the gateway and the routing peer share that subnet, which is the only option when the destination host is not the customer's to configure. The page previously offered the gateway only for a different-subnet destination, and its persistent-config steps assume host access throughout.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe documentation updates clarify automatic routing peer failover timing, masquerade-off return-route placement, and high-availability configurations that keep return traffic aligned with the active peer. ChangesRouting and masquerade documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/pages/manage/networks/masquerade.mdx`:
- Line 184: Update the recovery interruption statement in the masquerade
failover documentation to make it conditional: recovery can cause a second brief
interruption when clients return before the route does, and the behavior depends
on convergence timing. Preserve the surrounding explanation about automatic
client switching and virtual IP pinning.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: da772466-f08a-4d07-8b2a-ce88319d735a
📒 Files selected for processing (2)
src/pages/manage/networks/how-routing-peers-work.mdxsrc/pages/manage/networks/masquerade.mdx
| Do not point the return route at both peers at once, as two static routes or an equal-cost pair. Replies reaching the peer a client is not using are silently discarded, so both ends look healthy while traffic fails. | ||
| </Warning> | ||
|
|
||
| Established connections reset on failover, as they do with masquerade on. Recovery causes a second brief interruption, because clients return to the primary faster than the route does, and you cannot avoid it by pinning the virtual IP to the standby: the client's switch back is automatic. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | grep -F 'src/pages/manage/networks/masquerade.mdx' || true
echo "== relevant MDX section =="
if [ -f src/pages/manage/networks/masquerade.mdx ]; then
wc -l src/pages/manage/networks/masquerade.mdx
sed -n '150,210p' src/pages/manage/networks/masquerade.mdx
fi
echo "== search related docs for VRRP/keepalived/router failoff wording =="
rg -n "VRRP|keepalived|router|health check|failover|masquerade|second brief|return to the primary|switch back" src/pages/docs src/pages/manage public docs -S || true
echo "== repo config/build hints =="
for f in package.json netbird setup.sh; do
[ -f "$f" ] && { echo "--- $f ---"; sed -n '1,220p' "$f"; }
done
echo "== git status/stat =="
git diff --stat || true
git status --short || trueRepository: netbirdio/docs
Length of output: 23780
🌐 Web query:
NetBird routing peers failure detection recovery failback documentation seconds clients switch back primary
💡 Result:
NetBird handles routing peer failure, recovery, and failback through an automated high availability (HA) mechanism that operates continuously [1][2][3]. Failure Detection and Failover When you assign multiple routing peers to a network, NetBird automatically monitors the availability and health of those peers [4][3][5]. If a primary routing peer (the one with the lowest metric) becomes unreachable, NetBird clients immediately detect the failure and fail over to an available standby routing peer [1][2]. This process is designed to be immediate, ensuring that traffic continues to flow through the remaining healthy peer(s) [1][2][6]. Recovery and Failback When the primary routing peer comes back online and restores connectivity, NetBird clients automatically detect its return and switch back to it immediately [1]. This ensures that the network traffic is again routed through the preferred, lower-metric peer [1]. Key Technical Details: - Metric-based Selection: Routing peers are assigned metrics. A lower metric indicates a primary peer, while higher metrics indicate standby peers [1][4]. If peers are assigned the same metric, NetBird balances traffic among them based on latency [4][2]. - Automatic Switching: Failover and failback are triggered automatically by the NetBird client's route management system, which continuously evaluates peer status (e.g., connection status, latency, and metrics) [1][5]. - Connection Handling: While the failover and failback processes happen near-instantaneously, any established TCP connections passing through the previous routing peer will reset when the switch occurs, requiring applications to re-establish those connections [1]. - Configuration: High availability is achieved by adding multiple routing peers to the same network [4][7]. It is recommended to keep these peers in different failure domains to ensure genuine redundancy [4]. If you experience issues where a client fails to switch, it may indicate a stale route or a synchronization delay, which have been addressed in recent versions of NetBird to ensure robust automatic recovery [6][8]. Verify that your routing peers are correctly configured in a routing group or assigned individually with appropriate metrics [4][7].
Citations:
- 1: https://docs.netbird.io/manage/networks/how-routing-peers-work
- 2: https://docs.netbird.io/manage/integrations/kubernetes/use-cases/route-to-a-kubernetes-service
- 3: https://docs.netbird.io/use-cases/kubernetes/routing-peers-and-kubernetes
- 4: https://docs.netbird.io/how-to/networks?form=enterprise
- 5: https://deepwiki.com/netbirdio/netbird/5.5-route-management
- 6: Network Map not updating after loss of HA routing peer netbird#3886
- 7: https://docs.netbird.io/manage/network-routes
- 8: [client] fix stale routing peer on overlapping-prefix network removal netbird#6799
🌐 Web query:
VRRP keepalived failover convergence versus host/application reconnect after routing peer failure
💡 Result:
The convergence process following a routing peer or VRRP (keepalived) failure consists of two distinct phases: network-layer re-convergence and application-layer recovery [1]. Network-layer convergence (VRRP/Keepalived) The time taken for keepalived to fail over is primarily governed by the VRRP advertisement interval and the number of missed advertisements allowed before declaring the Master node dead [2][3][4]. - Default Behavior: With standard configuration (1-second advertisement, 3-second hold time), failover typically takes 3-4 seconds [2][4]. - Optimization: Convergence can be accelerated by tuning these timers (e.g., using sub-second advertisements if supported by the version and network environment) and ensuring Gratuitous ARP (GARP) behavior is optimized [2][5]. - Hidden Latency: Even after the backup node assumes the Master state, traffic may remain unreachable due to upstream switch ARP cache timeouts [2]. Configuring keepalived to send multiple, repeated GARPs can mitigate this by forcing upstream devices to update their MAC tables faster [2]. Application/Host Reconnect Network convergence is only the "gate opener"; it does not guarantee immediate restoration of traffic for active sessions [1]. - TCP Implications: A VRRP failover typically forces existing TCP connections to reset because the new Master often lacks the conntrack/stateful information of the original Master [6][7]. Unless state synchronization (such as conntrackd or application-level state replication) is in place, clients must re-establish sessions, which depends on application timeouts and retry logic [6][7]. - Physics of Failure: For established connections, a failover event often results in a brief window of dropped packets (RTO) [8]. Modern applications usually handle these short outages gracefully through automatic retries [3][4]. For more critical, high-availability scenarios, simple network-level redundancy is often insufficient; solutions such as Protective ReRoute (PRR) or stateful synchronization are required to mask these failures from the application layer [7][9]. Summary of Key Differences - VRRP/Keepalived Convergence: Focuses on the transition of the Virtual IP (VIP) and MAC ownership at the L2/L3 boundary [2][10]. It is deterministic and governed by timer configurations [3][4]. - Application Reconnect: Depends on the interaction between the network outage duration and the application's tolerance for connectivity loss [4][1]. Even if the network converges in 1 second, the application may require much longer to detect a "broken" session and initiate a reconnect, depending on its internal keepalive/timeout settings [4]. Top Results: [2], [3], [4], [8], [7], [1]
Citations:
- 1: https://cordero.me/why-bgp-failover-takes-30-seconds-but-failback-takes-2/
- 2: https://ialloc.org/keepalived-vrrp-failover-in-production-multi-node-setup/
- 3: https://devopsil.com/articles/keepalived-vrrp-failover
- 4: https://stackoverflow.com/questions/20489086/will-a-vrrp-fail-over-cause-tcp-connection-closed-and-need-to-re-connect
- 5: https://serverfault.com/questions/894268/keepalived-failover-downtime
- 6: vrrp master switchover lead to disconnection of tcp connections acassen/keepalived#2254
- 7: https://support.versa-networks.com/support/solutions/articles/23000029633-dia-failover-with-active-standby-ha-architecture-packet-flow-and-operational-guide
- 8: https://supportportal.juniper.net/s/article/Is-RTO-expected-during-VRRP-failovers
- 9: https://www2.cs.uh.edu/~gnawali/courses/cosc6377-f23/papers/Wetherall23.pdf
- 10: https://www.potaroo.net/ietf/rfc/rfc9568.html
Qualify the recovery interruption claim.
The supported routes setup is VRRP/keepalived or router health checks, but convergence timing can differ across configs. “Recovery causes a second brief interruption” is better as “Recovery can cause a second brief interruption if clients return before the return route moves; behavior depends on convergence timing.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/pages/manage/networks/masquerade.mdx` at line 184, Update the recovery
interruption statement in the masquerade failover documentation to make it
conditional: recovery can cause a second brief interruption when clients return
before the route does, and the behavior depends on convergence timing. Preserve
the surrounding explanation about automatic client switching and virtual IP
pinning.
A routing peer can stay enrolled and report healthy, with its WireGuard interface present, while being unable to forward from the tunnel to the destination subnet: a failed LAN interface, or a firewall change. Nothing fails over in that state. NetBird sees a reachable peer, and a health check that only looks for the interface sees a healthy one, so the return route stays pointed at a peer that forwards nothing until someone intervenes. Say that the check has to reach a destination through the tunnel.
Measured on a routing-peer pair with masquerade off: a held TCP connection survives the switch. Four failovers, graceful shutdown and hard kill, all showed the same signature — one round trip with multi-second latency, then the same connection continuing with monotonic sequence numbers. No resets, and no RST on the wire in any of them. The reason is a property of masquerade being off: the addresses on the connection do not change when another peer takes over, so the flow is portable between peers, and the routed ACL accepts on addresses rather than on connection state, so the standby forwards mid-stream packets it never saw a handshake for. Note this is specific to masquerade being off. The general high-availability section still says established connections reset, which is the masquerade-on default, where the standby would translate to a different source address. Also trims the mechanism guidance to name the pattern rather than lean on any one implementation, since which mechanism is right may change.
Review of the branch turned up three problems. The Networks overview still said disabling masquerade 'breaks high availability', one click from the page arguing it does not. Reworded to say it makes high availability something you arrange rather than something you get. Neither reset/stall sentence named its masquerade scope, so a reader with masquerade off met both and they flatly disagreed. The general failover section now scopes its reset claim to masquerade on, names the reason (the standby translates to a different source address), and links onward. 'Stalls for the length of the failover' understated it. Measured stalls were 6 to 13 seconds against a 6 second failover, because the sender waits for its next retransmit after the path returns. Says so, with the range, and keeps the claim hedged to what was observed rather than asserting connections are never broken.
2 participants
Why
Two pages state that high availability "stops working" when masquerade is off:
manage/networks/how-routing-peers-work— "High availability also stops working with masquerade off, because return traffic must flow back through one specific routing peer's LAN address"manage/networks/masquerade— "High availability stops working"That reads as a flat impossibility, and it turns away a legitimate use case: preserving the client source IP for auditing, per-user database permissions, or firewall policy keyed on the real source.
The true statement is narrower. NetBird's half of a failover still works, clients move to the standby peer on their own. What is missing is the return route inside the destination network following that move. That is a condition to meet, not a dead end.
What changed
masquerade.mdxBoth mechanisms live outside NetBird, so the section names the pattern without shipping a third-party configuration we would then own.
how-routing-peers-work.mdxNotes for review
#high-availability-with-masquerade-off(new) and#high-availability(already referenced from the Kubernetes use-case page).Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit