dhcp: fix boot time sync when a BMC USB NIC wins the DHCP race - #333
Open
juninhoojl wants to merge 1 commit into
Open
dhcp: fix boot time sync when a BMC USB NIC wins the DHCP race#333juninhoojl wants to merge 1 commit into
juninhoojl wants to merge 1 commit into
Conversation
On machines whose BMC exposes a virtual USB NIC, the one shot dhcpcd call exits as soon as that interface gets its link local lease from the BMC, which happens well before the real NICs finish negotiating carrier. The boot then continues with no default route, no DNS and no DHCP option 42, so the ntpd call that follows dies on "bad address 'pool.ntp.org'" and the clock is never set. The dhcpcd service picks up the real lease a moment later, but nothing retries the time sync, and openntpd cannot recover on its own because it only slews. A wrong clock then hard fails the tink-worker image pull on TLS certificate validity. Three fixes to files/dhcp.sh: - Catch SIGALRM. busybox ntpd uses it for its own timeouts, and the alarm terminated this script along with it, so the existing retry and manual fallback logic never ran. Catching rather than ignoring keeps ntpd's own timeout working: a caught signal is reset to its default disposition in the child, while an ignored one is inherited. - Prefer the NTP servers leased via DHCP option 42, falling back to pool.ntp.org. This drops the dependency on name resolution for setting the clock and makes time sync work on isolated provisioning networks that have no outbound DNS, or no internet access at all. - Retry the one shot dhcpcd until some interface actually holds the default route, excluding interfaces that hold a lease without one. A plain re-run cannot help: -1 exits after the first interface is configured, `persistent` leaves the BMC lease in place on exit, and the lease directory survives across invocations, so the next call is satisfied from that cached lease sooner than the real NICs can finish negotiating. `waitip 4` does not help either, as the BMC lease already supplies an IPv4 address. Excluding the routeless interfaces forces dhcpcd to race the ones still unconfigured; an interface that has not leased yet is never excluded, so a slow real NIC is still picked up. The time sync logic is factored into sync_time/try_ntp_servers/ ntp_servers/default_route_iface so both the option 42 preference and the manual date fallback share one retry path. Signed-off-by: José Corrêa <jose.correa@latitude.sh>
Member
|
Hi @juninhoojl -- have you tried Captain on that same machine? It's a standard systemd replacement for HookOS, which should make handling this kind of stuff simpler (eg plain networkd, timesyncd, etc). Try it out: https://github.com/tinkerbell/captain |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On machines whose BMC exposes a virtual USB NIC, Hook boots with an unset clock and the
tink-workerimage pull then fails on TLS certificate validity.The sequence:
dhcpcd -1infiles/dhcp.shexits as soon as any allowed interface is configured.ntpdcall that follows dies onbad address 'pool.ntp.org'.dhcpcdservice picks up the real lease a moment later, but nothing retries the time sync, andopenntpdcan't recover on its own because it only slews.Fixes
Three changes, all in
files/dhcp.sh:1. Catch
SIGALRM. busyboxntpduses it for its own timeouts, and the alarm was terminating this script along with it — so the existing retry and manual-datefallback never ran at all. Catching rather than ignoring is deliberate: a caught signal is reset to its default disposition in the child, while an ignored one is inherited, sontpd's own timeout keeps working.2. Prefer NTP servers from DHCP option 42, falling back to
pool.ntp.org. This removes the dependency on name resolution for setting the clock, so time sync works on isolated provisioning networks with no outbound DNS — or no internet at all.option ntp_serversis already requested indhcpcd.conf, so the lease data was there and simply unused.3. Retry the one-shot
dhcpcduntil an interface actually holds the default route, excluding interfaces that hold a lease without one.The exclusion is the important part — a plain re-run cannot fix this:
-1exits after the first interface is configuredpersistent(indhcpcd.conf) leaves the BMC lease configured on exit/var/lib/dhcpcdis bind-mounted, so the lease file survives the runso each retry can be satisfied from the cached BMC lease faster than the previous attempt, converging on the same routeless result.
waitip 4doesn't help either, since the BMC lease already supplies an IPv4 address. Excluding the routeless interfaces forcesdhcpcdto race the ones still unconfigured. An interface that hasn't leased yet is never excluded, so a slow real NIC is still picked up.The time-sync logic is factored into
sync_time/try_ntp_servers/ntp_servers/default_route_ifaceso the option-42 preference and the manual-datefallback share one retry path rather than duplicating thentpdinvocation.Testing
Verified on affected bare-metal hardware (Broadcom NICs + BMC virtual USB NIC) where the boot previously came up with an unset clock:
tink-workerstarts, and the image pull succeedspool.ntp.orgfallback still works where DNS and egress existbuild.sh lint(shellcheck + shellfmt) passes;sh -n files/dhcp.shcleanBehaviour is unchanged on machines without a BMC USB NIC: the first
dhcpcd -1gets a routed lease and the loop breaks on the first iteration.Notes
files/dhcp.shonly — no other files touched.--denyinterfaces(-Z) is confirmed present in thelinuxkit/dhcpcdimage currently pinned inbash/hook-lk-containers.sh.