Skip to content

dhcp: fix boot time sync when a BMC USB NIC wins the DHCP race - #333

Open
juninhoojl wants to merge 1 commit into
tinkerbell:mainfrom
juninhoojl:fix-boot-time-sync-bmc-usb-nic
Open

dhcp: fix boot time sync when a BMC USB NIC wins the DHCP race#333
juninhoojl wants to merge 1 commit into
tinkerbell:mainfrom
juninhoojl:fix-boot-time-sync-bmc-usb-nic

Conversation

@juninhoojl

Copy link
Copy Markdown

Problem

On machines whose BMC exposes a virtual USB NIC, Hook boots with an unset clock and the tink-worker image pull then fails on TLS certificate validity.

The sequence:

  1. dhcpcd -1 in files/dhcp.sh exits as soon as any allowed interface is configured.
  2. On these machines that first lease is a link-local one handed out by the BMC's own USB NIC, which has carrier immediately — well before the real NICs finish negotiating.
  3. Boot continues with no default route, no DNS and no DHCP option 42.
  4. The ntpd call that follows dies on bad address 'pool.ntp.org'.
  5. The dhcpcd service picks up the real lease a moment later, but nothing retries the time sync, and openntpd can't recover on its own because it only slews.

Fixes

Three changes, all in files/dhcp.sh:

1. Catch SIGALRM. busybox ntpd uses it for its own timeouts, and the alarm was terminating this script along with it — so the existing retry and manual-date fallback never ran at all. Catching rather than ignoring is deliberate: a caught signal is reset to its default disposition in the child, while an ignored one is inherited, so ntpd's own timeout keeps working.

2. Prefer NTP servers from DHCP option 42, falling back to pool.ntp.org. This removes the dependency on name resolution for setting the clock, so time sync works on isolated provisioning networks with no outbound DNS — or no internet at all. option ntp_servers is already requested in dhcpcd.conf, so the lease data was there and simply unused.

3. Retry the one-shot dhcpcd until an interface actually holds the default route, excluding interfaces that hold a lease without one.

The exclusion is the important part — a plain re-run cannot fix this:

  • -1 exits after the first interface is configured
  • persistent (in dhcpcd.conf) leaves the BMC lease configured on exit
  • /var/lib/dhcpcd is bind-mounted, so the lease file survives the run

so each retry can be satisfied from the cached BMC lease faster than the previous attempt, converging on the same routeless result. waitip 4 doesn't help either, since the BMC lease already supplies an IPv4 address. Excluding the routeless interfaces forces dhcpcd to race the ones still unconfigured. An interface that hasn't leased yet is never excluded, so a slow real NIC is still picked up.

The time-sync logic is factored into sync_time / try_ntp_servers / ntp_servers / default_route_iface so the option-42 preference and the manual-date fallback share one retry path rather than duplicating the ntpd invocation.

Testing

Verified on affected bare-metal hardware (Broadcom NICs + BMC virtual USB NIC) where the boot previously came up with an unset clock:

  • clock is now correct before tink-worker starts, and the image pull succeeds
  • with option 42 served, the leased server is used and no DNS lookup occurs
  • with option 42 absent, the pool.ntp.org fallback still works where DNS and egress exist
  • build.sh lint (shellcheck + shellfmt) passes; sh -n files/dhcp.sh clean

Behaviour is unchanged on machines without a BMC USB NIC: the first dhcpcd -1 gets a routed lease and the loop breaks on the first iteration.

Notes

  • Scoped to files/dhcp.sh only — no other files touched.
  • --denyinterfaces (-Z) is confirmed present in the linuxkit/dhcpcd image currently pinned in bash/hook-lk-containers.sh.
  • Parts of this change were developed with assistance from Claude Code.

On machines whose BMC exposes a virtual USB NIC, the one shot dhcpcd call
exits as soon as that interface gets its link local lease from the BMC,
which happens well before the real NICs finish negotiating carrier. The
boot then continues with no default route, no DNS and no DHCP option 42,
so the ntpd call that follows dies on "bad address 'pool.ntp.org'" and the
clock is never set. The dhcpcd service picks up the real lease a moment
later, but nothing retries the time sync, and openntpd cannot recover on
its own because it only slews. A wrong clock then hard fails the
tink-worker image pull on TLS certificate validity.

Three fixes to files/dhcp.sh:

- Catch SIGALRM. busybox ntpd uses it for its own timeouts, and the alarm
  terminated this script along with it, so the existing retry and manual
  fallback logic never ran. Catching rather than ignoring keeps ntpd's own
  timeout working: a caught signal is reset to its default disposition in
  the child, while an ignored one is inherited.

- Prefer the NTP servers leased via DHCP option 42, falling back to
  pool.ntp.org. This drops the dependency on name resolution for setting
  the clock and makes time sync work on isolated provisioning networks
  that have no outbound DNS, or no internet access at all.

- Retry the one shot dhcpcd until some interface actually holds the
  default route, excluding interfaces that hold a lease without one. A
  plain re-run cannot help: -1 exits after the first interface is
  configured, `persistent` leaves the BMC lease in place on exit, and the
  lease directory survives across invocations, so the next call is
  satisfied from that cached lease sooner than the real NICs can finish
  negotiating. `waitip 4` does not help either, as the BMC lease already
  supplies an IPv4 address. Excluding the routeless interfaces forces
  dhcpcd to race the ones still unconfigured; an interface that has not
  leased yet is never excluded, so a slow real NIC is still picked up.

The time sync logic is factored into sync_time/try_ntp_servers/
ntp_servers/default_route_iface so both the option 42 preference and the
manual date fallback share one retry path.

Signed-off-by: José Corrêa <jose.correa@latitude.sh>
@rpardini

Copy link
Copy Markdown
Member

Hi @juninhoojl -- have you tried Captain on that same machine? It's a standard systemd replacement for HookOS, which should make handling this kind of stuff simpler (eg plain networkd, timesyncd, etc). Try it out: https://github.com/tinkerbell/captain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants