kubevirt: pin Flatcar VMs to PCI root complex to preserve Ignition#2057
Open
mihiragrawal wants to merge 1 commit into
Open
kubevirt: pin Flatcar VMs to PCI root complex to preserve Ignition#2057mihiragrawal wants to merge 1 commit into
mihiragrawal wants to merge 1 commit into
Conversation
On KubeVirt >= 1.6.4, Flatcar worker VMs boot without their Ignition configuration and never join the cluster. KubeVirt's PCIe-hotplug port reservation re-defines the libvirt domain a second time; that path runs an XML round-trip that cannot preserve the qemu namespace, so it drops the <qemu:commandline> "-fw_cfg opt/com.coreos/config" argument that carries Flatcar's stage-1 Ignition. cloud-init (Ubuntu, noCloud) is unaffected because it is delivered as a disk, not a qemu command-line arg. Setting the kubevirt.io/placePCIDevicesOnRootComplex annotation makes KubeVirt skip the extra hotplug ports and the second domain define, so the fw_cfg argument survives. Worker VMs never hotplug PCI devices, so the lost hotplug capacity has no practical impact. Scoped to Flatcar to avoid changing behaviour for other operating systems. Workaround; remove once the upstream fix is in the deployed KubeVirt. Refs: kubevirt/kubevirt#16901 (root cause), kubevirt/kubevirt#18460 (fix)
kubermatic-bot
added
release-note
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubermatic-bot
added
the
size/XS
mihiragrawal
added a commit
to mihiragrawal/docs
that referenced
this pull request
Jul 23, 2026
Document the KubeVirt >= 1.6.4 regression that drops the fw_cfg (opt/com.coreos/config) qemu argument for Flatcar VMs, so Flatcar worker nodes never receive their Ignition config and fail to join the user cluster. Ubuntu (cloud-init/noCloud) is unaffected. The page covers the problem, root cause, the machine-controller workaround (the kubevirt.io/placePCIDevicesOnRootComplex annotation, kubermatic/machine-controller#2057) plus a manual MachineDeployment fallback, the Flatcar-only scope, the trade-off (disables unused PCIe hotplug on worker VMs), and upstream references (kubevirt/kubevirt#16901, kubevirt/kubevirt#18460).
Author
|
/retest |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
On KubeVirt 1.6.4 and newer, Flatcar worker VMs boot but never receive their Ignition configuration, so the node never joins the cluster. KubeVirt's PCIe-hotplug port reservation re-defines the libvirt domain a second time, and that path runs an XML round-trip that cannot preserve the qemu namespace — dropping the
<qemu:commandline>-fw_cfg opt/com.coreos/configargument that carries Flatcar's stage-1 Ignition. This PR sets thekubevirt.io/placePCIDevicesOnRootComplexannotation on Flatcar VMs, which makes KubeVirt skip the extra hotplug ports and the second domain define, so the fw_cfg argument survives. cloud-init based OSes (e.g. Ubuntu) are unaffected because they are delivered as a disk rather than a qemu command-line argument.Which issue(s) this PR fixes:
Fixes #
None (root cause is upstream KubeVirt — see Special notes)
What type of PR is this?
Special notes for your reviewer:
xml.Unmarshalround-trip inApplySidecarHookssilently drops the qemu-namespaced command line, which became fatal once KubeVirt 1.6.4 added a second domain define in the PCIe-hotplug port path. Upstream refs: [sidecars] XML round-trip via xml.Unmarshal/xml.Marshal drops or corrupts namespace-scoped domain sections (qemu:commandline/qemu:arg) kubevirt/kubevirt#16901 (root cause), sidecar hooks: preserve QEMUCmd across XML round-trip kubevirt/kubevirt#18460 (fix, still open, norelease-1.6backport yet). This PR is the workaround until that fix ships in the KubeVirt version we deploy.spec.template.metadata.annotationson the MachineDeployment, since machine-controller merges those after this default.testdatafixture currently exercises Flatcar (none setkubevirt.io/ignitiondata), so no golden files change. Happy to add a Flatcar fixture + case if you'd like the path covered.Does this PR introduce a user-facing change? Then add your Release Note here:
Documentation: