Add Known Issues page: Flatcar Ignition failure on KubeVirt 1.6.x

[mihiragrawal]

What this PR does / why we need it

Adds a Known Issues page to the Kubermatic Virtualization docs, documenting
the KubeVirt 1.6.x defect that breaks Flatcar worker-node provisioning.

On KubeVirt v1.6.4 and newer, Flatcar worker VMs boot and get an IP but never
receive their Ignition configuration, so the node never joins the user cluster.
KubeVirt's PCIe-hotplug port reservation re-defines the libvirt domain a second
time, and that XML round-trip drops the qemu:commandline -fw_cfg opt/com.coreos/config argument that carries Flatcar's Ignition. Ubuntu
(cloud-init / noCloud) is unaffected. This is an upstream KubeVirt bug, not a
Kubermatic Virtualization or machine-controller defect.

The page documents the problem, root cause, the workaround (the
kubevirt.io/placePCIDevicesOnRootComplex annotation — set automatically for
Flatcar by kubermatic/machine-controller#2057, with a manual MachineDeployment
fallback), the Flatcar-only scope, the trade-off (disables unused PCIe hotplug),
and upstream references.

Related

• Workaround: kubevirt: pin Flatcar VMs to PCI root complex to preserve Ignition machine-controller#2057
• Upstream root cause: [sidecars] XML round-trip via xml.Unmarshal/xml.Marshal drops or corrupts namespace-scoped domain sections (qemu:commandline/qemu:arg) kubevirt/kubevirt#16901
• Upstream fix (open): sidecar hooks: preserve QEMUCmd across XML round-trip kubevirt/kubevirt#18460

[kubermatic-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign simontheleg for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[archups]

Please take care of review comments for workaround section.

Also please add known issue under KKP docs as well to main and supported releases.

/hold until MC release is out. And make necessary changes in the workaround to include how to pin to the MC release containing the fix until it is available as part of KKP release using following change in KubermaticConfiguration

spec:
  userCluster:
    machineController:
      imageTag: v1.xx.xx

[archups]

Suggested change

	Note: the annotation must sit on `spec.template.metadata.annotations` (this is
	what reaches the VM). It only takes effect on newly created machines, so
	existing Flatcar nodes must be recreated for the fix to apply.
	**Note:** The annotation must sit on `spec.template.metadata.annotations`, not the
	MD's top-level metadata. This is what reaches the VM.

[archups]

Update this changes across all the occurrence.

[archups]

Suggested change

	KubeVirt exposes a per-VM annotation, `kubevirt.io/placePCIDevicesOnRootComplex`,
	that disables the extra hotplug-port reservation and therefore skips the second
	domain write, so the `-fw_cfg` argument is preserved.
	machine-controller sets this annotation automatically for Flatcar machines as of
	[kubermatic/machine-controller#2057](https://github.com/kubermatic/machine-controller/pull/2057).
	Deploy a machine-controller version that includes this change and recreate the
	affected Flatcar nodes.
	If you cannot update machine-controller yet, add the annotation manually to the
	Flatcar worker pool's `MachineDeployment` in the user cluster (namespace
	`kube-system`) and roll the nodes:
	KubeVirt exposes a per-VM annotation, `kubevirt.io/placePCIDevicesOnRootComplex`,
	that preserves the `-fw_cfg` argument on Flatcar nodes. Set it to `"true"` using
	one of the options below.
	**Existing user cluster (affected MachineDeployment)**
	Patch the `MachineDeployment` in the user cluster (namespace `kube-system`), then recreate the Flatcar nodes:
	`kubectl -n kube-system patch machinedeployment <flatcar-worker-pool> --type merge -p '{"spec":{"template":{"metadata":{"annotations":{"kubevirt.io/placePCIDevicesOnRootComplex":"true"}}}}}'`
	The annotation only takes effect on newly created machines, so existing Flatcar nodes must be recreated for the fix to apply.
	**New user cluster / MachineDeployment**
	Add the annotation under `spec.template.metadata.annotations` at creation time, either in the MD YAML, or directly from the KKP dashboard during user cluster or MachineDeployment creation:

[archups]

Update this Workaround changes across all the occurrence.