feat!: migrate to k3s + cloud-init; drop CRC/AAP

[xnoto]

Summary

Migrates tfroot-libvirt off the retired CRC/AAP stack onto a self-bootstrapping k3s VM, then iterates on the cloud-init bootstrap until the cluster reliably comes up green on Fedora 44.

Provisioning structure

• BREAKING: drops the AAP/Ansible Automation Platform integration entirely — provider, awx_* sops keys, runner module's enable_aap arguments.
• Runner VM: ports configure_runner.yml into cloud-init runcmd. Self-installs the GHA binary, registers via the org github_token PAT (shared with tfroot-github), runs as a systemd service. The PAT-bearing installer lives in /run/ so it doesn't survive reboot. installdependencies.sh is invoked because Fedora 44 doesn't ship libicu by default. Re-applies are idempotent against the persistent xfs /opt/actions-runner volume.
• k3s VM: new module "k3s" (6 vCPU / 16 GiB / 100 GiB) on a dedicated libvirt_pool "cluster" backed by /mnt/nvme/cluster. Cloud-init: SELinux permissive, k3s with Traefik+ServiceLB disabled, then a four-step bootstrap chain — sops-age-keys Secret → upstream cert-manager (with DNS-01 nameservers patched onto the controller args) → argocd-operator config/default + self-signed Issuer/Certificate to provision its webhook-server-cert → kubectl apply -k of kustomize-cluster/bootstrap.
• CI: runs on the arc-dind self-hosted runner with the GHCR tfroot-runner container.
• SSH credentials for the libvirt provider come out of sops and are materialized into .terraform/libvirt-ssh/ by the make libvirt-ssh target (auto-run by make init). No ~/.ssh/id_rsa required.

Operator instructions

This PR does not auto-rotate state. After merge:

1. Drop orphaned sops keys: awx_controller, awx_username, awx_password, proxyhost (already done in this branch).
2. Add github_token matching tfroot-github/secrets/secrets.yaml, plus sops_age_key, ops_ssh_privkey, hero_known_hosts, runner_ip_addr (already done in this branch).
3. One-time host setup: ssh user@hero 'sudo mkdir -p /mnt/nvme/cluster' (done).
4. make init then make apply. Boot-disk creation is ~5–7 minutes per VM; if you hit Volume Upload Failed: unexpected EOF just re-run.
5. Old GHA runner instance will orphan in the GitHub org on rebuild — delete via the API.

See AGENTS.md (also refreshed in this PR) for SSH access patterns, common apply hiccups, and recovery operations.

Test plan

• Pre-commit (terraform_validate, tflint, terraform_fmt, terraform_docs, sundry hooks) passes locally
• CI OpenTofu job passes
• make init && make apply from a fresh checkout produces 12 resources
• k3s VM boots, k3s service active, cert-manager + argocd-operator + ArgoCD CR + 3 Apps come up clean
• bootstrap-secrets Application reaches Synced + Healthy
• Runner VM registers a fresh runner with label libvirt, status online

Pairs with

• kustomize-cluster PR: drops OLM artifacts so gitops-operators can sync on k3s.
• images PR: gh-cli switches to numeric USER 1000 so runAsNonRoot passes for ci-token-sync.

🤖 Generated with Claude Code

[github-actions]

OpenTofu Plan

OpenTofu will perform the following actions:

  # module.k3s.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: k3s
            local-hostname: k3s
        EOT
      + name           = "k3s_commoninit"
      + network_config = <<-EOT
            version: 2
            ethernets:
              enp1s0:
                dhcp4: true
              enp2s0:
                dhcp4: false
                addresses:
                  - 192.168.102.2/24
        EOT
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

  # module.k3s.libvirt_volume.cloudinit will be updated in-place
  ~ resource "libvirt_volume" "cloudinit" {
      ~ allocation = 49152 -> (known after apply)
      ~ capacity   = 47104 -> (known after apply)
      ~ create     = {
          ~ content = {
              ~ url = "/Users/hatch/.claude/plugins/cache/context-mode/context-mode/1.0.15/terraform-provider-libvirt-cloudinit/cloudinit-bfb5c6617dbb44ba.iso" -> (known after apply)
            }
        }
        id         = "/mnt/nvme/cluster/k3s_cloudinit.iso"
        name       = "k3s_cloudinit.iso"
      ~ physical   = 47104 -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.runner.libvirt_cloudinit_disk.commoninit will be created
  + resource "libvirt_cloudinit_disk" "commoninit" {
      + id             = (known after apply)
      + meta_data      = <<-EOT
            instance-id: runner
            local-hostname: runner
        EOT
      + name           = "runner_commoninit"
      + network_config = (sensitive value)
      + path           = (known after apply)
      + size           = (known after apply)
      + user_data      = (sensitive value)
    }

  # module.runner.libvirt_volume.cloudinit will be updated in-place
  ~ resource "libvirt_volume" "cloudinit" {
      ~ allocation = 49152 -> (known after apply)
      ~ capacity   = 47104 -> (known after apply)
      ~ create     = {
          ~ content = {
              ~ url = "/Users/hatch/.claude/plugins/cache/context-mode/context-mode/1.0.15/terraform-provider-libvirt-cloudinit/cloudinit-b063448360566560.iso" -> (known after apply)
            }
        }
        id         = "/var/lib/libvirt/images/runner_cloudinit.iso"
        name       = "runner_cloudinit.iso"
      ~ physical   = 47104 -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 2 to add, 2 to change, 0 to destroy.