See intro for what this is about.

Per previous day, entirely written by Claude.

Got incus VMs working end-to-end today. Also started a terraform repo to provision them properly.

AlmaLinux 9’s QEMU is a stripped-down RHEL build — no SPICE, no virtio-9p. incus passes -spice and -runas on the command line and expects these to work. I ended up writing a wrapper script that strips the unsupported flags before calling /usr/libexec/qemu-kvm. Annoying but it works.

A few other AlmaLinux 9 / incus compatibility issues I hit:

• incusd looks for incus-agent at /usr/libexec/incus/incus-agent but the package doesn’t put it there. Symlink required.
• virtiofsd is installed to /usr/libexec/virtiofsd but incusd hardcodes /usr/lib/virtiofsd. Another symlink.
• genisoimage isn’t installed by default but incus needs it to create the agent config ISO disk.
• OVMF_CODE.fd is missing from /usr/share/OVMF, which is the path incus probes for UEFI firmware at startup.
• Without br_netfilter loaded, bridge traffic from VM tap interfaces never reaches the IP netfilter stack, so incus’s nftables DHCP/DNS rules never match. Also need to put incusbr0 in the firewalld trusted zone.

For networking, I first tried macvlan (VMs get a router address but the hypervisor can’t reach them) then switched to a proper bridge. Created a NetworkManager br0 over eno2 so VMs get a DHCP address from the router and the hypervisor can also talk to them. Getting this to actually survive a reboot without NetworkManager getting confused by competing connection profiles took a few attempts.

For terraform, the base almalinux/9/cloud image has no SSH server and no user. Added cloud-init to install openssh-server and create my user with an SSH key. Hit one more AlmaLinux quirk: even with key auth, PAM’s account phase blocks accounts with a locked shadow entry (!!). Fixed by setting passwd: "*" in the cloud-init users block.

Added will to the Ansible inventory as the machine I’ll run terraform from. It gets incus installed but not the full hypervisor setup — just enough to reach the remote incus API on the hypers.

Commits #

ansible #

• Add will host, hyper group, and expand incus role
• Symlink qemu-kvm to qemu-system-x86_64
• Replace symlink with QEMU wrapper script for AlmaLinux 9
• Load br_netfilter and trust incusbr0 in firewalld
• Configure br0 bridge over eno2
• Fix br0 bridge activation
• Delete System eno2 profile before reboot
• Update hyper1/hyper2 IPs after bridge MAC change

terraform #

• Initial skeleton: AlmaLinux 9 VM on hyper1 via incus
• Initial terraform config for incus VM provisioning
• Document AlmaLinux 9 QEMU compatibility workarounds
• Fix VM networking and add cloud-init SSH config
• Add macvlan NIC
• Switch VM NIC from macvlan to bridged br0
• Fix cloud-init user creation for PAM key auth