See intro for what this is about.

Per previous days, entirely written by Claude.

A big day. Went from three bare hypervisors with Incus installed to a full GitOps-managed Kubernetes cluster with monitoring, TLS, and a CI runner.

The rough order of events:

Got a three-node k3s cluster running with embedded etcd. All three nodes are servers — no dedicated agents. k3s comes bundled with its own load balancer (klipper-lb / servicelb) but I wanted MetalLB for proper L2 ARP advertisement, so I had to disable klipper-lb explicitly in the k3s config before MetalLB would take over the traefik LoadBalancer service.

MetalLB gives Traefik a stable VIP at 10.1.1.200 out of a pool of 10.1.1.200–220. The VIP is pinned via a metallb.universe.tf/loadBalancerIPs annotation on the traefik service.

dnsmasq on will now provides split-horizon DNS for the whole LAN: *.lab.implicit.net resolves to the Traefik VIP, so any service with a Kubernetes Ingress at foo.lab.implicit.net is reachable from anywhere on the network without any per-service DNS entry.

Also added a git.implicit.net override to the same VIP. Gitea runs on a bare metal box at 10.1.1.251. Routing both HTTPS and SSH through Traefik required two separate mechanisms: an Ingress for HTTPS (port 443) and a Traefik IngressRouteTCP for SSH (port 22). The TCP route needed an extra Traefik entrypoint configured via a HelmChartConfig patch. Without the DNS override, in-cluster pods would resolve git.implicit.net to the public IP, go out the router, come back in via port forwarding, and break — so CoreDNS forwards to will, and will answers with the internal VIP.

cert-manager handles TLS for everything. It uses Cloudflare DNS-01 for Let’s Encrypt challenges so it can issue certs for *.lab.implicit.net even though those names are never publicly resolvable. The Cloudflare API token lives in an Ansible Vault file and gets written into a Kubernetes secret by the cert-manager role.

VictoriaMetrics went in as the monitoring stack, deployed via the victoria-metrics-k8s-stack Helm chart. I’m running the cluster topology (vminsert, vmselect, vmstorage ×3) rather than vmsingle. Hit two issues: the annotation payload for the Helm release was too large for a regular kubectl apply (Kubernetes stores the last-applied config in an annotation, and the VictoriaMetrics chart is big) — fixed by switching to server-side apply. Also some of the default alerting rules fire false positives on k3s because the scheduler and controller manager are embedded in the k3s binary rather than running as separate services; disabled those rules in the Helm values.

ArgoCD watches the gitops/ directory tree in the ansible repo and reconciles the cluster. The deploy key setup is automated by the Ansible role: generates an ed25519 key, registers the public half with Gitea via the API, stores the private half in a Kubernetes secret. One gotcha: ssh-keygen prompts for a passphrase interactively unless you explicitly pass -N "". Without that flag it blocks the playbook.

Set up a Gitea Actions runner inside the cluster. The runner pod is three containers sharing a volume for the Docker socket: an init container that registers with Gitea, a DinD (Docker-in-Docker) sidecar for the daemon, and the actual act_runner. The init container was waiting for the Docker daemon using docker info before starting — this kept timing out because the daemon wasn’t ready yet. Fixed by waiting on the socket file instead, which appears as soon as dockerd starts even before it’s accepting connections.

Scaffolded the eloise FastAPI app and gave it a CI pipeline via Gitea Actions.

Also wrote a detailed runbook in the homelab repo covering the full stack from PXE to GitOps.

Commits #

ansible #

• Integrate k3s and VictoriaMetrics into site.yml via roles and group_vars
• Move will host_vars into management group_vars
• Add HA k3s, MetalLB, cert-manager, dnsmasq, NTP, and HTTPS ingress
• Disable klipper-lb (servicelb) on k3s servers so MetalLB takes sole control
• Single source of truth for Traefik VIP; pin MetalLB assignment explicitly
• Add README.md to every role documenting purpose, variables, and implementation
• Migrate in-cluster resources from Ansible to ArgoCD GitOps
• Generate passphrase-free ArgoCD deploy key; auto-register with Gitea API
• Disable k3s false-positive alert rules in victoria-metrics Helm values
• Use server-side apply for victoria-metrics to avoid annotation size limit
• Install ArgoCD CLI via get_url in argocd role
• Replace hardcoded Gitea IP with https://git.implicit.net hostname
• Add split-horizon DNS override for git.implicit.net
• Route git.implicit.net through Traefik for split-protocol access
• Add gitea-runner and eloise gitops apps
• Fix runner start.sh to wait on socket file not docker CLI

eloise #

• Add Dockerfile and Gitea Actions CI workflow

homelab #

• Initial runbook: PXE to VictoriaMetrics deployment guide