See intro for what this is about.

Per previous days, entirely written by Claude.

A day of debugging CI, fixing the eloise deployment, and splitting repos and docs into their proper homes.

CI pipeline: three bugs in a trench coat #

The Gitea Actions runner was running but every job was failing. Three separate issues stacked on top of each other:

DinD on k3s needs DOCKER_DRIVER=vfs. k3s uses containerd with overlayfs as the storage driver. When you run Docker-in-Docker on top of that, the DinD daemon tries to use overlayfs too — but nested overlayfs isn’t supported and fails with cryptic “invalid output path: failed to save dying container to disk” errors. The fix is to force the inner Docker daemon to use the vfs storage driver, which is slower but doesn’t need kernel overlay support.

act_runner auto-mounts the Docker socket. The act_runner binary automatically mounts /var/run/docker.sock from the pod into each job container. I had also added -v /var/run/docker.sock:/var/run/docker.sock to the runner’s container.options config, resulting in a duplicate mount and a “mount point already in use” error. Removed the manual option.

GITHUB_TOKEN can’t push to the container registry. Gitea auto-generates a GITHUB_TOKEN (called GITEA_TOKEN internally) for each workflow run, scoped to the repo. It doesn’t have write:package permission, so attempting to docker login with it gives an authentication error. The fix is to create a personal access token with write:package scope and store it as a repo secret (REGISTRY_TOKEN), then use that secret in the login step.

eloise deployment: two more bugs #

Even after CI was pushing images successfully, the pod wasn’t coming up.

Python relative imports need a package context. The app used a relative import (from .db import ...). Running uvicorn main:app from inside the app directory treats the file as a top-level module, not a package, and relative imports fail with ImportError: attempted relative import with no known parent package. Fixed by restructuring the Dockerfile to copy the source into a subdirectory named after the package and invoking uvicorn as uvicorn eloise.main:app from the parent directory.

Private Gitea registry returns 404, not 401. Kubernetes tries to pull the image, gets a 404 (Gitea’s private registry returns 404 to unauthenticated requests rather than 401), and reports ImagePullBackOff. The fix is an imagePullSecret pointing at a pre-created docker-registry secret with a PAT that has read:package scope.

gitops repo split #

Extracted the gitops/ directory out of the ansible repo into a dedicated chris/gitops repo. Updated all ArgoCD Application manifests to point at the new repo URL, and updated the ArgoCD repo secret in-cluster. ArgoCD re-synced cleanly after the switch.

cobbler sync #

The cobbler git repo had drifted from the real Cobbler state on disk — netboot_enabled was true in the repo for all systems but false on the server (correct after first provision), and two objects (the alma9.7 distro and the roles::base management class) were missing from the repo entirely. Copied the real state in.

terraform #

Expanded the terraform config from a single test VM to the full 15-VM fleet. All VMs are defined in a locals map and created via for_each, with fixed MAC addresses for stable DHCP assignment, explicit CPU/memory limits, and sized root disks. The three k3s nodes get 4 vCPU / 8 GB / 30 GB; the general-purpose VMs get 2 vCPU / 4 GB / 10 GB.

Documentation #

Reorganised docs across three repos to put things where they belong:

• homelab/infrastructure.md — hosts, network layout, Cobbler, k3s cluster detail
• gitops/README.md — deployed workloads and day-two ops for adding services
• ansible/README.md — Ansible-only: roles, inventory, secrets, playbook ops

Commits #

ansible #

• Fix register.sh to cd /data before writing .runner state file
• Use vfs storage driver for DinD to avoid nested overlayfs failure on k3s
• Remove duplicate docker.sock mount from runner container options
• Add imagePullSecret for Gitea container registry
• Add comprehensive architecture documentation
• Add Cobbler PXE provisioning section to architecture docs
• Move gitops manifests to dedicated chris/gitops repo
• Move GitOps and workload docs to gitops repo, add pointer
• Scope README to ansible only; move infra docs to homelab repo

cobbler #

• Sync cobbler state: netboot disabled post-provision, add missing objects

eloise #

• Use REGISTRY_TOKEN for container registry auth
• Fix relative import error by running app as a package

gitops #

• Initial import from ansible repo
• Update Application manifests to reference gitops repo
• Add README documenting GitOps architecture and deployed workloads

homelab #

• Update runbook to reflect current stack
• Add ops log for 2026-06-12 session
• Add infrastructure.md; move arch reference from ansible README

terraform #

• Expand VM definitions to full 15-VM fleet with for_each