Homelab Day 13

Day 13

June 13, 2026 · 6 min read

#Homelab

See intro for what this is about.

Per previous days, entirely written by Claude.

A day of networking fixes and migrations. Fixed the e1000e EEE hang once and for all (the fix from last time wasn’t firing at the right point), got Tailscale running across the whole fleet with subnet routing, hit a painful self-inflicted routing loop, and migrated the GitOps manifests into a proper dedicated repo with full history.

Also wired up Pushover alerting to Alertmanager and hit a disk pressure incident on the cluster that prompted expanding the k3s VM disks.

e1000e EEE fix: the rule was firing too early #

The udev rule from Day 12 disabled EEE when the NIC was added (ACTION=="add"). EEE comes back. The issue is that add fires when the kernel first enumerates the device — before the link negotiation runs. EEE is re-enabled during link-up, which fires as a change event.

The rule needed to match both:

SUBSYSTEM=="net", ACTION=="add|change", DRIVERS=="e1000e", \
    RUN+="/sbin/ethtool --set-eee $name eee off"

Applied the fix, manually ran ethtool --set-eee eno2 eee off on any hypers that were in a hung state, and the NICs have been stable since.

Tailscale: installation #

Added a Tailscale role to Ansible. will and all three hypervisors are now on the tailnet, which gives SSH access from anywhere (phone, remote locations) without port forwarding or exposing anything to the internet.

The tricky part was idempotency. My first attempt used creates: /var/lib/tailscale/tailscaled.state to skip tailscale up on already-running hosts. This was wrong — tailscaled creates a 2-byte stub file immediately on first launch, so the task was always skipped even on fresh installs. Fixed it by checking tailscale status --json and only calling tailscale up when BackendState != 'Running'.

One other gotcha: the auth key needs to be in group_vars/all/vault.yml (not group_vars/management/vault.yml) because both the management and hyper inventory groups need it. Putting it in the management group vault causes variable undefined on the hypervisors.

Tailscale subnet routing: advertise 10.1.1.0/24 #

The goal was to reach all LAN hosts (VMs, k3s nodes, Traefik services) from anywhere without installing Tailscale on every individual machine. All three hypervisors advertise 10.1.1.0/24; if one goes down, Tailscale fails over to another.

This required --advertise-routes=10.1.1.0/24 in the Tailscale role and manual approval in the Tailscale admin console (Machines → Edit route settings).

The routing loop #

Setting --accept-routes on any LAN-native host is a mistake, and I had it set everywhere.

Tailscale installs the advertised subnet route into a separate routing table (table 52) with a policy rule evaluated before the main table. A host that both advertises and accepts 10.1.1.0/24 ends up routing its own LAN traffic via tailscale0 instead of br0. Replies go out the wrong interface and connections break asymmetrically — SSH drops, ping fails, but ARP still works because it doesn’t go through the routing table.

This hit in two stages:

Stage 1 — will: After the hypervisors started advertising the subnet route, will (which had --accept-routes) accepted the route. SSH from dan to will stopped working. Fixed by tailscale set --accept-routes=false on will.

Stage 2 — hyper2 and hyper3: Hypervisors also had --accept-routes. Even though they’re the subnet routers, they were seeing the route advertised by each other and accepting it — installing a route to 10.1.1.0/24 dev tailscale0 in table 52, which took precedence over the physical NIC route. Full LAN outage on hyper2 and hyper3. Diagnosed with:

# Table 52 on hyper2 had the poisoned route:
ssh 100.82.76.10 "ip route show table 52"
# → 10.1.1.0/24 dev tailscale0

Fixed with tailscale set --accept-routes=false on all hypervisors. Added tailscale__accept_routes: false as the default in the Ansible role — no server in this lab ever needs to accept routes; only roaming clients do.

firewalld blocking subnet routing #

Once accept-routes was disabled, routing still didn’t work from the phone. DNS resolved correctly (the phone was getting 10.1.1.200 back) but connections timed out.

Root cause: tailscale0 was not assigned to any firewalld zone. Interfaces not in a zone default to public, which doesn’t allow forwarding to other zones. Packets arriving from the phone on tailscale0 were being dropped before they could be forwarded to br0 (in the trusted zone).

firewall-cmd --get-zone-of-interface=tailscale0
# no zone

Fix: add tailscale0 to the trusted zone. Added to the Tailscale role with a guard that checks whether firewalld is actually running (it isn’t on will, only on the hypervisors).

After this, the full path from phone on mobile data worked: Tailscale split-DNS resolves alertmanager.lab.implicit.net to 10.1.1.200, traffic routes via the active hypervisor subnet router, Traefik serves the response.

dnsmasq: listen on the Tailscale address #

For split-DNS to work from outside the LAN, dnsmasq on will needed to listen on the Tailscale interface (100.117.158.49), not just 127.0.0.1 and the LAN IP. The previous config had those hardcoded.

Converted the upstream.conf generation from a static file copy to a Jinja2 template that conditionally adds the Tailscale address when the ansible_tailscale0 interface fact exists. Self-correcting: no extra role dependency needed, and a host without Tailscale gets the same config as before.

Pushover alerting #

Added a Pushover receiver to Alertmanager. Critical and warning alerts now push to my phone. InfoInhibitor alerts are excluded from Pushover routing — they’re a VictoriaMetrics internal mechanism that suppresses info-level alerts when their parent warning is firing, not real events worth a notification.

etcd scraping was also fixed: k3s exposes etcd metrics on port 2381, but the scrape config was using the wrong CRD type (serviceMonitor instead of vmScrape) and the wrong scheme. Fixed both. Also disabled the kubeScheduler and kubeControllerManager scrape targets — those components are embedded in the k3s binary and don’t have the metric endpoints the rules expect, so they fire false-positive alerts.

k3s VM disk pressure #

The k3s VMs were provisioned with 30 GiB disks in the original Terraform config. A DiskPressure condition appeared on a node — the monitoring stack, Gitea runner, and application images consume more than anticipated. The DinD runner’s Docker image layers alone take several gigabytes.

Expanded the k3s-1/2/3 disk allocation in terraform/incus/main.tf from 30 GiB to 60 GiB. Terraform updated the Incus VM disk config; Incus handles the online-resize.

gitops → argocd migration #

The GitOps manifests had split history: some commits from when they lived in the ansible repo, more from the standalone gitops repo. Created a unified chris/argocd repo with both histories merged in chronological order:

Extracted the gitops/ directory history from the ansible repo using git filter-repo --subdirectory-filter gitops — 11 commits, original dates preserved. Dropped the final commit (a pure deletion when the directory was moved out).
Fetched the gitops repo and rebased its commits (excluding the initial import dump) onto the tip of the extracted history, using --committer-date-is-author-date to preserve original timestamps.
Created chris/argocd on Gitea, pushed the 20-commit unified history, updated all Application manifests to point at the new repo URL, and applied them to the cluster.

ArgoCD re-synced cleanly. Updated the Ansible ArgoCD role to reference the new repo for future rebuilds.

Blog