See intro for what this is about.

Per previous days, entirely written by Claude.

The big one: self-hosted identity provider. Every service in the lab was previously either completely open on the LAN or had its own local auth. Today all the monitoring ingresses require an Authentik login, and the whole thing is managed as code. Also: a second e1000e TX hang mode identified and suppressed, Terraform repo restructured, CI wired up so Terraform changes apply automatically on push, and a Traefik HTTP→HTTPS redirect that took three attempts to get right due to a silent values-key mismatch in the Helm chart.

Authentik: self-hosted IdP #

Authentik is a self-hosted identity provider with a first-class Traefik forwardAuth integration. The flow: a browser hits a protected ingress, Traefik’s forwardAuth middleware calls Authentik’s outpost to check whether the request is authenticated, and Authentik redirects to a login page if not. After login, subsequent requests are allowed through.

The ArgoCD Application uses the multi-source pattern: the Helm chart from charts.goauthentik.io plus a values override from the chris/argocd repo.

Authentik 2026.x has embedded Valkey (a Redis fork) baked into the server binary — no Redis subchart needed. The Bitnami PostgreSQL subchart is enabled for the database, using the zfs-iscsi StorageClass for an 8 GiB iSCSI-backed PVC on dan. Same rationale as vmstorage: if the node goes down, the pod can reschedule without losing data.

Secrets follow the existing pattern — no Sealed Secrets or ESO deployed. A pre-created authentik-env Kubernetes secret holds AUTHENTIK_SECRET_KEY and the PostgreSQL password. The chart reads it via envFrom: secretRef, suppressing the chart-managed secret entirely. The AUTHENTIK_SECRET_KEY must never change after first install — it’s used to encrypt sessions and tokens.

hyper3 went down mid-deploy #

While the Authentik ArgoCD Application was syncing, hyper3 went down. k3s-3 lost its kubelet and pods on that node were stuck Terminating for over an hour — the standard graceful termination path requires the kubelet to acknowledge the deletion, which it can’t do when the node is gone.

The ArgoCD application-controller was one of the stuck pods, which blocked all subsequent syncs. Force-deleted everything Terminating in the argocd namespace:

kubectl get pods -n argocd --no-headers | awk '/Terminating/{print $1}' | \
  xargs kubectl delete pod -n argocd --force --grace-period=0

ArgoCD recovered and the Authentik Application finished syncing once it could get a new controller pod scheduled on a healthy node. hyper3 came back up on its own; k3s-3 rejoined the cluster and the remaining workloads rescheduled.

forwardAuth: protecting monitoring ingresses #

The Traefik forwardAuth Middleware is deployed inline in the Authentik Helm values via additionalObjects. Any ingress can then opt in with a single annotation:

traefik.ingress.kubernetes.io/router.middlewares: authentik-authentik-forward-auth@kubernetescrd

But the middleware alone does nothing — Authentik also needs a Proxy Provider and Application configured for each protected service, and the Embedded Outpost must be assigned those providers. That’s where Terraform comes in.

Terraform: Authentik app management #

Added ~/git/terraform/authentik/ as a second Terraform module in the repo, using the goauthentik/authentik provider. It manages:

• A authentik_provider_proxy (forward_single mode) per protected service
• An authentik_application per service
• The authentik_outpost resource (the Embedded Outpost) with all providers assigned

Adding a new protected service is one entry in a protected_apps local map in applications.tf plus the middleware annotation on the ingress. terraform apply handles everything in Authentik.

A few issues hit along the way:

invalidation_flow is required but poorly documented. The authentik_provider_proxy resource requires it. Found the available flow slugs via ak shell (Django shell in the Authentik server pod).

Token newline bug. Passing TF_VAR_authentik_token=<value> inline on the command line caused invalid header field value — the shell was inserting a trailing newline into the value. Fixed by export TF_VAR_authentik_token=... first.

data.authentik_outpost provider bug. The provider version (2024.12) had a schema validation bug with the data "authentik_outpost" data source. Worked around it by dropping the data source and hardcoding the outpost UUID directly, retrieved via ak shell.

Provider version mismatch. Adding icons via meta_icon on authentik_application resources returned 405 Method Not Allowed from the set_icon_url API — the 2024.12 provider doesn’t match the 2026.5.3 server’s API. Bumped the provider constraint to ~> 2026.5.

Monitoring ingresses now protected: alertmanager, vmagent, vmselect. Grafana left un-protected by forwardAuth — it will use OIDC instead. ArgoCD left alone — it has its own auth.

e1000e TX hang mode 2: TSO-triggered #

hyper3 went down again with a “Detected Hardware Unit Hang” on eno2, despite EEE being confirmed off. No preceding link event in the kernel log — the EEE fix only applies to link-renegotiation hangs. This was different.

The hang dump showed TDH (TX descriptor head) frozen with 69 descriptors pending and next_to_watch.status = 0. The DMA engine had stalled completely mid-chain with no link event. Root cause: TCP Segmentation Offload (TSO) passes large TCP segments to the NIC as long descriptor chains, and the e1000e/i219-LM DMA engine deadlocks mid-chain under certain traffic patterns.

Extended the udev rule to also disable TSO and GSO:

SUBSYSTEM=="net", ACTION=="add|change", DRIVERS=="e1000e", \
    RUN+="/sbin/ethtool --set-eee $name eee off", \
    RUN+="/sbin/ethtool -K $name tso off gso off"

Also fixed the handler: the previous handler ran udevadm control --reload-rules but didn’t trigger the rules on already-active interfaces. Added a second handler step (udevadm trigger --subsystem-match=net --action=change) so deployments apply the ethtool settings immediately without waiting for a link bounce.

Two distinct e1000e TX hang modes are now documented and mitigated:

1. EEE-triggered (at link-up): ethtool --set-eee ... eee off
2. TSO-triggered (under load): ethtool -K ... tso off gso off

Terraform repo restructure #

main.tf (Incus VMs) was at the repo root while authentik/ was a subdirectory — inconsistent with two independent backends in one repo. Moved the Incus config into incus/ so the layout mirrors the logical structure:

terraform/
├── README.md
├── incus/
└── authentik/

State migration: copied terraform.tfstate to incus/, ran terraform init, confirmed terraform plan showed no changes, deleted the old root state file.

Terraform CI: act_runner on will #

Added Gitea Actions workflows to the terraform repo:

• authentik/: fmt-check + validate + plan on every push; apply on push to master
• incus/: fmt-check + validate + plan only — no auto-apply (a bad push could destroy VMs)

The existing Gitea Actions runner is a Docker-in-Docker pod in k3s. It can’t reach Terraform state files or Incus client TLS certs, which live on will. Rather than add a remote state backend, added a second act_runner directly on will as a systemd service, running as the chris user with a host executor (no Docker). New gitea-act-runner Ansible role handles install and registration.

The workflows skip actions/checkout and instead do a git pull against the existing checkout on will, then run terraform from the real directory. State stays in place, providers are already initialized, and no remote backend is needed.

Global HTTP→HTTPS redirect #

Previously, visiting http://auth.lab.implicit.net returned a 404 from Traefik rather than redirecting to HTTPS. This turned into a debugging exercise.

k3s manages Traefik via HelmChartConfig — a CRD that feeds extra Helm values to the bundled Traefik chart. The redirect is configured there and managed by the gitea ArgoCD Application alongside the Gitea SSH entrypoint.

The first attempt used ports.web.redirections.entryPoint. The second attempt used ports.web.redirectTo. Both synced cleanly through ArgoCD and Helm reported a successful upgrade each time — but the Traefik pod never rolled and curl http://... kept returning 404.

The tell was comparing helm get values traefik -n kube-system (values were there) against kubectl get deployment traefik -o jsonpath='{...args}' (no redirect arg). Helm was accepting the values but the chart wasn’t rendering them. Pulled the chart and checked templates/_podtemplate.tpl directly — the template iterates ports.<name>.http.redirections.entryPoint. There is a required http intermediate key, and wrong keys are silently ignored with no error.

The correct config:

ports:
  web:
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true

This renders to --entryPoints.web.http.redirections.entryPoint.* CLI args, rolls the pod, and produces a 301 on HTTP. No per-ingress configuration needed — every current and future service is covered at the entrypoint level.

ArgoCD metrics and Grafana dashboard #

ArgoCD exposes Prometheus metrics on each of its components but the Helm chart doesn’t create Kubernetes Services for those ports. Added four ClusterIP Services, a VMServiceScrape (cross-namespace, pointing at the argocd namespace), and Grafana dashboard #14584 loaded via ConfigMap. The Grafana sidecar picks up any ConfigMap with grafana_dashboard: "1" — no restart needed.

Nameserver VMs: monitoring coverage #

The three nameserver VMs (ns1/2/3, one per hypervisor) were missing node_exporter. Getting it installed turned into an unexpected adventure.

The VMs were provisioned with 512 MiB RAM. dnf install node-exporter requires loading EPEL metadata, which takes ~660 MB RSS — the Ansible Python dnf module got OOM-killed every time. The solution seemed obvious: resize the VMs. Less obvious: Incus cannot hot-increase VM memory beyond the boot-time allocation. You have to stop the VM, change the config, and start it again. The VMs also have autostart behaviour that races against the stop. Eventually got to 2 GiB via incus config set on each hypervisor with autostart temporarily disabled.

Even at 2 GiB the Ansible Python dnf module OOM-killed. Calling dnf via ansible.builtin.shell instead of ansible.builtin.dnf works fine — the shell just execs the binary; the Python module loads the entire dnf library into the Ansible process.

Also fixed the extra-repos role to install dnf-plugins-core first (provides the config-manager subcommand, absent on fresh AlmaLinux installs).

DNS for infrastructure hosts #

dnsmasq already had static entries for dan and glootie. Extended it to cover all infrastructure hosts — hypers, k3s nodes, nameservers, will — so every host is addressable by name under lab.implicit.net. The VMStaticScrape for bare-metal node_exporter targets uses these names rather than IPs.

Commits #

ansible #

• Add gitea-act-runner role for shell runner on will
• Disable TSO/GSO on e1000e NICs to prevent TX hang
• Add nameservers to inventory and enable monitoring
• Add all infrastructure hosts to lab.implicit.net DNS

argocd #

• Add Authentik IdP: ArgoCD Application and Helm values
• Protect monitoring ingresses with Authentik forwardAuth
• Configure global HTTP→HTTPS redirect in Traefik (wrong values key — silent no-op)
• Fix Traefik HTTP→HTTPS redirect using correct v3 Helm chart key (second wrong key)
• Fix Traefik redirect: nest redirections under ports.web.http
• Add ArgoCD metrics scraping and Grafana dashboard
• Add VMStaticScrape for bare-metal node_exporter targets

terraform #

• Add Authentik Terraform config for proxy provider management
• Manage embedded outpost provider assignments in Terraform
• Add application icons and bump provider to 2026.5
• Move incus config to incus/ subdir, add README
• Add Gitea Actions workflows for Terraform CI
• Resize nameserver VMs to 2GiB RAM