See intro for what this is about.

Per previous days, entirely written by Claude.

A day mostly spent cleaning up after the gitops→argocd migration, then adding persistent storage for vmstorage using iSCSI-backed PVCs. Also hit a good lesson about Helm values: a typo at a non-existent key path is silently dropped.

victoria-metrics operator deadlock #

Alertmanager was firing TooManyScrapeErrors for the victoria-metrics-operator target. The operator pod showed Running / 1/1 Ready but port 8080 (metrics) had gone dark. The liveness probe was a TCP socket check on port 8081 (the health endpoint), which was still accepting connections — so kubelet saw the pod as healthy and never restarted it.

Root cause: on startup, the operator tries to expose PSI (Pressure Stall Information) cgroup metrics. k3s nodes don’t expose cpu.pressure in the cgroup hierarchy. The operator logs the failure and continues, but the goroutine involved doesn’t fully unwind — it deadlocks the metrics HTTP server on port 8080 while leaving 8081 (a separately written handler) responsive. The pod then sits in this half-alive state indefinitely until manually restarted.

Immediate fix: kubectl rollout restart deployment/victoria-metrics-operator. Port 8080 responded within seconds.

Prevention: add an HTTP liveness probe on port 8080 so the next deadlock triggers a kubelet restart within 45 seconds (3 × 15s period).

The probe fix that did nothing #

The liveness probe override was nested under the wrong Helm key:

# wrong — operator: is a real key but livenessProbe under it is silently ignored
victoria-metrics-operator:
  operator:
    livenessProbe:
      httpGet:
        path: "/metrics"
        port: 8080

Helm silently drops values at unrecognised key paths. ArgoCD reported Synced / Healthy because the rendered manifest was unchanged — the running pod still had the original tcpSocket:8081 probe. The operator deadlocked again several hours later.

The correct key, confirmed by helm show values:

victoria-metrics-operator:
  probe:
    liveness:
      httpGet:
        path: "/metrics"
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 15
      failureThreshold: 3

The lesson: Synced/Healthy in ArgoCD means the cluster matches the rendered chart. It does not mean your override did anything. Verify against the live object with kubectl get ... -o jsonpath=..., not the sync status.

ArgoCD migration fallout: four separate errors #

The gitops→argocd migration from Day 13 left a mess in ArgoCD’s internal state that produced four different errors.

“not our ref”: ArgoCD fetches commits by SHA (git fetch origin <sha>). This requires uploadpack.allowAnySHA1InWant = true on the Git server, which the newly created chris/argocd repo didn’t have. Setting it in Gitea’s app.ini only affects repos created after the change; existing repos are not retroactively updated. Fix: sudo git config --system uploadpack.allowAnySHA1InWant true on the Gitea host (/etc/gitconfig, applies to all repos immediately).

“Host key verification failed”: After the repo-server pod restarted during debugging, it loaded argocd-ssh-known-hosts-cm fresh — and git.implicit.net was not in it. Previous connections had been using insecure: "true" in the repository secret, which bypasses host key checking. Added the three key types for git.implicit.net (RSA, ECDSA, ed25519) via ssh-keyscan.

“Permission denied (publickey)”: The application status objects contained references to old repository URLs from before the migration (ansible.git, gitops.git). ArgoCD’s repo-server was trying to fetch those old SHAs from the old repos to render history in the UI — and had no credentials for them.

Stale operationState: Even after adding credentials for the old repos, the underlying issue was that status.history and status.operationState across six applications contained stale repo URLs and SHAs. The cleanest fix was to clear them:

for app in cert-manager eloise gitea gitea-runner metallb monitoring-ingress victoria-metrics; do
  kubectl patch app -n argocd $app \
    --type=json -p='[{"op":"replace","path":"/status/history","value":[]}]'
done

All seven applications went clean after this.

democratic-csi: iSCSI-backed PVCs #

Up to this point, vmstorage was using local-path PVCs, which pin pods to the node where the data lives. If that node is down, the pod can’t reschedule.

Added democratic-csi, which provisions iSCSI volumes from a TrueNAS-compatible target. The storage class (zfs-iscsi) uses the org.democratic-csi.iscsi provisioner and creates ZFS datasets on dan (10.1.1.251) on demand.

Advantages over local-path:

• Volumes are network-attached — pods can reschedule to any node
• reclaimPolicy: Retain — PVs survive PVC deletion (safe for databases)
• allowVolumeExpansion: true — resize is a one-liner

The Helm chart installs the controller and node components. The controller needs credentials to reach the TrueNAS API; the node components handle iSCSI initiator setup on each k3s node.

Migrated vmstorage to use the new zfs-iscsi StorageClass. The three vmstorage pods each got a 10 GiB PVC backed by a ZFS dataset on dan.

Commits #

ansible #

• Add democratic-csi iSCSI storage provisioning for vmstorage persistence

argocd #

• Add HTTP liveness probe on port 8080 for victoria-metrics-operator
• Fix operator liveness probe to correct subchart key (probe.liveness)
• Add democratic-csi driver and move vmstorage to zfs-iscsi StorageClass