Skip to content

Cluster Recovery & Wiki Launch

Date: 2026-07-26 Category: infra Status: Complete 🟢

Summary

Full recovery from cluster crash: MetalLB operator rebuilt, kube-vip load balancer restored, all ingresses verified working, and wiki.emilsturzenegger.ch launched.

What Happened

The k3s cluster experienced a full crash affecting all nodes. After node restart and etcd recovery:

MetalLB

  • MetalLB operator pod was missing on the new deployment but the CRDs remained
  • FRR pods were stuck in CrashLoopBackOff due to stale FRR configuration
  • Fix: Deleted stuck pods, forced fresh operator deployment — all FRR pods reached 5/5 Ready

kube-vip (VIP Load Balancer)

  • The VIP 192.168.30.104 was pingable (ARP working) but ports 80/443 were not forwarding
  • kube-vip pods restarted during cluster crash with API connectivity errors before the cluster was healthy, preventing the static loadBalancers config from being applied
  • Fix: Force-deleted and recreated all kube-vip pods once the cluster was healthy — leader elected on k3s-node-2, port forwarding activated immediately

Verification Results

All 7 ingress services returned valid HTTP responses through the VIP:

Service HTTP Code Notes
Jellyfin 302 → /web/ Redirect to web UI
Sonarr 200 ✅
Radarr 200 ✅
Prowlarr 200 ✅
SABnzbd 200 ✅
Bazarr 200 ✅
Seerr 307 Redirect

wiki.emilsturzenegger.ch Launched

  • MkDocs Material theme with three-tab navigation: Updates & Reports, Lab, Products
  • Source in GitLab: emilsturzenegger/homelab/docs
  • Deployed as a K8s service in the wiki namespace (nginx + MkDocs initContainer pattern)
  • DNS record added to Technitium DNS

Lessons Learned

  1. kube-vip pods need a clean restart after cluster recovery — stale API connectivity errors at startup prevent the static loadBalancers config from applying
  2. MetalLB FRR config persists across pod restarts — when the operator is recreated, old FRR config from the old node pool can cause CrashLoopBackOff. Delete the old FRRConfig/FRRK8sConfig CRDs to let the new operator generate fresh config
  3. Verify the full ingress chain after any cluster-level restart — ping alone (ARP) isn't enough; ports 80/443 must be confirmed