home-dc-kubernetes, Part 1: The Rebuild Source of Truth

Most homelabs are a pile of services held together by muscle memory. Mine used to be too — until I deleted it. Not by accident: deliberately, so I could prove the repo was telling the truth.

This is home-dc-kubernetes: the repository that can rebuild my entire home data centre from a git clone plus a handful of secrets. Not “restore from backup” — rebuild from declarative source. This series walks through how it works, layer by layer.

(If you read my earlier “From Swarm to Kubernetes” post, you’ll recognise the story — that article referenced the repo under its old name, project-homelab. It’s been renamed; everything I describe here is the current state.)

What the repo actually is #

The README’s first line is the mission statement:

This repository is the rebuild source of truth for the Kubernetes homelab.

It manages two clusters, deliberately different:

One repo, two clusters, one control plane: Argo CD runs on the app cluster and manages both. The infra cluster is a registered Argo target, not a separate GitOps installation. That’s the pattern worth stealing.

The rebuild flow — seven stages #

The repo is organised as a staged recovery runbook, not just a pile of manifests:

Stage 0: Recovery inputs    age.key, kubeconfigs, tfvars, CEPH_KEYRING
Stage 1: Workstation        clone → nix develop → task deps
Stage 2: Infrastructure     OpenTofu: proxmox, cloudflare, localdns
Stage 3: App cluster Talos  generate config → bootstrap
Stage 4: App cluster GitOps apps:bootstrap → sync-argo → reconcile
Stage 5: Infra cluster      physical Talos nodes → register with Argo hub
Stage 6: Storage, DNS, pub  Ceph CSI, Cloudflare tunnel/DNS, Doppler
Stage 7: Final validation   task validate, verify:cluster, clusters:status

Each stage has an exit check. Stage 7 is explicit about what “done” means: both Lens contexts exist, every Argo Application is Synced and Healthy, no manifest targets in-cluster, no duplicate paths in git status. Rebuild is a checklist, not a hope.

Why “rebuild” instead of “backup” #

Backup asks: can I get my data back? Rebuild asks: can I get my infrastructure back? The difference matters. Backups rot silently — nobody tests the restore until the fire. Rebuild-from-source is tested every time a commit lands, because the manifests are the same artifacts CI validates and Argo syncs.

The repo even treats local-only inputs as a first-class recovery problem. The README doesn’t just say “secrets live here” — it documents where each secret comes from, what happens if it’s lost, and when losing it means regenerating a cluster identity instead of restoring it. The age.key loss path is a documented procedure (task template:encrypt-secrets), not a panic.

What runs on it #

Two clusters, ~25 namespaced workloads. The app cluster carries the media stack (Jellyfin, Sonarr, Radarr, Immich, qBittorrent, SabNZBD, Tdarr, Prowlarr), productivity (code-server, Linkwarden, Stirling PDF), monitoring (kube-prometheus-stack), network (Cloudflare DNS + tunnel, Envoy Gateway, Technitium), and web (Glance homepage). The infra cluster runs core services: Kestra automation, Uptime Kuma, Pulse agents, Tailscale, local DNS.

Notable stack choices, all documented in docs/architecture/IMPLEMENTATION_DECISIONS.md:

• CNI: Cilium, with kube-proxy replacement enabled
• Ingress: Envoy Gateway (Traefik demoted to “historical design idea”)
• Storage: Ceph CSI via RBD on the Proxmox Ceph pool; Longhorn deferred
• Secrets: SOPS + age in git, Doppler operator for runtime secrets
• GitOps: Argo CD, app-of-apps layout, cluster-aware paths
• Access: Tailscale subnet routers for node and service access

What’s next in this series #

• Part 2 — Talos as the Kubernetes OS: machine configs, SOPS secrets, bootstrap
• Part 3 — Argo CD app-of-apps: the cluster-aware manifest layout
• Part 4 — the platform layer: Cilium, Envoy Gateway, Ceph CSI, Cloudflare
• Part 5 — the app catalog: what actually runs
• Part 6 — the operating model: Taskfile, CI, Renovate, recovery drills