home-dc-kubernetes, Part 3: Argo CD App-of-Apps and the Cluster-Aware Layout

Table of Contents

The Nervous System of the Homelab #

In Part 1 we established the “Rebuild Source of Truth”. In Part 2 we made the foundation immutable with Talos. Now we get to the part that keeps everything honest: Argo CD.

The goal was simple: never trust a kubectl apply that happened once and was forgotten. If Git is the only source of truth, the cluster is just a mirror — and a mirror can be rebuilt from nothing.

Argo CD Hub managing App and Infra clusters — The Argo Hub model: one management instance controlling both the production App cluster and the underlying Infra cluster.

The App-of-Apps Pattern #

Managing 40+ applications individually in the Argo UI is a non-starter. Instead, the repo uses the app-of-apps pattern: a Root Application that doesn’t manage pods — it manages other Application manifests.

When the Root App syncs, it reads a folder in the repo and says: “everything in here is another app that needs to exist”. One sync at the top propagates down the tree to every service in the lab.

The hierarchy: Root App → Cluster App (app/infra) → Namespace App (network/storage) → The Workload.

Cluster-Aware Manifest Layout #

A flat folder structure is the classic GitOps mistake. Here, the directory structure is the design document:

kubernetes/argo/apps/
├── app-cluster/          # the workload cluster (Talos VMs)
│   ├── network/          # cloudflare-tunnel, envoy-gateway, technitium-dns
│   ├── media/            # jellyfin, sonarr, radarr, immich, ...
│   ├── productivity/     # code-server, linkwarden, n8n, sterling-pdf
│   ├── monitoring/       # kube-prometheus-stack
│   └── storage/          # ceph-csi
└── infra-cluster/        # the physical infrastructure cluster
    ├── network/          # cloudflare-tunnel-infra, tailscale-infra
    ├── monitoring/       # uptime-kuma, pulse
    └── storage/          # local-path-provisioner

The filesystem maps explicitly to the physical and virtual boundaries of the hardware. If a file lives under infra-cluster/network/, you know exactly where it deploys and what it does — without opening a single YAML.

Symmetry with Isolation #

Both clusters need a Cloudflare Tunnel, but they get two distinct manifests:

app-cluster/network/cloudflare-tunnel
infra-cluster/network/cloudflare-tunnel-infra

This is deliberate: a mistake in the app cluster’s networking must never take down the infra tunnel and lock me out of the lab. Same pattern, separate blast radius.

The Templating Layer: Jinja2 + Helm #

The manifests under apps/ are the rendered results. The real logic lives in templates/config/ as Jinja2 templates. Repository name, branch, and common settings are templated once — so a dev → main promotion or a repo rename touches one file, not fifty.

# templates/config/kubernetes/components/common/settings.yaml.j2
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: projects
  namespace: argo-system
spec:
  project: default
  source:
    repoURL: "https://github.com/#{ repository_name }#.git"
    path: kubernetes/argo/settings
    targetRevision: #{ repository_branch }#
  destination:
    name: app-cluster

Why This Matters #

The layout answers the three questions every ops person asks at 2am during an outage:

What is running? → read the folder tree
Where does it run? → app-cluster/ vs infra-cluster/
What version? → the pinned targetRevision in each Application

Next up: the Platform Layer — Cilium, Envoy Gateway, Ceph CSI, and how Cloudflare ties it all together.