Swarm - the origin story #

After a few iterations and failed experiments (and pretty much 8 years of constant rebuild, merging technologies I used at work - including Kubernetes and Dell servers) into un-Godly hybrid abomination and fusions of technology that I think would put the Adeptus Mechanicus to shame, I have decided to go back to basics and finally get to a Homelab state that I am 95% happy with and I think I will stick to it now!

Before I start discussing the “how” however, I want to give a bit of background information on this project, and my decisions and reasons.

I started building homelabs probably around 2016. The “gateway drug” was a humble Raspberry Pi (can’t remember the version; 1 or 2?). The feeling when you figure out how to get a web server running and see your Nginx welcome page is something that I will never forget. Yeah, laugh if you want…but still!

Any way, as I was always moving in Infrastructure and networking, iterations of hardware came hard and fast.

• 3x cheap mini PC from Ebay? - check
• old and run-down switches and routers? - check
• Dell R730 server from Facebook Marketplace? - check
• 3x fairly expensive mini PC? - check (actually I am running those now!)

During the built-up (both personal and work) experience, I have also crystalized a few things:

1. I want my infrastructure to be fairly resilient and reproducable
   1. No “pets”, everything is “cattle” (this was actually demonstrated a few days ago, as one of my VM-s started misbehaving; took an entire 20 min to rebuild, mainly as the disk copy took like 12 min)
   2. The code should be in version control of course - but the secrets aren’t (not even encrypted; sorry SOPS!)
   3. GitOps would be ideal, but I am not shying away from some manual script execution
2. I want to be able to spin up new servers and have them join the cluster with minimal effort
3. I want to be able to spin up new services with minimal effort

The plan #

I have decided to go with Docker Swarm. I have used Kubernetes in the past, but I have found it to be a bit too complex for my needs. I have also found that Docker Swarm is a bit more lightweight and easier to manage. I have also found that Docker Swarm is a bit more resilient and easier to recover from failures.

I have also decided to go with Ansible for the provisioning and configuration of the servers. I have used Ansible in the past, and I have found it to be a bit more lightweight and easier to manage than other tools like Puppet or Chef. I have also found that Ansible is a bit more resilient and easier to recover from failures.

The stack #

I have also decided to go with a few other tools:

• Terraform/Opentofu for the provisioning of the infrastructure (ok I am partial for this tool, and got to admit, sometimes I almost think this is the “hammer” and everything is a “nail”)
  • Nowadays I even provision my network - I use Unifi BTW! - with this tool, not to mention my Cloudflare tunnels, DNS and you-name-it.
• Ansible is my main go-to when it comes to configuration management on Linux
  • I did some foray to use Puppet, but while I liked the pull mode, I prefer the straightforwardness of Ansible
  • In certain cases I complement it with some Bash scripts - temporarily mostly - but as of yet I did not come across any reasonable challenge that it could not solve
  • I now am at the point where I can just simply set IP up my Proxmox nodes in an event of a rebuild (perhaps I could also ommit this by using DHCP reservation, but currently the last rebuild was about 1.5 years ago…so not too often, I could say…)
• Now here is my recent favourite: Doppler. Doppler solves a forever-problem I had: how to manage secrets in a way that is secure, reproducible and easy to manage. Sure, .env files are easy to manage, but they are not secure. Vault is secure, but it is not easy to manage. Doppler is a bit of both. I have used it for a few months now, and I am very happy with it.
• Of course it would not be fair to not give credit to AI. Simple, tedious tasks, and/or tasks that took days of head banging / Github - and soul - searching, now reduced to minutes. Consistency is now almost guaranteed. Sure, the results are not always perfect, but Amp (replacing Gemini, after some weird and wonderful cost overrun…) has a good efficiency in getting me to the right direction.

Conclusion #

In the next few posts I will cover the build and my choices. But if you are only here for the repo - please feel free to check out. Be warned: it is a work in progress, and will probably always be evolving - but it is a good starting point for anyone who wants to build a similar setup.