Homeops

Don’t be fooled. Having a home server is really just hundreds of hours of badblocks.

I’ve been hard at work converting my architecture to support orchestrated deployments with multiple nodes. I haven’t quite achieved high availability, and I’m not likely to take things that far. However, the process has taught me a lot about Kubernetes.

Setup and usage are inspired by a homelab GitOps template and the k8s-at-home community. You can find similar setups with the k8s at home search. Historical revisions of my homelab setup had rootless Podman containers deployed with Ansible as systemd units. Prior to that, I used docker-compose to orchestrate containers on a single node.

Setup

Here are some of my goals:

• Flux GitOps with this repository (cluster directory)
• Ansible node provisioning and K3s setup (Ansible roles and playbooks)
• Terraform DNS records (Terraform)
• SOPS secrets stored in Git
• Renovate bot dependency updates
• WireGuard VPN pod gateway via paid service
• WireGuard VPN proxy hosted on VPS
• Cloudflared HTTP tunnel
• K8s gateway for local DNS resolution to the cluster and the NGINX ingress controller
• Both internal & external services with a service gateway
• OIDC authentication with LDAP
• Automatic Cloudflare DNS updates
• MetalLB bare-metal K8s network load balancing
• Calico CNI
• ZFS
• JBOD mergerfs union NFS with SnapRAID backup for low-touch media files
• Restic backups to remote and local buckets
• go-task shorthand for useful commands (Taskfile and taskfiles)

Some questions

• Why use ECC RAM?
  • Hacker News discussion
  • If you love your data, use ECC RAM
  • Error rates increase rapidly with rising altitude

Hardware

I finally upgraded my media server chassis to a Supermicro CSE-826. For almost seven years, I used a Node 804, a popular hobbyist case that fits eight 3.5-inch drives. I use old desktop hardware for this NAS and other nodes.

I used a widely known and inexpensive method to add SATA storage through a Host Bus Adapter (HBA). I purchased a Dell PERC H310 a long while back. Mine did come from overseas, but it turned out to be legit. This video shows how it can be flashed with LSI 9211-8i firmware in IT mode (see also 1 and 2).

Here are other recommended controllers.