~/homelab-kubecraft
A production-shaped cluster, run by one person who is still learning it.
I'm a sysadmin moving into DevOps, and this homelab is where the KubeCraft course meets real hardware. Three nodes, two CPU architectures, GitOps with Flux, secrets encrypted in the repo, and a public app behind a Cloudflare Tunnel. Everything below links to the commit or the post that proves it, including the parts that went wrong.
k3sv1.36.2nodes3archamd64 + arm64ram64 GBapps6open ports0fluxsynced
# Hardware
| Role | Node | Device | OS | Arch | RAM | Label |
|---|---|---|---|---|---|---|
| control plane | k3s-server | 2018 MacBook Pro (T2) | Ubuntu Server, headless | amd64 | 16 GB | light |
| worker | dell-node | Dell Latitude 7490 | Fedora | amd64 | 32 GB | heavy |
| worker | biggie-smalls | M2 MacBook Air | Fedora Asahi Remix | arm64 | 16 GB | compute |
Workloads pick nodes by label, not hostname, so I can swap hardware without touching manifests. Total spend on hardware: nothing. It was all in a drawer.
# Stack
| Layer | Tool | Job | Since | Status |
|---|---|---|---|---|
| distro | k3s | lightweight Kubernetes on old laptops | Aug 2026 | DONE |
| gitops | Flux | reconciles the cluster from the repo, prunes drift | Aug 2026 | DONE |
| packaging | Helm + Kustomize | HelmRelease for monitoring, base + staging overlays | Sep 2026 | DONE |
| secrets | SOPS + age | encrypted secrets in git, decrypted by Flux | Oct 2026 | DONE |
| ingress | Traefik | routes traffic to apps inside the cluster | Oct 2026 | DONE |
| exposure | Cloudflare Tunnel | outbound tunnel, no inbound ports on the router | Oct 2026 | DONE |
| metrics | kube-prometheus-stack | Prometheus, Grafana, alert rules | Sep 2026 | DONE |
| storage | local-path | k3s default; fast, free, pinned to one node | Sep 2026 | DONE |
| dashboards | Grafana (own) | node health, SLO, error budget | — | NOW |
| tls | cert-manager | automatic certificates (self-signed today) | — | NEXT |
| logs | Loki + Alloy | cluster-wide log search | — | NEXT |
| networking | Cilium | CNI + LoadBalancer IPAM | — | NEXT |
| database | CloudNativePG | Postgres with backups and tested restore | — | NEXT |
| iac | Terraform | cloud infrastructure as code | — | NEXT |
# How it deploys
GitHub · homelab-kubecraftsource of truth · SOPS-encrypted secrets
Flux pulls every 1 to 10 minutes
FluxKustomize + HelmRelease · decrypts with age
K3S CLUSTER · 3 NODES
control planeamd64 · light
workeramd64 · heavy
workerarm64 · compute
appslinkding · audiobookshelf · mealie · ntfy · homepage · littlelink
monitoringkube-prometheus-stack · Grafana
cloudflaredoutbound tunnel · no open ports
Internetyou, through Cloudflare
Nothing is applied by hand. Every box above is a folder in the repo.
# Things that broke
Linkding stuck Pending for seven hours
Impact: Old pod stuck Terminating, replacement stuck Pending, one node NotReady. Cause: A worker laptop went to sleep. The local-path volume is pinned to that node, so nothing else could take the pod. Fix: Woke the laptop. Kubernetes recovered on its own. Lesson: Disable sleep on every node, and read the scheduler's per-node reasons before guessing.
read the post → The PVC that never showed up
Impact: No storage created even though Flux had the commit. Cause: Switching to Recreate clashed with rollingUpdate fields Kubernetes had filled in by default. Flux's dry-run rejected the whole kustomization. Fix: Deleted the Deployment and let Flux rebuild it clean. Lesson: Check flux get kustomizations first. Set Recreate in the first commit, not later.
read the post → TLS handshake failure on a public hostname
Impact: ERR_SSL_VERSION_OR_CIPHER_MISMATCH in the browser. Cause: Cloudflare Universal SSL only covers one subdomain level. links.lab.example.com was one level too deep. Fix: Flattened the hostname to links.example.com. Lesson: A TLS error can come from the name, not the setup.
read the post → Flux bootstrap refused
Impact: Lost an afternoon before the cluster had GitOps at all. Cause: The fine-grained GitHub token had no Administration permission, which repo creation needs. Fix: Used a classic token with the full repo scope. Lesson: Check token scopes, then GITHUB_TOKEN, then the repo name, in that order.
read the post → Capstone deploy broken by one stray letter
Impact: Flux apply failed on the first line of a manifest. Cause: An extra vim i turned apiVersion into iapiVersion. Fix: Deleted the character, reconciled with --with-source. Lesson: Run head -1 on anything you pasted.
read the post → # Decisions
| ID | Decision | Why | Tradeoff | Status |
|---|---|---|---|---|
| ADR-001 | Flux GitOps instead of kubectl applypost → | The manifests lived in my shell history and in my head, which are two places I do not trust. With Flux the repo is the source of truth and the cluster gets pulled back to it if it drifts. | prune: true is also the flag that deletes your workload the moment you delete a file. | accepted |
| ADR-002 | Cloudflare Tunnel instead of opening router portspost → | I did not want to open a port and start advertising my front door. With a tunnel my router never has to accept an inbound connection at all. | Universal SSL only covers one subdomain level, so hostnames had to be flattened. Deeper names cost extra. | accepted |
| ADR-003 | Label nodes by what they can do, not by brandpost → | When dell-node eventually dies, every workload asking for workload=heavy follows the label to whatever replaces it. Also, honestly, dell-node felt ugly in a YAML file. | One more thing to keep consistent when a node is rebuilt. | accepted |
| ADR-004 | Recreate strategy for single-replica apps with one volumepost → | A rolling update deadlocks: the new pod cannot mount the volume while the old pod still holds it. On a single-replica homelab app with one RWO volume, that is exactly the right trade. | A few seconds of downtime on every deploy. | accepted |
| ADR-005 | Let the Helm chart generate the Ingress instead of writing itpost → | I do not edit a chart. I give it values. The chart owns the Ingress, so an upgrade does not fight my hand-written copy. | A wrong value name does not fail. It just gets ignored. | accepted |
| ADR-006 | SOPS + age for secrets in gitpost → | A plaintext password in values ends up in git history forever. A Secret created by hand was a problem I had deferred, not fixed. Encrypting with SOPS before it goes near git fixes it. | One more key to protect. Lose the age key and every secret in the repo is unreadable. | accepted |
| ADR-007 | k3s default local-path storage, for nowpost → | It is the default in k3s and it is fast and free. | The volume is pinned to one node, so three nodes give that app no resilience at all. Shared storage and volume backups are on the roadmap. | revisiting |
# Roadmap · 24/97 done · 20 goals · updated 2026-10-13
01 · Kubernetes cluster operations4/6
02 · Cluster provisioning & lifecycle0/4
- NEXT First look at Omni
- NEXT Cluster architecture, bootstrapping + secrets
- NEXT Provisioning with Ansible
- NEXT Kubernetes from dev containers
03 · Infrastructure as Code0/5
- NEXT Clouds are APIs
- NEXT Intro to Infrastructure as Code
- NEXT Terraform modules
- NEXT AKS with Terraform
- NEXT Ansible playbooks for node prep
04 · Cloud: AWS0/1
- NEXT Scoping the first AWS project
05 · Cloud: Azure0/6
- NEXT Setting up the repo + Azure
- NEXT Azure Kubernetes Service
- NEXT Best practices for AKS ops
- NEXT Fixing the new staging environment
- NEXT Production cluster + customer onboarding
- NEXT Reviewing the production cluster
06 · GitOps with Flux2/7
07 · CI/CD with GitHub Actions0/6
- NEXT Repository settings
- NEXT Linting with ruff (pre-commit + GHA)
- NEXT Tests, coverage, dev dependencies
- NEXT Frontend CI
- NEXT DRY in pipelines
- NEXT Automatic versioning + release creation
08 · Containers & image security0/6
- NEXT Building a Python image
- NEXT Containerizing the backend with uv
- NEXT Scanning images with Trivy
- NEXT Smaller base image, optimized layers
- NEXT Securing the image
- NEXT Building + pushing images, frontend included
09 · Local testing with k3d0/5
- NEXT Intro to k3d
- NEXT Manifests directory + local build script
- NEXT Minimal cluster exercise
- NEXT End-to-end testing with k3d + Python
- NEXT e2e testing in the pipeline
10 · Helm & packaging4/5
11 · Observability1/5
- DONE kube-prometheus-stack post →
- NOW My own Grafana dashboard
- NEXT Alert rules in Prometheus, delivered to ntfy
- NEXT Logs with Loki
- NEXT Production-grade monitoring
12 · SRE practice0/5
- NEXT Define an SLO for one app
- NEXT Error budget tracked in Grafana
- NEXT Runbook for each alert
- NEXT Postmortem template, blameless
- NEXT Capacity notes per node
13 · Bare-metal networking0/3
- NEXT Bootstrapping + installing Cilium
- NEXT Cilium LoadBalancer IPAM
- NEXT External DNS
14 · Ingress & TLS3/6
15 · Identity & access1/3
- DONE Securing Linkding properly
- NEXT Single sign-on in front of apps
- NEXT AKS hardening: securing cluster access
16 · Secrets management2/3
17 · Storage & databases1/4
- DONE Persistent volumes with local-path post →
- NEXT Shared storage across nodes
- NEXT Intro to databases in Kubernetes
- NEXT Postgres database for n8n
18 · Backups & disaster recovery0/4
- NEXT Volume backups
- NEXT Backups with the CNPG operator
- NEXT Restoring backups with CNPG
- NEXT Tested restore, timed and documented
19 · Dev workflow & scripting0/5
- NEXT mise.toml + setup script
- NEXT Conventional commits, Commitizen, pre-commit
- NEXT Python project + packaging with uv
- NEXT Backend + frontend code walkthroughs
- NEXT Go for tooling, later
# Build logs · 11 posts
devops KubeCraft Capstone: Can I Deploy Audiobookshelf With Nothing but Git? Oct 13, 2026 devops Exercise - Ingress with Traefik Oct 12, 2026 devops Letting Linkding Out of the House Oct 5, 2026 devops Why Didn't My PVC Show Up in the Cluster? Sep 15, 2026 devops Why Won't My LoadBalancer Load Balance Anything? Sep 3, 2026 devops Where Does the Grafana Password Actually Come From? Sep 2, 2026 thoughts NotReady: A Love Story Sep 1, 2026 devops Joining the Dell as a k3s worker node Aug 31, 2026 devops Setting Up GitOps with Flux on a Kubernetes Cluster Aug 31, 2026 devops Fedora Asahi Remix on Apple Silicon: A Runbook Aug 26, 2026 devops The M2 Air as a homelab node Aug 24, 2026
Craven Craven · DevOps / SRE
Open to DevOps and SRE roles.
Happy to walk through the cluster live and show you how a change gets from git to running.