The Homelab Mesh

Five machines and a paranoia budget.

Real failover. Real verified backups. Real monitoring.At home, without enterprise cloud pricesor pretending one gaming PC is infrastructure.

02 / PROBLEM

The failure mode

I wanted the resilience and observability I build for a living — real failover, real verified backups, real monitoring — at home, without paying enterprise cloud prices or pretending a single gaming PC that also runs a media server counts as infrastructure.

03 / CONSTRAINTS

Operating envelope

Consumer and prosumer hardware only, no rack-mount budget.
No public IP exposure of anything internal — remote access without opening a single port to the internet.
A genuinely mixed fleet: a Linux box as primary compute, a repurposed Windows machine for parallel workloads, a Mac Mini as a fallback when the primary is down.
A time budget of nights and weekends, solo.
And the real test: it has to survive me ignoring it for two weeks straight.

04 / THE SYSTEM

How it holds together

Read the system narrative

Five machines, five defined roles: a primary compute node (the brain), a secondary Windows node for parallel and overflow work, a Mac Mini fallback, a lightweight watchdog node that does nothing but watch the others, and network-attached storage as the backup tier. A Tailscale mesh overlay replaces port-forwarding entirely — every machine is reachable from anywhere without a single exposed port on the home router. Cloudflare tunnels and Workers handle the handful of things that genuinely need to be public (a status page, a couple of small tools), so “public-facing” and “port-forwarded” are never the same sentence in this design. Monitoring lives on a machine that isn’t the thing being monitored, on purpose — a watchdog that dies with the host it’s watching isn’t a watchdog. Backups are tiered by retention and, critically, get scheduled restore rehearsals: a backup nobody has restored is a rumor, not a backup.

05 / WHAT BROKE

INC-20260709

Incident replay

Symptom: a service got “fixed” by restarting its container, and the fix appeared to work — until it didn’t, an hour later.
Root cause: the actual code lived baked into the image at build time, not in the mounted config volume the restart refreshed — I chased a phantom bug for over an hour before realizing I’d never rebuilt anything, only restarted a container running the same stale code the whole time.
Fix: “restart vs. rebuild” is now a house rule with a name.

SYMPTOM: a service got “fixed” by restarting its container, and the fix appeared to work — until it didn’t, an hour later. ROOT CAUSE: the actual code lived baked into the image at build time, not in the mounted config volume the restart refreshed — I chased a phantom bug for over an hour before realizing I’d never rebuilt anything, only restarted a container running the same stale code the whole time. FIX: “restart vs. rebuild” is now a house rule with a name. Separately: an unrelated system update quietly flipped a background helper service from automatic to manual, which meant a nightly backup job had been silently not running for weeks — the schedule still fired, the log still said “done,” but the thing it depended on to actually execute had gone dark, and nothing about the green log line said so.

06 / RETROSPECTIVE

What I’d do differently

“Restart vs. rebuild” should have been a rule before it cost me an hour. More importantly: any unattended job now needs an out-of-band check that it actually did the thing, not just that it exited zero — a scheduler’s own self-report is exactly the kind of single signal that lied to me once already, elsewhere, and I should have generalized the lesson faster.

07 / SPEC PLATE

Build record

Status: live LIVE
Stack: 5-node mesh (Linux / Windows / macOS), Tailscale overlay, Docker fleets, Cloudflare tunnels, Grafana + uptime watchdog.
Scars: restart vs. rebuild.
Last incident: 2026-07-09