Mission Control | Chadron Bryant

Five machines. A growing pile of AI agents.A couple dozen tool integrations.No single place to ask: is any of this actually okay right now?

02 / PROBLEM

The failure mode

I had five machines, a growing pile of AI agents, a couple dozen tool integrations, and no single place to ask “is any of this actually okay right now.” The honest answer, most days, was “check four different terminals and hope.” A platform team builds an internal ops console as a matter of course; I hadn’t, because it felt like overhead for an audience of one. It isn’t. An audience of one is exactly who needs it most, because there’s no second person to notice what you missed.

03 / CONSTRAINTS

Operating envelope

It had to aggregate status across genuinely heterogeneous things — compute nodes, model endpoints, stored credentials, and tool inventories — none of which expose a common health format.
It had to be a control surface, which means it couldn’t be casually public: a dashboard that shows credential health and mesh topology is itself sensitive, so it needed real authentication, not a bookmarked local URL.
It had to stay cheap to run continuously.
And it had to earn its own existence — a status page nobody opens is worse than no status page, because it creates false confidence that “someone would notice.”

04 / THE SYSTEM

How it holds together

MISSION CONTROL / SELF-TEST SELECT A DIAL FOR THE FINDING

Read the system narrative

A small backend service polls every subsystem on a schedule — compute nodes, model endpoints, the tool inventory across every integration, and a dedicated credential-health check — and a frontend renders the whole thing as one authenticated dashboard. The credential-health view is the part I actually care about: rather than trusting that a secret exists because it was set once, it periodically re-checks that each stored credential still authenticates against the thing it’s supposed to unlock, and surfaces drift instead of assuming permanence. A separate view tracks storage and backup posture (which backend, which retention tier, when it last actually completed) so “are backups running” stops being a question I have to go dig for. None of it is decorative — every tile on the dashboard maps to something that has actually broken silently at least once.

05 / WHAT BROKE

INC-20260621

Incident replay

Symptom: the credential-health tab, in its first real week of use, reported several credentials as fine when I had reason to believe otherwise for at least one of them.
Root cause: the health check resolved which environment’s secrets it was auditing based on an assumed default working context rather than the actual active one — a classic “it works on my machine” bug, except the machine in question was the monitoring tool itself.
Fix: fixed the resolution logic so the credential-health check always determines its target context explicitly rather than assuming a default, and added the same check to itself that it applies to everything else — a monitor that can silently monitor the wrong thing is worse than no monitor.

SYMPTOM: the credential-health tab, in its first real week of use, reported several credentials as fine when I had reason to believe otherwise for at least one of them. ROOT CAUSE: the health check resolved which environment’s secrets it was auditing based on an assumed default working context rather than the actual active one — a classic “it works on my machine” bug, except the machine in question was the monitoring tool itself. FIX: fixed the resolution logic so the credential-health check always determines its target context explicitly rather than assuming a default, and added the same check to itself that it applies to everything else — a monitor that can silently monitor the wrong thing is worse than no monitor. The bug fix, once shipped, immediately did the job the dashboard was built to do: the very first correct pass of the credentials view turned up three real, previously invisible problems — a token that had silently drifted out of a healthy state, a service account nobody had checked against its actual expiry, and a stored secret that no longer matched what it was supposed to unlock. None of these were new problems. They just hadn’t had anywhere to show up before.

06 / RETROSPECTIVE

What I’d do differently

I’d write a self-check into every monitor before shipping it, on the assumption that any tool whose entire job is “tell me the truth about X” is the one place where being subtly wrong is most expensive. I also underestimated how much of the value would come from the first correct read rather than from ongoing use — if I’d known that, I’d have built the credentials view first instead of last.

07 / SPEC PLATE

Build record

Status: live LIVE
Stack: lightweight backend service + web frontend, cross-mesh status collectors, credential-health probes.
Scars: the auditor that needed auditing.
Last incident: 2026-06-21