The purpose of an observability system is to answer questions during failure. That sentence sounds obvious until you audit a real monitoring stack against it. The metrics server stores data on the storage cluster it monitors. The dashboards resolve their datasource through the DNS server whose outage you are trying to diagnose. Alerts leave the network through the same uplink whose loss they should report. Everything works beautifully until the one moment it exists for.
Observability that disappears during an incident is decoration.
The observability stack is infrastructure
Monitoring tools are usually deployed as applications: installed onto the platform, wired into convenient shared services, given the same dependencies as any other workload. But the observability stack has a different requirement profile from any other workload. It is the one system whose value is concentrated in the moments when other systems are failing. That inverts the usual engineering priorities: for telemetry, graceful degradation under partial infrastructure failure matters more than feature richness during normal operation.
The practical technique is to run a dependency audit on the monitoring path itself. For each stage, ask what has to be alive for it to work: collection, transport, storage, evaluation, alert routing, notification delivery, and the human-facing surfaces used during diagnosis. Then intersect that list with the failure scenarios the system is supposed to explain. Every intersection is a blind spot with a schedule: it will activate precisely during the incident it should have illuminated.
Common self-dependencies
- Metrics stored on the storage platform being monitored, so storage incidents erase their own evidence.
- Dashboards and alerting reachable only through the network segment whose health they report.
- Alert delivery that exits through a single uplink, a single DNS path, or a single notification vendor.
- Monitoring hosts joined to the same identity provider they are supposed to watch, so authentication outages lock out the diagnosis.
- The quiet failure mode: nothing alerts because the alerting pipeline itself is down, and silence reads as health.
Designing for the bad day
The goal is not a monitoring stack with no dependencies. That does not exist. The goal is a stack whose dependencies are known, minimized, and different from the things it watches. Independence is bought with deliberate placement and verified with tests that break things on purpose.
- Separate the failure domains: telemetry storage and evaluation should not share the platform, power, and network path of the systems they observe.
- Route alert delivery over more than one path, and prove the paths are independent rather than assuming it.
- Watch the watcher: a dead-man signal from the alerting pipeline to an observer outside the estate converts silent monitoring death into a loud, unambiguous event.
- Preserve evidence by design: an incident should leave enough telemetry behind to reconstruct it even when part of the stack was down.
- Rehearse: the only trustworthy claim about incident-time behavior is one that has been observed during a deliberately induced failure.
None of this is exotic. It is ordinary systems engineering applied to a subsystem that usually escapes it because it is labeled a tool instead of infrastructure. The test of an observability platform is simple to state and demanding to pass: during the worst hour of the year, can it still answer what happened, when, and why?