Observability projects tend to start with dashboards and end with alert fatigue. The more useful order is to decide what would constitute a bad day for a user, alert on that, and add detail as incidents teach you what you were missing.
Start from the user, not the machine
Processor utilisation is a symptom; a slow checkout is a problem. Define a small number of service-level indicators that describe the experience – success rate and latency for the critical path – and alert when those degrade. Resource metrics then become the tools you use to diagnose, rather than the things that wake you.
| Layer | Signals |
|---|---|
| User journey | Success rate and latency of the critical path. |
| Service | Request rate, error rate, duration percentiles. |
| Dependencies | Error rate and latency of each downstream call. |
| Saturation | CPU, memory, disk, connections, queue depth. |
| Cost | Daily spend, as an anomaly detector. |
Alerts worth having
- Every alert must be actionable: if nobody would do anything, it is not an alert.
- Every alert needs an owner and a documented first step.
- Alert on symptoms users feel, not on every threshold crossing.
- Use windows and rates, so one transient error does not page anyone.
- Route by urgency: some things page, most things wait for the morning.
Logs, metrics and traces do different jobs
- Metrics tell you something is wrong and how widespread it is. Cheap to keep.
- Logs tell you what happened in a specific case. Expensive at volume; set retention deliberately.
- Traces tell you where the time went across services. Sample rather than capturing everything.
Turn it on before you need it
Enable audit and application logging when the account is created, not after the first incident. Retrospective observability does not exist; you can only ever investigate what you were already recording.
Summary
Measure the user experience, alert only on things worth acting on, keep the alert channel trustworthy, and enable logging from day one. Detail can be added; history cannot be recovered.
