From signal to incident
A failed check on its own is just a signal. A signal can be transient, can disappear, can be forgotten. An incident, on the other hand, is a trackable record: it carries a start time, affected resource, monitoring type, severity, and review and resolution status.
Root incident and affected incident
When a switch goes down, every dependent check fails at the same time. Looking at this cluster without separating it out pushes the operator to start from a random alarm.
The root incident marks the most likely starting point of the cluster; the other records are grouped as affected incidents. This is a priority indicator, not a declaration of definitive root cause.
Reducing alert fatigue
Notification noise is a configuration problem, not a tool problem. The following practices noticeably reduce noise.
- Separate severity levels according to genuinely different business impacts.
- Use root/affected grouping; do not send a separate notification for every record.
- Split notification channels by responsible team.
- Review the threshold and retry settings of checks that produce false positives.
- Record operator feedback on resolved incidents.