A monitoring setup that pages someone eleven times a week has already failed, regardless of how complete its coverage is. People stop reading alerts that are usually nothing, and the twelfth one — the real one — gets the same treatment.
Alert on symptoms, not causes
High CPU is not an incident. Checkout requests failing is an incident. If a cause does not produce a symptom your users can feel, it belongs on a dashboard where someone will look at it during working hours.
This collapses an alert list dramatically, usually to a handful of rules per service covering error rate, latency and the specific business events that have to keep happening.
Tie thresholds to an objective
Pick a service level objective — say, 99.5% of checkout requests succeed over thirty days — and alert on the rate at which you are consuming the error budget rather than on instantaneous thresholds.
A brief spike that consumes a tiny fraction of the budget does not need a person at three in the morning. A sustained elevation that will exhaust it by Thursday does.
Every alert needs a runbook
If the person receiving the page cannot act on it, it is not an alert, it is a notification. The rule we apply is simple: an alert without a linked runbook describing the first three things to check gets deleted or downgraded.
Writing that runbook often reveals that nobody knows what to do about the condition, which is useful information on its own.
Optimise for time to restore
Chasing an uptime figure encourages heroics. Optimising for how quickly service is restored encourages the things that actually help: fast rollbacks, feature flags, and deployments small enough to reason about at speed.
Working on something this touches on? Tell us about it — we are happy to talk it through whether or not it turns into a project.