scryops obs ARTICLES
Article 5 min read Alerting SLOs On-Call Observability

Alert Fatigue Is an Observability Problem

Every alert that fires is doing its job. That is the problem. The model is wrong, not the thresholds.
Night time pages are heart attacks, not diabetes.
Charity Majors
Charity Majors On Call Shouldn’t Suck · charity.wtf

Alert fatigue is the expected result of traditional monitoring when it’s working correctly.

Every alert fired as configured. Threshold crossed, notification sent, system did its job. The alerts are not broken. The model is. You built a machine that watches numbers and shouts when one crosses a line. In a complex system, that machine shouts all the time. The on-call team tunes it out. The alerts that matter get lost in the noise.

The fix is not fewer alerts — it’s alerting on the right signal.

The Machine That Always Cries Wolf

A static threshold assumes anything above X is wrong. In real systems, that is almost never true. Daily traffic, weekend shifts, deployments, or seasonal spikes all break that rule. What matters at 2pm on Tuesday does not matter at 3am on Sunday.

Static threshold versus a real 24-hour traffic patternA service metric over 24 hours follows a diurnal curve: low overnight, a large midday and afternoon peak. A single flat alert threshold is drawn across it. During the normal afternoon peak the metric rises above the line and fires false alarms. At 3am a real incident spikes the metric well above its overnight baseline, but it stays just under the daytime-tuned threshold, so no alert fires and the incident is missed.STATIC THRESHOLD vs. REALITY// one fixed line, 24 hours of real trafficservice latency / error rateALERT THRESHOLD✗ FALSE ALARMSnormal afternoon peaktrips a fixed line✗ MISSEDreal 3am incident —never crosses the line,so no page fires00:0006:0012:0018:0024:0003:00observedexpected patternstatic threshold✗false alarm / missed
Fig. — A single static line can’t be right at both 3am and 2pm. Tune it for the daytime peak and it goes blind to a real overnight incident; tune it for the quiet hours and it screams through every normal afternoon.

It’s a feedback loop. The noisier the alerts, the more you ignore them. The more you ignore, the more real issues slip by. So you add more alerts to catch what you missed, and the noise just gets louder.

The alert-fatigue feedback loopA self-reinforcing loop. High alert volume leads to desensitization as the on-call team tunes alerts out, which lets real issues slip by, which prompts the team to add still more alerts, which raises the volume again — and the cycle repeats, louder each round.THE ALERT-FATIGUE LOOP// noisier alerts → more ignored → more missed → more addedHigh alert volumeDesensitizationMissed issuesMore alerts addedtuned outslip byadd more↻ and the cycle repeats — louder every round
Fig. — Every fix points inward: more alerts to catch what you missed, which is exactly what made you miss things. The loop has to be broken, not tuned.

To fix alert fatigue, you need to rethink what you measure—not just tweak the numbers.

A Better Question

If static thresholds ask “did this number cross a line?”, SLO-based alerting asks “is the user experience degrading, and how fast?” That’s a fundamentally different question — and it’s the one that actually matters.

Burn rate alerts fire when your error budget drains faster than the baseline depletion rate allows. They do not care about 2pm or 3am. They do not care if p99 crossed some random threshold. They ask: at this rate, when do you run out of budget? And they give you a head start.

If you’re burning through your budget 14 times faster than the rate that would exhaust it over the full 30-day window, someone gets paged right away. If it’s 6 times faster — sustained across both a one-hour and a six-hour window — it’s urgent but not a fire drill. A burn rate below 2× means you have roughly two weeks of budget remaining at that rate; that earns a ticket for investigation during business hours, not a 3am page. The response matches how quickly you’re heading for trouble, not just whether a number jumped.

Burn-rate triageBurn-rate triage as two questions. First: is the SLO error budget burning? If not, monitor and log it. If yes, ask how fast it is burning, which sets a proportional response. A slow burn (under 2 times the sustainable rate, weeks of budget left) becomes a ticket to investigate this week. A moderate burn (around 6 times, on the one-hour and six-hour windows) notifies the team and escalates cross-functionally if on-call cannot resolve it. A fast burn (14 times on the one-hour window) pages on-call and is declared a major incident if users are severely hit.BURN-RATE TRIAGE// two questions decide the responseIncident detectedSLO budget burning?no → monitor & logyesHow fast is itburning?SLOW · under 2× · weeks of budget leftTicket / Slack — investigate this weekMODERATE · ~6× · 1h + 6h windowsNotify the team — escalate if on-call can’t resolveFAST · 14× · 1h windowPage on-call — major incident if users hit hard
Fig. — Two questions — is the budget burning, and how fast — and the response picks itself.

Teams who move past static thresholds use multi-window, multi-burn-rate alerts. These catch the slow burns and creeping trends that old alerts miss.

The Alert Audit

Even if you’ve switched to SLO-based alerts, you probably still have threshold alerts that were never retired. Start by reviewing them.

For each active alert, ask one question: when this fires, what does the on-call person actually do? If the honest answer is check a few things and close it, the alert is not actionable. Fix it or remove it. An alert that does not drive action is just noise.

As you clean up, track three things: how many alerts fire, how many lead to real action, and how fast people respond. You want fewer alerts, more action, and faster response. If all three improve, the cleanup is working. If only the volume drops, you might be missing something important.

ALERT FATIGUE  ·  BY THE NUMBERS  ·  INDUSTRY DATA 2024–2026
~2,000
alerts per week at a typical enterprise
3%
of them genuinely need immediate action
67%
of engineers ignore or dismiss alerts without investigating
44%
had an outage caused by a suppressed or ignored alert
ACTIONABLE-ALERT RATE — WHERE TEAMS ACTUALLY LAND
Most teams today
<30%
Healthy range
30–50%
Trustworthy — this guide’s bar
70%+
0%25%50%75%100%
SOURCES: CATCHPOINT SRE REPORT · INCIDENT.IO 2025 · NEUBIRD STATE OF PRODUCTION RELIABILITY 2026
FIGURES ARE INDUSTRY-SURVEY MEDIANS — DIRECTIONAL, NOT ABSOLUTE
Fig. — The gap between alert volume and alert value is the whole problem. When 97% of what fires is ignorable, people learn to ignore all of it — including the 3% that matters.

Route Alerts by Severity

Not every alert needs the same channel. P0 issues that burn your error budget fast go to PagerDuty and wake someone up. P1 issues go to an incident channel with a clear 30-minute response goal. Everything else goes somewhere passive: a Slack channel, a ticket queue, or a weekly review. The person asleep at 3am should only be woken for alerts where delayed response directly accelerates budget burn.

Suppress alerts during planned maintenance windows. An alert that fires during an intentional deployment is not signal — it’s confirmation that you deployed.

ALERT ROUTING  ·  MATCH URGENCY TO INTERRUPTION
SEVERITYTRIGGERROUTEWHO IT WAKES
P0FAST BURN
14× burn over 1 hourerror budget vanishing now
PagerDuty — wake on-callimmediate incident response
WAKE
SOMEONE
P1MODERATE
6× burn over 6 hoursdegrading, not yet critical
Incident channel — 30-min responseteam picks it up
BUSINESS
HOURS
P2SLOW / INFO
2× burn over 3 daysor purely informational
Passive Slack / ticket queuereviewed weekly
NO
INTERRUPTION
The person asleep at 3am should only ever hear from the top lane. Everything else can wait for morning — and if it can’t wait but also can’t wake someone, it’s in the wrong lane.
Fig. — Not every fire alarm needs the fire department. Routing by burn rate keeps the 3am page rare enough that people still trust it.

What Trustworthy Looks Like

The goal is not zero alerts. A system that never alerts lacks visibility into failures.

The goal is alerts you trust. A small set that fires rarely, means something every time, and tells the person what to look at. That is what makes on-call tenable and alerts worth trusting.

Maintain and protect your trusted alert set. Require that new alerts prove their value before adding them.

How to get started. Audit alerts over 30 days. For each alert, mark it as actionable if it led to a real change or investigation, or as noise if it was only acknowledged and closed. If fewer than 70 percent are actionable — and most noisy alert sets aren’t close to that — start with the loudest offenders.
BawkwardNOOB · your noisiest alert“BWOCK!! WOLF!! ...ok, the wind. WOLF!! ...the nightly cron. WOLF!! ...a deploy, probably fine. One of these is the real outage, I swear it. You will have to check every single one to find out which. That is my whole gift. I am level 1. I will never stop.”
The Cucco is every untuned alert given a body: never lying, never useful, impossible to ignore.