scryops obs ARTICLES
Article 5 min read Alerting SLOs On-Call Observability

Alert Fatigue Is an Observability Problem

Every alert that fires is doing its job. That is the problem. The model is wrong, not the thresholds.
Night time pages are heart attacks, not diabetes.
Charity Majors
Charity Majors On Call Shouldn’t Suck · charity.wtf

Alert fatigue is the expected result of traditional monitoring when it’s working correctly.

Every alert fired as configured. Threshold crossed, notification sent, system did its job. The alerts are not broken. The model is. You built a machine that watches numbers and shouts when one crosses a line. In a complex system, that machine shouts all the time. The on-call team tunes it out. The alerts that matter get lost in the noise.

The fix is not fewer alerts — it’s alerting on the right signal.

The Machine That Always Cries Wolf

A static threshold assumes anything above X is wrong. In real systems, that is almost never true. Daily traffic, weekend shifts, deployments, or seasonal spikes all break that rule. What matters at 2pm on Tuesday does not matter at 3am on Sunday.

Static threshold versus a real 24-hour traffic patternA service metric over 24 hours follows a diurnal curve: low overnight, a large midday and afternoon peak. A single dashed threshold rule is drawn across it. During the normal afternoon peak the metric rises above the line, marked with a hollow square where it crosses — a false alarm. At 3am a real incident spikes the metric well above its overnight baseline, marked with an X, but it stays just under the daytime-tuned threshold, so no alert fires and the incident is missed.STATIC THRESHOLD vs. REALITY// one fixed line, 24 hours of real trafficservice latency / error rateALERT THRESHOLD□ FALSE ALARMnormal afternoon peaktrips a fixed line✗ MISSEDreal 3am incident —never crosses the line,so no page fires00:0006:0012:0018:0024:0003:00
Fig. — A single static line can’t be right at both 3am and 2pm. Tune it for the daytime peak and it goes blind to a real overnight incident; tune it for the quiet hours and it screams through every normal afternoon.

It’s a feedback loop. The noisier the alerts, the more you ignore them. The more you ignore, the more real issues slip by. So you add more alerts to catch what you missed, and the noise just gets louder.

The alert-fatigue feedback loopA self-reinforcing loop of four numbered steps. Step one, high alert volume, leads to step two, desensitization, as the on-call team tunes alerts out. That lets step three, real issues, slip by, which prompts step four, adding still more alerts. Step four is marked with a heavier border and a tinted fill because it is the intervention point — the place to break the loop. A dashed return edge carries step four back to step one, and the cycle repeats, louder each round. The sequence is carried by numbers and the intervention point by border weight, not by colour.THE ALERT-FATIGUE LOOP// noisier alerts → more ignored → more missed → more added1High alert volume2Desensitization3Missed issues4More alerts addedtuned outslip byadd more↻ back to step 1 — and the cycle repeats, louder every round
Fig. — Every fix points inward: more alerts to catch what you missed, which is exactly what made you miss things. Step 4 carries the heavy border because that is where the loop breaks — it has to be broken, not tuned.

To fix alert fatigue, you need to rethink what you measure—not just tweak the numbers.

A Better Question

If static thresholds ask “did this number cross a line?”, SLO-based alerting asks “is the user experience degrading, and how fast?” That’s a fundamentally different question — and it’s the one that actually matters.

Burn rate alerts fire when your error budget drains faster than the baseline depletion rate allows. They do not care about 2pm or 3am. They do not care if p99 crossed some random threshold. They ask: at this rate, when do you run out of budget? And they give you a head start.

If you’re burning through your budget 14 times faster than the rate that would exhaust it over the full 30-day window, someone gets paged right away. If it’s 6 times faster — sustained across both a one-hour and a six-hour window — it’s urgent but not a fire drill. A burn rate below 2× means you have roughly two weeks of budget remaining at that rate; that earns a ticket for investigation during business hours, not a 3am page. The response matches how quickly you’re heading for trouble, not just whether a number jumped.

Fig. — Two questions — is the budget burning, and how fast — and the response picks itself.

Teams who move past static thresholds use multi-window, multi-burn-rate alerts. These catch the slow burns and creeping trends that old alerts miss.

The Alert Audit

Even if you’ve switched to SLO-based alerts, you probably still have threshold alerts that were never retired. Start by reviewing them.

For each active alert, ask one question: when this fires, what does the on-call person actually do? If the honest answer is check a few things and close it, the alert is not actionable. Fix it or remove it. An alert that does not drive action is just noise.

As you clean up, track three things: how many alerts fire, how many lead to real action, and how fast people respond. You want fewer alerts, more action, and faster response. If all three improve, the cleanup is working. If only the volume drops, you might be missing something important.

ALERT FATIGUE  ·  BY THE NUMBERS  ·  INDUSTRY DATA 2024–2026
~2,000
alerts per week at a typical enterprise
3%
of them genuinely need immediate action
67%
of engineers ignore or dismiss alerts without investigating
44%
had an outage caused by a suppressed or ignored alert
ACTIONABLE-ALERT RATE — WHERE TEAMS ACTUALLY LAND
Most teams today
<30%
Healthy range
30–50%
Trustworthy — this guide’s bar
70%+
0%25%50%75%100%
SOURCES: CATCHPOINT SRE REPORT · INCIDENT.IO 2025 · NEUBIRD STATE OF PRODUCTION RELIABILITY 2026
FIGURES ARE INDUSTRY-SURVEY MEDIANS — DIRECTIONAL, NOT ABSOLUTE
Fig. — The gap between alert volume and alert value is the whole problem. When 97% of what fires is ignorable, people learn to ignore all of it — including the 3% that matters.

Route Alerts by Severity

Not every alert needs the same channel. P0 issues that burn your error budget fast go to PagerDuty and wake someone up. P1 issues go to an incident channel with a clear 30-minute response goal. Everything else goes somewhere passive: a Slack channel, a ticket queue, or a weekly review. The person asleep at 3am should only be woken for alerts where delayed response directly accelerates budget burn.

Suppress alerts during planned maintenance windows. An alert that fires during an intentional deployment is not signal — it’s confirmation that you deployed.

ALERT ROUTING  ·  MATCH URGENCY TO INTERRUPTION
SEVERITYTRIGGERROUTEWHO IT WAKES
P0 FAST BURN
14× burn over 1 hourerror budget vanishing now
PagerDuty — wake on-callimmediate incident response
WAKE
SOMEONE
P1 MODERATE
6× burn over 6 hoursdegrading, not yet critical
Incident channel — 30-min responseteam picks it up
BUSINESS
HOURS
P2 SLOW / INFO
2× burn over 3 daysor purely informational
Passive Slack / ticket queuereviewed weekly
NO
INTERRUPTION
The person asleep at 3am should only ever hear from the top lane. Everything else can wait for morning — and if it can’t wait but also can’t wake someone, it’s in the wrong lane.
Fig. — Not every fire alarm needs the fire department. Routing by burn rate keeps the 3am page rare enough that people still trust it.

What Trustworthy Looks Like

The goal is not zero alerts. A system that never alerts lacks visibility into failures.

The goal is alerts you trust. A small set that fires rarely, means something every time, and tells the person what to look at. That is what makes on-call tenable and alerts worth trusting.

Maintain and protect your trusted alert set. Require that new alerts prove their value before adding them.

The trustworthy-alert-set test. Over a 30-day window, an actionable rate below 70 percent is the sign of a set that still cries wolf — and most untuned alert sets don’t come close to that number. The gap between “acknowledged and closed” and “led to a real change or investigation” is exactly the gap this piece is about.
Bawkward NOOB · your noisiest alert

BWOCK!! WOLF!! ...ok, the wind. WOLF!! ...the nightly cron. WOLF!! ...a deploy, probably fine. One of these is the real outage, I swear it. You will have to check every single one to find out which. That is my whole gift. I am level 1. I will never stop.

The Cucco is every untuned alert given a body: never lying, never useful, impossible to ignore.

This is the same reactive-vs-proactive argument that runs through the whole publication — see Observability 1.0 meant forensics. Observability 2.0 means prevention. for the broader case.