Quiet Hours
A rules engine that routes production alerts to whoever’s on call and holds the rest until morning.
Context
Every production alert paged the whole on-call rotation, including warnings nobody could act on at night. People started muting the channel, which is the one thing an alerting system cannot survive.
Idea
Routing rules became data owned by each team, with a quiet window per rule. Anything below critical inside that window is held and delivered in the morning. Before a team switches a rule on, a replay mode runs it against the last thirty days of real alerts.
| Row | Time | Severity | Rule | Decision |
|---|---|---|---|---|
| 0000 | 23:12 | warning | payments | hold until 08:00 |
| 0010 | 23:40 | critical | payments | page owner |
| 0020 | 02:05 | info | search | hold until 08:00 |
| 0030 | 06:48 | warning | search | hold until 08:00 |
| 0040 | 08:15 | warning | search | page owner |
Details
Rules as data
Teams edit their own rules in a reviewed file. The engine loads them at startup and refuses to start on an invalid rule instead of guessing.
Replay before rollout
The same routing function runs over historical alerts and prints who would have been woken up, and when. Most rule changes were settled by reading that output.
Critical always pages
No quiet window can hold a critical alert. That rule lives in code, not configuration, and it has its own test.
func Route(a Alert, rules []Rule, now time.Time) Decision { for _, r := range rules { if !r.Matches(a) { continue } if r.Quiet.Contains(now) && a.Severity < SeverityCritical { return Hold(r.Owner, r.Quiet.End(now)) } return Page(r.Owner) } return Page(DefaultOwner)}Outcome
Rolled out one team at a time. Because every team saw its own replay first, nobody switched a rule on blind.