prometheus and alertmanager end 2 end alerting delay

how long does it take to get an alert?

alertmanager

alertmanager the alerting component of prometheus. So how long can it take between an event happening and you getting notified about it? Since they're both written by the same team, they work more or less the same way: timer based event loops.

timer event loop

So what triggers things to happen?

• prometheus scrape is ticker based
• prometheus rules is ticker based
• alertmanager dispatch/group wait is reset timer based

delay

In the most unfortunate case:

• wait up to $scrape_interval to get the initial data
• wait up to $evaluation_interval to check if alert is firing
• wait up to $for (round up to nearest $evaluation_interval) to trigger the alert
• wait up to $group_wait to send initial notification
  • updates are sent at $group_wait ticks

Now say you have another on call system layered on top. It too is likely to be evaluation_interval / delay based, so your total delay is even longer....