alerts in yaml

weird apis

alerts in yaml

For unfortunate reasons, I've had to come up with an interface for configuring alerts through yaml (that gets templated through helm, then terraform, then datadog messaging dsl).

This was the structure I ended up with:

• all object based to allow easy threshold overrides in different files
• a single source of truth for id (the alert title)
• canned alerts for common stuff

 1alert-config:
 2  metric_based_alerts:
 3    "alert 1":
 4      query:
 5        predefined: "type-1"
 6        threshold: 55
 7      message: |
 8        oops stuff broke        
 9
10    "alert 2":
11      query:
12        raw: "sum(last_5m):sum:my.metric{instance:INSTACE}"
13        threshold: 22
14      message: |
15        another thing broke        
16
17  alert_based_slos:
18    "slo 1":
19      metric_alerts:
20        - "alert 1"
21        - "alert 2"
22      target: 99.9