Routing Budget Alerts to Slack and PagerDuty

The specific bottleneck this page solves is the single-channel alert firehose: once a webhook handler starts emitting a verdict for every budget threshold crossed by every service in every account, posting all of it to one Slack channel or paging everyone on every trigger produces exactly the outcome you were trying to avoid — alerts get muted, on-call burns out, and the one alert that actually mattered (a runaway autoscaling group, not a routine month-end true-up) gets lost in the scroll. A budget overage on a low-spend dev account is informational; the same overage on a production billing account tagged to an SLA-bound team is pageable. Routing has to be severity-aware, owner-aware, deduplicated so a flapping threshold doesn’t open ten PagerDuty incidents, and rate-limited so a burst of simultaneous triggers doesn’t itself get throttled by Slack or PagerDuty and silently drop alerts. This page covers the routing layer that sits downstream of alert generation inside Budget Alert Automation with Webhooks — the component that takes a normalized alert and decides where it goes, how urgently, and whether it goes at all.

# Root Cause & Failure Modes

Naive routing designs break in three predictable ways once alert volume passes a handful of accounts:

• One Slack channel, no severity split. Every budget crossing — a 50 USD dev-sandbox nudge and a 50,000 USD production overrun — lands in the same feed with the same visual weight. Humans stop reading the channel within a week, which means the channel is not a control, it is decoration.
• Paging on every trigger, no dedup. A budget hovering right at its threshold can flap across the boundary for hours as usage accrues throughout the day. Without a stable dedup key, each crossing opens a new PagerDuty incident instead of updating the existing one, and the on-call engineer gets paged repeatedly for what is functionally one ongoing event.
• No suppression window and no backoff. A cost-anomaly detector that re-scores every account on a tight schedule can emit a burst of alerts in the same few seconds if a shared dependency (a mispriced SKU, a billing export delay) trips multiple thresholds at once. Firing all of them synchronously runs straight into provider rate limits: Slack incoming webhooks degrade sharply past roughly one message per second sustained, returning 429 with a Retry-After header, and PagerDuty’s Events API v2 enforces a documented ceiling on the order of 61 events per minute per routing key before it also starts returning 429. A router with no backoff just drops those alerts on the floor; a router with no suppression window re-sends them redundantly the moment it retries.

The fix has three parts that map directly onto the three failures: a routing table keyed on (severity, owner_team) so destinations are explicit rather than one-size-fits-all; a stable dedup_key derived from the alert’s identity (not its timestamp) so repeated firings collapse into one PagerDuty incident and one Slack thread; and a suppression window combined with exponential backoff so bursts are smoothed rather than dropped or duplicated.

# Production Pipeline Architecture

The router is a thin, stateless-except-for-suppression layer between the alert source and the two destination APIs. It runs as four phases, and it assumes the alert has already been normalized and deduplicated at the source — that normalization work is the subject of the sibling page Building Budget Alert Webhook Handlers, which covers verifying inbound webhook signatures and shaping the raw provider payload into the BudgetAlert record this router consumes.

1. Classify. Map the alert’s severity field (info / warning / critical) and owner_team tag to a routing-table entry. The table is data, not code — a dict of owner_team → RouteTarget(slack_webhook, pagerduty_routing_key) — so onboarding a new team is a config change, not a deploy.
2. Derive the dedup key. Hash the stable identity fields — account ID, service, budget ID — never the timestamp or the current spend figure, both of which change between successive firings of the same underlying event.
3. Check suppression. Look up the dedup key in a short-lived cache. If an alert with that key was dispatched within the suppression window, skip the Slack post (avoid re-notifying a human about something already visible) but still allow the PagerDuty call through, because PagerDuty’s own dedup logic uses the same key to update the existing incident rather than opening a new one — the suppression window and PagerDuty’s server-side grouping are complementary, not redundant.
4. Dispatch with backoff. POST to each destination independently, wrapped in a retry with exponential backoff and jitter so a 429 or transient 5xx from one provider never blocks or drops the alert to the other.

The severity mapping this router keys off of is produced by the scoring stage documented in Building Cost Anomaly Detection Models — that page covers how a raw deviation becomes a typed severity label before it ever reaches this router.

# Step-by-Step Python Implementation

The module below is a self-contained router: a BudgetAlert record, a routing table, a suppression cache, and two dispatch methods with independent retry policies. It is idempotent by design — running it twice on the same alert within the suppression window sends at most one new Slack message and exactly one PagerDuty event (trigger or update, never a duplicate incident).

import hashlib
import logging
import os
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional

import requests
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
logger = logging.getLogger(__name__)

PAGERDUTY_EVENTS_URL = "https://events.pagerduty.com/v2/enqueue"
SUPPRESSION_WINDOW_SECONDS = int(os.getenv("SUPPRESSION_WINDOW_SECONDS", "900"))  # 15 min


class Severity(str, Enum):
    INFO = "info"
    WARNING = "warning"
    CRITICAL = "critical"


# PagerDuty's Events API v2 only accepts these four severities on the payload.
_PD_SEVERITY = {Severity.INFO: "info", Severity.WARNING: "warning", Severity.CRITICAL: "critical"}


@dataclass(frozen=True)
class BudgetAlert:
    account_id: str
    service: str
    budget_id: str
    owner_team: str
    severity: Severity
    current_spend: float
    budget_amount: float
    triggered_at: float  # epoch seconds; excluded from the dedup key on purpose


@dataclass(frozen=True)
class RouteTarget:
    slack_webhook: Optional[str] = None
    pagerduty_routing_key: Optional[str] = None


class AlertRouter:
    """Routes normalized budget alerts to Slack and/or PagerDuty by severity + owner."""

    def __init__(self, routing_table: dict[str, RouteTarget], suppression_window: int = SUPPRESSION_WINDOW_SECONDS):
        self.routing_table = routing_table
        self.suppression_window = suppression_window
        self._last_sent: dict[str, float] = {}  # dedup_key -> epoch of last Slack dispatch

    @staticmethod
    def dedup_key(alert: BudgetAlert) -> str:
        # Identity fields only — never triggered_at or current_spend, or every
        # flap across the threshold would mint a fresh key and defeat grouping.
        raw = f"{alert.account_id}:{alert.service}:{alert.budget_id}"
        return hashlib.sha256(raw.encode()).hexdigest()[:40]  # PagerDuty dedup_key max is 255 chars; well under it

    def _is_suppressed(self, key: str) -> bool:
        last = self._last_sent.get(key)
        return last is not None and (time.monotonic() - last) < self.suppression_window

    def route(self, alert: BudgetAlert) -> None:
        target = self.routing_table.get(alert.owner_team)
        if target is None:
            logger.warning("No route configured for owner_team=%s; dropping alert.", alert.owner_team)
            return

        key = self.dedup_key(alert)
        suppressed = self._is_suppressed(key)

        # INFO never pages. WARNING and CRITICAL both notify Slack (subject to
        # suppression). Only CRITICAL pages, and paging bypasses Slack
        # suppression because PagerDuty's own grouping handles the repeat.
        if target.slack_webhook and alert.severity in (Severity.WARNING, Severity.CRITICAL) and not suppressed:
            self._send_slack(alert, target.slack_webhook)
            self._last_sent[key] = time.monotonic()
        elif suppressed:
            logger.info("Suppressed duplicate Slack notification for dedup_key=%s", key)

        if target.pagerduty_routing_key and alert.severity == Severity.CRITICAL:
            self._send_pagerduty(alert, target.pagerduty_routing_key, key)

    @retry(
        retry=retry_if_exception_type(requests.RequestException),
        stop=stop_after_attempt(4),
        wait=wait_exponential(multiplier=1, min=1, max=20),
        reraise=True,
    )
    def _send_slack(self, alert: BudgetAlert, webhook_url: str) -> None:
        overage_pct = round((alert.current_spend / alert.budget_amount - 1) * 100, 1)
        payload = {
            "text": (
                f":rotating_light: *{alert.severity.upper()}* budget alert — "
                f"`{alert.service}` in `{alert.account_id}` is at "
                f"${alert.current_spend:,.2f} of a ${alert.budget_amount:,.2f} budget "
                f"({overage_pct:+.1f}%)."
            )
        }
        resp = requests.post(webhook_url, json=payload, timeout=5)
        if resp.status_code == 429:
            retry_after = int(resp.headers.get("Retry-After", "1"))
            logger.warning("Slack rate-limited us; Retry-After=%ss", retry_after)
            time.sleep(retry_after)
            resp.raise_for_status()  # let tenacity retry the raised exception
        resp.raise_for_status()
        logger.info("Slack notification sent for budget_id=%s", alert.budget_id)

    @retry(
        retry=retry_if_exception_type(requests.RequestException),
        stop=stop_after_attempt(4),
        wait=wait_exponential(multiplier=1, min=1, max=20),
        reraise=True,
    )
    def _send_pagerduty(self, alert: BudgetAlert, routing_key: str, dedup_key: str) -> None:
        payload = {
            "routing_key": routing_key,
            "event_action": "trigger",
            "dedup_key": dedup_key,  # same key across flaps updates one open incident
            "payload": {
                "summary": f"{alert.service} in {alert.account_id} exceeds budget {alert.budget_id}",
                "source": alert.account_id,
                "severity": _PD_SEVERITY[alert.severity],
                "custom_details": {
                    "current_spend": alert.current_spend,
                    "budget_amount": alert.budget_amount,
                    "owner_team": alert.owner_team,
                },
            },
        }
        resp = requests.post(PAGERDUTY_EVENTS_URL, json=payload, timeout=5)
        resp.raise_for_status()
        logger.info("PagerDuty event enqueued, dedup_key=%s", dedup_key)


if __name__ == "__main__":
    routing_table = {
        "platform-infra": RouteTarget(
            slack_webhook=os.environ["SLACK_WEBHOOK_PLATFORM"],
            pagerduty_routing_key=os.environ["PAGERDUTY_ROUTING_KEY_PLATFORM"],
        ),
    }
    router = AlertRouter(routing_table)
    sample = BudgetAlert(
        account_id="123456789012",
        service="AmazonEC2",
        budget_id="prod-ec2-monthly",
        owner_team="platform-infra",
        severity=Severity.CRITICAL,
        current_spend=54200.11,
        budget_amount=45000.00,
        triggered_at=time.time(),
    )
    router.route(sample)

# Verification & Testing

Correctness here means “the right destinations, exactly once, within the window” — verify each independently:

• Dedup stability. Construct two BudgetAlert instances that differ only in current_spend and triggered_at; assert AlertRouter.dedup_key() returns the identical string for both. If it doesn’t, a field that changes between flaps has leaked into the hash.
• Suppression boundary. Call route() twice back-to-back for the same alert with suppression_window=5; assert the mocked Slack POST fires once. Advance a monkeypatched time.monotonic() past the window and call again; assert it fires a second time.
• Severity gating. Assert an INFO alert never calls the PagerDuty send method and a WARNING alert never pages, using unittest.mock.patch.object on _send_pagerduty and asserting assert_not_called().
• Backoff on 429. Mock requests.post to return a 429 with Retry-After: 2 on the first call and 200 on the second; assert the method completes successfully and that at least one retry occurred, confirming the tenacity policy is wired to the right exception type.

# Common Pitfalls Checklist

• Hashing the timestamp into the dedup key. Every flap becomes a “new” event and PagerDuty opens a fresh incident each time — derive the key only from account, service, and budget identity.
• Suppressing PagerDuty the same way as Slack. A page that never fires again because an earlier one is still “suppressed” hides a genuinely escalating incident — let PagerDuty’s server-side dedup grouping own the repeat-page decision, don’t gate it client-side too.
• Sharing one requests.Session across a burst without backoff. A tight loop over dozens of accounts crossing thresholds at once will hit Slack’s or PagerDuty’s rate limit almost immediately — always wrap dispatch calls in retry-with-backoff, not just error logging.
• Routing table baked into code instead of config. Hardcoded if owner_team == "..." chains force a deploy for every new team onboarded — keep the table as external config the router loads at startup.
• Treating a 2xx from Slack as delivery confirmation. Slack’s webhook endpoint returns 200 even if the channel is archived or the workflow step is misconfigured — periodically send a synthetic canary alert and confirm it’s visible, don’t rely solely on HTTP status.

# Frequently Asked Questions

# Related

• Budget Alert Automation with Webhooks — the parent page covering the full webhook-driven alerting pipeline this router plugs into.
• Building Budget Alert Webhook Handlers — the upstream handler that verifies inbound signatures and normalizes raw payloads into the BudgetAlert record this router consumes.
• Building Cost Anomaly Detection Models — the scoring stage that produces the severity label this router keys its routing decisions on.
• Cloud Cost Anomaly Detection & Budget Automation — the overall discipline this routing layer supports, from baseline modeling through CI/CD guardrails.

Up: Budget Alert Automation with Webhooks · Home: Cloud Cost Optimization & FinOps Automation