Budget Alert Automation with Webhooks

A budget threshold breach is worthless as a financial signal until it reaches a human or an automated enforcement action within minutes, exactly once, with the right urgency attached. Every hyperscaler ships a native budget-notification primitive — AWS Budgets publishes to an Amazon SNS topic, GCP Budgets publishes to a Pub/Sub topic, and Azure Cost Management invokes an action-group webhook — but none of the three delivers a normalized payload, guarantees single delivery, or knows anything about your on-call routing. This page covers the alert-and-enforce stage of Cloud Cost Anomaly Detection & Budget Automation: the engineering problem of turning three incompatible, at-least-once notification streams into one idempotent, authenticated, severity-routed alert pipeline that fans out to Slack and PagerDuty without paging on-call twice for the same breach.

# Architecture Context & Data-Flow Position

Budget notifications sit downstream of the statistical detection work covered in Building Cost Anomaly Detection Models: anomaly models catch spend that deviates from a learned baseline, while budget alerts catch spend that crosses an explicit, human-set ceiling. Both feed the same enforcement surface, so this pipeline is designed to accept alerts from either source through one receiver contract. The three provider notification mechanisms are fundamentally at-least-once delivery systems layered on different transports, which means the receiver — not the provider — is where correctness has to live.

The API-level shape of each provider’s notification differs enough that a thin per-provider adapter has to run before anything touches shared logic:

Two structural constraints follow directly from this table. First, none of the three transports is exactly-once — SNS retries HTTP/S subscribers, Pub/Sub redelivers unacknowledged messages, and Azure’s action-group webhook call can itself be retried by whatever orchestrator sits in front of it — so deduplication has to be a first-class stage, not an afterthought. Second, none of the three is low-latency; budget evaluation runs on a batch cadence measured in hours, not seconds, which sets the realistic floor for how “real time” this pipeline can ever be. The deep implementation of the receiver itself — routes, request validation, and provider-specific parsing — lives in Building Budget Alert Webhook Handlers; the fan-out and on-call routing logic is expanded in Routing Budget Alerts to Slack and PagerDuty.

# Core Implementation Patterns

# 1. Receiver Authentication & Signature Verification

Never trust an inbound webhook body on transport alone. AWS’s raw SNS delivery is itself signed with an X.509 certificate you can validate at the subscribing function, but once you fan three providers into one internal receiver, the simplest and most portable pattern is to have each per-provider adapter re-sign the normalized envelope with a shared HMAC-SHA256 secret before forwarding it — the same approach Stripe and Slack use for their own outbound webhooks. Sign over timestamp + "." + raw_body, not just the body, so a captured signature cannot be replayed indefinitely, and compare digests with hmac.compare_digest to avoid a timing side-channel. Reject anything outside a 5-minute clock-skew window.

# 2. Idempotent Deduplication

Because every transport is at-least-once, the receiver must collapse retried deliveries of the same breach into a single downstream action while still treating a genuinely new breach (a higher threshold crossed on the same budget) as distinct. The dedup key is a content hash of provider + account_id + budget_name + notification_type + threshold_pct — never a provider-supplied message ID, since SNS, Pub/Sub, and Azure each mint IDs differently and a new ID on every redelivery would defeat the purpose. The dedup store’s TTL must exceed the longest plausible redelivery window across all three providers, which in practice means sizing it against Pub/Sub’s message retention rather than SNS’s much shorter retry cycle.

# 3. Retry With Backoff on Downstream Delivery

Slack and PagerDuty are themselves rate-limited and occasionally unavailable, so outbound delivery needs its own bounded retry independent of the inbound provider’s retry policy. Wrap outbound POSTs in tenacity’s exponential backoff restricted to transient network errors (ConnectionError, Timeout) — never retry on a 4xx, which signals a malformed request that will fail identically on every attempt. Critically, if all retries to a downstream sink are exhausted, do not mark the alert as delivered in the dedup store; let the upstream provider’s own retry policy hand the event back rather than silently dropping a breach that never reached anyone.

# 4. Severity Mapping & Routing Rules

A FORECASTED notification at 50% of budget and an ACTUAL notification at 100% of budget are not the same event, and routing them identically either floods on-call with noise or buries a real overspend in a Slack channel nobody watches. Classify severity from notification_type and threshold_pct — ACTUAL breaches at or above 100% are CRITICAL and page; anything at or above a 90% threshold is WARNING and posts to Slack; everything else is INFO. Keep the severity-to-destination mapping in a data table, not branching code, so adding a new destination or tightening a threshold is a config change. Route by team using the same cost_center / owner tags validated upstream in Resource Tagging & Validation Pipelines — a budget without a resolvable owner tag should route to a fallback FinOps channel rather than vanish.

# Production-Grade Python Webhook Receiver

The module below is the full receiver: HMAC signature verification, per-provider normalization into a shared Alert dataclass, an in-memory TTL dedup store (swap for Redis or DynamoDB in production), tenacity-backed retries to Slack and PagerDuty, and a Flask app with a /webhook/<provider> route. Dependencies: flask>=3.0, tenacity>=8.2, requests>=2.31.

"""
Budget alert webhook receiver.

Normalizes AWS Budgets (via SNS), GCP Budgets (via Pub/Sub push), and Azure
Cost Management (via Action Group webhook) notifications into one Alert
model, verifies an HMAC-SHA256 signature applied by the upstream provider
adapter, deduplicates on a content hash, and fans the surviving alert out to
Slack and PagerDuty with bounded retries.
"""
from __future__ import annotations

import hashlib
import hmac
import json
import logging
import os
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, Optional

import requests
from flask import Flask, request, jsonify
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential,
)

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%SZ",
)
logger = logging.getLogger("finops.budget_webhook")

# Shared secret used to verify the HMAC signature the upstream per-provider
# normalizer (Lambda / Cloud Function / Azure Function) attaches to every
# forwarded event. Rotate via a secrets manager; never hardcode in git.
WEBHOOK_SECRET = os.environ.get("BUDGET_WEBHOOK_SECRET", "").encode("utf-8")
SIGNATURE_HEADER = "X-Signature-256"
TIMESTAMP_HEADER = "X-Signature-Timestamp"
MAX_CLOCK_SKEW_SECONDS = 300  # reject requests signed more than 5 minutes ago

# How long a dedup key is remembered. Must exceed the longest redelivery
# window across providers -- SNS's default backoff plus Pub/Sub's message
# retention -- or a legitimate redelivery would ship twice.
DEDUP_TTL_SECONDS = 24 * 60 * 60

SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL", "")
PAGERDUTY_ROUTING_KEY = os.environ.get("PAGERDUTY_ROUTING_KEY", "")
PAGERDUTY_EVENTS_URL = "https://events.pagerduty.com/v2/enqueue"

# Severity -> destinations. INFO stays in Slack only; WARNING and CRITICAL
# also page. Extend this table, never branch on severity in code.
ROUTING_RULES: Dict[str, Dict[str, bool]] = {
    "INFO": {"slack": True, "pagerduty": False},
    "WARNING": {"slack": True, "pagerduty": False},
    "CRITICAL": {"slack": True, "pagerduty": True},
}


@dataclass
class Alert:
    """The normalized alert model every provider payload is mapped onto."""

    provider: str                 # aws | gcp | azure
    account_id: str
    budget_name: str
    notification_type: str        # ACTUAL | FORECASTED
    threshold_pct: float
    actual_amount: float
    budget_limit: float
    currency: str
    received_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
    severity: str = "INFO"
    dedup_key: str = ""

    def __post_init__(self) -> None:
        self.severity = self._classify_severity()
        if not self.dedup_key:
            self.dedup_key = self._compute_dedup_key()

    def _classify_severity(self) -> str:
        # Forecasted breaches under 100% are early warnings; an actual spend
        # breach at or above 100% of the limit is the only CRITICAL case.
        if self.notification_type == "ACTUAL" and self.actual_amount >= self.budget_limit:
            return "CRITICAL"
        if self.threshold_pct >= 90:
            return "WARNING"
        return "INFO"

    def _compute_dedup_key(self) -> str:
        # Two independent redeliveries of the SAME breach must collapse to
        # the SAME key; a NEW breach (new threshold crossed) must not.
        canonical = "|".join([
            self.provider,
            self.account_id,
            self.budget_name,
            self.notification_type,
            f"{self.threshold_pct:.0f}",
        ])
        return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:32]


class DedupStore:
    """In-memory TTL cache. Swap for Redis SETNX or a DynamoDB conditional
    write in production -- a single process's memory does not survive a
    redeploy or scale past one instance."""

    def __init__(self, ttl_seconds: int = DEDUP_TTL_SECONDS) -> None:
        self._ttl = ttl_seconds
        self._seen: Dict[str, float] = {}

    def seen_recently(self, key: str) -> bool:
        self._evict_expired()
        return key in self._seen

    def remember(self, key: str) -> None:
        self._seen[key] = time.time()

    def _evict_expired(self) -> None:
        cutoff = time.time() - self._ttl
        expired = [k for k, t in self._seen.items() if t < cutoff]
        for k in expired:
            del self._seen[k]


def verify_signature(raw_body: bytes, signature: str, timestamp: str) -> bool:
    """Constant-time HMAC-SHA256 check over `timestamp.raw_body`, mirroring
    Stripe/Slack-style signed webhooks so a captured signature cannot be
    replayed indefinitely."""
    if not signature or not timestamp:
        return False
    try:
        skew = abs(time.time() - float(timestamp))
    except ValueError:
        return False
    if skew > MAX_CLOCK_SKEW_SECONDS:
        logger.warning("rejected signature: clock skew %.0fs exceeds tolerance", skew)
        return False
    signing_payload = f"{timestamp}.".encode("utf-8") + raw_body
    expected = hmac.new(WEBHOOK_SECRET, signing_payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)


def normalize_aws(payload: Dict[str, Any]) -> Alert:
    """AWS Budgets publishes a JSON string as the SNS Message body."""
    msg = json.loads(payload["Message"]) if isinstance(payload.get("Message"), str) else payload
    return Alert(
        provider="aws",
        account_id=msg["accountId"],
        budget_name=msg["budgetName"],
        notification_type=msg["notificationType"],
        threshold_pct=float(msg["threshold"]),
        actual_amount=float(msg["actualSpend"]),
        budget_limit=float(msg["budgetLimit"]),
        currency=msg.get("currency", "USD"),
    )


def normalize_gcp(payload: Dict[str, Any]) -> Alert:
    """GCP delivers a Pub/Sub push envelope; the budget fields are carried
    in message.attributes by the upstream provider adapter."""
    data = payload["message"]["attributes"]
    return Alert(
        provider="gcp",
        account_id=data["billingAccountId"],
        budget_name=data["budgetDisplayName"],
        notification_type="FORECASTED" if data.get("forecastedSpend") else "ACTUAL",
        threshold_pct=float(data["alertThresholdExceeded"]) * 100,
        actual_amount=float(data["costAmount"]),
        budget_limit=float(data["budgetAmount"]),
        currency=data.get("currencyCode", "USD"),
    )


def normalize_azure(payload: Dict[str, Any]) -> Alert:
    """Azure Action Groups invoke the webhook with a `data` envelope."""
    data = payload["data"]
    return Alert(
        provider="azure",
        account_id=data["SubscriptionId"],
        budget_name=data["BudgetName"],
        notification_type="FORECASTED" if data.get("BudgetType") == "Forecasted" else "ACTUAL",
        threshold_pct=float(data["SpendingThreshold"]),
        actual_amount=float(data["SpendingAmount"]),
        budget_limit=float(data["Budget"]),
        currency=data.get("Unit", "USD"),
    )


NORMALIZERS = {"aws": normalize_aws, "gcp": normalize_gcp, "azure": normalize_azure}

TRANSIENT_HTTP = (requests.ConnectionError, requests.Timeout)


@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential(multiplier=1, min=1, max=15),
    retry=retry_if_exception_type(TRANSIENT_HTTP),
    reraise=True,
)
def _post_with_retry(url: str, json_body: Dict[str, Any], headers: Optional[Dict[str, str]] = None) -> requests.Response:
    resp = requests.post(url, json=json_body, headers=headers or {}, timeout=5)
    resp.raise_for_status()
    return resp


def send_to_slack(alert: Alert) -> None:
    text = (
        f"*[{alert.severity}] {alert.provider.upper()} budget `{alert.budget_name}`*\n"
        f"{alert.notification_type} spend {alert.actual_amount:.2f} {alert.currency} "
        f"({alert.threshold_pct:.0f}% of {alert.budget_limit:.2f} limit)"
    )
    _post_with_retry(SLACK_WEBHOOK_URL, {"text": text})
    logger.info("delivered dedup_key=%s to slack", alert.dedup_key)


def send_to_pagerduty(alert: Alert) -> None:
    body = {
        "routing_key": PAGERDUTY_ROUTING_KEY,
        "event_action": "trigger",
        "dedup_key": alert.dedup_key,
        "payload": {
            "summary": f"{alert.provider.upper()} budget {alert.budget_name} breached {alert.threshold_pct:.0f}%",
            "severity": "critical" if alert.severity == "CRITICAL" else "warning",
            "source": alert.account_id,
            "custom_details": {
                "actual_amount": alert.actual_amount,
                "budget_limit": alert.budget_limit,
                "notification_type": alert.notification_type,
            },
        },
    }
    _post_with_retry(PAGERDUTY_EVENTS_URL, body)
    logger.info("delivered dedup_key=%s to pagerduty", alert.dedup_key)


def route(alert: Alert) -> None:
    rules = ROUTING_RULES.get(alert.severity, ROUTING_RULES["INFO"])
    if rules["slack"]:
        send_to_slack(alert)
    if rules["pagerduty"]:
        send_to_pagerduty(alert)


app = Flask(__name__)
dedup_store = DedupStore()


@app.route("/webhook/<provider>", methods=["POST"])
def receive(provider: str):
    if provider not in NORMALIZERS:
        return jsonify({"error": "unknown provider"}), 404

    raw_body = request.get_data()
    signature = request.headers.get(SIGNATURE_HEADER, "")
    timestamp = request.headers.get(TIMESTAMP_HEADER, "")
    if not verify_signature(raw_body, signature, timestamp):
        logger.warning("signature verification failed for provider=%s", provider)
        return jsonify({"error": "invalid signature"}), 401

    try:
        payload = json.loads(raw_body)
        alert = NORMALIZERS[provider](payload)
    except (KeyError, ValueError, TypeError) as exc:
        logger.error("malformed payload from provider=%s: %s", provider, exc)
        return jsonify({"error": "malformed payload"}), 400

    if dedup_store.seen_recently(alert.dedup_key):
        logger.info("dropped duplicate dedup_key=%s provider=%s", alert.dedup_key, provider)
        return jsonify({"status": "duplicate", "dedup_key": alert.dedup_key}), 200

    try:
        route(alert)
    except TRANSIENT_HTTP as exc:
        # Downstream stayed down through every retry. Do not remember the
        # dedup key -- the provider's own retry policy will hand it back.
        logger.error("delivery failed after retries dedup_key=%s: %s", alert.dedup_key, exc)
        return jsonify({"status": "delivery_failed", "dedup_key": alert.dedup_key}), 502

    dedup_store.remember(alert.dedup_key)
    return jsonify({"status": "delivered", "severity": alert.severity, "dedup_key": alert.dedup_key}), 200


@app.route("/healthz", methods=["GET"])
def healthz():
    return jsonify({"status": "ok"}), 200


if __name__ == "__main__":
    if not WEBHOOK_SECRET:
        raise RuntimeError("BUDGET_WEBHOOK_SECRET must be set before starting the receiver")
    app.run(host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))

The design choices that matter at scale: severity is computed once in Alert.__post_init__ so routing logic never re-derives it inconsistently; the dedup key deliberately excludes provider-issued message IDs because those differ on every redelivery by design; retries on downstream delivery are scoped to network-transient exceptions only, never to a 4xx that will fail identically forever; and a failed delivery after retries is not marked as seen, which relies on the upstream provider’s own at-least-once guarantee to hand the alert back rather than silently swallowing a real breach.

# Schema Reference Table

This is the field-level map from each provider’s raw notification payload to the normalized Alert model. Version this table alongside the code — a provider renaming or restructuring a field (which AWS, GCP, and Azure have all done at least once) is a one-line update here, not a debugging session in production.

# Operational Considerations

• SNS delivery retries. Amazon SNS’s default HTTP/S delivery policy issues a small number of immediate retries followed by a fixed backoff phase — out of the box, roughly 3 retries spaced ~20 seconds apart before the delivery attempt is abandoned. For a budget-alert topic, replace the default with an explicit delivery policy that extends the backoff phase (AWS supports configuring it out to tens of thousands of retries over multiple weeks) so a receiver outage of a few minutes does not silently drop a breach notification.
• Pub/Sub acknowledgement deadline. Push subscriptions default to a 10-second ack deadline, extendable up to 600 seconds. If the receiver takes longer than the configured deadline to return 200, Pub/Sub treats the message as unacknowledged and redelivers it — which is precisely why the receiver must ack fast (verify, normalize, dedup-check, enqueue) rather than block on a slow downstream Slack/PagerDuty call inline. Unacknowledged messages are retried with exponential backoff until the subscription’s message retention expires (default 7 days, configurable up to 31).
• Azure evaluation cadence. Cost Management budgets are evaluated a limited number of times per day against actual and amortized cost data, not continuously — a spend spike can take up to ~8 hours to surface as an action-group invocation. This is a hard ceiling on how “real time” budget-based alerting can be; anomaly detection models, not budget thresholds, are the right tool when minutes matter.
• Idempotency store sizing. Size the dedup TTL against the longest redelivery window in play, not the shortest — in this stack that is Pub/Sub’s retention window, not SNS’s retry backoff. Under-sizing it is the most common cause of duplicate pages.
• Monitoring hooks. Emit metrics for signature-verification failures (a spike usually means a rotated secret didn’t propagate), dedup hit rate, delivery failures per downstream sink, and end-to-end latency from provider notification timestamp to Slack/PagerDuty delivery. Alert if signature failures exceed a low baseline — that is either a misconfigured adapter or an attempted forgery.

# Troubleshooting

The same budget breach pages on-call twice within a minute. Root cause: the dedup key incorporated a provider-issued message ID (SNS MessageId, Pub/Sub messageId) instead of a content hash, so a legitimate redelivery looked like a new event. Detection: two PagerDuty incidents share identical budget_name, threshold_pct, and notification_type within the provider’s retry window. Remediation: rebuild the dedup key from budget content fields only, as Alert._compute_dedup_key does, and backfill the dedup store’s TTL to cover the longest provider redelivery window.

Signature verification rejects legitimate requests after a secret rotation. Root cause: the per-provider adapter and the central receiver were updated with the new BUDGET_WEBHOOK_SECRET out of order, so requests signed with the new secret fail against a receiver still holding the old one. Detection: a step-function spike in 401 responses immediately following a rotation deploy. Remediation: accept both the current and previous secret for a bounded overlap window during rotation, and verify signatures against raw request bytes — never against a re-serialized JSON object, since re-encoding can silently change whitespace and break the HMAC.

SNS subscription stays PendingConfirmation forever. Root cause: the receiver’s endpoint never handled the SNS SubscriptionConfirmation message type and auto-confirmed by fetching SubscribeURL. Detection: no alerts arrive from AWS at all, and the SNS console shows the subscription unconfirmed. Remediation: add explicit handling for x-amz-sns-message-type: SubscriptionConfirmation in the provider adapter in front of this receiver, distinct from the Notification type this page’s handler expects.

Azure alerts stop arriving after a subscription move or budget edit. Root cause: action groups are scoped to the subscription or resource group the budget was created under; moving a subscription between management groups or editing the budget’s scope can silently detach the action group. Detection: the last successful Azure delivery timestamp in your monitoring hook stalls while Cost Management shows the budget still active. Remediation: re-attach the action group explicitly after any scope change, and alert on “no delivery from provider=azure in N hours” rather than trusting Azure to notify you of the misconfiguration.

PagerDuty floods on a single forecasted breach. Root cause: severity classification treated FORECASTED notifications at a low threshold the same as ACTUAL breaches at 100%, so routine early warnings paged on-call. Detection: PagerDuty incident volume spikes for notification_type=FORECASTED events. Remediation: tighten the ROUTING_RULES table so only ACTUAL breaches at or above the configured limit reach CRITICAL, and route FORECASTED warnings to Slack only, as covered in depth in Routing Budget Alerts to Slack and PagerDuty.

# Frequently Asked Questions

# Related

• Cloud Cost Anomaly Detection & Budget Automation — the parent reference defining where alerting and enforcement sit in the broader anomaly-detection pipeline.
• Building Cost Anomaly Detection Models — the statistical detection stage that feeds the same alert-and-enforce surface as provider budget thresholds.
• Building Budget Alert Webhook Handlers — the deep implementation of the receiver’s routes, request validation, and per-provider parsing.
• Routing Budget Alerts to Slack and PagerDuty — the full severity-mapping and on-call routing rules referenced in this page’s Core Implementation Patterns.
• Handling Billing API Rate Limits & Retries — the retry, backoff, and quota patterns behind this page’s tenacity-based downstream delivery.

Up: Cloud Cost Anomaly Detection & Budget Automation · Home: Cloud Cost Optimization & FinOps Automation