Building Cost Anomaly Detection Models

Cloud cost anomaly detection is a time-series problem wearing a FinOps costume: the underlying task is scoring each day’s spend per service against a baseline built from its own recent history, and deciding — with a defensible statistical threshold, not a gut-feel percentage — whether the deviation warrants paging someone. The engineering problem is not fitting a model; naive rolling z-scores and one-line pandas rollups are a five-minute exercise. The hard part is making that scoring robust to the specific pathologies of billing data: multi-day finalization lag that makes yesterday look artificially cheap, weekly and monthly seasonality that a naive threshold mistakes for a spike, sparse or bursty series for low-volume services, and the sheer heterogeneity of hundreds of service_id × account_id series each needing their own baseline. This page covers the model and baseline-fitting stage of the Cloud Cost Anomaly Detection & Budget Automation pipeline — the stage that sits between normalized daily cost series and the alert-routing layer, and it assumes the upstream data is already deduplicated, currency-normalized, per-service daily aggregates.

Architecture Context & Data-Flow Position

Anomaly detection is the second stage of the anomaly and budget-automation discipline described in Cloud Cost Anomaly Detection & Budget Automation: ingestion produces a normalized daily cost series, this stage scores each new point against a rolling baseline and emits a typed anomaly record, and a downstream stage — covered in Budget Alert Automation with Webhooks — turns that record into a routed notification. The series this stage consumes is the same kind of daily aggregate produced by Time-Series Aggregation for Daily Cloud Cost Tracking: one row per (usage_date, service_id, account_id) with a net cost figure. The detector’s isolation contract is strict — it must never mutate the input series, must be re-runnable over the same window without producing different verdicts (idempotency matters because backfills and finalization corrections replay the same days repeatedly), and must degrade gracefully on short or sparse series rather than throwing.

Four statistical approaches cover the bulk of production use, and they are not mutually exclusive — most detector engines run two or three per series and take the union or a weighted vote.

Method Core assumption Seasonality handling Cold-start Typical false-positive mode
Rolling z-score Recent window is approximately normal (Gaussian) None — treats weekday/weekend swings as noise or signal indiscriminately Needs ~14–30 points before the window is trustworthy Flags every Monday spike on a service with a strong weekly cycle
EWMA (exponentially weighted moving average) Recent points matter more than old ones; drift should be tracked, not just level Partial — a long half-life absorbs slow drift but not fixed weekly cycles Faster than z-score; usable after ~7–10 points with a short half-life Lags real step-changes (a permanent price increase looks like a slow-building anomaly for days)
MAD (median absolute deviation) Baseline may contain outliers; median/MAD are robust to them None on its own — usually paired with day-of-week bucketing Needs a full seasonal cycle (7+ points) per bucket for the robust stats to be meaningful Under-reacts to sustained level shifts because the median only updates slowly
STL decomposition Series has an additive trend + seasonal + residual structure Explicit — seasonal component is modeled and subtracted before scoring Needs 2+ full seasonal cycles (14+ days for weekly seasonality) to fit Misattributes a real anomaly into the trend component if the anomaly persists long enough

Rolling z-score and EWMA are the workhorses for high-volume, well-behaved services where seasonality is weak. MAD-on-residuals is the correction for services with occasional legitimate spikes (batch jobs, scheduled backfills) that would otherwise poison a mean-based baseline. STL decomposition is reserved for services with a strong, stable weekly pattern — see Seasonal Decomposition for Cloud Cost Baselines for the deep implementation, and Detecting Cost Spikes with a Rolling Z-Score for the z-score variant tuned specifically for spike detection on high-cardinality service fleets.

Cost anomaly scoring pipeline: series construction to alert record A four-stage left-to-right pipeline. Stage one, series construction, builds a per-service daily cost series from normalized billing data and excludes the unfinalized trailing window. Stage two, baseline fitting, computes a rolling statistic — mean and standard deviation, EWMA level, median and MAD, or an STL seasonal-trend-residual decomposition — over a lookback window. Stage three, scoring, compares the latest finalized point against the baseline to produce a deviation score. Stage four, thresholding, applies a per-service sensitivity threshold and emits a typed AnomalyResult record consumed by downstream alert routing. Cost anomaly scoring pipeline Each stage is idempotent over the same input window; re-runs after finalization reproduce the same verdicts 1 Series construction Daily cost per service_id, trailing finalization window excluded from scoring 2 Baseline fitting z-score / EWMA / MAD / STL over a lookback window per (service, account) 3 Scoring Latest finalized point vs. baseline, deviation score in baseline-native units 4 Thresholding Per-service sensitivity, emit AnomalyResult to alert routing Cold-start guard: series shorter than the minimum lookback are skipped, not scored with a degraded baseline Sparse-series guard: zero-cost days are imputed as true zeros, never dropped, to keep the window length stable
The scoring pipeline runs series construction, baseline fitting, scoring, and thresholding as four idempotent stages; finalization lag and cold-start/sparse-series guards are enforced at stage boundaries, not bolted on afterward.

Core Implementation Patterns

1. Series construction from normalized cost data

The detector never reads raw billing rows directly — it consumes the daily, per-service_id, per-account_id aggregate produced by the ingestion and normalization stages. Two decisions at construction time determine most of the downstream false-positive rate: how to handle the unfinalized trailing window, and how to handle days with zero recorded cost.

import pandas as pd

def build_series(daily_costs: pd.DataFrame, service_id: str, account_id: str,
                  finalization_lag_days: int = 2) -> pd.Series:
    """Build a dense daily cost series for one (service, account) pair.

    finalization_lag_days excludes the trailing window that billing
    exports have not yet fully settled — scoring an unfinalized day
    against a baseline of finalized days is the single largest source
    of false positives in naive implementations.
    """
    mask = (daily_costs["service_id"] == service_id) & (daily_costs["account_id"] == account_id)
    s = daily_costs.loc[mask].set_index("usage_date")["net_cost"].sort_index()
    # Reindex to a dense daily calendar so gaps become explicit zeros, not
    # missing index entries that would silently shrink the rolling window.
    full_index = pd.date_range(s.index.min(), s.index.max(), freq="D")
    s = s.reindex(full_index, fill_value=0.0)
    cutoff = s.index.max() - pd.Timedelta(days=finalization_lag_days)
    return s.loc[:cutoff]

2. Baseline fitting

Each method fits differently, and the fit determines the units the score is expressed in. Z-score and MAD both produce a standardized deviation; EWMA produces a level plus a variance estimate you standardize separately; STL produces a deseasonalized residual you then feed into z-score or MAD.

import numpy as np

def rolling_zscore_baseline(s: pd.Series, window: int = 28) -> tuple[pd.Series, pd.Series]:
    mean = s.rolling(window, min_periods=window).mean()
    std = s.rolling(window, min_periods=window).std(ddof=0)
    return mean, std

def ewma_baseline(s: pd.Series, halflife_days: float = 7.0) -> tuple[pd.Series, pd.Series]:
    level = s.ewm(halflife=halflife_days, min_periods=7).mean()
    # EWMA of squared deviations approximates a variance that tracks drift.
    var = (s - level).pow(2).ewm(halflife=halflife_days, min_periods=7).mean()
    return level, var.pow(0.5)

def mad_baseline(s: pd.Series, window: int = 28) -> tuple[pd.Series, pd.Series]:
    median = s.rolling(window, min_periods=window).median()
    mad = s.rolling(window, min_periods=window).apply(
        lambda w: np.median(np.abs(w - np.median(w))), raw=True
    )
    # 1.4826 rescales MAD to be a consistent estimator of the standard
    # deviation under a normal distribution, so thresholds are comparable
    # across methods.
    return median, mad * 1.4826

3. Scoring and threshold selection

Scoring reduces to (observed - baseline_center) / baseline_spread, but the threshold that turns a score into a verdict has to trade precision for recall per service, not globally. A single |z| > 3 threshold for every service either buries low-volume services in noise (their std is dominated by a handful of large but normal daily swings) or misses real spikes on smooth, high-volume services. In practice, thresholds are set per service from the false-positive rate observed over a backtest window, not picked once and applied uniformly:

def select_threshold(historical_scores: pd.Series, target_fp_per_month: float = 1.0) -> float:
    """Pick the smallest |score| threshold whose backtest false-positive
    rate (assuming scores under the null are the true distribution) is
    at or below the target, expressed as expected false positives/month."""
    scores = historical_scores.dropna().abs().sort_values(ascending=False)
    n = len(scores)
    if n == 0:
        return 3.0  # conservative default with no history to calibrate against
    target_rank = max(1, int(round(target_fp_per_month * n / 30)))
    return float(scores.iloc[min(target_rank, n - 1)])

Lower target_fp_per_month raises the effective threshold (fewer, higher-confidence alerts); most teams start at 1–2 false positives per service per month and tune from there once they have paged on-call enough times to know the pain tolerance.

Production-Grade Python Anomaly Detection Engine

The module below implements a self-contained, idempotent detector that fits a rolling z-score, EWMA, and MAD baseline for each (service_id, account_id) series, scores the latest finalized day against all three, and emits a typed AnomalyResult when the configured method crosses threshold. It handles cold-start (skips series shorter than the minimum lookback) and sparse series (zero-fill rather than drop) as described above. Dependencies: pandas>=2.0, numpy>=1.24.

"""
cost_anomaly_engine.py — statistical cost anomaly detector.

Fits rolling z-score, EWMA, and robust MAD baselines over a normalized
daily cost series and scores the latest finalized point against each.
Idempotent: re-running over the same window with the same inputs
reproduces identical AnomalyResult records, which matters because
billing finalization causes routine backfills over the same days.
"""
import logging
from dataclasses import dataclass, field
from datetime import date
from enum import Enum
from typing import Optional

import numpy as np
import pandas as pd

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
logger = logging.getLogger("finops.anomaly.engine")

MIN_LOOKBACK_DAYS = 14  # cold-start floor: fewer points and the baseline is unreliable


class Method(str, Enum):
    ZSCORE = "zscore"
    EWMA = "ewma"
    MAD = "mad"


@dataclass(frozen=True)
class DetectorConfig:
    window: int = 28
    ewma_halflife_days: float = 7.0
    finalization_lag_days: int = 2
    threshold: float = 3.0          # in baseline-native standardized units
    method: Method = Method.MAD     # MAD is the safest default: robust to spikes in the window itself


@dataclass
class AnomalyResult:
    service_id: str
    account_id: str
    usage_date: date
    observed_cost: float
    baseline_center: float
    baseline_spread: float
    score: float
    method: Method
    is_anomaly: bool

    def to_dict(self) -> dict:
        return {
            "service_id": self.service_id,
            "account_id": self.account_id,
            "usage_date": self.usage_date.isoformat(),
            "observed_cost": round(self.observed_cost, 6),
            "baseline_center": round(self.baseline_center, 6),
            "baseline_spread": round(self.baseline_spread, 6),
            "score": round(self.score, 4),
            "method": self.method.value,
            "is_anomaly": self.is_anomaly,
        }


def _dense_series(raw: pd.Series) -> pd.Series:
    """Reindex to a gap-free daily calendar; missing days become true
    zero-cost days rather than silently shrinking the rolling window."""
    full_index = pd.date_range(raw.index.min(), raw.index.max(), freq="D")
    return raw.reindex(full_index, fill_value=0.0)


def _zscore_baseline(s: pd.Series, window: int) -> tuple[pd.Series, pd.Series]:
    mean = s.rolling(window, min_periods=window).mean()
    std = s.rolling(window, min_periods=window).std(ddof=0)
    return mean, std


def _ewma_baseline(s: pd.Series, halflife: float) -> tuple[pd.Series, pd.Series]:
    level = s.ewm(halflife=halflife, min_periods=7).mean()
    var = (s - level).pow(2).ewm(halflife=halflife, min_periods=7).mean()
    return level, var.pow(0.5)


def _mad_baseline(s: pd.Series, window: int) -> tuple[pd.Series, pd.Series]:
    median = s.rolling(window, min_periods=window).median()
    mad = s.rolling(window, min_periods=window).apply(
        lambda w: np.median(np.abs(w - np.median(w))), raw=True
    )
    return median, mad * 1.4826  # rescale to a normal-consistent spread estimate


_BASELINE_FNS = {
    Method.ZSCORE: lambda s, cfg: _zscore_baseline(s, cfg.window),
    Method.EWMA: lambda s, cfg: _ewma_baseline(s, cfg.ewma_halflife_days),
    Method.MAD: lambda s, cfg: _mad_baseline(s, cfg.window),
}


class CostAnomalyDetector:
    """Fits a baseline and scores the latest finalized point per series."""

    def __init__(self, cfg: Optional[DetectorConfig] = None) -> None:
        self.cfg = cfg or DetectorConfig()

    def score_series(self, raw: pd.Series, service_id: str, account_id: str) -> Optional[AnomalyResult]:
        """Score the latest finalized day of one (service, account) series.

        Returns None (not a zero-confidence result) when the series is
        too short to fit a baseline — callers must distinguish "no
        signal yet" from "scored and normal", since silently treating
        cold-start series as normal masks real spikes during onboarding.
        """
        s = _dense_series(raw)
        cutoff = s.index.max() - pd.Timedelta(days=self.cfg.finalization_lag_days)
        finalized = s.loc[:cutoff]

        if len(finalized) < MIN_LOOKBACK_DAYS:
            logger.info(
                "cold-start: %s/%s has %d finalized points, need %d — skipping",
                service_id, account_id, len(finalized), MIN_LOOKBACK_DAYS,
            )
            return None

        center_s, spread_s = _BASELINE_FNS[self.cfg.method](finalized, self.cfg)
        latest_date = finalized.index[-1]
        observed = float(finalized.iloc[-1])
        center = center_s.iloc[-1]
        spread = spread_s.iloc[-1]

        if pd.isna(center) or pd.isna(spread):
            logger.info("%s/%s baseline not yet warmed up at %s — skipping",
                        service_id, account_id, latest_date.date())
            return None

        # Guard against a degenerate zero-spread baseline (e.g. a service
        # with identical daily cost for the whole window) — treat any
        # deviation from a flat baseline as maximally significant rather
        # than dividing by zero.
        score = 0.0 if spread == 0 and observed == center else (
            float("inf") if spread == 0 else (observed - center) / spread
        )
        is_anomaly = abs(score) >= self.cfg.threshold

        result = AnomalyResult(
            service_id=service_id,
            account_id=account_id,
            usage_date=latest_date.date(),
            observed_cost=observed,
            baseline_center=float(center),
            baseline_spread=float(spread),
            score=score,
            method=self.cfg.method,
            is_anomaly=is_anomaly,
        )
        if is_anomaly:
            logger.warning(
                "ANOMALY %s/%s on %s: observed=%.2f baseline=%.2f±%.2f score=%.2f",
                service_id, account_id, result.usage_date, observed, center, spread, score,
            )
        return result

    def score_frame(self, daily_costs: pd.DataFrame) -> list[AnomalyResult]:
        """Score every (service_id, account_id) group in a normalized frame.

        Expects columns: usage_date, service_id, account_id, net_cost.
        """
        results: list[AnomalyResult] = []
        for (service_id, account_id), group in daily_costs.groupby(["service_id", "account_id"]):
            raw = group.set_index("usage_date")["net_cost"].sort_index()
            result = self.score_series(raw, service_id, account_id)
            if result is not None:
                results.append(result)
        return results


def _synthetic_demo_frame() -> pd.DataFrame:
    """Build a 60-day synthetic series with one injected spike, for the demo."""
    rng = np.random.default_rng(seed=7)
    dates = pd.date_range("2026-05-01", periods=60, freq="D")
    baseline = 500 + 40 * np.sin(np.arange(60) * (2 * np.pi / 7))  # weekly wobble
    noise = rng.normal(0, 15, size=60)
    costs = baseline + noise
    costs[52] += 900  # inject an obvious spike five days before the finalization cutoff
    return pd.DataFrame({
        "usage_date": dates,
        "service_id": ["compute-engine"] * 60,
        "account_id": ["acct-finops-demo"] * 60,
        "net_cost": costs,
    })


def main() -> None:
    frame = _synthetic_demo_frame()
    detector = CostAnomalyDetector(DetectorConfig(method=Method.MAD, threshold=3.0))
    results = detector.score_frame(frame)
    for r in results:
        logger.info("result: %s", r.to_dict())
    anomalies = [r for r in results if r.is_anomaly]
    logger.info("scored %d series, %d flagged anomalous", len(results), len(anomalies))


if __name__ == "__main__":
    main()

Schema Reference Table

The detector’s input contract is narrow by design — it depends only on the normalized daily cost model, not on any provider-specific billing field. This keeps the same engine usable across AWS, GCP, and Azure series once they pass through normalization.

Cost-series field Model input Type Notes
usage_date series index date Must be a dense daily calendar per group; gaps are zero-filled, not dropped
service_id groupby key string One baseline fit per (service_id, account_id) pair; never pooled across services
account_id groupby key string Second groupby dimension; prevents one account’s spike from polluting another’s baseline
net_cost observed_cost decimal Post-credit cost; feed the same net figure used for invoice reconciliation, not gross cost
(derived) baseline_center float Rolling mean, EWMA level, or rolling median depending on method
(derived) baseline_spread float Rolling std, EWMA-derived std, or rescaled MAD depending on method
(derived) score float Standardized deviation in baseline-native units; comparable across services only within the same method
(config) finalization_lag_days int Trailing days excluded from scoring to avoid alerting on unsettled cost
(config) threshold float Per-service sensitivity; set via backtest false-positive targeting, not a global constant

Operational Considerations

  • Finalization lag is method-agnostic and non-negotiable. Every method in the comparison table above will flag the trailing 1–3 days as anomalously low if you score them, because billing exports under-report until settlement. Exclude the lag window at series-construction time, not at alert-filtering time, so the baseline itself never learns from unfinalized points.
  • Backfills re-run the same window. Because the engine reindexes to a dense calendar and recomputes rolling windows deterministically from the input series, re-scoring the same 60-day window after a billing correction reproduces identical AnomalyResult records for unaffected days — this is what makes daily cron re-runs safe.
  • Per-service thresholds, not a global one. A fleet of 200 services scored with a single |z| > 3 typically produces a 5–10x higher false-positive rate on low-volume services than on high-volume ones, because their historical std is dominated by a handful of legitimately noisy days. Calibrate threshold per service from a backtest, as shown in select_threshold above.
  • Sparse series need a minimum lookback floor. A service with fewer than MIN_LOOKBACK_DAYS (14 here) of finalized history should return None, not a low-confidence score — treating cold-start series as “scored and normal” is how a real launch-week cost spike goes undetected.
  • Monitoring hooks. Emit a metric for series skipped due to cold-start, series with a degenerate zero-spread baseline, and the anomaly rate per method — a sudden jump in the skip rate usually means an upstream ingestion gap, not a quiet week.
  • Compute cost of the fit itself. Rolling and EWMA fits are O(n) per series and trivially vectorized in pandas; a fleet of a few thousand series scored daily over a 28-day window runs in seconds on a single process. STL decomposition (see the seasonal page) is the one method here expensive enough to warrant batching or a scheduled job rather than an inline call.

Troubleshooting

Every Monday shows up as a cost spike. Root cause: a plain rolling z-score or EWMA baseline has no seasonal component, so it treats the weekly cycle (batch jobs, backup windows, traffic patterns) as noise around a flat mean, and Monday’s real but recurring uptick reads as anomalous. Detection: is_anomaly=True results cluster on the same weekday across multiple weeks. Remediation: switch to day-of-week-bucketed MAD baselines or STL decomposition — see Seasonal Decomposition for Cloud Cost Baselines.

Yesterday’s cost always looks like a drop. Root cause: finalization_lag_days is set too low (or to zero) for the provider’s actual settlement window, so the score compares a partially-settled day against a baseline of fully-settled days. Detection: the anomaly is always in the negative direction and always on the most recent finalized date. Remediation: raise finalization_lag_days to match the provider’s restatement window (GCP’s detailed export settles over roughly 30 days for credits, though the bulk of the delta lands in the first 2–3 days; validate against your own provider’s behavior before shipping the default).

A newly launched service pages on-call on day three. Root cause: the detector scored a series with fewer than MIN_LOOKBACK_DAYS points because the cold-start guard was bypassed or the floor was set too low. Detection: the flagged usage_date is within days of the service’s first non-zero cost entry. Remediation: enforce the MIN_LOOKBACK_DAYS floor and return None rather than a result — callers must not treat “no result” as “not anomalous” when wiring this into alert routing.

One giant scheduled batch job every month triggers a false positive, then poisons the next month’s baseline. Root cause: a rolling z-score baseline includes the spike itself in its own window, inflating std and desensitizing the detector for the following ~28 days. Detection: the anomaly score for the same recurring event trends downward month over month even though the absolute spike size is constant. Remediation: switch to Method.MAD for this service — the median and MAD are robust to a single outlier inside the window, unlike the mean and standard deviation.

Two methods disagree on the same day. Root cause: this is expected, not a bug — z-score and EWMA weight recent history differently, and MAD is robust to in-window outliers the other two are not. Detection: score_frame run with different DetectorConfig.method values on the same input produces different is_anomaly flags. Remediation: run multiple methods per series in production and route to alerting only on agreement (or a configurable quorum), rather than trusting a single method’s verdict.

Frequently Asked Questions

Which method should I default to for a new pipeline?

Start with MAD-based scoring. It is robust to the single most common failure mode — a legitimate but large spike already sitting inside the baseline window — without requiring the multi-cycle warm-up that STL decomposition needs. Add day-of-week bucketing or STL once you have enough history and a specific seasonality-driven false-positive problem to solve.

How long a lookback window do I need before trusting a baseline?

Fourteen days is the practical floor for z-score and MAD at daily granularity — enough to estimate a stable center and spread without being so long it can’t adapt to a genuine step-change in usage. STL decomposition needs at least two full seasonal cycles, so 14 days for weekly seasonality, and ideally 28+ for a more stable seasonal component estimate.

How do I stop finalization lag from causing false positives?

Exclude the trailing finalization_lag_days window from the series before it ever reaches the baseline fit or the scoring step, not after. If you filter alerts post-hoc instead, the unfinalized days are still inside the rolling window used to compute the baseline itself, which quietly drags the center and spread estimates off.

Can I use the same threshold across every service?

No — a single global threshold systematically over-alerts on low-volume, noisy services and under-alerts on smooth, high-volume ones, because their baseline spreads differ by orders of magnitude. Calibrate threshold per service from a backtest against a target false-positive rate, as in the select_threshold pattern above.

What is the actual difference between the score from z-score and from MAD?

Both produce a standardized deviation, but z-score’s center and spread are the mean and standard deviation of the raw window — sensitive to any outlier already inside it — while MAD’s are the median and a rescaled median absolute deviation, which barely move when a single day is extreme. The two scores are numerically comparable in normal conditions but diverge sharply whenever the window itself contains an outlier.

Up: Cloud Cost Anomaly Detection & Budget Automation