Blocking Terraform Plans That Exceed Cost Budgets

A cost-diff comment on a pull request is advisory. It renders a markdown table, a reviewer skims it, and nothing about GitHub’s merge button changes state — the PR merges whether the delta is $12 or $12,000. The only mechanism that actually stops a runaway-spend plan from shipping is a required status check with a failure conclusion, wired into branch protection so the merge button stays locked until it clears. That single distinction — commenting versus gating — is the entire engineering problem this page solves: taking the priced delta a cost-diff step already computed and turning it into a pass/warn/block verdict that either sets a GitHub check-run to failure and exits the CI job non-zero, or lets a sanctioned override through with an audit trail. Get the policy design wrong and you get one of two outcomes in production: a gate so strict it blocks every quarter’s legitimate traffic growth and gets disabled within a month, or a gate so loose it never fires and might as well be the advisory comment it replaced.

# Root Cause & Failure Modes

Three failure patterns show up once teams try to move from “post a comment” to “block the merge”:

• The comment step never sets a conclusion. A workflow step that runs python cost_diff.py and prints a table always exits 0. Even if a human reads “delta: +$4,200 (38%)” in the comment, the job succeeds, the required check (if one even exists) reports green, and branch protection has nothing to act on. Blocking requires a distinct check-run — commonly named something like cost-budget-gate — that a workflow step deliberately fails.
• Binary fail-only policy blocks legitimate growth. A flat percentage cap (say, 20% over baseline) does not know the difference between an unplanned regression and a planned autoscaling change ahead of a product launch. Once a gate blocks a legitimate merge two or three times, engineers route around it — either by disabling the required check in branch protection settings or by getting an admin to force-merge — and the control is now weaker than the advisory comment it replaced, because it also destroyed trust in the tooling.
• No sanctioned override path. Without an explicit, auditable way to say “yes, this breach is expected, ship it,” the only escape hatch is an out-of-band admin override that leaves no record of who approved the overage or why. A gate that cannot be overridden through the same review surface as the code change gets bypassed around that surface instead.

The fix has three parts, matching the three failures: a distinct check-run that the workflow can fail deliberately, a policy that distinguishes a marginal breach from a severe one instead of a single fail-or-pass cutoff, and a labeled, re-triggerable override path that both unblocks the merge and records who authorized it. This builds directly on the plan-pricing and delta computation covered in CI/CD Cost Guardrails and the PR-comment formatting in GitHub Actions Cost-Diff Reporting — this page picks up after the delta is computed and turns it into an enforced verdict.

# Production Pipeline Architecture

The gate runs as a fourth stage after the pricing and diffing already covered in the parent guardrail engine, and it is deliberately separated into its own job so a policy change never requires touching the pricing logic:

1. Ingest the priced delta. Read the delta.json artifact the cost-diff step already wrote — service, baseline_usd, proposed_usd, delta_usd, delta_pct — rather than recomputing it. The gate trusts the upstream engine’s arithmetic and owns only the policy decision.
2. Evaluate policy. Look up the service’s PolicyRule (absolute dollar cap, percent-of-baseline cap, and a growth allowance) and classify the delta as pass, warn, or block. Breaching only one of the two caps is a warn — visible in the check-run summary but non-blocking; breaching both simultaneously is a block.
3. Resolve overrides. Only on a block verdict, query the pull request’s labels through the GitHub API for a specific override label. Its presence does not silently pass the check — it flips the conclusion while logging exactly who has the authority to apply that label and forcing a written justification in the PR itself.
4. Emit the gate. Post a cost-budget-gate check-run with a success or failure conclusion, and exit the CI job with a matching status code — 0 on pass, warn, or sanctioned override; 1 on an unoverridden block. Branch protection reads the check-run, not the exit code directly, but the two must agree or a reviewer sees a green check on a job GitHub still shows as failed.

This mirrors the same plan-time enforcement pattern used for Setting Up FinOps Governance Boundaries in Terraform — a policy evaluated against a plan before apply, not against a resource after it exists — applied specifically to the cost-delta dimension instead of module or provider constraints.

# Step-by-Step Python Implementation

The module below is the gate itself: it loads the delta and a per-service policy file, evaluates a pass/warn/block verdict, checks for the override label only when needed, posts the check-run, and returns the exit code the workflow step propagates.

import argparse
import json
import logging
import sys
from dataclasses import dataclass
from enum import Enum
from typing import Optional

import requests

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(name)s | %(message)s",
)
logger = logging.getLogger("cicd.budget_gate")

OVERRIDE_LABEL = "cost-override-approved"
GITHUB_API = "https://api.github.com"


class Verdict(str, Enum):
    PASS = "pass"
    WARN = "warn"
    BLOCK = "block"


@dataclass
class PolicyRule:
    """Per-service budget policy. growth_allowance_pct widens the percent
    cap for services with a documented, expected growth trend, so a
    forecasted traffic ramp doesn't read as an anomaly."""

    service: str
    max_delta_usd: float
    max_delta_pct: float
    growth_allowance_pct: float = 0.0


@dataclass
class CostDelta:
    """One priced delta, as emitted by the upstream cost-diff engine."""

    service: str
    baseline_usd: float
    proposed_usd: float
    delta_usd: float
    delta_pct: float


def load_policy(policy_path: str) -> dict[str, PolicyRule]:
    with open(policy_path, "r", encoding="utf-8") as fh:
        raw = json.load(fh)
    rules = {
        entry["service"]: PolicyRule(
            service=entry["service"],
            max_delta_usd=float(entry["max_delta_usd"]),
            max_delta_pct=float(entry["max_delta_pct"]),
            growth_allowance_pct=float(entry.get("growth_allowance_pct", 0.0)),
        )
        for entry in raw["services"]
    }
    logger.info("loaded policy for %d service(s)", len(rules))
    return rules


def load_delta(delta_path: str) -> CostDelta:
    with open(delta_path, "r", encoding="utf-8") as fh:
        raw = json.load(fh)
    return CostDelta(
        service=raw["service"],
        baseline_usd=float(raw["baseline_usd"]),
        proposed_usd=float(raw["proposed_usd"]),
        delta_usd=float(raw["delta_usd"]),
        delta_pct=float(raw["delta_pct"]),
    )


def evaluate(delta: CostDelta, policy: dict[str, PolicyRule]) -> Verdict:
    """A breach of ONE cap is a warn; breaching BOTH simultaneously blocks.
    This is what keeps a single-dimension breach (a big workspace growing a
    modest, expected 12%) from hard-blocking on its own."""
    if delta.delta_usd <= 0:
        return Verdict.PASS
    rule = policy.get(delta.service)
    if rule is None:
        logger.warning("no policy rule for service=%s; defaulting to warn", delta.service)
        return Verdict.WARN

    effective_pct_cap = rule.max_delta_pct + rule.growth_allowance_pct
    over_usd = delta.delta_usd > rule.max_delta_usd
    over_pct = delta.delta_pct > effective_pct_cap

    if over_usd and over_pct:
        return Verdict.BLOCK
    if over_usd or over_pct:
        return Verdict.WARN
    return Verdict.PASS


def check_override_label(repo: str, pr_number: int, token: str) -> Optional[str]:
    """Return the override label's name if present on the PR, else None."""
    resp = requests.get(
        f"{GITHUB_API}/repos/{repo}/issues/{pr_number}/labels",
        headers={"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"},
        timeout=15,
    )
    resp.raise_for_status()
    labels = {item["name"] for item in resp.json()}
    if OVERRIDE_LABEL in labels:
        logger.info("override label '%s' present on PR #%d", OVERRIDE_LABEL, pr_number)
        return OVERRIDE_LABEL
    return None


def post_check_run(
    repo: str, sha: str, token: str, verdict: Verdict, delta: CostDelta, override: Optional[str]
) -> None:
    """Set the cost-budget-gate conclusion branch protection reads."""
    if verdict == Verdict.BLOCK and override:
        conclusion, summary = "success", (
            f"BLOCK overridden by label '{override}'. delta=${delta.delta_usd:,.2f} "
            f"({delta.delta_pct:+.1f}%) for {delta.service} — merge is audited via this label."
        )
    elif verdict == Verdict.BLOCK:
        conclusion, summary = "failure", (
            f"Budget breach: delta=${delta.delta_usd:,.2f} ({delta.delta_pct:+.1f}%) for "
            f"{delta.service} exceeds both the dollar and percent caps."
        )
    else:
        conclusion, summary = "success", (
            f"{verdict.value.upper()}: delta=${delta.delta_usd:,.2f} "
            f"({delta.delta_pct:+.1f}%) for {delta.service}."
        )

    resp = requests.post(
        f"{GITHUB_API}/repos/{repo}/check-runs",
        headers={"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"},
        json={
            "name": "cost-budget-gate",
            "head_sha": sha,
            "status": "completed",
            "conclusion": conclusion,
            "output": {"title": "Cost budget gate", "summary": summary},
        },
        timeout=15,
    )
    resp.raise_for_status()
    logger.info("posted check-run conclusion=%s for sha=%s", conclusion, sha)


def main() -> int:
    parser = argparse.ArgumentParser(
        description="Block a Terraform plan whose priced cost delta breaches budget policy."
    )
    parser.add_argument("--delta-file", required=True)
    parser.add_argument("--policy-file", required=True)
    parser.add_argument("--repo", required=True, help="owner/repo")
    parser.add_argument("--pr-number", required=True, type=int)
    parser.add_argument("--sha", required=True)
    parser.add_argument("--token", required=True, help="token with checks:write, issues:read")
    args = parser.parse_args()

    delta = load_delta(args.delta_file)
    policy = load_policy(args.policy_file)
    verdict = evaluate(delta, policy)

    override = check_override_label(args.repo, args.pr_number, args.token) if verdict == Verdict.BLOCK else None
    post_check_run(args.repo, args.sha, args.token, verdict, delta, override)

    if verdict == Verdict.BLOCK and not override:
        logger.error("blocking merge: %s breaches budget with no override label", delta.service)
        return 1
    if verdict == Verdict.BLOCK and override:
        logger.warning("verdict BLOCK overridden by '%s' for %s; audit this merge", override, delta.service)
    return 0


if __name__ == "__main__":
    sys.exit(main())

The workflow step that invokes it needs to re-run when the override label is added or removed, not just on synchronize, or a stale failure conclusion sits on the PR until the next commit:

name: cost-budget-gate
on:
  pull_request:
    types: [opened, synchronize, labeled, unlabeled]
    paths: ["**.tf"]

permissions:
  checks: write
  issues: read
  contents: read

jobs:
  budget-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Terraform plan
        run: |
          terraform init -input=false
          terraform plan -out=plan.tfplan -input=false
          terraform show -json plan.tfplan > plan.json
      - name: Price and diff against baseline
        run: python cost_diff.py plan.json baseline.json > delta.json
      - name: Evaluate budget gate
        run: |
          python budget_gate.py \
            --delta-file delta.json \
            --policy-file policy.json \
            --repo "$" \
            --pr-number "$" \
            --sha "$" \
            --token "$"

policy.json holds the per-service caps that load_policy reads, for example {"services": [{"service": "AmazonRDS", "max_delta_usd": 500, "max_delta_pct": 20, "growth_allowance_pct": 10}]} — the growth_allowance_pct is the mechanism for a documented, time-bound expansion (a launch, a migration) that widens the percent cap without touching the dollar cap or the default policy for every other service.

# Verification & Testing

Treat the gate’s exit code as the thing under test, not just the printed summary:

• Matrix the four states. Run main() against fixtures for pass, warn, block without override, and block with the override label mocked present. Assert the exit code is 0 for the first three and the fourth, and 1 only for unoverridden block.
• Assert the conclusion matches the exit code. A test that stubs requests.post and captures the JSON body should confirm conclusion == "failure" if and only if main() returns 1. A mismatch here is the exact failure mode that lets a reviewer see a passing check on a job GitHub Actions still reports as failed.
• Boundary-test the growth allowance. Construct a delta whose delta_pct sits between max_delta_pct and max_delta_pct + growth_allowance_pct and confirm it resolves to warn, not block — this is the specific case the allowance exists to protect.
• Dry-run the label check against a real PR. Point check_override_label at a disposable test PR, add and remove cost-override-approved, and confirm the function’s return value flips accordingly before wiring it into a required check.

# Common Pitfalls Checklist

• Workflow doesn’t trigger on labeled/unlabeled. Adding the override label after a block verdict leaves the stale failure conclusion in place until the next push — fix by including both event types in the workflow’s on.pull_request.types.
• Treating any single-cap breach as a hard block. A flat “breach either cap and fail” policy (fine for the advisory comment) blocks routine growth constantly at the gate stage — reserve block for breaching both caps and use warn for one.
• Applying one global growth allowance to every service. A blanket allowance masks a genuine anomaly in a service with no growth plan — scope growth_allowance_pct per service in the policy file, and expire it explicitly rather than leaving it permanent.
• Letting anyone apply the override label. If the label isn’t restricted via a CODEOWNERS-style branch protection rule or a required reviewer team, it becomes a self-service bypass — restrict who can apply cost-override-approved the same way you restrict who can approve the PR itself.
• Swallowing the check-run POST failure. If post_check_run raises and the exception isn’t handled, the CI job fails for the wrong reason and the actual budget verdict never reaches the PR — let raise_for_status() propagate so the job fails loudly rather than exiting 0 with no recorded conclusion.

# Frequently Asked Questions

# Related

• CI/CD Cost Guardrails — the parent engine that prices the Terraform plan and computes the delta this gate consumes.
• GitHub Actions Cost-Diff Reporting — the PR-comment formatting stage that runs alongside this gate in the same workflow.
• Setting Up FinOps Governance Boundaries in Terraform — the account- and module-level plan-time controls this gate complements at the cost-delta dimension.
• Cloud Cost Anomaly Detection & Budget Automation — the parent reference positioning plan-time blocking against reactive anomaly detection and budget alerting.

Up: CI/CD Cost Guardrails