Azure Cost Management API Integration

The Azure Cost Management Query API is the acquisition channel for Azure inside a multi-cloud billing system: an authenticated, synchronous REST endpoint that returns aggregated or line-item consumption on demand. It is the Azure-specific implementation of the acquisition stage defined in the Cloud Billing Data Ingestion & Parsing pipeline, and it behaves nothing like the file-drop model used elsewhere. Where the AWS CUR to Data Lake Pipeline reacts to objects landing in S3 and the GCP BigQuery Billing Export Sync reads from a managed export table, Azure forces you to issue explicit HTTP POST requests against Microsoft.CostManagement/query, page through nextLink cursors, and survive aggressive rate limits. This page covers the constraints that dictate reliability — scope-bound RBAC, payload schema, throttling, and dimension mapping — then ships a self-contained Python ingestion engine and the failure modes that take it down at 3 a.m.

# Architecture Context & Data-Flow Position

This component sits at the boundary between Azure’s management plane and your analytical store. It is the only stage permitted to hold Azure credentials and talk to management.azure.com; everything downstream — normalization, allocation, persistence — operates on the canonical rows it emits. Because the API is synchronous and rate-limited, the engine owns retry logic, idempotent emission, and explicit error boundaries so a transient 429 or 503 never corrupts pipeline state. The same idempotency contract enforced across the parent pipeline applies here: every page of rows carries a deterministic key so re-runs over an overlapping window converge rather than double-count.

The request surface is small but unforgiving. The fields below define every production call; getting the scope and api-version wrong is the most common source of silent breakage.

Token acquisition, scope resolution, and base setup mirror the console configuration walked through in Azure Cost Management Setup; this page assumes that the subscription, billing account, and directory consent already exist and focuses on programmatic ingestion.

# Core Implementation Patterns

# 1. Least-Privilege IAM with Entra ID

Azure Cost Management exclusively enforces Microsoft Entra ID (formerly Azure AD) authentication. Interactive credentials, personal access tokens, and shared secrets are prohibited in automated pipelines. Provision a dedicated Service Principal (SPN) or a workload Managed Identity and assign exactly one role at the tier you query:

• Cost Management Reader — subscription or resource group scopes.
• Billing Reader — Enterprise Agreement (EA) or Microsoft Customer Agreement (MCA) billing account scopes.

While the azure-identity SDK offers DefaultAzureCredential for environment-agnostic resolution, bind credentials explicitly via AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_CLIENT_SECRET in CI/CD runners and containers. The convenience chain otherwise falls back to a developer’s cached Azure CLI login, which works on a laptop and fails in production. Insufficient scope yields 403 Forbidden; a cross-tenant query lacking directory consent fails during the OAuth 2.0 token exchange itself, before any cost query runs. Validate the scope URI before execution and rotate the client secret on a schedule aligned to your compliance mandate.

# 2. Query Construction & Scope Resolution

The query body declares type, timeframe, and a dataset. Use ActualCost for finalized billed charges, AmortizedCost to spread reservation and savings-plan purchases across their term, or Usage for raw metered consumption. The dataset governs granularity, aggregation, grouping, and OData filter expressions.

def build_query(scope_type: str = "ResourceGroup") -> dict:
    return {
        "type": "ActualCost",
        "timeframe": "BillingMonthToDate",
        "dataset": {
            "granularity": "Daily",
            "aggregation": {
                "totalCost": {"name": "PreTaxCost", "function": "Sum"}
            },
            "grouping": [{"type": "Dimension", "name": scope_type}],
        },
    }

Always pin the api-version query parameter (for example 2023-11-01). Omitting it resolves to a legacy default whose response envelope differs, and unpinned versions are the usual cause of a pipeline that “suddenly” returns a different column order after a Microsoft platform update.

# 3. Pagination & Rate-Limit Handling

The API returns at most one page of rows plus a properties.nextLink cursor; offset paging is unsupported. Follow nextLink verbatim — it already encodes the scope, api-version, and continuation token, so the follow-up request must send no body. Throttling is enforced per scope and surfaces as 429 TooManyRequests with a Retry-After header; 503/504 appear under management-plane contention. Honour Retry-After exactly, and fall back to exponential backoff with jitter for 5xx. The detailed provider retry matrix and Retry-After parsing behaviour are shared across channels in Handling Billing API Rate Limits & Retries; the deeper edge cases around overlapping windows and cursor reuse live in the Azure Cost API Pagination and Deduplication Guide.

# 4. Metric & Dimension Mapping

The response is column-oriented: properties.columns lists names and types, and each entry in properties.rows is a positional array aligned to those columns. Never assume a fixed column order — build a name→index map from columns first, then project rows through it. Map Azure dimensions (ResourceGroup, MeterCategory, ServiceName, currency) onto the canonical model so allocation logic stays provider-agnostic. Committed-usage reconciliation — Azure Reservations versus Savings Plans — should align with the amortization schedules described in Reserved Instance Mapping Logic.

# Production-Grade Python Ingestion Engine

The engine below is self-contained. It resolves a token through azure-identity, decorates the request with a retry policy that respects Retry-After and backs off 5xx with jitter, walks the nextLink cursor, projects column-oriented rows into a typed dataclass, and assigns a deterministic key for downstream upserts. Structured logging carries the scope and a correlation id without leaking payloads, and a __main__ guard shows a subscription-scoped run.

import os
import time
import uuid
import json
import logging
from dataclasses import dataclass, asdict
from decimal import Decimal
from functools import wraps
from typing import Callable, Dict, List, Optional

import requests
from azure.identity import DefaultAzureCredential

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
)
logger = logging.getLogger("azure_cost_ingestion")

MANAGEMENT_BASE = "https://management.azure.com"
TOKEN_AUDIENCE = "https://management.azure.com/.default"
API_VERSION = "2023-11-01"
REQUEST_TIMEOUT = 300  # seconds; cap runaway management-plane calls


@dataclass(frozen=True)
class CostRecord:
    """Canonical row emitted to the normalization stage."""
    scope: str
    usage_date: str
    resource_group: str
    pre_tax_cost: Decimal
    currency: str
    dedup_key: str

    def to_json(self) -> Dict:
        d = asdict(self)
        d["pre_tax_cost"] = str(self.pre_tax_cost)  # Decimal is not JSON-native
        return d


def with_retries(max_retries: int = 5, max_backoff: float = 30.0) -> Callable:
    """Retry decorator: honour Retry-After on 429, jittered backoff on 5xx."""
    def decorator(fn: Callable) -> Callable:
        @wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_retries + 1):
                try:
                    return fn(*args, **kwargs)
                except requests.exceptions.HTTPError as exc:
                    status = exc.response.status_code
                    if status == 429:
                        wait = int(exc.response.headers.get("Retry-After", 60))
                        logger.warning(
                            "Throttled (429); waiting %ss [attempt %d/%d]",
                            wait, attempt, max_retries,
                        )
                    elif status >= 500:
                        jitter = os.urandom(1)[0] / 255.0
                        wait = min(2 ** attempt + jitter, max_backoff)
                        logger.warning(
                            "Server error %d; backing off %.2fs [attempt %d/%d]",
                            status, wait, attempt, max_retries,
                        )
                    else:
                        logger.error("Fatal HTTP %d: %s", status, exc.response.text)
                        raise
                    if attempt == max_retries:
                        raise
                    time.sleep(wait)
                except requests.exceptions.RequestException as exc:
                    logger.error("Transport error: %s [attempt %d/%d]",
                                 exc, attempt, max_retries)
                    if attempt == max_retries:
                        raise
                    time.sleep(min(2 ** attempt, max_backoff))
            raise RuntimeError("Max retries exceeded for cost query")
        return wrapper
    return decorator


class AzureCostIngestionClient:
    def __init__(self, scope: str, api_version: str = API_VERSION):
        self.scope = scope.strip("/")
        self.api_version = api_version
        self.credential = DefaultAzureCredential()
        self.session = requests.Session()
        self.correlation_id = str(uuid.uuid4())

    def _token(self) -> str:
        return self.credential.get_token(TOKEN_AUDIENCE).token

    @with_retries()
    def _post(self, url: str, payload: Optional[Dict]) -> Dict:
        headers = {
            "Authorization": f"Bearer {self._token()}",
            "Content-Type": "application/json",
            "x-ms-client-request-id": self.correlation_id,
        }
        resp = self.session.post(
            url, headers=headers, json=payload, timeout=REQUEST_TIMEOUT
        )
        resp.raise_for_status()
        return resp.json()

    @staticmethod
    def _project(props: Dict, scope: str) -> List[CostRecord]:
        """Map column-oriented response into canonical records by column name."""
        cols = {c["name"]: i for i, c in enumerate(props.get("columns", []))}
        cost_i = cols.get("PreTaxCost")
        date_i = cols.get("UsageDate")
        rg_i = cols.get("ResourceGroup")
        cur_i = cols.get("Currency")
        records: List[CostRecord] = []
        for row in props.get("rows", []):
            usage_date = str(row[date_i]) if date_i is not None else ""
            rg = str(row[rg_i]) if rg_i is not None else "unassigned"
            currency = str(row[cur_i]) if cur_i is not None else "USD"
            cost = Decimal(str(row[cost_i])) if cost_i is not None else Decimal("0")
            dedup_key = f"{scope}|{rg}|{usage_date}|{currency}"
            records.append(CostRecord(
                scope=scope,
                usage_date=usage_date,
                resource_group=rg,
                pre_tax_cost=cost,
                currency=currency,
                dedup_key=dedup_key,
            ))
        return records

    def build_payload(self, timeframe: str, start: Optional[str],
                      end: Optional[str]) -> Dict:
        payload = {
            "type": "ActualCost",
            "timeframe": timeframe,
            "dataset": {
                "granularity": "Daily",
                "aggregation": {
                    "totalCost": {"name": "PreTaxCost", "function": "Sum"}
                },
                "grouping": [{"type": "Dimension", "name": "ResourceGroup"}],
            },
        }
        if timeframe == "Custom" and start and end:
            payload["timePeriod"] = {"from": start, "to": end}
        return payload

    def fetch(self, timeframe: str = "BillingMonthToDate",
              start: Optional[str] = None,
              end: Optional[str] = None) -> List[CostRecord]:
        url = (f"{MANAGEMENT_BASE}/{self.scope}/providers/"
               f"Microsoft.CostManagement/query?api-version={self.api_version}")
        payload: Optional[Dict] = self.build_payload(timeframe, start, end)

        records: List[CostRecord] = []
        pages = 0
        while url:
            data = self._post(url, payload)
            props = data.get("properties", {})
            records.extend(self._project(props, self.scope))
            pages += 1
            url = props.get("nextLink")
            payload = None  # nextLink carries scope + api-version + cursor
        logger.info(
            "Ingested %d records across %d page(s) for scope %s [cid=%s]",
            len(records), pages, self.scope, self.correlation_id,
        )
        return records


if __name__ == "__main__":
    subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"]
    client = AzureCostIngestionClient(scope=f"subscriptions/{subscription_id}")
    rows = client.fetch(timeframe="BillingMonthToDate")
    print(json.dumps([r.to_json() for r in rows[:3]], indent=2))

# Schema Reference Table

The Query API response is positional; the projection step above resolves columns by name before mapping onto the unified dimensional model shared across providers.

Aligning these fields with the AWS and GCP equivalents is the job of Cross-Cloud Cost Allocation Strategies, which defines the shared taxonomy this table targets.

# Operational Considerations

• Rate limits. The Cost Management query path enforces per-scope throttling and returns 429 with a Retry-After header (commonly 30–60 seconds). Plan for serialized queries per scope rather than fan-out; parallelism across different scopes is safe, parallelism within one scope is not.
• Eventual consistency. Cost data finalizes 24–72 hours after usage. Treat the trailing three days as provisional, re-ingest them on each run, and rely on the dedup_key to overwrite provisional rows deterministically rather than appending duplicates.
• Amortized vs actual. ActualCost books a reservation purchase as a lump charge on the purchase date; AmortizedCost spreads it across the commitment term. Pick one per pipeline and document it — mixing the two double-counts commitments.
• Currency normalization. Multi-currency tenants return rows in each subscription’s billing currency. Capture the rate timestamp at ingestion; do not convert with a single end-of-month rate.
• Monitoring hooks. Emit metrics for request latency, token-acquisition duration, pagination depth, and 429 rate. Persist output as Parquet/Delta partitioned by billing_period and scope, validate schema conformity before downstream routing, and wire Azure Monitor alerts on quota consumption and service-health events.

# Troubleshooting

403 Forbidden on an otherwise valid query. Root cause: the principal’s RBAC role does not match the scope tier — for example Cost Management Reader on a subscription while querying a billingAccounts scope. Detection: the error body names the missing action (Microsoft.CostManagement/query/action). Remediation: assign Billing Reader at the billing-account scope, or drop the query down to the subscription scope the SPN actually owns.

AADSTS token-exchange failure before any cost call. Root cause: cross-tenant query without admin consent, or a stale/rotated client secret. Detection: failure occurs inside credential.get_token, not in _post. Remediation: grant directory consent for the multi-tenant app and confirm AZURE_TENANT_ID targets the resource tenant, not the home tenant.

Truncated totals / missing rows. Root cause: the nextLink cursor was dropped, or a body was resent on the continuation request and silently ignored. Detection: row count is an exact multiple of the page size. Remediation: follow nextLink verbatim with payload=None, exactly as the engine does.

KeyError / shifted columns after a quiet period. Root cause: an unpinned api-version resolved to a schema with a different column order. Detection: projection reads cost values into the date field. Remediation: pin api-version and always resolve columns through the columns name→index map rather than fixed offsets.

Persistent 429 despite backoff. Root cause: concurrent runners hitting the same scope, so each honours Retry-After but they collectively exceed the budget. Detection: 429 rate stays high while per-request latency is normal. Remediation: serialize per-scope ingestion behind a lease/lock and parallelize only across distinct scopes.

# Frequently Asked Questions

# Related

• Cloud Billing Data Ingestion & Parsing — the parent pipeline whose acquisition stage this Azure integration implements.
• Azure Cost API Pagination and Deduplication Guide — deep dive on nextLink traversal and overlapping-window deduplication.
• Handling Billing API Rate Limits & Retries — the shared retry matrix and Retry-After semantics behind the backoff decorator.
• Azure Cost Management Setup — console-level scope, billing-account, and consent configuration this page assumes.
• Cross-Cloud Cost Allocation Strategies — the unified taxonomy the schema reference table maps Azure dimensions onto.

Up: Cloud Cost Optimization & FinOps Automation · Parent reference: Cloud Billing Data Ingestion & Parsing