Backfilling Tags with boto3 Batch Operations

The specific bottleneck this page solves is retro-tagging an existing fleet — thousands of resources that predate your tagging policy and now need cost-center, owner, or environment applied after the fact. The obvious approach, a for loop calling tag_resources once per ARN, breaks on two fronts at once. First, the Resource Groups Tagging API’s TagResources call hard-caps you at 20 ARNs per request — a loop that ignores this either fails outright on lists longer than 20 or silently only tags the first 20 depending on how it was written. Second, even a correctly chunked loop that fires requests as fast as the SDK allows will eventually collide with the account’s request-rate quota and start receiving ThrottlingException, and a script with no backoff either crashes mid-run or, worse, keeps retrying in a tight loop and makes the throttling worse. Neither failure is cosmetic: a backfill job that dies at resource 4,000 of 12,000 with no record of what succeeded either has to restart from zero (re-tagging the same 4,000 resources, burning API quota for no reason) or leaves an operator guessing which resources are actually done.

This page builds the batch backfiller that sits downstream of a validation pass — the list of NON_COMPLIANT ARNs comes from the discovery and validation stages described in Automated Tag Remediation Workflows — and applies the missing tags safely, in bulk, with the ability to stop and resume without re-doing work or double-writing.

# Root Cause & Failure Modes

Three things make a naive backfill loop unsafe at fleet scale:

• The 20-ARN write ceiling. TagResources accepts a ResourceARNList of at most 20 entries per call. Anything larger is rejected outright by the API, so the chunking has to happen in your code, not left to the SDK.
• Throttling under sustained write volume. Once a backfill job settles into a steady stream of TagResources calls, the account’s request-rate quota for the tagging API becomes the limiting factor — and it tightens further if AWS Config, CloudFormation drift detection, or another remediation job is hitting the same API concurrently. A script that treats ThrottlingException as a fatal error dies partway through a multi-thousand-resource run; one that retries without backoff amplifies the throttling instead of recovering from it.
• Blind overwrite risk. A backfill’s job is to fill gaps, not to reset every resource to a fixed tag set. If the write step sends the full desired tag map to every ARN regardless of what is already there, it silently clobbers tags a resource owner set intentionally — a cost-center override, a do-not-delete flag — with the backfill’s default value. That is a correctness bug, not just a performance one, and it is the reason this backfiller reads current tags before deciding what to write.

None of these are solved by “add a try/except around the loop.” They require chunking to the API’s actual limit, exponential backoff keyed to the specific throttling error codes, and a read-before-write step that computes only the delta.

# Production Pipeline Architecture

The backfiller runs as four phases, each of which can fail and resume independently:

1. Load and filter. Read the full ARN list (typically the NON_COMPLIANT output of a validation sweep) and subtract any ARN already recorded as done in a checkpoint file from a prior run.
2. Batched read. Fetch current tags for the pending ARNs via GetResources, which accepts up to 100 ARNs per ResourceARNList — a wider ceiling than the write path, so reads and writes are chunked at different sizes.
3. Delta computation and batched write. For each 20-ARN write chunk, compute the tags each resource is actually missing against the desired set, skip resources that already have everything, and call TagResources only with the keys that are absent.
4. Checkpoint. After each write chunk — success, no-op, or dry-run — record the ARNs as done, so a crash or a manual Ctrl-C loses at most one chunk of progress on restart.

This mirrors the safety posture of the sibling workflow in Safe Tag Remediation Without Overwriting IaC State: both refuse to write a tag value the resource didn’t ask for, they just guard against different sources of unwanted overwrite — Terraform-managed state there, an operator’s manual tag here. The retry and backoff shape below is the same pattern used against a different API surface in Handling Billing API Rate Limits & Retries: treat throttling as an expected, retryable condition, never a terminal one.

# Step-by-Step Python Implementation

The module below is self-contained: it chunks reads at 100 ARNs and writes at 20, retries only on throttling-shaped errors with tenacity, computes a read-before-write delta so it never overwrites an existing value, supports --dry-run, and checkpoints after every chunk so a re-run resumes instead of restarting.

import argparse
import json
import logging
import os
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, Iterable, List, Set

import boto3
from botocore.exceptions import ClientError
from tenacity import retry, retry_if_exception, stop_after_attempt, wait_exponential

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
logger = logging.getLogger(__name__)

TAG_WRITE_CHUNK = 20   # TagResources hard cap: at most 20 ARNs per call
TAG_READ_CHUNK = 100   # GetResources hard cap: at most 100 ARNs per ResourceARNList
THROTTLE_CODES = {"ThrottlingException", "TooManyRequestsException"}


def _is_throttle(exc: BaseException) -> bool:
    return isinstance(exc, ClientError) and exc.response["Error"]["Code"] in THROTTLE_CODES


@dataclass
class BackfillCheckpoint:
    """Tracks which ARNs are already handled so a re-run resumes instead of restarting."""

    path: Path
    done: Set[str] = field(default_factory=set)

    @classmethod
    def load(cls, path: Path) -> "BackfillCheckpoint":
        if path.exists():
            data = json.loads(path.read_text())
            return cls(path=path, done=set(data.get("done", [])))
        return cls(path=path)

    def mark(self, arns: Iterable[str]) -> None:
        self.done.update(arns)
        # Rewritten whole, not appended, so a truncated write never leaves invalid JSON behind.
        self.path.write_text(json.dumps({"done": sorted(self.done)}))


def chunk(items: List[str], size: int) -> Iterable[List[str]]:
    for i in range(0, len(items), size):
        yield items[i : i + size]


@retry(retry=retry_if_exception(_is_throttle), wait=wait_exponential(multiplier=1, max=30), stop=stop_after_attempt(6))
def _get_existing_tags(client, arns: List[str]) -> Dict[str, Dict[str, str]]:
    """Read-before-write: current tags for up to 100 ARNs, so writes touch only missing keys."""
    existing: Dict[str, Dict[str, str]] = {}
    paginator = client.get_paginator("get_resources")
    for page in paginator.paginate(ResourceARNList=arns):
        for mapping in page["ResourceTagMappingList"]:
            existing[mapping["ResourceARN"]] = {t["Key"]: t["Value"] for t in mapping["Tags"]}
    return existing


@retry(retry=retry_if_exception(_is_throttle), wait=wait_exponential(multiplier=1, max=30), stop=stop_after_attempt(6))
def _tag_resources(client, arns: List[str], tags: Dict[str, str]) -> None:
    resp = client.tag_resources(ResourceARNList=arns, Tags=tags)
    failed = resp.get("FailedResourcesMap", {})
    if failed:
        # TagResources reports partial failures in the body instead of raising —
        # surface them explicitly so they aren't silently checkpointed as done.
        for arn, detail in failed.items():
            logger.error("Failed to tag %s: %s (%s)", arn, detail.get("ErrorMessage"), detail.get("ErrorCode"))
        raise RuntimeError(f"{len(failed)} ARNs failed within batch")


def backfill(arns: List[str], desired_tags: Dict[str, str], client, checkpoint: BackfillCheckpoint, dry_run: bool) -> int:
    pending = [a for a in arns if a not in checkpoint.done]
    logger.info("%d ARNs pending, %d already checkpointed as done", len(pending), len(checkpoint.done))
    applied = 0
    for read_batch in chunk(pending, TAG_READ_CHUNK):
        existing = _get_existing_tags(client, read_batch)
        for write_batch in chunk(read_batch, TAG_WRITE_CHUNK):
            targets: List[str] = []
            batch_missing: Dict[str, str] = {}
            for arn in write_batch:
                current = existing.get(arn, {})
                missing = {k: v for k, v in desired_tags.items() if k not in current}
                if missing:
                    targets.append(arn)
                    batch_missing.update(missing)  # union is safe: desired_tags is constant for the run
            if not targets:
                checkpoint.mark(write_batch)  # already fully tagged — still checkpoint, so it's never re-read
                continue
            if dry_run:
                logger.info("[dry-run] would apply %s to %d resource(s)", list(batch_missing), len(targets))
            else:
                _tag_resources(client, targets, batch_missing)
                applied += len(targets)
            checkpoint.mark(write_batch)
    return applied


def main() -> None:
    parser = argparse.ArgumentParser(description="Backfill missing tags across a batch of ARNs.")
    parser.add_argument("--arns-file", required=True, help="Newline-delimited file of resource ARNs to backfill")
    parser.add_argument("--tags", required=True, help='JSON object of tags to fill in, e.g. \'{"cost-center":"unassigned"}\'')
    parser.add_argument("--checkpoint-file", default="backfill_checkpoint.json")
    parser.add_argument("--dry-run", action="store_true", help="Log intended writes without calling TagResources")
    parser.add_argument("--region", default=os.getenv("AWS_REGION", "us-east-1"))
    args = parser.parse_args()

    arns = [line.strip() for line in Path(args.arns_file).read_text().splitlines() if line.strip()]
    desired_tags = json.loads(args.tags)
    checkpoint = BackfillCheckpoint.load(Path(args.checkpoint_file))
    client = boto3.client("resourcegroupstaggingapi", region_name=args.region)

    applied = backfill(arns, desired_tags, client, checkpoint, args.dry_run)
    logger.info("Backfill complete: %d resource(s) newly tagged, %d total checkpointed.", applied, len(checkpoint.done))


if __name__ == "__main__":
    main()

The two _get_existing_tags and _tag_resources calls are the only network boundaries, and both carry the same tenacity retry: six attempts with exponential backoff capped at 30 seconds, triggered only on ThrottlingException or TooManyRequestsException so a real permissions or validation error still fails fast instead of retrying uselessly six times.

# Verification & Testing

• Dry-run diff first. Always run with --dry-run against the full ARN list before the real pass and inspect the logged would apply lines. A backfill that shows more targets than expected usually means desired_tags contains a key that overlaps a value already in wide use — check the read step’s output before writing.
• Idempotency assertion. Run the script twice in a row without deleting the checkpoint. The second run should log zero newly-tagged resources — every ARN is either checkpointed or already carries every desired key. If the second run tags anything, the delta computation or the checkpoint write has a bug.
• Spot-check via GetResources post-run. Query a random sample of the ARNs with TagFilters set to the backfilled keys and confirm every sampled resource now returns them, and that any pre-existing tag values on those resources are unchanged from a pre-run snapshot.
• Partial-failure replay. Kill the process mid-run (kill -9 the process, not a graceful signal) and restart it. The checkpoint file should let it resume from the last completed chunk with no duplicate TagResources calls against already-tagged ARNs.

# Common Pitfalls Checklist

• Sending the full desired tag map to every ARN. Skips the read step and overwrites operator-set values — always compute missing per-resource against the current tags before writing.
• Chunking writes at more than 20 ARNs. TagResources rejects the call outright — chunk with TAG_WRITE_CHUNK = 20, and don’t reuse the same chunk size for the read path, which allows up to 100.
• Treating every ClientError as retryable. Retrying an AccessDeniedException or a malformed-ARN error six times just delays a failure that will never succeed — scope the retry predicate to throttling codes only, as _is_throttle does here.
• Checkpointing before the write confirms success. Marking a chunk done before _tag_resources returns means a mid-call crash looks like a completed backfill on resume — checkpoint only after the call (or dry-run log) completes.
• Ignoring FailedResourcesMap. A TagResources call can return 200 OK while individual ARNs inside the batch failed — always inspect the response body, not just exceptions, or those resources get silently checkpointed as tagged when they weren’t.

# Frequently Asked Questions

# Related

• Automated Tag Remediation Workflows — the event-driven remediation layer this batch backfiller complements for one-off or scheduled retro-tagging sweeps.
• Safe Tag Remediation Without Overwriting IaC State — the sibling guard that prevents remediation writes from colliding with Terraform-managed tag values.
• Handling Billing API Rate Limits & Retries — the same exponential-backoff-on-throttling pattern applied to billing APIs instead of the tagging API.
• Resource Tagging & Validation Pipelines — the four-stage discovery-to-remediation architecture this backfiller’s ARN input comes from.

Up: Automated Tag Remediation Workflows · Home: Cloud Cost Optimization & FinOps Automation