Safe Tag Remediation Without Overwriting IaC State

The specific failure this page solves is the drift loop: a remediation worker sees a resource missing cost_center, calls tag:TagResources to add it, and the fix appears to work — until the next terraform apply runs a plan against state that never knew the key existed, sees a tag present on the real resource that isn’t in the .tf config, and either reverts it (if the resource’s tags block is exhaustively managed) or leaves it alone only by accident. If the module manages tags as a complete map rather than a merge, the very next apply deletes the key the remediator just added. The remediator’s next sweep sees it missing again and reapplies it. Two automated systems now fight over the same field forever, each one “fixing” what the other considers correct, and every cycle shows up as a spurious change in the apply plan that erodes trust in both systems. This is the mutation-safety gap in the Automated Tag Remediation Workflows worker: routing a verdict to apply-missing is not sufficient on its own — the worker also has to know which keys it is allowed to own.

Root Cause & Failure Modes

Terraform and CloudFormation both compute drift by diffing the desired state in configuration against the actual state of the live resource, and Terraform additionally diffs desired state against what it last wrote into its own state file. A tag added out-of-band — by a remediator, by console click-ops, by any process outside the IaC pipeline — is invisible to both diffs until the next plan runs, at which point it becomes a mismatch that one side “resolves.” Three specific configuration shapes turn that mismatch into a recurring loop rather than a one-time correction:

  • Exhaustive tags = {} blocks. When a resource’s tags are set as a complete map literal, Terraform treats anything not in that map as drift to be removed. A remediator-added cost_center gets deleted on the next apply, and the remediation sweep re-adds it on its next run. The fight repeats every deploy cycle indefinitely.
  • ignore_changes scoped too narrowly. Some modules add lifecycle { ignore_changes = [tags["Name"]] } for one key, correctly excluding it from drift detection, but leave every other key exhaustively managed. A remediator that assumes “this module ignores tag drift” because of one ignored key will still get reverted on every other key it touches.
  • CloudFormation stacks with DeletionPolicy and drift detection disabled. Without scheduled drift detection, CloudFormation never surfaces the mismatch automatically, but the next stack update still overwrites the resource’s tags wholesale from the template — so the loop is silent until a deploy, then it reverts everything at once, which is harder to trace back to the remediator than Terraform’s per-apply plan diff.

The number that matters operationally: every one of these loops costs a full remediation-to-revert cycle per deploy, and on a team deploying multiple times a day that is not an occasional annoyance, it is a permanent, compounding source of noisy terraform plan diffs that makes real drift harder to spot in the noise. The fix has to happen before the write, not after — the remediator needs to know, per key, whether IaC considers itself the owner.

Production Pipeline Architecture

There are three valid patterns for keeping remediation and IaC from fighting, and they are not mutually exclusive — most estates run the first two together and reserve the third for keys that genuinely need to originate from a human-reviewed change.

  1. Add only non-managed keys. The remediator maintains an explicit allowlist or a discovered set of IaC-managed keys per resource type (or reads it from a managed_keys annotation the IaC pipeline itself publishes) and only ever writes keys outside that set. This is the pattern implemented below: compute the delta between what a resource is missing and what IaC owns, and apply only the intersection that belongs to neither the current tag set nor the managed set.
  2. Exclude IaC-managed keys structurally. Where the IaC tooling supports it, scope every module’s tags argument to a merge() of a base map plus ignore_changes on any key the remediator might touch, so drift detection on those specific keys is structurally impossible rather than avoided by convention. This is the durable long-term fix and the one covered in depth in Enforcing Tag Policies in Terraform, which handles the enforcement side of the same boundary this page handles from the remediation side.
  3. Write back to the source instead of the resource. For estates where every resource is provisioned through IaC and console mutation is disallowed entirely, the correct remediation target is not the cloud API at all — it is a pull request against the IaC repository that adds the missing tag to the .tf or template source, reviewed and merged like any other change. This trades immediacy for correctness: the fix lands only after the next apply, but it can never drift because it is the desired state.

All three patterns share the same prerequisite: the remediator must be able to answer “does IaC own this key on this resource?” before it writes anything. The Automated Tag Remediation Workflows worker’s apply-missing action already reads current tags before writing; safe remediation adds a second read — the managed-key set — and intersects both checks before computing what to apply. This is a narrower, more conservative version of the estate-wide backfill covered in Backfilling Tags with boto3 Batch Operations: that page’s batch job assumes the target keys are safe to add across an entire resource population, while this page’s per-resource delta assumes nothing until it has checked.

The Terraform drift loop versus a safe-delta remediation write Two panels compare tag remediation approaches. The left panel shows the broken loop: a remediator calls TagResources to add a missing key, the next terraform apply diffs actual state against an exhaustive tags map, sees an unrecognized key, and reverts it, after which the remediation sweep detects the key missing again and reapplies it, forming an endless cycle drawn as a circular arrow. The right panel shows the fix: the remediator reads both the current tag set and a set of IaC-managed keys, computes a safe delta containing only keys that are neither present nor IaC-owned, and applies just that delta, so the next terraform apply finds no unrecognized key and the loop never starts. BROKEN: write direct to resource Remediator TagResources Live resource tag now present terraform apply diffs exhaustive tags map key reverted unrecognized key removed next sweep sees key missing again ENDLESS LOOP one revert-and-reapply cycle per deploy, forever CORRECT: safe-delta write read current_tags(resource) read iac_managed_keys(resource) safe_delta = desired − current − iac_managed TagResources(safe_delta only) skipped keys logged, not written next terraform apply finds no unrecognized key loop never starts
Writing straight to the resource races the next terraform apply and creates an endless revert-and-reapply cycle. Computing a safe delta that excludes IaC-managed keys before writing avoids the race entirely.

Step-by-Step Python Implementation

The module below extends the read-before-write check from the parent remediation worker with a second read: the set of tag keys a given resource’s IaC module already manages. It computes the safe delta — keys that are neither present on the live resource nor claimed by IaC — applies only that delta, and logs every key it skips along with the reason, so an audit of a --dry-run pass tells you exactly what would happen and why.

import argparse
import logging
from dataclasses import dataclass, field
from typing import Dict, Set

import boto3
from botocore.exceptions import ClientError

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%SZ",
)
logger = logging.getLogger("finops.safe_tag_remediator")


@dataclass
class RemediationPlan:
    """The outcome of computing a safe delta for one resource."""

    resource_arn: str
    safe_to_apply: Dict[str, str] = field(default_factory=dict)
    skipped_already_present: Set[str] = field(default_factory=set)
    skipped_iac_managed: Set[str] = field(default_factory=set)

    @property
    def has_work(self) -> bool:
        return bool(self.safe_to_apply)


class SafeTagRemediator:
    """Applies only tag keys that IaC does not own and the resource does not
    already carry, so a remediation write can never be reverted by the next
    terraform apply or CloudFormation stack update."""

    def __init__(self, dry_run: bool = False) -> None:
        self.dry_run = dry_run
        self.tagging = boto3.client("resourcegroupstaggingapi")

    def current_tags(self, resource_arn: str) -> Dict[str, str]:
        resp = self.tagging.get_resources(ResourceARNList=[resource_arn])
        tags: Dict[str, str] = {}
        for mapping in resp.get("ResourceTagMappingList", []):
            for tag in mapping.get("Tags", []):
                tags[tag["Key"]] = tag["Value"]
        return tags

    def compute_delta(
        self,
        resource_arn: str,
        desired_tags: Dict[str, str],
        iac_managed_keys: Set[str],
    ) -> RemediationPlan:
        """The core safety rule: a key is safe to apply only if it is absent
        from current state AND not claimed by IaC. Either condition alone is
        not enough — a key IaC does not manage yet but a human just set by
        hand is still 'already present' and must not be overwritten, and a
        key that is genuinely missing but IaC-managed must be left for the
        source-of-truth pipeline to add, not patched in directly."""
        current = self.current_tags(resource_arn)
        plan = RemediationPlan(resource_arn=resource_arn)

        for key, value in desired_tags.items():
            if key in current:
                plan.skipped_already_present.add(key)
                continue
            if key in iac_managed_keys:
                plan.skipped_iac_managed.add(key)
                continue
            plan.safe_to_apply[key] = value

        return plan

    def apply(self, plan: RemediationPlan) -> None:
        if not plan.has_work:
            logger.info(
                "resource=%s no safe keys to apply (present=%s, iac_managed=%s)",
                plan.resource_arn,
                sorted(plan.skipped_already_present),
                sorted(plan.skipped_iac_managed),
            )
            return

        for key in sorted(plan.skipped_already_present):
            logger.info("resource=%s skip key=%s reason=already_present", plan.resource_arn, key)
        for key in sorted(plan.skipped_iac_managed):
            logger.info("resource=%s skip key=%s reason=iac_managed", plan.resource_arn, key)

        if self.dry_run:
            logger.info("resource=%s [dry-run] would apply=%s", plan.resource_arn, plan.safe_to_apply)
            return

        try:
            self.tagging.tag_resources(
                ResourceARNList=[plan.resource_arn],
                Tags=plan.safe_to_apply,
            )
            logger.info("resource=%s applied=%s", plan.resource_arn, plan.safe_to_apply)
        except ClientError as exc:
            logger.error("resource=%s tag_resources failed: %s", plan.resource_arn, exc)
            raise


def main() -> None:
    parser = argparse.ArgumentParser(description="Safe tag remediation excluding IaC-managed keys")
    parser.add_argument("--resource-arn", required=True)
    parser.add_argument("--dry-run", action="store_true", help="Log the plan without writing")
    args = parser.parse_args()

    # In production, desired_tags comes from the validation verdict and
    # iac_managed_keys comes from a manifest the IaC pipeline publishes
    # (e.g. a `managed_keys` output written alongside each module's state).
    desired_tags = {"cost_center": "cc-4471", "environment": "prod", "owner": "platform-team"}
    iac_managed_keys = {"environment", "Name"}

    remediator = SafeTagRemediator(dry_run=args.dry_run)
    plan = remediator.compute_delta(args.resource_arn, desired_tags, iac_managed_keys)
    remediator.apply(plan)


if __name__ == "__main__":
    main()

The design choice that matters most: compute_delta never sees a boolean “safe or not” for the whole resource, only a per-key classification. A resource can simultaneously have one key that’s already correct, one that IaC owns and hasn’t set yet, and one that’s genuinely safe to backfill — treating the whole write as atomic-or-nothing would either block the legitimately safe key or risk the IaC-owned one.

Verification & Testing

  • Dry-run against a known-drifting resource. Pick a resource you know has an ignore_changes-scoped key plus a genuinely unmanaged gap, run with --dry-run, and confirm the managed key shows up under skipped_iac_managed while only the unmanaged gap appears in safe_to_apply.
  • Post-apply plan check. After a live run, immediately run terraform plan against the affected module. A clean plan (no changes) confirms the delta computation correctly excluded every key Terraform considers its own; any diff means the managed-key set is incomplete.
  • Idempotency assertion. Run the remediator twice against the same resource with no state change in between. The second run’s safe_to_apply should be empty, since every key it could safely add is now already present.
  • Managed-key manifest freshness. Assert the iac_managed_keys source (a manifest, a tag on the module, or a state query) was refreshed after the module’s last apply — a stale manifest is the most common way a “safe” delta accidentally includes a key IaC actually owns now.

Common Pitfalls Checklist

  • Trusting a stale managed-keys manifest. If it’s generated at deploy time and never refreshed, a module change that starts managing a new key silently reopens the drift loop — regenerate the manifest on every apply, not on a schedule.
  • Treating “resource has a managed-by: terraform marker tag” as blanket coverage. That marker says the resource exists under IaC, not which specific keys it manages — always check the per-key set, never infer safety from a single marker.
  • Applying the whole desired-tags map when only part of it is safe. Skipping the per-key delta and writing everything the verdict proposed reintroduces the loop for whichever keys happened to be IaC-managed — always intersect against both current state and the managed-key set.
  • Forgetting that PR-based remediation has latency. Writing back to the IaC repository is drift-proof but not instant; don’t route time-sensitive compliance fixes (like a security-classification tag) through a PR queue that might sit unreviewed for days.

Frequently Asked Questions

Why not just add ignore_changes for every tag key IaC manages and skip the delta computation entirely?

ignore_changes prevents Terraform from reverting a key once it’s outside the config, but it does nothing to stop the remediator from writing to a key IaC still actively sets — you’d get a live resource whose tag value never matches what’s in .tf, which is a quieter but equally real correctness problem. The delta computation and ignore_changes solve two different halves of the same boundary and work best combined, as described in Enforcing Tag Policies in Terraform.

How do I generate the iac_managed_keys set without hand-maintaining a list?

The most reliable source is the IaC pipeline itself: have each module emit its tags map (or its keys) as a Terraform output, and publish that output to a small lookup table keyed by resource ARN or module path during every successful apply. Falling back to static per-resource-type allowlists works but drifts out of sync with actual module changes faster than a pipeline-generated manifest does.

Is writing back to the IaC repository ever the better default over a direct safe-delta write?

Yes, for any estate where console or API mutation outside IaC is disallowed by policy — in that world the safe-delta write is itself a policy violation regardless of correctness. In estates that tolerate limited out-of-band tagging for cost-allocation keys specifically, the direct safe-delta write is faster to land and just as drift-proof, since it never touches IaC-owned keys in the first place.

What happens if a key is missing from both current tags and the IaC-managed set, but IaC starts managing it next week?

The remediator’s write becomes the value already in the live resource by the time IaC picks it up, so the next apply either matches (no diff) if the config’s default equals what the remediator wrote, or shows a one-time, intentional diff if it doesn’t — which is a normal, single-cycle config change, not a loop, because after that apply the key is now correctly excluded from future safe-delta computations via a refreshed manifest.

Up: Automated Tag Remediation Workflows