How to Review Terraform Plans Without Destroying Production: 10 Critical Traps & Automated Guardrails

1. The 2 AM Disaster: Anatomy of an Unplanned Database Drop

It is 2:15 AM on a Wednesday. An on-call engineer pushes a seemingly benign pull request to update the database naming scheme to match the new company domain standard. The pull request title reads: "chore: align rds naming with core conventions".

The reviewer glances at the pull request diff, notes that only one string changed in HCL, and hits approve. The automated CI/CD pipeline triggers. Thirty seconds later, the company Slack channels explode: customer authentication fails, checkout services throw 500 errors, and the main PostgreSQL database instance is gone.

This is not a hypothetical horror story. Skimming the output of terraform plan is the single most common cause of self-inflicted production outages across enterprise cloud infrastructure.

2. Decoding the CLI Symbols: What Really Triggers Destruction

Terraform communicates intent through three primary symbols and one compound operator. Understanding the exact mechanical implication of each symbol is the first line of defense:

• + (Create): Terraform will create a brand new resource. Usually safe, unless it collides with existing unmanaged cloud resources or exhausts account quotas.
• ~ (Update in-place): Terraform modifies attributes on the live resource via an API Update call. Safe for tags, alarm thresholds, and non-structural configurations.
• - (Destroy): Terraform deletes the resource entirely. If attached to persistent storage, data will be lost unless backups or retention policies exist.
• -/+ (Destroy and then create / Replacement): The most dangerous symbol in Infrastructure as Code. Modifying an immutable attribute forces Terraform to tear down the existing resource before spinning up a replacement.

# An unmistakable red flag in any pull request:
-/+ resource "aws_db_instance" "primary" {
      ~ identifier = "core-prod" -> "core-production" # forces replacement
    }

Plan: 1 to add, 0 to change, 1 to destroy.

3. The 5 Deadliest Terraform Traps in Production

Trap 1: The RDS Database Identifier Trap

As highlighted above, changing the identifier attribute on aws_db_instance forces replacement. If you need dynamic naming, always use identifier_prefix, or perform a manual blue/green read replica promotion before updating state.

Trap 2: Refactoring count to for_each Without State Migration

Converting a module from integer indexing (count) to string map keys (for_each) changes every resource address in the state file:

# What Terraform plans without state migration:
- aws_instance.web[0] (destroyed)
- aws_instance.web[1] (destroyed)
+ aws_instance.web["api"] (created)
+ aws_instance.web["auth"] (created)

Terraform does not realize you are merely organizing existing instances. It schedules the destruction of all running instances simultaneously.

Trap 3: Subnet CIDR Block Shifts

You cannot change an existing AWS VPC subnet CIDR block in-place. If an engineer modifies cidr_block = "10.0.1.0/24" to "10.0.10.0/24", Terraform will attempt to delete the subnet. The AWS API will either block the call with a dependency error or terminate attached Elastic Network Interfaces (ENIs), severing Kubernetes worker nodes from the control plane.

Trap 4: Inline vs Standalone Security Group Rules Collision

Terraform allows declaring firewall rules either as inline blocks inside aws_security_group or as standalone aws_security_group_rule resources. If a developer touches the security group resource and leaves an empty ingress = [] block, Terraform will flush all standalone rules created elsewhere, closing public HTTPS ports instantly.

Trap 5: Accidental force_destroy = true on S3 Buckets

By default, AWS prevents deleting an S3 bucket that contains objects. Adding force_destroy = true overrides this safety mechanism, enabling Terraform to recursively purge millions of production documents and backups in a single command.

4. Declarative moved {} Blocks: Zero-Downtime State Refactoring

Prior to Terraform 1.1, fixing state address mismatches required running dangerous manual CLI commands like terraform state mv aws_instance.web[0] aws_instance.web["api"] directly against remote state files.

Modern Terraform solves this declaratively through moved blocks committed directly into version control:

# Safe refactoring from count to for_each without downtime:
moved {
  from = aws_instance.web[0]
  to   = aws_instance.web["api"]
}

moved {
  from = aws_instance.web[1]
  to   = aws_instance.web["auth"]
}

When Terraform encounters these blocks, it automatically maps the existing cloud resource IDs to the new addresses in memory. The plan output will read: Plan: 0 to add, 0 to change, 0 to destroy, completely eliminating downtime.

5. Architectural Guardrails: lifecycle { prevent_destroy = true } & OPA

Never rely exclusively on human reviewers during code reviews. Implement code-level and CI/CD policy-as-code guardrails:

resource "aws_db_instance" "production_core" {
  identifier        = "prod-core-db"
  allocated_storage = 500
  instance_class    = "db.r6g.2xlarge"
  engine            = "postgres"

  # Absolute safeguard against accidental drops:
  lifecycle {
    prevent_destroy = true
  }
}

When prevent_destroy = true is present, Terraform will reject any plan that requires destroying or replacing the resource with a fatal error:

Error: Instance cannot be destroyed
Resource aws_db_instance.production_core has lifecycle.prevent_destroy set, but the plan calls for destruction.

6. Production PR Review Runbook: The 7-Point Checklist

Adopt this standard checklist in your engineering team's GitHub Pull Request templates:

1. Check the Plan Summary Line: Look at Plan: X to add, Y to change, Z to destroy. If Z > 0, demand an architectural justification.
2. Search for Replacement Triggers: Run a text search for forces replacement and -/+. If present on any database, network, or IAM resource, reject the PR immediately unless downtime is scheduled.
3. Verify State Address Continuity: If module names or loop constructs changed, confirm that corresponding moved {} blocks are included.
4. Inspect Network Boundaries: Ensure Route Tables, NAT Gateways, and Subnet CIDRs are not mutated in-place.
5. Verify Target Workspace & Environment: Double check that the plan was executed against the correct workspace (e.g., production vs staging).
6. Enforce prevent_destroy on State: Verify that databases, KMS keys, and S3 buckets maintain explicit lifecycle locks.
7. Ensure Automated CI/CD Execution: Never accept manual terminal outputs pasted into PR descriptions. Plans must be executed by trusted CI/CD runner environments.

8. Staff SRE Interview Scenarios: How to Answer Terraform Drift & Outages

In Senior and Staff DevOps interviews, interviewers frequently present candidates with ambiguous Terraform failure scenarios. Here is how top practitioners structure their answers:

• Scenario A: terraform plan shows unexpected changes to resources you didn't touch:
  Read our verified model answer in the handbook: terraform plan Shows Unexpected Changes — Investigation Steps →
• Scenario B: Someone manually modified AWS resources in the web console:
  Read the production remediation runbook: Someone Manually Changes Terraform Infrastructure — What Happens? →
• Scenario C: Two engineers ran terraform apply simultaneously on the same state:
  Explore state locking architecture: Two Developers Ran Terraform Apply on the Same Workspace →

9. Frequently Asked Questions