Autoscaling group tags are destroyed and recreated for an update in-place

[ghost]

This issue was originally opened by @Lynty as hashicorp/terraform#25949. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Hello,
I have a Service Control Policy configured in an AWS account that denies creation and updating of an autoscaling group if it does not have some mandatory tags. I am using this AWS example to accomplish this. It uses a Null condition operator to check if a condition key is present at the time of authorization. If it is true (the key doesn't exist — it is null), the SCP prevents resource creation.

The issue I am coming across is that regardless of a valid or invalid change to a tag, a terraform plan output shows a change in place for the autoscaling group resource but actually destroys and recreates the changed tag. A terraform apply will trigger the SCP and deny updating the autoscaling group due to the tag being momentarily destroyed/marked as null.

Terraform v0.12.24
+ provider.aws v2.52.0

Terraform Configuration Files

resource "aws_autoscaling_group" "webserver" {
  name_prefix               = "webserver"
  max_size                  = 1
  min_size                  = 0
  desired_capacity          = 1
  health_check_grace_period = 300
  health_check_type         = "EC2"
  force_delete              = true

  lifecycle {
    create_before_destroy = true
  }

  launch_configuration = aws_launch_configuration.webserver.name
  # this will become the private_subnets once we have a bastion
  vpc_zone_identifier = var.public_subnets

  tag {
    key                 = "environment"
    value               = var.env
    propagate_at_launch = "true"
  }
  tag {
    key                 = "project"
    value               = "mvp"
    propagate_at_launch = "true"
  }
  tag {
    key                 = "Name"
    value               = "${terraform.workspace}-${var.env}-webserver"
    propagate_at_launch = "true"
  }
  tag {
    key                 = "workspace"
    value               = terraform.workspace
    propagate_at_launch = "true"
  }
  tag {
    key                 = "component"
    value               = "webserver"
    propagate_at_launch = "true"
  }
  tag {
    key                 = "terraform"
    value               = "True"
    propagate_at_launch = "true"
  }
}

Service Control Policy Document

data "aws_iam_policy_document" "scp_mandatory_ec2_tags" {
  version = "2012-10-17"
  statement {
    sid = "DenyAutoscalingGroupWithNoEnvironmentTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/environment"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoscalingGroupWithNoProjectTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/project"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoscalingGroupWithNoNameTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/Name"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoscalingGroupWithNoComponentTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/component"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoscalingGroupWithNoWorkspaceTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/workspace"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoscalingGroupWithNoGxPTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/gxp"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoscalingGroupWithNoTerraformTag"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "Null"
      variable = "aws:RequestTag/terraform"

      values = [
        "true",
      ]
    }
  }

  statement {
    sid = "DenyAutoScalingGroupWithoutGxPBoolean"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "StringNotEquals"
      variable = "aws:RequestTag/gxp"

      values = [
        "True",
        "False"
      ]
    }
  }

  statement {
    sid = "DenyAutoScalingGroupWithoutTerraformBoolean"

    actions = [
      "autoscaling:CreateAutoScalingGroup",
    ]

    effect = "Deny"

    resources = [
      "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*",
    ]

    condition {
      test     = "StringNotEquals"
      variable = "aws:RequestTag/terraform"

      values = [
        "True",
        "False"
      ]
    }
  }
}

Expected Behavior

environment is a mandatory tag. I should spin up an ec2 instance in an autoscaling group with tag environment:dev. While the instance is running, I should be able to change the tag value to something like environment:stage without triggering the Service Control Policy with terraform.

Actual Behavior

SCP is triggered and blocks autoscaling resource update due to seeing a tag being destroyed/marked as null.
terraform plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_autoscaling_group.webserver: Refreshing state... [id=webserver20200820173423399100000001]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_autoscaling_group.webserver will be updated in-place
  ~ resource "aws_autoscaling_group" "webserver" {
        arn                       = "arn:aws:autoscaling:us-west-2:104568625881:autoScalingGroup:624943a4-2ab9-4585-90ca-c30a185c14d3:autoScalingGroupName/webserver20200820173423399100000001"
        availability_zones        = [
            "us-west-2a",
            "us-west-2b",
        ]
        default_cooldown          = 300
        desired_capacity          = 1
        enabled_metrics           = []
        force_delete              = true
        health_check_grace_period = 300
        health_check_type         = "EC2"
        id                        = "webserver20200820173423399100000001"
        launch_configuration      = "webserver20200819192444336200000002"
        load_balancers            = []
        max_instance_lifetime     = 0
        max_size                  = 1
        metrics_granularity       = "1Minute"
        min_size                  = 0
        name                      = "webserver20200820173423399100000001"
        name_prefix               = "webserver"
        protect_from_scale_in     = false
        service_linked_role_arn   = "arn:aws:iam::104568625881:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        suspended_processes       = []
        target_group_arns         = []
        termination_policies      = []
        vpc_zone_identifier       = [
            "subnet-0cac79a2fb0d14a70",
            "subnet-0d9d17f1e0c6f5eba",
        ]
        wait_for_capacity_timeout = "10m"

        tag {
            key                 = "Name"
            propagate_at_launch = true
            value               = "ldong5-opsdev-webserver"
        }
        tag {
            key                 = "component"
            propagate_at_launch = true
            value               = "webserver"
        }
      - tag {
          - key                 = "environment" -> null
          - propagate_at_launch = true -> null
          - value               = "opsdev" -> null
        }
      + tag {
          + key                 = "environment"
          + propagate_at_launch = true
          + value               = "test"
        }
        tag {
            key                 = "gxp"
            propagate_at_launch = true
            value               = "False"
        }
        tag {
            key                 = "project"
            propagate_at_launch = true
            value               = "mvp"
        }
        tag {
            key                 = "terraform"
            propagate_at_launch = true
            value               = "True"
        }
        tag {
            key                 = "workspace"
            propagate_at_launch = true
            value               = "ldong5"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Steps to Reproduce

1. terraform init
2. terraform apply
3. Change a tag value to a different valid value
4. terraform apply

[ewbankkit]

Hmm. The AWS EC2 Auto Scaling API has a method to create or update tags but our code

https://github.com/terraform-providers/terraform-provider-aws/blob/51f8bae0d4eee82dc79bf3236d26cbcd93b6d560/aws/autoscaling_tags.go#L133-L162

isn't so subtle and for updated tags will delete the tag and then create it again.

The current code is being refactored in #13868 to allow Auto Scaling tags to be handled the same as other service tags and it looks like this PR should fix this issue by updating the tag in-place (@bflad can you confirm?).

[bflad]

Ha yeah, that was going to be my comment here. I believe #13868 will likely get this much closer to the desired behavior since its standardized and if it doesn't we should re-review things afterwards. 👍

[bflad]

The fix for this should be merged now and will release with version 3.5.0 of the Terraform AWS Provider, later this week. If its still causing trouble after that release, we can re-evaluate with the newer logic.

[ghost]

This has been released in version 3.5.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!