Global Tags - Note: Objects have changed outside of Terraform every run

[ashtonian]

Description

Currently getting a change every run even though nothing is changing. I think this is an issue with setting the global tags on the provider.

Versions

• Terraform:
  Terraform v1.0.6
  on darwin_amd64
• Provider(s):

+ provider registry.terraform.io/hashicorp/aws v3.57.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.3.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.4.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Set global tags on provider, setup eks, run tf apply.

no

yes

tf apply

Code Snippet to Reproduce

# Configure the AWS Provider
provider "aws" {
  region = local.region

  // This is necessary so that tags required for eks can be applied to the vpc without changes to the vpc wiping them out.
  // https://stackoverflow.com/questions/57495581/terraform-eks-tagging
  // https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/resource-tagging
  ignore_tags {
    key_prefixes = ["kubernetes.io/", "k8s.io/"]
    keys         = concat(keys(local.global_tags), [])

  }
  default_tags {
    // https://github.com/hashicorp/terraform-provider-aws/issues/18311#issuecomment-860868546
    tags = local.global_tags
  }
}
locals{
  global_tags = {
    Environment  = local.environment
    TF_Timestamp = timestamp()
    TF_state     = "us-east-1:s3:stateblah"
    TF_lock      = "us-east-1:dynamo:keyblah"
    TF_source    = "https://github.com/blah"
  }
}


module "eks" {
  // https://github.com/terraform-aws-modules/terraform-aws-eks
  source          = "terraform-aws-modules/eks/aws"
  version         = "17.18.0"
  cluster_name    = local.cluster_name
  cluster_version = var.version_eks
  subnets         = var.private_subnets
  enable_irsa     = true
  # tags = {
  #   Name = local.cluster_name
  # }

  vpc_id = var.vpc_id

  workers_group_defaults = {
    root_volume_type     = "gp2"
    bootstrap_extra_args = "--enable-docker-bridge true"
  }

  # valid keys and defaults - https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/local.tf#L50
  worker_groups_launch_template = [
    {
      name               = local.worker_group_1_name
      launch_template_id = local.worker_group_1_template_id
      # ami_id                               = var.ami_id
      asg_desired_capacity                 = var.asg_capacity_desired
      asg_max_size                         = var.asg_capacity_max
      asg_min_size                         = var.asg_capacity_min
      key_name                             = var.key_name
      instance_type                        = var.instance_type
      additional_userdata                  = "echo foo bar"
      additional_security_group_ids        = [aws_security_group.main-node.id]
      metadata_http_put_response_hop_limit = 3 # required for node terminator to work
      update_default_version               = true
      instance_refresh_enabled             = true
      asg_recreate_on_change               = true
      capacity_rebalance                   = true
      root_volume_size                     = "30"
      root_volume_type                     = "gp3"
      max_instance_lifetime                = 86400 // 30 days = 2592000 // TODO: test & tune
      instance_refresh_triggers            = ["tag"]
      tags = [
        {
          key                 = "aws-node-termination-handler/managed" # used by termination handler
          value               = ""
          propagate_at_launch = true
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/enabled"
          "propagate_at_launch" = false //TODO: test true ?
          "value"               = "true"
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/${local.cluster_name}"
          "propagate_at_launch" = false
          "value"               = "owned"
        }
      ],
      // https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules/node_groups
      node_groups = {
        "${local.worker_group_1_name}" = {
          launch_template_id = local.worker_group_1_template_id
          # taints =
        }
      }
    },
    # {
    #   name                    = "spot-1"
    #   override_instance_types = ["m5.large", "m5a.large", "m5d.large", "m5ad.large"]
    #   spot_instance_pools     = 4
    #   asg_max_size            = 5
    #   asg_desired_capacity    = 5
    #   kubelet_extra_args      = "--node-labels=node.kubernetes.io/lifecycle=spot"
    #   public_ip               = true
    # },
  ]

  # worker_additional_security_group_ids = [aws_security_group.all_worker_mgmt.id]
  map_roles = local.map_roles
  map_users = local.map_users
  # map_accounts                         = var.map_accounts
}

Expected behavior

Apply only to run when there are changes.

Actual behavior

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.eks.module.eks.aws_autoscaling_group.workers_launch_template[0] has been changed
  ~ resource "aws_autoscaling_group" "workers_launch_template" {
        id                        = "lattice-dev-platform20210909075419913700000004"
        name                      = "lattice-dev-platform20210909075419913700000004"
        # (23 unchanged attributes hidden)



      - tag {
          - key                 = "k8s.io/cluster-autoscaler/enabled" -> null
          - propagate_at_launch = false -> null
          - value               = "true" -> null
        }
      - tag {
          - key                 = "k8s.io/cluster-autoscaler/lattice-dev" -> null
          - propagate_at_launch = false -> null
          - value               = "owned" -> null
        }
      - tag {
          - key                 = "kubernetes.io/cluster/lattice-dev" -> null
          - propagate_at_launch = true -> null
          - value               = "owned" -> null
        }
        # (4 unchanged blocks hidden)
    }

  # module.eks.module.aws_node_termination_handler_sqs.aws_sqs_queue.this[0] has been changed
  ~ resource "aws_sqs_queue" "this" {
        id                                = "https://sqs.us-east-2.amazonaws.com/876783672983/lattice-dev-eks-termination"
        name                              = "lattice-dev-eks-termination"
      ~ policy                            = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                      ~ Principal = {
                          ~ Service = [
                              - "sqs.amazonaws.com",
                                "events.amazonaws.com",
                              + "sqs.amazonaws.com",
                            ]
                        }
                        # (4 unchanged elements hidden)
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        tags                              = {}
        # (11 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:

  # module.eks.data.aws_eks_cluster_auth.cluster will be read during apply
  # (config refers to values not yet known)
 <= data "aws_eks_cluster_auth" "cluster"  {
      ~ id    = "lattice-dev" -> (known after apply)
        name  = "lattice-dev"
      ~ token = (sensitive value)
    }

  # module.eks.module.eks.data.http.wait_for_cluster[0] will be read during apply
  # (config refers to values not yet known)
 <= data "http" "wait_for_cluster"  {
      ~ body             = "ok" -> (known after apply)
      ~ id               = "mycluster/healthz" -> (known after apply)
      - insecure         = false -> null
      ~ response_headers = {
          - "Cache-Control"                     = "no-cache, private"
          - "Content-Length"                    = "2"
          - "Content-Type"                      = "text/plain; charset=utf-8"
          - "Date"                              = "Thu, 09 Sep 2021 17:53:50 GMT"
          - "X-Content-Type-Options"            = "nosniff"
          - "X-Kubernetes-Pf-Flowschema-Uid"    = "47b04dbd-ec0a-4f43-8fae-f3b23c0f3511"
          - "X-Kubernetes-Pf-Prioritylevel-Uid" = "ac29553c-be4b-4bfc-acd3-69fa0cd4b143"
        } -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.eks.module.eks.aws_autoscaling_group.workers_launch_template[0] will be updated in-place
  ~ resource "aws_autoscaling_group" "workers_launch_template" {
        id                        = "lattice-dev-platform20210909075419913700000004"
        name                      = "lattice-dev-platform20210909075419913700000004"
        # (23 unchanged attributes hidden)



      + tag {
          + key                 = "k8s.io/cluster-autoscaler/enabled"
          + propagate_at_launch = false
          + value               = "true"
        }
      + tag {
          + key                 = "k8s.io/cluster-autoscaler/lattice-dev"
          + propagate_at_launch = false
          + value               = "owned"
        }
      + tag {
          + key                 = "kubernetes.io/cluster/lattice-dev"
          + propagate_at_launch = true
          + value               = "owned"
        }
        # (4 unchanged blocks hidden)
    }

  # module.eks.module.eks.aws_security_group.workers[0] will be updated in-place
  ~ resource "aws_security_group" "workers" {
        id                     = "sg-0401d7099efd3e08a"
        name                   = "lattice-dev20210909063823857900000005"
      ~ tags                   = {
          + "kubernetes.io/cluster/lattice-dev" = "owned"
            # (1 unchanged element hidden)
        }
        # (9 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

[daroga0002]

this doesnt seems related to module itself but default_tags in aws provider which is quite buggy.

Here you will see a lot of issues in provider which can causing your problems:
https://github.com/hashicorp/terraform-provider-aws/issues?q=is%3Aissue+is%3Aopen+default_tags

[ashtonian]

So I still have this issue when default_tags is not present, but when ignore_tags is set.

I've updated the provider to use the following:

provider "aws" {
  region = local.region
  ignore_tags {
    key_prefixes = ["kubernetes.io/", "k8s.io/"]
  }

This is another workaround to prevent the vpc module from overriding subnet tags that are defined elsewhere.

[stephennancekivell]

Im seeing this problem just with the wait_for_cluster response headers. Should we add an ignore_changes for this ?

data "http" "wait_for_cluster"  {
        ...
      ~ response_headers = {
          ~ "Date"                              = "Tue, 14 Sep 2021 04:03:03 GMT" -> "Tue, 14 Sep 2021 04:03:11 GMT"
            ..
        }
    }

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/data.tf#L89

[daroga0002]

Should we add an ignore_changes for this ?

good idea, do you will be able to create a PR for this?

[stephennancekivell]

Hi @daroga0002, Im happy to 😊

[stephennancekivell]

It seems we cant add ignore_changes to wait_for_cluster. Terraform doesn't allow that for data.

Error: Unsupported lifecycle block

  on root.tf line 15, in data "http" "example":
  15:   lifecycle {

Data resources do not have lifecycle settings, so a lifecycle block is not
allowed.

One way I can see of changing this is having the http provider optionally exclude the headers. Thats a bit outside my expertise though. Im not sure what other options there are ?

I've just updated our terraform diff checker to ignore these.

I was able to reproduce with this https://gist.github.com/stephennancekivell/a2c06ce216de3999295c738668d53682

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity since being marked as stale.

[lucasreed]

I've opened up a new PR to the aws http provider to allow a field here called exclude_response_headers: terraform-aws-modules/terraform-provider-http#10

If that is added and released, we could make use of it in this module to exclude the Date header.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.