Tags are not propagated from EKS managed node group to auto-scaling groups

[cthiebault]

Description

We are using cluster-autoscaler.
It needs tags on ASG to know which ASG to use to start or stop a new node.

We tags EKS Managed Node Groups but not all tags are propagated to ASG :-(

Versions

Terraform v1.1.5
on linux_amd64

• provider registry.terraform.io/hashicorp/aws v4.0.0
• provider registry.terraform.io/hashicorp/cloudinit v2.2.0
• provider registry.terraform.io/hashicorp/kubernetes v2.8.0
• provider registry.terraform.io/hashicorp/null v3.1.0
• provider registry.terraform.io/hashicorp/tls v3.1.0
• provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Reproduction

locals {
  cluster_name = "my-cluster"
}
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "~> 18.7.1"
  cluster_version = "1.21"
  cluster_name    = local.cluster_name
  tags            = {
    "k8s.io/cluster-autoscaler/enabled"               = true
    "k8s.io/cluster-autoscaler/${local.cluster_name}" = true
    "kubernetes.io/cluster/${local.cluster_name}"     = "owned"
  }
  eks_managed_node_group_defaults = {
    ami_type                = "BOTTLEROCKET_x86_64"
    platform                = "bottlerocket"
    instance_types          = [ "t3.medium" ]
    disk_size               = 50
    ebs_optimized           = false
    disable_api_termination = false
    enable_monitoring       = true
    force_update_version    = true
  }
  eks_managed_node_groups = {
    test = {
      name            = "test"
      use_name_prefix = true
      min_size        = 0
      max_size        = 2
      desired_size    = 0
      labels          = {
        env = "test"
      }
      tags = {
        "k8s.io/cluster-autoscaler/node-template/label/env" = "test"
      }
    }
    staging = {
      name            = "staging"
      use_name_prefix = true
      min_size        = 0
      max_size        = 2
      desired_size    = 0
      labels          = {
        env = "staging"
      }
      tags = {
        "k8s.io/cluster-autoscaler/node-template/label/env" = "staging"
      }
    }
  }
}

Here are the tags for test EKS managed node group:

• Name=test
• kubernetes.io/cluster/my-cluster=owned
• k8s.io/cluster-autoscaler/enabled=true
• k8s.io/cluster-autoscaler/my-cluster=true
• k8s.io/cluster-autoscaler/node-template/label/env=test

And the tags for ASG created by test EKS managed node group:

• eks:cluster-name=my-cluster
• eks:nodegroup-name=test-20220216130904674900000025
• k8s.io/cluster-autoscaler/enabled=true
• k8s.io/cluster-autoscaler/my-cluster=owned
• kubernetes.io/cluster/my-cluster

Tag k8s.io/cluster-autoscaler/node-template/label/env=test specified in EKS managed node group is not propagated to ASG :-(
Cluster autoscaler needs this tags k8s.io/cluster-autoscaler/node-template/... to find which ASG to use.

Any ideas?

[cthiebault]

Oups... it may be a duplicate of #860

[bryantbiggs]

The EKS managed node group service manages the autoscaling group created. You can use https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group_tag to tag the autoscaling groups after they are created

[cthiebault]

Thanks for the pointer @bryantbiggs!

I'm now able to add tags to created ASG using aws_autoscaling_group_tag:

locals {
  staging-asg-name = module.eks.eks_managed_node_groups["staging"].node_group_resources[0].autoscaling_groups[0].name
  staging-asg-tags = [
    {
      tagKey   = "k8s.io/cluster-autoscaler/node-template/label/env"
      tagValue = "staging"
    },
  ]
}

resource "aws_autoscaling_group_tag" "staging" {
  for_each = { for k, v in local.staging-asg-tags : k => v }

  autoscaling_group_name = local.staging-asg-name
  tag {
    key                 = each.value.tagKey
    value               = each.value.tagValue
    propagate_at_launch = false
  }
}

I'm sure there is a way to loop over EKS managed node groups, find their ASG and replicate tag from node groups to ASG,
but I'm not strong enough with Terraform for_each :-/

[joncolby]

@cthiebault - I had the same problem and a similar workaround. It would be nice if the module provided a way to handle ASG specific tags.

I used an double loop (we use a locals to define worker groups, and inside it we define a field asg_tags)

locals {
self_managed_node_groups = {

 foo-worker-group = {
    name = foo
    instance_type = "m5.large"
    min_size      = 2
    desired_size  = 2
    max_size      = 5
    ....

  // the `asg_tags` field is not recognised by the module.self_managed_node_groups, but we use it in the loop below
   asg_tags = {
       mytag1 = "aaa"
       mytag2 = "bbb"
  }

  ...
 }
}


  asg_tags = flatten([
    for i in local.self_managed_node_groups : [
      for k, v in i.asg_tags : {
        id                  = join("-", [i.name, k])
        name                = i.name
        key                 = k
        value               = v
      }
    ]
  ])
}

}



resource "aws_autoscaling_group_tag" "this" {
  for_each = { for t in local.asg_tags : t.id => t }

  autoscaling_group_name = module.eks.self_managed_node_groups[each.value.name].autoscaling_group_name

  tag {
    key                 = each.value.key
    value               = each.value.value
    propagate_at_launch = true
  }

[bryantbiggs]

its not possible for it to exist natively within this module due to lifecycle constraints #1558 (comment)

[bryantbiggs]

closing for now - see links/references shared for context

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.