Apply taints using taint block in aws_eks_node_group resource

[cvittoriasona]

what

• Use the taint block in aws-eks-node-group to apply taints instead of kubelet args now that they're fully supported.
• Uprev AWS provider requirement to >=3.43 to use the taint block in aws-eks-node-group
• Adds tests to validate taints are applied to nodegroup successfully.

why

• Allows var.userdata_override_base64 with var.kubernetes_taints
• Enforces valid taints as part of tf plan step.
• Taints show up in the plan rather than obfuscated in the base64 encoded user_data

references

• EKS Managed Node Groups support Kubernetes Taints - applies to existing & new nodegroups, so no compatibility concerns.
• taint block added in AWS provider v3.43.0

[jamengual]

/test all

[Nuru]

@cvittoriasona Thank you for this contribution. We appreciate the effort.

This module is not currently accepting PRs due to the master branch containing breaking changes that will be replaced. Until the master branch is fixed, PRs are on hold.

Meanwhile, would you please verify that you can use var.userdata_override_base64 with var.kubernetes_taints. The AWS documentation says:

When specifying an AMI, Amazon EKS doesn't merge any user data. Rather, you're responsible for supplying the required bootstrap commands for nodes to join the cluster. If your nodes fail to join the cluster, the Amazon EKS CreateNodegroup and UpdateNodegroupVersion actions also fail.

When the user supplies userdata or taints, this module specifies a launch template with a custom AMI ID, and currentlyvar.kubernetes_taints is ignored if you use var.userdata_override_base64. The thing is, I am pretty sure that the taint block also works by setting userdata parameters passed to bootstrap.sh so taints will continue to be ignored when you specify var.userdata_override_base64.

So we are probably going to reject this request because it imposes the extra requirement on AWS Terraform provider version for not enough gain.

[cvittoriasona]

@Nuru - fair enough; I only question if specifying kubernetes_taints using this module should cause the module to specify a custom AMI ID and override of the EKS API managed user_data bootstrap script, which changes the behavior of the EKS Managed Node Group, now that taints are directly supported.

This is probably more complex than is absolutely needed for this module, but would be interested in:

1. Applying labels/taints through EKS API using the labels and taint attribute/block for aws_eks_node_group
2. Appending to EKS-managed user_data using cloudinit_config per docs. Supports:
   a. EKS-API managed defaults and options, for ex taints/labels, for kubelet
   b. Docker daemon config changes or other user bootstrap script changes that don't touch the kubelet
3. Complete override of user_data in the launch template. Allows for full customization of the bootstrap script. Similar implementation to what exists.

That said, this change would need to be reworked anyway to support the above.

[Nuru]

@cvittoriasona worte

@Nuru - fair enough; I only question if specifying kubernetes_taints using this module should cause the module to specify a custom AMI ID and override of the EKS API managed user_data bootstrap script, which changes the behavior of the EKS Managed Node Group, now that taints are directly supported.

This module does not "change the behavior of the EKS Managed Node Group", it mimics it. The user_data this module supplies is a simple modification of what EKS supplies, just adding features. I am pretty sure that all the resource attribute options do is make the same kind of modifications to user data that this module makes. I encourage you to investigate and report back.