docs: Complete upgrade guide docs for migration and changes applied

README.md

@@ -287,7 +287,7 @@ We are grateful to the community for contributing bugfixes and improvements! Ple
 
 ## License
 
-Apache 2 Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-aws-rds-aurora/tree/master/LICENSE) for full details.
+Apache 2 Licensed. See [LICENSE](https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/LICENSE) for full details.
 
 ## Additional information for users from Russia and Belarus
 

docs/UPGRADE-20.0.md

@@ -19,6 +19,7 @@ To give users advanced notice and provide some future direction for this module,
 
 1. The `aws-auth` sub-module will be removed entirely from the project. Since this sub-module is captured in the v20.x releases, users can continue using it even after the module moves forward with the next major version. The long term strategy and direction is cluster access entry and to rely only on the AWS Terraform provider.
 2. The default value for `authentication_mode` will change to `API`. Aligning with point 1 above, this is a one way change, but users are free to specify the value of their choosing in place of this default (when the change is made). This module will proceed with an EKS API first strategy.
+3. The launch template and autoscaling group usage contained within the EKS managed nodegroup and self-managed nodegroup sub-modules *might be replaced with the [`terraform-aws-autoscaling`](https://github.com/terraform-aws-modules/terraform-aws-autoscaling) module. At minimum, it makes sense to replace most of functionality in the self-managed nodegroup module with this external module, but its not yet clear if there is any benefit of using it in the EKS managed nodegroup sub-module. The interface that users interact with will stay the same, the changes will be internal to the implementation and we will do everything we can to keep the disruption to a minimum.
 
 ## Additional changes
 
@@ -98,12 +99,6 @@ To give users advanced notice and provide some future direction for this module,
    - `access_entries`
    - `cloudwatch_log_group_class`
 
-   - EKS managed nodegroup
-      - `create_access_entry`
-
-   - Fargate Profile
-      - `create_access_entry`
-
    - Karpenter
       - `iam_policy_name`
       - `iam_policy_use_name_prefix`
@@ -136,12 +131,6 @@ To give users advanced notice and provide some future direction for this module,
 
    - `access_entries`
 
-   - EKS managed nodegroup
-      - `access_entry_arn`
-
-   - Fargate Profile
-      - `access_entry_arn`
-
    - Karpenter
       - `node_access_entry_arn`
 
@@ -150,4 +139,124 @@ To give users advanced notice and provide some future direction for this module,
 
 ## Upgrade Migrations
 
+### Diff of Before (v19.21) vs After (v20.0)
+
+```diff
+ module "eks" {
+   source  = "terraform-aws-modules/eks/aws"
+-  version = "~> 19.21"
++  version = "~> 20.0"
+
+# If you want to maintain the current default behavior of v19.x
++  kms_key_enable_default_policy = false
+
+-   manage_aws_auth_configmap = true
+
+-   aws_auth_roles = [
+-     {
+-       rolearn  = "arn:aws:iam::66666666666:role/role1"
+-       username = "role1"
+-       groups   = ["custom-role-group"]
+-     },
+-   ]
+
+-   aws_auth_users = [
+-     {
+-       userarn  = "arn:aws:iam::66666666666:user/user1"
+-       username = "user1"
+-       groups   = ["custom-users-group"]
+-     },
+-   ]
+}
+
++ module "eks" {
++   source  = "terraform-aws-modules/eks/aws//modules/aws-auth"
++   version = "~> 20.0"
+
++   manage_aws_auth_configmap = true
+
++   aws_auth_roles = [
++     {
++       rolearn  = "arn:aws:iam::66666666666:role/role1"
++       username = "role1"
++       groups   = ["custom-role-group"]
++     },
++   ]
+
++   aws_auth_users = [
++     {
++       userarn  = "arn:aws:iam::66666666666:user/user1"
++       username = "user1"
++       groups   = ["custom-users-group"]
++     },
++   ]
++ }
+```
+
+### Karpenter Diff of Before (v19.21) vs After (v20.0)
+
+```diff
+ module "eks" {
+   source  = "terraform-aws-modules/eks/aws//modules/karpenter"
+-  version = "~> 19.21"
++  version = "~> 20.0"
+
+# If you wish to maintain the current default behavior of v19.x
++  enable_irsa             = true
++  create_instance_profile = true
+
+# To avoid any resource re-creation
++  iam_role_name          = "KarpenterIRSA-${module.eks.cluster_name}"
++  iam_role_description   = "Karpenter IAM role for service account"
++  iam_policy_name        = "KarpenterIRSA-${module.eks.cluster_name}"
++  iam_policy_description = "Karpenter IAM role for service account"
+}
+```
+
 ## Terraform State Moves
+
+#### ⚠️ Authentication Mode Changes ⚠️
+
+Changing the `authentication_mode` is a one-way decision. See [announcement blog](https://aws.amazon.com/blogs/containers/a-deep-dive-into-simplified-amazon-eks-access-management-controls/) for further details:
+
+> Switching authentication modes on an existing cluster is a one-way operation. You can switch from CONFIG_MAP to API_AND_CONFIG_MAP. You can then switch from API_AND_CONFIG_MAP to API. You cannot revert these operations in the opposite direction. Meaning you cannot switch back to CONFIG_MAP or API_AND_CONFIG_MAP from API. And you cannot switch back to CONFIG_MAP from API_AND_CONFIG_MAP.
+
+### authentication_mode = "CONFIG_MAP"
+
+If using `authentication_mode = "CONFIG_MAP"`, before making any changes, you will first need to remove the configmap from the statefile to avoid any disruptions:
+
+```sh
+terraform state rm 'module.eks.kubernetes_config_map_v1_data.aws_auth[0]'
+terraform state rm 'module.eks.kubernetes_config_map.aws_auth[0]' # include if Terraform created the original configmap
+```
+
+Once the configmap has been removed from the statefile, you can add the new `aws-auth` sub-module and copy the relevant definitions from the EKS module over to the new `aws-auth` sub-module definition (see before after diff above).
+
+⚠️ 
+You will need to add entries for any IAM roles used by nodegroups and/or Fargate profiles - the module no longer handles this in the background on behalf of users.
+⚠️
+
+When you apply the changes with the new sub-module, the configmap in the cluster will get updated with the contents provided in the sub-module definition, so please be sure all of the necessary entries are added before applying the changes.
+
+### authentication_mode = "API_AND_CONFIG_MAP"
+
+When using `authentication_mode = "API_AND_CONFIG_MAP"` and there are entries that will remain in the configmap (entries that cannot be replaced by cluster access entry), you will first need to update the `authentication_mode` on the cluster to `"API_AND_CONFIG_MAP"`. To help make this upgrade process easier, a copy of the changes defined in the [`v20.0.0`](https://github.com/terraform-aws-modules/terraform-aws-eks/pull/2858) PR have been captured [here](https://github.com/clowdhaus/terraform-aws-eks-v20-migrate) but with the `aws-auth` components still provided in the module. This means you get the equivalent of the `v20.0.0` module, but it still includes support for the `aws-auth` configmap. You can follow the provided README on that interim migration module for the order of execution and return here once the `authentication_mode` has been updated to `"API_AND_CONFIG_MAP"`.
+
+Once the `authentication_mode` has been updated, next you will need to remove the configmap from the statefile to avoid any disruptions:
+
+```sh
+terraform state rm 'module.eks.kubernetes_config_map_v1_data.aws_auth[0]'
+terraform state rm 'module.eks.kubernetes_config_map.aws_auth[0]' # include if Terraform created the original configmap
+```
+
+ℹ️
+If you are using Terraform `v1.7+`, you can utilize the [`remove`](https://developer.hashicorp.com/terraform/language/resources/syntax#removing-resources) to facilitate both the removal of the configmap through code. You can create a fork/clone of the provided [migration module](https://github.com/clowdhaus/terraform-aws-eks-migrate-v19-to-v20) and add the `remove` blocks and apply those changes before proceeding. We do not want to force users onto the bleeding edge with this module, so we have not included `remove` support at this time.
+ℹ️
+
+Once the configmap has been removed from the statefile, you can add the new `aws-auth` sub-module and copy the relevant definitions from the EKS module over to the new `aws-auth` sub-module definition (see before after diff above). When you apply the changes with the new sub-module, the configmap in the cluster will get updated with the contents provided in the sub-module definition, so please be sure all of the necessary entries are added before applying the changes. In the before/example above - the configmap would remove any entries for roles used by nodegroups and/or Fargate Profiles, but maintain the custom entries for users and roles passed into the module definition.
+
+### authentication_mode = "API"
+
+In order to switch to `API` only using cluster access entry, you first need to update the `authentication_mode` on the cluster to `API_AND_CONFIG_MAP` without modifying the `aws-auth` configmap. To help make this upgrade process easier, a copy of the changes defined in the [`v20.0.0`](https://github.com/terraform-aws-modules/terraform-aws-eks/pull/2858) PR have been captured [here](https://github.com/clowdhaus/terraform-aws-eks-v20-migrate) but with the `aws-auth` components still provided in the module. This means you get the equivalent of the `v20.0.0` module, but it still includes support for the `aws-auth` configmap. You can follow the provided README on that interim migration module for the order of execution and return here once the `authentication_mode` has been updated to `"API_AND_CONFIG_MAP"`.
+
+Once the `authentication_mode` has been updated, you can update the `authentication_mode` on the cluster to `API` and remove the `aws-auth` configmap components.

examples/eks_managed_node_group/README.md

@@ -49,7 +49,6 @@ Note that this example may create resources which cost money. Run `terraform des
 | <a name="module_eks_managed_node_group"></a> [eks\_managed\_node\_group](#module\_eks\_managed\_node\_group) | ../../modules/eks-managed-node-group | n/a |
 | <a name="module_key_pair"></a> [key\_pair](#module\_key\_pair) | terraform-aws-modules/key-pair/aws | ~> 2.0 |
 | <a name="module_vpc"></a> [vpc](#module\_vpc) | terraform-aws-modules/vpc/aws | ~> 5.0 |
-| <a name="module_vpc_cni_irsa"></a> [vpc\_cni\_irsa](#module\_vpc\_cni\_irsa) | terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks | ~> 5.0 |
 
 ## Resources
 

examples/eks_managed_node_group/main.tf

@@ -7,7 +7,7 @@ data "aws_availability_zones" "available" {}
 
 locals {
   name            = "ex-${replace(basename(path.cwd), "_", "-")}"
-  cluster_version = "1.29"
+  cluster_version = "1.27"
   region          = "eu-west-1"
 
   vpc_cidr = "10.0.0.0/16"
@@ -31,9 +31,6 @@ module "eks" {
   cluster_version                = local.cluster_version
   cluster_endpoint_public_access = true
 
-  # Gives Terraform identity admin access to cluster
-  enable_cluster_creator_admin_permissions = true
-
   # IPV6
   cluster_ip_family          = "ipv6"
   create_cni_ipv6_iam_policy = true
@@ -46,9 +43,8 @@ module "eks" {
       most_recent = true
     }
     vpc-cni = {
-      most_recent              = true
-      before_compute           = true
-      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
+      most_recent    = true
+      before_compute = true
       configuration_values = jsonencode({
         env = {
           # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
@@ -248,12 +244,6 @@ module "eks" {
   tags = local.tags
 }
 
-module "disabled_eks_managed_node_group" {
-  source = "../../modules/eks-managed-node-group"
-
-  create = false
-}
-
 module "disabled_eks" {
   source = "../.."
 
@@ -294,6 +284,12 @@ module "eks_managed_node_group" {
   tags = merge(local.tags, { Separate = "eks-managed-node-group" })
 }
 
+module "disabled_eks_managed_node_group" {
+  source = "../../modules/eks-managed-node-group"
+
+  create = false
+}
+
 ################################################################################
 # Supporting Resources
 ################################################################################
@@ -333,24 +329,6 @@ module "vpc" {
   tags = local.tags
 }
 
-module "vpc_cni_irsa" {
-  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
-  version = "~> 5.0"
-
-  role_name_prefix      = "VPC-CNI-IRSA"
-  attach_vpc_cni_policy = true
-  vpc_cni_enable_ipv6   = true
-
-  oidc_providers = {
-    main = {
-      provider_arn               = module.eks.oidc_provider_arn
-      namespace_service_accounts = ["kube-system:aws-node"]
-    }
-  }
-
-  tags = local.tags
-}
-
 module "ebs_kms_key" {
   source  = "terraform-aws-modules/kms/aws"
   version = "~> 2.1"

examples/fargate_profile/main.tf

@@ -30,9 +30,6 @@ module "eks" {
   cluster_version                = local.cluster_version
   cluster_endpoint_public_access = true
 
-  # Gives Terraform identity admin access to cluster
-  enable_cluster_creator_admin_permissions = true
-
   cluster_addons = {
     kube-proxy = {}
     vpc-cni    = {}
@@ -92,12 +89,6 @@ module "eks" {
   tags = local.tags
 }
 
-module "disabled_fargate_profile" {
-  source = "../../modules/fargate-profile"
-
-  create = false
-}
-
 ################################################################################
 # Sub-Module Usage on Existing/Separate Cluster
 ################################################################################
@@ -116,6 +107,12 @@ module "fargate_profile" {
   tags = merge(local.tags, { Separate = "fargate-profile" })
 }
 
+module "disabled_fargate_profile" {
+  source = "../../modules/fargate-profile"
+
+  create = false
+}
+
 ################################################################################
 # Supporting Resources
 ################################################################################

examples/karpenter/README.md

@@ -41,6 +41,9 @@ kubectl delete node -l karpenter.sh/provisioner-name=default
 2. Remove the resources created by Terraform
 
 ```bash
+# Necessary to avoid removing Terraform's permissions too soon before its finished
+# cleaning up the resources it deployed inside the clsuter
+terraform state rm module.eks.aws_eks_access_entry.this["cluster_creator_admin"] || true
 terraform destroy
 ```
 

examples/karpenter/main.tf

@@ -42,7 +42,7 @@ data "aws_ecrpublic_authorization_token" "token" {
 
 locals {
   name            = "ex-${replace(basename(path.cwd), "_", "-")}"
-  cluster_version = "1.29"
+  cluster_version = "1.28"
   region          = "eu-west-1"
 
   vpc_cidr = "10.0.0.0/16"
@@ -66,7 +66,8 @@ module "eks" {
   cluster_version                = local.cluster_version
   cluster_endpoint_public_access = true
 
-  # Gives Terraform identity admin access to cluster
+  # Gives Terraform identity admin access to cluster which will
+  # allow deploying resources (Karpenter) into the cluster
   enable_cluster_creator_admin_permissions = true
 
   cluster_addons = {
@@ -149,6 +150,17 @@ module "karpenter" {
   tags = local.tags
 }
 
+module "karpenter_disabled" {
+  source = "../../modules/karpenter"
+
+  create = false
+}
+
+################################################################################
+# Karpenter Helm chart & manifests
+# Not required; just to demonstrate functionality of the sub-module
+################################################################################
+
 resource "helm_release" "karpenter" {
   namespace           = "karpenter"
   create_namespace    = true
@@ -158,6 +170,7 @@ resource "helm_release" "karpenter" {
   repository_password = data.aws_ecrpublic_authorization_token.token.password
   chart               = "karpenter"
   version             = "v0.33.1"
+  wait                = false
 
   values = [
     <<-EOT
@@ -264,12 +277,6 @@ resource "kubectl_manifest" "karpenter_example_deployment" {
   ]
 }
 
-module "karpenter_disabled" {
-  source = "../../modules/karpenter"
-
-  create = false
-}
-
 ################################################################################
 # Supporting Resources
 ################################################################################

examples/outposts/README.md

@@ -36,6 +36,13 @@ $ terraform apply
 
 Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
 
+```bash
+# Necessary to avoid removing Terraform's permissions too soon before its finished
+# cleaning up the resources it deployed inside the clsuter
+terraform state rm module.eks.aws_eks_access_entry.this["cluster_creator_admin"] || true
+terraform destroy
+```
+
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Requirements
 

examples/outposts/main.tf

@@ -29,7 +29,8 @@ module "eks" {
   cluster_endpoint_public_access  = false # Not available on Outpost
   cluster_endpoint_private_access = true
 
-  # Gives Terraform identity admin access to cluster
+  # Gives Terraform identity admin access to cluster which will
+  # allow deploying resources (EBS storage class) into the cluster
   enable_cluster_creator_admin_permissions = true
 
   vpc_id     = data.aws_vpc.this.id

examples/outposts/prerequisites/main.tf

@@ -56,7 +56,7 @@ module "ssm_bastion_ec2" {
     rm terraform_${local.terraform_version}_linux_amd64.zip 2> /dev/null
 
     # Install kubectl
-    curl -LO https://dl.k8s.io/release/v1.27.0/bin/linux/amd64/kubectl
+    curl -LO https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl
     install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
 
     # Remove default awscli which is v1 - we want latest v2