Init Push

.gitignore

@@ -0,0 +1,37 @@
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Exclude all .tfvars files, which are likely to contain sentitive data, such as
+# password, private keys, and other secrets. These should not be part of version
+# control as they are data points which are potentially sensitive and subject
+# to change depending on the environment.
+#
+*.tfvars
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+#
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
+
+.DS_Store

README.md

@@ -0,0 +1,77 @@
+# Elastic Kubernetes Service (EKS)
+
+This is an opinionated terraform module to bootstrap an EKS Cluster using Terraform.
+
+Features enabled:
+
+* Logging using Amazon Cloudwatch
+
+* Firewall security measures, allowing only required control-plane to nodes communication.
+
+* IAM Accounts for Service Accounts
+
+* VPC Native cluster
+
+* Cluster Autoscaler IAM Roles and Helm release installed (configurable)
+
+* Metrics server configured and fully functional
+
+* Updatable nodes through AWS Autoscaling instance refreshes
+
+* Non-default SA for nodes
+
+* Usage of containerd as runtime (configurable on the example file)
+
+* `aws-auth` management
+
+## Usage
+
+You can find a fully functional, production-ready example on the `examples/` folder.
+
+### Requirements
+
+| Name | Version |
+| --------- | ------- |
+| terraform | >= 1.0 |
+
+### Important Note
+
+This module requires the `kubectl` client to be installed, since it uses a local exec provisioner.
+
+### Providers
+
+| Name | Version |
+| ----------- | -------- |
+| local | ~> 2.1.0 |
+| aws | ~> 3.73.0 |
+| kubernetes | ~> 2.5.1 |
+| null | >= 3.0 |
+| helm | 2.4.0 |
+
+### Inputs
+
+| Name                      | Description                                                                                                                                        | Type                                                                                            | Default                                             | Required |
+|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|-----------------------------------------------------|----------|
+| environment               | The environment where this cluster will be deployed. All names will be generated from this variable                                                | string                                                                                          | devgurus-dev                                        | no       |
+| kubernetes_version        | The Kubernetes version that the module will try to bootstrap                                                                                       | string                                                                                          | 1.21                                                | no       |
+| self_managed_node_groups  | An object representing all node-groups that you want to create. They should be generated according the terraform-aws-modules/eks/aws documentation | any                                                                                             | n/a                                                 | yes      |
+| vpc_cidr                  | VPC's CIDR to be created by the VPC module                                                                                                         | string                                                                                          | 10.0.0.0/16                                         | no       |
+| vpc_private_subnets       | VPC's private subnets to be created by the VPC module                                                                                              | list(string)                                                                                    | [ "10.0.1.0/24" ,  "10.0.2.0/24" ,  "10.0.3.0/24" ] | no       |
+| vpc_public_subnets        | VPC's public subnets to be created by the VPC module                                                                                               | list(string)                                                                                    | [ "10.0.4.0/24" ,  "10.0.5.0/24" ,  "10.0.6.0/24" ] | no       |
+| enable_cluster_autoscaler | Whether to create a Helm release installing cluster-autoscaler resources or not                                                                    | bool                                                                                            | false                                               | no       |
+| map_users                 | An array of objects that represent what IAM users have access to the EKS Cluster                                                                   | list(object({     userarn  =  string     username =  string     groups   = list( string )   })) | []                                                  | no       |
+| map_roles                 | An array of objects that represent what IAM roles have access to the EKS Cluster                                                                   | list(object({     rolearn  =  string     username =  string     groups   = list( string )   })) | []                                                  | no       |
+
+### Outputs
+
+| Name                     | Description                  |
+| ------------------------ | ---------------------------- |
+| cluster_arn              | The Amazon Resource Name (ARN) of the cluster             |
+| cluster_certificate_authority_data           | Base64 encoded certificate data required to communicate with the cluster |
+| cluster_endpoint              | Endpoint for your Kubernetes API server        |
+| cluster_id             | The name/id of the EKS cluster. Will block on cluster creation until the cluster is really ready                 |
+| cluster_oidc_issuer_url        | The URL on the EKS cluster for the OpenID Connect identity provider             |
+| oidc_provider_arn          | The ARN of the OIDC Provider              |
+| cloudwatch_log_group_name          | Name of cloudwatch log group created              |
+| cloudwatch_log_group_arn          | Arn of cloudwatch log group created              |
+| self_managed_node_groups_autoscaling_group_names | The names of the self managed ASG created by the module |

addons.tf

@@ -0,0 +1,63 @@
+#
+# The cluster autoscaler will:
+#   1) Check the cluster every 10s to find unschedulable pods, and if required, it'll contact AWS to scale up the cluster
+#   2) Check if there is any underused node, and if possible, will reschedule all pods to another nodes and remove the node
+# 
+#   We're using all autoscaler defaults, but they can be overriden using the Helm extraArgs parameter:
+#     --scan-interval controls how often the autoscaler checks for unschedulable resources (default 10)
+#     --scale-down-utilization-threshold how low the usage has to be to consider node removal (default 50%)
+#     --new-pod-scale-up-delay how old the pods have to be in order to be taken into account for the autoschedule algorithm (default 10s)
+#   More information here https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md
+resource "helm_release" "cluster_autoscaler" {
+  depends_on = [
+    module.eks_cluster
+  ]
+
+  count = var.enable_cluster_autoscaler == true ? 1 : 0
+
+  name             = "cluster-autoscaler"
+  namespace        = "cluster-autoscaler"
+  repository       = "https://kubernetes.github.io/autoscaler"
+  chart            = "cluster-autoscaler"
+  version          = "9.21.0"
+  create_namespace = true
+
+  set {
+    name  = "awsRegion"
+    value = data.aws_region.current.name
+  }
+  set {
+    name  = "rbac.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
+    value = aws_iam_role.eks_cluster_autoscaler_role[0].arn
+    type  = "string"
+  }
+  set {
+    name  = "autoDiscovery.clusterName"
+    value = local.cluster_name
+  }
+  set {
+    name  = "rbac.create"
+    value = "true"
+  }
+}
+
+resource "helm_release" "metrics_server" {
+  depends_on = [
+    module.eks_cluster
+  ]
+
+  # count = var.enable_cluster_autoscaler == true ? 1 : 0
+
+  name             = "metrics-server"
+  namespace        = "metrics-server"
+  version          = "3.8.2"
+  repository       = "https://kubernetes-sigs.github.io/metrics-server/"
+  chart            = "metrics-server"
+  create_namespace = true
+
+  set {
+    name  = "replicas"
+    value = 2
+  }
+
+}

eks.tf

@@ -0,0 +1,91 @@
+data "aws_availability_zones" "available" {}
+
+data "aws_eks_cluster" "cluster" {
+  name = module.eks_cluster.cluster_id
+}
+
+data "aws_eks_cluster_auth" "this" {
+  name = module.eks_cluster.cluster_id
+}
+
+module "eks_cluster" {
+  source  = "terraform-aws-modules/eks/aws"
+  version = "~> 18.29.0"
+
+  cluster_name    = local.cluster_name
+  cluster_version = var.kubernetes_version
+
+  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
+
+  manage_aws_auth_configmap = var.manage_aws_auth_configmap
+  create_aws_auth_configmap = var.create_aws_auth_configmap
+
+  aws_auth_roles = var.map_roles
+  aws_auth_users = var.map_users
+
+  # # Enabling encryption on AWS EKS secrets using a customer-created key
+  cluster_encryption_config = [{
+    provider_key_arn = aws_kms_key.eks_kms_key.arn
+    resources        = ["secrets"]
+  }]
+
+
+  # Enabling this, we allow EKS to manage this components for us (upgrading and maintaining)
+  cluster_addons = {
+    kube-proxy = {}
+    vpc-cni = {
+      resolve_conflicts = "OVERWRITE"
+    }
+  }
+
+  vpc_id     = module.vpc.vpc_id
+  subnet_ids = module.vpc.private_subnets
+
+  # IRSA enabled to create an OpenID trust between our cluster and IAM, in order to map AWS Roles to Kubernetes SA's
+  enable_irsa = true
+
+  self_managed_node_group_defaults = {
+    update_launch_template_default_version = true
+    iam_role_additional_policies           = ["arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"]
+  }
+
+
+  self_managed_node_groups = var.self_managed_node_groups
+
+  node_security_group_additional_rules = {
+    ms_4443_ing = {
+      description                   = "Cluster API to metrics server 4443 ingress port"
+      protocol                      = "tcp"
+      from_port                     = 4443
+      to_port                       = 4443
+      type                          = "ingress"
+      source_cluster_security_group = true
+    }
+
+    ms_443_ing = {
+      description                   = "Cluster API to metrics server 15017 ingress port"
+      protocol                      = "tcp"
+      from_port                     = 15017
+      to_port                       = 15017
+      type                          = "ingress"
+      source_cluster_security_group = true
+    }
+    node_to_node_ig = {
+      description = "Node to node ingress traffic"
+      from_port   = 1
+      to_port     = 65535
+      protocol    = "all"
+      type        = "ingress"
+      self        = true
+    }
+    node_to_node_eg = {
+      description = "Node to node egress traffic"
+      from_port   = 1
+      to_port     = 65535
+      protocol    = "all"
+      type        = "egress"
+      self        = true
+    }
+  }
+
+}

examples/spot_eks_cluster.tf

@@ -0,0 +1,71 @@
+module "eks-cluster" {
+  source             = "../."
+  environment        = "eks-spot-demo"
+  kubernetes_version = "1.23"
+
+  enable_cluster_autoscaler = true
+
+  # Networking configuration 
+  vpc_cidr            = "10.0.0.0/16"
+  vpc_private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
+  vpc_public_subnets  = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
+
+
+  create_aws_auth_configmap = true
+
+  map_users = [
+    {
+      userarn  = "arn:aws:iam::xxxxxxxx:user/youremail@yourdomain.com"
+      username = "youremail@yourdomain"
+      groups   = ["system:masters"]
+    },
+    {
+      userarn  = "arn:aws:iam::xxxxxxxx:user/youremail2@yourdomain.com"
+      username = "youremail2@yourdomain.com"
+      groups   = ["system:masters"]
+    }
+  ]
+
+  map_roles = [
+    {
+      rolearn  = "arn:aws:iam::xxxxxxxx:role/YourRoleARN"
+      username = "yourroleusername"
+      groups   = ["system:masters"]
+    }
+  ]
+
+  # Here, we define all node pools that we want to create
+
+  self_managed_node_groups = {
+    spot_pool = {
+      name = "spool-node-pool"
+
+      instance_type = "t3.medium"
+
+      max_size     = 6
+      desired_size = 2
+
+      # If we want to run the whole pool with spot instances we create this block
+      instance_market_options = {
+        market_type = "spot"
+      }
+
+      # Dockershim is the default runtime. Here, we switch it with containerd
+      bootstrap_extra_args = "--container-runtime containerd"
+
+
+      # SSM Agent installation. Extracted from the AWS EKS module documentation
+      post_bootstrap_user_data = <<-EOT
+      cd /tmp
+      sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
+      sudo systemctl enable amazon-ssm-agent
+      sudo systemctl start amazon-ssm-agent
+      EOT
+
+      autoscaling_group_tags = {
+        "k8s.io/cluster-autoscaler/eks-spot-demo-eks-cluster" = "owned"
+        "k8s.io/cluster-autoscaler/enabled"                   = "true"
+      }
+    }
+  }
+}

iam_policies.tf

@@ -0,0 +1,41 @@
+resource "aws_iam_policy" "autoscaler_modify_asg" {
+  name        = "ClusterAutoscalerPolicy-${var.environment}"
+  count       = var.enable_cluster_autoscaler == true ? 1 : 0
+  description = "Policy created to allow the Cluster autoscaler service to access the underlying AWS ASG"
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [
+      {
+        Action = [
+          "autoscaling:DescribeAutoScalingGroups",
+          "autoscaling:DescribeAutoScalingInstances",
+          "autoscaling:DescribeLaunchConfigurations",
+          "autoscaling:DescribeTags",
+          "ec2:DescribeInstanceTypes",
+          "ec2:DescribeLaunchTemplateVersions",
+          "ec2:GetInstanceTypesFromInstanceRequirements",
+          "eks:DescribeNodegroup"
+        ]
+        Effect   = "Allow"
+        Resource = "*"
+      },
+      {
+        Action = [
+          "autoscaling:SetDesiredCapacity",
+          "autoscaling:TerminateInstanceInAutoScalingGroup",
+          "autoscaling:UpdateAutoScalingGroup",
+        ]
+        Effect   = "Allow"
+        Resource = "*"
+
+        Condition = {
+          "StringEquals" = {
+            "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/${local.cluster_name}" = ["owned"],
+            "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled"               = ["true"]
+          }
+        }
+      }
+
+    ]
+  })
+}