Support target_group_arns for managed node groups

[DmitriyStoyanov]

Is your request related to a new offering from AWS?

This feature request opened in aws provider for node groups
aws/containers-roadmap#709

but separate part can be added in module to support this feature

Is your request related to a problem? Please describe.

For the moment, we used nginx ingress controller and worker groups attached to alb via property target_group_arns
(A list of Application LoadBalancer (ALB) target group ARNs to be associated to the autoscaling group)

When we added managed node groups we found that alb target group does not contain instances from managed node groups. So ingress cannot work on these instances.

Describe the solution you'd like.

option as a target_group_arns for node_groups

Describe alternatives you've considered.

Currently to mitigate this issue we created separate resource (thanks to aws/containers-roadmap#709 (comment) ) :

resource "aws_autoscaling_attachment" "autoscaling_attachment" {
  for_each = {for name, value in module.eks.node_groups : name => lookup(lookup(lookup(value, "resources")[0], "autoscaling_groups")[0], "name")}
  autoscaling_group_name = each.value
  alb_target_group_arn   = aws_lb_target_group.alb.arn
}

So this part can be added in node_group submodule

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[daroga0002]

why you cannot use normal service Type=Loadbalancer or something like https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/nlb/ ?

[DmitriyStoyanov]

From my case, I had previously created LB for nginx ingress for whole cluster and it was attached to ALB group via target_group_arns. But when I started to add managed nodegroups, faced, that this balancer cannot be attached to these node groups because of unavailability of such parameter in node group submodule. So it is added as separate aws_autoscaling_attachment terraform resource. If it will be added to submodule, it can be used as the same parameter directly during module description.

[daroga0002]

I really dont understand why you cannot use cloud provider or mentioned https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/service/nlb/ ?

[DmitriyStoyanov]

Because ALB already exist in separate terraform module, with management of Security groups, WAF, Certificates, etc... in separate module.
And by the way, one of our cluster in China, so you cannot just recreate your LB, if you already pinned it with static IP's and applied ICP License for that stuff.

[jaimehrubiks]

@daroga0002 Many of us don't use service type "LoadBalancer" because we prefer to use terraform when creating infrastructure. That way we can manage security groups, dns, and so on in a much controlled way than when kubernetes creates it automatically.

That said I use this without trouble

resource "aws_autoscaling_attachment" "node_groups--ELB_NAME" {
for_each = module.eks.node_groups

autoscaling_group_name = lookup(lookup(lookup(each.value, "resources")[0], "autoscaling_groups")[0], "name")
elb = aws_elb.elb_ELB_NAME.id
}

[DmitriyStoyanov]

From my side I already solved this issue by creation of

resource "aws_autoscaling_attachment" "autoscaling_attachment" {
  for_each = {for name, value in module.eks.node_groups : name => lookup(lookup(lookup(value, "resources")[0], "autoscaling_groups")[0], "name")}
  autoscaling_group_name = each.value
  alb_target_group_arn   = aws_lb_target_group.alb.arn
}

And just created this issue, to possible add this ability in the module as we already have for asg groups.

Or possible if someone also will face with the same issue, the same approach can be used.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kkapoor1987]

@max-rocket-internet @antonbabenko can we pass target_group_arns to LTs for NodeGroups. This would provide similar behavior as we have for ASGs where we expose target_group_arns as input param and that to LTs
Just like this
#219 (comment)

[daroga0002]

I am still sceptical for this approach as this is not best pattern to follow. If we want create such resources then we should rather follow this approach https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/targetgroupbinding/targetgroupbinding/

[kkapoor1987]

@daroga0002 We have ALB created by different TF module and MNG creation creates ASGs which would not have target_group_arns as in self-managed ASGs. I am not sure why we have to treat MNG ASGs differently than self-managed ASG where module allows to pass target_group_arns and set it in LT

[daroga0002]

Because Managed Node groups are Managed and there is no such option over AWS EKS so we dont want to make something what is not supported by AWS itself as later we have problem with managing such configurations and etc.

[kkapoor1987]

@daroga0002 MNG Still allows us to pass LT which can be used to set target_group_arns. This would make target_group_arns exactly like we have for unmanaged ASGs

[daroga0002]

MNG Still allows us to pass LT

yup, so then use that approach which is valid one

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days

[antonbabenko]

This issue has been resolved in version 18.0.0 🎉

[mprzygrodzki]

@antonbabenko - marked as resolved, but I still don't understand the context how. I've same situation as @kkapoor1987 where mentioned before, we have deployed our own ALB and things related to it via TF - and now and in the near future there's no possibility to use mentioned AWS ALB Controller. I've also started to migrate our self managed workers to use managed worker groups directly from module in 18.7.2. So what's the final point for this topic ? Is there any possibility to pass target_group_arn somehow to managed node group ?

@kkapoor1987 - did you managed to pass that issue ? If so - would you be kind to show me how ?

[bryantbiggs]

@mprzygrodzki there aren't any options to pass target_group_arn anywhere - as far as I can tell, this is not supported in launch templates so I'm not sure how others were doing this previously (as mentioned above). The issue was closed due to stale activity, but I suspect because there was a reference of this issue in v18 somewhere, the release automation commented here (a bit of a fluke if you will).

That said, there isn't anything for us to do here in this module. Users can take the outputs and grab the autoscaling groups created and create the ALB attachments outside of this module

[mprzygrodzki]

@bryantbiggs - ok, thanks for clarification.

[kkapoor1987]

@mprzygrodzki I was able to make it work by following #1539 (comment)

Its not perfect but if you have ASG info then you can aws_autoscaling_attachment. It seems like module was creating attachments for self managed worker groups but not for managed worker group nodes. I think we are also moving away from aws_autoscaling_attachment and adopting alb controller to that as its much cleaner approach. Please, let me know if you still need help I can walk you through the aws_autoscaling_attachment hack we have in place

[mconigliaro]

It got easier in 18.x:

resource "aws_autoscaling_attachment" "this" {
  for_each               = toset(module.eks.eks_managed_node_groups["generic"].node_group_autoscaling_group_names)
  autoscaling_group_name = each.value
  alb_target_group_arn   = aws_lb_target_group.lb.arn
}

[mprzygrodzki]

@kkapoor1987 @mconigliaro - yes, i'm using similar - as far as I remember, the only thing which breaks a little this approach is that you have to create node groups first cause for_each is not waiting for them.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.