[EKS] [Fargate]: Add ability to know which fargate combination resource is chosen to a POD

[nbluis]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?
The ability to know for each POD which combination of fargate resources has been choosen.

Which service(s) is this request for?
EKS Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
According the EKS Fargate documentation and here.
Fargate will choose the properly (rounded up) combination resources for the POD according requested resources of all containers inside the POD
(I am intentionally abstracting the details about init-containers and long-time-running-containers but this information is counted in detail in the documentation if necessary.)

The problem that occurs and especially in complex PODs (with many containers) is that it is currently really difficult to predict which combination of resources was selected when deploying the POD. This makes pretty impossible to predict or even know the monthly cost of a deployment within a fargate cluster, since billing only accumulates total memory and cpu used by fargate.

If I make a kubectl get nodes I retrieve the list of nodes from all the PODs in the fargate. However, the resources described by kubectl describe node node_name do not match the combinations described in the documentation in any way.

Ex. I have pods that have no definition of resources as coredns (it is assumed that they should use the lowest existing combination 0.25 CPU + 0.5 Mem) but the node presents me with 2CPUs + ~4GBMem for each coredns POD. (I am assuming it was this combination chosen by Fargate and this information is not true)

It is extremely important to know how much each deployment costs, mainly to be able to make a correct decision if I will deploy using fargate or node_groups.

If we can know for each POD (or fargate node) which resource combination was chose (maybe with a custom label) it would allow us to know how much each POD of a service costs monthly.

Are you currently working around this issue?
No

Additional context
No

Attachments

[ghdiska]

We are extensively using EKS on Fargate in prod and non-prod environments but we can only guess the final Fargate size according to the configurations matrix provided by the public documentation. Given the various vCPU/memory combinations we are using, a section of the EKS UI or an API to call would be useful to know the final assigned sizing by the service and then estimate the expected pricing.

[Sceat]

@nbluis Thanks for this issue i was becoming crazy by not finding anything on the topic, it is pretty critical for a small startup to estimate the infrastructure cost, i was pretty scared seing this below even with properly configured low requests! Does that mean it is not related to fargate selected combination ?

This is also confusing, how can we be sure that we will not pay more than requested, pretty ambiguous to consume 5% of something made to fit exactly

[mikestef9]

We recently published a blog with more information on how to monitor metrics on EKS/Fargate, including details on how a fargate task size is chosen

https://aws.amazon.com/blogs/containers/monitoring-amazon-eks-on-aws-fargate-using-prometheus-and-grafana/

[TBBle]

It'd be really handy if the EKS Fargate support used something like Virtual Kubelet so that Fargate looked like a normal node (albeit with dynamic capacity!) so that existing monitoring setups and expectations worked. (Edit: I realised this was a more-general idea, so raised a separate request for this)

In the meantime, perhaps a custom Prometheus metrics endpoint to wrap-up and expose all those calculations decribed in the blog-post would support at least the monitoring use-cases, even if the Node object itself is going to be incorrect.

[tlopo]

Has anyone ever managed to schedule a pod that uses less than 2CPUs and 4G Memory? No matter how little resources I specify in the pod definition I end up with 2CPUs 4G Memory, is this an undocumented (or perhaps documented but I failed to find) minimum requirement?

[booleanbetrayal]

Also very curious about this and am beginning to wonder if we're being mis-billed according to the pricing documentation we've read. Some transparency over this process would be appreciated.

[ghdiska]

In some case ( ex old java ) pod view "fargate node resource" and not the resources provisioned. with the latest eks you can see pod
Annotations: CapacityProvisioned: 0.25vCPU 0.5GB

[mreferre]

As @ghdiska already noted we have recently added an annotation to the pod that allows you to easily get the size of the Fargate task that was picked. Before this annotation was available the customer did not have a way to know (from us) what that size was. It is important to understand though that the size you see in the "worker nodes" is not what you are billed for (and not even the capacity you are entitled to use because that is capped by the cgroup we create to map the task size we picked). This obviously means customers are not (and were not) over-billed. What may happen is that we may pick an instance that happens to be arbitrarily larger than the pod/task size requested. Given this was the only size visible to the user this was obviously and understandably creating confusion (hence the addition at the pod level of the new annotation).

HTH.