[EKS][Fargate] [request]: Increase the ulimit for EKS Fargate

[abhishek181]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Increase the ulimit hard limit for the EKS Fargate

Which service(s) is this request for?
EKS, Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Currently the hard limit for the ulimit is set to 4096 for a pod deployed in EKS Fargate. So, the soft limit of ulimit can only be set to the hard limit and it cannot be increased than the hard limit. Sometimes, there can be a requirement for higher ulimit than 4096.

Are you currently working around this issue?
No workaround as of now.

[casret]

ulimit -n unlimited also raises an error instead of going to the hard limit, which is annoying.

[booleanbetrayal]

Also experiencing this on our EKS Fargate cluster, leaving us to even wonder if this is a Fargate deal-killer. Meanwhile, it would be nice to have some sort of interface in order to take the ambiguity out of these limits; something AWS could respect - kubernetes/kubernetes#3595

[mlanner-aws]

Just to follow up on this one, is it nofile in particular you are concerned with or are there other limits that are of concern?

[casret]

For me nofile. Just got a EMFILE error this morning in fact.

[booleanbetrayal]

@mlanner-aws - Definitely hitting nofile issues right off the bat with the 1024 soft / 4096 hard limits. Haven't had a chance to test any other constraints, but i suspect we would want to uncap memlock from the current 64k defaults as well.

[booleanbetrayal]

@mlanner-aws - Any ETA on getting this fixed in the Fargate EKS worker node AMIs? Can the nofile limit be set to the value of sysctl fs.file-max ?

[mlanner-aws]

No ETA yet, we're still discussing and prioritizing. It's unlikely we'll be able to get to this in 2020.

[booleanbetrayal]

@mlanner-aws - I want to believe that EKS Fargate is a viable product offering, but right now it feels like it isn't. I'm sure that an arbitrary hard-cap of 4096 nofile breaks significant numbers of prospective workloads right out the gate. Given the AMI history behind ulimit, I'm wondering what sort of quality control goes into the EKS Fargate product line, at this point. Finally, the fact that it may potentially take several months to update approximately 10 lines of code, makes me wonder what we'd be paying for, and whether or not it would just be reverted again in the future. If there is an escape hatch that I'm unaware of, please let me know. As it stands, we are in the process of evaluating potential alternatives.

PS - Shamelessly leveraging my Brent-powers to (hopefully) find an advocate in @brentley

[wshamim1]

+1

[jsidhu]

The hard limits for nofile and nproc have been raised to 65535.

Hard limit:

# ulimit -Ha
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        10240
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) unlimited
process              65535
nofiles              65535
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

Soft limit

# ulimit -a
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        10240
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) unlimited
process              1024
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
rtprio               0

[booleanbetrayal]

Thank you @jsidhu !

[styfle]

I'm still seeing the 4096 nofile limit for ECS Fargate.

[jsidhu]

For ECS Fargate, you will need to edit your task definition and provide the relevant config for limits

https://docs.aws.amazon.com/AmazonECS/latest/userguide/task_definition_parameters.html#container_definition_limits

"ulimits": [
      {
        "name": "core"|"cpu"|"data"|"fsize"|"locks"|"memlock"|"msgqueue"|"nice"|"nofile"|"nproc"|"rss"|"rtprio"|"rttime"|"sigpending"|"stack",
        "softLimit": integer,
        "hardLimit": integer
      }
      ...
    ]

[SaloniSonpal]

Now the nofile and nproc soft limit is 1024 and the hard limit is 65535 for EKS/Fargate pods.
https://docs.aws.amazon.com/eks/latest/userguide/fargate.html#fargate-considerations

[tomaszdudek7]

What is the maximum nofile for ECS Fargate? Is it 65535 as well? I can't find the exact value anywhere in the docs. All I see concerns EKS/Fargate and I am looking for ECS limits.

[danielcompton]

@tomaszdudek7 from Task Definition Parameters - Resource Limits

Amazon ECS tasks hosted on Fargate use the default resource limit values set by the operating system with the exception of the nofile resource limit parameter which Fargate overrides. The nofile resource limit sets a restriction on the number of open files that a container can use. The default nofile soft limit is 1024 and hard limit is 1048576.

I assume that 1048576 is also the maximum?

[sunnynazar]

@danielcompton - this is bit confusing. This link tells default limits differently at least for hard limit.
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Ulimit.html

Amazon ECS tasks hosted on AWS Fargate use the default resource limit values set by the operating system with the exception of the nofile resource limit parameter which AWS Fargate overrides. The nofile resource limit sets a restriction on the number of open files that a container can use. The default nofile soft limit is 1024 and hard limit is 4096.

I assume that maximum hard limit can be set to 1048576

[ngoyal16]

@SaloniSonpal can you please share some example of how to increase in case of eks.. because not able to find any relevant information on internet.

[gain620]

@SaloniSonpal can you please share some example of how to increase in case of eks.. because not able to find any relevant information on internet.

@ngoyal16 unfortunately i don't think it is possible to increase on eks :(

[gbajson-area22]

Now the nofile and nproc soft limit is 1024 and the hard limit is 65535 for EKS/Fargate pods. https://docs.aws.amazon.com/eks/latest/userguide/fargate.html#fargate-considerations

In the following doc it's only mentioned that "The default nofile and nproc soft limit is 1024 and the hard limit is 65535 for Fargate Pods.".

Though, It's not clear if it's possible to change it.

[parthbulusu]

I tried multiple ways to increase the soft limit using k8s deployment yml as described in the following
https://repost.aws/knowledge-center/eks-fargate-increase-ulimit-values
but nothing worked! Please help if there is a solution.