Error: ENOSPC: System limit for number of file watchers reached

[ismailyenigul]

What happened:
I can't build a react native app in the pod runs on EKS cluster 1.6
I got Error: ENOSPC: System limit for number of file watchers reached due small value of fs.inotify.max_user_watches

What you expected to happen:
Increases fs.inotify.max_user_watches = 8192 value to something bigger

Anything else we need to know?:

Environment:

• AWS Region:
  eu-west-1

• Instance Type(s): m5.large

• EKS Platform version 1.16

• AMI Version: AMI ID amazon-eks-node-1.16-v20200609 (ami-0a3879f5c5e608624)

• Release information (run cat /etc/eks/release on a node):

[ajcann]

We're seeing this as well.

This looks like it may be a resurfacing of
#278
#233

In the us-west-2 EKS-optimized GPU AMI - ami-0489da41209229234 - we are seeing:

[ec2-user ~]$ cat /etc/sysconfig/docker
# The max number of open files for the daemon itself, and all
# running containers.  The default value of 1048576 mirrors the value
# used by the systemd service unit.
DAEMON_MAXFILES=1048576

# Additional startup options for the Docker daemon, for example:
# OPTIONS="--ip-forward=true --iptables=true"
# By default we limit the number of open files per container
OPTIONS="--bridge=none --default-ulimit nofile=2048:8192 --log-driver=json-file --log-opt max-size=10m --log-opt max-file=10 --live-restore=true --max-concurrent-downloads=10"

# How many seconds the sysvinit script waits for the pidfile to appear
# when starting the daemon.
DAEMON_PIDFILE_TIMEOUT=10


[ec2-user@ip-10-52-46-213 ~]$ cat /etc/sysconfig/docker
# The max number of open files for the daemon itself, and all
# running containers.  The default value of 1048576 mirrors the value
# used by the systemd service unit.
DAEMON_MAXFILES=1048576

# Additional startup options for the Docker daemon, for example:
# OPTIONS="--ip-forward=true --iptables=true"
# By default we limit the number of open files per container
OPTIONS="--bridge=none --default-ulimit nofile=2048:8192 --log-driver=json-file --log-opt max-size=10m --log-opt max-file=10 --live-restore=true --max-concurrent-downloads=10"

# How many seconds the sysvinit script waits for the pidfile to appear
# when starting the daemon.
DAEMON_PIDFILE_TIMEOUT=10


[ec2-user ~]$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

And for the non-gpu instances we are seeing 8192 for fs.inotify.max_user_watches though ulimit is unlimited and the docker sysconfig file has no mention of ulimits.

[itssimon]

Hitting the same issue with fs.inotify.max_user_watches being set to only 8192 with the non-GPU AMI amazon-eks-node-1.16-v20200423 (ami-04889bf8ec2eaeeac).

[Jeffwan]

Seems this happens frequently in some web servers.

We currently don't have specific setting for fs.inotify.max_user_watches. Thinking a more flexible way to handle this. Seems only docker run has --sysctl option support and max_user_watches are not part of it. We probably need a change at AMI level to better support this. I can file a PR later.

In the meanwhile, you can check following solution to mitigate the issue. sysctl settings are usually global kernel-level settings and containers share the host's kernel. they usually share the same sysctl values.

# Before
sudo docker run -it --rm ubuntu bash -c "cat /proc/sys/fs/inotify/max_user_watches"
8192

# our mitigation 
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# After
sudo docker run -it --rm ubuntu bash -c "cat /proc/sys/fs/inotify/max_user_watches"
524288

[ismailyenigul]

Hi @Jeffwan

I am using AWS EKS 1.16 and EKS node max_user_watches value is 8192 and I am getting the following value on AMI

# sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

If I increase this value on EKS worker node, the pods can see this value. so there is nothing to do with docker. It should be fixed on AMI

[Jeffwan]

Right, container inherit node setting for this sys setting. Are you ok with custom change by your own or you think it's better to come up a generic way to configure when we build AMI, for example, pass options etc to make it customizable? @ismailyenigul

[ismailyenigul]

I think the latter option is better. I can do it my own but this is not required only by me.
Maybe we can set the variables via userdata commands.
GKE can pass custom linux-sysctl variables with the following option

-linux-sysctls=KEY=VALUE,[KEY=VALUE,…]
Linux kernel parameters to be applied to all nodes in the new cluster's default node pool as well as the pods running on the nodes.
Example:

gcloud alpha container clusters create example-cluster --linux-sysctls="net.core.somaxconn

https://cloud.google.com/sdk/gcloud/reference/alpha/container/clusters/create

[jakegibson]

Hitting this as well on EKS 1.18 - t3.2xlarge nodes

[medkbadri]

Hi,
Is it safe to increase fs.inotify.max_user_watches on host
What are the costs of increasing it?

[elgalu]

is this fixed?

[cartermckinnon]

I'm seeing the following on the latest AMI's:

fs.inotify.max_user_watches=524288

So I'm going to close this. Let us know if this remains an issue.