Max pids is set to 32768

[dza89]

What happened:

Nodes crashed because we didn't set a max pids per pod.

What you expected to happen:

Question is why this is set to 32768? Which seems to be the max value for 32-bit.
Can we raise it? I'll make the PR.

How to reproduce it (as minimally and precisely as possible):

Run a lot of Java :')

[cartermckinnon]

I don't have any concerns with increasing this value, because the current default is mostly arbitrary. However, we don't want to mask issues such as DataDog/datadog-agent#12997, so we shouldn't increase it too dramatically.

@dza89 for your use-case, what value has proved stable? Increasing by a factor of 2 seems like a good first step.

[dza89]

We actually hit the same issue as datadog in kube. We implementated a pod pid limit to make sure it doesn't happen again.
But even so, 32768 is just really low, I just checked my arch machine and it's 4194304. We raised it to 262144, but that value is completely arbitrary.

[cartermckinnon]

Oh interesting, it sounds like Arch is just using the maximum possible value. From proc manual:

/proc/sys/kernel/pid_max (since Linux 2.5.34)
This file specifies the value at which PIDs wrap around
(i.e., the value in this file is one greater than the
maximum PID). PIDs greater than this value are not
allocated; thus, the value in this file also acts as a
system-wide limit on the total number of processes and
threads. The default value for this file, 32768, results
in the same range of PIDs as on earlier kernels. On
32-bit platforms, 32768 is the maximum value for pid_max.
On 64-bit systems, pid_max can be set to any value up to
2^22 (PID_MAX_LIMIT, approximately 4 million).

I think I've decided this is the right approach; if we "remove" this limit, users can manage PID usage entirely with Kubernetes via Pod PID limits.

[dza89]

Nice to hear, the only thing I just realize, is the eks ami build the right place to set this? Shouldn't this be fixed at the source? amzn2-ami-minimal-hvm?

[tzneal]

To offer a contrarian point of view, with no limit I think you'll end up seeing machines hard-lock rather than error about being unable to fork/exec which is harder to diagnose.

Just looking for guidance elsewhere, the RHEL 9 STIG recommends it be set to 65536:

https://static.open-scap.org/ssg-guides/ssg-rhel9-guide-index.html

[cartermckinnon]

I think it makes sense to configure it here, because the AL2 AMI is not assuming Kubernetes usage (and it is late in its support cycle).

@tzneal the failure mode would change, for sure. I think we'd see physical resource exhaustion (which users can obviously already run into and need to monitor for, control with resource limits, etc.).

Looks like the value seen in an Arch install is likely a result of systemd version 243, which changed the default to the maximum value: https://github.com/systemd/systemd/blob/d08ff4506c2989abc0afb9f7d0912d35a8d23da2/NEWS#L6483-L6498. I believe AL2022 is on systemd 244, so I'll check what limit is used there. Also probably worth checking Bottlerocket.

[bbende]

Is there any update on this issue? has it already been addressed, or is it planing to be? Thanks

[gpchelkin]

Some more guidance:
https://kubernetes.io/docs/concepts/policy/pid-limiting/

Note: On certain Linux installations, the operating system sets the PIDs limit to a low default, such as 32768. Consider raising the value of /proc/sys/kernel/pid_max.

[bkosaraju]

32bit Max limit - 32768
64bit Max limit - 4194304

In case if any one after this, having this set at user data block will give temporary fix before the AMI patched -

echo 2194304 > /proc/sys/kernel/pid_max

[sfc-gh-jpollard]

We (at Snowflake) are now including that patch/work-around in our userData block, but we'd very much appreciate if this could be baked-in by default for 64 bit AWS instance!

[dims]

@sfc-gh-jpollard wanna drop a quick PR?

[bryantbiggs]

FWIW - on AL2023 the max pids are set at 4194304 so this would only apply to AL2