Kafka Streams applications keep getting OOMKilled

[ngc4579]

We are in the process of establishing a CDC pipeline involving Kafka, Kafka Connect and a bunch of Kafka Streams applications on top of a (managed) Kubernetes 1.29 cluster with Flatcar operated nodes (version 3815.2.5).

For a long time we've had lots of issues with Kafka Streams applications running out of memory, eventually being OOMKilled when hitting the configured Pod limits. This happened even with very conservative JVM / off-heap settings with lots of headroom, yet somehow none of our configurations ever solved the issue. Memory consumption of the Pods kept slowly increasing (over days) until hitting the limits.

Now configuring Kafka Streams applications at times can be a tricky task as it involves JNI components (RocksDB) that are not managed by JVM and utilize their own configuration. But even with RocksDB off-heap memory strictly bounded and using an alternative malloc implementation (jemalloc) to avoid fragmentation, the issue persisted...

... Until just recently we changed the node OS to Ubuntu - and suddenly the applications seem to be run much more stable, we had no OOMKilled events since the change.

Now I wonder if there is anything in the underlying Flatcar architecture that might contribute to this situation, and whether this would be the place to track potential issues down. E.g. are there circumstances regarding memory management that might lead to incompatibilities with jemalloc (or tcmalloc as well) that might render those non-functional? In fact the behaviour might look like some kind of memory leak, yet there are no such (known) issues with neither Kafka Streams libraries nor RocksDB, and I tend to think this might be more an issue of memory fragmentation (?) not being taken care of sufficiently - even more because on an alternative OS the issues seems to be gone magically.

I'd be happy to provide additional information if needed, as otherwise we are perfectly satisfied with Flatcar as the driver of our worker nodes (and are even using it on other worker groups not related to Kafka workload).

[tormath1]

Hello and thanks for sharing this experience with us. How did you install kubernetes components on Flatcar vs Ubuntu? First thing that comes to my mind is that systemd-oomd is enabled by default on Ubuntu and not on Flatcar. That could explain why applications are OOMKilled in a first place. What is your container runtime interface?

[ngc4579]

Well actually I have no real insight into how K8s components are installed on the underlying nodes as it's a managed K8s distribution. As far as I know, things are provisioned using Terraform, but I've got no further config options than choosing from Flatcar / Ubuntu as node OS.

Apart from that - systemd-oomd seems to be inactive on both variants.

Ubuntu:

# systemctl status systemd-oomd
Unit systemd-oomd.service could not be found.

Flatcar:

# systemctl status systemd-oomd
○ systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer
     Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; disabled; preset: disabled)
     Active: inactive (dead)
TriggeredBy: ○ systemd-oomd.socket
       Docs: man:systemd-oomd.service(8)
             man:org.freedesktop.oom1(5)

Container runtime is containerd in both cases.

[ader1990]

I could find a difference in Flatcar vs Ubuntu (in my setups, would be great to get a confirmation):

# Flatcar
sysctl vm.overcommit_memory
vm.overcommit_memory = 1

# Ubuntu
sysctl vm.overcommit_memory
vm.overcommit_memory = 0

See: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting

It might be that the Kafka/Java knows that the Ubuntu might not let it overcommit and its memory cleanup is less active?

[ngc4579]

Hm, interestingly vm.overcommit_memory is the same on both platforms.

Flatcar:

root@shoot--420214--<redacted>-z1-74f67-4wdd4:/# sysctl vm.overcommit_memory
vm.overcommit_memory = 1

Ubuntu (22.4.4):

shoot--420214--<redacted>-z1-8659c-klfxz / # sysctl vm.overcommit_memory
vm.overcommit_memory = 1