Too much rss memory after upgrading from jdk17 to jdk 21

[pax95]

After upgrading from jdk 17 to. jdk 21 we see increased rss memory usage compared to jdk 17 running on gke k8s.

This is a midsized spring-boot app that on jdk17 consumed around 400-> 500Mbi rss memory. Now it's about 750 Mbi.

I'm not able to see any GC activity releasing memory.
To me it seems that jdk21 has issues on k8s ?

The app is build using buildpacks with defaults, and started using ->

Jan 24 04:19:58 [epg-online-prod-prod](https://my.papertrailapp.com/systems/epg-online-prod-prod/events?focus=1686652011712102402&selected=1686652011712102402) [epg-online-epg-online-prod-282fs](https://my.papertrailapp.com/events?q=program%3Aepg-online-epg-online-prod-282fs&focus=1686652011712102402&selected=1686652011712102402) Setting Active Processor Count to 8
Jan 24 04:19:58 [epg-online-prod-prod](https://my.papertrailapp.com/systems/epg-online-prod-prod/events?focus=1686652011712102403&selected=1686652011712102403) [epg-online-epg-online-prod-282fs](https://my.papertrailapp.com/events?q=program%3Aepg-online-epg-online-prod-282fs&focus=1686652011712102403&selected=1686652011712102403) Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx201770K -XX:MaxMetaspaceSize=105429K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 800M, Thread Count: 250, Loaded Class Count: 16200, Headroom: 0%)
Jan 24 04:19:58 [epg-online-prod-prod](https://my.papertrailapp.com/systems/epg-online-prod-prod/events?focus=1686652011712102404&selected=1686652011712102404) [epg-online-epg-online-prod-282fs](https://my.papertrailapp.com/events?q=program%3Aepg-online-epg-online-prod-282fs&focus=1686652011712102404&selected=1686652011712102404) Enabling Java Native Memory Tracking
Jan 24 04:19:58 [epg-online-prod-prod](https://my.papertrailapp.com/systems/epg-online-prod-prod/events?focus=1686652011712102405&selected=1686652011712102405) [epg-online-epg-online-prod-282fs](https://my.papertrailapp.com/events?q=program%3Aepg-online-epg-online-prod-282fs&focus=1686652011712102405&selected=1686652011712102405) Adding 137 container CA certificates to JVM truststore
Jan 24 04:19:58 [epg-online-prod-prod](https://my.papertrailapp.com/systems/epg-online-prod-prod/events?focus=1686652011712102413&selected=1686652011712102413) [epg-online-epg-online-prod-282fs](https://my.papertrailapp.com/events?q=program%3Aepg-online-epg-online-prod-282fs&focus=1686652011712102413&selected=1686652011712102413) Spring Cloud Bindings Enabled
Jan 24 04:19:58 [epg-online-prod-prod](https://my.papertrailapp.com/systems/epg-online-prod-prod/events?focus=1686652011712102414&selected=1686652011712102414) [epg-online-epg-online-prod-282fs](https://my.papertrailapp.com/events?q=program%3Aepg-online-epg-online-prod-282fs&focus=1686652011712102414&selected=1686652011712102414) Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=8 -XX:MaxDirectMemorySize=10M -Xmx201770K -XX:MaxMetaspaceSize=105429K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

Currently is just increasing ->

[dmikusa]

Unfortunately, there's not a lot we can do to help with this. A major Java version upgrade can cause changes in behavior like this, and until there are more specific details and understandings of what is happening it's beyond the scope of what we as the Java Buildpacks team can help you investigate.

That said, We do provide some access to tooling in the JVM that can help debug memory issues. Specifically we enable Java NMT native memory tracing so you can track memory usage in your apps. By default it dumps NMT output when the app exits but you could trigger a manual dump with jcmd or jattach. We also make it easy to take flight recordings with JFR that can help you debug.

Also, Paketo provides Buildpacks that help integrate with several APM tools which might also be useful here.

If you are able to narrow this down and there is some specific thing the Buildpack is doing, like a JVM flag we're setting, that is causing problems we'd be happy to help with that.

[tata-dr-dk]

+1

I am observing similar behaviour, with a tiny Quarkus application having one camel route running a scheduled job each 30 minutes. After upgrading to jdk 21, memory usage has gone wild as can be seen below:

[dmikusa]

This isn't something that the Paketo buildpacks team can help you with. A major Java upgrade can have all sorts of changes, none of which are related to buildpacks. We just install the JVM for you.

You need to investigate your application and see where it is consuming memory specifically. When you know specifically what has changed, then you can start to look for causes. You might also want to look at the Java 21 release notes and see if there are known issues or changes in behavior that may contribute to what you're seeing.

I just want to be upfront here so you're not waiting/hoping this thread is going to provide you with a solution.

[edeandrea]

@franz1981 any thoughts?

[franz1981]

Very first stupid question:

• the problem existing even without NMT statistics on right?

Secondary, I will try to make sure if the used RSS is:

• heap peak mapped usage (eg some leak will cause heap expansions slowly and steady, increasing RSS)
• JVM related (compiler threads due to changes on bytecode compilation - the so-called new ClassFile-API)
• (other uncknown)

To verify which of these 3 is responsible, ideally, you would use async-profiler -e mprotect which sadly require high-level privileges; I suggest to verify than if the issue exist even outside containers, which will make far-more easier to troubleshoot and observe using tools like the one I've mentioned (more info on it at quarkusio/quarkus#36204, which indeed was luckly on bare-metal and easier to use low-level tools to figure it out what's going on).

If using async-profiler isn't an option, NMT is indeed the way to go and collect snapshots to understand what could be the part of the JDK (if any) which can be responsible and has increased it's usage.

A "cheap trick" to see if things could be magically improved is to enable autotrimming (see redhat-developer/vscode-java#3395 (comment)) and eventually G1 intervals pauses, but both won't tell what's going on, just would likely workaround, if works, the potential regression in production (if any) till a proper investigation is completed.

Maybe unrelated, but ss a personal preference, I would be very scared to configure things like MaxMetaspaceSize/ReservedCodeCacheSize/MaxDirectMemorySize for few reasons:

• wrong metaspace sizing can cause deflation loops of reflective requests, turning them to be super slow and maybe consuming again and again compiler native memory to re-inflate them (which happen every each 14 reflective invocation)
• ReservedCodeCacheSize could cause the JIT compiler to be disabled (!!!): if a warning like that is missed, you end up with bad performance without failing-fast...
• MaxDirectMemorySize: it can have some interesting adverse effects if Netty is in the usage path and, for some reason, cannot use Unsafe to allocate its Arena (OODirectMemory thrown, which is not as OOM, and will let applications to be living zombies...)

[pax95]

@franz1981 We haven't tried running without NMT statistics. We could try that.
We would aslo be able to gather some NMT snapshots over a period for troubleshooting.
First we need to setup a isolated workload and reproduce the problem here. BBL

[pax95]

Thanks @dmikusa
I'm aware of that Paketo buildpacks proberly is not the right place to discuss this, but maybe this thread will help others.

Currently all hands are on downgrading to jdk 17, where things works as expected. See rss bellow.

If time allows I will investigate the issue later.

BTW I wonder if the memory settings or availableProcessors settings calculated by the memory calculator has any influence.
And we have only seen this issue when running on k8s.

[dmikusa]

Until there's more detail, it's hard to say and I don't want to speculate. If there are issues with something the buildpacks are doing, rest assured, we'll engage and get things corrected.