Sbt-assembly is using lots of native memory

[ashangit]

Issue description

On some of our build relying on sbt-assembly to create shaded jar we have observed big usage of native memory.
For a 6G max heap JVM, the pod memory usage is above 12G

Analysing the native usage with libtcmalloc_and_profiler.so.4 and google-pprof we observed that the native usage is due to native usage of Inflater used to unzip jar:

Src code ref of the native inflater code: Java_java_util_zip_Inflater_init and Java_java_util_zip_Inflater_inflateBytesBytes

Proposal

Current code keep all jar file handler which leads to all native inflater being kept with all the memory those are using

In order to avoid this, the proposal is to load all entry data from each jar as ByteArrayInputStream instead of keeping the stream and immediately close the jar once all entries are loaded

Here the native memory usage after this change

The counterpart is an increase of the heap usage.
On our specific build we had to increase heap size to 7G but pod memory usage has decreased below 10G:

[eed3si9n]

I don't know if this is a good idea. Maybe it would be better to create a setting to tweak the parallelism of .par to reduce the concurrent calls on very big JARs?

[ashangit]

Updating parallelism of .par should not change the native memory usage as we will still keep all those jar handler

[eed3si9n]

I guess you're right. The PR as it stands requires heap equivalent to all JAR file size right?

[ashangit]

No it will store the inflated data for each entries of each jar. From heapdump on our build job it represent around 5GB of byte[] while using the "stream" version it use around 3.5GB of byte[]. Stream version also used lots of byte[] because even if it is a stream the stream as already load/initiate a chunk of bytes per entry which seems to represent a big percent of an entry (probably because each of them are mostly only little text file).
Another approach would be to not load the stream in https://github.com/sbt/sbt-assembly/blob/develop/src/main/scala/sbtassembly/Assembly.scala#L304. Just get needed information for each entry and close the jar file handler immediately
And load the stream only in createJar.
This would reduce the heap used (even with current release) and reduce off heap size
Do you think it is possible or there are some other usage of the stream that would be uneasy to extract?

[eed3si9n]

Jar Jar Abrams needs to read the streams to process various rules against the stream of bytecode I think. Similar to what it's doing () => jarFile.getInputStream(entry) a lazy function that creates a stream on demand, I guess we can further make it lazier to provide:

name => {
  val jarFile = lookupJarFile(jar.data)
  val entry = jarFile.getEntry(name)
  jarFile.getInputStream(entry)
}

where we can keep a LRU cache of JAR files in lookupJarFile(...)?

[ashangit]

The shader that rely on this jarjarabrams Shader seems to already load the whole stream into a ByteArrayInputStream
From jarjar-abrams bytecodeShader signature: https://github.com/eed3si9n/jarjar-abrams/blob/develop/core/src/main/scala/com/eed3si9n/jarjarabrams/Shader.scala#L71

I'm not sure to see the gain in keeping the stream as we will load all the bytes in the shader. Also wondering if we rely on such mechanism with LRU cache if we could not reach some bugs with LRU cache closing the file handler while still having an input stream open from it