ijar tool fails with segmentation fault

[sholokhov]

Description of the problem / feature request:

I'm trying to migrate from bazel 4.2.2 to bazel 5.0.0 on Scala monorepo (with rules_scala) and started to see the following error:

ERROR: /home/sholokhov/code/xite/3rdparty/jvm/com/github/pureconfig/BUILD:20:13: StampWithIjar 3rdparty/jvm/com/github/pureconfig/pureconfig_core.stamp/com_github_pureconfig_pureconfig_core_2_13-stamped.jar failed: (Segmentation fault): ijar failed: error executing command 
  (cd /home/sholokhov/.cache/bazel/_bazel_sholokhov/d1dee1b302471130bebbc47f621d2c9b/sandbox/linux-sandbox/58/execroot/xite && \
  exec env - \
  external/remote_java_tools_linux/java_tools/ijar/ijar --nostrip_jar --target_label //3rdparty/jvm/com/github/pureconfig:pureconfig_core external/com_github_pureconfig_pureconfig_core_2_13/jar/com_github_pureconfig_pureconfig_core_2_13.jar bazel-out/k8-fastbuild/bin/3rdparty/jvm/com/github/pureconfig/pureconfig_core.stamp/com_github_pureconfig_pureconfig_core_2_13-stamped.jar)
# Configuration: 9f72dbfc6067cba8c7182abbe35db7759aa1626e954083725070236a790fcf64
# Execution platform: @local_config_platform//:host

Looks like ijar fails with segfault during jar stamping step.
It works perfectly fine with bazel 4 series (tested on 4.2.1 and 4.2.2).

What operating system are you running Bazel on?

Linux nzxt 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

What's the output of bazel info release?

release 5.0.0

[cushon]

There was another report of this in slack

Do you have a repro, or can you share the jar it's crashing on in the bug?

[sholokhov]

Apparently this is a problem for specific jar (pureconfig), I was able to find similar issues for the same Scala artifact:

• Segmentation fault while calling singlejar_local with pureconfig-core_2.12-0.14.1.jar #13943 (comment)
• pureconfig-core_2.12-0.14.1.jar might have zip issue pureconfig/pureconfig#1142

Its weird that it has been working for us on bazel 4.x and earlier but started to break on 5.x. Closing this as a duplicate of #13943.

[ahmedyarub]

It's not just for a specific jar. I can see a lot of other complaints about some random jars failing including us. I have some jars that I need urgently for my project and I can extract them on MacOS without any problem, but when used as a part of build I get that segmentation fault.

[cushon]

Do you have a repro, or can you share one of the jars it's crashing on in the bug?

[ahmedyarub]

We build some internal plugins that use compiled versions of IntelliJ-Community edition compiled by us. We have been doing this for years without any problem. Then suddenly when upgraded to Java 17 + latest EAP the files started throwing these errors. Using --nous_ijars mitigated the problems but that is not really a solution. I'll consult my company and try to get you binaries and/or scripts.

[cushon]

There were some other reports of ijar segfaults here:

• Segmentation fault while calling singlejar_local with pureconfig-core_2.12-0.14.1.jar #13943 (comment)
• Segmentation fault while calling singlejar_local with pureconfig-core_2.12-0.14.1.jar #13943 (comment)
• Segmentation fault while calling singlejar_local with pureconfig-core_2.12-0.14.1.jar #13943 (comment)

For anyone seeing this, can you share one of the jars ijar is crashing on in the bug?

[youmna-rabie]

@cushon I'm seeing this
ERROR: /private/var/tmp/_bazel_youmna.rabie/a3f73e9053c6872ac891025dc8e34184/external/commons_codec_commons_codec/jar/BUILD:3:12: Extracting interface @commons_codec_commons_codec//jar:jar failed: (Segmentation fault): ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar external/commons_codec_commons_codec/jar/commons_codec_commons_codec.jar ... (remaining 3 arguments skipped)

On

Build label: 5.1.1
Build target: bazel-out/darwin_arm64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Apr 8 15:58:49 2022 (1649433529)
Build timestamp: 1649433529
Build timestamp as int: 1649433529

[cushon]

@youmna-rabie can you share one of the jars ijar is crashing on, perhaps by attaching it to a comment on this bug?

[youmna-rabie]

@cushon I'm not able to share a JAR on here as it's not a supported file type, but here's a Google Drive link with the jar folder.

[cushon]

I can reproduce with the version of ijar in Bazel 5.1.1, but not with the latest versions of ijar, this bug seems to have been fixed. Can you try using a newer version of Bazel?

[youmna-rabie]

Do you know the lowest version of bazel that would have fixed this? This is a legacy code base that isn't easily upgrade-able unfortunately :/

[cushon]

The ijar fix was 3954a18

I think Bazel 7.x is the first version to include that, specifically in 9f0b235

[cushon]

Actually 6.3.0 also had the fix: dc1c7b3

[youmna-rabie]

Thanks, upgrading to 6.3.0 resolved it!