[BUG]binary-dedup.sh script fails on mac

[tgravescs]

Describe the bug
looks like xargs on mac doesn't support --arg-file option:

     [exec] + xargs --arg-file=/Users/tgraves/Documents/workspace/spark-rapids/dist/target/spark3xx-common.txt -P 6 -n 1 -I% bash -c 'remove_duplicates "$@"' _ %
     [exec] xargs: illegal option -- -
     [exec] usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements]] [-J replstr]
     [exec]              [-L number] [-n number [-x]] [-P maxprocs] [-s size]
     [exec]              [utility [argument ...]]

line: xargs --arg-file="$SPARK3XX_COMMON_TXT" -P 6 -n 1 -I% bash -c 'remove_duplicates "$@"' _ %

instead of using the --arg-file option I think we can just cat the the file and | it into xargs

Note there is a line at the end that uses it too:

xargs --arg-file="$UNSHIMMED_LIST_TXT" -P 6 -n 100 -I% \
But that didn't fail so we probably have a bug where it should fail.

[gerashegalov]

What's your MacOS version?

on 11.6 (Big Sur), the first failure I encountered is even earlier than --arg-file
we need to use shasum instead of sha1sum on Mac

      [exec] xargs: sha1sum: No such file or directory

then I see the the --arg-file error

then the build fails for me though as expected

    [exec] + xargs --arg-file=/Users/user/gits/NVIDIA/spark-rapids/dist/target/spark3xx-common.txt -P 6 -n 1 -I% bash -c 'remove_duplicates "$@"' _ %
     [exec] xargs: illegal option -- -
     [exec] usage: xargs [-0opt] [-E eofstr] [-I replstr [-R replacements] [-S replsize]]
     [exec]              [-J replstr] [-L number] [-n number [-x]] [-P maxprocs]
     [exec]              [-s size] [utility [argument ...]]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Distribution 21.12.0-SNAPSHOT:
[INFO] 
[INFO] RAPIDS Accelerator for Apache Spark Distribution ... FAILURE [ 15.119 s]
[INFO] RAPIDS Accelerator for Apache Spark UDF Examples ... SKIPPED
[INFO] RAPIDS Accelerator for Apache Spark Tests .......... SKIPPED
[INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
[INFO] RAPIDS Accelerator for Apache Spark Tests For 3.1.X+ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.266 s
[INFO] Finished at: 2021-10-14T12:39:58-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (create-parallel-world) on project rapids-4-spark_2.12: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="/Users/user/gits/NVIDIA/spark-rapids/dist/target" executable="/Users/user/gits/NVIDIA/spark-rapids/dist/scripts/binary-dedupe.sh"/>... @ 14:174 in /Users/user/gits/NVIDIA/spark-rapids/dist/target/antrun/build-main.xml

[gerashegalov]

and once I fix it it's incredibly slow, probably need to just compile a single list of files to delete in larger batches

[tgravescs]

yes I had to install sha1sum as well, but I figured that was env setup like install maven. we may want to document but should just be brew install md5sha1sum

[tgravescs]

versions fail on Mojava 10.14.6 and Catalina 10.15.7

[gerashegalov]

do you see shasum on Mojava and Catalina?
it was already there on Big Sur

which shasum
/usr/bin/shasum

so we could just automatically pick it on Mac without having to document an additional download

[tgravescs]

yes /usr/bin/shasum exists on both.