Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] compressed event logs are not processed #2622

Closed
gerashegalov opened this issue Jun 7, 2021 · 1 comment · Fixed by #2626
Closed

[BUG] compressed event logs are not processed #2622

gerashegalov opened this issue Jun 7, 2021 · 1 comment · Fixed by #2626
Assignees
Labels
bug Something isn't working

Comments

@gerashegalov
Copy link
Collaborator

Describe the bug
The profiling tool path filter out file names containing a . . Thus lz4 event log files produced with the config spark.eventLog.compress=true which is standard for production deployments are skipped. The remainder of the logic relying on the SHS event log reader supports lz4 (default), snappy, zstd , lzf

Steps/Code to reproduce bug
Run some integration tests e.g.:

 SPARK_HOME=~/dist/spark-3.1.1-bin-hadoop3.2 \
 NUM_LOCAL_EXECS=0 \
./integration_tests/run_pyspark_from_build.sh -k large_orderby_nested

This will generate compressed event logs under integration_tests/target/run_dir/eventlog_gw0

Then run:

~/dist/spark-3.1.1-bin-hadoop3.2/bin/spark-submit --class com.nvidia.spark.rapids.tool.profiling.ProfileMain ~/gits/NVIDIA/spark-rapids/tools/target/rapids-4-spark-tools_2.12-21.08.0-SNAPSHOT.jar -o /tmp/prof1/ ~/gits/NVIDIA/spark-rapids/integration_tests/target/run_dir/eventlog_gw0

The tool silently quits: I think we need an actionable log message for legitimately empty input directories

Expected behavior
The tool should generate reports for SHS-supported compression codecs. I verified that it works if you modify the filter https://github.com/NVIDIA/spark-rapids/blame/3e303c0dae26d1a8ef01fa271ee5f0f43759e865/tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileMain.scala#L73

and

https://github.com/NVIDIA/spark-rapids/blame/3e303c0dae26d1a8ef01fa271ee5f0f43759e865/tools/src/main/scala/com/nvidia/spark/rapids/tool/profiling/ProfileMain.scala#L91

Environment details (please complete the following information)
local dev

Additional context
N/A

@gerashegalov gerashegalov added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jun 7, 2021
@gerashegalov gerashegalov self-assigned this Jun 8, 2021
@gerashegalov gerashegalov removed the ? - Needs Triage Need team to review and classify label Jun 8, 2021
@gerashegalov gerashegalov added this to the June 7 - June 18 milestone Jun 8, 2021
@gerashegalov
Copy link
Collaborator Author

Fixed but not auto-closed by #2626

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant