Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] test event logs should be compressed #2644

Closed
rongou opened this issue Jun 8, 2021 · 1 comment
Closed

[BUG] test event logs should be compressed #2644

rongou opened this issue Jun 8, 2021 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@rongou
Copy link
Collaborator

rongou commented Jun 8, 2021

Describe the bug
We've merged a couple test event log files that are pretty big:

ls -lh tools/src/test/resources/spark-events-profiling
total 103M
-rw-rw-r-- 1 rou rou 343K Jun  8 10:52 eventlog_minimal_events
-rw-rw-r-- 1 rou rou  64M Jun  8 10:52 executors_removed_eventlog
-rw-rw-r-- 1 rou rou 1.7M Jun  8 10:52 malformed_json_eventlog
-rw-rw-r-- 1 rou rou 2.2M Jun  8 10:52 rapids_join_eventlog
-rw-rw-r-- 1 rou rou 2.2M Jun  8 10:52 rapids_join_eventlog2
-rw-rw-r-- 1 rou rou 110K Jun  8 10:52 rp_nosql_eventlog
-rw-rw-r-- 1 rou rou 1.7M Jun  8 10:52 rp_sql_eventlog
-rw-rw-r-- 1 rou rou  32M Jun  8 10:52 task_job_failure_eventlog

This causes git to complain with a warning:

remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: warning: See http://git.io/iEPt8g for more information.
remote: warning: File tools/src/test/resources/spark-events-profiling/executors_removed_eventlog is 63.56 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB

Steps/Code to reproduce bug
The files are already merged.

Expected behavior
We should compress these log files and change the tests to decompress them on the fly.

Additional context
We may want to change the git history so these large files don't show up in every git clone.

@rongou rongou added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jun 8, 2021
@nartal1 nartal1 self-assigned this Jun 9, 2021
@nartal1 nartal1 removed the ? - Needs Triage Need team to review and classify label Jun 9, 2021
@tgravescs
Copy link
Collaborator

we removed them, this required force push. @nartal1 please make a pr to add those tests back in as I removed the files and commented out the tests.

gerashegalov added a commit that referenced this issue Jun 11, 2021
Prevents large files as in #2644. 

Contributes to #2420, #2674
    
Signed-off-by: Gera Shegalov <gera@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants