Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dynamic class file deduplication across shims in dist jar build #3545

Merged
merged 7 commits into from
Sep 20, 2021
Merged

Dynamic class file deduplication across shims in dist jar build #3545

merged 7 commits into from
Sep 20, 2021

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented Sep 19, 2021 •
edited
Loading

  • uses a binary diff to collect a list of class files identical for all shims in the current build (reduces the jar from 115M to 83M)
  • fixes duplicate storage of the shared resources from unshimmed-base, down to 76M

Numbers are for the biggest jar profile snapshotsWithDatabricks

Signed-off-by: Gera Shegalov gera@apache.org

@gerashegalov
wip
215d6cb
@gerashegalov
Add a profile-oblivious script dedupping classes
0b4da6d 
- uses a binary diff to collect a list of class files identicall for all
  shims in the current build

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this pull request Sep 19, 2021
Closed
21 tasks
@gerashegalov
Exclude unshimmed-base from subdirs
c236121 
Signed-off-by: Gera Shegalov <gera@apache.org>
@sameerz sameerz added this to the Sep 13 - Sep 24 milestone Sep 19, 2021
@gerashegalov gerashegalov marked this pull request as ready for review September 20, 2021 07:27
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov
revert doc
a34925e 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
Copy link
Collaborator Author

build

jlowe
jlowe reviewed Sep 20, 2021
View reviewed changes
build/buildall Show resolved Hide resolved
dist/scripts/binary-dedupe.sh Outdated Show resolved Hide resolved
revans2
revans2 reviewed Sep 20, 2021
View reviewed changes
Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a small nit.

dist/scripts/binary-dedupe.sh Outdated Show resolved Hide resolved
@gerashegalov
Reviews:
05f2a22 
- added brief -q output to suppress content diff
- added a unique label to protect from a match in file name
- added a regex
- class files for different classes can't be identical becuase class files
  store class names

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov gerashegalov merged commit 19b090b into NVIDIA:branch-21.10 Sep 20, 2021
@gerashegalov gerashegalov deleted the binaryDiffDedupe branch September 20, 2021 19:58
@tgravescs tgravescs mentioned this pull request Sep 21, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

@jlowe jlowe jlowe approved these changes

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@tgravescs tgravescs Awaiting requested review from tgravescs tgravescs is a code owner

Assignees

@gerashegalov gerashegalov

Labels
Spark 3.2+
Projects
None yet
Milestone
Sep 13 - Sep 24
Development

Successfully merging this pull request may close these issues.
4 participants
@gerashegalov @jlowe @revans2 @sameerz