Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce disk space usage #59

Open
rhaas80 opened this issue Apr 6, 2023 · 3 comments
Open

reduce disk space usage #59

rhaas80 opened this issue Apr 6, 2023 · 3 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@rhaas80
Copy link
Member

rhaas80 commented Apr 6, 2023

So the disk space warning is indeed back in: https://github.com/EinsteinToolkit/tests/actions/runs/4585746865 and that run actually died. One wonders which repository has the largest impact.

If gh-pages then there are in principle ways to reduce its on-disk size significantly though I more suspect the culprit is at least partially the compiled ET code.
If I do a:

git clone -b master https://github.com/einsteintoolkit/tests master
cd master
git submodule update --init --recursive --remote --jobs 4
cd ../
git clone -b gh-pages --depth 1 https://github.com/einsteintoolkit/tests gh-pages
git clone -b scripts --depth 1 https://github.com/einsteintoolkit/tests scripts

then the different checkouts are sized (du -hs *):

19G     gh-pages
2.1G    master
356K    scripts

So gh-pages is actually sizable and the majority is actually in the checked out files (and not the .git git objects directory which is only 486MB).

Since there are only a very few files we actually modify (other than add new files) we should be able to start with a "checkout" that has nothing actually checked out (using git clone's --no-checkout option) and operate on that one after manually checking out the couple of files we do modify (eg version.txt). This means one has to manually add the files that one wants to add or modify and we need to check that github's checkout action does not "helpfully" run a git commit -a at the end of the worflow (which it may well do) since that would record all the never checked out files as deleted files.

@rhaas80 rhaas80 added the bug Something isn't working label Apr 6, 2023
@rhaas80
Copy link
Member Author

rhaas80 commented Jun 13, 2023

This has become urgent now since it prevents tests from running:

image

failed due to out of disk space errors (the failures just before were actual test failures).

@rhaas80 rhaas80 added the help wanted Extra attention is needed label Jun 13, 2023
@rhaas80
Copy link
Member Author

rhaas80 commented Jun 13, 2023

As a stopgap measure I have removed some of the old versions from the checked out data:

git rm -r version_?? version_1?? version_2?? version_3??

in git hash b8f63046f5 "temoprarily remove some records files to free space" of tests which can be reverted if and when we have a proper fix (using --no-checkout alluded in the description).

einsteintoolkit-updater pushed a commit that referenced this issue Jan 26, 2024
Submodule repos/GRHayL da2e140..e69b470:
  > Scupp/newtests (#59)
@Hrishikesh-Kalyanaraman
Copy link
Collaborator

Hrishikesh-Kalyanaraman commented Aug 30, 2024

For some reason, I hadn't added the fetch-depth = 1 fix yet. I've added it in a pr. Will work on the --no-checkout fix now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants