Significantly less available disk space in macos-14-arm64 #10511
Comments
|
Hi @ehuss We are looking into it. Will keep you posted. |
|
Hey @ehuss! Please, see the description of GitHub hosted runners bellow: Thank you for bringing this fact to our attention. Most likely, it is related to the fact that a new Xcode with all SDKs and simulators was added to the assembly, but this still fits into the stated service agreement and is not a regression. Please, see the description of GitHub hosted runners bellow: Just in case, we will of course check what exactly led to such a sharp decrease in space and will try to do something about it, but unfortunately we cannot make any promises. MacOS images are the most voluminous and it is quite difficult to remove something from there without negatively affecting the experience of other users. If your builds require additional space, CPU or RAM, then I recommend paying attention to Larger Runners: |
|
Is there any equivalent cleanup command as per #10386 but for macOS runners instead? |
|
I appreciate you looking into this, but regarding the recommendation to use larger runners for macOS, I am curious: which one is the one with more than 14GiB storage (as promised by service contract), exactly?
|
|
Hey @nth10sd! We'll definitely propose something like that to you later. It's a good temporary solution I think. |
|
Hey @workingjubilee! Nice input. I checked up with the corresponding team and it seams that you are right - there is no option to extend disk space for macOS as it might be done for the Ubuntu/Windows images. Given the above and all the information in general, the best solution would be to try to increase the free space on our side - we will try to work on this. As an interim solution, we will develop a couple of small scripts that can be implemented in your CI in order to free up space on the runner by removing components that are not needed at the moment. cc: @ehuss |
|
APFS seems to support reflinks/file clones. Has deduplication been applied to the runner images? If not then that might save some space. |
|
We are suffering from the same issue too |
…orkingjubilee Try to reduce space usage in dist CI We have had recurrent CI problems as a result of GitHub adding a new version of Xcode to its runners[^0], which has consumed ~40GB of space which served as padding. Try to reduce the number of Xcodes on our systems, because we only use Xcode 14 in actual practice. Also, try to move files instead of pointlessly copy them when we're at the end of the job. I could not resist addressing a few shellcheck lints while I was at it. [^0]: actions/runner-images#10511
…orkingjubilee Try to reduce space usage in dist CI We have had recurrent CI problems as a result of GitHub adding a new version of Xcode to its runners[^0], which has consumed ~40GB of space which served as padding. Try to reduce the number of Xcodes on our systems, because we only use Xcode 14 in actual practice. Also, try to move files instead of pointlessly copy them when we're at the end of the job. I could not resist addressing a few shellcheck lints while I was at it. [^0]: actions/runner-images#10511
…ilee Try to reduce space usage in dist CI We have had recurrent CI problems as a result of GitHub adding a new version of Xcode to its runners[^0], which has consumed ~40GB of space which served as padding. Try to reduce the number of Xcodes on our systems, because we only use Xcode 14 in actual practice. Also, try to move files instead of pointlessly copy them when we're at the end of the job. I could not resist addressing a few shellcheck lints while I was at it. [^0]: actions/runner-images#10511
|
+1 for tracking. Also running into this. |
|
Suffering of this as well +1 |
|
+1 also on our side |
|
You can 👍 the issue instead of adding new comments. |
|
👍 |
|
We encountered this as a sudden failure of Homebrew to install the Searching the issues in this repo didn't turn up anything -- hopefully, mentioning Homebrew in this comment will help someone searching in the future. I was eventually able to discover where the Homebrew installer logs are recorded and saw messages that the package needed 8.89 GB but only 7.99 GB was available. The install works on macos-12 and macos-13 runners, which still have enough space. But this alerted to me how much space the |
|
I ran into this issue on our macos runner. After doing a bit of sleuthing with We do use Xcode as part of our build, but we only need one of those versions of Xcode (not all of them). So the solution in our case was to delete a bunch of the old ones: # We only use Xcode 15.4
- name: Remove unused applications
run: |
df -hI /dev/disk3s1s1
sudo rm -rf /Applications/Xcode_14.3.1.app
sudo rm -rf /Applications/Xcode_15.0.1.app
sudo rm -rf /Applications/Xcode_15.1.app
sudo rm -rf /Applications/Xcode_15.2.app
sudo rm -rf /Applications/Xcode_15.3.app
df -hI /dev/disk3s1s1... which frees up about 30GB of space. |
|
Here is a step that will delete all beta versions of Xcode unless you explicitly selected one: |
|
In addition to the above suggestions to delete unused versions of Xcode, we were able to add a significant amount of space by deleting unused simulators and clearing the simulator cache: |
Summary: This diff does two things: 1. Brings back D9464, which was previously removed in D11202 2. Implements the changes suggested in [this GitHub comment](actions/runner-images#10511 (comment)) to delete iOS simulators and associated caches Test Plan: I landed the changes to `android_ci.yml` directly on master and confirmed that the GitHub Actions run succeeded: https://github.com/CommE2E/comm/actions/runs/10722767078/job/29734470192 Reviewers: varun, will Subscribers:
Summary: This diff does two things: 1. Brings back D9464, which was previously removed in D11202 2. Implements the changes suggested in [this GitHub comment](actions/runner-images#10511 (comment)) to delete iOS simulators and associated caches Test Plan: I landed the changes to `android_ci.yml` directly on master and confirmed that the GitHub Actions run succeeded: https://github.com/CommE2E/comm/actions/runs/10722767078/job/29734470192 Reviewers: varun, will Reviewed By: varun Subscribers: tomek Differential Revision: https://phab.comm.dev/D13247
|
I have tried below script on my unit test job, and after cleaning up the space and when I rerun the job, all Xcodes version are still there. |
This should mitigate the effects of a GitHub regression that reduced space on these runners: actions/runner-images#10511 After this, we're mostly fine, but builds that happen to compile enough of the codebase can bump past it. Hopefully with this PR we'll have plenty of space.
|
For other folks finding this... Definitely bit our project too, and seems like a pretty sharp regression. We found removing Xcode versions was very slow though and for limited space. The tip to remove the iOS simulators (and maybe only add the one you need) look much more promising. Happens that we're just testing on macOS, not iOS, so even better for us. In case folks want to see a full PR that adds this: |
This should mitigate the effects of a GitHub regression that reduced space on these runners: actions/runner-images#10511 After this, we're mostly fine, but builds that happen to compile enough of the codebase can bump past it. With this PR we have over 50 GiB of space which is more than we need. See a test run here: https://github.com/carbon-language/carbon-lang/actions/runs/10780743574
|
Also ran into this issue. My actions are constantly failing unless I do some serious runner cleanup beforehand. Upon investigation, it seems like every Xcode folder in Running: Outputs: Is this expected? Why suddenly all versions have a sibling |
|
Hey @rodperottoni! It's just a symlink. It takes no space. |
|
|
|
I can confirm, maybe we could avoid to free up space by removing xcode in our Action? Initial space mac os 14 
After clean up 
|
At your discretion. I think you can disable it for now to save execution time if the specified space is enough for you. I will notify you and other participants separately about all further changes as they happen. |
|
👋 For those tracking these changes or encountering issues for the first time, here’s some context and clarification. As you may know, we recently faced significant challenges where many users began running out of disk space during tests and builds on Here’s how we’ve approached resolving the issue so far:
However, we’ve discovered that some projects rely on multiple major versions of Current Strategy 📏Our updated approach balances reducing installed
Recommendations 🦮
|
Description
The 20240827.4 release of the macos-14-arm64 image seems to have significantly less available disk space than the previous release (20240818.4). I wanted to check if this was intended or expected. This is unfortunately causing our builds to fail.
20240827.4 initial space:
compared to 20240818.4 initial space:
Platforms affected
Runner images affected
Image version and build link
20240827.4
https://github.com/rust-lang-ci/rust/actions/runs/10612800738/job/29415273318
Is it regression?
Yes, 20240818.4 works. https://github.com/rust-lang-ci/rust/actions/runs/10622377375/job/29446525166
Expected behavior
Would like to have more than 18GiB free when starting.
Actual behavior
Only 18GiB free when starting.
Repro steps
The text was updated successfully, but these errors were encountered: