'No space left on device' for some alpine-linux and x86-64_linux static docker nodes

[smlambert]

test-docker-alpine317-x64-2 (which has a label hw.dockerhost.dockerhost-equinix-ubuntu2204-x64-1) and test-docker-ubuntu2204-x64-2 (which has a label hw.dockerhost.dockerhost-equinix-ubuntu2204-x64-1) have many test jobs that are failing with 'No space left on device'

To make it easy for the infrastructure team to repeat and diagnose, please
answer the following questions:

• test suite/name (e.g, BUILD_LIST, TARGET, CUSTOM_TARGET)?
• a link into recent Test_ job on https://ci.adoptium.net which showed the failure
  • https://ci.adoptium.net/job/Test_openjdk17_hs_sanity.functional_x86-64_linux/263/
  • https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_x86-64_alpine-linux_testList_2/61/
  • https://ci.adoptium.net/job/Test_openjdk20_hs_sanity.system_x86-64_linux/83/
• Hyperlink to re-run in Grinder:
• Is there an existing issue elsewhere covering this?
• Which machine(s) does it work on?
• Which machine(s) does it fail on? test-docker-alpine317-x64-2 and test-docker-ubuntu2204-x64-2

[sxa]

I've issued docker system prune on this host as an interim measure but I will note that the nagios alerts in #infrastructure-bot don't seem to have been firing for the last week.

Both x86 dockerhosts seem to be in the same state so I'll prune the other one too.

[sxa]

On the second one is was the docker build cache that was chewing just under 700Gb:

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          70        9         37.11GB   37.11GB (100%)
Containers      9         9         42.87GB   0B (0%)
Local Volumes   0         0         0B        0B
Build Cache     6278      0         695.4GB   695.4GB

so I've just issued docker builder prune on their to reclaim the space.

[sxa]

infrastructure bot has been fixed by @steelhead31 so is now producing alerts again. Closing