GPU runners are stalled

[jhamman]

All active PRs are currently failing to get GPU CI runners:

GPU Test V3 / py=3.11, np=2.0, deps=minimal (pull_request) Queued — Waiting to run this check...

The actions are all reporting something like this:

Requested labels: gpu-runner
Job defined at: zarr-developers/zarr-python/.github/workflows/gpu_test.yml@refs/heads/v3
Waiting for a runner to pick up this job...
Job is waiting for a runner from 'gpu-runner' to come online.

I've been merging without this action running for PRs that are obviously unrelated.

[jhamman]

With @rabernat's help, I've determined this is not related to the switch to main. I have filed a support ticket with GitHub about this.

[jhamman]

GitHub (GitHub Support)Oct 15, 2024, 2:19 PM UTC

Hello,

Thank you for reaching out to GitHub Support and bringing this issue to our attention.

We are currently experiencing an incident affecting our GPU runners.

After thorough investigation, we have identified the root cause to be an issue with the NVIDIA GPU-Optimized VMI image provided by NVIDIA, which we utilize for our GPU Runners.

Unfortunately, the latest image available on the Azure Marketplace appears to be faulty, and previous versions are currently unavailable.

This issue has also been publicly reported on NVIDIA's page.

Please rest assured that our Engineering team is actively working on resolving this issue in collaboration with NVIDIA.

We understand the importance of this service to your work and are committed to restoring full functionality as soon as possible.

At this time, we do not have a timeline when this will be resolved, but we will notify you immediately once it is resolved.

Thank you for your patience and understanding.

Best regards,

David
GitHub Support

[jhamman]

Seems to be fixed. Will reopen if this happens again.