Build for GPU on CircleCI

[luhenry]

This change adds just enough support to build Velox with GPU support on
CircleCI. That will help make sure the build with VELOX_ENABLE_GPU
doesn't regress, even if it doesn't test just yet.

Co-authored-by: Sergei Lewis slewis@rivosinc.com

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	ef7cd95
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/65d8b167efad3f0008006b7c

[luhenry]

It depends on #8828 for support of CUDA on the CircleCI Docker image

[luhenry]

After trying to use the GPU executors, I'm coming to the realization that they don't seem compatible with running the steps in Docker. @assignUser @kgpai I'm not sure what to do next? I think we still want to go ahead and add the building on a Linux/Docker executor for now, and we can add a downstream test job that uses the gpu executor.

[kgpai]

After trying to use the GPU executors, I'm coming to the realization that they don't seem compatible with running the steps in Docker

Sorry can you add more detail - what is the problem you are seeing ?

[kgpai]

Hi @luhenry I think this is what should be done:

1. We have 4-core-ubuntu-gpu-t4 as GPU supported runners in GHA
2. These are in GHA so changes to Add CUDA to the CircleCI docker image #8828 cant be used (as thats specific to CCI)
3. You will need to make changes to scripts/centos-8... or scripts/ubuntu dockerfiles
4. Setup a new GHA job that uses these images and the new runner above along with changes in this PR.

[assignUser]

@kgpai @luhenry We are a bit blocked with the GHA migration due to limitations of actions/cache when used to cache a build cache like ccache but I have a fix almost ready.

I haven't used the official cuda capable runners yet but with our self-hosted ones it's important to get the match between host and docker cuda versions correct, so either we run this without docker or use a cuda base image. I'll checkout the gpu runners and see what works.

Do we want to run this on every PR? Or would a scheduled nightly job be enough to spot regressions (+ maybe on PRs with direct changes to the experimental features?)

[luhenry]

Sorry can you add more detail - what is the problem you are seeing ?
It is a limitation of the CircleCI GPU runners where you need to specify a machine: making it impossible to specify a docker:.

On using GHA to build and run with GPU, when are you expecting to have all of this ready to go? At the moment, with #8828, we would be able today to build the CUDA code on CircleCI.

I'm happy to do the work to add a specific GHA workflow/job to build and test on GHA. But if that's not going to be merged in the coming weeks/months because it's blocked on something else, I would much rather get what's currently available merged (even if it's only building, given it doesn't even builds today) and improve things later on (even if it means the work I'm doing today goes to trash and I need to do more later).

Do we want to run this on every PR? Or would a scheduled nightly job be enough to spot regressions (+ maybe on PRs with direct changes to the experimental features?)

I think that the building part should be run on every PR as it's only building and there shouldn't be any flakiness/slowness to it. For the testing, I think it entirely depends on the capacity and availability of the runners with GPUs. If there are enough, the tests shouldn't take much longer than a non-GPU test run.

@pedroerp what do you think?

[kgpai]

On using GHA to build and run with GPU, when are you expecting to have all of this ready to go? At the moment, with #8828, we would be able today to build the CUDA code on CircleCI.

We expect this to happen in a few weeks at most. While we can have this working in CCI we will still need to do get this working in GHA as we are migrating in the very short term to GHA from CCI - thus changes to CCI are additional and wasted effort.

I think that the building part should be run on every PR as it's only building and there shouldn't be any flakiness/slowness to it. For the testing, I think it entirely depends on the capacity and availability of the runners with GPUs. If there are enough, the tests shouldn't take much longer than a non-GPU test run.

This makes sense, we can have the gpu part build / PR but only exercise the code , say nightly on a gpu.

[luhenry]

Closed in favor of #9335