Google cloud CI with GPUs in docker #2152

ssheorey · 2020-08-08T01:40:11Z

Entire CI run takes ~25-30 minutes and runs build configurations in parallel.

Tasks:

Build docker image on GitHub, push to google container registry, create VM, run CI in docker container. Delete VM and docker image.
If resources are available, can handle up to 13 simultaneous commit CI runs (one per GCE US zone with Tesla T4 GPUs). For simultaneous commits, will run into quota limit in one zone and switch to next zone.
Update CI TF version from 2.0 -> 2.3
Test and enable Pytorch + cuDNN: Need cuDNN detection or auto-install.
Add build config with Ubuntu 20.04: Linker error with Filament.

This change is

Each command tested to work. Need to fix authentication and test with GH actions on github.

This reverts commit e480144.

Start from base Ubuntu 18.04 image Multiple commits can be processed at the same time. Tries 5 times to create VM if not successful.

gcloud CLI reboot disables future ssh

Moved all workflow logic to shell script gce-ubuntu-docker-run.sh) CI_CONFIG_ID used instead of build matrix to limit CI to a few build combinations TODO: change cuda to 10.1 cuDNN 7 add cudnn to docker image

Always run VM and container image delete steps.

This will hit the quota limit (each zone quota is 4GPUs

update-docs · 2020-08-08T01:40:14Z

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

yxlao

Reviewed 4 of 7 files at r1, 2 of 2 files at r2.
Reviewable status: 6 of 7 files reviewed, 14 unresolved discussions (waiting on @benjaminum, @germanros1987, @ssheorey, and @yxlao)

.github/workflows/gce-ubuntu-docker.yml, line 9 at r1 (raw file):

    branches:
      - master
      - ssheorey/gce-gpu-ci-docker

this could be removed eventually since the branch is in pull_request already

.github/workflows/gce-ubuntu-docker.yml, line 29 at r2 (raw file):

      # Setup gcloud CLI
      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master

give this step a name, so it looks nicer in web UI

.github/workflows/gce-ubuntu-docker.yml, line 31 at r2 (raw file):

      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
        with:
          version: '302.0.0'    # Fixed to avoid dependency on API changes

put this in global env so it is easier to change?

.github/workflows/gce-ubuntu-docker.yml, line 38 at r2 (raw file):

      # Build the Docker image
      - name: Build

name: Build docker image

.github/workflows/gce-ubuntu-docker.yml, line 43 at r2 (raw file):

      # Push the Docker image to Google Container Registry
      - name: Publish

name: Push docker image

.github/workflows/gce-ubuntu-docker.yml, line 46 at r2 (raw file):

        id: docker-push
        run: |
          ./util/docker/open3d-gpu/scripts/gce-ubuntu-docker-run.sh docker-push

if the docker image is built and consumed in local machine, do we need to push it to some repository?

.github/workflows/gce-ubuntu-docker.yml, line 92 at r2 (raw file):

  cleanup:
    name: Cleanup container images and delete VM

"Cleanup container images and delete VM" seems too long, can we use a shorter name for this? Also seems like other job names are pretty long to be displayed on the webpage, we can consider shorter names.

.github/workflows/gce-ubuntu-docker.yml, line 99 at r2 (raw file):

    steps:
      - name: GCloud setup
        uses: actions/checkout@v2       # Checkout code

do we need to checkout code again? also it looks like we are having nested folders
Open3D/Open3D/

.github/workflows/gce-ubuntu-readme.md, line 8 at r2 (raw file):

- Clones the repository and checks out the specific commit
- Installs dependencies (including NVIDIA drivers and CUDA)
- Builds the code and runs tests (`run-ci.sh`)

do we need to update workflow for docker?

util/docker/open3d-gpu/scripts/env-setup.sh, line 3 at r2 (raw file):

#!/usr/bin/env bash

# Run this script from the CMakeLists.txt (top-level) directory

is this needed?

util/docker/open3d-gpu/scripts/gce-ubuntu-docker-run.sh, line 7 at r2 (raw file):

# exit when any command fails
set -e

remove trailing space (probably enable this automatically in the editor)

util/docker/open3d-gpu/scripts/gce-ubuntu-docker-run.sh, line 102 at r2 (raw file):

        until (( TRIES>=5 )) || gcloud compute images create ${VM_IMAGE} --source-disk=$VM_IMAGE \
            --source-disk-zone=${GCE_INSTANCE_ZONE[$GCE_ZID]} \
            --family=ubuntu-os-docker-gpu-1804-lts \

shall we specify it in the # Container configuration section?

util/docker/open3d-gpu/scripts/gce-ubuntu-docker-run.sh, line 120 at r2 (raw file):

            --boot-disk-size=$GCE_BOOT_DISK_SIZE \
            --boot-disk-type=$GCE_BOOT_DISK_TYPE \
            --image-family=ubuntu-os-docker-gpu-1804-lts \

same, shall we specify it in the # Container configuration section?

util/docker/open3d-gpu/scripts/gce-ubuntu-docker-run.sh, line 134 at r2 (raw file):

            --env NPROC=$NPROC \
            --env SHARED=${SHARED[$CI_CONFIG_ID]} \
            --env BUILD_DEPENDENCY_FROM_SOURCE=${BUILD_DEPENDENCY_FROM_SOURCE[$CI_CONFIG_ID]} \

seems like this has been removed from master

+ fix for propogating GCE_ZID to next step

ssheorey

Reviewable status: 3 of 7 files reviewed, 14 unresolved discussions (waiting on @benjaminum, @germanros1987, and @yxlao)

.github/workflows/gce-ubuntu-docker.yml, line 9 at r1 (raw file):