[discussion] Increase the availability of CI in katib

[gaocegege]

/kind discussion

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Now CI hurts the development of katib, we need to find a way to solve the problem.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/cc @richardsliu @johnugeorge @hougangliu

[gaocegege]

Some suggestion:

• btwn , do we need to build images in parallel ?. i don't think, it will make a huge difference in build time. we can fix this later. From @johnugeorge
• Should we build the code instead of building the image in the CI? From @gaocegege

[richardsliu]

List of issues we have seen in the past couple of days:

• RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'cloudbuild.googleapis.com/get_requests' and limit 'GetRequestsPerMinutePerUser' of service '
  cloudbuild.googleapis.com' for consumer 'project_number:593963025935'.

This one seems to be resolved for the moment after requesting quota increase.

• Exception occurred: HTTPSConnectionPool(host='35.196.213.148', port=443): Max retries exceeded with url: /apis/argoproj.io/v1alpha1/namespaces/kubeflow-test-infra/workflows/kubeflow-katib-presubmit-e2e-v1alpha1-519-db486d7-0611-b811 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff439d6f950>: Failed to establish a new connection: [Errno 111] Connection refused',))

Transient error?

• level=error msg="handle object: merging object with existing state: Get https://35.196.213.148/apis/argoproj.io/v1alpha1/namespaces/kubeflow-test-infra/workflows/kubeflow-katib-presubmit-e2e-v1alpha2-512-4c2ae63-9536-0842: net/http: TLS handshake timeout"

Transient error?

[hougangliu]

maybe we can make katib build-.sh wiser, only build corresponding new images when related files changed. saying if only pkg/db/ changed, only katib-manager image should be built

[gaocegege]

We do not need GPU cluster, thus we can release the constraints.

katib/test/scripts/v1alpha2/create-cluster.sh

Line 35 in 78a4563

--accelerator type=nvidia-tesla-k80,count=1 \

[gaocegege]

After #525 , The building time is reduced to 10-20mins.

[andreyvelich]

What if we move to go modules, in that case we don't need vendor folder and dep ensure?

[gaocegege]

Go module also needs to download packages from the remote. I prefer storing vendor whether we use dep or not.

[gaocegege]

There is one thing we can do: Jeremy suggested batching the request to gcloud container builds.

Now we send single-image building requests one by one. Jeremy said we can do it in one request.

[gaocegege]

Find a note in https://cloud.google.com/sdk/gcloud/reference/builds/submit:

You can also run a build locally using the separate component: gcloud components install cloud-build-local.

These variants are also available:

$ gcloud alpha builds submit
$ gcloud beta builds submit

[gaocegege]

/close