Request: project information needed for ingress-gce jobs when migrating to k8s-infra-prow-build cluster

[sawsa307]

Currently, in our e2e job we need a GCP project for two purposes:

1. We build an image based on the PR that triggers it, and push and store it temporary in the project https://github.com/kubernetes/ingress-gce/blob/master/hack/run-e2e-gce.sh#L58.
2. We need project information from /etc/gce.conf to start our controller, but it is unclear that if it still exists after migrating to use k8s-infra-prow-build cluster.

[sawsa307]

cc @swetharepakula

[sawsa307]

/sig k8s-infra
/sig testing

[sawsa307]

@rjsadow @ameukam

[ameukam]

We build an image based on the PR that triggers it, and push and store it temporary in the project https://github.com/kubernetes/ingress-gce/blob/master/hack/run-e2e-gce.sh#L58.

The service account used in the community infra should be able to build and push after the migration. what's the existing configuration for the GCR service ?

We need project information from /etc/gce.conf to start our controller, but it is unclear that if it still exists after migrating to use k8s-infra-prow-build cluster.

I think this is created during the job execution unless the clusters used for the tests are not cleaned up when the tests are done.

[ameukam]

We also need to drop the project type: https://github.com/kubernetes/ingress-gce/blob/5a636516043dd4fbdb56e905d4814771e2d6d44e/hack/boskos.sh#L25

Boskos will provide a gce project for the tests.

[sawsa307]

We also need to drop the project type: https://github.com/kubernetes/ingress-gce/blob/5a636516043dd4fbdb56e905d4814771e2d6d44e/hack/boskos.sh#L25

Boskos will provide a gce project for the tests.

Yes I'll include this as part of the clean up.

[sawsa307]

The service account used in the community infra should be able to build and push after the migration. what's the existing configuration for the GCR service ?

The permission should not be a problem. My concern is more about which path we should push to? Is it okay we use gcloud config get project to get project and use `gcr.io/$GCLOUD_PROJECT" as image path here?

I think this is created during the job execution unless the clusters used for the tests are not cleaned up when the tests are done.

Got it, so it should not be a problem. Thanks!

[ameukam]

The permission should not be a problem. My concern is more about which path we should push to? Is it okay we use gcloud config get project to get project and use `gcr.io/$GCLOUD_PROJECT" as image path here?

GCR service is enabled for all the projects in the boskos. For a community-owned boskos project, some images are already pushed:

gcloud container images list --project k8s-infra-e2e-boskos-002
NAME
gcr.io/k8s-infra-e2e-boskos-002/cloud-controller-manager
gcr.io/k8s-infra-e2e-boskos-002/gcp-filestore-csi-driver
gcr.io/k8s-infra-e2e-boskos-002/gcp-persistent-disk-csi-driver
gcr.io/k8s-infra-e2e-boskos-002/local-volume-provisioner
gcr.io/k8s-infra-e2e-boskos-002/vpa-admission-controller
gcr.io/k8s-infra-e2e-boskos-002/vpa-admission-controller-amd64
gcr.io/k8s-infra-e2e-boskos-002/vpa-recommender
gcr.io/k8s-infra-e2e-boskos-002/vpa-recommender-amd64
gcr.io/k8s-infra-e2e-boskos-002/vpa-updater
gcr.io/k8s-infra-e2e-boskos-002/vpa-updater-amd64

[sawsa307]

The permission should not be a problem. My concern is more about which path we should push to? Is it okay we use gcloud config get project to get project and use `gcr.io/$GCLOUD_PROJECT" as image path here?

GCR service is enabled for all the projects in the boskos. For a community-owned boskos project, some images are already pushed:

gcloud container images list --project k8s-infra-e2e-boskos-002
NAME
gcr.io/k8s-infra-e2e-boskos-002/cloud-controller-manager
gcr.io/k8s-infra-e2e-boskos-002/gcp-filestore-csi-driver
gcr.io/k8s-infra-e2e-boskos-002/gcp-persistent-disk-csi-driver
gcr.io/k8s-infra-e2e-boskos-002/local-volume-provisioner
gcr.io/k8s-infra-e2e-boskos-002/vpa-admission-controller
gcr.io/k8s-infra-e2e-boskos-002/vpa-admission-controller-amd64
gcr.io/k8s-infra-e2e-boskos-002/vpa-recommender
gcr.io/k8s-infra-e2e-boskos-002/vpa-recommender-amd64
gcr.io/k8s-infra-e2e-boskos-002/vpa-updater
gcr.io/k8s-infra-e2e-boskos-002/vpa-updater-amd64

Oh thats great! I think this matches our expectation.
Currently, we rent a boskos project of type ingress-project using POST API with Curl https://github.com/kubernetes/ingress-gce/blob/master/hack/boskos.sh#L25-L27. Do you know the proper project type if we want to use k8s-infra-e2e-boskos-002 as boskos project or any guide on the process? Thank you!

[ameukam]

@sawsa307 Already mentioned you can drop the project type. Boskos will provide a project.

[sawsa307]

https://github.com/kubernetes/ingress-gce/blob/5a636516043dd4fbdb56e905d4814771e2d6d44e/hack/boskos.sh#L25

Oh sorry I misread, I thought you were talking about getting rid of project type here.

You can drop the project type. Boskos will provide a project.

Currently we are doing boskos project rental ourselves, and then do kubernetes_e2e.py to run our e2e tests.
The project type is a required part of the project rental URL. When you say "drop" the project type, do you mean we do gce-project as the project type?

[ameukam]

Currently we are doing boskos project rental ourselves, and then do kubernetes_e2e.py to run our e2e tests.
The project type is a required part of the project rental URL. When you say "drop" the project type, do you mean we do gce-project as the project type?

Yes. The type gce-project exists also in the community infrastructure.

[ameukam]

@sawsa307 Any update about this ?

[sawsa307]

We don’t see any errors in our presubmit after migration so we should be good! Closing this issue, thanks for the help!

[sawsa307]

I also added a note under #31789 to mark this as done.