Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-e2e-gce-network-proxy-grpc

[spiffxp]

What should be cleaned up or changed:

This is part of #18550

To properly monitor the outcome of this, you should be a member of k8s-infra-prow-viewers@kubernetes.io. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

Migrate pull-kubernetes-e2e-gce-network-proxy-grpc to k8s-infra-prow-build by adding a cluster: k8s-infra-prow-build field to the job:

• Make sure any release-branch variants are also updated
• Reference this issue, but do not "fix" it; instead leave this open to monitor the behavior of the job before/after
• Here's an example PR that did this for pull-kubernetes-integration: Migrate pull-kubernetes-integration to k8s-infra-prow-build #18814

NOTE: migrating this job is not as straightforward as some of the other #18550 issues, because:

• it doesn't run for every single PR
• so in terms of appropriate amount of PR traffic, either manually trigger this job, or look for open PRs that have triggered this job (e.g. using the job-history link below), and explicitly /test <this-job-name> on them

Once the PR has merged, note the date/time it merged. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=pull-kubernetes-e2e-gce-network-proxy-grpc
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-gce-network-proxy-grpc&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-e2e-gce-network-proxy-grpc
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-network-proxy-grpc
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for pull-kubernetes-e2e-gce-network-proxy-grpc for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for pull-kubernetes-e2e-gce-network-proxy-grpc for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
• https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1
  • are there still plenty of free projects in the "GCE project (k8s-infra) usage status" graph?

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra
/sig testing
/area jobs
/help

[bartsmykla]

/assign
/remove-help

[bartsmykla]

@spiffxp I opened the PR to test if migration succeeded and the job actually failed with the error:

W0820 08:13:15.836] ERROR: (gcloud.compute.project-info.describe) Could not fetch resource:
W0820 08:13:15.836]  - Required 'compute.projects.get' permission for 'projects/k8s-network-proxy-e2e' 

[spiffxp]

So it turns out I missed this bit in the job definition: - --gcp-project=k8s-network-proxy-e2e

You should delete that and see if the job still functions (this is equivalent to setting --gcp-project-type=gce-project)

If it fails, then we'll need to create a custom project type and understand what's unique about k8s-network-proxy-e2e. I would reach out to whomever added/authored the job to start.

[spiffxp]

I've opened #19079 to try out the fix I suggested in #18853 (comment)

[spiffxp]

Per kubernetes/kubernetes#94127 (comment) I think that worked

[RobertKielty]

Hi @bartsmykla, have you had a chance to submit the changes?
cc @spiffxp

[spiffxp]

This job was made non-blocking via #19033 I would like to see it restored to blocking if we're confident it's been fixed

[spiffxp]

I'm keeping an eye on https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=94127 and think we should trigger a few more runs

[spiffxp]

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-network-proxy-grpc

It appears to be intermittently timing out

If I look back to before we started messing with this job (e.g. https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-network-proxy-grpc?buildId=1299167300326264832), I see it also intermittently times out

Since it's pre-existing behavior, I'm willing to call this good enough.

[bartsmykla]

I agree and think we could call this done

[spiffxp]

/close

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.