-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wg-k8s-infra: canary prowjobs for sig-scalability #22430
wg-k8s-infra: canary prowjobs for sig-scalability #22430
Conversation
Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ameukam, dims The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/assign @BenTheElder @spiffxp |
xref: kubernetes/k8s.io#1469. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/hold
If you expect these to fail and just want to see how, then remove the hold. But I'm not ok with the 5k node jobs as-is.
I would much rather see provisioning of a special 5k node project for the 5k node jobs. Like modify infra/gcp/prow/ensure-e2e-projects.sh
to add a k8s-infra-e2e-scale-5k-project
to E2E_MANUAL_PROJECTS
and then pin the jobs to that.
- --gcp-master-image=gci | ||
- --gcp-node-image=gci | ||
- --gcp-node-size=e2-small | ||
- --gcp-nodes=5000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need a quota request for this. I'd rather avoid giving all scalability projects this kind of quota, we should pin to a specific project for this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@spiffxp I'll make the quota requests in k8s-infra-e2e-scale-5k-project
. See kubernetes/k8s.io#2225
- --env=KUBE_DNS_MEMORY_LIMIT=300Mi | ||
- --extract=ci/latest-fast | ||
- --extract-ci-bucket=k8s-release-dev | ||
- --gcp-nodes=5000 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same. This should use the same project as correctness. What's the node type used here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Took a lot at a successful job of ci-kubernetes-e2e-gce-scale-performance
: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1401585162642264064/build-log.txt.
The node type appears to be e2-standard-32.
- --extract=ci/latest | ||
- --gcp-node-image=gci | ||
- --gcp-node-size=e2-standard-8 | ||
- --gcp-nodes=84 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honestly not sure if this will need quota increase or not
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to increase the quota for this job. the smallest quota in us-east1 is 1250 (for all the projects with type scalability-project
)
There are various GCP quotas apart from CPU one that need to be lifted to run 5k node jobs (to be copied from legacy project). Are you tracking this anywhere? |
@jkaniuk For the moment nothing is tracked. Do you have an existing document with the list of quotas that need to be raised ? I can create a new document if not. |
Please do, I do not believe we have anything yet. |
Add e2e projects for sig-scalability prowjobs that need 5K nodes. Initial suggestion : kubernetes/test-infra#22430 (review) Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
Add e2e projects for sig-scalability prowjobs that need 5K nodes. Initial suggestion : kubernetes/test-infra#22430 (review) Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
@jkaniuk I created https://docs.google.com/spreadsheets/d/1v9ynsUx3pcMJKVaHViuVqxVtRP1OmGIadvw2uro7D7Y/edit#gid=0. Feel free to adjust at your own convenience. |
Add e2e project for sig-scalability prowjobs that need 5K nodes. Initial suggestion : kubernetes/test-infra#22430 (review) Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
Add e2e project for sig-scalability prowjobs that need 5K nodes. Initial suggestion : kubernetes/test-infra#22430 (review) Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
Ref: kubernetes/k8s.io#1469 Add canaries running on `k8s-infra-prow-build` for some sig-scalability periodics prowjobs. Switch from `--gcp-project=foo` to `--gcp-project=k8s-infra-e2e-scale-5k-project` Signed-off-by: Arnaud Meukam <ameukam@gmail.com>
398e9b8
to
9ee0994
Compare
/lgtm please feel free to remove hold if you are ready. |
/hold cancel |
@ameukam: Updated the
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Add canary jobs running on
k8s-infra-prow-build
for some sig-scalability periodics prowjobs.Switch from
--gcp-project=foo
to--gcp-project=k8s-infra-e2e-scale-5k-project