Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Add fine-grained service rate limiters #2201

Merged
merged 1 commit into from
Jan 15, 2021
Merged

✨ Add fine-grained service rate limiters #2201

merged 1 commit into from
Jan 15, 2021

Conversation

sedefsavas
Copy link
Contributor

What this PR does / why we need it:
Rebase of #2033

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 13, 2021
@k8s-ci-robot k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jan 13, 2021
@sedefsavas sedefsavas mentioned this pull request Jan 13, 2021
Closed
@sedefsavas sedefsavas force-pushed the rebase-ratelimiter branch 4 times, most recently from b24926e to de03cdc Compare January 14, 2021 17:38
@sedefsavas sedefsavas changed the title ✨ WIP: Rebase ratelimiter ✨ Rebase ratelimiter Jan 14, 2021
@sedefsavas
Copy link
Contributor Author

Did not added ratelimiter to MachinePool, it needs further investigation on how to incorporate there best.

/hold
Running e2e test.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 14, 2021
@randomvariable
Copy link
Member

Did not added ratelimiter to MachinePool, it needs further investigation on how to incorporate there best.

Can you create a follow up issue if there isn't one?

@randomvariable
Copy link
Member

can you squash these commits?

@randomvariable randomvariable changed the title ✨ Rebase ratelimiter ✨ Add fine-grained service rate limiters Jan 14, 2021
Adds a AWS request throttler based upon per-operation type
d3db52a 
request buckets that can be reset when AWS returns that it's
throttling us.

Based loosely on https://github.com/aws/aws-app-mesh-controller-for-k8s/tree/master/pkg/aws/throttle
but much simplified.

Uses a forked version of Golang's rate limiter to implement the bucket
reset.

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html

Signed-off-by: Naadir Jeewa <jeewan@vmware.com>

Minor fixes
@sedefsavas
Copy link
Contributor Author

Scale tests that are on this PR (#2034) are not passing due to docker rate limits:

Failed to pull image "calico/cni:v3.13.2": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.13.2": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:bbf7e3ac3f80d0a356a6c27b095bd313d1106f8ed84f85850816ed79295843c1: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

But this something we need to worry in that PR.

@sedefsavas
Copy link
Contributor Author

I will create the rebased test PR once this is merged.

This was referenced Jan 14, 2021
Closed
Closed
@sedefsavas
Copy link
Contributor Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 14, 2021
@sedefsavas
Copy link
Contributor Author

/test pull-cluster-api-provider-aws-e2e

@randomvariable
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 15, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: randomvariable

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 15, 2021
@randomvariable
Copy link
Member

On the scale test issue, we should probably bump Calico to the latest version. Going to assume that they've maybe got a new public registry that doesn't get hit by the limits?

@k8s-ci-robot k8s-ci-robot merged commit 7470b49 into kubernetes-sigs:master Jan 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michaelbeaumont michaelbeaumont Awaiting requested review from michaelbeaumont

@vincepri vincepri Awaiting requested review from vincepri

Assignees

@randomvariable randomvariable

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sedefsavas @randomvariable @k8s-ci-robot