Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS EC2 API Rate exceeded #364

Closed
tofanadrian3000 opened this issue May 2, 2023 · 5 comments
Closed

AWS EC2 API Rate exceeded #364

tofanadrian3000 opened this issue May 2, 2023 · 5 comments

Comments

@tofanadrian3000
Copy link

Issue Details

Describe the bug
Whenever the "Manage Jenkins" -> "Manage Nodes and Clouds" -> "Configure Clouds" page is being opened, the EC2 fleet plugin loads all the fleets that were added. As far as I can tell, this load procedure, involves loading all the AWS Auto Scaling Groups on one's AWS account using the EC2 API for all the fleets that were added ordered by the name of the fleet (not by the order that can be seen on the page itself).

If someone has 100 EC2 fleets, the plugin will make 100 API calls in order to retrieve the ASG lists (which doesn't change from one fleet to another). Therefore, at some point, the API's requests/second rate can be exceeded and the following error occurs:
WARNING com.amazon.jenkins.ec2fleet.EC2FleetCloud$DescriptorImpl doFillFleetItems
Cannot describe fleets in eu-central-1 or by endpoint
com.amazonaws.services.autoscaling.model.AmazonAutoScalingException: Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: f4886158-05d0-4bfb-80e2-db990dfb300b; Proxy: null)

Whenever this error occurs, the plugin is using the first (alphabetically sorted) ASG that it can find in a previous list for the "EC2 Fleet
" field. This fill happens only on the Jenkins page, but if someone is not careful about it and saves the changes, then multiple EC2 fleets can be mismatched (instead of using the right ASG, they will use the first ASG that was found by the plugin).

To Reproduce

  1. Add more ec2 fleets than the API requests/second limit, pointing to multiple ASGs.
  2. Open the "Manage Jenkins" -> "Manage Nodes and Clouds" -> "Configure Clouds" page
  3. Check the ASGs behind the latest (alphabetically sorted) EC2 fleets. They would probably point to the first (alphabetically sorted) ASG from the AWS account that is being used.

Environment Details

Plugin Version?
2.5.1

Jenkins Version?
2.319.2

Spot Fleet or ASG?
ASG

Label based fleet?
Yes - but I don't think it's relevant in this case

Linux or Windows?
Both

EC2Fleet Configuration as Code
N/A

Anything else unique about your setup?
Not really

@benipeled
Copy link

It's probably solved by jenkinsci/jenkins#7658 (perfect improvement) added on Jenkins 2.403

@tofanadrian3000
Copy link
Author

It's probably solved by jenkinsci/jenkins#7658 (perfect improvement) added on Jenkins 2.403

Thanks for your response, @benipeled! Unfortunately, that's not included into an LTS release at this time but once it is, I'll try it.

@pdk27
Copy link
Collaborator

pdk27 commented Jun 28, 2023

Related Issue: #347

@naraharip2017
Copy link
Collaborator

Issues addressed as part of #385. Please, re-open if experiencing any additional issues.

@pdk27
Copy link
Collaborator

pdk27 commented Jul 14, 2023

Released in ec2-fleet-2.7.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants