fix: Add delay to prevent ssm rate limits if instance count is greater than max throughput

[devsh4]

Fix for bug #1841 which prevents hitting AWS SSM rate limits by adding a delay between subsequent putParameter calls.

Note: 25ms delay is based on the AWS service quotas as they only allow 40 requests per second to the parameter store by default.

[Tests]:

• Pre commit checks ✅
• Ran build locally ✅
• Ran linter ✅
• Ran jest tests ✅

Related PR (never got merged): #1843

[devsh4]

@npalm Looks like the jest step is failing because the coverage is dropping below 92%, however all the tests itself are passing.

Jest: "global" coverage threshold for branches (92%) not met: 91.37%

[npalm]

Guess this is caused since condional delay black is not touched in the test. This requires to add a test with adding 40 runners. Right now not more time to dig in further. Will check later.

[github-actions]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.

[devsh4]

@npalm Any chance you can look at this? I tried adding some tests with 40 runners but it didn't help.

[npalm]

Yes will do. But will be after next week.

[npalm]

@GuptaNavdeep1983 would you have tim to have a look on this PR as well?

[devsh4]

Any update on this? Looking to upgrade the module version but cannot until we have this PR merged in.

[npalm]

Sorry no updates, bit restricted by time. Sorry.

[GuptaNavdeep1983]

@devsh4 can you have a look at the recent changes?

[npalm]

The lodash throttle function seems not delay but discoarding calls duing the thrutttle time. I think for delay API call sw do not nee a extra labrary. We can use simple the setTimeout fucntion to avoid we hitting the API to hard.

Not sure why we start only throttling above 40, maybe better to start earlier. There is a unit test for creating 2 instances. Which failes when you set the limit 40 to 2. This because putParameter is never called.

[devsh4]

I agree with @npalm, we should revert this to what implementation I had before as I had tried something similar before and it didn't work.

@GuptaNavdeep1983 can you please revert this to the original change I had made? See more context below.

[npalm]

Ther refered AWS document only mention GetParameters to be rate limitted. The Lambda only calls putParatameter. I can't see a specific limit for the putParamaters. Hoever I have seen RateLimits as well when we try to scale aggressive. AWS also used a bucket to rate limite. As soon you bucket is empty ou got rate limitted. The moudle makes several AWS calls. PutParameter wil certainly tak a part in this.

But not sure if delaying the SSM put parameter calal will solve the problem. Once solving the proble here, maybe call the GetParamter from the instance start can be the next limit.

[devsh4]

AWS service quotas

@npalm yes I believe putParameter call does play a role in hitting rate limits. We are running the forked version of this repo with the initial change I made here and it has been running seamlessly since the past 6 months for spinning up more than ~250 runners at once without hitting rate limits.

[npalm]

AWS service quotas

@npalm yes I believe putParameter call does play a role in hitting rate limits. We are running the forked version of this repo with the initial change I made here and it has been running seamlessly since the past 6 months for spinning up more than ~250 runners at once without hitting rate limits.

Can you refer to the fork you running here (in case it is OS)?

[devsh4]

@npalm Sure forked version is here, it's linked above as well. FYI we are using Linux runners.

[devsh4]

@npalm Any update on this?

[github-actions]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed if no further activity occurs. Thank you for your contributions.