Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After setting up minimum spare instances - instances are not deprovisioned if demand is lower than requested #342

Closed
h-okon opened this issue Aug 10, 2022 · 1 comment · Fixed by #343
Labels

Comments

@h-okon
Copy link

h-okon commented Aug 10, 2022

When minimum spare instances is set - the instances do not deprovision regardless of the demand.

Describe the bug
When I use any number in minimum spare instances, the fleets do not change their size to a lower size. They go up to the maximum limit and stay at that limit even though there is no load / demand for the fleet.

To Reproduce

  1. Create cloud fleet with following parameters:
    Minimum Cluster Size - 1
    Maximum Cluster Size - 200
    Minimum spare size - 5
  2. Create another cloud fleet with simillar parameters
  3. Run some jobs on those workers, give some load so that the plugin scales the fleet to almost max.
  4. Run some jobs on the second worker
  5. After workload is done on the first fleet, and no further workload is given to the first fleet, we expect the plugin to rescale down to 5 nodes. It did not rescale at all and will be left at 200.
    etc.

Environment Details

Plugin Version?
2.5.1

Jenkins Version?
2.319.3

Spot Fleet or ASG?
Spot Fleet

Label based fleet?
Yes

Linux or Windows?
Linux

EC2Fleet Configuration as Code
We do not use JCasC.

Anything else unique about your setup?
No

@h-okon h-okon added the bug label Aug 10, 2022
@h-okon
Copy link
Author

h-okon commented Aug 10, 2022

Why this is happening?
Ec2FleetCloud.java:844 - getCurrentSpareInstanceCount
uses
enkins.get().getComputers()
and checks:
if (computer instanceof EC2FleetNodeComputer && !computer.isIdle())
to check how many workers are busy.

Just as a recap method should return a number of currently available spare instances for given fleet. This method is used to determine whether there should be a descaling action performed.

Instead this function calculates the number of available instances via following equation:

countOfInstances - currentBusyInstances
where:

  • countOfInstances is a number of instances in a fleet (from FleetStateStats) - this is correct
  • currentBusyInstances - number of busy instances on a whole jenkins instances, not only within one given fleet

This is okay if we have one fleet only, and no local workers
This would not work if we have either:

  • local slaves that can be busy as well,
  • another cloud fleets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
1 participant