[Fix] Fix computation of excess workload and available capacity

[pdk27]

Description

The plugin computes current demand and available capacity in order to identify excess workload and provision capacity for it. This code is run when jobs are added to queues.
Problems with current computations:

• Available capacity includes both connectingExecutors and plannedCapacitySnapshot. This can lead to availableCapacity being much higher than reality, esp. for huge fleets and a busy Jenkins with numerous builds running / queued.
• Current demand changes as capacity is provisioned but queue length remains same. This can be misleading and makes code and logs less readable, especially when dealing with negative current demand.

com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply In NodeProvisioner.StrategyDecision -> apply StrategyState{label=spot-workers, 
snapshot=LoadStatisticsSnapshot{definedExecutors=0, onlineExecutors=0, connectingExecutors=0, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}

currentDemand 1 availableCapacity 0 (availableExecutors 0 connectingExecutors 0 plannedCapacitySnapshot 0 additionalPlannedCapacity 0)

com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply In NodeProvisioner.StrategyDecision -> apply StrategyState{label=spot-workers, 
snapshot=LoadStatisticsSnapshot{definedExecutors=0, onlineExecutors=0, connectingExecutors=2, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=2, additionalPlannedCapacity=0}

currentDemand -3 availableCapacity 4 (availableExecutors 0 connectingExecutors 2 plannedCapacitySnapshot 2 additionalPlannedCapacity 0)

• Logs are misleading, show provisioning completed messages after planning to provision.

Context:

The plugin keeps track of new capacity planned to provision with plannedCapacitySnapshot, which is required to avoid over-provisioning. The plugin also controls when the Java futures for planned nodes are resolved - after Jenkins has established a connection to the new node or connection fails due to timeout i.e.plannedCapacitySnapshot includes nodes in ‘connecting’ state.

Reference of other plugins:

• Ec2-plugin: (doesn’t change what’s tracked in plannedCapacitySnapshot) - https://github.com/jenkinsci/ec2-plugin/blob/master/src/main/java/hudson/plugins/ec2/NoDelayProvisionerStrategy.java#L31
• Kubernetes-plugin - https://github.com/jenkinsci/kubernetes-ci-plugin/blob/master/src/main/java/com/elasticbox/jenkins/k8s/plugin/slaves/KubernetesSlaveProvisioningStrategy.java#L24

Related Issues:

• Huge lag in provisioning workers, workers become suspended but not deleted #322
• Plugin doesn't request new node due to incorrect currentDemand value #359

Testing done

Tested that results match expectations with a snapshot version of the plugin with:

• various configurations
• changing plugin config
• scaling ASG directly in AWS console
• multiple Jenkins (free style) projects

Sample logs:

• excess workload:

>> Provisioning:
In NodeProvisioner.StrategyDecision -> apply StrategyState{label=spot-workers, snapshot=LoadStatisticsSnapshot{definedExecutors=2, onlineExecutors=2, connectingExecutors=0, busyExecutors=2, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}  
label [spot-workers]: queueLength 1 availableCapacity 0 (availableExecutors 0 plannedCapacitySnapshot 0 additionalPlannedCapacity 0)  
EC2FleetCloud [spot-workers] excessWorkload 1  
EC2FleetCloud [spot-workers] to provision = 1  
Planned 1 new nodes  
Started provisioning FleetNode-EC2FleetCloud-6 from EC2FleetCloud with 1 executors. Remaining excess workload: 0  
Provisioning completed 

>> Update cycle (actual provisioning using EC2 APIs):
EC2FleetCloud [spot-workers] start cloud com.amazon.jenkins.ec2fleet.EC2FleetCloud@2a1fd5ad  
EC2FleetCloud [spot-workers] Set target capacity to '3'  

>> ... After successful connection...

[hudson.slaves.NodeProvisioner lambda$update$6] FleetNode-EC2FleetCloud-3 provisioning successfully completed. We have now 4 computer(s)  

Before:

In NodeProvisioner.StrategyDecision -> apply StrategyState{label=spot-workers, snapshot=LoadStatisticsSnapshot{definedExecutors=3, onlineExecutors=3, connectingExecutors=0, busyExecutors=3, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}  
label [spot-workers]: currentDemand 1 availableCapacity 0 (availableExecutors 0 connectingExecutors 0 plannedCapacitySnapshot 0 additionalPlannedCapacity 0)  
EC2FleetCloud [spot-workers] excessWorkload 1  
EC2FleetCloud [spot-workers] to provision = 1  
Planned 1 new nodes  
After provisioning currentDemand=-1  
label [spot-workers]: currentDemand is less than 1, not provisioning  
Provisioning completed  

• no excess workload:

In NodeProvisioner.StrategyDecision -> apply StrategyState{label=spot-workers, snapshot=LoadStatisticsSnapshot{definedExecutors=5, onlineExecutors=3, connectingExecutors=0, busyExecutors=3, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=2, additionalPlannedCapacity=0}  
label [spot-workers]: queueLength 1 availableCapacity 2 (availableExecutors 0 plannedCapacitySnapshot 2 additionalPlannedCapacity 0)  
label [spot-workers]: No excess workload, provisioning not needed. 

Before:

In NodeProvisioner.StrategyDecision -> apply StrategyState{label=spot-workers, snapshot=LoadStatisticsSnapshot{definedExecutors=3, onlineExecutors=2, connectingExecutors=0, busyExecutors=2, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=1, additionalPlannedCapacity=0}  
label [spot-workers]: currentDemand 0 availableCapacity 1 (availableExecutors 0 connectingExecutors 0 plannedCapacitySnapshot 1 additionalPlannedCapacity 0)  
label [spot-workers]: currentDemand is less than 1, not provisioning  
Provisioning completed  

[cjerad]

lgtm 👍