Packed: Fleet scaled down removes nodes from least used Nodes

This implements the strategy such that when a Fleet is scaled down, and has the "Packed" strategy, it removed GameServers from Nodes that have the least GameServers running on them. This also fixes up some issues with the OpenAPI validation as well.
googleforgames · Oct 25, 2018 · ed1c1d1 · ed1c1d1
1 parent 597344e
commit ed1c1d1
Show file tree

Hide file tree

Showing 13 changed files with 210 additions and 153 deletions.
diff --git a/docs/scheduling_autoscaling.md b/docs/scheduling_autoscaling.md
@@ -7,14 +7,23 @@
 Table of Contents
 =================
 
-* [Fleet Autoscaling](#fleet-autoscaling)
-* [Autoscalng Concepts](#autoscalng-concepts)
-   * [Allocation Scheduling](#allocation-scheduling)
-* [Fleet Scheduling](#fleet-scheduling)
-   * [Packed](#packed)
-      * [Allocation Scheduling Strategy](#allocation-scheduling-strategy)
-   * [Distributed](#distributed)
-      * [Allocation Scheduling Stategy](#allocation-scheduling-stategy)
+   * [Scheduling and Autoscaling](#scheduling-and-autoscaling)
+   * [Table of Contents](#table-of-contents)
+      * [Fleet Autoscaling](#fleet-autoscaling)
+      * [Autoscaling Concepts](#autoscaling-concepts)
+         * [Allocation Scheduling](#allocation-scheduling)
+         * [Pod Scheduling](#pod-scheduling)
+         * [Fleet Scale Down Strategy](#fleet-scale-down-strategy)
+      * [Fleet Scheduling](#fleet-scheduling)
+         * [Packed](#packed)
+            * [Allocation Scheduling Strategy](#allocation-scheduling-strategy)
+            * [Pod Scheduling Strategy](#pod-scheduling-strategy)
+            * [Fleet Scale Down Strategy](#fleet-scale-down-strategy-1)
+         * [Distributed](#distributed)
+            * [Allocation Scheduling Strategy](#allocation-scheduling-strategy-1)
+            * [Pod Scheduling Strategy](#pod-scheduling-strategy-1)
+            * [Fleet Scale Down Strategy](#fleet-scale-down-strategy-2)
+
 
 Scheduling and autoscaling go hand in hand, as where in the cluster `GameServers` are provisioned
 impacts how to autoscale fleets up and down (or if you would even want to)
@@ -42,6 +51,11 @@ Each `GameServer` is backed by a Kubernetes [`Pod`](https://kubernetes.io/docs/c
 refers to the strategy that is in place that determines which node in the Kubernetes cluster the Pod is assigned to,
 when it is created.
 
+### Fleet Scale Down Strategy
+
+Fleet Scale Down strategy refers to the order in which the `GameServers` that belong to a `Fleet` are deleted, 
+when Fleets are shrunk in size.
+
 ## Fleet Scheduling
 
 There are two scheduling strategies for Fleets - each designed for different types of Kubernetes Environments.
@@ -74,8 +88,7 @@ for the infrastructure you use.
 It attempts to _pack_ as much as possible into the smallest set of nodes, to make
 scaling infrastructure down as easy as possible.
 
-Currently, Allocation scheduling is the only aspect this strategy affects, but in future releases it will
-also affect `GameServer` `Pod` scheduling, and `Fleet` scale down scheduling as well.
+This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.
 
 #### Allocation Scheduling Strategy
 
@@ -91,6 +104,11 @@ topology. This attempts to group together `GameServer` Pods within as few nodes
 > The default Kubernetes scheduler doesn't do a perfect job of packing, but it's a good enough job for what we need - 
   at least at this stage. 
 
+#### Fleet Scale Down Strategy
+
+With the "Packed" strategy, Fleets will remove `Ready` `GameServers` from Nodes with the _least_ number of `Ready` and 
+`Allocated` `GameServers` on them. Attempting to empty Nodes so that they can be safely removed.
+
 ### Distributed
 
 ```yaml
@@ -118,8 +136,7 @@ on bare metal, and the cluster size rarely changes, if at all.
 This attempts to distribute the load across the entire cluster as much as possible, to take advantage of the static
 size of the cluster.
 
-Currently, the only thing the scheduling strategy affects is Allocation scheduling, but in future releases it will
-also affect `GameServer` `Pod` scheduling, and `Fleet` scaledown scheduling as well.
+This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.
 
 #### Allocation Scheduling Strategy
 
@@ -129,4 +146,9 @@ number of allocated `GameServers` on them.
 #### Pod Scheduling Strategy
 
 Under the "Distributed" strategy, `Pod` scheduling is provided by the default Kubernetes scheduler, which will attempt
-to distribute the `GameServer` `Pods` across as many nodes as possible.
+to distribute the `GameServer` `Pods` across as many nodes as possible.
+
+#### Fleet Scale Down Strategy
+
+With the "Distributed" strategy, Fleets will remove `Ready` `GameServers` from Nodes with at random, to ensure
+a distributed load is maintained.
diff --git a/examples/simple-udp/fleet.yaml b/examples/simple-udp/fleet.yaml
@@ -25,7 +25,6 @@ spec:
     spec:
       ports:
       - name: default
-        portPolicy: "dynamic"
         containerPort: 7654
       template:
         spec:

diff --git a/install/helm/agones/templates/crds/_gameserverspecvalidation.yaml b/install/helm/agones/templates/crds/_gameserverspecvalidation.yaml
@@ -57,7 +57,7 @@ properties:
       ports:
         title: array of ports to expose on the game server container
         type: array
-        minItems: 0 # make this 1 in 0.4.0
+        minItems: 1
         required:
           - containerPort
         items:
@@ -91,34 +91,11 @@ properties:
               type: integer
               minimum: 1
               maximum: 65535
-      portPolicy: # remove this in 0.4.0
-        title: the port policy that will be applied to the game server
-        description: |
-            portPolicy has two options:
-            - "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
-            - "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
-            port is available. When static is the policy specified, `hostPort` is required to be populated
+      scheduling:
         type: string
         enum:
-        - dynamic
-        - static
-      protocol: # remove this in 0.4.0
-        title: Protocol being used. Defaults to UDP. TCP is the only other option
-        type: string
-        enum:
-        - UDP
-        - TCP
-      containerPort: # remove this in 0.4.0
-        title: The port that is being opened on the game server process
-        type: integer
-        minimum: 1
-        maximum: 65535
-      hostPort: # remove this in 0.4.0
-        title: The port exposed on the host
-        description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
-        type: integer
-        minimum: 1
-        maximum: 65535
+        - Packed
+        - Distributed
       health:
         type: object
         title: Health checking for the running game server

diff --git a/install/helm/agones/templates/crds/fleet.yaml b/install/helm/agones/templates/crds/fleet.yaml
@@ -43,9 +43,15 @@ spec:
             replicas:
               type: integer
               minimum: 0
+            scheduling:
+              type: string
+              enum:
+              - Packed
+              - Distributed
             strategy:
               properties:
                 type:
+                  type: string
                   enum:
                     - Recreate
                     - RollingUpdate

diff --git a/install/helm/agones/templates/crds/fleetautoscaler.yaml b/install/helm/agones/templates/crds/fleetautoscaler.yaml
@@ -50,8 +50,9 @@ spec:
                 - type
               properties:
                 type:
+                  type: string
                   enum:
-                    - Buffer
+                  - Buffer
                 buffer:
                   required:
                     - maxReplicas

diff --git a/install/helm/agones/templates/crds/gameserverset.yaml b/install/helm/agones/templates/crds/gameserverset.yaml
@@ -44,5 +44,10 @@ spec:
             replicas:
               type: integer
               minimum: 0
+            scheduling:
+              type: string
+              enum:
+              - Packed
+              - Distributed
             template:
               {{- include "gameserver.validation" . | indent 14 }}
diff --git a/install/yaml/install.yaml b/install/yaml/install.yaml
@@ -184,9 +184,15 @@ spec:
             replicas:
               type: integer
               minimum: 0
+            scheduling:
+              type: string
+              enum:
+              - Packed
+              - Distributed
             strategy:
               properties:
                 type:
+                  type: string
                   enum:
                     - Recreate
                     - RollingUpdate
@@ -234,7 +240,7 @@ spec:
                     ports:
                       title: array of ports to expose on the game server container
                       type: array
-                      minItems: 0 # make this 1 in 0.4.0
+                      minItems: 1
                       required:
                         - containerPort
                       items:
@@ -268,34 +274,11 @@ spec:
                             type: integer
                             minimum: 1
                             maximum: 65535
-                    portPolicy: # remove this in 0.4.0
-                      title: the port policy that will be applied to the game server
-                      description: |
-                          portPolicy has two options:
-                          - "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
-                          - "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
-                          port is available. When static is the policy specified, `hostPort` is required to be populated
+                    scheduling:
                       type: string
                       enum:
-                      - dynamic
-                      - static
-                    protocol: # remove this in 0.4.0
-                      title: Protocol being used. Defaults to UDP. TCP is the only other option
-                      type: string
-                      enum:
-                      - UDP
-                      - TCP
-                    containerPort: # remove this in 0.4.0
-                      title: The port that is being opened on the game server process
-                      type: integer
-                      minimum: 1
-                      maximum: 65535
-                    hostPort: # remove this in 0.4.0
-                      title: The port exposed on the host
-                      description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
-                      type: integer
-                      minimum: 1
-                      maximum: 65535
+                      - Packed
+                      - Distributed
                     health:
                       type: object
                       title: Health checking for the running game server
@@ -421,8 +404,9 @@ spec:
                 - type
               properties:
                 type:
+                  type: string
                   enum:
-                    - Buffer
+                  - Buffer
                 buffer:
                   required:
                     - maxReplicas
@@ -515,7 +499,7 @@ spec:
             ports:
               title: array of ports to expose on the game server container
               type: array
-              minItems: 0 # make this 1 in 0.4.0
+              minItems: 1
               required:
                 - containerPort
               items:
@@ -549,34 +533,11 @@ spec:
                     type: integer
                     minimum: 1
                     maximum: 65535
-            portPolicy: # remove this in 0.4.0
-              title: the port policy that will be applied to the game server
-              description: |
-                  portPolicy has two options:
-                  - "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
-                  - "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
-                  port is available. When static is the policy specified, `hostPort` is required to be populated
-              type: string
-              enum:
-              - dynamic
-              - static
-            protocol: # remove this in 0.4.0
-              title: Protocol being used. Defaults to UDP. TCP is the only other option
+            scheduling:
               type: string
               enum:
-              - UDP
-              - TCP
-            containerPort: # remove this in 0.4.0
-              title: The port that is being opened on the game server process
-              type: integer
-              minimum: 1
-              maximum: 65535
-            hostPort: # remove this in 0.4.0
-              title: The port exposed on the host
-              description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
-              type: integer
-              minimum: 1
-              maximum: 65535
+              - Packed
+              - Distributed
             health:
               type: object
               title: Health checking for the running game server
@@ -647,6 +608,11 @@ spec:
             replicas:
               type: integer
               minimum: 0
+            scheduling:
+              type: string
+              enum:
+              - Packed
+              - Distributed
             template:              
               required:
               - spec
@@ -691,7 +657,7 @@ spec:
                     ports:
                       title: array of ports to expose on the game server container
                       type: array
-                      minItems: 0 # make this 1 in 0.4.0
+                      minItems: 1
                       required:
                         - containerPort
                       items:
@@ -725,34 +691,11 @@ spec:
                             type: integer
                             minimum: 1
                             maximum: 65535
-                    portPolicy: # remove this in 0.4.0
-                      title: the port policy that will be applied to the game server
-                      description: |
-                          portPolicy has two options:
-                          - "dynamic" (default) the system allocates a free hostPort for the gameserver, for game clients to connect to
-                          - "static", user defines the hostPort that the game client will connect to. Then onus is on the user to ensure that the
-                          port is available. When static is the policy specified, `hostPort` is required to be populated
+                    scheduling:
                       type: string
                       enum:
-                      - dynamic
-                      - static
-                    protocol: # remove this in 0.4.0
-                      title: Protocol being used. Defaults to UDP. TCP is the only other option
-                      type: string
-                      enum:
-                      - UDP
-                      - TCP
-                    containerPort: # remove this in 0.4.0
-                      title: The port that is being opened on the game server process
-                      type: integer
-                      minimum: 1
-                      maximum: 65535
-                    hostPort: # remove this in 0.4.0
-                      title: The port exposed on the host
-                      description: Only required when `portPolicy` is "static". Overwritten when portPolicy is "dynamic".
-                      type: integer
-                      minimum: 1
-                      maximum: 65535
+                      - Packed
+                      - Distributed
                     health:
                       type: object
                       title: Health checking for the running game server

diff --git a/pkg/apis/stable/v1alpha1/gameserver.go b/pkg/apis/stable/v1alpha1/gameserver.go
@@ -118,7 +118,7 @@ type GameServerSpec struct {
 	// Health configures health checking
 	Health Health `json:"health,omitempty"`
 	// Scheduling strategy. Defaults to "Packed".
-	Scheduling SchedulingStrategy `json:"scheduling"`
+	Scheduling SchedulingStrategy `json:"scheduling,omitempty"`
 	// Template describes the Pod that will be created for the GameServer
 	Template corev1.PodTemplateSpec `json:"template"`
 }

diff --git a/pkg/apis/stable/v1alpha1/gameserverset.go b/pkg/apis/stable/v1alpha1/gameserverset.go
@@ -57,7 +57,7 @@ type GameServerSetSpec struct {
 	// Replicas are the number of GameServers that should be in this set
 	Replicas int32 `json:"replicas"`
 	// Scheduling strategy. Defaults to "Packed".
-	Scheduling SchedulingStrategy `json:"scheduling"`
+	Scheduling SchedulingStrategy `json:"scheduling,omitempty"`
 	// Template the GameServer template to apply for this GameServerSet
 	Template GameServerTemplateSpec `json:"template"`
 }