Refactor downscales and add unit tests

[sebgl]

This refactors the StatefulSets downscale code into multiple smaller
functions, easier to test.
It also adds unit tests for most of those functions, and generally covers
the entire downscale codebase with unit tests.

Relates #1287.

[sebgl]

This refactoring does not limit the number of master nodes we can remove at once.
I'll reintroduce it as part of #1281.
Edit: that wasn't part of the original code either. Will definitely introduce it with #1281.

[david-kow]

Looks good, few comments.

[david-kow]

Nice.

[sebgl]

Jenkins test this please

[david-kow]

Jenkins test this please

[barkbay]

Not sure if it is related to this PR but I think I managed to have a race condition:

I quickly changed the number of expected Pods from 5 -> 3 -> 5
Pods have not been deleted:

NAME                                               READY   AGE   CONTAINERS      IMAGES
statefulset.apps/elasticsearch-sample-es-default   5/5     33m   elasticsearch   docker.elastic.co/elasticsearch/elasticsearch:7.2.0
NAME                                    READY   STATUS    RESTARTS   AGE   IP          NODE                                                 NOMINATED NODE   READINESS GATES
pod/elasticsearch-sample-es-default-0   1/1     Running   0          27m   10.60.1.3   gke-michael-dev-cluster-default-pool-e885d7bd-0f29   <none>           <none>
pod/elasticsearch-sample-es-default-1   1/1     Running   0          27m   10.60.0.9   gke-michael-dev-cluster-default-pool-e4e8e5d9-tt13   <none>           <none>
pod/elasticsearch-sample-es-default-2   1/1     Running   0          27m   10.60.2.7   gke-michael-dev-cluster-default-pool-88a37176-xfx3   <none>           <none>
pod/elasticsearch-sample-es-default-3   1/1     Running   0          27m   10.60.0.8   gke-michael-dev-cluster-default-pool-e4e8e5d9-tt13   <none>           <none>
pod/elasticsearch-sample-es-default-4   1/1     Running   0          27m   10.60.1.2   gke-michael-dev-cluster-default-pool-e885d7bd-0f29   <none>           <none>
pod/kibana-sample-kb-d48d76cdd-8s74l    1/1     Running   0          19m   10.60.1.4   gke-michael-dev-cluster-default-pool-e885d7bd-0f29   <none>           <none>

But I have the two last nodes excluded:

{
  "persistent" : { },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "exclude" : {
            "_name" : "elasticsearch-sample-es-default-4,elasticsearch-sample-es-default-3"
          }
        }
      }
    }
  }
}

[sebgl]

@barkbay good catch!
This is because we return early, but should not:

// scheduleDataMigrations requests Elasticsearch to migrate data away from leavingNodes.
func scheduleDataMigrations(esClient esclient.Client, leavingNodes []string) error {
	if len(leavingNodes) == 0 {
		return nil
	}
	log.V(1).Info("Migrating data away from nodes", "nodes", leavingNodes)
	return migration.MigrateData(esClient, leavingNodes)
}

I'll fix this.
Thanks!

[sebgl]

Last commit should correctly reset shard allocation excludes, similar to how done before.
I created #1522 to optimize for not doing it at every single reconciliation.

I think there is a potential race where we'd clear allocation excludes before the pod is completely deleted (if it's still being terminated for example). I'd like to handle that with another issue: #1523.

[barkbay]

LGTM

[anyasabo]

LGTM, just one question about the fakes

[david-kow]

LGTM