Add only one master node at a time

[sebgl]

To better match zen1 and zen2 requirements (especially zen 1),
we should only create one master node at a time.

This is achieved here by:

• slowly incrementing master StatefulSets replicas, as long as the
  upscaleState allows us to do so
• inspecting the current nodes at upscaleState initialization. If the
  cluster is not bootstrapped yet, any number of masters can be created.
  Else, we allow master node creation only if there's not a master in the
  process of being created already.
• because the above point can hardly tolerate an out-of-date cache, we
  check pods vs. StatefulSet spec expectations beforehand.
• zen1 & zen2 configuration in elasticsearch.yml now needs to be patched
  after adjusting the number of replicas

Some side-effects:

• cluster bootstrap/UUID annotation is moved to its own file since not
  only related to zen2 anymore
• downscale expectations checks can be removed, since checked at a
  higher level
• ESState is shared between upscales and rolling upgrades

[sebgl]

When moving to onDelete, I would expect this to become:

1. check that the expected number of pods is running (same first half of the function)
2. check our deletions expectations: if we did client.Delete(pod, ...) at some point, we should make sure this pod has indeed disappeared from our cache here. This would replace the second half of the function: we would not really care about revision here. As long as all pods are there, and no deletion is in progress (for potential pod recreation by the sset controller), I think we should be fine?

[sebgl]

So after this state we may have updated some ssets, so we'll end up with sset update conflicts in next steps (downscale, rolling upgrade). Which is fine, but I'm wondering whether we should maybe either:

• requeue here if the downscale step did any change
• re-check expectations here to catch any change

[barkbay]

We may also recheck expectations before downscale and handleRollingUpgrades It would make it explicit and maybe less error prone when this code will be updated.

[barkbay]

Should we not update the expectation for this statefulset if it is updated ?
e.g.:

1. an upscale from 2 to 3 masters has just be done
2. in the meantime a new statefulset with some new masters has been added by the user
3. if we have a stale cache for the first statefulset we would add a fourth master with a m_m_n of 2

[barkbay]

My above comment assumes that the new statefulset is at the top of the list returned by adjustResources and would have precedence over the first one.

[pebrc]

I think you are correct. Also maybe I am missing something but it looks to me like the limiting is only happening on replica increases of existing ssets but does not limit the addition of completely new ones

[pebrc]

Yes so I just confirmed that my reading of the code is correct:
If you go from

apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elasticsearch-test
spec:
  version: 7.3.0
  nodes:
  - name: master-1
    config:
      node.master: true
      node.data: true
    nodeCount: 3

to

apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  name: elasticsearch-test
spec:
  version: 7.3.0
  nodes:
  - name: master-1
    config:
      node.master: true
      node.data: true
    nodeCount: 3
  - name: master-2
    config:
      node.master: true
      node.data: true
    nodeCount: 3

the new master nodes in the master-2 sset will be created all at once.

[pebrc]

Discussed with @barkbay that we are going to merge as is, so that work on #1442 can build on top of it. We will create two follow up issues to track the issue around expectations on new stateful sets and limiting master creation on new stateful sets