Implement rolling upgrades with StatefulSets

[sebgl]

Handle cluster rolling upgrades with StatefulSets.

Relies on manually updating the rollingUpdate.Partition index of a sset when we're ready to restart the ES node.
For now, it only handles 1 node restart at a time, mostly because we rely on cluster health green checks.
It also introduces 2 additional concepts:

• StatefulSet updates expectations, based on UID + Generation: there are certain ES calls we don't want to make if our StatefulSet cache is out of date
• "lazy" ES client calls: some ES calls we do to handle the upgrade cannot rely on out-of-date information from observers. We need fresh ES information on the state of the cluster before eg. re-enabling shard allocations. However we don't want to perform these calls at every single reconciliation. I see this as a "stop-gap" implementation before we go on with a larger refactoring of sync/async ES calls.

This is still WIP since many things are still missing:

• changeBudget considerations.
• zen1/zen2 handling.
• Unit tests, everywhere. I still think a large portion of this code may be refactored in the short-term to account for change budget & zen1/zen2. Keeping unit tests for a later PR.
• E2E tests, kept for a later PR.

[thbkrkr]

The algorithm in operators/pkg/controller/elasticsearch/driver/upgrade.go looks good to me. Nice work!

[pebrc]

license header should be on top?

[sebgl]

Yes! I'll merge that one and fix it in #1262.

[pebrc]

I wasn't able to go as deep as I wanted but I found the overall logic pretty easy to follow which is a good sign and an improvement over what we had previously