live app upgrade

[joel-bluedata]

This would be one of the most useful & impressive features we could add in the near term.

App upgrade could be driven by an edit to an existing KubeDirectorApp CR, preserving its name but changing its version string. Obviously currently we block such edits altogether if any clusters are using the app, but if we support upgrade we can allow them.

• We may need to limit what fields can be changed, but I think we might be surprised at how completely we can change the app spec.
• We may need to limit how the version string is allowed to be changed. E.g. don't go backwards (semantic-version-wise) if any current clusters are using it?

If cluster reconciliation notices that the app version is different from what it is, it can kick off a statefulset update.

• We probably want the update strategy for our statefulsets to be OnDelete so we can manage the staging explicitly.
• It would be useful for the KubeDirectorApp to state if there are ordering requirements between the roles when updating.

This is obviously a lifecycle event, and post-update we need to be able to trigger a script hook insider the containers to do app-specific stuff.

The really tricky question here is what to do with any directories mounted from persistent volumes. Here's a strawman:

• When the initcontainer comes back up after upgrade, it notices if an upgrade has happened (various ways to drive this).
• If this is first restart post-upgrade, the existing /mnt dir is moved to /old_mnt and a new /mnt dir is constructed from the potentially new set of persist dirs and potentially new image. (Any existing old_mnt is discarded.)
• The app pod would need to always mount /old_mnt as well as its usual mounts.
• The script for the upgrade event hook can copy over the stuff from old_mnt that it cares about. Can also clean stuff out of old_mnt to save space.

This has some big downsides. For one, you must ALWAYS implement an upgrade script hook, or you lose your persisted data. For two, you have to keep overhead around to potentially double your persistent data storage on update. So yeah it's just a strawman but perhaps it is pointing in the general direction of a solution.

One way we could tackle those drawbacks is that a KDApp could identify what specific app versions it supports upgrading from, and what to do with the persisted directories when coming from each of those various older versions. Ideally some or all of the persisted directories could be left alone on upgrade. We may also need to get more fine-grained about specifying which subdirectories (or even individual files?) should be on the persistent storage; current app CRs use a broad brush.

Statefulset update is the trickiest part but not the only thing to manage; we'd also need to enhance handleMemberServiceConfig so that it can support reconciling a service object to change its ports.

==========

A related issue is how much are concerned about detecting/preventing out-of-band fiddling with the statefulset spec, i.e. trying to do updates to a KDCluster without actually editing the KDApp. If we truly want/need to block this, we would need another admission controller watching statefulsets.

[kmathur2]

Rough notes -

1. As a first step, we should allow just individual role's image upgrades. This itself would be a huge step forward.
2. We should leverage statefulset inplace upgrade feature for already running kdcluster in the namespace
3. We should unblock/remove validation to update kdapp role images even if kdcluster exists.
4. The kdapp reconciller should update statefulset images for kdcluster's in configured state only
5. Should we introduce new field in status stanza of kdcluster to show that a role was upgraded or being upgraded?
6. @joel-bluedata do we have to worry about the back-up CRs? (joel says: probably not)
7. If they have persistent dirs, statefulset upgrade should take care of that, but we should verify.
8. Need to decide if distroID should be updated? (joel says: just the version, not the distroID)

[joel-bluedata]

The Ampool guys had an interesting request here. When one of their workers goes down and comes back up, there's a subsequent period of time in which it could be bad for another worker to go down, before the system has done some necessary rebalancing/repopulating. In general this idea of application-gated upgrade progress is probably a good thing to support (added value).

The in-container app code knows when it is OK to proceed with the next pod upgrade, but how would we interface that with whatever code/controller is actually doing the pod upgrades?

KD could control the upgrade progress in various ways:

• The "vanilla" use of RollingUpdate proceeds to the next pod whenever the upgraded pod comes back as Ready. This is too eager in this case, but we could use custom readinessGates on the pod so we can explicitly set pod readiness from KubeDirector.

• Or, we could set a partition value for the RollingUpdate so that the next pod won't immediately be upgraded, and then let KD explicitly keep decreasing that value to allow the next pod to be upgraded.

• Or, we could use the OnDelete strategy instead of RollingUpdate, and have KD just explicitly delete each pod when it is OK to upgrade it. (This might be cleanest?)

The other side of this question is how KD knows when it is OK to proceed. In the case of Ampool we could e.g. run a script hook in the controller container to see what it says, but I'm wondering how the idea of "what to do to learn if it's ok to keep upgrading" can be generalized and expressed in the kdapp.

Food for thought.

[joel-bluedata]

Will be filing a bunch of finer-grained issues for work on the live-app-upgrade branch.