Recreate Pods if necessary after volume expansion

[sebgl]

K8s 1.15 supports ExpandInUsePersistentVolumes as Beta. Compatible storage drivers can resize the filesystem while in use. Storage drivers that are not compatible require the user to manually recreate the Pods.

We added support for volume expansion in #3752, but currently we don't restart Pods that may need to be restarted for the filesystem to be resized. We could include those as part of the regular rolling upgrade mechanism.

The condition to check is not trivial though:
Once resized, a Pod is updated with the following status condition: Waiting for user to (re-)start a pod to finish file system resize of volume on node. However this condition also appears (briefly) when the filesystem can be resized while in use, in which case we don't want to recreate the Pod. We could do something based on the last transition time of that condition, but it sounds brittle.
There might be a better condition to check. To investigate.

[david-kow]

An easy way out would be making this configurable on the operator level - don't rotate by default, always rotate if enabled. This wouldn't cover cases were multiple storage classes are used and resize support differs, but maybe that's fine.