Document upgrade process for production ECK deployments

[pebrc]

While the current upgrade guide explicitly mentions the fact that all clusters will go through a rolling restart for example when upgrading from a beta version to the 1.0 release, this is still very subtle and can be missed. In addition just rolling out the ECK upgrade maybe problematic if the fleet of managed clusters is large as it might lead to resource contention if all clusters are restarted at the same time etc.

Suggestion: create additional documentation around the upgrade process that outlines mitigating strategies. For example:

• pausing all reconciliations before the upgrade (but see Rename pause annotation #2594 )
• rolling out the operator upgrade
• successively un-pausing the reconciliation of the managed resources starting with the lowest priority resources and moving to the higher priority resources after that