Fix76: Scale STS / PVC vertically to support resizing of druid nodes

[AdheipSingh]

Fixes #76

Description

Sequence Flow

- For every node in array of nodes do the following if the node is statefulset
1. Get sts set
2. Get pvc referenced by that sts on label
3. GetDesiredPVCSize from CR
4. GetCurrentPVCSize from sts 
5. Validate the request, shrinkage of pvc not possible, so currentSize cannot be greater than desiredSize, reject request here
5. if GetDesiredPVCSize != GetCurrentPVCSize {
    then delete sts with cascade false
    }
6. Operator on reconcile created the sts, since it sees their is no sts present
7. GetPVCSizeofSts from pvc
8. If GetDesiredPVCSize != GetPVCSizeofSts {
      using deepcopy patch the pvc, dont delete the pvc, no use of update CRUD just patch 
   }

Know Issues

• This function to be called before STS Creation starts, so on a new druid where no sts is created this function will log saying it cannot find any sts for that nodeType ( cold, hot, mm etc )
• The program does not return here, continues to other flow, after sts is created on next reconcile this error should go away
• 10 sec reconcile being a bit fast, so at times patch may take few more sec, so their might be a error log saying sts could not be updated, but goes away quickly

TODO:

• the operator should not allow shrinkage of pvc claims, this should be validated by the operator and logged/sendEvents.
  Here's a sample log which shall be thrown

{"level":"error","ts":1606744048.5905652,"logger":"druid_operator_handler","msg":"Failed to Patch [PVC:cold-data-master-in-druid-historicals-cold-0] due to [PersistentVolumeClaim \"cold-data-master-in-druid-historicals-cold-0\" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value]","error":"Failed to Patch [PVC:cold-data-master-in-druid-historicals-cold-0] due to [PersistentVolumeClaim \"cold-data-master-in-druid-historicals-cold-0\" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value]"

• To validate that no other change apart from pvc size in condition

---

This PR has:

• been tested on a real K8S cluster to ensure creation of a brand new Druid cluster works.
• been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

---

Key changed/added files in this PR

• handler.go

@himanshug @nishantmonu51

[AdheipSingh]

Here is a sequence of logs

{"level":"info","ts":1606914406.0898578,"logger":"druid_operator_handler","msg":"Detected Change in VolumeClaimTemplate Sizes for Statefuleset [master-in-druid-historicals-cold] in Namespace [master-in], desVolumeClaimTemplateSize: [74Gi], currVolumeClaimTemplateSize: [73Gi]\n, deleteing STS [master-in-druid-historicals-cold] with casacde=false]"}
{"level":"info","ts":1606914407.2999537,"logger":"druid_operator_handler","msg":"[StatefuleSet:master-in-druid-historicals-cold] successfully deleted with casacde=false","name":"master-in","namespace":"master-in"}
{"level":"info","ts":1606914407.3025851,"logger":"druid_operator_handler","msg":"[PVC:cold-data-master-in-druid-historicals-cold-0] successfully Patched with [Size:73Gi]","name":"master-in","namespace":"master-in"}
{"level":"error","ts":1606914408.527233,"logger":"druid_operator_handler","msg":"Failed to update [StatefulSet:master-in-druid-historicals-cold] due to [StatefulSet.apps \"master-in-druid-historicals-cold\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden]

So the last log is fine, since it takes a bit time for reconcile to run and create new sts with the desired spec.

[AdheipSingh]

[AdheipSingh]

@himanshug can you pls review this

[AdheipSingh]

https://travis-ci.org/github/druid-io/druid-operator/builds/752248477#L234
rate limit 🤯

[himanshug]

just updating that I am aware of this one ... I did take a quick look today and I maybe wrong but it felt somewhat complex on first look for handling a very special case. I couldn't spend enough time to think through about how best to simplify it though or if that is even possible.

[AdheipSingh]

just updating that I am aware of this one ... I did take a quick look today and I maybe wrong but it felt somewhat complex on first look for handling a very special case. I couldn't spend enough time to think through about how best to simplify it though or if that is even possible.

:)
i am pretty sure this is the best way to upgrade sts, we have been using this in-house and haven't faced any downtime or disruptions during updates. The only function which i am not using is the checkIfStsHasEverythingElseSame since for me i really dont care if anything is updated or not. So thats why i requested your review/suggestion on the best way to check that ( open to change )

The only hack which i used is to sleep , this was just used so that the operator does not throw an error log since after deletion it takes some time to upgrade, regardless of the error log their is no issue, just wanted to keep logs clean :)

time.Sleep(10 * time.Second)

ill be happy to explain any issues/queries

[AdheipSingh]

@himanshug any update on this here ?

[AdheipSingh]

@himanshug over to you for review.

[nishantmonu51]

👍