reconcile sts/deploy if replica count changed

[himanshug]

Fixes #108

Description

This PR introduces the mechanism to do comparison of resources generated by druid-operator so as to reconcile them if they were changed directly. currently it only does so for the Deployment/StatefulSet replica counts but could be extended further as needed.

---

This PR has:

• been tested on a real K8S cluster to ensure creation of a brand new Druid cluster works.
• been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• [] added documentation for new or modified features or behaviors.

---

[AdheipSingh]

@himanshug in case we add spec here, wonder if this will conflict on scaling sts #97

Regardless are you planning to keep only replicas for now or extend for spec too

[himanshug]

for now, I am leaving it at just replica count mismatch ... but we can add more stuff here based on real world needs.

from feature perspective .. #97 and this , are different things ... however both might end up reusing some code for checking equality if that makes sense.

[AdheipSingh]

question :-

1. What will happen if someone delete's a deployment/sts, with this PR will operator create the sts/deploy ?

[himanshug]

What will happen if someone delete's a deployment/sts, with this PR will operator create the sts/deploy ?

operator would recreate those in 10secs in the worst case.

[connie-ru-wang]

Hi, is it possible that this PR broke the HPA functionality? I have a working hpa in a cluster with operator:0.0.4, but the hpa doesn't work in a cluster with operator:0.0.5 because it looks like the operator immediately reconciles the sts size back down when the hpa tries to scale up.

[AdheipSingh]

@connie-ru-wang
yeah i think that might have broken the HPA, we can add a condition to check in case HPA is enabled, or maybe add a flag for force maintaining the replica count at the operator level.
Regardless do you mind telling for which components are you using HPA for ?

cc @himanshug

[AdheipSingh]

BTW @connie-ru-wang it would be nice if you can create a separate issue for this, so that it can be tracked.
Thanks

[connie-ru-wang]

@AdheipSingh I've been testing HPA with routers, brokers, historicals. Although for historicals, most likely we will need v2beta2 support for the specific behavior policies that will prevent thrashing. https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior

Also just opened an issue for this: #160