Allow overriding ctrl manager graceful shutdown timeout

[relu]

Overriding the default GracefulShutdownTimeout option given to the controller manager with a default of 0 (no timeout) since the helm operations are sensitive to interruption and can lead to leaving the HelmRelease in a bad state.

This will also allow users to override the option via a CLI flag -graceful-shutdown-timeout how much time to wait before forcibly exiting.

[relu]

I've added a commit to remove the readinessProbe as well since without this change graceful shutdown won't work anyway.
The alternative would be to adjust its failureThreshold so it allows the container to run for a longer period of time (needing to match the terminationGracePeriodSeconds which we have set to 600 by default) before being terminated, but I don't think this is quite the right solution.
All in all, I don't really think there's much value in having the readinessProbe configured because the helm-controller does not expose a service endpoint and thus readiness doesn't quite make sense anyway in its runtime lifecycle context.

[hiddeco]

Sorry for the late reply, was floored by COVID the past couple of days.

I am concerned about the readiness probe being removed for scenarios in which someone has configured multiple replicas, as this would report the non-elected Pod as being Ready while it is not actually handling reconciliation requests. Which does touch the runtime lifecycle context without it exposing a Service endpoint.

[stefanprodan]

I don't think we should remove readiness in this PR. If we decided to do this, it should be done across all controllers which don't have ClusterIP services like KC, IRC, IAC.

[relu]

Sorry for the late reply, was floored by COVID the past couple of days.

No worries, I'm sorry to hear that, hope you're feeling better now!

Thanks for the feedback. I agree with both of your assessments. I think ideally this problem should be addressed somehow at the level of the controller-runtime, having the behavior changed so the probes are not shut down during graceful termination.

Going to remove the second commit.

[stefanprodan]

LGTM

Thanks @relu

PS. Please add the new flag to the controller options documentation here: https://github.com/fluxcd/website/blob/main/content/en/flux/components/helm/options.md