Run helm test after chart deployed

[rndstr]

Once a Helm chart is deployed, the operator could run helm test and record its result as a HelmRelease condition.

What is a helm test?

A test in a helm chart lives under the templates/ directory and is a pod definition that specifies a container with a given command to run.

https://github.com/helm/helm/blob/master/docs/chart_tests.md
https://github.com/helm/helm/blob/master/docs/helm/helm_test.md

[hiddeco]

It would probably be best to make it configurable per release through a boolean flag in the HelmRelease spec (and probably incorporate the --cleanup and --parallel options too).

One tricky part is that helm test is a blocking operation, and we only have one worker processing items from the working queue, which means that if the tests take a long time, all other scheduled releases will stall until the tests have completed.

Another thing that may require some attention is #2006, which reacts on status updates, making it likely that if for some reason the roll-out of an upgrade fails but the tests are already running, the release is rolled back during testing. One solution I can think of to overcome this is to --wait on HelmReleases with rollbacks enabled, and rollback and return early when this results in an error, so the tests are never started. This equals to what --atomic does in the Helm client.

Edit (2019-06-03): I made changes to #2006 to mimic --atomic, so it should be easier to run the tests at the right moment.

[stefanprodan]

Alternative solution using Flagger fluxcd/flagger#196

[squaremo]

After a bit of discussion IRL, we decided that since helm test can take an arbitrary duration to run, it is not a good fit for the helm operator which seeks to respond to changes quickly. The expectation would be that updating a HelmRelease would be reflected in a release in some reasonable time, but it would be difficult to maintain that.

An alternative is to drive helm test at a higher (outer?) layer -- flagger will support this, for example, given suitably prepared charts.