Desired manifest not updated on secret change

[mook-as]

Describe the bug
After a QuarksSecret referenced in a BOSHDeployment has been rotated, the corresponding desired-manifest is not updated.

To Reproduce

• Deploy KubeCF
• Trigger a rotation on the var-locket-database-password QuarksSecret.
• Watch the database-seeder job trigger.
• Watch the diego-api QuarksStatefulSet not restart and pick up the new password.
• Delete the diego-api pod(s) and see them fail to connect to the database (because they're using old credentials).

Expected behavior

• The desired manifest should update.
• The affected pods should restart.

Environment

• CF-Operator v6.0.4-0.gb44bb859
• KubeCF master-ish.

Additional context
This means a KubeCF user can rotate secrets, but once they do the whole cluster is busted.

If I spam the quarks.cloudfoundry.org/restart-on-update annotation on all my instance groups it seems to work. But I was under the impression that updateOnConfigChange was supposed to take care of that for me.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174916357

The labels on this github issue will be updated when the story is started.

[rohitsakala]

Hi @mook-as , fixed this with #1159. Please reopen if the issue still exists.