feat(vector): Add option to roll vector pods on secrets and extra objects change

[anas-aso]

Changes to secrets or extra manifests defined does not trigger a rolling restart.
This PR adds an option to do that.

Normally I would turn .Values.rollWorkload into an object, then defined something like this

rollWorkload:
  configMap: true
  secrets: false
  extraObjects: false

but that would be a breaking change. Maybe something to consider for the next major release of this chart.

[jszwedko]

Thanks @anas-aso ! I agree that it'd be nice to turn rollWorkload into an object, but your approach avoids the breaking change. Out of curiosity, do you think it'd make sense to have rollWorkload roll if any of the config map, secrets, or extra objects changes? Or is it useful to be able to configure which changes will cause a roll?

[anas-aso]

do you think it'd make sense to have rollWorkload roll if any of the config map, secrets, or extra objects changes? Or is it useful to be able to configure which changes will cause a roll

@jszwedko depends on the use case. For things where Vector's hot reload works (--watch-config), I would prefer to not recreate the pods. But for things like environment variables read from a k8s secret (example below), a rolling restart is a must.

env:
  - name: MY_SINK_TOKEN
    valueFrom:
      secretKeyRef:
        name: vector-aggregator
        key: MY_SINK_TOKEN

My use case is simple:

1. hot reload Vector on config change (faster deployment time)
2. recreate pods when an env variable read from a k8s secrets is changed.

I understand the concern of adding more knobs to the chart, so I am open to other suggestions if you have any.

[jszwedko]

Gotcha, I see, this change seems reasonable then to be able to only role when, for example, secrets change but not otherwise.