[tempo-distributed] Tempo components doesn't reload config dynamically - Must Required Feature

[vaibhhavv]

As a user of tempo-distributed helm chart v1.16.0 into various production use cases, whenever I want to build a quick fix for any production issue, let's say I am changing some configurations in tempo.yaml or overrides.yaml configuration files.
After applying the configurations all the components pods restart and then load the new configuration.

The drawbacks related to the following approach are as follows:
1. If I am changing a parameter related to the querier, still the distributor is restarting along with all other components that have a dependency or no dependency on that config file.
2. Why does any component not work with the configuration auto reload approach? An automatic reload feature would be much more suitable for tempo-distributed architecture.

If we can do that, the following could be the benefits for the tempo:
1. Only dependent components should try to reload the configuration -> less downtime -> environment more stable.
2. We should have a built-in logic in all the dependent components for an automatic reload of configuration directly into the pod or as a sidecar -> Components will not have to restart every time -> Components like ingester which takes longer to load the WAL and get up, this time will get eliminated -> will lead tempo towards more stability.

If you also feel the need for this feature, please support the issue.

Thanks!