Decouple individual listener initialization during startup

[howardjohn]

Title: Decouple individual listener initialization during startup

Description:
Currently, all listeners are started at once during initialization. This means if Listener A is warming, Listener B will not start. This includes static listeners.

This behavior is discussed a bit in #6904 (though its also largely focusing on how this is exposed in config_dump).

This ends up being operationally risky. We expose stats through a (static) listener, but these cannot be accessed for pods that are stuck warming - when things are going wrong is exactly the time when we really want to see metrics to understand what is going on. Additionally, we serve health checks through (static) listeners. These should fail, but it would be nice to able to fail because we explicitly said "This is not ready yet for " rather than just refusing the connection entirely.

In general I am most concerned with static listeners starting, though I think that is fairly specific to the way we set things up today and doesn't seem like a general case.

One (of, I am sure, many) thing to keep in mind is there are dependencies between listeners for bind_to_port=false listeners. Likely these would need to be tied together in some way for initialization purposes

[nezdolik]

cc @kyessenov @ggreenway @alyssawilk

[ggreenway]

The idea makes sense to me, but as you've noted, you'll have to do this with great care, and probably not change the default behavior, or else you'll break peoples startup behavior.

[alyssawilk]

+1 to reasonable idea and implementing with great care. If well designed we might be able to get away with reload protecting if the differences are expected to be generally helpful and not harmful but worst case we can always add a knob.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.