cluster manager: dynamically inserted clusters should be warming at first

[Shikugawa]

This is related issue of #11120.
I think that the startup behavior of clusters which are triggered by addOrUpdate() is strange.
Facing problem is that dynamically inserted cluster which depends on SDS is always active even though depending TLS certificate is not initialized because some of issue occurred, like there is no cert info on DiscoveryResponse.
I think that this problem is caused by current behavior of ClusterManager. In current implementation, clusters inserted dynamically before all of init clusters haven't initialized will be marked immediately active. As a result, there is no indicator that inserted cluster is active or warming.
In this situation, dynamically inserted cluster (calls cluster_1) should walk state change like,

1. Stats is indicating that cluster_1 is warming when addOrUpdate() is called before all of init clusters are not initialized.
2. While there is no TLS certificate, SDS subscription belongs to cluster_1's transport socket factory is send DiscoveryRequest repeatedly until it reaches to initial_fetch_timeout, and stats is indicating warming. When it retrieved TLS certificate, the state of cluster_1 should change from warming to active.
3. After it reaches to initial_fetch_timeout and there is no certificate, stats should indicate cluster_1 is warming eternally.

It may be wrong understanding about cluster manager. But I think that we need to consider the initializing cluster's state change to resolve this problem.

cc. @howardjohn

[mattklein123]

I think this is a duplicate of #11120 or a more general solution to it? Can we track this issue in one place?

[Shikugawa]

@mattklein123 I think that this is a part of #11120. So not duplicate. More general solution is required. This issue should be focused on Cluster Management. And #11120 should be focused on SDS. In my thought, we should track the changes of behavior of dynamically inserted cluster in this issue.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[Shikugawa]

/nostale