Feat: Cluster Topology Awareness

[proost]

Summary

Adding Cluster Topology Awareness to Cluster Client. Cluster Topology Awareness makes usage of “Elasticache Auto Scaling With Replicas” more efficient

Problem Definition

Background

Elasticache Redis supports Auto Scaling With Replicas. Depending on target util or schedule, cluster can scale out or scale in automatically.

Currently ruedis can handle scale-in situation. When nodes disappear, client refresh cluster cache and connections.

But when scale-out situation, especially replicas added, rueidis can’t notice that cluster is changed. It is blocker to use Elasticache Auto Scaling With Replicas effectively.

Goals

• Periodically sync with redis cluster.

Non-Goals

• Always refresh rueidis client.
• Immeditately sync when redis cluster is changed.

Plan

ClusterTopologyRefreshmentOption

Cluster topology refreshment must need to how refreshment will work. Definitely, we must need two options.

• ScanInterval : how frequently scan the redis cluster.
• RefreshCoolingDownPeriod
  • A duration that rueidis client can update cluster cache from last cluster cache is changed.
  • Change redis cluster takes lots of time. Because of this, refreshment can happen too often. Even if cluster might be changed, refreshment can be delayed.
  • This option does not affect refreshment when got error from redis cluster. This option only works cluster topology refreshment loop only.

// ClusterTopologyRefreshmentOption is the options for the cluster topology refreshment.
type ClusterTopologyRefreshmentOption struct {
	// ScanInterval is the interval to scan the cluster topology.
	// If the value is zero, refreshment will be disabled.
	ScanInterval time.Duration

	// RefreshCoolingDownPeriod is the period to wait before the next refreshment.
	// If the value is zero, the scan and refreshment will always be triggered when cluster topology is not up-to-date.
	// If the value is more than zero, the scan and refreshment will be triggered only when the last refreshment is older than the period.
	RefreshCoolingDownPeriod time.Duration
}

Cluster Topology Refreshment Loop

• loop starts in newClusterClient with goroutine. loop ends when Close is called. To manage lifecycle of goroutine, i will use context.Context .
• we should store when last refreshment happen. Using lastRefreshTime , we calculate whether cooling down period is over or not.
• lastRefreshTime is updated when _refresh is called. Purpose of RefreshCoolingDownPeriod is delaying refreshment. So Where refreshment happen doesn’t matter. we don’t want update cluster cache too frequently.

func (c *clusterClient) runClusterTopologyRefreshment(ctx context.Context) {
	ticker := time.NewTicker(c.opt.ClusterTopologyRefreshmentOption.ScanInterval)
	defer ticker.Stop()
	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			lastRefreshTime := c.lastRefreshTime.Load()
			if time.Since(*lastRefreshTime) <= c.opt.ClusterTopologyRefreshmentOption.RefreshCoolingDownPeriod {
				continue
			}

			// 1. scan cluster
			// 2. calculate diffs cluster topology
			// 3. refresh if there is diffs 
		}
	}
}

• scanning cluster code and calculating diffs cluster topology code will same as in _refresh
• when there are diffs or failed to scan cluster, then i call lazyrefresh

[rueian]

Hi @proost, thank you for the proposal. I think it is great, but I would like it to be more simplified:

1. A RefreshCoolingDownPeriod option seems unnecessary to me. We should just use lazyrefresh which cools down for 1 second. I also believe there is no good reason to use other period values.
2. ClusterTopologyRefreshmentOption.ScanInterval is too long. Maybe just ShardsRefreshInterval.
3. A simple channel is enough to exit the goroutine. No need to use context.Context.

[proost]

@rueian
I agree 1, 3. I will change plan.

you are right. ClusterTopologyRefreshmentOption.ScanInterval is too long. But maybe later someone want more additional options to ClusterTopologyRefreshmentOption like lettuce case. In that case, wrapping with struct is more reasonable to me. What do you think about it?

[rueian]

okay, but ClusterTopologyRefreshmentOption is too limited. How about ClusterOption? We can put all cluster-related options into it.

[proost]

close due to #638