High Availability with Multi-AZ Deployments

[mattisonchao]

Motivation

As the distributed storage systems, we must let Oxia support the AZ ensemble placement function, enhancing fault tolerance and disaster recovery.
We can even deploy the Oxia server into different AZs, but the coordinator might still assign shard replicas for the servers in the same zone. Therefore, the primary purpose of this proposal is to support the Oxia coordinator in coordinating data shards into different time zones to satisfy the AZ ensemble placement policy.

Goal

• Support AZ awareness for oxia-coordinator, which includes ensemble generation and load balancing.
• Support different AZ policies at the namespace level. Such as basic zonal or regional.

Out of scope

I understand it's a big topic here; we can make many improvements related to the load balancer and shards assignments. However, we should focus only on the main purpose of this proposal. We can talk about others in the future. So, there are several topics we will not discuss in the current proposal:

• namespace AZ preference.
• observer(async replication) mechanism.
• ...

High-Level Design

Based on the above diagram, you will understand the primary purpose of AZ-aware placement.

Zonal ensemble placement

This rule will let the coordinator assign all the shards in the same zone (green colour). It will help some namespace uses for lower latency, cross-zone, cost-free, multi-replica disaster recovery cases.

Reginal ensemble placement

This is the default rule for ensemble placement, which will assign all shard replicas in the different zones. Regional AZ gives the oxia cluster a very high level of disaster recovery ability—which is very important for metadata(coordination) service.

Details Design

Model

1. Introduce an AvailableZone field for the ServerAddress struct.

type ServerAddress struct {
	//AvailableZone
	AvailableZone string `json:"availableZone" yaml:"availableZone"`

	// Public is the endpoint that is advertised to clients
	Public string `json:"public" yaml:"public"`

	// Internal is the endpoint for server->server RPCs
	Internal string `json:"internal" yaml:"internal"`
}

2. Introduce an AvailableZoneRule field for NamespaceConfig struct.

type AvailableZoneRule string

const (
	AZRuleZonal    AvailableZoneRule = "zonal"
	AZRuleRegional AvailableZoneRule = "regional"
)

type NamespaceConfig struct {
	Name              string            `json:"name" yaml:"name"`
	AvailableZoneRule AvailableZoneRule `json:"availableZoneRule" yaml:"availableZoneRule"`
	InitialShardCount uint32            `json:"initialShardCount" yaml:"initialShardCount"`
	ReplicationFactor uint32            `json:"replicationFactor" yaml:"replicationFactor"`
}

Component

This proposal will introduce a new component AZRankClusterRebalancer with a ClusterRebalancer interface to support calculating ensemble with AZ awareness and server shards ranking.

The interface definition is as follows:

type ClusterRebalancer interface {

	GetEnsemble(shard int64, namespace *model.NamespaceConfig, candidates []model.ServerAddress, status *model.ClusterStatus) []model.ServerAddress

	RebalanceShards(configs *model.ClusterConfig, status *model.ClusterStatus) []SwapNodeAction
}

Algorithm

This proposal only focuses on the AZ awareness ensemble placement algorithm. We will reuse and be compatible with the existing ranking algorithm.

Therefore, the primary point here is that:

• Assign the shards replication to different servers according to the zone, namespace AZ rule, and sever ranking.
• If the eligible server's number is less than the replication. The coordinator will assign two replicas to the same server and set metrics and alerts.

Disaster recovery

We have several points to consider in disaster recovery. For example. If a zone server crashes or is deleted. The rule of shards assigning is as follows:

• eligibleServer= !highestRank && server.contains(terminatedServerShard)
• Assign the terminated server's shards to the same zone(avoiding the cross-zone traffic cost) based on the ranking.
• If the same zone doesn't have an eligible server, move to regional assigning.

Others

none.

[lsytj0413]

Perhaps rename ServerAddress to another name？it contains AvailableZone and sound didn't like only address semantic.

[mattisonchao]

might be, but I would like to use another proposal to change it. :)