diff --git a/docs/source/index.rst b/docs/source/index.rst index e52285b85c2..582c201f927 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -15,6 +15,7 @@ Scylla Operator Documentation migration nodeoperations/index exposing + multidc/index performance upgrade releases @@ -55,6 +56,7 @@ Currently it supports: * :doc:`Setting up Monitoring using Prometheus and Grafana ` * :doc:`Node operations ` * :doc:`Exposing ScyllaCluster to other networks ` +* :doc:`Deploying multi-datacenter ScyllaDB clusters in Kubernetes ` * :doc:`Performance tuning [Experimental] ` * :doc:`Upgrade procedures ` * :doc:`Releases ` diff --git a/docs/source/multidc/eks.md b/docs/source/multidc/eks.md new file mode 100644 index 00000000000..bdba26668f6 --- /dev/null +++ b/docs/source/multidc/eks.md @@ -0,0 +1,168 @@ +# Build multiple Amazon EKS clusters with inter-Kubernetes networking + +This document describes the process of creating multiple Amazon EKS clusters in different regions, using separate VPCs, and explains the steps necessary for configuring inter-Kubernetes networking between the clusters. +The interconnected clusters can serve as a platform for [deploying a multi-datacenter ScyllaDB cluster](multidc.md). + +This guide will walk you through the process of creating and configuring EKS clusters in two distinct regions. Although it is only an example setup, it can easily be built upon to create infrastructure tailored to your specific needs. +For simplicity, several predefined values are used throughout the document. The values are only exemplary and can be adjusted to your preference. + +## Prerequisites + +To follow the below guide, you first need to install and configure the tools that you will need to create and manage AWS and Kubernetes resources: +- eksctl – A command line tool for working with EKS clusters. +- kubectl – A command line tool for working with Kubernetes clusters. + +For more information see [Getting started with Amazon EKS – eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) in AWS documentation. + +## Create EKS clusters + +### Create the first EKS cluster + +Below is the required specification for the first cluster. + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: scylladb-us-east-1 + region: us-east-1 + +availabilityZones: +- us-east-1a +- us-east-1b +- us-east-1c + +vpc: + cidr: 10.0.0.0/16 + +nodeGroups: + ... +``` + +Specify the first cluster's configuration file and save it as `cluster-us-east-1.yaml`. +Refer to [Creating an EKS cluster](../../eks#creating-an-eks-cluster) section of ScyllaDB Operator documentation for the reference of the configuration of node groups. + +To deploy the first cluster, use the below command: +```shell +eksctl create cluster -f=cluster-us-east-1.yaml +``` + +Run the following command to learn the status and VPC ID of the cluster: +```shell +eksctl get cluster --name=scylladb-us-east-1 --region=us-east-1 +``` + +You will need to get the cluster's context for future operations. To do so, use the below command: +```shell +kubectl config current-context +``` + +For any `kubectl` commands that you will want to run against this cluster, use the `--context` flag with the value returned by the above command. + +#### Deploy ScyllaDB Operator + +Once the cluster is ready, refer to [Deploying Scylla on a Kubernetes Cluster](../generic.md) to deploy the ScyllaDB Operator and its prerequisites. + +#### Prepare nodes for running ScyllaDB + +Then, prepare the nodes for running ScyllaDB workloads and deploy a volume provisioner following the steps described in [Deploying Scylla on EKS](../../eks#prerequisites) in ScyllaDB Operator documentation. + +### Create the second EKS cluster + +Below is the required specification for the second cluster. As was the case with the first cluster, the provided values are only exemplary and can be adjusted according to your needs. + +``` caution:: + It is required that the VPCs of the two EKS clusters have non-overlapping IPv4 network ranges. +``` + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: scylladb-us-east-2 + region: us-east-2 + +availabilityZones: +- us-east-2a +- us-east-2b +- us-east-2c + +vpc: + cidr: 172.16.0.0/16 + +nodeGroups: + ... +``` + +Follow analogous steps to create the second EKS cluster and prepare it for running ScyllaDB. + +## Configure the network + +The prepared Kubernetes clusters each have a dedicated VPC network. +To be able to route the traffic between the two VPC networks, you need to create a networking connection between them, otherwise known as [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html). + +### Create VPC peering + +Refer to [Create a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html#create-vpc-peering-connection-local) in AWS documentation for instructions on creating a VPC peering connection between the two earlier created VPCs. + +In this example, the ID of the created VPC peering connection is `pcx-08077dcc008fbbab6`. + +### Update route tables + +To enable private IPv4 traffic between the instances in the VPC peered network, you need to establish a communication channel by adding a route to the route tables associated with all the subnets associated with the instances for both VPCs. +The destination of the new route in a given route table is the CIDR of the VPC of the other cluster and the target is the ID of the VPC peering connection. + +The following is an example of the route tables that enable communication of instances in two peered VPCs. Each table has a local route and the added route which sends traffic targeted at the other VPC to the peered network connection. The other preconfigured routes are omitted for readability. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Route tableDestinationTarget
eksctl-scylladb-us-east-1-cluster/PublicRouteTable10.0.0.0/16local
172.16.0.0/16pcx-08077dcc008fbbab6
eksctl-scylladb-us-east-2-cluster/PublicRouteTable172.16.0.0/16local
10.0.0.0/16pcx-08077dcc008fbbab6
+ + +Refer to [Update your route tables for a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-routing.html) in AWS documentation for more information. + +### Update security groups + +To allow traffic to flow to and from instances associated with security groups in the peered VPC, you need to update the inbound rules of the VPCs' shared security groups. + +Below is an example of the inbound rules that to be added to the corresponding security groups of the two VPCs. + +| Security group name | Type | Protocol | Port range | Source | +|--------------------------------------------------------------------------------|-------------|----------|------------|----------------------| +| eksctl-scylladb-us-east-1-cluster-ClusterSharedNodeSecurityGroup-TD05V9EVU3B8 | All traffic | All | All | Custom 172.16.0.0/16 | +| eksctl-scylladb-us-east-2-cluster-ClusterSharedNodeSecurityGroup-1FR9YDLU0VE7M | All traffic | All | All | Custom 10.0.0.0/16 | + +The names of the shared security groups of your VPCs should be similar to the ones presented in the example. + +--- + +Having followed the above steps, you should now have a platform prepared for deploying a multi-datacenter ScyllaDB cluster. +Refer to [Deploy a multi-datacenter ScyllaDB cluster in multiple interconnected Kubernetes clusters](multidc.md) in ScyllaDB Operator documentation for guidance. diff --git a/docs/source/multidc/gke.md b/docs/source/multidc/gke.md new file mode 100644 index 00000000000..9c353ed0d9c --- /dev/null +++ b/docs/source/multidc/gke.md @@ -0,0 +1,156 @@ +# Build multiple GKE clusters with inter-Kubernetes networking + +This document describes the process of creating multiple GKE clusters in a shared VPC and explains the steps necessary for configuring inter-Kubernetes networking between clusters in different regions. +The interconnected clusters can serve as a platform for [deploying a Multi Datacenter ScyllaDB cluster](multidc.md). + +This guide will walk you through the process of creating and configuring GKE clusters in two distinct regions. Although it is only an example setup, it can easily be built upon to create infrastructure tailored to your specific needs. +For simplicity, several predefined values are used throughout the document. The values are only exemplary and can be adjusted to your preference. + +## Prerequisites + +To follow the below guide, you first need to install and configure the following tools that you will need to create and manage GCP and Kubernetes resources: +- gcloud CLI - Google Cloud Command Line Interface, a command line tool for working with Google Cloud resources and services directly. +- kubectl – A command line tool for working with Kubernetes clusters. + +See [Install the Google Cloud CLI](https://cloud.google.com/sdk/docs/install-sdk) in GCP documentation and [Install Tools](https://kubernetes.io/docs/tasks/tools/) in Kubernetes documentation for reference. + +## Create and configure a VPC network + +For the clusters to have inter-Kubernetes networking, you will create a virtual network shared between all the instances, with dedicated subnets for each of the clusters. +To create the subnets manually, create the network in custom subnet mode. + +### Create the VPC network + +Run the below command to create the network: +```shell +gcloud compute networks create scylladb --subnet-mode=custom +``` + +With the VPC network created, create a dedicated subnet with secondary CIDR ranges for their Pod and Service pools in each region which the clusters will reside in. + +### Create VPC network subnets + +To create a subnet for the first cluster in region `us-east1`, run the below command: +```shell +gcloud compute networks subnets create scylladb-us-east1 \ + --region=us-east1 \ + --network=scylladb \ + --range=10.0.0.0/20 \ + --secondary-range='cluster=10.1.0.0/16,services=10.2.0.0/20' +``` + +To create a subnet for the second cluster in region `us-west1`, run the below command: +```shell +gcloud compute networks subnets create scylladb-us-west1 \ + --region=us-west1 \ + --network=scylladb \ + --range=172.16.0.0/20 \ + --secondary-range='cluster=172.17.0.0/16,services=172.18.0.0/20' +``` + +``` caution:: + It is required that the IPv4 address ranges of the subnets allocated for the GKE clusters do not overlap. +``` + +Refer to [Create a VPC-native cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips) and [Alias IP ranges](https://cloud.google.com/vpc/docs/alias-ip) in GKE documentation for more information about VPC native clusters and alias IP ranges. + +## Create GKE clusters + +With the VPC network created, you will now create two VPC native GKE clusters in dedicated regions. + +### Create the first GKE cluster + +Run the following command to create the first GKE cluster in the `us-east1` region: +```shell +gcloud container clusters create scylladb-us-east1 \ + --location=us-east1-b \ + --node-locations='us-east1-b,us-east1-c' \ + --machine-type=n1-standard-8 \ + --num-nodes=1 \ + --disk-type=pd-ssd \ + --disk-size=20 \ + --image-type=UBUNTU_CONTAINERD \ + --no-enable-autoupgrade \ + --no-enable-autorepair \ + --enable-ip-alias \ + --network=scylladb \ + --subnetwork=scylladb-us-east1 \ + --cluster-secondary-range-name=cluster \ + --services-secondary-range-name=services +``` + +Refer to [Creating a GKE cluster](../../gke#creating-a-gke-cluster) section of ScyllaDB Operator documentation for more information regarding the configuration and deployment of additional node pools, including the one dedicated for ScyllaDB nodes. + +You will need to get the cluster's context for future operations. To do so, use the below command: +```shell +kubectl config current-context +``` + +For any `kubectl` commands that you will want to run against this cluster, use the `--context` flag with the value returned by the above command. + +#### Deploy ScyllaDB Operator + +Once the cluster is ready, refer to [Deploying Scylla on a Kubernetes Cluster](../generic.md) to deploy the ScyllaDB Operator and its prerequisites. + +#### Prepare nodes for running ScyllaDB + +Then, prepare the nodes for running ScyllaDB workloads and deploy a volume provisioner following the steps described in [Deploying Scylla on GKE](../gke.md) page of the documentation. + +### Create the second GKE cluster + +Run the following command to create the second GKE cluster in the `us-west1` region: +```shell +gcloud container clusters create scylladb-us-west1 \ + --location=us-west1-b \ + --node-locations='us-west1-b,us-west1-c' \ + --machine-type=n1-standard-8 \ + --num-nodes=1 \ + --disk-type=pd-ssd \ + --disk-size=20 \ + --image-type=UBUNTU_CONTAINERD \ + --no-enable-autoupgrade \ + --no-enable-autorepair \ + --enable-ip-alias \ + --network=scylladb \ + --subnetwork=scylladb-us-west1 \ + --cluster-secondary-range-name=cluster \ + --services-secondary-range-name=services +``` + +Follow analogous steps to create the second GKE cluster and prepare it for running ScyllaDB. + +## Configure the firewall rules + +When creating a cluster, GKE creates several ingress firewall rules that enable the instances to communicate with each other. +To establish interconnectivity between the two created Kubernetes clusters, you will now add the allocated IPv4 address ranges to their corresponding source address ranges. + +First, retrieve the name of the firewall rule associated with the first cluster, which permits traffic between all Pods on a cluster, as required by the Kubernetes networking model. +The rule name is in the following format: `gke-[cluster-name]-[cluster-hash]-all`. + +To retrieve it, run the below command: +```shell +gcloud compute firewall-rules list --filter='name~gke-scylladb-us-east1-.*-all' +``` + +The output should resemble the following: +```console +NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED +gke-scylladb-us-east1-f17db261-all scylladb INGRESS 1000 udp,icmp,esp,ah,sctp,tcp False +``` + +Modify the rule by updating the rule's source ranges with the allocated Pod IPv4 address ranges of both clusters: +```shell +gcloud compute firewall-rules update gke-scylladb-us-east1-f17db261-all --source-ranges='10.1.0.0/16,172.17.0.0/16' +``` + +Follow the analogous steps for the other cluster. In this example, its corresponding firewall rule name is `gke-scylladb-us-west1-0bb60902-all`. To update it, you would run: +```shell +gcloud compute firewall-rules update gke-scylladb-us-west1-0bb60902-all --source-ranges='10.1.0.0/16,172.17.0.0/16' +``` + +Refer to [Automatically created firewall rules](https://cloud.google.com/kubernetes-engine/docs/concepts/firewall-rules) in GKE documentation for more information. + +--- + +Having followed the above steps, you should now have a platform prepared for deploying a multi-datacenter ScyllaDB cluster. +Refer to [Deploy a multi-datacenter ScyllaDB cluster in multiple interconnected Kubernetes clusters](multidc.md) in ScyllaDB Operator documentation for guidance. diff --git a/docs/source/multidc/index.rst b/docs/source/multidc/index.rst new file mode 100644 index 00000000000..52a87126e8a --- /dev/null +++ b/docs/source/multidc/index.rst @@ -0,0 +1,25 @@ +========================================================== +Deploying multi-datacenter ScyllaDB clusters in Kubernetes +========================================================== + +Prepare a platform for a multi datacenter ScyllaDB cluster deployment: + +.. toctree:: + :hidden: + :maxdepth: 2 + + eks + gke + +* :doc:`Build multiple Amazon EKS clusters with Inter-Kubernetes networking ` +* :doc:`Build multiple GKE clusters with Inter-Kubernetes networking ` + +Deploy a multi-datacenter ScyllaDB cluster in Kubernetes: + +.. toctree:: + :hidden: + :maxdepth: 2 + + multidc + +* :doc:`Deploy a multi-datacenter ScyllaDB cluster in multiple interconnected Kubernetes clusters ` diff --git a/docs/source/multidc/multidc.md b/docs/source/multidc/multidc.md new file mode 100644 index 00000000000..018085c9141 --- /dev/null +++ b/docs/source/multidc/multidc.md @@ -0,0 +1,563 @@ +# Deploy a multi-datacenter ScyllaDB cluster in multiple interconnected Kubernetes clusters + +This document describes the process of deploying a Multi Datacenter ScyllaDB cluster in multiple interconnected Kubernetes clusters. + +This guide will walk you through the example procedure of deploying two datacenters in distinct regions of a selected cloud provider. + +``` note:: + This guide is dedicated to deploying multi-datacenter ScyllaDB clusters and does not discuss unrelated configuration options. + For details of ScyllaDB cluster deployments and their configuration, refer to `Deploying Scylla on a Kubernetes Cluster <../generic.md>`_ in ScyllaDB Operator documentation. +``` + +## Prerequisites + +As this document describes the procedure of deploying a Multi Datacenter ScyllaDB cluster, you are expected to have the required infrastructure prepared. +Let's assume two interconnected Kubernetes clusters, capable of communicating with each other over PodIPs, with each cluster meeting the following requirements: +- a node pool dedicated to ScyllaDB nodes composed of at least 3 nodes running in different zones (with unique `topology.kubernetes.io/zone` label), configured to run ScyllaDB, each labeled with `scylla.scylladb.com/node-type: scylla` +- running ScyllaDB Operator and its prerequisites +- running a storage provisioner capable of provisioning XFS volumes of StorageClass `scylladb-local-xfs` in each of the nodes dedicated to ScyllaDB instances + +You can refer to one of our guides describing the process of preparing such infrastructure: +- [Build multiple Amazon EKS clusters with Inter-Kubernetes networking](eks.md) +- [Build multiple GKE clusters with Inter-Kubernetes networking](gke.md) + +Additionally, to follow the below guide, you need to install and configure the following tools that you will need to manage Kubernetes resources: +- kubectl – A command line tool for working with Kubernetes clusters. + +See [Install Tools](https://kubernetes.io/docs/tasks/tools/) in Kubernetes documentation for reference. + +## Multi Datacenter ScyllaDB Cluster + +In v1.11, ScyllaDB Operator introduced support for manual multi-datacenter ScyllaDB cluster deployments. + +``` warning:: + ScyllaDB Operator only supports *manual configuration* of multi-datacenter ScyllaDB clusters. + In other words, although ScyllaCluster API exposes the machinery necessary for setting up multi-datacenter ScylaDB clusters, the ScyllaDB Operator only automates operations for a single datacenter. + + Operations related to multiple datacenters may require manual intervention of a human operator. + Most notably, destroying one of the Kubernetes clusters or ScyllaDB datacenters is going to leave DN nodes behind in other datacenters, and their removal has to be carried out manually. +``` + +The main mechanism used to set up a manual multi-datacenter ScyllaDB cluster is a field in ScyllaCluster's specification - `externalSeeds`. + +### External seeds + +The `externalSeeds` field in ScyllaCluster's specification enables control over external seeds that are propagated to ScyllaDB binary as `--seed-provider-parameters seeds=`. +In this context, external should be understood as "external to the datacenter being specified by the API". +The provided seeds are used by the nodes as initial points of contact, which allows them to discover the cluster ring topology when joining it. + +Refer to [Scylla Seed Nodes](https://opensource.docs.scylladb.com/stable/kb/seed-nodes.html) in ScyllaDB documentation for more information regarding the function of seed nodes in ScyllaDB. +For more details regarding the function and implementation of external seeds, refer to [the original enhancement proposal](https://github.com/scylladb/scylla-operator/tree/v1.11/enhancements/proposals/1304-external-seeds). + +### Networking + +Since this guide assumes interconnectivity over PodIPs of the Kubernetes clusters, you are going to configure the ScyllaDB cluster's nodes to communicate over PodIPs. +This is enabled by a subset of `exposeOptions` specified in ScyllaCluster API, introduced in v1.11. + +For this particular setup, define the ScyllaClusers as follows: +```yaml +apiVersion: scylla.scylladb.com/v1 +kind: ScyllaCluster +spec: + exposeOptions: + nodeService: + type: Headless + broadcastOptions: + clients: + type: PodIP + nodes: + type: PodIP +``` + +However, other configuration options allow for the manual deployment of multi-datacenter ScyllaDB clusters in different network setups. For details, refer to [Exposing ScyllaClusters](../exposing.md) in ScyllaDB Operator documentation. + +#### Deploy a multi-datacenter ScyllaDB Cluster + +#### Using context + +Let's specify contexts for `kubectl` commands used throughout the guide. +To retrieve the context of your current cluster, run: +```shell +kubectl config current-context +``` + +Save the contexts of the two clusters, which you are going to deploy the datacenters in, as `CONTEXT_DC1` and `CONTEXT_DC2` environment variables correspondingly. + +#### Deploy the first datacenter + +First, run the below command to create a dedicated 'scylla' namespace: +```shell +kubectl --context="${CONTEXT_DC1}" create ns scylla +``` + +For this guide, let's assume that your cluster is running in `us-east-1` region and the nodes dedicated to running ScyllaDB nodes are running in zones `us-east-1a`, `us-east-1b` and `us-east-1c` correspondingly. If that is not the case, adjust the manifest accordingly. + +``` caution:: + The ``.spec.name`` field of the ScyllaCluster objects represents the ScyllaDB cluster name and has to be consistent across all datacenters of this ScyllaDB cluster. + The names of the datacenters, specified in ``.spec.datacenter.name``, have to be unique across the entire multi-datacenter cluster. + + For more information see `Create a ScyllaDB Cluster - Multi Data Centers (DC) `_ in ScyllaDB documentation. +``` + +Save the ScyllaCluster manifest in `dc1.yaml`: +```yaml +apiVersion: scylla.scylladb.com/v1 +kind: ScyllaCluster +metadata: + name: scylla-cluster + namespace: scylla +spec: + version: 5.2.7 + agentVersion: 3.1.2 + cpuset: true + sysctls: + - "fs.aio-max-nr=2097152" + automaticOrphanedNodeCleanup: true + exposeOptions: + broadcastOptions: + clients: + type: PodIP + nodes: + type: PodIP + nodeService: + type: Headless + datacenter: + name: us-east-1 + racks: + - name: a + members: 1 + storage: + storageClassName: scylladb-local-xfs + capacity: 1800G + agentResources: + requests: + cpu: 100m + memory: 250M + limits: + cpu: 100m + memory: 250M + resources: + requests: + cpu: 7 + memory: 56G + limits: + cpu: 7 + memory: 56G + placement: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: scylla + scylla/cluster: scylla-cluster + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: topology.kubernetes.io/zone + operator: In + values: + - us-east-1a + - key: scylla.scylladb.com/node-type + operator: In + values: + - scylla + tolerations: + - key: role + operator: Equal + value: scylla-clusters + effect: NoSchedule + - name: b + members: 1 + storage: + storageClassName: scylladb-local-xfs + capacity: 1800G + agentResources: + requests: + cpu: 100m + memory: 250M + limits: + cpu: 100m + memory: 250M + resources: + requests: + cpu: 7 + memory: 56G + limits: + cpu: 7 + memory: 56G + placement: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: scylla + scylla/cluster: scylla-cluster + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: topology.kubernetes.io/zone + operator: In + values: + - us-east-1b + - key: scylla.scylladb.com/node-type + operator: In + values: + - scylla + tolerations: + - key: role + operator: Equal + value: scylla-clusters + effect: NoSchedule + - name: c + members: 1 + storage: + storageClassName: scylladb-local-xfs + capacity: 1800G + agentResources: + requests: + cpu: 100m + memory: 250M + limits: + cpu: 100m + memory: 250M + resources: + requests: + cpu: 7 + memory: 56G + limits: + cpu: 7 + memory: 56G + placement: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: scylla + scylla/cluster: scylla-cluster + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: topology.kubernetes.io/zone + operator: In + values: + - us-east-1c + - key: scylla.scylladb.com/node-type + operator: In + values: + - scylla + tolerations: + - key: role + operator: Equal + value: scylla-clusters + effect: NoSchedule +``` + +Apply the manifest: +```shell +kubectl --context="${CONTEXT_DC1}" apply --server-side -f=dc1.yaml +``` + +Wait for the cluster to be fully rolled out: +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla wait --for='condition=Progressing=False' scyllaclusters.scylla.scylladb.com/scylla-cluster +``` +```console +scyllacluster.scylla.scylladb.com/scylla-cluster condition met +``` + +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla wait --for='condition=Degraded=False' scyllaclusters.scylla.scylladb.com/scylla-cluster +``` +```console +scyllacluster.scylla.scylladb.com/scylla-cluster condition met +``` + +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla wait --for='condition=Available=True' scyllaclusters.scylla.scylladb.com/scylla-cluster +``` +```console +scyllacluster.scylla.scylladb.com/scylla-cluster condition met +``` + +You can now verify that all the nodes of your cluster are in UN state: +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla exec -it pod/scylla-cluster-us-east-1-a-0 -c=scylla -- nodetool status +``` + +The expected output should look similar to the below: +```console +Datacenter: us-east-1 +===================== +Status=Up/Down +|/ State=Normal/Leaving/Joining/Moving +-- Address Load Tokens Owns Host ID Rack +UN 10.0.70.195 290 KB 256 ? 494277b9-121c-4af9-bd63-3d0a7b9305f7 c +UN 10.0.59.24 559 KB 256 ? a3a98e08-0dfd-4a25-a96a-c5ab2f47eb37 b +UN 10.0.19.237 107 KB 256 ? 64b6292a-327f-4128-852a-6004039f402e a +``` + +##### Retrieve PodIPs of ScyllaDB nodes for use as external seeds + +``` warning:: + Due to the ephemeral nature of PodIPs, it is ill-advised to use them as seeds in production environments. + This is because there is a high likelihood that the Pods of your ScyllaDB clusters will change their IPs during the cluster's lifecycle, and so the provided seeds will no longer point to the ScyllaDB nodes. + It is undesired, as the seeds provided on node's startup may serve as fallback contact points when all of the node's peers are unreachable. + In production environments, it is recommended that you use domain names or non-ephemeral IP addresses as external seeds. + PodIPs are being used in this example for the sheer simplicity of this setup. +``` + +Use the below commands and their expected outputs as a reference for retrieving the PodIPs used by the cluster for inter-node communication. +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla get pod/scylla-cluster-us-east-1-a-0 --template='{{ .status.podIP }}' +``` +```console +10.0.19.237 +``` + +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla get pod/scylla-cluster-us-east-1-b-0 --template='{{ .status.podIP }}' +``` +```console +10.0.59.24 +``` + +```shell +kubectl --context="${CONTEXT_DC1}" -n=scylla get pod/scylla-cluster-us-east-1-c-0 --template='{{ .status.podIP }}' +``` +```console +10.0.70.195 +``` + +You are going to utilize the retrieved addresses as seeds for the other datacenter. + +#### Deploy the second datacenter + +To deploy the second datacenter, you will follow similar steps. + +First, create a dedicated 'scylla' namespace: +```shell +kubectl --context="${CONTEXT_DC2}" create ns scylla +``` + +Replace the values in `.spec.externalSeeds` of the below manifest with the Pod IP addresses that you retrieved earlier. +The provided values are going to serve as initial contact points for the joining nodes of the second datacenter. + +For this guide, let's assume that the second cluster is running in `us-east-2` region and the nodes dedicated for running ScyllaDB nodes are running in zones `us-east-2a`, `us-east-2b` and `us-east-2c` correspondingly. If that is not the case, adjust the manifest accordingly. +Having configured it, save the manifest as `dc2.yaml`: +```yaml +apiVersion: scylla.scylladb.com/v1 +kind: ScyllaCluster +metadata: + name: scylla-cluster + namespace: scylla +spec: + version: 5.2.7 + agentVersion: 3.1.2 + cpuset: true + sysctls: + - "fs.aio-max-nr=2097152" + automaticOrphanedNodeCleanup: true + exposeOptions: + broadcastOptions: + clients: + type: PodIP + nodes: + type: PodIP + nodeService: + type: Headless + externalSeeds: + - 10.0.19.237 + - 10.0.59.24 + - 10.0.70.195 + datacenter: + name: us-east-2 + racks: + - name: a + members: 1 + storage: + storageClassName: scylladb-local-xfs + capacity: 1800G + agentResources: + requests: + cpu: 100m + memory: 250M + limits: + cpu: 100m + memory: 250M + resources: + requests: + cpu: 7 + memory: 56G + limits: + cpu: 7 + memory: 56G + placement: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: scylla + scylla/cluster: scylla-cluster + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: topology.kubernetes.io/zone + operator: In + values: + - us-east-2a + - key: scylla.scylladb.com/node-type + operator: In + values: + - scylla + tolerations: + - key: role + operator: Equal + value: scylla-clusters + effect: NoSchedule + - name: b + members: 1 + storage: + storageClassName: scylladb-local-xfs + capacity: 1800G + agentResources: + requests: + cpu: 100m + memory: 250M + limits: + cpu: 100m + memory: 250M + resources: + requests: + cpu: 7 + memory: 56G + limits: + cpu: 7 + memory: 56G + placement: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: scylla + scylla/cluster: scylla-cluster + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: topology.kubernetes.io/zone + operator: In + values: + - us-east-2b + - key: scylla.scylladb.com/node-type + operator: In + values: + - scylla + tolerations: + - key: role + operator: Equal + value: scylla-clusters + effect: NoSchedule + - name: c + members: 1 + storage: + storageClassName: scylladb-local-xfs + capacity: 1800G + agentResources: + requests: + cpu: 100m + memory: 250M + limits: + cpu: 100m + memory: 250M + resources: + requests: + cpu: 7 + memory: 56G + limits: + cpu: 7 + memory: 56G + placement: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: scylla + scylla/cluster: scylla-cluster + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: topology.kubernetes.io/zone + operator: In + values: + - us-east-2c + - key: scylla.scylladb.com/node-type + operator: In + values: + - scylla + tolerations: + - key: role + operator: Equal + value: scylla-clusters + effect: NoSchedule +``` + +To apply the manifest, run: +```shell +kubectl --context="${CONTEXT_DC2}" -n=scylla apply --server-side -f=dc2.yaml +``` + +Wait for the second datacenter to roll out: +```shell +kubectl --context="${CONTEXT_DC2}" -n=scylla wait --for='condition=Progressing=False' scyllaclusters.scylla.scylladb.com/scylla-cluster +``` +```console +scyllacluster.scylla.scylladb.com/scylla-cluster condition met +``` + +```shell +kubectl --context="${CONTEXT_DC2}" -n=scylla wait --for='condition=Degraded=False' scyllaclusters.scylla.scylladb.com/scylla-cluster +``` +```console +scyllacluster.scylla.scylladb.com/scylla-cluster condition met +``` + +```shell +kubectl --context="${CONTEXT_DC2}" -n=scylla wait --for='condition=Available=True' scyllaclusters.scylla.scylladb.com/scylla-cluster +``` +```console +scyllacluster.scylla.scylladb.com/scylla-cluster condition met +``` + +You can verify that the nodes have joined the existing cluster and that you are now running a multi-datacenter ScyllaDB cluster by running `nodetool status` with the below command: +```shell +kubectl --context="${CONTEXT_DC2}" -n=scylla exec -it pod/scylla-cluster-us-east-2-a-0 -c=scylla -- nodetool status +``` +```console +Datacenter: us-east-1 +===================== +Status=Up/Down +|/ State=Normal/Leaving/Joining/Moving +-- Address Load Tokens Owns Host ID Rack +UN 10.0.70.195 705 KB 256 ? 494277b9-121c-4af9-bd63-3d0a7b9305f7 c +UN 10.0.59.24 764 KB 256 ? a3a98e08-0dfd-4a25-a96a-c5ab2f47eb37 b +UN 10.0.19.237 634 KB 256 ? 64b6292a-327f-4128-852a-6004039f402e a +Datacenter: us-east-2 +===================== +Status=Up/Down +|/ State=Normal/Leaving/Joining/Moving +-- Address Load Tokens Owns Host ID Rack +UN 172.16.39.209 336 KB 256 ? 7c30ea55-7a4f-4d93-86f7-c881772ebe62 b +UN 172.16.25.18 759 KB 256 ? 665dde7e-e420-4db3-8c54-ca71efd39b2e a +UN 172.16.87.27 503 KB 256 ? c19c89cb-e24c-4062-9df4-2aa90ab29a99 c +```