ansible scylla node: Deploying a Scylla cluster

Deploying a Scylla cluster from scratch

This guide will follow the steps required to deploy a Scylla cluster using the ansible-scylla-node role

Requirements

An inventory file in a supported format (ini/yaml)
Ansible 2.8 or higher
SSH access to all the Scylla nodes configured

Variables

The defaults/main.yml file of the role is highly commented. It can be found here: https://github.com/scylladb/scylla-ansible-roles/blob/master/ansible-scylla-node/defaults/main.yml

Since different operating systems require different dependencies, keys and configuration, the role is designed to be flexible, but might require some careful data gathering before it is used.

To keep things easy, it is recommended to keep the defaults and set only the necessary variables. Open https://scylladb.com and move to the downloads page. There, select your preferred version and platform:

Next, scroll down to get the additional information.

For CentOS all you need is the repo URL:

For Debian you will need additional parameters:

At this point we should have the following parameters:

RHEL/CentOS:

scylla_repos:
  - http://repositories.scylladb.com/scylla/repo/UUID/centos/scylladb-4.1.repo

Debian/Ubuntu:

scylla_dependencies:
  - software-properties-common
  - apt-transport-https
  - gnupg2
  - dirmngr

scylla_repo_keyfile_urls:
  - https://download.opensuse.org/repositories/home:/scylladb:/scylla-3rdparty-stretch/Debian_10.0/Release.key

scylla_repos:
  - http://repositories.scylladb.com/scylla/repo/UUID/debian/scylladb-4.1-buster.list

scylla_repo_keyserver: 'hkp://keyserver.ubuntu.com:80'

scylla_repo_keys:
  - 5e08fbd8b5d6ec9c

NOTE: Since the requirements for different OS, Scylla distribution and architecture might differ, it is a generally good idea to go through the Scylla downloads page first and populate these variables carefully.

Scylla Manager integration

In case Scylla Manager will also be installed in order to configure the cluster, we need to populate some additional variables and update the ones we already have. Open the Manager download page and grab the information it presents:

For CentOS/RHEL all we need is the Manager repo:

scylla_manager_enabled: true
scylla_manager_repo_url: http://downloads.scylladb.com/rpm/centos/scylladb-manager-2.1.repo

For Debian/Ubuntu some additional variables need to be set:

scylla_manager_enabled: true
scylla_manager_repo_url: http://downloads.scylladb.com/deb/debian/scylladb-manager-2.1-stretch.list
scylla_manager_repo_keys:
  - 6B2BFD3660EF3F5B
  - 17723034C56D4B19

Also, if you see additional dependencies that were not yet mentioned in scylla_dependencies above, add those to the list.

Finally, if the Scylla Manager agents will be dealing with backups, you can set scylla_manager_agent_config like this:

scylla_manager_agent_config: |
  s3:
    access_key_id: 12345678
    secret_access_key: QWerty123456789
    provider: AWS
    region: us-east-1c
    endpoint: https://foo.bar.baz
    server_side_encryption:
    sse_kms_key_id:
    upload_concurrency: 2
    chunk_size: 50M
    use_accelerate_endpoint: false

For full syntax and options, refer to the Scylla Manager documentation.

Scylla Monitoring Integration

If the setup will also have Scylla-Monitoring attached, the role can generate and produce a ready to use scylla_servers.yml - the configuration file that will be used later on by the ansible_scylla_monitoring role in order to configure the monitoring stack to monitor the deployed Scylla cluster.

Install type

Scylla can be installed online (connect to repos and install from them) or offline to disable any download attempts (some packages, like node_exporter will need to be installed manually)

RAID configuration

RAID configuration is flexible with this role. You have several options:

Set the scylla_raid_setup variable:
```
scylla_raid_setup:
  - /dev/nvme0n1
  - /dev/nvme0n2
  - /dev/nvme0n3
```
This setting will attempt to take the 3 disks in the given list and build a RAID-0 array named md0 on top of them, them mount it under /var/lib/scylla. If md0 already exists and contains all 3 disks, the task will skip. If the disks are already formatted or md0 isn't available, the task will fail.
Use another role or automation to configure RAID. Make sure scylla_raid_setup is not set.
Skip RAID configuration altogether (not recommended unless simply testing Scylla outside production). Make sure scylla_raid_setup is not set.

ELRepo kernel - EL7 distributions only

Since CentOS comes with a rather old kernel, the role has the option to install the current latest kernel available for it on ELRepo. To install an ELRepo kernel set

elrepo_kernel: True

Bare installation (skip configuration)

It is possible to install the Scylla binaries but leave the database unconfigured (useful for some specific use cases). To do this set:

install_only: false

Scylla configuration

The following variables are used to configure the Scylla cluster and nodes.

Set the Cluster name (required):

scylla_cluster_name: MyCluster

Set the Scylla NIC (required)

Leave this at the default, unless you want all the nodes to have Scylla listening on a specified NIC, i.e. eth0

# default:
scylla_nic: "{{ ansible_default_ipv4.interface }}"

Scylla version and edition (required)

The version should be either a specific number, e.g. 4.1.2 (for OSS) or 2020.1.2 (for Enterprise) or simply use latest
The edition can either be enterprise or oss

scylla_version: latest
scylla_edition: oss

Scylla snitch (required)

This variable determines the snitch the cluster will be configured to use. The available snitches and their descriptions are available in the Scylla Administration guide.

NOTE: Some of the snitches require additional per-node settings in the inventory file

scylla_snitch: GossipingPropertyFileSnitch

# if GossipingPropertyFileSnitch is used, also set gpfs_prefer_local
gpfs_prefer_local: "true"

IO configuration (required)

Since it is the absolute best practice to run Scylla on homogeneous clusters, it is assumed all the nodes will have the same disk type and count, you have two options:

Allow the role to measure the node IO throughput on a single node and propagate the gathered benchmark data as a setting to all the nodes (recommended):

scylla_io_probe: True

Set the IO figures manually:

scylla_io_probe: False
io_properties:
  disks:
    - mountpoint: /var/lib/scylla/data
      read_iops: 10962999
      read_bandwidth: 348093056
      write_iops: 9722999
      write_bandwidth: 352182944

Scylla seeds list (required)

This is an important setting that will populate the cluster's seed list. If running on a multi-DC cluster, it is a good idea to have a few seeds in each cluster. The default setting simply sets the first 3 nodes in the inventory as seeds, which is usually sub-optimal, so this variable is important to set:

scylla_seeds:
  - 10.1.0.1
  - 10.1.0.2
  - 10.1.0.3
  - 10.2.0.1
  - 10.2.0.2
  - 10.2.0.3

Additional address settings (best to leave at default):

These can be altered and reset, but at the default, as shown below, they will use the IPv4 address of the previously set scylla_nic

scylla_listen_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_rpc_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_broadcast_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"

Scylla API configuration (best to leave at default):

scylla_api_address: '127.0.0.1'
scylla_api_port: '10000'

SSL configuration for Scylla nodes:

If no certificates are present, the role will generate a self-signed CA and node certificates locally and push them out to all the nodes. WARNING: if a local ./ssl directory is not present and populated by existing certificates, it will be generated and certificates on the nodes will be overwritten with new certificates. To use existing certificates:

To use your own CA, place it in ./ssl/ca/SCYLLA_CLUSTER_NAME-ca.pem and derived per-node certificates will be generated from it, if they are not present.
To use your own node certificates, create a directory per hostname (same as in the inventory) in ./ssl/HOSTNAME and place the .pem file as HOSTNAME.pem and the .crt file as HOSTNAME.crt

The playbook will use the CA certificate to generate a signed host certificate for the .pem file and push them into the nodes. To skip this configuration entirely, set enabled: false for both options or comment out the scylla_ssl variable.

scylla_ssl:
  cert_path: /etc/scylla/ # path on the node, where the certificates will be stored
  internode:
    enabled: true
    internode_encryption: all
  client:
    enabled: false

All other scylla.yaml parameters:

Set them under the catchall scylla_yaml_params variable as follows:

scylla_yaml_params:
  authorizer: CassandraAuthorizer
  authenticator: PasswordAuthenticator
  enable_sstables_mc_format: true
  internode_compression: all

NOTE: In order to avoid outages, changes to scylla.yaml will NOT trigger a service restart. Please use the `rolling_restart.yml` playbook if cluster-wide configuration changes have occurred.

SWAP on Scylla nodes

Scylla does not require swap to be enabled, but it is highly recommended to have some swap allocated. The role does not configure swap, just like it doesn't require for RAID settings to be in place. It is recommended to use another role to configure swap, or provision the target nodes with a swap partition in place.

Resulting parameters file:

Here's an example settings file for a cluster:

Cluster name "MyCluster"
Enterprise 2020.1 latest
Internode SSL enabled, Client SSL disabled
etc...

# parameters.yml

# swap configuration for the swap role
swap_file_size_mb: 2048

#RAID
scylla_raid_setup: 
  - /dev/nvme0n1
  - /dev/nvme0n2
  - /dev/nvme0n3

# Scylla Manager agent
scylla_manager_agent_config: |
  gcs:
    service_account_file: /etc/scylla/gcs.json
scylla_manager_repo_url: http://downloads.scylladb.com/rpm/centos/scylladb-manager-2.1.repo
scylla_manager_enabled: true

# monitoring configuration
generate_monitoring_config: true

# Scylla configuration
scylla_api_port: 10000
scylla_api_address: 127.0.0.1
scylla_broadcast_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_rpc_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_listen_address: "{{ vars['ansible_'~scylla_nic].ipv4.address }}"
scylla_seeds:
  - 10.142.0.3
  - 10.142.0.4
  - 10.128.0.3
  - 10.128.0.4
enable_mc_format: true
scylla_io_probe: true
scylla_snitch: GossipingPropertyFileSnitch
scylla_edition: enterprise
scylla_version: latest
scylla_dependencies:
  - curl
  - wget

elrepo_kernel: true

scylla_repos:
  - http://downloads.scylladb.com.s3.amazonaws.com/rpm/centos/scylla-2020.1.repo

install_type: online
scylla_nic: "{{ ansible_default_ipv4.interface }}"
install_only: false
scylla_cluster_name: MyCluster
scylla_ssl:
  cert_path: /etc/scylla/
  internode:
    enabled: true
    internode_encryption: all
  client:
    enabled: false

scylla_yaml_params:
  authorizer: CassandraAuthorizer
  authenticator: PasswordAuthenticator
  enable_sstables_mc_format: true
  internode_compression: all

Inventory

Some settings are relevant on a per-node basis, and cannot be worked around using ansible's magic variables.

For GossipingPropertyFileSnitch each node has to be configured with a specific dc and rack.
For Cloud-aware snitches (EC2Snitch, Ec2MultiRegionSnitch, GoogleCloudSnitch) dc_suffix needs to be set.

Here is our example inventory, using the GPFS snitch across two DCs, note how each DC has 2 seeds in the scylla_seeds variable above.

# inventory.ini
[scylla]
10.142.0.3 dc=us-east1 rack=rack-b
10.142.0.4 dc=us-east1 rack=rack-b
10.142.0.5 dc=us-east1 rack=rack-b
10.128.0.2 dc=us-central1 rack=rack-a
10.128.0.3 dc=us-central1 rack=rack-a
10.128.0.4 dc=us-central1 rack=rack-a

Playbook and execution

Here is an example playbook using the role:

# scylla_nodes.yml
- name: Scylla node
  hosts: scylla
  any_errors_fatal: true
  tasks:
    # Workaround for some Debian versions that might not have XFS support out of the box
    - name: install XFSprogs
      package:
        name: xfsprogs
        state: present
      become: true
  roles:
    # This is a nice working role for setting up swap
    - { name: geerlingguy.swap, become: true } 
    - ansible-scylla-node

And the playbook execution:

ansible-playbook -i inventory.ini -e '@parameters.yml' scylla_nodes.yml

Outputs

When the playbook is done running, depending on the settings provided, you will end up with some produced files:

scyllamgr_auth_token.txt: this is the token configured in all the scylla-manager-agent configuration files across the cluster. Use it with the ansible-scylla-manager role to connect the Manager to the Scylla cluster.
scylla_servers.yml: this file will be useful for setting up Scylla Monitoring, use it with the ansible-scylla-monitoring role in order to configure monitoring for the cluster.
ssl/ca and ssl/HOST directories and their contents: These are the certificates generated and deployed in case SSL was enabled with the ansible-scylla-node role. Please store these safely

Additional variables and tuning

sysctl

The role has the facility to adjust sysctl settings, that might be required by the Scylla team or by your cloud platform support for optimal performance. The defaults/main.yml settings contain a set of recommended sysctl settings under the sysctl variable as well as a flag determining whether or not sysctl changes will be applied at all called update_sysctl which has to be set to True for changes to take place.

ELRepo kernel

When using the ELRepo kernel, it will get installed but not activated until the next reboot. In order to switch to the new kernel, please make sure you restart the nodes after applying the role. If done in an active cluster, perform a rolling restart either manually or using the rolling restart playbook.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ansible scylla node: Deploying a Scylla cluster

Deploying a Scylla cluster from scratch

Requirements

Variables

Scylla Manager integration

Scylla Monitoring Integration

Install type

RAID configuration

ELRepo kernel - EL7 distributions only

Bare installation (skip configuration)

Scylla configuration

Set the Cluster name (required):

Set the Scylla NIC (required)

Scylla version and edition (required)

Scylla snitch (required)

NOTE: Some of the snitches require additional per-node settings in the inventory file

IO configuration (required)

Scylla seeds list (required)

Additional address settings (best to leave at default):

Scylla API configuration (best to leave at default):

SSL configuration for Scylla nodes:

All other scylla.yaml parameters:

NOTE: In order to avoid outages, changes to scylla.yaml will NOT trigger a service restart. Please use the `rolling_restart.yml` playbook if cluster-wide configuration changes have occurred.

SWAP on Scylla nodes

Resulting parameters file:

Inventory

Playbook and execution

Outputs

Additional variables and tuning

sysctl

ELRepo kernel

Clone this wiki locally

ansible scylla node: Deploying a Scylla cluster

Deploying a Scylla cluster from scratch

Requirements

Variables

Scylla Manager integration

Scylla Monitoring Integration

Install type

RAID configuration

ELRepo kernel - EL7 distributions only

Bare installation (skip configuration)

Scylla configuration

Set the Cluster name (required):

Set the Scylla NIC (required)

Scylla version and edition (required)

Scylla snitch (required)

NOTE: Some of the snitches require additional per-node settings in the inventory file

IO configuration (required)

Scylla seeds list (required)

Additional address settings (best to leave at default):

Scylla API configuration (best to leave at default):

SSL configuration for Scylla nodes:

All other scylla.yaml parameters:

NOTE: In order to avoid outages, changes to scylla.yaml will NOT trigger a service restart. Please use the rolling_restart.yml playbook if cluster-wide configuration changes have occurred.

SWAP on Scylla nodes

Resulting parameters file:

Inventory

Playbook and execution

Outputs

Additional variables and tuning

sysctl

ELRepo kernel

Clone this wiki locally

NOTE: In order to avoid outages, changes to scylla.yaml will NOT trigger a service restart. Please use the `rolling_restart.yml` playbook if cluster-wide configuration changes have occurred.