Support Astra as storage backend

src/docs/content/docs/backends/_index.md

@@ -7,20 +7,23 @@ weight = 10
 
 # Backends
 
-Cassandra Reaper can be used with either an ephemeral memory storage or persistent database. For persistent scalable database storage, a Cassandra cluster can be set up to back Reaper. To use a Cassandra cluster as the backed storage for Reaper set `storageType` to a value of **cassandra** in the Reaper configuration file. Alternatively, a relational database storage; either H2 or Postgres can be set up to back Reaper. To use one of the relational database options as the backed storage for Reaper set `storageType` to a value of either **h2** or **postrges** in the Reaper configuration file.
+Cassandra Reaper can be used with either an ephemeral memory storage or persistent database. For persistent scalable database storage, a Cassandra cluster can be set up to back Reaper. To use a Cassandra cluster as the backed storage for Reaper set `storageType` to a value of **cassandra** in the Reaper configuration file. **Astra** can also be used as storage backend by setting `storageType` to a value of **astra**. With purge settings tuned appropriately, Reaper's workload should fit perfectly into the free tier of Astra.
+Alternatively, a relational database storage; either H2 or Postgres can be set up to back Reaper. To use one of the relational database options as the backed storage for Reaper set `storageType` to a value of either **h2** or **postrges** in the Reaper configuration file.
 
 Further information on the available storage options is provided in the following section.
 
 * [In-Memory]({{<ref "memory.md" >}})
 * [Cassanda]({{<ref "cassandra.md">}})
 * [PostgresQL]({{<ref "postgres.md">}})
 * [H2]({{<ref "h2.md">}})
+* [Astra]({{<ref "astra.md">}})
 
 Sample YAML files are available in the *[src/packaging/resource](https://github.com/thelastpickle/cassandra-reaper/tree/master/src/packaging/resource)* directory for each of the above storage options:
 
 * cassandra-reaper-memory.yaml
 * cassandra-reaper-cassandra.yaml
 * cassandra-reaper-postgres.yaml
 * cassandra-reaper-h2.yaml
+* cassandra-reaper-astra.yaml
 
 For configuring other aspects of the service, see the available configuration options in the [Configuration Reference](../configuration).

src/docs/content/docs/backends/astra.md

@@ -0,0 +1,53 @@
++++
+[menu.docs]
+name = "Astra"
+parent = "backends"
+weight = 5
++++
+
+# Astra Backend
+
+To use Astra (Cassandra managed service by Datastax) as the persistent storage for Reaper, the `storageType` setting must be set to **astra** in the Reaper configuration YAML file. In addition, the connection details for the Astra cluster being used to store Reaper data must be specified in the configuration YAML file. An example of how to configure Astra as persistent storage for Reaper can be found in the *[cassandra-reaper-astra.yaml](https://github.com/thelastpickle/cassandra-reaper/blob/master/src/packaging/resource/cassandra-reaper-astra.yaml)*.
+
+```yaml
+storageType: astra
+cassandra:
+  clusterName: "reaper"
+  contactPoints: ["astra host from the secure bundle config.json file"]
+  keyspace: reaper_db
+  port: <cql port found in the secure bundle cqlshrc file>
+  authProvider:
+    type: plainText
+    username: reaper
+    password: ReaperOnAstraRocks
+  ssl:
+    type: jdk
+```
+
+The CQL port to connect to can be found in the `cqlshrc` file of Astra's secure connect bundle. The port found in `config.json` is the metadata port which cannot be used for CQL connections. 
+
+The Astra backend provides the same capabilities as [the Cassanda backend]({{<ref "cassandra.md">}}).
+
+Schema initialization and migration will be done automatically upon startup.
+
+## SSL settings
+
+Astra enables client to node encryption by default, which requires some additional setup in Reaper.
+After installing Reaper and configuring the yaml file, copy the `cassandra-reaper-ssl.properties` file to the `/etc/cassandra-reaper` directory (the temmplate can be found under `/etc/cassandra-reaper/configs/`) and configure it as follows:
+
+```
+-Djavax.net.ssl.keyStore=/path/to/identity.jks
+-Djavax.net.ssl.keyStorePassword=keystore_password
+
+-Djavax.net.ssl.trustStore=/path/to/trustStore.jks
+-Djavax.net.ssl.trustStorePassword=truststore_password
+
+# Comment the following line when using the Astra backend
+# unless JMX encryption is enabled with the same keystore/truststore
+# -Dssl.enable=true
+```
+
+The truststore and keystore (identity) files can be found in the secure connect bundle which should be downloaded from your Astra dashboard. The passwords will be found in the `config.json` file of that same bundle.
+Make sure you comment the `-Dssl.enable=true` line as it enables JMX encryption.
+
+If both CQL and JMX encryption need to be enabled, then JMX encryption must be configured to use the same truststore/keystore than Astra, and the `-Dssl.enable=true` should be left uncommented.

src/docs/content/docs/configuration/reaper_specific.md

@@ -397,7 +397,7 @@ Note that to bind the service to all interfaces use value **0.0.0.0** or leave t
 
 Type: *String*
 
-The storage type to use in which Reaper will store its control data. The value must be either **cassandra**, **h2**, **memory**, or **postgres**. If the recommended (persistent) storage type **cassandra**, **h2**, or **postgres** is being used, the database client parameters must be specified in the respective `cassandra`, `h2`, or `postgres` section in the configuration file. See the example settings in provided the *[src/packaging/resources](https://github.com/thelastpickle/cassandra-reaper/tree/master/src/packaging/resource)* directory of the repository.
+The storage type to use in which Reaper will store its control data. The value must be either **cassandra**, **astra**, **h2**, **memory**, or **postgres**. If the recommended (persistent) storage type **cassandra**, **h2**, or **postgres** is being used, the database client parameters must be specified in the respective `cassandra`, `h2`, or `postgres` section in the configuration file. See the example settings in provided the *[src/packaging/resources](https://github.com/thelastpickle/cassandra-reaper/tree/master/src/packaging/resource)* directory of the repository.
 
 <br/>
 

src/packaging/resource/cassandra-reaper-astra.yaml

@@ -0,0 +1,138 @@
+# Copyright 2020-2020 DataStax, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Cassandra Reaper Configuration Example.
+# See a bit more complete example in:
+# src/server/src/test/resources/cassandra-reaper.yaml
+segmentCountPerNode: 64
+repairParallelism: DATACENTER_AWARE
+repairIntensity: 0.9
+scheduleDaysBetween: 7
+repairRunThreadCount: 15
+hangingRepairTimeoutMins: 30
+storageType: astra
+enableCrossOrigin: true
+incrementalRepair: false
+blacklistTwcsTables: true
+enableDynamicSeedList: true
+repairManagerSchedulingIntervalSeconds: 10
+activateQueryLogger: false
+jmxConnectionTimeoutInSeconds: 5
+useAddressTranslator: false
+# purgeRecordsAfterInDays: 30
+# numberOfRunsToKeepPerUnit: 10
+
+# datacenterAvailability has three possible values: ALL | LOCAL | EACH | SIDECAR
+# the correct value to use depends on whether jmx ports to C* nodes in remote datacenters are accessible
+# If the reaper has access to all node jmx ports, across all datacenters, then configure to ALL.
+# If jmx access is only available to nodes in the same datacenter as reaper in running in, then configure to LOCAL.
+# If there's a reaper instance running in every datacenter, and it's important that nodes under duress are not involved in repairs,
+#    then configure to EACH.
+# If jmx access is restricted to localhost, then configure to SIDECAR.
+#
+# The default is ALL
+datacenterAvailability: ALL
+
+logging:
+  level: INFO
+  loggers:
+    com.datastax.driver.core.QueryLogger.NORMAL:
+      level: DEBUG
+      additive: false
+      appenders:
+        - type: file
+          currentLogFilename: /var/log/cassandra-reaper/query-logger.log
+          archivedLogFilenamePattern: query-logger-%d.log.gz
+          archivedFileCount: 2
+    io.dropwizard: WARN
+    org.eclipse.jetty: WARN
+  appenders:
+    - type: console
+      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
+      threshold: WARN
+    - type: file
+      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
+      currentLogFilename: /var/log/cassandra-reaper/reaper.log
+      archivedLogFilenamePattern: /var/log/cassandra-reaper/reaper-%d.log.gz
+      archivedFileCount: 99
+
+server:
+  type: default
+  applicationConnectors:
+    - type: http
+      port: 8080
+      bindHost: 0.0.0.0
+  adminConnectors:
+    - type: http
+      port: 8081
+      bindHost: 0.0.0.0
+  requestLog:
+    appenders: []
+
+cassandra:
+  clusterName: "reaper"
+  contactPoints: ["astra host from the secure bundle config.json file"]
+  keyspace: reaper_db
+  port: <cql port found in the secure bundle cqlshrc file>
+  authProvider:
+    type: plainText
+    username: reaper
+    password: ReaperOnAstraRocks
+  ssl:
+    type: jdk
+  loadBalancingPolicy:
+    type: tokenAware
+    shuffleReplicas: true
+    subPolicy:
+      type: dcAwareRoundRobin
+      localDC:
+      usedHostsPerRemoteDC: 0
+      allowRemoteDCsForLocalConsistencyLevel: false
+autoScheduling:
+  enabled: false
+  initialDelayPeriod: PT15S
+  periodBetweenPolls: PT10M
+  timeBeforeFirstSchedule: PT5M
+  scheduleSpreadPeriod: PT6H
+  excludedKeyspaces:
+    - keyspace1
+    - keyspace2
+  excludedClusters:
+#   - cluster1
+
+
+# Uncomment the following to enable dropwizard metrics
+#  Configure to the reporter of your choice
+#  Reaper also provides prometheus metrics on the admin port at /prometheusMetrics
+
+#metrics:
+#  frequency: 1 minute
+#  reporters:
+#    - type: log
+#      logger: metrics
+
+# Uncomment the following to enable encryption and the ability to store jmx credentials in
+#  the backend data source where SOME_SYSTEM_PROPERTY_KEY is a system property that can be
+#  used to lookup the shared symmetric key.
+#  ie: export SOME_SYSTEM_PROPERTY=YOUR_SHARED_SYMMETRIC_KEY
+
+#cryptograph:
+#  type: symmetric
+#  systemPropertySecret: SOME_SYSTEM_PROPERTY
+
+# Authentication is enabled by default
+accessControl:
+  sessionTimeout: PT10M
+  shiro:
+    iniConfigs: ["classpath:shiro.ini"]

src/packaging/resource/cassandra-reaper-ssl.properties

@@ -32,4 +32,6 @@
 -Djavax.net.ssl.trustStore=/path/to/truststore.jks
 -Djavax.net.ssl.trustStorePassword=truststore_password
 
+# Comment the following line when using the Astra backend
+# unless JMX encryption is enabled with the same keystore/truststore
 -Dssl.enable=true

src/server/pom.xml

@@ -150,7 +150,13 @@
         <dependency>
             <groupId>com.datastax.cassandra</groupId>
             <artifactId>cassandra-driver-core</artifactId>
-            <version>3.7.1</version>
+            <version>3.10.1</version>
+            <exclusions>
+	            <exclusion>
+		            <groupId>com.fasterxml.jackson.core</groupId>
+	                <artifactId>jackson-databind</artifactId>
+	            </exclusion>
+	        </exclusions>
         </dependency>
         <dependency>
             <groupId>org.postgresql</groupId>

src/server/src/checkstyle/java.header

@@ -1,16 +1,16 @@
-/*
-( * Copyright 2014-2017 Spotify AB)?( * Copyright \d\d\d\d-\d\d\d\d Stefan Podkowinski)?
-( * Copyright \d\d\d\d-\d\d\d\d The Last Pickle Ltd)?
- *
- * Licensed under the Apache License, Version 2.0 \(the "License"\);
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
+\/\*
+(^ \* Copyright \d\d\d\d-\d\d\d\d Spotify AB$|^ \* Copyright \d\d\d\d-\d\d\d\d Stefan Podkowinski$|^ \*$)?
+(^ \* Copyright \d\d\d\d-\d\d\d\d The Last Pickle Ltd$)+(^ \* Copyright \d\d\d\d-\d\d\d\d DataStax, Inc.$|^ \*$){0,1}
+( \*|\* Licensed under the Apache License, Version 2.0 \(the "License"\);)+
+( \* Licensed under the Apache License, Version 2.0 \(the "License"\);| \* you may not use this file except in compliance with the License\.)+
+( \* you may not use this file except in compliance with the License\.| \* You may obtain a copy of the License at)+
+( \* You may obtain a copy of the License at| \*)+
+( \*| \*     http:\/\/www\.apache\.org\/licenses\/LICENSE-2\.0)+
+( \*     http:\/\/www\.apache\.org\/licenses\/LICENSE-2\.0| \*)+
+( \*| \* Unless required by applicable law or agreed to in writing, software)+
+( \* Unless required by applicable law or agreed to in writing, software|\* distributed under the License is distributed on an "AS IS" BASIS,)+
+( \* distributed under the License is distributed on an "AS IS" BASIS,| \* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied\.)+
+( \* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied\.| \* See the License for the specific language governing permissions and)+
+( \* See the License for the specific language governing permissions and| \* limitations under the License\.)+
+( \* limitations under the License\.| \*)+
+( \*)+

src/server/src/main/java/io/cassandrareaper/ReaperApplication.java

@@ -1,6 +1,7 @@
 /*
  * Copyright 2014-2017 Spotify AB
  * Copyright 2016-2019 The Last Pickle Ltd
+ * Copyright 2020-2020 DataStax, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -67,6 +68,7 @@
 import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
 import io.dropwizard.Application;
 import io.dropwizard.assets.AssetsBundle;
 import io.dropwizard.client.HttpClientBuilder;
@@ -468,8 +470,11 @@ private IStorage initializeStorage(ReaperApplicationConfiguration config, Enviro
 
     if ("memory".equalsIgnoreCase(config.getStorageType())) {
       storage = new MemoryStorage();
-    } else if ("cassandra".equalsIgnoreCase(config.getStorageType())) {
-      storage = new CassandraStorage(context.reaperInstanceId, config, environment);
+    } else if (Lists.newArrayList("cassandra", "astra").contains(config.getStorageType())) {
+      CassandraStorage.CassandraMode mode = config.getStorageType().equals("cassandra")
+          ? CassandraStorage.CassandraMode.CASSANDRA
+          : CassandraStorage.CassandraMode.ASTRA;
+      storage = new CassandraStorage(context.reaperInstanceId, config, environment, mode);
     } else if ("postgres".equalsIgnoreCase(config.getStorageType())
         || "h2".equalsIgnoreCase(config.getStorageType())
         || "database".equalsIgnoreCase(config.getStorageType())) {