extended_README.md

Benchmark your Astra DB with NoSQLBench

Time: 2 hours. Difficulty: Intermediate. Start Building!

The goal of this workshop is to get you familiar with the powerful and versatile tool NoSQLBench. With that, you can perform industry-grade, robust benchmarks aimed at several (distributed) target systems, especially NoSQL databases.

Today you'll be benchmarking Astra DB, a database-as-a-service built on top of Apache Cassandra. Along the way, you will learn the basics of NoSQLBench.

In this repository you will find all material and references you need:

• NoSQLBench Discord
• NoSQLBench homepage
• Exercises
• Step-by-step guide
• DataStaxDevs Discord server to keep in touch with us
• Our Q&A forum (think StackOverflow for Cassandra and all things DataStax)
• Slide deck

Table of Contents

1. Before you start
2. Create your Astra DB instance
3. Launch Gitpod and setup NoSQLBench
4. Run benchmarks
5. Workloads
6. Homework assignment

Before you start

Heads up: these instructions are available in two forms: a short and to-the-point one, with just the useful commands if you are watching us live; and a longer one (this one), with lots of explanations and details, designed for those who follow this workshop at their own pace. Please choose what best suits you!

If you are not watching us live, please look at the provided presentation material for more coverage of the theory part. There, you will learn more about:

• benchmarking a database;
• Apache Cassandra™, a distributed NoSQL highly available database;
• Astra DB, a database-as-a-service in the cloud, built on Cassandra, ready to use in minutes for free with a few clicks;
• the main features and general principles behind the NoSQLBench benchmarking tool.

FAQ

• What are the prerequisites?

This workshop is aimed at data architects, solution architects, developers, or anybody who wants to get serious about measuring the performance of their data-intensive system. You should know what a (distributed) database is, and have a general understanding of the challenges of communicating over a network.

• Do I need to install a database or anything on my machine?

No, no need to install anything. You will do everything in the browser. (That being said, the knowledge you gain today will probably be best put to use once you install NoSQLBench on some client machine to run tests.)

You can also choose to work on your machine instead of using Gitpod: there's no problem with that, just a few setup and operational changes to keep in mind. We will not provide live support in this case, though, assuming you know what you are doing.

• Is there anything to pay?

No. All materials, services and software used in this workshop is free.

• Do you cover NoSQLBench 4 or 5?

Ah, I see you are a connoisseur. We focus on the newly-release NoSQLBench 5, but we provide tips and remarks aimed at those still using nb4. See here for a bit of context.

Homework

To complete the workshop and get a verified "NoSQLBench" badge, follow these instructions:

1. Do the hands-on practice, either during the workshop or by following the instructions in this README;
2. (optional) Complete the "Lab" assignment as detailed here;
3. Fill the submission form here. Answer the theory questions and (optionally) provide a screenshot of the completed "Lab" part;
4. give us a few days to process your submission: you should receive your well-earned badge in your email inbox!

Create your Astra DB instance

First you must create a database: an instance of Astra DB, which you will then benchmark with NoSQLBench.

Don't worry, you will create it within the "Free Tier", which offers quite a generous free allowance in terms of monthly I/O (about 40M operations per month) and storage (80 GB).

You need to:

• create an Astra DB instance as explained here, with database name = workshops and keyspace name = nbkeyspace;
• (this will happen automatically with the previous one) generate and retrieve a DB Token as explained here. Important: use the role "DB Administrator" if manually creating the token.
• generate and download a Secure Connect Bundle as explained here;

Moreover, keep the Astra DB dashboard open: it will be useful later. In particular, locate the Health tab and the CQL Console.

Launch Gitpod and setup NoSQLBench

Gitpod is an IDE in the cloud (modeled after VSCode). It comes with a full "virtual machine" (actually a Kubernetes-managed container), which you will use as if it were your own computer (e.g. downloading files, executing programs and scripts, even launching containers from within it).

Now ctrl-click on the Gitpod button below. It will:

• spawn your own Gitpod container;
• clone this repository in it;
• preinstall the required dependencies.

Note that you may have to authenticate through Github in the process. (Also make sure you "Open in new tab" to keep this readme open alongside.)

In a few minutes, a full IDE will be ready in your browser, with a file explorer on the left, a file editor on the top, and a console (bash) below it.

There are many more other features, probably familiar to those who have experience with VSCode. Feel free to play around a bit!

Install NoSQLBench

Download the latest stable version of NoSQLBench to your Gitpod environment. First make sure you are in the top-level repository directory with your Gitpod console: type pwd and see if the output is /workspace/workshop-nosqlbench.

Please download the "Linux binary" distribution of NoSQLBench: as instructed here, you can obtain the latest stable binary (it's a few hundred megabytes) with

curl -L -O https://github.com/nosqlbench/nosqlbench/releases/latest/download/nb5

and, when the download is finished, make it executable and move it, out of convenience, to a directory which is part of the search path:

chmod +x nb5
sudo mv nb5 /usr/local/bin/

Ok, now check that the program starts: invoking

nb5 --version

should output the program version (something like 5.17.3 or higher).

You will probably see a message like Picked up JAVA_TOOL_OPTIONS ... when launching nb5. You can ignore it: it is a consequence of some settings by Gitpod and does not have to do with NoSQLBench itself.

Note that if your Gitpod instance gets hibernated (which happens after some inactive time) and you restart it later, recall that only the contents of /workspace will be restored: hence, you would have to repeat the installation.

Version used

This workshop is built for the newly-released NoSQLBench 5. Compared to version 4, there are many differences both under the hood and for the user. We will provide tips and remarks to help users of version 4 as well throughout the following.

Tell me more

NoSQLBench 5 comes with a completely redesigned architecture for drivers, which grants all drivers (present and future) a set of nice features for free and makes it easier to develop new ones.

On a more user-facing side, among several improvements, the error handling logic has been enhanced; the workload yaml syntax has been streamlined; new and upgraded drivers are available for various databases.

Tip: in the NoSQLBench repo itself, you can find plenty of information on the specifications for the workload yaml syntax.

Great care has been taken to maintain backward compatibility (e.g. in the workload yaml syntax); in the following, some of the most relevant changes will be highlighted, especially concerning the workload syntax, to support users of version 4 as well.

Upload the Secure Connect Bundle to Gitpod

NoSQLBench will need to connect to your Astra DB database: to do so, the Secure Connect Bundle zip file must be uploaded to your Gitpod environment.

Locate, with the file explorer on your computer, the bundle file that you downloaded earlier (it should be called secure-connect-workshops.zip) and simply drag-and-drop it to the file navigator panel ("Explorer") on the left of the Gitpod view.

Show me

Once you drop it you will see it listed in the file explorer itself. As a check, you can issue the command

ls /workspace/workshop-nosqlbench/secure*zip -lh

so that you get the absolute path to your bundle file (and also verify that it is the correct size, about 12-13 KB). As an aside, note that, as per best practices with NoSQLBench, (relative) paths to files would start with ./.

Show me

Configure the Astra DB parameters

If you want NoSQLBench to be able to access the Astra DB instance, you have to pass it the required connection parameters and secrets. To do so, set up a .env file, which will make your life easier later.

Copy the provided template file to a new one and open it in the Gitpod file editor:

cp .env.sample .env
gp open .env
# (you can also simply locate the file
#  in the Explorer and click on it)

Insert the "Client ID" and "Client Secret" of the DB Token you created earlier and, if necessary, adjust the other variables.

Show me what the .env file would look like

Now, source this file to make the definitions therein available to this shell:

. .env

To check that the file has been sourced, you can try with:

echo ${ASTRA_DB_KEYSPACE_NAME}

and make sure the output is not an empty line.

(Note that you will have to source the file in any new shell you plan to use).

Run benchmarks

Everything is set to start running the tool.

A short dry run

From a user's perspective, NoSQLBench sends operations (e.g. reads, writes) to the target system (e.g. a database) and record how the latter responds (service time, number of ops/second and failures).

Try launching this very short "dry-run benchmark", that instead of actually reaching the database simply prints a series of CQL statements to the console (as specified by the driver=stdout parameter):

nb5 cql-keyvalue2 astra                 \
    driver=stdout                       \
    rampup-cycles=10                    \
    main-cycles=10                      \
    keyspace=${ASTRA_DB_KEYSPACE_NAME}

You will see 21 (fully-formed, valid CQL) statements being printed: one CREATE TABLE, then ten INSERTs and then another ten between SELECTs and further INSERTs.

Note: we will use workload cql-keyvalue2 throughout. This is functionally identical to the cql-keyvalue workload but is expressed in the newer syntax for yaml workloads, which comes handy when later dissecting its content. If you are working with NoSQLBench 4, remember to drop the trailing 2 from the workload name in the following!

These three blocks correspond to the three phases defined in the cql-keyvalue2 workload: schema, rampup and main. The most relevant benchmark results are from the main phase, which mimics the kind of load the database would be subject to when operating normally.

For most workloads, a three-phase structure is the most reasonable choice: first the schema is set up, then an amount of data is poured in, and finally the target system is subject to a mixture of reads and writes. The third and last phase (main) in most cases is designed to generate contention on the server, which is the very situation one usually wants to put to test.

Now re-launch the above dry run and look for differences in the output:

nb5 cql-keyvalue2 astra                 \
    driver=stdout                       \
    rampup-cycles=10                    \
    main-cycles=10                      \
    keyspace=${ASTRA_DB_KEYSPACE_NAME}

do you notice that the output of this second run is identical to the first, down to the actual values used in the INSERTs and DELETEs? Indeed an important goal for any benchmark is its reproducibility: here, among other consequences, this means that the operations (their progression and contents alike) should be completely determined with no trace of actual randomness.

You can also peek at the logs directory now: it is created automatically and populated with some information from the benchmark at each execution of nb.

Benchmark your Astra DB

It is now time to start hitting the database! What you launched earlier is the cql-keyvalue2 workload, one of the several ready-to-use workloads included with NoSQLBench. In particular, you ran the astra scenario, which determines a particular way the workload is to be unfolded and executed.

This time you will run with driver=cql to actually reach the database: for that to work, you will provide all connection parameters set up earlier.

Note: for this workload, as you could check by examining its definition (see the Inspect "cql-keyvalue" section below), you could leave the driver specification out since cql is the default.

The next run will ask NoSQLBench to perform a substantial amount of operations, in order to collect enough statistical support for the results.

On the other hand, you will rate-limit the operations sent to the database (with the cyclerate parameter). This is mainly because often the goal of a benchmark is to verify whether the system can withstand a known, controlled workload; but in this case there are also other practical reasons: (1) you don't want to consume too much of your Free Tier monthly allowance in this "functional demo"; moreover, (2) if you are running on the Free Tier you may otherwise hit the guardrails and the performance limitations specific to free-tier Astra databases.

Here is the full command to launch:

nb5 cql-keyvalue2                                                         \
    astra                                                                 \
    username=${ASTRA_DB_CLIENT_ID}                                        \
    password=${ASTRA_DB_CLIENT_SECRET}                                    \
    secureconnectbundle=${ASTRA_DB_BUNDLE_PATH}                           \
    keyspace=${ASTRA_DB_KEYSPACE_NAME}                                    \
    cyclerate=50                                                          \
    driver=cql                                                            \
    main-cycles=9000                                                      \
    rampup-cycles=9000                                                    \
    errors='OverloadedException:warn'                                     \
    --progress console:5s                                                 \
    --log-histograms 'histogram_hdr_data.log:.*.main.result.*:20s'        \
    --log-histostats 'hdrstats.log:.*.main.result.*:20s'

Show me the command breakdown

Note that some of the parameters (e.g. keyspace) are workload-specific.

command	meaning
cql-keyvalue2	workload
astra	scenario
username	authentication
password	authentication
secureconnectbundle	Astra DB connection parameters
keyspace	target keyspace
cyclerate	rate-limiting (cycles per second)
driver=cql	driver to use (CQL, for AstraDB/Cassandra)
main-cycles	how many operations in the "main" phase
rampup-cycles	how many operations in the "rampup" phase
errors	behaviour if errors occur during benchmarking
--progress console	frequency of console prints
--log-histograms	write data to HDR file (see later)
--log-histostats	write some basic stats to a file (see later)

This way of invoking nb, the "named scenario" way, is not the only one: it is also possible to have a finer-grained control over what activities should run with a full-fledged CLI scripting syntax.

Note: the syntax of the errors parameter has been improved in NoSQLBench 5 to allow for a finer control (with multiple directives, such as errors='NoNodeAvailable.*:ignore;InvalidQueryException.*:counter;OverloadedException:warn'). On version 4 you should revert to a simpler parameter, such as errors=count, instead of the above.

Note: if you were targeting a "regular" Cassandra instance (as opposed to Astra DB), the command line above would change a little: the scenario would be default, you would need to provide a parameter such as hosts=192.168.1.1,192.168.1.2, and you would have additionally to provide a parameter localdc=datacenter1 (change to reflect the name of one of your Cassandra datacenters). The latter is required starting from NoSQLBench 5, but is a good practice with earlier versions too. Further, there would be no secureconnectbundle parameter. As for username and password, ... well, that depends on how the Cassandra cluster is configured. As far as benchmark results are concerned, in the next steps you will have to look at metric with "default" in their name instead of "astra". Note that NoSQLBench uses the very same CQL drivers to access the database, regardless of whether Astra DB or a Cassandra cluster.

The above command should run for approximately six minutes, during which NoSQLBench sends a constant stream of I/O operations to the database and collects timing information on how it responds. You will see a console output keeping you updated on the progress of the currently-running phase (schema/rampup/main).

If the execution fails with, "Cannot construct cloud config from the cloudConfigUrl ...", chances are your (free-tier) database is currently in Standby status. To resume it, open its Health tab or the CQL Console tab in the Astra DB UI and wait two or three minutes before retrying.

While this runs, have a look around.

Database contents

By this point, you may have a question: "but what is being written to the database, exactly?

To find out, connect to the database and inspect the contents of the keyspace. There are several ways one could connect to Astra DB: using Cassandra drivers with most programming languages; through the HTTP requests enabled by the Stargate Data API; or directly with a client that "speaks" the CQL language. Today you will make use of the CQL Console available in the browser within the Astra UI.

Choose your database in the Astra main dashboard and click on it; next, go to the "CQL Console" tab in the main panel. In a few seconds the console will open in your browser, already connected to your database and waiting for your input.

Show me how to get to the CQL Console in Astra

Commands entered in the CQL Console are terminated with a semicolon (;) and can span multiple lines. Run them with the Enter key. If you want to interrupt the command you are entering, hit Ctrl-C to be brought back to the prompt. See here for more references to the CQL language commands.

Start by telling the console that you will be using the nbkeyspace keyspace:

USE nbkeyspace;

Check what tables have been created by NoSQLBench in this keyspace:

DESC TABLES;

You should see table keyvalue listed as the sole output. Look at a a few lines from this table:

SELECT * FROM keyvalue LIMIT 20;

Show me what the output looks like

Ok, mystery solved. It looks like the table contains simple key-value pairs, with two columns seemingly of numeric type. Check with:

DESC TABLE keyvalue;

Surprise! As a matter of fact both columns are of type TEXT (that is, variable-length strings). Indeed, most key-value stores admit string keys and (unless specific schemas are enforced) the values are also either binary blobs or ASCII sequences: this workload could then be adapted to benchmarking other key-value databases.

Show me what the output looks like

Database health

By now, the benchmark is probably still running. That's good: you get a chance to inspect the "database health" while NoSQLBench is writing to it. Remember you instructed the tool to work at a steady rate in terms of operations per second.

Locate your database in the Astra main dashboard and click on it; next, go to the "Health" tab in the main panel. You will see what essentially is a Grafana dashboard, with a handful of plots being displayed within the tab - all related to how the database is performing in terms of reads and writes.

Show me the Database Health tab in Astra UI

Tip: you can customize the width of the time window in the graphs and the update frequency. Today it could make sense to set these to 15 minutes and 5 seconds respectively.

Take a look at the plot labeled "Requests Combined": this describes the amount of write and read requests per second received by the database. You will note that the total fluctuates around the value provided with the cyclerate parameter during the whole test: but during rampup it will be all writes, while the main phase will be an equal mixture of reads and writes.

Now turn your attention to the "Write Latency"/"Read Latency" plots. These provide quantities related to the "latency", as experienced by the coordinator node, involved in servicing write/read requests. In particular, some reference percentiles are reported as they vary in time: indeed, when describing the performance of the target system, percentiles are a much better tool than averages or maximum values. Note down the values you read from the plot in the middle of the main phase.

Show me "sample values" one could read from the graph

Below is a real-life example of the values that could result from a cql-keyvalue2 benchmark session in the main phase:

Percentile	Write Latency	Read Latency
P50	709 µs	935 µs
P75	831 µs	1.31 ms
P90	904 µs	1.53 ms
P95	1.04 ms	1.77 ms
P99	2.45 ms	15.6 ms

What is the meaning of a percentile? Well, if one says "P75 for reading is 13 milliseconds," that means "75% of the read requests are serviced within 13 ms from being received by the server." This kind of metric, in most cases, is more meaningful than quoting the maximum value recorded (for one, the max tends to fluctuate way more).

Quoting high percentiles such as P95, P99 or similar is a standard practice when expressing the performance of a data system.

Final summary in "logs/"

By now the benchmark should have finished. If it hasn't yet, give it time to complete: you will see a summary being printed, after which you will get the console prompt back.

Each run of NoSQLBench generates timestamped files in the logs directory (which is created automatically if not present). The on-screen summary is a shortened form of the data found in the summary files there.

Now open the most recent *.summary file in that directory, corresponding to the nb invocation that just completed. Find the metric called cqlkeyvalue2_astra_main.result-success: this corresponds to events of the type "a command was issued to the database during the main phase and succeeded (and the time needed for it to complete, as seen from the client side, was recorded)".

The related metric cqlkeyvalue2_astra_main.result is similar, but comprises all events regardless of whether they succeeded or not. In "healthy" cases, these two are identical or at most very close to each other. This difference is sometimes characterized as the "goodput/throughput" dichotomy.

Under that metric title, you will see something similar to:

cqlkeyvalue2_astra_main.result-success
             count = 15000
         mean rate = 50.00 calls/second
     1-minute rate = 49.94 calls/second
     5-minute rate = 50.29 calls/second
    15-minute rate = 50.57 calls/second

This tells you the total count and the per-minute rate (as averaged over different time windows) for this operation.

NoSQLBench keeps track of count/timing information for many operations, which range from substitution of variables in the workload commands to actual sending of an I/O operation to the database.

Additional "histostats" datafile

You instructed NoSQLBench to output timing information in two other formats:

First, with the --log-histostats option you get a file with a listing, for selected metrics, of the min, max and some percentiles as measured over a desired time window. Try to look at the file hdrstats.log in the Gitpod editor and figure out its structure.

If you want to graphically look at the data thus collected, we provide a sample script for plotting. Simply enter the command (try -h for more options)

./hdr_tool/histostats_quick_plotter.py \
    hdrstats.log \
    -m cqlkeyvalue2_astra_main.result-success

and then open, in the Gitpod editor, the hdrstats.png image just created.

Show me the generated "histostats" plot

The version of the plotter script included in this repo is for educational purposes only: for general use, please head to the official release page.

Tip: you can use Ctrl-mousewheel to zoom on the image in Gitpod.

Look at the values of, say, the P90 percentile: it will be larger than both the read and write corresponding percentiles given in the Astra DB "health" tab. That's because this time you are seeing things from the vantage point of the testing client, and the (Gitpod-to-Astra) communication over the network is included in the "service time".

Besides, the comparison is made somewhat murkier by the fact that NoSQLBench measures "cycles", which means reads and writes in the main phase; while the Astra health dashboard keeps these two separate. A way to make this comparison more apples-to-apples would be to "instrument" the metric collection: adding the instrument=true parameter to the invocation will attach a separate timer to each of the named metric in the workload definition, and each of these metrics will be featured e.g. in the summary file or anywhere for further inspection.

HDR extensive histogram data

Second, the --log-histograms option has the effect that NoSQLBench writes the whole of the measurements it takes in the HDR file format.

This is an optimized way to save the full histogram of the measured service time using exact counts over very thin fixed-width bins. Try to inspect the contents of file histogram_hdr_data.log: you will notice the data is stored in ASCII-encoded long strings.

Fortunately for you, we provide a sophisticated plotting tool for this file, which may serve as inspiration or be used directly for your own benchmarks.

Try running the command (run with -h to see more options)

./hdr_tool/hdr_tool.py \
    histogram_hdr_data.log \
    -b -c -s \
    -p SampleData \
    -m cqlkeyvalue2_astra_main.result-success

and look at the SampleData*.png plots that are generated:

• the "base plot" is the full collected histogram of the chosen metric;
• the "stability plot" plots measurements from each time window separately, to help assessing whether the main phase is really a steady state;
• the "percentile plot" reformulates the histogram in terms of percentiles.

Show me the plots generated by the HDR file

The version of the plotter script included in this repo is for educational purposes only: for general use, please head to the official release page.

Again, the timings are larger than those found on the Astra health tab (i.e. on server-side): these measurements are reported "as seen by the testing client".

Tip: this script is just a demonstration of the versatility of the HDR format: you can invent your own post-analysis tool and attach it to the data collected by NoSQLBench. Look at the (Python) source code of the tool for inspiration! (Or simply take the tool and use it on your data.)

Metrics, metrics, metrics

On top of everything you have seen, it is possible to have NoSQLBench start a Grafana dashboard locally alongside the benchmark itself, powered behind the scenes by Prometheus-based real-time metric collection.

All it takes is the additional --docker-metrics option to the command line, and Docker must be available on your system (Note: Gitpod comes with Docker preinstalled).

You can try launching another benchmark as follows (note the last option and the fact that this time we dropped the driver parameter):

nb5 cql-keyvalue2                                                         \
    astra                                                                 \
    username=${ASTRA_DB_CLIENT_ID}                                        \
    password=${ASTRA_DB_CLIENT_SECRET}                                    \
    secureconnectbundle=${ASTRA_DB_BUNDLE_PATH}                           \
    keyspace=${ASTRA_DB_KEYSPACE_NAME}                                    \
    cyclerate=50                                                          \
    rampup-cycles=15000                                                   \
    main-cycles=15000                                                     \
    errors='OverloadedException:warn'                                     \
    --progress console:5s                                                 \
    --docker-metrics

This command might take a few additional seconds to start the first time, as Docker images are being downloaded and the containers are started. Then, successful start of the Grafana dashboard should be logged, at which point the usual cql-keyvalue2 workload will start.

Note: on some versions of NoSQLBench 5 you may see an error such as Java HttpClient was not authorized, [...] :WWW-Authenticate header missing for response code 401. This is due to the Grafana server not complying with the HTTP 401 specifications, and can be fixed by running the script ./_grafana_docker_key_fetcher.sh and then re-starting the above nb5 command. (the script manually generates an API key for interacting with Grafana, storing it where then nb5 expects to find it.)

Show me the run with Docker metrics

It is possible that Gitpod will detect new services running on some local ports and automatically open the Grafana dashboard in its mini-browser. Better to ignore it and re-open the URL in a new, actual browser tab as instructed below.

Grafana dashboard

The Grafana container is running and exposed on local port 3000. Luckily, Gitpod is kind enough to map local ports to externally-accessible addresses, such as https://3000-datastaxdevs-workshopnos-nq5y2jf5uw9.ws-eu38.gitpod.io. Your URL will be different: to construct it, simply prepend a 3000- to the URL of the Gitpod IDE. Then, open this address in a new tab.

The default credentials to log in to Grafana are ... admin/admin. Once you're in, don't bother to reset your password (click "Skip"). You'll get to the Grafana landing page. Find the "Dashboards" icon in the leftmost menu bar and pick the "Manage" menu item: finally, click on the "NB4 Dashboard" item you should see listed there. Congratulations, you are seeing the data coming from NoSQLBench.

Show me how to get to the Grafana plots

You may find it convenient to set the update frequency to something like 10 seconds and the displayed time window to 5 minutes or so (upper-right controls).

The dashboard comprises several (interactive) plots, updated in real time. Let's see some of them and their significance.

Show me the dashboard contents

• "Ops and Successful Ops". One-minute averages of the total operations dispatched per second by NoSQLBench. This will match the reported per-second averages found in the *.summary file. There are separate curves for each phase, so you will clearly see rampup leaving the room to main at some point during the test.
• "p75/p99 client overhead". This is related to timing of operations internal to NoSQLBench (note the scale, barely a few micro-seconds). You should keep an eye on those to check that the benchmarking client is not experiencing bottlenecks of sorts.
• "service time distribution/range". Here you can see percentiles for "service time" relative to the various phases, a measurement corresponding to the "success" metric introduced earlier. These quantities then directly relate to the HDR histogram data analyzed above and should match them. Note that the "min" and "max" are denoted "p0" and "p100" here (indeed this is what they are, by definition).

A glance at Prometheus

If you want to look at the "raw" source of data feeding the Grafana plots, head to the Prometheus UI, exposed at local port 9090. If you are on Gitpod, similarly to what you did for Grafana earlier, you get there by prepend a 9090- to the domain name you are using your IDE at. Then, open this address in a new tab.

Show me the Prometheus UI

Click on the "world" icon next to the "Execute" button in the search bar: in the dialog that appears you can look for specific metrics. Try to look for result_success and confirm, then click "Execute".

The search results are many datapoints, pertaining to several metrics related to the service time. Still, those are for different phases and, most important, different measurements (percentiles, averages, counts), which does not really make sense to inspect together.

Tip: switch to the "Graph" view for a more immediate visualization. The graphs display "raw" data, hence are in units of nanoseconds.

Fortunately, each datapoint is fully equipped with detailed metadata, which allow for sophisticated filtering. Further, you can use the flexible, powerful query language used by Prometheus to even perform aggregations on top of the data queries.

Just to pique your interest, try pasting these examples and click "Execute":

# filtering by metadata
{__name__="result_success", type="pctile", alias=~".*main.*"}

# aggregation
avg_over_time({__name__="result_success", type="pctile", alias=~".*main.*"}[10m])

# another aggregation, + filtering
max_over_time({__name__="result_success", type="pctile", alias=~".*main.*"}[10m])

Also, note that Prometheus exposes a REST access, so you can imagine, for instance, a running scripts that tracks the metrics and responds to them in real time.

Workloads

We mentioned that cql-keyvalue2 is one of several ready-to-use workloads, but also that you can easily build your own: for example, to test a specific data model or ensure you closely mimic your application's request pattern.

So, now it's time for a closer inspection of workloads. Each workload is built as a yaml file, defining its scenarios and the sequences of phases each scenario is made of.

Phases, in turn, consist of several statements that the driver will know how to "execute" (i.e. send to the target system as operations to execute).

There is quite some freedom in creating workloads: in the following you will just explore some of this space. Look into the reference documentation for more.

Tip: feel free to interrupt the previous benchmark, if it still runs, with Ctrl-C. You won't need it anymore.

Inspect "cql-keyvalue"

Ask NoSQLBench to dump to a file the yaml defining the workload you just ran:

    nb5 --copy cql-keyvalue2

(you can also get a comprehensive list of all available workloads with nb5 --list-workloads, by the way, and a more fine-grained output with nb5 --list-scenarios.)

A file cql-keyvalue2.yaml is created in the working directory. You can open it (clicking on it in the Gitpod explorer or by running gp open cql-keyvalue2.yaml).

This is not a line-by-line analysis, but broadly speaking there are three important sections:

• scenarios. This defines which phases constitute each scenario. More specifically, each phase is expressed as an inline script, which in turn follows the NoSQLBench CLI scripting syntax.
• bindings. Each binding defines a particular recipe to generate a sequence of pseudo-random values. Emphasis is on the "pseudo", since bindings are (usually) fully deterministic, reproducible, mappings of cycle numbers to values. The sequence of "keys" produced as the the rampup phase of key-value workload unfolds, for instance, is defined here by the seq_key binding. Bindings are defined using a functional compositional approach, starting from a set of available building blocks, and can generate values of many different data types.
• blocks. Here, groups of "statements" (better: operations) are defined for use within scenarios. Within the body of operations, bindings can be employed ({binding_name}), which implies they will take values along the generated sequence as the cycle number unfolds; similarly, template parameters can be used (<<parameter_name:default>>, or TEMPLATE(parameter_name,default)) to be replaced according to settings passed through the command-line or when defining the scenario composition in the "scenarios" section.

Try to track what happens when you invoke the cql-keyvalue2 astra workload/scenario (Note: this focuses on the version 5 syntax):

1. the schema phase is invoked: it consists of running, with the cql driver, the blocks whose naming matches the selector tags==block:schema. Also some parameters are passed to the execution. (Note that the schema phase differs between an Astra DB target and a regular Cassandra target, while later phases do not.)
2. When schema is over, the rampup is started. This runs all blocks matching tags==block:rampup, and passes the parameters threads and cycles. The latter will preferrably read the value passed through command-line (look for rampup-cycles=... in the benchmark command you ran earlier), with a default.
3. Running rampup then entails running the block with the tag block: rampup (the tag is automatically attached to the block from its name). In the block, a consistency level is defined, again using template parameters. The statement (or operation) that is repeatedly executed is an INSERT statement, which writes rows to a DB table:
4. The INSERT targets a table in a keyspace whose name (as well as the table name itself) can be provided through a command-line parameter (compare with the keyspace=... part of the benchmark command you ran earlier). The default keyspace is baselines.
5. In this INSERT statement, also the bindings seq_key and seq_value occur: that is, you can imagine a loop with an index i looping over rampup-cycles integer values: each time the INSERT statement is executed, "key" and "value" will be equal to the corresponding mapping functions "evaluated at i".
6. When these INSERT statements are all executed, the rampup phase will be done and the execution will turn to the main phase. The tag filtering expression (tags==block:"main.*") this time matches two operations. They will be combined in an alternating fashion, according to their ratio (see here for details), with the final result, in this case, of reproducing a mixed read-write workload.
7. The scenario will then have completed.

Note: The yaml file corresponding to this workload (embedded in NoSQLBench) is an egregious example of the improvements in the workload syntax offered in version 5. You can inspect the yaml files shipping with nb4 and nb5 (both provided in this repo for your convenience) and observe the differences:

• One can do without "tags" and, since lists are generally replaced by key-value (ordered) maps, use the keys themselves to refer to blocks of operations in defining phases;

• Operation-specific parameters (such as ratio) are next to the statement body, and in general information logically pertaining to the same item stays packed together;
• The map-based syntax allows for a way more synthetic workload definition yaml.

Show me the differences

Play with workloads

A good way to understand workload construction is to start from simple ones.

To run the following examples please go to the appropriate subdirectory:

cd workloads

Note: The two example yaml files that follow are also provided in the "nb4 list-based syntax" (as *-nb4.yaml files). Just keep in mind that modern nb5 can run any format, while the modern map-based syntax is not guaranteed to be parseable by NoSQLBench 4.

Example 1: talking about food

Take a look at simple-workload.yaml. This defines two operations (formerly called "statements") that are to be interleaved according to their ratio parameter. As you saw earlier, there are parts of the operation body that depend on the cycle number as specified by the binding functions defined under bindings.

Run the workload with:

nb5 run driver=stdout workload=simple-workload cycles=12

The driver here is stdout, so each operation will simply print its body to screen. As you can see, these are not valid CQL statements, indeed one would get errors if invoking with drivers=cql. This is an important point: the operation body itself can take any form, since its actual syntactical validation will happen at driver level. NoSQLBench offers several drivers, and it is important to match statement bodies with the proper drivers (stdout in particular can take basically anything as input).

Bindings are defined in a functional way, by providing a sequence of functions that will be composed, left-to-right. It is implied that the input to the first such function is the cycle number, so for instance the binding:

 multiplier: AddHashRange(50); Mod(20); Clamp(2,20); NumberNameToString()

means: the multiplier binding will be a function that takes the cycle number as input, adds a pseudorandom 0-50 value to it, takes the modulo-20 of the result, adjusts it so that it lies within the [2, 20] interval, and finally produces the spelled-out name in words of the resulting number.

For a general introduction to bindings and a list of the (many) available functions, please see here. You can also find several valueble example "recipes" in the /nbr/src/main/resources/examples directory of NoSQLBench source code.

Example 2: animal meeting

The previous example, albeit silly, was meant to show the basics of building workloads. An important feature is the possibility to package several workflows into a single sequence that can then be run at once ("named scenarios").

The need to perform a schema initialization and to execute a rampup phase before doing the actual main benchmarking is, as discussed earlier, almost universal: named scenarios have been designed with that need in mind.

Try to run the following example scenario:

nb5 workload-with-phases default driver=stdout

You see that here several clearly distinct phases take place. If you inspect the structure of workload-with-phases.yaml, you can see that the file begins by defining the default scenario as a sequence of run invocations, each involving certain statement blocks selected through tags.

This yaml file also shows usage of template parameters: with (interchangeable) expressions such as TEMPLATE(rampup-cycles,5) and <<act_ratio:1>> NoSQLBench can use a command-line-provided parameter (with a default value) inside the workload specs. Try to re-run the workload adding rampup-cycles=10 act_ratio=11 to see the difference.

Note: keep in mind that act_ratio is constrained in its values so that the sum of the ratio parameters for the operations in main divides exactly the number of cycles in that phase. This sum constitutes the ("stride").

Homework assignment

The "Lab" part of the homework, which requires you to finalize a workload yaml and make it work according to specifications, is detailed on this page.

To submit your homework, please use this form.