Time: 2 hours. Difficulty: Intermediate. Start Building!
The goal of this workshop is to get you familiar with the powerful and versatile
tool NoSQLBench
. With that, you can perform
industry-grade, robust benchmarks
aimed at several (distributed) target systems, especially NoSQL databases.
Today you'll be benchmarking Astra DB, a database-as-a-service built on top of Apache Cassandra. Along the way, you will learn the basics of NoSQLBench.
In this repository you will find all material and references you need:
- NoSQLBench Discord
- NoSQLBench homepage
- Exercises
- Step-by-step guide
- DataStaxDevs Discord server to keep in touch with us
- Our Q&A forum (think StackOverflow for Cassandra and all things DataStax)
- Slide deck
- Before you start
- Create your Astra DB instance
- Launch Gitpod and setup NoSQLBench
- Run benchmarks
- Workloads
- Homework assignment
Heads up: these instructions are available in two forms: a short and to-the-point one, with just the useful commands if you are watching us live; and a longer one (this one), with lots of explanations and details, designed for those who follow this workshop at their own pace. Please choose what best suits you!
If you are not watching us live, please look at the provided presentation material for more coverage of the theory part. There, you will learn more about:
- benchmarking a database;
- Apache Cassandra™, a distributed NoSQL highly available database;
- Astra DB, a database-as-a-service in the cloud, built on Cassandra, ready to use in minutes for free with a few clicks;
- the main features and general principles behind the NoSQLBench benchmarking tool.
- What are the prerequisites?
This workshop is aimed at data architects, solution architects, developers, or anybody who wants to get serious about measuring the performance of their data-intensive system. You should know what a (distributed) database is, and have a general understanding of the challenges of communicating over a network.
- Do I need to install a database or anything on my machine?
No, no need to install anything. You will do everything in the browser. (That being said, the knowledge you gain today will probably be best put to use once you install NoSQLBench on some client machine to run tests.)
You can also choose to work on your machine instead of using Gitpod: there's no problem with that, just a few setup and operational changes to keep in mind. We will not provide live support in this case, though, assuming you know what you are doing.
- Is there anything to pay?
No. All materials, services and software used in this workshop is free.
- Do you cover NoSQLBench 4 or 5?
Ah, I see you are a connoisseur. We focus on the newly-release NoSQLBench 5, but we provide tips and remarks aimed at those still using nb4. See here for a bit of context.
To complete the workshop and get a verified "NoSQLBench" badge, follow these instructions:
- Do the hands-on practice, either during the workshop or by following the instructions in this README;
- (optional) Complete the "Lab" assignment as detailed here;
- Fill the submission form here. Answer the theory questions and (optionally) provide a screenshot of the completed "Lab" part;
- give us a few days to process your submission: you should receive your well-earned badge in your email inbox!
First you must create a database: an instance of Astra DB, which you will then benchmark with NoSQLBench.
Don't worry, you will create it within the "Free Tier", which offers quite a generous free allowance in terms of monthly I/O (about 40M operations per month) and storage (80 GB).
You need to:
- create an Astra DB instance as explained here, with database name =
workshops
and keyspace name =nbkeyspace
; - (this will happen automatically with the previous one) generate and retrieve a DB Token as explained here. Important: use the role "DB Administrator" if manually creating the token.
- generate and download a Secure Connect Bundle as explained here;
Moreover, keep the Astra DB dashboard open: it will be useful later. In particular, locate the Health tab and the CQL Console.
Gitpod is an IDE in the cloud (modeled after VSCode). It comes with a full "virtual machine" (actually a Kubernetes-managed container), which you will use as if it were your own computer (e.g. downloading files, executing programs and scripts, even launching containers from within it).
Now ctrl-click on the Gitpod button below. It will:
- spawn your own Gitpod container;
- clone this repository in it;
- preinstall the required dependencies.
Note that you may have to authenticate through Github in the process. (Also make sure you "Open in new tab" to keep this readme open alongside.)
In a few minutes, a full IDE will be ready in your browser, with a file
explorer on the left, a file editor on the top, and a console (bash
) below it.
There are many more other features, probably familiar to those who have experience with VSCode. Feel free to play around a bit!
Download the latest stable version of NoSQLBench to your Gitpod environment.
First make sure you are in the top-level repository directory with your
Gitpod console: type pwd
and see if the output is /workspace/workshop-nosqlbench
.
Please download the "Linux binary" distribution of NoSQLBench: as instructed here, you can obtain the latest stable binary (it's a few hundred megabytes) with
curl -L -O https://github.com/nosqlbench/nosqlbench/releases/latest/download/nb5
and, when the download is finished, make it executable and move it, out of convenience, to a directory which is part of the search path:
chmod +x nb5
sudo mv nb5 /usr/local/bin/
Ok, now check that the program starts: invoking
nb5 --version
should output the program version (something like 5.17.3
or higher).
You will probably see a message like
Picked up JAVA_TOOL_OPTIONS ...
when launchingnb5
. You can ignore it: it is a consequence of some settings by Gitpod and does not have to do with NoSQLBench itself.
Note that if your Gitpod instance gets hibernated (which happens after some inactive time) and you restart it later, recall that only the contents of
/workspace
will be restored: hence, you would have to repeat the installation.
This workshop is built for the newly-released NoSQLBench 5. Compared to version 4, there are many differences both under the hood and for the user. We will provide tips and remarks to help users of version 4 as well throughout the following.
Tell me more
NoSQLBench 5 comes with a completely redesigned architecture for drivers, which grants all drivers (present and future) a set of nice features for free and makes it easier to develop new ones.
On a more user-facing side, among several improvements,
the error handling logic has been enhanced; the workload
yaml
syntax has been streamlined; new and upgraded drivers are available
for various databases.
Tip: in the NoSQLBench repo itself, you can find plenty of information on the specifications for the workload
yaml
syntax.
Great care has been taken to maintain backward compatibility (e.g. in the workload yaml syntax); in the following, some of the most relevant changes will be highlighted, especially concerning the workload syntax, to support users of version 4 as well.
NoSQLBench will need to connect to your Astra DB database: to do so, the Secure Connect Bundle zip file must be uploaded to your Gitpod environment.
Locate, with the file explorer on your computer, the bundle file that
you downloaded earlier (it should be called
secure-connect-workshops.zip
)
and simply drag-and-drop it to the file navigator panel
("Explorer") on the left of the Gitpod view.
Once you drop it you will see it listed in the file explorer itself. As a check, you can issue the command
ls /workspace/workshop-nosqlbench/secure*zip -lh
so that you get the absolute path to your bundle file (and also verify that it is
the correct size, about 12-13 KB).
As an aside, note that, as per best practices with NoSQLBench, (relative)
paths to files would start with ./
.
If you want NoSQLBench to be able to access the Astra DB instance, you have to pass it
the required connection parameters and secrets. To do so, set up
a .env
file, which will make your life easier later.
Copy the provided template file to a new one and open it in the Gitpod file editor:
cp .env.sample .env
gp open .env
# (you can also simply locate the file
# in the Explorer and click on it)
Insert the "Client ID" and "Client Secret" of the DB Token you created earlier and, if necessary, adjust the other variables.
Now, source this file to make the definitions therein available to this shell:
. .env
To check that the file has been sourced, you can try with:
echo ${ASTRA_DB_KEYSPACE_NAME}
and make sure the output is not an empty line.
(Note that you will have to source the file in any new shell you plan to use).
Everything is set to start running the tool.
From a user's perspective, NoSQLBench sends operations (e.g. reads, writes) to the target system (e.g. a database) and record how the latter responds (service time, number of ops/second and failures).
Try launching this very short "dry-run benchmark", that instead of actually
reaching the database simply prints a series of CQL statements to the console
(as specified by the driver=stdout
parameter):
nb5 cql-keyvalue2 astra \
driver=stdout \
rampup-cycles=10 \
main-cycles=10 \
keyspace=${ASTRA_DB_KEYSPACE_NAME}
You will see 21 (fully-formed, valid CQL) statements being printed:
one CREATE TABLE
, then ten INSERT
s
and then another ten between SELECT
s and further INSERT
s.
Note: we will use workload
cql-keyvalue2
throughout. This is functionally identical to thecql-keyvalue
workload but is expressed in the newer syntax foryaml
workloads, which comes handy when later dissecting its content. If you are working with NoSQLBench 4, remember to drop the trailing2
from the workload name in the following!
These three blocks correspond
to the three phases defined in the cql-keyvalue2
workload: schema,
rampup and main. The most relevant benchmark results are from the
main phase, which mimics the kind of load the database would be subject to
when operating normally.
For most workloads, a three-phase structure is the most reasonable choice: first the schema is set up, then an amount of data is poured in, and finally the target system is subject to a mixture of reads and writes. The third and last phase (main) in most cases is designed to generate contention on the server, which is the very situation one usually wants to put to test.
Now re-launch the above dry run and look for differences in the output:
nb5 cql-keyvalue2 astra \
driver=stdout \
rampup-cycles=10 \
main-cycles=10 \
keyspace=${ASTRA_DB_KEYSPACE_NAME}
do you notice that the output of this second run is identical to the first,
down to the actual values used in the INSERT
s and DELETE
s?
Indeed an important goal for any benchmark is its reproducibility:
here, among other
consequences, this means that the operations (their progression and contents alike)
should be completely determined with no trace of actual randomness.
You can also peek at the logs
directory now: it is created automatically and
populated with some information from the benchmark at each execution of nb
.
It is now time to start hitting the database!
What you launched earlier is the cql-keyvalue2
workload, one of the several
ready-to-use workloads included with NoSQLBench.
In particular, you ran the astra
scenario, which determines a particular
way the workload is to be unfolded and executed.
This time you will run with driver=cql
to actually reach the database:
for that to work, you will provide all connection parameters set up earlier.
Note: for this workload, as you could check by examining its definition (see the Inspect "cql-keyvalue" section below), you could leave the driver specification out since
cql
is the default.
The next run will ask NoSQLBench to perform a substantial amount of operations, in order to collect enough statistical support for the results.
On the other hand, you will rate-limit the operations sent to the database (with the
cyclerate
parameter). This is mainly because often the goal of a benchmark is to verify whether the system can withstand a known, controlled workload; but in this case there are also other practical reasons: (1) you don't want to consume too much of your Free Tier monthly allowance in this "functional demo"; moreover, (2) if you are running on the Free Tier you may otherwise hit the guardrails and the performance limitations specific to free-tier Astra databases.
Here is the full command to launch:
nb5 cql-keyvalue2 \
astra \
username=${ASTRA_DB_CLIENT_ID} \
password=${ASTRA_DB_CLIENT_SECRET} \
secureconnectbundle=${ASTRA_DB_BUNDLE_PATH} \
keyspace=${ASTRA_DB_KEYSPACE_NAME} \
cyclerate=50 \
driver=cql \
main-cycles=9000 \
rampup-cycles=9000 \
errors='OverloadedException:warn' \
--progress console:5s \
--log-histograms 'histogram_hdr_data.log:.*.main.result.*:20s' \
--log-histostats 'hdrstats.log:.*.main.result.*:20s'
Show me the command breakdown
Note that some of the parameters (e.g. keyspace
) are workload-specific.
command | meaning |
---|---|
cql-keyvalue2 |
workload |
astra |
scenario |
username |
authentication |
password |
authentication |
secureconnectbundle |
Astra DB connection parameters |
keyspace |
target keyspace |
cyclerate |
rate-limiting (cycles per second) |
driver=cql |
driver to use (CQL, for AstraDB/Cassandra) |
main-cycles |
how many operations in the "main" phase |
rampup-cycles |
how many operations in the "rampup" phase |
errors |
behaviour if errors occur during benchmarking |
--progress console |
frequency of console prints |
--log-histograms |
write data to HDR file (see later) |
--log-histostats |
write some basic stats to a file (see later) |
This way of invoking nb
, the "named scenario"
way, is not the only one: it is also possible to have a finer-grained control over what activities should
run with a full-fledged CLI scripting syntax.
Note: the syntax of the
errors
parameter has been improved in NoSQLBench 5 to allow for a finer control (with multiple directives, such aserrors='NoNodeAvailable.*:ignore;InvalidQueryException.*:counter;OverloadedException:warn'
). On version 4 you should revert to a simpler parameter, such aserrors=count
, instead of the above.
Note: if you were targeting a "regular" Cassandra instance (as opposed to Astra DB), the command line above would change a little: the scenario would be
default
, you would need to provide a parameter such ashosts=192.168.1.1,192.168.1.2
, and you would have additionally to provide a parameterlocaldc=datacenter1
(change to reflect the name of one of your Cassandra datacenters). The latter is required starting from NoSQLBench 5, but is a good practice with earlier versions too. Further, there would be nosecureconnectbundle
parameter. As for username and password, ... well, that depends on how the Cassandra cluster is configured. As far as benchmark results are concerned, in the next steps you will have to look at metric with "default" in their name instead of "astra". Note that NoSQLBench uses the very same CQL drivers to access the database, regardless of whether Astra DB or a Cassandra cluster.
The above command should run for approximately six minutes, during which NoSQLBench
sends a constant stream of I/O operations to the database and collects timing
information on how it responds. You will see a console output keeping you
updated on the progress of the
currently-running phase (schema
/rampup
/main
).
If the execution fails with,
"Cannot construct cloud config from the cloudConfigUrl ..."
, chances are your (free-tier) database is currently in Standby status. To resume it, open its Health tab or the CQL Console tab in the Astra DB UI and wait two or three minutes before retrying.
While this runs, have a look around.
By this point, you may have a question: "but what is being written to the database, exactly?
To find out, connect to the database and inspect the contents of the keyspace. There are several ways one could connect to Astra DB: using Cassandra drivers with most programming languages; through the HTTP requests enabled by the Stargate Data API; or directly with a client that "speaks" the CQL language. Today you will make use of the CQL Console available in the browser within the Astra UI.
Choose your database in the Astra main dashboard and click on it; next, go to the "CQL Console" tab in the main panel. In a few seconds the console will open in your browser, already connected to your database and waiting for your input.
Commands entered in the CQL Console are terminated with a semicolon (
;
) and can span multiple lines. Run them with theEnter
key. If you want to interrupt the command you are entering, hitCtrl-C
to be brought back to the prompt. See here for more references to the CQL language commands.
Start by telling the console that you will be using the nbkeyspace
keyspace:
USE nbkeyspace;
Check what tables have been created by NoSQLBench in this keyspace:
DESC TABLES;
You should see table keyvalue
listed as the sole output.
Look at a a few lines from this table:
SELECT * FROM keyvalue LIMIT 20;
Ok, mystery solved. It looks like the table contains simple key-value pairs, with two columns seemingly of numeric type. Check with:
DESC TABLE keyvalue;
Surprise! As a matter of fact both columns are of type TEXT
(that is,
variable-length strings). Indeed, most key-value stores admit string keys
and (unless specific schemas are enforced) the values are also either binary blobs
or ASCII sequences: this workload could then be adapted to
benchmarking other key-value databases.
By now, the benchmark is probably still running. That's good: you get a chance to inspect the "database health" while NoSQLBench is writing to it. Remember you instructed the tool to work at a steady rate in terms of operations per second.
Locate your database in the Astra main dashboard and click on it; next, go to the "Health" tab in the main panel. You will see what essentially is a Grafana dashboard, with a handful of plots being displayed within the tab - all related to how the database is performing in terms of reads and writes.
Tip: you can customize the width of the time window in the graphs and the update frequency. Today it could make sense to set these to 15 minutes and 5 seconds respectively.
Take a look at the plot labeled "Requests Combined": this describes the amount
of write and read requests per second received by the database. You will note
that the total fluctuates around the value provided with the cyclerate
parameter during the whole test: but during rampup it will be all writes,
while the main phase will be an equal mixture of reads and writes.
Now turn your attention to the "Write Latency"/"Read Latency" plots. These provide quantities related to the "latency", as experienced by the coordinator node, involved in servicing write/read requests. In particular, some reference percentiles are reported as they vary in time: indeed, when describing the performance of the target system, percentiles are a much better tool than averages or maximum values. Note down the values you read from the plot in the middle of the main phase.
Show me "sample values" one could read from the graph
Below is a real-life example of the values that could result from a cql-keyvalue2
benchmark session in the main phase:
Percentile | Write Latency | Read Latency |
---|---|---|
P50 | 709 µs | 935 µs |
P75 | 831 µs | 1.31 ms |
P90 | 904 µs | 1.53 ms |
P95 | 1.04 ms | 1.77 ms |
P99 | 2.45 ms | 15.6 ms |
What is the meaning of a percentile? Well, if one says "P75 for reading is 13 milliseconds," that means "75% of the read requests are serviced within 13 ms from being received by the server." This kind of metric, in most cases, is more meaningful than quoting the maximum value recorded (for one, the max tends to fluctuate way more).
Quoting high percentiles such as P95, P99 or similar is a standard practice when expressing the performance of a data system.
By now the benchmark should have finished. If it hasn't yet, give it time to complete: you will see a summary being printed, after which you will get the console prompt back.
Each run of NoSQLBench generates timestamped files in the logs
directory (which is created automatically if not present). The on-screen
summary is a shortened form of the data found in the summary files there.
Now open the most recent *.summary
file in that directory, corresponding
to the nb
invocation that just completed.
Find the metric called cqlkeyvalue2_astra_main.result-success
: this
corresponds to events of the type "a command was issued to the database during
the main phase and succeeded (and the time needed for it to complete, as seen from the client
side, was recorded)".
The related metric
cqlkeyvalue2_astra_main.result
is similar, but comprises all events regardless of whether they succeeded or not. In "healthy" cases, these two are identical or at most very close to each other. This difference is sometimes characterized as the "goodput/throughput" dichotomy.
Under that metric title, you will see something similar to:
cqlkeyvalue2_astra_main.result-success
count = 15000
mean rate = 50.00 calls/second
1-minute rate = 49.94 calls/second
5-minute rate = 50.29 calls/second
15-minute rate = 50.57 calls/second
This tells you the total count and the per-minute rate (as averaged over different time windows) for this operation.
NoSQLBench keeps track of count/timing information for many operations, which range from substitution of variables in the workload commands to actual sending of an I/O operation to the database.
You instructed NoSQLBench to output timing information in two other formats:
First, with the --log-histostats
option you get a file with
a listing, for selected metrics, of the min, max and some percentiles as
measured over a desired time window. Try to look at the file hdrstats.log
in the Gitpod editor and figure out its structure.
If you want to graphically look at the data thus collected, we provide a sample
script for plotting. Simply enter the command (try -h
for more options)
./hdr_tool/histostats_quick_plotter.py \
hdrstats.log \
-m cqlkeyvalue2_astra_main.result-success
and then open, in the Gitpod editor, the hdrstats.png
image just created.
The version of the plotter script included in this repo is for educational purposes only: for general use, please head to the official release page.
Tip: you can use Ctrl-mousewheel to zoom on the image in Gitpod.
Look at the values of, say, the P90 percentile: it will be larger than both the read and write corresponding percentiles given in the Astra DB "health" tab. That's because this time you are seeing things from the vantage point of the testing client, and the (Gitpod-to-Astra) communication over the network is included in the "service time".
Besides, the comparison is made somewhat murkier by the fact that NoSQLBench measures "cycles", which means reads and writes in the main phase; while the Astra health dashboard keeps these two separate. A way to make this comparison more apples-to-apples would be to "instrument" the metric collection: adding the
instrument=true
parameter to the invocation will attach a separate timer to each of the named metric in the workload definition, and each of these metrics will be featured e.g. in the summary file or anywhere for further inspection.
Second, the --log-histograms
option has the effect that NoSQLBench writes
the whole of the measurements it takes in the HDR file format.
This is an optimized way to save the full histogram of the measured service time
using exact counts over very thin fixed-width bins. Try to inspect the contents
of file histogram_hdr_data.log
: you will notice the data is stored in ASCII-encoded long strings.
Fortunately for you, we provide a sophisticated plotting tool for this file, which may serve as inspiration or be used directly for your own benchmarks.
Try running the command (run with -h
to see more options)
./hdr_tool/hdr_tool.py \
histogram_hdr_data.log \
-b -c -s \
-p SampleData \
-m cqlkeyvalue2_astra_main.result-success
and look at the SampleData*.png
plots that are generated:
- the "base plot" is the full collected histogram of the chosen metric;
- the "stability plot" plots measurements from each time window separately, to help assessing whether the main phase is really a steady state;
- the "percentile plot" reformulates the histogram in terms of percentiles.
The version of the plotter script included in this repo is for educational purposes only: for general use, please head to the official release page.
Again, the timings are larger than those found on the Astra health tab (i.e. on server-side): these measurements are reported "as seen by the testing client".
Tip: this script is just a demonstration of the versatility of the HDR format: you can invent your own post-analysis tool and attach it to the data collected by NoSQLBench. Look at the (Python) source code of the tool for inspiration! (Or simply take the tool and use it on your data.)
On top of everything you have seen, it is possible to have NoSQLBench start a Grafana dashboard locally alongside the benchmark itself, powered behind the scenes by Prometheus-based real-time metric collection.
All it takes is the additional --docker-metrics
option to the command line,
and Docker must be available on your system (Note: Gitpod comes with Docker preinstalled).
You can try launching another benchmark as follows (note the last option
and the fact that this time we dropped the driver
parameter):
nb5 cql-keyvalue2 \
astra \
username=${ASTRA_DB_CLIENT_ID} \
password=${ASTRA_DB_CLIENT_SECRET} \
secureconnectbundle=${ASTRA_DB_BUNDLE_PATH} \
keyspace=${ASTRA_DB_KEYSPACE_NAME} \
cyclerate=50 \
rampup-cycles=15000 \
main-cycles=15000 \
errors='OverloadedException:warn' \
--progress console:5s \
--docker-metrics
This command might take a few additional seconds to start the first time,
as Docker images are being downloaded and the containers are started.
Then, successful start of the Grafana dashboard
should be logged, at which point the usual cql-keyvalue2
workload will start.
Note: on some versions of NoSQLBench 5 you may see an error such as
Java HttpClient was not authorized, [...] :WWW-Authenticate header missing for response code 401
. This is due to the Grafana server not complying with the HTTP 401 specifications, and can be fixed by running the script./_grafana_docker_key_fetcher.sh
and then re-starting the abovenb5
command. (the script manually generates an API key for interacting with Grafana, storing it where thennb5
expects to find it.)
It is possible that Gitpod will detect new services running on some local ports and automatically open the Grafana dashboard in its mini-browser. Better to ignore it and re-open the URL in a new, actual browser tab as instructed below.
The Grafana container is running and exposed on local port 3000. Luckily,
Gitpod is kind enough to map local ports to externally-accessible addresses,
such as https://3000-datastaxdevs-workshopnos-nq5y2jf5uw9.ws-eu38.gitpod.io
.
Your URL will be different: to construct it, simply prepend a 3000-
to the URL of the Gitpod IDE. Then, open this address in a new tab.
The default credentials to log in to Grafana are ... admin/admin
. Once you're
in, don't bother to reset your password (click "Skip"). You'll get to the Grafana
landing page. Find the "Dashboards" icon in the leftmost menu bar and pick the
"Manage" menu item: finally, click on the "NB4 Dashboard" item you should see
listed there. Congratulations, you are seeing the data coming from NoSQLBench.
You may find it convenient to set the update frequency to something like 10 seconds and the displayed time window to 5 minutes or so (upper-right controls).
The dashboard comprises several (interactive) plots, updated in real time. Let's see some of them and their significance.
- "Ops and Successful Ops". One-minute averages of the total operations dispatched per second by NoSQLBench. This will match the reported per-second averages found in the
*.summary
file. There are separate curves for each phase, so you will clearly see rampup leaving the room to main at some point during the test. - "p75/p99 client overhead". This is related to timing of operations internal to NoSQLBench (note the scale, barely a few micro-seconds). You should keep an eye on those to check that the benchmarking client is not experiencing bottlenecks of sorts.
- "service time distribution/range". Here you can see percentiles for "service time" relative to the various phases, a measurement corresponding to the "success" metric introduced earlier. These quantities then directly relate to the HDR histogram data analyzed above and should match them. Note that the "min" and "max" are denoted "p0" and "p100" here (indeed this is what they are, by definition).
If you want to look at the "raw" source of data feeding the Grafana plots,
head to the Prometheus UI, exposed at local port 9090. If you are on Gitpod,
similarly to what you did for Grafana earlier,
you get there by prepend a 9090-
to the domain name you are using your IDE at. Then, open this address
in a new tab.
Click on the "world" icon next to the "Execute" button in the search bar:
in the dialog that appears you can look for specific metrics.
Try to look for result_success
and confirm, then click "Execute".
The search results are many datapoints, pertaining to several metrics related to the service time. Still, those are for different phases and, most important, different measurements (percentiles, averages, counts), which does not really make sense to inspect together.
Tip: switch to the "Graph" view for a more immediate visualization. The graphs display "raw" data, hence are in units of nanoseconds.
Fortunately, each datapoint is fully equipped with detailed metadata, which allow for sophisticated filtering. Further, you can use the flexible, powerful query language used by Prometheus to even perform aggregations on top of the data queries.
Just to pique your interest, try pasting these examples and click "Execute":
# filtering by metadata
{__name__="result_success", type="pctile", alias=~".*main.*"}
# aggregation
avg_over_time({__name__="result_success", type="pctile", alias=~".*main.*"}[10m])
# another aggregation, + filtering
max_over_time({__name__="result_success", type="pctile", alias=~".*main.*"}[10m])
Also, note that Prometheus exposes a REST access, so you can imagine, for instance, a running scripts that tracks the metrics and responds to them in real time.
We mentioned that cql-keyvalue2
is one of several ready-to-use workloads,
but also that you can easily build your own: for example, to test a specific
data model or ensure you closely mimic your application's request pattern.
So, now it's time for a closer inspection of workloads. Each workload is built as
a yaml
file, defining its scenarios and the sequences of phases each scenario
is made of.
Phases, in turn, consist of several statements that the driver will know how to "execute" (i.e. send to the target system as operations to execute).
There is quite some freedom in creating workloads: in the following you will just explore some of this space. Look into the reference documentation for more.
Tip: feel free to interrupt the previous benchmark, if it still runs, with Ctrl-C. You won't need it anymore.
Ask NoSQLBench to dump to a file the yaml
defining the workload
you just ran:
nb5 --copy cql-keyvalue2
(you can also get a comprehensive list of all available workloads with
nb5 --list-workloads
, by the way, and a more fine-grained output with
nb5 --list-scenarios
.)
A file cql-keyvalue2.yaml
is created in the working directory.
You can open it (clicking on it in the Gitpod explorer or by running
gp open cql-keyvalue2.yaml
).
This is not a line-by-line analysis, but broadly speaking there are three important sections:
- scenarios. This defines which phases constitute each scenario. More specifically, each phase is expressed as an inline script, which in turn follows the NoSQLBench CLI scripting syntax.
- bindings. Each binding defines a particular recipe to generate a sequence of pseudo-random values. Emphasis is on the "pseudo", since bindings are (usually) fully deterministic, reproducible, mappings of cycle numbers to values. The sequence of "keys" produced as the the rampup phase of key-value workload unfolds, for instance, is defined here by the
seq_key
binding. Bindings are defined using a functional compositional approach, starting from a set of available building blocks, and can generate values of many different data types. - blocks. Here, groups of "statements" (better: operations) are defined for use within scenarios. Within the body of operations, bindings can be employed (
{binding_name}
), which implies they will take values along the generated sequence as the cycle number unfolds; similarly, template parameters can be used (<<parameter_name:default>>
, orTEMPLATE(parameter_name,default)
) to be replaced according to settings passed through the command-line or when defining the scenario composition in the "scenarios" section.
Try to track what happens when you invoke the cql-keyvalue2 astra
workload/scenario (Note: this focuses on the version 5 syntax):
- the schema phase is invoked: it consists of running, with the
cql
driver, the blocks whose naming matches the selectortags==block:schema
. Also some parameters are passed to the execution. (Note that the schema phase differs between an Astra DB target and a regular Cassandra target, while later phases do not.) - When schema is over, the rampup is started. This runs all blocks matching
tags==block:rampup
, and passes the parametersthreads
andcycles
. The latter will preferrably read the value passed through command-line (look forrampup-cycles=...
in the benchmark command you ran earlier), with a default. - Running rampup then entails running the block with the tag
block: rampup
(the tag is automatically attached to the block from its name). In the block, a consistency level is defined, again using template parameters. The statement (or operation) that is repeatedly executed is anINSERT
statement, which writes rows to a DB table: - The
INSERT
targets a table in a keyspace whose name (as well as the table name itself) can be provided through a command-line parameter (compare with thekeyspace=...
part of the benchmark command you ran earlier). The default keyspace isbaselines
. - In this
INSERT
statement, also the bindingsseq_key
andseq_value
occur: that is, you can imagine a loop with an indexi
looping overrampup-cycles
integer values: each time theINSERT
statement is executed, "key" and "value" will be equal to the corresponding mapping functions "evaluated ati
". - When these
INSERT
statements are all executed, the rampup phase will be done and the execution will turn to the main phase. The tag filtering expression (tags==block:"main.*"
) this time matches two operations. They will be combined in an alternating fashion, according to theirratio
(see here for details), with the final result, in this case, of reproducing a mixed read-write workload. - The scenario will then have completed.
Note: The yaml file corresponding to this workload (embedded in NoSQLBench) is an egregious example of the improvements in the workload syntax offered in version 5. You can inspect the yaml files shipping with nb4 and nb5 (both provided in this repo for your convenience) and observe the differences:
- One can do without "tags" and, since lists are generally replaced by key-value (ordered) maps, use the keys themselves to refer to blocks of operations in defining phases;
- Operation-specific parameters (such as
ratio
) are next to the statement body, and in general information logically pertaining to the same item stays packed together; - The map-based syntax allows for a way more synthetic workload definition yaml.
A good way to understand workload construction is to start from simple ones.
To run the following examples please go to the appropriate subdirectory:
cd workloads
Note: The two example yaml files that follow are also provided
in the "nb4 list-based syntax" (as *-nb4.yaml
files).
Just keep in mind that modern nb5
can run any format, while
the modern map-based syntax is not guaranteed to be
parseable by NoSQLBench 4.
Take a look at simple-workload.yaml
. This defines two
operations
(formerly called "statements") that are to be interleaved according
to their
ratio
parameter. As you saw earlier, there are parts of the
operation body that depend on the cycle number
as specified by the binding functions defined under bindings
.
Run the workload with:
nb5 run driver=stdout workload=simple-workload cycles=12
The driver here is stdout
, so each operation will simply print its body
to screen. As you can see, these are not valid CQL statements, indeed one would
get errors if invoking with drivers=cql
. This is an important point: the
operation body itself can take any form, since its actual syntactical validation
will happen at driver level. NoSQLBench offers several drivers, and it is
important to match statement bodies with the proper drivers (stdout
in
particular can take basically anything as input).
Bindings are defined in a functional way, by providing a sequence of functions that will be composed, left-to-right. It is implied that the input to the first such function is the cycle number, so for instance the binding:
multiplier: AddHashRange(50); Mod(20); Clamp(2,20); NumberNameToString()
means: the multiplier
binding will be a function that takes the cycle number as input,
adds a pseudorandom 0-50 value
to it, takes the modulo-20 of the result,
adjusts it
so that it lies within the [2, 20] interval, and finally
produces the spelled-out name in words
of the resulting number.
For a general introduction to bindings and a list of the (many) available
functions, please see here. You can also find several valueble example "recipes"
in the /nbr/src/main/resources/examples
directory of NoSQLBench source code.
The previous example, albeit silly, was meant to show the basics of building workloads. An important feature is the possibility to package several workflows into a single sequence that can then be run at once ("named scenarios").
The need to perform a schema initialization and to execute a rampup phase before doing the actual main benchmarking is, as discussed earlier, almost universal: named scenarios have been designed with that need in mind.
Try to run the following example scenario:
nb5 workload-with-phases default driver=stdout
You see that here several clearly distinct phases take place.
If you inspect the structure of workload-with-phases.yaml
,
you can see that the file begins by defining the default
scenario
as a sequence of run
invocations, each involving certain statement blocks
selected through tags.
This yaml
file also shows usage of template parameters:
with (interchangeable) expressions such as TEMPLATE(rampup-cycles,5)
and <<act_ratio:1>>
NoSQLBench can use a command-line-provided parameter (with a default value)
inside the workload specs. Try to re-run the workload adding
rampup-cycles=10 act_ratio=11
to see the difference.
Note: keep in mind that
act_ratio
is constrained in its values so that the sum of theratio
parameters for the operations inmain
divides exactly the number of cycles in that phase. This sum constitutes the ("stride").
The "Lab" part of the homework, which requires you to finalize
a workload yaml
and make it work according to specifications,
is detailed on this page.
To submit your homework, please use this form.