This repo contains a tool for benchmarking Weaviate performance.
- 📊 results and context can be found in the Weaviate documentation
- 💬 discuss the results on our Slack channel or Twitter
There are two components you will need to run for the benchmarks:
weaviate
the standard Weaviate imagebenchmarker
a go based benchmarking tool
You can run both as containers on the same machine via Docker compose.
For replicating our benchmarks we recommend setting up two separate machines.
Machine description | CPU type | CPUs | Memory | Disk size | Disk type | Misc. |
---|---|---|---|---|---|---|
Machine to run Weaviate | c2 | 30 | 120GB | 500GB | SSD | Ubuntu 22.04 with Docker-compose |
Machine to run benchmark script | N2 | 8 | 64GB | 500GB | SSD | Ubuntu 22.04 with Docker-compose |
Clone this repo and cd into it $ git clone https://github.com/semi-technologies/weaviate-benchmarking && cd weaviate-benchmarking
Run the following command to spin up Weaviate: $ docker-compose up weaviate -d
Copy the IP address and amount of CPU cores this machine has.
Check if the Weaviate machine is available: curl http://{IP OF WEAVIATE INSTANCE}/v1/meta
. Note that the instance runs on port 8080
, e.g., http://10.128.15.12:8080/v1/meta
.
You will also need to allow port 50051
for gRPC and 21121
for metrics. You can verify this via nc -zv localhost 50051
.
Clone this repo and cd into it $ git clone https://github.com/semi-technologies/weaviate-benchmarking && cd weaviate-benchmarking
Download the files into a datasets folder as outlined below.
mkdir datasets && \
curl -o ./datasets/dbpedia-100k-openai-ada002-angular.hdf5 https://storage.googleapis.com/ann-datasets/custom/dbpedia-100k-openai-ada002-angular.hdf5 \
curl -o ./datasets/deep-image-96-angular.hdf5 https://ann-benchmarks.com/deep-image-96-angular.hdf5 && \
curl -o ./datasets/mnist-784-euclidean.hdf5 https://ann-benchmarks.com/mnist-784-euclidean.hdf5 && \
curl -o ./datasets/gist-960-euclidean.hdf5 https://ann-benchmarks.com/gist-960-euclidean.hdf5
Run a single performance test on an ann-benchmarks hdf5 dataset.
DATASET=./datasets/dbpedia-100k-openai-ada002-angular.hdf5 DISTANCE=cosine docker compose run benchmarker
For more details on additional configuration options see the help options.
docker compose run benchmarker /app/benchmarker ann-benchmark -h