Update reproduction instructions for BEIR (with tarball downloads) (#…

…2346)
castorini · Jan 19, 2024 · ba472b8 · ba472b8
1 parent 5ef8b18
commit ba472b8
Show file tree

Hide file tree

Showing 291 changed files with 2,991 additions and 407 deletions.
diff --git a/README.md b/README.md
@@ -207,11 +207,13 @@ See individual pages for details!
 Key:
 
 + F1 = "flat" baseline (Lucene analyzer)
-+ F2 = "flat" baselinse (pre-tokenized with `bert-base-uncased` tokenizer)
++ F2 = "flat" baseline (pre-tokenized with `bert-base-uncased` tokenizer)
 + MF = "multifield" baseline (Lucene analyzer)
 + U1 = uniCOIL (noexp)
 + S1 = SPLADE++ CoCondenser-EnsembleDistil
 
+See instructions below the table for how to reproduce results for a model on all BEIR corpora "in one go".
+
 | Corpus                  |                                      F1                                       |                                        F2                                        |                                         MF                                          |                                           U1                                           |                                          S1                                           |
 |-------------------------|:-----------------------------------------------------------------------------:|:--------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------:|:--------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------:|
 | TREC-COVID              |       [+](docs/regressions/regressions-beir-v1.0.0-trec-covid-flat.md)        |       [+](docs/regressions/regressions-beir-v1.0.0-trec-covid-flat-wp.md)        |       [+](docs/regressions/regressions-beir-v1.0.0-trec-covid-multifield.md)        |       [+](docs/regressions/regressions-beir-v1.0.0-trec-covid-unicoil-noexp.md)        |       [+](docs/regressions/regressions-beir-v1.0.0-trec-covid-splade-pp-ed.md)        |
@@ -244,6 +246,36 @@ Key:
 | Climate-FEVER           |      [+](docs/regressions/regressions-beir-v1.0.0-climate-fever-flat.md)      |      [+](docs/regressions/regressions-beir-v1.0.0-climate-fever-flat-wp.md)      |      [+](docs/regressions/regressions-beir-v1.0.0-climate-fever-multifield.md)      |      [+](docs/regressions/regressions-beir-v1.0.0-climate-fever-unicoil-noexp.md)      |      [+](docs/regressions/regressions-beir-v1.0.0-climate-fever-splade-pp-ed.md)      |
 | SciFact                 |         [+](docs/regressions/regressions-beir-v1.0.0-scifact-flat.md)         |         [+](docs/regressions/regressions-beir-v1.0.0-scifact-flat-wp.md)         |         [+](docs/regressions/regressions-beir-v1.0.0-scifact-multifield.md)         |         [+](docs/regressions/regressions-beir-v1.0.0-scifact-unicoil-noexp.md)         |         [+](docs/regressions/regressions-beir-v1.0.0-scifact-splade-pp-ed.md)         |
 
+To reproduce the SPLADE++ CoCondenser-EnsembleDistil results, start by downloading the collection:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-splade-pp-ed.tar -P collections/
+tar xvf collections/beir-v1.0.0-splade-pp-ed.tar -C collections/
+```
+
+The tarball is 42 GB and has MD5 checksum `9c7de5b444a788c9e74c340bf833173b`.
+Once you've unpacked the data, the following commands will loop over all BEIR corpora and run the regressions:
+
+```bash
+MODEL="splade-pp-ed"; CORPORA=(trec-covid bioasq nfcorpus nq hotpotqa fiqa signal1m trec-news robust04 arguana webis-touche2020 cqadupstack-android cqadupstack-english cqadupstack-gaming cqadupstack-gis cqadupstack-mathematica cqadupstack-physics cqadupstack-programmers cqadupstack-stats cqadupstack-tex cqadupstack-unix cqadupstack-webmasters cqadupstack-wordpress quora dbpedia-entity scidocs fever climate-fever scifact); for c in "${CORPORA[@]}"
+do
+    echo "Running $c..."
+    python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-${c}-${MODEL} > logs/log.beir-v1.0.0-${c}-${MODEL} 2>&1
+done
+```
+
+You can verify the results by examining the log files in `logs/`.
+
+For the other models, modify the above commands as follows:
+
+| Key | Corpus                          | Checksum                           | `MODEL`         |
+|:----|:--------------------------------|:-----------------------------------|:----------------|
+| F1  | `beir-v1.0.0-corpus.tar`        | `faefd5281b662c72ce03d22021e4ff6b` | `flat`          |
+| F2  | `beir-v1.0.0-corpus-wp.tar`     | `3cf8f3dcdcadd49362965dd4466e6ff2` | `flat-wp`       |
+| MF  | `beir-v1.0.0-corpus.tar`        | `faefd5281b662c72ce03d22021e4ff6b` | `multifield`    |
+| U1  | `beir-v1.0.0-unicoil-noexp.tar` | `4fd04d2af816a6637fc12922cccc8a83` | `unicoil-noexp` |
+| S1  | `beir-v1.0.0-splade-pp-ed.tar`  | `9c7de5b444a788c9e74c340bf833173b` | `splade-pp-ed`  |
+
 </details>
 <details>
 <summary>Cross-lingual and Multi-lingual Regressions</summary>

diff --git a/docs/regressions/regressions-beir-v1.0.0-arguana-flat-wp.md b/docs/regressions/regressions-beir-v1.0.0-arguana-flat-wp.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-flat-wp
 ```
 
+All the BEIR corpora, pre-tokenized with the `bert-base-uncased` tokenizer, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus-wp.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus-wp.tar -C collections/
+```
+
+The tarball is 13 GB and has MD5 checksum `3cf8f3dcdcadd49362965dd4466e6ff2`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-arguana-flat.md b/docs/regressions/regressions-beir-v1.0.0-arguana-flat.md
@@ -12,6 +12,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-flat
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-arguana-multifield.md b/docs/regressions/regressions-beir-v1.0.0-arguana-multifield.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-multifield
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-arguana-splade-pp-ed.md b/docs/regressions/regressions-beir-v1.0.0-arguana-splade-pp-ed.md
@@ -11,10 +11,19 @@ Note that this page is automatically generated from [this template](../../src/ma
 From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:
 
 ```
-python src/main/python/run_regression.py --index --verify --search \
-  --regression beir-v1.0.0-arguana-splade-pp-ed
+python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-splade-pp-ed
 ```
 
+All the BEIR corpora, encoded by the SPLADE++ CoCondenser-EnsembleDistil model, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-splade-pp-ed.tar -P collections/
+tar xvf collections/beir-v1.0.0-splade-pp-ed.tar -C collections/
+```
+
+The tarball is 42 GB and has MD5 checksum `9c7de5b444a788c9e74c340bf833173b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Sample indexing command:
@@ -66,8 +75,3 @@ With the above commands, you should be able to reproduce the following results:
 | BEIR (v1.0.0): ArguAna                                                                                       | 0.9744    |
 | **R@1000**                                                                                                   | **SPLADE++ (CoCondenser-EnsembleDistil)**|
 | BEIR (v1.0.0): ArguAna                                                                                       | 0.9950    |
-
-
-## Reproduction Log[*](../../docs/reproducibility.md)
-
-To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-arguana-splade-pp-ed.template) and run `bin/build.sh` to rebuild the documentation.
diff --git a/docs/regressions/regressions-beir-v1.0.0-arguana-unicoil-noexp.md b/docs/regressions/regressions-beir-v1.0.0-arguana-unicoil-noexp.md
@@ -16,6 +16,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-arguana-unicoil-noexp
 ```
 
+All the BEIR corpora, encoded by the uniCOIL-noexp model, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-unicoil-noexp.tar -P collections/
+tar xvf collections/beir-v1.0.0-unicoil-noexp.tar -C collections/
+```
+
+The tarball is 30 GB and has MD5 checksum `4fd04d2af816a6637fc12922cccc8a83`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-bioasq-flat-wp.md b/docs/regressions/regressions-beir-v1.0.0-bioasq-flat-wp.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq-flat-wp
 ```
 
+All the BEIR corpora, pre-tokenized with the `bert-base-uncased` tokenizer, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus-wp.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus-wp.tar -C collections/
+```
+
+The tarball is 13 GB and has MD5 checksum `3cf8f3dcdcadd49362965dd4466e6ff2`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-bioasq-flat.md b/docs/regressions/regressions-beir-v1.0.0-bioasq-flat.md
@@ -12,6 +12,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq-flat
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-bioasq-multifield.md b/docs/regressions/regressions-beir-v1.0.0-bioasq-multifield.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq-multifield
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-bioasq-splade-pp-ed.md b/docs/regressions/regressions-beir-v1.0.0-bioasq-splade-pp-ed.md
@@ -11,10 +11,19 @@ Note that this page is automatically generated from [this template](../../src/ma
 From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:
 
 ```
-python src/main/python/run_regression.py --index --verify --search \
-  --regression beir-v1.0.0-bioasq-splade-pp-ed
+python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq-splade-pp-ed
 ```
 
+All the BEIR corpora, encoded by the SPLADE++ CoCondenser-EnsembleDistil model, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-splade-pp-ed.tar -P collections/
+tar xvf collections/beir-v1.0.0-splade-pp-ed.tar -C collections/
+```
+
+The tarball is 42 GB and has MD5 checksum `9c7de5b444a788c9e74c340bf833173b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Sample indexing command:
@@ -66,8 +75,3 @@ With the above commands, you should be able to reproduce the following results:
 | BEIR (v1.0.0): BioASQ                                                                                        | 0.7385    |
 | **R@1000**                                                                                                   | **SPLADE++ (CoCondenser-EnsembleDistil)**|
 | BEIR (v1.0.0): BioASQ                                                                                        | 0.8757    |
-
-
-## Reproduction Log[*](../../docs/reproducibility.md)
-
-To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-bioasq-splade-pp-ed.template) and run `bin/build.sh` to rebuild the documentation.
diff --git a/docs/regressions/regressions-beir-v1.0.0-bioasq-unicoil-noexp.md b/docs/regressions/regressions-beir-v1.0.0-bioasq-unicoil-noexp.md
@@ -16,6 +16,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-bioasq-unicoil-noexp
 ```
 
+All the BEIR corpora, encoded by the uniCOIL-noexp model, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-unicoil-noexp.tar -P collections/
+tar xvf collections/beir-v1.0.0-unicoil-noexp.tar -C collections/
+```
+
+The tarball is 30 GB and has MD5 checksum `4fd04d2af816a6637fc12922cccc8a83`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-climate-fever-flat-wp.md b/docs/regressions/regressions-beir-v1.0.0-climate-fever-flat-wp.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-climate-fever-flat-wp
 ```
 
+All the BEIR corpora, pre-tokenized with the `bert-base-uncased` tokenizer, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus-wp.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus-wp.tar -C collections/
+```
+
+The tarball is 13 GB and has MD5 checksum `3cf8f3dcdcadd49362965dd4466e6ff2`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-climate-fever-flat.md b/docs/regressions/regressions-beir-v1.0.0-climate-fever-flat.md
@@ -12,6 +12,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-climate-fever-flat
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-climate-fever-multifield.md b/docs/regressions/regressions-beir-v1.0.0-climate-fever-multifield.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-climate-fever-multifield
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-climate-fever-splade-pp-ed.md b/docs/regressions/regressions-beir-v1.0.0-climate-fever-splade-pp-ed.md
@@ -11,10 +11,19 @@ Note that this page is automatically generated from [this template](../../src/ma
 From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:
 
 ```
-python src/main/python/run_regression.py --index --verify --search \
-  --regression beir-v1.0.0-climate-fever-splade-pp-ed
+python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-climate-fever-splade-pp-ed
 ```
 
+All the BEIR corpora, encoded by the SPLADE++ CoCondenser-EnsembleDistil model, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-splade-pp-ed.tar -P collections/
+tar xvf collections/beir-v1.0.0-splade-pp-ed.tar -C collections/
+```
+
+The tarball is 42 GB and has MD5 checksum `9c7de5b444a788c9e74c340bf833173b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Sample indexing command:
@@ -66,8 +75,3 @@ With the above commands, you should be able to reproduce the following results:
 | BEIR (v1.0.0): Climate-FEVER                                                                                 | 0.5211    |
 | **R@1000**                                                                                                   | **SPLADE++ (CoCondenser-EnsembleDistil)**|
 | BEIR (v1.0.0): Climate-FEVER                                                                                 | 0.7183    |
-
-
-## Reproduction Log[*](../../docs/reproducibility.md)
-
-To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/beir-v1.0.0-climate-fever-splade-pp-ed.template) and run `bin/build.sh` to rebuild the documentation.
diff --git a/docs/regressions/regressions-beir-v1.0.0-climate-fever-unicoil-noexp.md b/docs/regressions/regressions-beir-v1.0.0-climate-fever-unicoil-noexp.md
@@ -16,6 +16,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-climate-fever-unicoil-noexp
 ```
 
+All the BEIR corpora, encoded by the uniCOIL-noexp model, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-unicoil-noexp.tar -P collections/
+tar xvf collections/beir-v1.0.0-unicoil-noexp.tar -C collections/
+```
+
+The tarball is 30 GB and has MD5 checksum `4fd04d2af816a6637fc12922cccc8a83`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-cqadupstack-android-flat-wp.md b/docs/regressions/regressions-beir-v1.0.0-cqadupstack-android-flat-wp.md
@@ -13,6 +13,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-cqadupstack-android-flat-wp
 ```
 
+All the BEIR corpora, pre-tokenized with the `bert-base-uncased` tokenizer, are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus-wp.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus-wp.tar -C collections/
+```
+
+The tarball is 13 GB and has MD5 checksum `3cf8f3dcdcadd49362965dd4466e6ff2`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command:

diff --git a/docs/regressions/regressions-beir-v1.0.0-cqadupstack-android-flat.md b/docs/regressions/regressions-beir-v1.0.0-cqadupstack-android-flat.md
@@ -12,6 +12,16 @@ From one of our Waterloo servers (e.g., `orca`), the following command will perf
 python src/main/python/run_regression.py --index --verify --search --regression beir-v1.0.0-cqadupstack-android-flat
 ```
 
+All the BEIR corpora are available for download:
+
+```bash
+wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-corpus.tar -P collections/
+tar xvf collections/beir-v1.0.0-corpus.tar -C collections/
+```
+
+The tarball is 14 GB and has MD5 checksum `faefd5281b662c72ce03d22021e4ff6b`.
+After download and unpacking the corpora, the `run_regression.py` command above should work without any issue.
+
 ## Indexing
 
 Typical indexing command: