Skip to content

Latest commit

 

History

History
162 lines (139 loc) · 9.6 KB

regressions-wiki-all-6-3-tamber-bm25.md

File metadata and controls

162 lines (139 loc) · 9.6 KB
Raw

Anserini Regressions: QA on Wikipedia Sliding Windows

Models: BM25

This page documents QA regression experiments on the wiki-all-6-3-tamber corpus, which is integrated into Anserini's regression testing framework. The exact configuration here is the 6/3 sentence sliding window corpus described in the following paper:

Manveer Singh Tamber, Ronak Pradeep, and Jimmy Lin. Pre-Processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering. Proceedings of the 45th European Conference on Information Retrieval (ECIR 2023), Part III, pages 163–176, April 2023, Dublin, Ireland.

The exact configurations for these regressions are stored in this YAML file. Note that this page is automatically generated from this template as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead.

From one of our Waterloo servers (e.g., orca), the following command will perform the complete regression, end to end:

python src/main/python/run_regression.py --index --verify --search --regression wiki-all-6-3-tamber-bm25

Indexing

Typical indexing command:

target/appassembler/bin/IndexCollection \
  -collection MrTyDiCollection \
  -input /path/to/wiki-all-6-3-tamber \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -generator DefaultLuceneDocumentGenerator \
  -threads 20 -storeRaw \
  >& logs/log.wiki-all-6-3-tamber &

The directory /path/to/wiki-all-6-3-tamber/should be a directory containing the wiki-all-6-3-tamber passages collection retrieved from here.

For additional details, see explanation of common indexing options.

Retrieval

Topics and qrels are stored here, which is linked to the Anserini repo as a submodule. The regression experiments here evaluate on the test sets of Natural Questions, TriviaQA, SQuAD, and WebQuestions.

After indexing has completed, you should be able to perform retrieval as follows:

target/appassembler/bin/SearchCollection \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -topics tools/topics-and-qrels/topics.dpr.nq.test.txt \
  -topicreader DprNq \
  -output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.nq.test.txt \
  -bm25 &
target/appassembler/bin/SearchCollection \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -topics tools/topics-and-qrels/topics.dpr.trivia.test.txt \
  -topicreader DprNq \
  -output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.trivia.test.txt \
  -bm25 &
target/appassembler/bin/SearchCollection \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -topics tools/topics-and-qrels/topics.dpr.squad.test.txt \
  -topicreader DprJsonl \
  -output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.squad.test.txt \
  -bm25 &
target/appassembler/bin/SearchCollection \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -topics tools/topics-and-qrels/topics.dpr.wq.test.txt \
  -topicreader DprJsonl \
  -output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.wq.test.txt \
  -bm25 &
target/appassembler/bin/SearchCollection \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -topics tools/topics-and-qrels/topics.dpr.curated.test.txt \
  -topicreader DprJsonl \
  -output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.curated.test.txt \
  -bm25 &
target/appassembler/bin/SearchCollection \
  -index indexes/lucene-index.wiki-all-6-3-tamber/ \
  -topics tools/topics-and-qrels/topics.nq.test.txt \
  -topicreader DprNq \
  -output runs/run.wiki-all-6-3-tamber.bm25.topics.nq.test.txt \
  -bm25 &

The trec format will need to be converted to DPR's JSON format for evaluation:

python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --index indexes/lucene-index.wiki-all-6-3-tamber/ \
  --topics dpr-nq-test \
  --input runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.nq.test.txt \
  --output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.nq.test.txt.json \
  --combine-title-text &
python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --index indexes/lucene-index.wiki-all-6-3-tamber/ \
  --topics dpr-trivia-test \
  --input runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.trivia.test.txt \
  --output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.trivia.test.txt.json \
  --combine-title-text &
python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --index indexes/lucene-index.wiki-all-6-3-tamber/ \
  --topics dpr-squad-test \
  --input runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.squad.test.txt \
  --output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.squad.test.txt.json \
  --combine-title-text &
python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --index indexes/lucene-index.wiki-all-6-3-tamber/ \
  --topics dpr-wq-test \
  --input runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.wq.test.txt \
  --output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.wq.test.txt.json \
  --combine-title-text &
python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --index indexes/lucene-index.wiki-all-6-3-tamber/ \
  --topics dpr-curated-test \
  --input runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.curated.test.txt \
  --output runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.curated.test.txt.json \
  --combine-title-text  --regex &
python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
  --index indexes/lucene-index.wiki-all-6-3-tamber/ \
  --topics nq-test \
  --input runs/run.wiki-all-6-3-tamber.bm25.topics.nq.test.txt \
  --output runs/run.wiki-all-6-3-tamber.bm25.topics.nq.test.txt.json \
  --combine-title-text &

Evaluation can be performed using scripts from pyserini:

python -m pyserini.eval.evaluate_dpr_retrieval --topk 20 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.nq.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 100 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.nq.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 20 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.trivia.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 100 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.trivia.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 20 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.squad.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 100 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.squad.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 20 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.wq.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 100 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.wq.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 20 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.curated.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 100 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.dpr.curated.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 20 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.nq.test.txt.json
python -m pyserini.eval.evaluate_dpr_retrieval --topk 100 --retrieval runs/run.wiki-all-6-3-tamber.bm25.topics.nq.test.txt.json

Effectiveness

With the above commands, you should be able to reproduce the following results:

top_20_accuracy BM25 (default parameters)
DPR: Natural Questions Test 0.6604
DPR: TriviaQA Test 0.7832
DPR: SQuAD Test 0.7265
DPR: WebQuestions Test 0.6403
DPR: CuratedTREC Test 0.8055
EfficientQA: Natural Questions Test 0.6665
top_100_accuracy BM25 (default parameters)
DPR: Natural Questions Test 0.8083
DPR: TriviaQA Test 0.8482
DPR: SQuAD Test 0.8325
DPR: WebQuestions Test 0.7874
DPR: CuratedTREC Test 0.9135
EfficientQA: Natural Questions Test 0.8166

Reproduction Log*

To add to this reproduction log, modify this template and run bin/build.sh to rebuild the documentation.