Add fusion regression #2611
Add fusion regression #2611
Conversation
…yserini convention
| qrel: qrels.beir-v1.0.0-robust04.test.txt | ||
|
|
||
| # Fusion Regression Test Configuration | ||
| runs: |
There was a problem hiding this comment.
Why don't we assume that the runs are generated by a regression in https://github.com/castorini/anserini/tree/master/src/main/resources/regression? E.g., beir-v1.0.0-robust04.flat.yaml
So we keep the yaml file and the run name (i.e., just take the run file name it generates).
There was a problem hiding this comment.
I established a test_2.yaml, do you mind taking a look to see if that is something you'd like?
Perform two regression runs for test2
One could generate the runs necessary for test 2 following
- https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.bge-base-en-v1.5.flat-int8.cached.md
- https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.bge-base-en-v1.5.flat.cached.md
Or do you want the fusion yaml file to contain information about the yaml files (the ones that produce runs for the fusion test) it depends on and perform those runs automatically when the fusion test is performed?
There was a problem hiding this comment.
Its name is now changed into src/main/resources/fuse_regression/fusion-regression-bge-flat-int8-robust04-2.yaml
| @@ -52,4 +52,4 @@ models: | |||
| R@100: | |||
| - 0.3746 | |||
| R@1000: | |||
| - 0.6345 | |||
| - 0.6345 | |||
There was a problem hiding this comment.
revert unnecessary changes please
|
|
||
| # Fusion Regression Test Configuration | ||
| runs: | ||
| - runs/runs.beir/run.inverted.beir-v1.0.0-robust04.flat.test.bm25 |
There was a problem hiding this comment.
record the regression yaml this run comes from?
docs/fuse-regressions/test-setup.md
Outdated
| mkdir -p runs/runs.beir | ||
|
|
||
| # Download the run files from Google Drive using their file IDs | ||
| wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1XVlVCDYQe3YjRzxplaeGbmW_0EFQCgm8' -O runs/runs.beir/run.inverted.beir-v1.0.0-robust04.multifield.test.bm25 |
There was a problem hiding this comment.
depend on the runs generated by the regression yaml config?
| output: runs/fuse/run.flat-int8.beir-v1.0.0-robust04.bge-base-en-v1.5.test.bge-flat-int8-cached.bge-flat-cached.fusion.rrf | ||
| results: | ||
| nDCG@10: | ||
| - 0.3 |
…bust04.flat.yaml back
There was a problem hiding this comment.
Why don't we start with fusion robust04 BM25 and dense:
Accessible here: https://github.com/castorini/anserini?tab=readme-ov-file#beir-v100-regressions
Added corresponding yaml. Do you like the structure of the yaml and the namings? If so, I could start on generating more. |
| @@ -0,0 +1,79 @@ | |||
| --- | |||
There was a problem hiding this comment.
let's remove this and keep only the above?
| k: 1000 | ||
| depth: 1000 | ||
| rrf_k: 60 | ||
| output: runs/fuse/run.beir-v1.0.0-robust04.flat.bm25.fuse-rrf.bge-base-en-v1.5.bge-flat-onnx.topics.beir-v1.0.0-robust04.test.txt |
There was a problem hiding this comment.
instead of a subdir, how about runs/run.fuse.XXXX?
There was a problem hiding this comment.
How about runs/runs.fuse.rrf.XXX and runs/runs.fuse.avg.XXX
|
hey @Stefan824 this is good for merging, but I pinged you on Slack for improvements. Please work on that next. Thanks! |
This PR should address issue #2595 if goes well.
run_fusion_regression.pyto handle fusion regression tasks. It behaves similarly torun_regression.py.anserini/docs/fuse-regressions/test-setup.md.