Evaluate on all topics instead of those with documents retrieved #108
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ryan-clancy
approved these changes
Jun 18, 2019
ryan-clancy
pushed a commit
that referenced
this pull request
Jun 26, 2019
* Add files via upload (#104) * Evaluate on all topics instead of those with documents retrieved (#108) * Added TREC7 and TREC8 qrels (primarily to avoid confusion with the WT qrels in `qrels/qrels.401-450.txt` * Missing TREC7 topics files * Change default to setting also used in the "official" evaluation * Add -c flag for trec_eval * Revert "Change default to setting also used in the "official" evaluation" This reverts commit 85b13df. * TREC Terabyte 04-06 topic/qrel files merged (#109)
ryan-clancy
pushed a commit
that referenced
this pull request
Jun 28, 2019
* Update Azure script * and make sure it's uncommented... * Enable timings for terrier and pisa * Capture logs and scp back to local. * Update robust04 runbook * Remove P@5 from OldDog config * robust04, core17, and core18 runbooks. * Update terrier version in runbook * Update core17 runbook * Update terrier runbook * Whoops... * Update olddog version * Add anserini-bm25prf to runbook * Update anserini-bm25prf * Update anserini-bm25prf * Update robust05 path for irc-centre2019 * Add ielab and galago to runbook * Write an additional file with the trec_eval results * Update runbook for official releases * Add NDCG@20 by default and fix anserini-bm25prf output directory * Don't create .trec_eval files for non-trec files * Merge master into azure-script (#111) * Add files via upload (#104) * Evaluate on all topics instead of those with documents retrieved (#108) * Added TREC7 and TREC8 qrels (primarily to avoid confusion with the WT qrels in `qrels/qrels.401-450.txt` * Missing TREC7 topics files * Change default to setting also used in the "official" evaluation * Add -c flag for trec_eval * Revert "Change default to setting also used in the "official" evaluation" This reverts commit 85b13df. * TREC Terabyte 04-06 topic/qrel files merged (#109)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Call
trec_eval -cto avoid missing queries without document results in a topic set.(Relevant for us when processing conjunctively.)
Addresses #105