Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve default evaluation setting #107

Closed
wants to merge 5 commits into from

Conversation

arjenpdevries
Copy link
Member

Address evaluation metrics to be reported on all topics (-c flag),
re: #105

And, report P@5 by default, just like the "official" evaluation at OSIRRC 2019.

@arjenpdevries arjenpdevries added the enhancement New feature or request label Jun 18, 2019
@lintool
Copy link
Member

lintool commented Jun 18, 2019

@arjenpdevries I'll let @r-clancy look at this in case he's in the middle of something else.

@ryan-clancy
Copy link
Member

I'll merge this, but for the rest of the official run through I'll keep P@30 in my local copy of jig so I can confirm P@30 numbers (which is what most people have in their README now).

@lintool
Copy link
Member

lintool commented Jun 18, 2019

@arjenpdevries would you be willing to change over to P@30 in OldDog to be consistent with others? The primary metrics we've decided on is NDCG@20, AP, and P@30, IIRC...

@arjenpdevries
Copy link
Member Author

Definitely!
@chriskamphuis can you update the table with the P@30 scores?

@chriskamphuis
Copy link
Member

After this merge we will find other values because of -c flag. I will update the values to those found when running with the -c flag.

@arjenpdevries
Copy link
Member Author

Probably should only merge the - c modification then?
Shall I revert the P. 5 and reopen another pull request?

@ryan-clancy
Copy link
Member

Probably should only merge the - c modification then?
Shall I revert the P. 5 and reopen another pull request?

Yeah, let's just do the -c change for now.

@arjenpdevries
Copy link
Member Author

Ok 👌 later tonight

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants