Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add logging for RetrievalEvaluator NaN values for similarity scores #1398

Conversation

KennethEnevoldsen
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen commented Nov 6, 2024

Fixes #1389

Ran through tests but couldn't find a case where it was actually used. It might be a thing we can simply remove.

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

@KennethEnevoldsen
Copy link
Contributor Author

@orionw or @Muennighoff will merge this in but would love to remove the -1 transformation if possible

@KennethEnevoldsen KennethEnevoldsen merged commit cc7a106 into main Nov 11, 2024
10 checks passed
@KennethEnevoldsen KennethEnevoldsen deleted the KennethEnevoldsen/issue-RetrievalEvaluator-NaN-values-for-similarity-scores branch November 11, 2024 08:17
@orionw
Copy link
Contributor

orionw commented Nov 11, 2024

I’m not aware of when it would happen, I think it was there when I started using MTEB. I assume like guessed in #1389 it’s for overflow or something.

Perhaps @Muennighoff may know more.

@Muennighoff
Copy link
Contributor

Yeah I'm also not sure, maybe when embeddings are NaN, but then that's a problem with the model and we probably shouldn't work around that 🤔 We can also remove it and wait until someone reports an issue with it

KennethEnevoldsen added a commit that referenced this pull request Nov 13, 2024
As discussed in #1398 NaN should never occur. Removing handling of them.

Fixes #1389
@KennethEnevoldsen KennethEnevoldsen mentioned this pull request Nov 13, 2024
2 tasks
@KennethEnevoldsen
Copy link
Contributor Author

removing them in #1444

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RetrievalEvaluator NaN values for similarity scores
3 participants