Fix problematic sample in Schelling Point #1534
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1533
Problem
There was a strange sample in the OWT dataset that looks like:
which repeats the word
bihl
>6000 times. This was triggering aopenai.BadRequestError
:Solution
This PR removes that problematic sample from the dataset. b01c56d
It would be more robust to do better error handling like we do here so that the eval doesn't crash in such cases, but this error handling is implemented in Solvers, which in-theory should work but doesn't due specifics of how this Schelling Point eval was implemented. Don't think this is currently a priority, so I've just added a clearer error message to users in case they (reasonably) try to run a Solver with this eval and it fails. a506f0c