Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to Match: case insensitive and strip #1421

Open
LoryPack opened this issue Nov 30, 2023 · 0 comments
Open

Improvements to Match: case insensitive and strip #1421

LoryPack opened this issue Nov 30, 2023 · 0 comments

Comments

@LoryPack
Copy link
Contributor

LoryPack commented Nov 30, 2023

Describe the feature or improvement you're requesting

The current implementation of the Match basic eval template is case-sensitive. This leads to results such as:

{'correct': False, 'expected': 'no', 'picked': None, 'sampled': 'No', 'options': ['no']}

Similarly, Match does not strip the sampled string from white spaces in the front. That causes the evaluation to fail for models using the Completion endpoint, as those are more likely to output spaces in the front. Example:

{'correct': False, 'expected': 'Mumbai', 'picked': None, 'sampled': ' Mumbai', 'options': ['Mumbai']}

It would be good to add an argument to Match allowing to require case insensitive behaviour and to determine whether the answer should be stripped of spaces. These can then be specified in the yaml file for a task.

Similar options can be added for the other templates, such as Includes and FuzzyMatch.

Additional context

No response

@LoryPack LoryPack changed the title Case insensitive Match Improvements to Match: case insensitive and strip Nov 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant