Use function calling to parse responses #18
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CC @wong-codaio |
manugoyal
approved these changes
Oct 10, 2023
js/llm.ts
Outdated
|
|
||
| const COT_SUFFIX = `Write out in a step by step manner your reasoning to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset. Then print only a single choice from {{__choices}} (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the answer by itself on a new line formatted as "Answer=X"`; | ||
| const COT_SUFFIX = | ||
| "Answer the question by calling the `select_choice` with your reasoning in a step-by-step matter to be sure that your conclusion is correct. Avoid simply stating the correct answer at the outset. Select a single choice by setting the `choice` parameter to a single choice from {{__choices}}."; |
There was a problem hiding this comment.
the select_choice function?
py/autoevals/llm.py
Outdated
| simply stating the correct answer at the outset. Then print only a single choice from {{__choices}} (without | ||
| quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the | ||
| answer by itself on a new line formatted as "Answer=X" | ||
| Answer the question by calling the `select_choice` with your reasoning in a step-by-step matter to be |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A substantial fraction (20%+) of responses have parse errors. This change uses function calling to reduce that.