Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issues with evaluation code #133

Merged
merged 4 commits into from
May 30, 2024
Merged

Fix issues with evaluation code #133

merged 4 commits into from
May 30, 2024

Conversation

jlewi
Copy link
Owner

@jlewi jlewi commented May 30, 2024

  • Fix Link Eval Results To Traces #131 - store the generate trace in the EvalExample this will make it possible to look up the trace associated with an evaluation result so we can see how it was generated.

  • To support this we add trace id to the GenerateResponse

  • Fix incorrect value in case where AI returns no code cells - Fix Eval Results - Have Distance -1 and Normalized Distance of 0 #132

    • We weren't actually computing the distance so we were getting whatever the initialization values were
    • This caused normalized distance to be 0 which is very misleading
    • If no code blocks are returned then we should set the distance to be the maximum possible distance
      • In which case normalized distance will end up being 1
    • To achieve that we should set the command to be the empty string and then compute the distance.

Copy link

netlify bot commented May 30, 2024

Deploy Preview for foyle canceled.

Name Link
🔨 Latest commit d2a884c
🔍 Latest deploy log https://app.netlify.com/sites/foyle/deploys/6657dbef6d8d0900082b6c74

@jlewi jlewi merged commit 5a63b46 into main May 30, 2024
5 checks passed
@jlewi jlewi deleted the jlewi/evalexamples branch May 30, 2024 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Eval Results - Have Distance -1 and Normalized Distance of 0 Link Eval Results To Traces
1 participant