Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: RAG QA Dataset v2 #1075

Open
jalling97 opened this issue Sep 19, 2024 · 0 comments
Open

feat: RAG QA Dataset v2 #1075

jalling97 opened this issue Sep 19, 2024 · 0 comments

Comments

@jalling97
Copy link
Contributor

User Story

As an evaluator of LFAI
I want to have metrics using in depth Question/Answer evaluations
So that I can measure RAG improvement

Additional context

The LFAI RAG qa v1 dataset is a useful dataset for measuring simple question and answer pairs, but a harder task is needed in order to provide better evaluations for LFAI.

As such, a new dataset is needed. This dataset will be different in the following ways:

  • The v1 version has only 8 documents in the dataset. v2 Will require more, at least double.
  • The type of documents had an initially varied scope to determine what topics may be easier to answer questions surrounding. The v2 version will be more narrowed in topic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant