Skip to content

Commit

Permalink
fix(publichealth-qa): ignore rows with None values in question or…
Browse files Browse the repository at this point in the history
… `answer` (#1565)
  • Loading branch information
dbuades authored Dec 8, 2024
1 parent 1d21818 commit 68bd8ac
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions mteb/tasks/Retrieval/multilingual/PublicHealthQARetrieval.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ def _load_publichealthqa_data(
answer_ids = {answer: _id for _id, answer in enumerate(set(data["answer"]))}

for row in data:
if row["question"] is None or row["answer"] is None:
# There are some questions and answers that are None in the original dataset, specifically in the Arabic subset.
continue
question = row["question"]
answer = row["answer"]
query_id = f"Q{question_ids[question]}"
Expand Down

0 comments on commit 68bd8ac

Please sign in to comment.