Remove multiple duplicates in choice sheets

[Sujanadh]

Issue:

• Duplicate choices after injecting mandatory fields #302

Description:

This PR addresses the issue of duplicate combinations of list_name and name fields in the "choices" sheet when allow_choices_duplicates is enabled. It ensures that only unique combinations are retained after the merging process, improving the accuracy of the choices sheet and preventing redundancy.

Key Changes:

Duplicate Removal in Choices Sheet:

Added logic to remove duplicates based on the list_name and name columns when processing the "choices" sheet.
Implemented using the drop_duplicates(subset=['list_name', 'name']) method, ensuring the choice lists have unique entries.

Before the Fix:
When allow_choices_duplicates was enabled, the "choices" sheet could have multiple duplicate entries for the same list_name and name combination, leading to redundant data in the final output.

After the Fix:
The function now ensures that only unique combinations of list_name and name are retained, improving the integrity of the choices data while retaining necessary survey structure.

[spwoodcock]

Isn't this regressing to the same functionality / issue we had before I removed the duplicate check? @manjitapandey had a form where yes/no choices had a different list name, but they were stripped out as we already have yes/no choices in our mandatory fields (giving an invalid form). I removed the duplicate check as it's valid to have duplicates in the choices sheet. Is this causing issues somehow?

[Sujanadh]

We used to check if name field in choice sheet is unique or not, and removed other duplicates despite different list_name and we changed allow_duplicate_fields to True resulting into multiple same choices resulted in to the error of not displaying the label , may be its due to missing translation but I am not sure, removing duplicated yes_no list name worked as expected.

[Sujanadh]

So in this PR we check for not only unique name field but the unique combination of name and list_name , so that we can have multiple yes_no choices with different list_name.

[manjitapandey]

Since its not because of the translation, rather the duplication of option. I believe we are good to go ahead with merging the PR and we also tested it on local the questions are visible now and no duplication of options.
@Sujanadh

[spwoodcock]

Noted Sujan, thanks for clarifying!

I can see in this form screenshot there is clearly an issue with the translation fields not being filled. Having a combination of both filled label and empty label::English etc will surely mean the label doesn't display at all (I could be wrong, but I thought I have seen this behaviour in xlsforms)

[rsavoye]

This may not make a difference, but you're supposed to have the 2 letter ISO abbreviation, so it'd be label::English(en), etc...

[Sujanadh]

For now, i have released the recent fix to work around, we can see this translation issue later on if we encounter related issue again.

[Sujanadh]

This issue was originally due to multiple label field for english lang, custom form already had label :: english(1) and our mandatory form has label :: english(en) . ODK Collect does not support mixed naming conventions for the label column.
The correct format for specifying labels in different languages is label::language(code), such as label::English (en) for English.

[Sujanadh]

proposed solution would be to detect invalid label and convert them to valid labels while injecting mandatory fields.

[spwoodcock]

The (1) is appended by Pandas when merging two dataframes that have the same column, but both have values present (so they can't merge into one column).

We have the same when we had conflicting values in the save_to column.

Perhaps we need to specify one of the columns to take precedent over the other, likely the user uploaded XLSForm value.
(no need to do column name correction, as this is too complicated and a bit too 'magic')