Should newlines be allowed on metadata upload

[wasade]

I recently pushed metadata to Qiita which unbeknownst to me contained newlines in some cells. I discovered this when downloading the pushed metadata after, and observed what appeared to be incorrect sample names in the resulting tab delimited file, coming from the contained newlines.

Should Qiita detect and error, or detect and replace, newline characters prior to accepting metadata updates?

[antgonza]

Thank you for reporting.

A couple of questions:

1. do you have a small example of the erroneous data? Basically, I'm not sure if the new lines were in the sample names, in the metadata columns or both.
2. where you using the GUI or an endpoint?

[wasade]

1. qiita-rc under 10317 should have invalid sample names in the metadata download (unless it's been wiped since last week). An example of what was observed in the download from qiita-rc is below. Newlines were in variable values.

$ grep -v "^10317" 10317_20210930-123208.txt | cut -c 1-20 | head
sample_name	acid_ref
fresh water fish: Se
fresh water fish: Se
fresh water fish: Se
fresh water fish: Se
fresh water fish: Se
Note: CAT died very 
have had diarrhea 2-
felt nauseous 2 time
Chickens

2. REST endpoint

[antgonza]

I can confirm this issue and figured out why this is happening: the REST endpoint is not calling load_template_to_dataframe which validates and clean up the metadata to create the dataframe.

We basically would need to replace this line here:

data = qiita_db.metadata_template.util.load_template_to_dataframe(self.request.body)

[wasade]

Okay cool. Let's see if #3152 does it