Replace join metadata and clades script with csvtk and tsv append

[j23414]

Description of proposed changes

In our effort to centralize ingest scripts, we identified an opportunity to streamline the process by replacing the join-metadata-and-clades.py script with csvtk rename and tsv-append in cases where customized score calculations are not required.

The specific columns from a nextclade TSV file to be renamed (if desired) and appended to the final metadata.tsv file are defined in a source-data/nextclade-field-map.tsv key-value file. This implementation is similar to the approach used in ncbi-source-field-map used elsewhere.

Related issue(s)

nextstrain/ingest#23

Checklist

• Checks pass
• Manual check

Manual check could look like:

git clone https://github.com/nextstrain/monkeypox.git old
git clone https://github.com/nextstrain/monkeypox.git new

cd new/ingest
git checkout replace_join_metadata_and_clades
nextstrain build . data/metadata.tsv

cd ../../old/ingest
nextstrain build . data/metadata.tsv

# Compare using daff diff since order of columns were not retained
daff diff data/metadata.tsv ../../new/ingest/data/metadata.tsv > diff.txt

[joverlee521]

Thanks for working through this @j23414! This approach makes sense to me but I'd be curious to see what others think.

I do want to note that this changes the order of columns in the final metadata TSV. That should be fine for our uses of the metadata TSV. I can't think of any downstream Augur commands that depend on column order.

There's currently an error that is blocking due to the use of the results directory and I've added other comments as well.

[joverlee521]

Thanks for fixing things up @j23414! I have one final suggestion to parameterize the --key-fields name but everything else looks good by inspection.

Also, not sure if you've seen this in this repo, but you can do a full test run of the ingest workflow with the Fetch and ingest (on branch) workflow. When you select to run on a branch with this workflow, the ingest pipeline will upload outputs to s3://nextstrain-data/files/workflows/monkeypox/branch/<branch-name>/.

[j23414]

Thanks @jover! Sorry for the delay; I've added the nextclade-key-field parameterization (cefff3b).

Thank you for highlighting the Fetch and ingest (on branch) workflow!
I'm encountering an error - ./bin/set-branch-ingest-config: No such file or directory with an exit code of 127 when I attempt to run it. Perhaps I'm overlooking something?

https://github.com/nextstrain/monkeypox/actions/runs/6475385391

Nevermind, I think that script got moved to phylogenetic/bin/set-branch-ingest-config.

[joverlee521]

Looks good to me! I think to good to merge as long as the trial run shows no difference in the output (except column order in the metadata.tsv).