Refactor workflow

[rneher]

This PR refactors the old work flow that was spread out over files like Snakefile_base, Snakefile_WHO, etc into a structure that follows the recommendations. In particular, it aims at splitting the workflows into substeps that have defined input and output files to minimize the inter-dependencies of different parts of the workflow. Complex input functions are avoided as much as possible to increase legibility.

TODO

• Rename new build outputs to match the current Auspice filenames (for example, flu_seasonal_h3n2_2y_ha.json from the refactored workflow needs to be flu_seasonal_h3n2_ha_2y.json to match the live builds). Some options include:
  • Allow the build config to define an auspice_name that parallels but remains decoupled from the build_name so the build name could be flu_seasonal_{lineage}_6m but the auspice_name could be flu_seasonal_{lineage}_{{segment}}_6m. There is some precedent for this approach in the auspice_json_prefix option in the ncov workflow.
  • Add a custom rule to rename public build outputs (e.g., with cp) like we do for dated JSONs in ncov.
  • Update on Jan 25, 2023: This is now implemented for the public builds through custom renaming rule plus an auspice_name build parameter. To get all of the properly named public builds we just need to run: snakemake --configfile profiles/nextstrain-public.yaml -- all_public. This all_public target will generate the expected files in the auspice_renamed/ directory.
• Root trees with the alignment reference and then remove that root sequence after rooting. Some options include:
  • Implement proposal in Include option to prune outgroup in augur tree or augur refine augur#340
  • Copy custom rule to prune outgroup after refine step from the workflow on master branch
  • Update on Jan 27, 2023 by @huddlej: I ended up copying and modifying the prune outgroup script to prune the outgroup using the same reference FASTA used for alignment. I also moved the pruning immediately after tree building and removed the midpoint rooting logic from the "sanitize" script to avoid undoing the rooting that happens during tree building.
• Port rules to build Auspice JSONs for private.nextflu.org
• Verify that public and private builds run and deploy as expected

Running public builds (40 trees)

nextstrain build \
  --aws-batch \
  --detach \
  --cpus 36 \
  --memory 72gib \
  . \
  all_public \
  -p \
  --configfile profiles/nextstrain-public.yaml

Output is in auspice_renamed/.

Running private.nextflu.org builds (80 trees)

nextstrain build \
  --aws-batch \
  --detach \
  --cpus 36 \
  --memory 72gib \
  . \
  all_who \
  -p \
  --configfile profiles/private.nextflu.org.yaml

Output is in auspice-who/.

Future TODOs

• Copy all reference sequences into their respective directories and split them into FASTA and GFF files (using a script to convert GenBank to FASTA + GFF)

[huddlej]

Thank you for putting this together, @rneher! This is already a major improvement over the current workflow both in the separation of concerns into separate rules and the abstraction of the build configs into separate config files. I like that we can use the powerful array builds approach when we want, but that the simpler flat config approach works just as well.

My biggest structural questions have to do with who we think will use this workflow:

1. Do we want to support this as a canonical seasonal flu workflow that other public health organizations can run on their own data like we do for SARS-CoV-2?
2. Can we support definition of specific input sequences and metadata in the config, when fauna inputs are not an option (both for our own internal uses and for external users).

In reviewing this and what a user's experience would be like working from GISAID, I noticed that we can now download ~20,000 sequences at once from GISAID. This means most people could run some powerful analyses from a couple of simple GISAID downloads. For an H3N2 HA and NA analysis, for example, this would produce two FASTA files like gisaid_epiflu_sequence_2015-2018_ha.fasta and gisaid_epiflu_sequence_2015-2018_na.fasta that would need to run through augur parse to get the separate sequences and metadata. Can we support this simple use case in this workflow even for our own work?

It would also be nice to eventually move all of the configuration bits out of the Snakefiles and into config files. For example, clock rates, gene names, etc. are all candidates to move into config files where they are more visible, easier for non-Snakemake experts to modify, etc.

[huddlej]

I didn’t understand why we needed this additional syntax/functionality for composing build directories. Is this the approach to creating "WHO" and "live" builds?

[rneher]

I this should be rethought. this was mostly my quick way of hacking together various targets for testing. But being able to specify an auspice dir has been useful to run different analysis and profile in the same directory and not struggle with clean removing stuff you want to keep.

[huddlej]

I like that fauna is abstracted away in this implementation.

Do we want to allow users (including us) to define their own input sequences/metadata as we do in ncov? This would provide a path for public health partners to use our workflow from GISAID downloads without needing access to fauna. The bigger question is: do we want to support this as a public Nextstrain workflow for flu?

[huddlej]

Is there a reason we don't pair strains by segment earlier than this in the workflow? It seems like the metadata merge would be the earliest place to ensure we have the same strains. Then we wouldn't need this additional rule to sanitize the trees...

[huddlej]

Just a note that the new script for joining metadata (scripts/join_metadata.py) supports inner joins of the data, so we could choose to limit analyses to strains with all requested segments at the beginning of the workflow. The steps that follow that join and work from the joined metadata should always pull the same strains for tree building...

[huddlej]

These top-level boolean keys like titer-models don't allow us to limit which segments to apply the models to from the config, so we'll run titer models on NA, PB1, etc. Could we move these types of parameters into the build definitions (like the vaccines option below) instead so we have more control over which builds use which annotations?

Then we can eliminate all of these additional segment tests that happen for glycosylation, LBI, etc.

[huddlej]

Instead of unpacking output from a wildcard function like this, what if we allow users to define the “titer priorities” file as a subsample input in the config file (e.g., priorities_file: "{build_name}/titer_priorities.tsv")? This would allow Snakemake to build the titer priorities on the fly, as needed, and also allow users to define their own priorities file. This would decouple the titers bit from the priorities bit.

[huddlej]

In our discussion today, we want to support users who download separate segment FASTAs from GISAID. We may need to support files like data/lineage/metadata_segment.tsv

[huddlej]

@joverlee521 made the point that we also want to support example data from open data with an example Auspice JSON so users can confirm that the basic workflow will work for them as a tutorial.

We also need to move the distance files, fitness components, etc. still.

I will try running the workflow with standard GISAID downloads and determine what minimal changes need to happen for this to work.

[huddlej]

Using this workflow more, I was surprised by this directory as the place to look for my output. The rest of our workflows use results/ with the contents of that directory depending on the pathogen workflow (zika stores a single set of outputs top-level while ncov nests build-specific outputs in subdirectories). Can we switch back to that here?