From 9769311887cc4f1e75faa65a5d29f170b764da9d Mon Sep 17 00:00:00 2001 From: John SJ Anderson Date: Wed, 7 Aug 2024 14:59:29 -0700 Subject: [PATCH 1/5] Add configuration for linkchecking [#1569] --- docs/conf.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/conf.py b/docs/conf.py index 1a97a5d65..207b86455 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -148,3 +148,18 @@ def prose_list(items): 'pandas': ('https://pandas.pydata.org/docs', None), 'treetime': ('https://treetime.readthedocs.io/en/stable/', None), } + +# -- Linkchecking ------------------------------------------------------------ + +## NOTE: for both sets of regular expressions that follow, the +## underlying linkchecker code uses `re.match()` to apply them to URLs +## — so there's already an implicit "only at the beginning of a +## string" matching happening, and something like a plain `r'google'` +## regular expression will _NOT_ match all google.com URLs. +linkcheck_ignore = [] +linkcheck_anchors_ignore_for_url = [ + # Github uses anchor-looking links for highlighting lines but + # handles the actual resolution with Javascript, so skip anchor + # checks for Github URLs: + r'^https://github\.com', +] From d1627bca1a6fc6a27e6b33c88275ba0d1a2726c6 Mon Sep 17 00:00:00 2001 From: John SJ Anderson Date: Wed, 7 Aug 2024 17:21:05 -0700 Subject: [PATCH 2/5] Convert "migrating from v5 to v6" to RST [#1569] --- docs/releases/migrating-v5-v6.md | 534 --------------------- docs/releases/migrating-v5-v6.rst | 751 ++++++++++++++++++++++++++++++ 2 files changed, 751 insertions(+), 534 deletions(-) delete mode 100644 docs/releases/migrating-v5-v6.md create mode 100644 docs/releases/migrating-v5-v6.rst diff --git a/docs/releases/migrating-v5-v6.md b/docs/releases/migrating-v5-v6.md deleted file mode 100644 index 7568617d9..000000000 --- a/docs/releases/migrating-v5-v6.md +++ /dev/null @@ -1,534 +0,0 @@ -# Migrating from augur v5 to v6 - - - - - - -Augur is a versatile bioinformatics toolkit in its own right, but it is developed in conjunction with [Auspice](https://nextstrain.github.io/auspice), the interactive visualisation tool behind what you see on [nextstrain.org](https://nextstrain.org). -Because of this, build pipelines often finish with `augur export` which produces JSONs that Auspice can visualise. - -As Auspice grows and improves, the format of the JSON files is changing too, to allow more functionality and flexibility. This means Augur's export function needs to change as well. -This page details how the new `augur export v2` works and what you need to do to start using it. - -Some important points: -* 'Old' JSON files (v1, made with Augur v5) still work with the new (v2) Auspice for the time being - but may not be supported in future versions. -* When you upgrade to Augur v6, `augur export` will no longer work. You'll need to specify which version of Auspice (v1 or v2) your JSON files should be. - - ---- - -* [Compatibility between Augur & Auspice versions](#compatibility-between-augur-auspice-versions) -* [Motivation behind changing JSON formats](#motivation-behind-changing-json-formats) -* [I just need my old run to work _right now_!](#i-just-need-my-old-run-to-work-right-now) -* [Terminology](#terminology) -* [Using command-line options to customise the visualisation](#using-command-line-options-to-customise-the-visualisation) -* [Using a config file to customise the visualisation](#using-a-config-file-to-customise-the-visualisation) -* [Prettifying metadata fields](#prettifying-metadata-fields) - ---- -## Compatibility between Augur & Auspice versions - -Augur v5 simply used `augur export` to produce two JSONs (tree + meta) for Auspice v1 - so we'll call these JSONs "v1" JSONs. - -The new Augur (v6) can still create "v1" JSONS, but can also create JSONs that work with the latest [Auspice release](https://nextstrain.github.io/auspice/releases/v2) - Auspice v2. The new format combines the tree and metadata into one JSON, which we'll call a "v2" JSON. - -[This page](auspice-compatibility) has the most up-to-date compatibility information between different Augur and Auspice versions. - -We understand how important backwards compatibility is - so for the time being "v1" JSONs will continue to work with Auspice v2. -However, we recommend switching to v2 JSONs - they have more features, are easier to work with, and future versions of Auspice may not support v1 JSONs! - - ---- - -## Motivation behind changing JSON formats - -With the release of [Auspice v2](https://nextstrain.github.io/auspice/releases/v2), we made some breaking changes to the JSON schema. -Why change formats? We were motivated by: - -* **Compactness**: Tree and Meta JSON files are now combined, so you only have to worry about one output file -* **Flexibility**: The new v2 JSONs allow us flexibility to include more features and data, and will let us move towards getting in line with existing conventions like GFF and BibTex -* **Ease of use**: Users commonly got confused by the 'config' file. For basic runs you can now specify everything you need to see your data right in the command-line - no 'config' file needed! For more advanced exports, you can still specify a config file with more detail. (See "[Using a Config File](#using-a-config-file-to-customise-the-visualisation)") - - ---- -## I just need my old run to work _right now_! - -We completely understand you may not have time to make the change right this second, and that there's nothing more frustrating than having a run break right before a presentation or deadline! - -If you want to keep using the old version _for now_, replace `augur export` with `augur export v1` - everything else remains the same. -You can use Auspice v1 or Auspice v2 (see [compatibility](auspice-compatibility)). - -To use the new version, use `augur export v2`. -You'll need to make a few small changes, but you'll be future-proofing your runs. -_(Future you will thank past you!)_ - ---- -## What needs changing to use `augur export v2`? - -> _Helpful hint: You can always get a full overview of the arguments for export v2 with `augur export v2 --help`_ - -### What's the same - -You still pass in your tree, metadata, and node-data files with `--tree`, `--metadata`, and `--node-data` - just like in `export v1`. -Similarly, you can pass in files containing colors, latitude, and longitude data using `--colors` and `--lat-longs`, respectively. - -### Different outputs - -Instead of specifying two output files (`--output-tree` and `--output-meta`) you now only need to specify one with `--output`. -For example, if your old files were `auspice/virus_AB_tree.json` and `auspice/virus_AB_meta.json`, you might want to call the single output `auspice/virus_AB.json` - or if you want to tell it apart from your v1 export, you might call it `auspice/virus_ABv2.json`. - -To export the reference sequence relative to which mutations have been identified, specify the `--include-root-sequence` flag. -This flag writes a JSON whose name is relative to the stem of the main output JSON. -For VCF input, this file will contain the reference sequence to which the VCF is mapped. -For example, if the main output is called `auspice/virus_AB.json`, the root sequence will be saved to `auspice/virus_AB_root-sequence.json`. - -### Other changed arguments - -The `--tree-name` argument has been removed, as auspice v2 no longer uses this. -See [the auspice docs](https://nextstrain.github.io/auspice/advanced-functionality/second-trees) for more information about how second trees are specified and displayed. - -### Command Line Options instead of (or in addition to) a config file - -One of the biggest changes in `augur export v2` is that you can pass much more in using the command-line, meaning 'config' files are no longer required. The 'config' or 'Auspice config' file defines a number of visualisation settings such as title, default displays, and which colorings to use. However, it's been a source of pain for many users! - -Many of these things can now be passed in by the command-line, but some options are only possible using the config file. You can always continue to put most things in the config file if you prefer. If you want to use a [config file](#using-a-config-file-to-customise-the-visualisation), you can pass this in with `--auspice-config`, but the format of this has changed (see the link). - -It's important to note that generally _any command line options you use will override the same option in your config file_. - -### Coloring traits is smarter - -Previously, anything you wanted to color by had to be in the config file. You always had to include a "gt" and "num_date" entry, and remember to add anything new to the file. - -We've made this smarter - `augur export v2` now automatically detects some traits and you can specify others on the command line. You can also control the color options in more detail using a config file. - -We'll cover how coloring works on the [command line](#id1) and how it works in [config files](#colorings) in more detail below. - -### Traits display exactly how you want - -Previously, auspice tried to make traits and locations look 'pretty' by auto-capitalizing them and removing underscores (which were required in multi-word traits). Auspice no longer does this for v2 JSONS, so you'll need to ensure your traits look exactly how you want them to display in auspice. You can read more about that [here](#prettifying-metadata-fields). - ---- -## Terminology - -#### Traits -Traits is the general term for certain data associated with nodes in the tree, for example "country", "serotype", or "age". -These may have been inferred for internal nodes by Augur functions like `augur traits` (confusingly named!) and `augur clades`, or they may only be available for tips and provided by the metadata TSV file. - -### Geographic Traits -Certain traits have a geographic interpretation, e.g. "country". -Auspice will attempt to display these traits on a map (and provide a drop-down to switch between them if there are more than one). - -> _Make sure that these have corresponding entry in the lat-longs TSV file supplied to `export`. See how to do this [here](https://docs.nextstrain.org/en/latest/guides/bioinformatics/lat_longs.html)._ - - - ---- -## Using command-line options to customise the visualisation - -As mentioned above, you can now replace most of the functionality of the "Auspice config" file with command line options. -We hope that for most users this means the config file isn't necessary (but it's always there is you need its advanced functionality). - -> Remember that generally _any command line options you use will override the same option in your config file_. - - -### Title - -Set the title displayed by Auspice via `--title` (previously this was the "title" field in your v1 config file). -If running directly from the command line, put your title in quotes (ex: `--title "Phylodynamics of my Pathogen"`). -If you are using Snakemake and passing the value using `params`, you'll need to double-quote the title using single and double quotes. For example: - -```python -params: - title = "'Phylodynamics of my Pathogen'" -shell: - "augur export v2 --title {params.title} ..." -``` - - -### Maintainers - -The maintainer(s) are displayed in the footer of Auspice and may have associated links. -These can be specified with `--maintainers` and you can now have more than one maintainer associated with your run. -Previously this was set by the "maintainer" field in your v1 config file and was limited to a single entry. - -If running directly from the command line, put each maintainer in quotes (ex: `--maintainers "Jane Doe" "Ravi Kupra"`). -If you have a URL associated with a maintainer (completely optional), then you can add them like so: - -``` ---maintainers "Jane Doe " "Ravi Kupra If you want to display the frequencies panel, you must specify "frequencies" _and_ ensure a tip frequency file is available for `auspice` to access. - - -### Traits - -_([What's a trait?](#traits))_ Traits will become coloring options in Auspice. Some are automatically included, and some can be defined on the command line. -The following rules are followed for which traits will be exported: - -1. Genotype and date (if present) are always automatically included as coloring options - you don't need to include them. -_(Previously these were "gt" and "numdate" in the "color_options" section of your v1 config file.)_ - -2. Traits contained in the node-data JSONs handed to `augur export` (using `--node-data`) will automatically be included. -These are often generated from the Augur commands `traits`, `clades` or `seqtraits`. - -3. Traits present in the metadata file can be included by specifying them with `metadata-color-by` (e.g: `--metadata-color-by country age host`). _(These must match column names of your metadata file.)_ - -The changes hopefully make things a little easier to use -- previously, if you had run `augur clades`, you had to remember to add `clade_membership` to the config file, and if you'd run `augur seqtraits` you had to add every resulting option. -Now, they'll be automatically included. -If you don't want them as a coloring option, don't pass in the files. - - -> _Note: You can't specify the title or type of a colouring option using just command-line - but `export v2` will make its best guess using the following rules:_ -_Excluding missing data, if a trait contains only 'True', 'False', 'Yes', 'No', '0' or '1', it will be set to 'boolean.'_ -_If it contains only numbers (integers and/or decimals), it will be set to 'continuous.'_ -_Otherwise, it will be set as 'discrete.'_ -_If you want to have more control over how your trait is interpreted, you should use a config file (see [below](#using-a-config-file-to-customise-the-visualisation))._ - - -### Geographic Traits - -Specify these traits using `--geo-resolutions`, e.g. `--geo-resolutions country region`. -Previously these were defined by the "geo" field in your v1 config file. - - -### What's not possible to set without a config file - -The command line arguments cover everything you need to get a basic run working in `augur` and `auspice`. -However, there are still some features that offer more options or are only available when you use a config file. - -Currently, using command line arguments: - -* It is not possible to set the default view options using only command-line arguments in `export v2`. You can read more about the defaults (and how to change them using a config file) [here](#display-defaults). -* When using `export v2` with only command-line arguments, every trait that's a coloring option and is either categorical or boolean will automatically be available to filter by. Find out how to specify what is a filter using a config file [here](#filters). - ---- -## Using a config file to customise the visualisation - -Traditionally you had to use an "Auspice config file" to customise the visualisation. -This is still available as an option, but you can now choose between exporting using just the command-line, or using a combination of the command-line and config file. - ->_Anything you can specify using the command-line arguments above can be done using a config file instead._ - -This section will detail the config file provided to `augur export v2` by the `--auspice-config` argument. -The format of the new config file **differs slightly** from previous versions of Augur. -If you try to use a previous version of the config file it _should mostly_ still work, but will print out warnings where keys have changed. - - -### Config file priority - -It is important to remember that if you set an option both in the config file _and_ in the command line, **the command line option will override the config file option**. -For example, if you set `"title"` in your config file as "A Title About Apples", and then import this config file using `--auspice-config` _and_ use `--title "Better Title Befitting Bears"`, the title displayed by Auspice will be "Better Title Befitting Bears". -To use the one in the config file, don't use `--title` in the command line. - - -There are a couple of exceptions to this: - -* There is no way to set default display views using command line only, so using "display_defaults" in your config file will set this. -* There is no way to modify the default filters displayed when using command line only, so using "filters" in your config file will set this. -* If you set color-by options in command-line using `--metadata-color-by` _and_ pass in a config file, only the things listed in `--metadata-color-by` will be coloring options, but if they have a 'title' and 'type' set in the config file, these will be used. - - -### Config file format - -The config file is a JSON file, and as such it's important that everything in your config file is enclosed in one pair of curly brackets. -These can be on a separate line at the very top and very bottom of your file. -Syntax is important - if you are getting errors, ensure all your brackets and quotation marks match up, and that commas separate items in the same pair of brackets. - -Export v2 config files are generally very similar to export v1, _but there are a few changes_. They are explained in detail below, or you can see [an example of converting a v1 config to v2](#updating-your-config-file). For more details, see [the complete JSON schema for v2 config files](https://github.com/nextstrain/augur/blob/-/augur/data/schema-auspice-config-v2.json). - -Here are the top-level keys of the config JSON in plain English: - - -#### title - -The title to be displayed by Auspice, unchanged from previous versions of the config file. -E.g. `"title": "Phylodynamics of my Pathogen"`. - -#### maintainers - -You can now have more than one maintainer associated with your run! -Specify one or as many maintainers as you wish via the following structure (`url`s are optional): - -``` -"maintainers": [ - {"name": "Jane Doe", "url": "www.janedoe.com"}, - {"name": "Ravi Kupra", "url": "www.ravikupra.co.uk"} -] -``` - -Previously this was the "maintainer" field in your v1 config file and used a different structure. - -#### build-url - -The build / repository URL to be displayed by Auspice, a new functionality in `augur export v2`, e.g. `"build_url": "https://github.com/nextstrain/zika"`. -This is an optional field. - -#### panels - -Optional and unchanged from previous versions of the config file, this defines the panels that Auspice will display. -If not set, Auspice will by default try to show the tree, map, and entropy panels, if data is available. -Options are "tree", "map", "entropy", and "frequencies" (e.g: `"panels": ["tree", "map"]`). - -> If you want to display the frequencies panel, you must specify "frequencies" _and_ ensure a tip frequency file is available for `auspice` to access. - -#### colorings - -These are a list of the traits which Auspice should display as options to color the tree & map. -In previous versions of the config file this was "color_options" and the current structure is similar, but hopefully easier to understand! - -For each trait you include, you can define: -* A required "key", which is used to lookup the values via node-data JSONs or other provided metadata. -* An optional "title" which will shown by Auspice when referring to this trait -- for instance you may have a trait called "ab1" which you want to show as "Age bracket 1" in the drop-down menus, legend, and filter. -* An optional, but highly recommended "type" which can be either 'ordinal', 'boolean', 'continuous', or 'categorical'. If you don't provide a type, augur will try to guess it (see how it guesses [here](#id1)). - -Unless you want to change the name displayed, you _no longer_ need to include `gt`, `num_date`, `clade_membership`, or `augur seqtraits` output (like clade or drug resistance information) in your config file - if that information is present, it will automatically be included. To exclude it, don't pass in the corresponding file to `--node-data`. - -> _Remember that if you are using `--metadata-color-by` on the command-line, only the traits given there will be color-by options!_ -_To include everything in your config file, don't use `--metadata-color-by`, but include all traits you want as coloring options in "colorings" in the config file._ - ->_Put another way, if a trait is listed in `--metadata-color-by` and not in the config, it will be included._ -_If a trait is in the config but not in `--metadata-color-by` it will be excluded._ -_If a trait is in both, but has `"title"` and `"type"` information in the config file, this information _will_ be used by export v2._ - -In short, if using a config file and the command line, ensure everything you want as a coloring option is in `--metadata-color-by`. -You only need to also include it `"colorings"` in the config file if you want to set the `"title"` and/or `"type"`. - -#### geo_resolutions - -This specifies the geographical traits you want Auspice to use. You can pass this in the same way as in the v1 config file, or you can now specify a title to be displayed by option, using a slightly different structure. - -For example, for many users, these might be "country" and "region", i.e. `"geo_resolutions: ["country", "region"]`. If you want to give them new titles, use the format `"geo_resolutions": [{"key": "country", "title": "Areas"}, {"key": "region", "title": "Global"}]`. - -You can also mix the two, if you just want a title for one location: `"geo_resolutions": [ {"key": "country", "title": "Areas"}, "region"]` - - -#### filters - -This specifies which traits you can filter by in Auspice. -E.g. `"filters": ["country", "region", "symptom", "age"]`. -If you don't include this option in your config file, all non-continuous traits that are coloring options will be included as filters. -If you don't want any filter options, include this option with an empty list, i.e. `"filters": []`. -This is the same as the "filters" field in previous config files, but the behavior has changed slightly. - - -#### display_defaults - -This allows you to specify the default view that users will see when they visualise the data in Auspice. -There are five options you can set here -- note they are similar to those in the previous config files but we have now standardised them to snake_case: - -* `geo_resolution` - Sets which of the "geo_resolutions" should be shown. _Default is 'country'_ -* `color_by` - Sets what trait should be used for coloring. _Default is 'country'_ -* `distance_measure` - Sets whether tree branch lengths are in 'time' or 'divergence'. -Options are `num_date` (time, default if available) or `div` (divergence). -* `layout` - Sets how the tree is visualised. -Options are `rect` (rectangular, default), `radial`, `unrooted`, and `clock`, corresponding to the four options normally shown on the left in Auspice. -* `map_triplicate` - Sets whether the map is extended / wrapped around, which can be useful if transmissions are worldwide. Set to `true' or 'false'. _Default 'false'_ - - - -### Config file examples - -Here is an example of how all of the above options would fit into a config file: -```json -{ - "title": "Phylodynamics of my Pathogen", - "maintainers": [ - {"name": "Jane Doe", "url": "www.janedoe.com"}, - {"name": "Ravi Kupra", "url": "www.ravikupra.co.uk"} - ], - "build_url": "https://github.com/nextstrain/zika", - "colorings": [ - { - "key": "age", - "title": "Host age", - "type": "continuous" - }, - { - "key": "hospitalized", - "type": "boolean" - }, - { - "key": "country", - "type": "categorical" - }, - { - "key": "region", - "type": "categorical" - } - ], - "geo_resolutions": [ - {"key":"country", "title": "Areas"}, - "region" - ], - "panels": ["tree", "map"], - "filters": ["country","region","symptom","age"], - "display_defaults": { - "color_by": "symptom", - "geo_resolution": "region", - "distance_measure": "div", - "map_triplicate": "true" - } -} -``` - - -If you want some examples of the new config files used in practice, you can see some in these builds: -* _TO DO_ - -### Updating your config file - -It's fairly easy to convert old export v1 config files to work with export v2. - -Here's an export v1 config file on the left, and an export v2 config file on the right. We've tried to line them up to highlight the differences: - -
-
-Export v1 config: - 
-{
-  "title": "Phylodynamics of Virus A",
-  "color_options": {
-    "gt": {
-      "menuItem": "genotype",
-      "legendTitle": "Genotype",
-      "type": "discrete",
-      "key": "genotype"
-    },
-    "num_date": {
-      "menuItem": "date",
-      "legendTitle": "Sampling date",
-      "type": "continuous",
-      "key": "num_date"
-    },
-    "age": {
-      "menuItem": "Host age",
-      "legendTitle": "Host Age (years)",
-      "type": "continuous",
-      "key": "age"
-    },
-    "host": {
-      "menuItem": "Animal",
-      "legendTitle": "Animal",
-      "type": "discrete",
-      "key": "host"
-    },
-    "country": {
-      "type": "discrete"
-    },
-    "region": {
-      "type": "discrete"
-    },
-    "clade_membership":{
-      "menuItem": "Clade",
-      "legendTitle": "Clade",
-      "type": "discrete",
-      "key": "clade_membership"
-    }
-  },
-  "geo": [
-    "country",
-    "region"
-  ],
-  "maintainer": [
-    "Hanna Kukk, Mohammad Fahir",
-    "http://vamuzlab.org"
-  ],
-  "filters": [
-    "country", "region"
-  ],
-  "defaults": {
-    "layout": "unrooted",
-    "colorBy": "age"
-  }
-}
-
-
-
-Export v2 config: - 
-{
-  "title": "Phylodynamics of Virus A",
-  "colorings": [












-    {
-      "key": "age",

-      "title": "Host age",

-      "type": "continuous"

-    },
-    {
-      "key": "host",

-      "title": "Animal",

-      "type": "categorical"

-    },
-    {
-      "key": "country",

-      "type": "categorical"
-    },
-    {
-      "key": "region",

-      "type": "categorical"
-    }






-  ],
-  "geo_resolutions": [
-      "country",
-      "region"
-    ],
-  "maintainers": [
-    {"name": "Hanna Kukk", "url": "http://vamuzlab.org"},
-    {"name": "Mohammad Fahir", "url": "http://mfahir.co.uk"}
-  ],
-  "filters": [
-    "country", "region"
-  ],
-  "display_defaults": {
-    "layout": "unrooted",
-    "color_by": "age"
-  }
-}
-
-
-
- -### Vaccine choices -In previous versions of augur, certain strains could be defined in the config file as `vaccine_choices` (auspice would display this as a cross over the tip in the tree). -This functionality is now specified via a node-data JSON (see the v6 release notes). - -## Prettifying metadata fields - -In Auspice v1, we automatically 'prettified' many metadata values. For example, a country value of 'new_zealand' would display as 'New Zealand', and a metadata column called 'age_range' would display as 'Age Range'. - -This worked well most of the time, but meant that users couldn't intentionally keep underscores or lower-case values. It also meant we had to detect exception cases like turning 'usa' into 'USA' rather than 'Usa'. - -In Auspice v2, all values are now displayed exactly as they arrive, allowing users to ensure every gene and abbreviation displays just as it should. However, this means that you should ensure your data looks exactly how you'd like it to display - change any 'new_zealand's in your metadata to 'New Zealand'! - -Don't forget to also change them in any custom lat-long and/or coloring files you are using. We've also become stricter about the format of the files that pass in color and lat-long information. Previously, it didn't matter if columns were separated by spaces or tabs - now, they must be separated by tabs. - -You can find out more about how to add [custom coloring](https://docs.nextstrain.org/en/latest/guides/bioinformatics/colors.html) and [lat-long](https://docs.nextstrain.org/en/latest/guides/bioinformatics/lat_longs.html) values. - -If you use the command `parse` to generate a metadata table from fields in a fasta header, you can use the flag `--prettify-fields` to apply some prettifying operations to specific metadata entries, see the documentation [`parse`](/usage/cli/parse). diff --git a/docs/releases/migrating-v5-v6.rst b/docs/releases/migrating-v5-v6.rst new file mode 100644 index 000000000..45b2e7678 --- /dev/null +++ b/docs/releases/migrating-v5-v6.rst @@ -0,0 +1,751 @@ +============================= +Migrating from augur v5 to v6 +============================= + +Augur is a versatile bioinformatics toolkit in its own right, but it +is developed in conjunction with `Auspice +`__, the +interactive visualisation tool behind what you see on `nextstrain.org +`__. Because of this, build pipelines often +finish with ``augur export`` which produces JSONs that Auspice can +visualise. + +As Auspice grows and improves, the format of the JSON files is +changing too, to allow more functionality and flexibility. This means +Augur's export function needs to change as well. This page details how +the new ``augur export v2`` works and what you need to do to start +using it. + +Some important points: + +* 'Old' JSON files (v1, made with Augur v5) still work with the new + (v2) Auspice for the time being - but may not be supported in future + versions. +* When you upgrade to Augur v6, ``augur export`` will no longer work. + You'll need to specify which version of Auspice (v1 or v2) your JSON + files should be. + +---- + +.. contents:: + :local: + :depth: 1 + +---- + +Compatibility between Augur & Auspice versions +---------------------------------------------- + +Augur v5 simply used ``augur export`` to produce two JSONs (tree + +meta) for Auspice v1 - so we'll call these JSONs "v1" JSONs. + +The new Augur (v6) can still create "v1" JSONS, but can also create +JSONs that work with the latest `Auspice release +`__ - +Auspice v2. The new format combines the tree and metadata into one +JSON, which we'll call a "v2" JSON. + +:doc:`This page <./auspice-compatibility>` has the most +up-to-date compatibility information between different Augur and +Auspice versions. + +We understand how important backwards compatibility is - so for the +time being "v1" JSONs will continue to work with Auspice v2. However, +we recommend switching to v2 JSONs - they have more features, are +easier to work with, and future versions of Auspice may not support v1 +JSONs! + + +Motivation behind changing JSON formats +--------------------------------------- + +With the release of `Auspice v2 +`__, +we made some breaking changes to the JSON schema. Why change formats? +We were motivated by: + +* **Compactness**: Tree and Meta JSON files are now combined, so you + only have to worry about one output file +* **Flexibility**: The new v2 JSONs allow us flexibility to include + more features and data, and will let us move towards getting in line + with existing conventions like GFF and BibTex +* **Ease of use**: Users commonly got confused by the 'config' file. + For basic runs you can now specify everything you need to see your + data right in the command-line - no 'config' file needed! For more + advanced exports, you can still specify a config file with more + detail. (See ":ref:`Using a Config File `") + +I just need my old run to work *right now*! +------------------------------------------- + +We completely understand you may not have time to make the change +right this second, and that there's nothing more frustrating than +having a run break right before a presentation or deadline! + +If you want to keep using the old version *for now*, replace ``augur +export`` with ``augur export v1`` - everything else remains the same. +You can use Auspice v1 or Auspice v2 (see :doc:`compatibility <./auspice-compatibility>`). + +To use the new version, use ``augur export v2``. You'll need to make a +few small changes, but you'll be future-proofing your runs. *(Future +you will thank past you!)* + +What needs changing to use ``augur export v2``? +----------------------------------------------- + +.. note:: Helpful hint: You can always get a full overview of the + arguments for export v2 with ``augur export v2 --help`` + +What's the same +*************** + +You still pass in your tree, metadata, and node-data files with +``--tree``, ``--metadata``, and ``--node-data`` - just like in ``export v1``. +Similarly, you can pass in files containing colors, latitude, and +longitude data using ``--colors`` and ``--lat-longs``, respectively. + +.. _different-outputs: + +Different outputs +***************** + +Instead of specifying two output files (``--output-tree`` and +``--output-meta``) you now only need to specify one with ``--output``. +For example, if your old files were ``auspice/virus_AB_tree.json`` and +``auspice/virus_AB_meta.json``, you might want to call the single +output ``auspice/virus_AB.json`` - or if you want to tell it apart +from your v1 export, you might call it ``auspice/virus_ABv2.json``. + +To export the reference sequence relative to which mutations have been +identified, specify the ``--include-root-sequence`` flag. This flag +writes a JSON whose name is relative to the stem of the main output +JSON. For VCF input, this file will contain the reference sequence to +which the VCF is mapped. For example, if the main output is called +``auspice/virus_AB.json``, the root sequence will be saved to +``auspice/virus_AB_root-sequence.json``. + +Other changed arguments +*********************** + +The ``--tree-name`` argument has been removed, as auspice v2 no longer +uses this. See `the auspice docs +`__ +for more information about how second trees are specified and +displayed. + +Command Line Options instead of (or in addition to) a config file +***************************************************************** + +One of the biggest changes in ``augur export v2`` is that you can pass +much more in using the command-line, meaning 'config' files are no +longer required. The 'config' or 'Auspice config' file defines a +number of visualisation settings such as title, default displays, and +which colorings to use. However, it's been a source of pain for many +users! + +Many of these things can now be passed in by the command-line, but +some options are only possible using the config file. You can always +continue to put most things in the config file if you prefer. If you +want to use a :ref:`config file +`, you can pass +this in with ``--auspice-config``, but the format of this has changed +(see the link). + +It's important to note that generally *any command line options you +use will override the same option in your config file*. + +Coloring traits is smarter +************************** + +Previously, anything you wanted to color by had to be in the config +file. You always had to include a ``gt`` and ``num_date`` entry, and +remember to add anything new to the file. + +We've made this smarter - ``augur export v2`` now automatically detects +some traits and you can specify others on the command line. You can +also control the color options in more detail using a config file. + +We'll cover how coloring works on the :ref:`command line +` and how it works in :ref:`config files ` in more detail +below. + +Traits display exactly how you want +*********************************** + +Previously, auspice tried to make traits and locations look 'pretty' +by auto-capitalizing them and removing underscores (which were +required in multi-word traits). Auspice no longer does this for v2 +JSONS, so you'll need to ensure your traits look exactly how you want +them to display in auspice. You can read more about that +:ref:`here `. + +----------------------------------------------------------------- + +Terminology +----------- + +.. _terminology-traits: + +Traits +****** + +Traits is the general term for certain data associated with nodes in +the tree, for example "country", "serotype", or "age". These may have +been inferred for internal nodes by Augur functions like ``augur +traits`` (confusingly named!) and ``augur clades``, or they may only be +available for tips and provided by the metadata TSV file. + +Geographic Traits +***************** + +Certain traits have a geographic interpretation, e.g. "country". +Auspice will attempt to display these traits on a map (and provide a +drop-down to switch between them if there are more than one). + +.. note:: Make sure that these have corresponding entry in the + lat-longs TSV file supplied to ``export``. See how to do this + `here `_. + +.. _command-line: + +Using command-line options to customise the visualisation +--------------------------------------------------------- + +As mentioned above, you can now replace most of the functionality of +the "Auspice config" file with command line options. We hope that for +most users this means the config file isn't necessary (but it's always +there is you need its advanced functionality). + +.. note:: Remember that generally *any command line options you use + will override the same option in your config file*. + +Title +***** + +Set the title displayed by Auspice via ``--title`` (previously this +was the "title" field in your v1 config file). If running directly +from the command line, put your title in quotes (ex: ``--title +"Phylodynamics of my Pathogen"``). If you are using Snakemake and +passing the value using ``params``, you'll need to double-quote the +title using single and double quotes. For example: + +.. code-block:: python + + params: + title = "'Phylodynamics of my Pathogen'" + shell: + "augur export v2 --title {params.title} ..." + +Maintainers +*********** + +The maintainer(s) are displayed in the footer of Auspice and may have +associated links. These can be specified with ``--maintainers`` and +you can now have more than one maintainer associated with your run. +Previously this was set by the "maintainer" field in your v1 config +file and was limited to a single entry. + +If running directly from the command line, put each maintainer in +quotes (ex: ``--maintainers "Jane Doe" "Ravi Kupra"``). If you have a +URL associated with a maintainer (completely optional), then you can +add them like so: + +.. code-block:: shell + + --maintainers "Jane Doe " "Ravi Kupra `) Traits will become coloring options +in Auspice. Some are automatically included, and some can be defined +on the command line. The following rules are followed for which traits +will be exported: + +1. Genotype and date (if present) are always automatically included as + coloring options - you don't need to include them. *(Previously + these were "gt" and "numdate" in the "color_options" section of + your v1 config file.)* +2. Traits contained in the node-data JSONs handed to ``augur export`` + (using ``--node-data``) will automatically be included. These are + often generated from the Augur commands ``traits``, ``clades`` or + ``seqtraits``. +3. Traits present in the metadata file can be included by specifying + them with ``metadata-color-by`` (e.g: ``--metadata-color-by country + age host``). *(These must match column names of your metadata + file.)* + +The changes hopefully make things a little easier to use -- +previously, if you had run ``augur clades``, you had to remember to add +``clade_membership`` to the config file, and if you'd run ``augur +seqtraits`` you had to add every resulting option. Now, they'll be +automatically included. If you don't want them as a coloring option, +don't pass in the files. + + +.. note:: You can't specify the title or type of a colouring option + using just command-line - but ``export v2`` will make its + best guess using the following rules: Excluding missing + data, if a trait contains only 'True', 'False', 'Yes', 'No', + '0' or '1', it will be set to 'boolean.' If it contains only + numbers (integers and/or decimals), it will be set to + 'continuous.' Otherwise, it will be set as 'discrete.' If + you want to have more control over how your trait is + interpreted, you should use a config file (see :ref:`below + `). + +Geographic Traits +***************** + +Specify these traits using ``--geo-resolutions``, e.g. +``--geo-resolutions country region``. Previously these were defined by +the "geo" field in your v1 config file. + +What's not possible to set without a config file +************************************************ + +The command line arguments cover everything you need to get a basic +run working in ``augur`` and ``auspice``. However, there are still +some features that offer more options or are only available when you +use a config file. + +Currently, using command line arguments: + +* It is not possible to set the default view options using only + command-line arguments in ``export v2``. You can read more about the + defaults (and how to change them using a config file) + :ref:`here `. +* When using ``export v2`` with only command-line arguments, every trait + that's a coloring option and is either categorical or boolean will + automatically be available to filter by. Find out how to specify + what is a filter using a config file :ref:`here `. + +----------------------------------------------------------------- + +.. _using-a-config-file-to-customise-the-visualisation: + +Using a config file to customise the visualisation +-------------------------------------------------- + +Traditionally you had to use an "Auspice config file" to customise the +visualisation. This is still available as an option, but you can now +choose between exporting using just the command-line, or using a +combination of the command-line and config file. + +.. note:: Anything you can specify using the command-line arguments + above can be done using a config file instead. + +This section will detail the config file provided to ``augur export v2`` +by the ``--auspice-config`` argument. The format of the new config file +**differs slightly** from previous versions of Augur. If you try to +use a previous version of the config file it *should mostly* still +work, but will print out warnings where keys have changed. + + +Config file priority +******************** + +It is important to remember that if you set an option both in the +config file *and* in the command line, **the command line option will +override the config file option**. For example, if you set ``"title"`` +in your config file as "A Title About Apples", and then import this +config file using ``--auspice-config`` *and* use ``--title "Better Title +Befitting Bears"``, the title displayed by Auspice will be "Better +Title Befitting Bears". To use the one in the config file, don't use +``--title`` in the command line. + +There are a couple of exceptions to this: + +* There is no way to set default display views using command line + only, so using "display_defaults" in your config file will set this. +* There is no way to modify the default filters displayed when using + command line only, so using "filters" in your config file will set + this. +* If you set color-by options in command-line using + ``--metadata-color-by`` *and* pass in a config file, only the things + listed in ``--metadata-color-by`` will be coloring options, but if + they have a 'title' and 'type' set in the config file, these will be + used. + +Config file format +****************** + +The config file is a JSON file, and as such it's important that +everything in your config file is enclosed in one pair of curly +brackets. These can be on a separate line at the very top and very +bottom of your file. Syntax is important - if you are getting errors, +ensure all your brackets and quotation marks match up, and that commas +separate items in the same pair of brackets. + +Export v2 config files are generally very similar to export v1, *but +there are a few changes*. They are explained in detail below, or you +can see :ref:`an example of converting a v1 config to v2 `. +For more details, see `the complete JSON schema for v2 config files +`__. + +Here are the top-level keys of the config JSON in plain English: + +title +,,,,, + +The title to be displayed by Auspice, unchanged from previous versions +of the config file. E.g. ``"title": "Phylodynamics of my Pathogen"``. + +maintainers +,,,,,,,,,,, + +You can now have more than one maintainer associated with your run! +Specify one or as many maintainers as you wish via the following +structure (The ``url`` keys are optional): + +.. code-block:: json + + "maintainers": [ + {"name": "Jane Doe", "url": "www.janedoe.com"}, + {"name": "Ravi Kupra", "url": "www.ravikupra.co.uk"} + ] + +Previously this was the "maintainer" field in your v1 config file and +used a different structure. + +build-url +,,,,,,,,, + +The build / repository URL to be displayed by Auspice, a new +functionality in ``augur export v2``, e.g. ``"build_url": +"https://github.com/nextstrain/zika"``. This is an optional field. + +panels +,,,,,, + +Optional and unchanged from previous versions of the config file, this +defines the panels that Auspice will display. If not set, Auspice will +by default try to show the tree, map, and entropy panels, if data is +available. Options are "tree", "map", "entropy", and "frequencies" +(e.g: ``"panels": ["tree", "map"]``). + +.. note:: If you want to display the frequencies panel, you must + specify "frequencies" *and* ensure a tip frequency file is + available for ``auspice`` to access. + +.. _colorings: + +colorings +,,,,,,,,, + +These are a list of the traits which Auspice should display as options +to color the tree & map. In previous versions of the config file this +was "color_options" and the current structure is similar, but +hopefully easier to understand! + +For each trait you include, you can define: + +* A required "key", which is used to lookup the values via node-data + JSONs or other provided metadata. +* An optional "title" which will shown by Auspice when referring to + this trait -- for instance you may have a trait called "ab1" which + you want to show as "Age bracket 1" in the drop-down menus, legend, + and filter. +* An optional, but highly recommended "type" which can be either + 'ordinal', 'boolean', 'continuous', or 'categorical'. If you don't + provide a type, augur will try to guess it (see how it guesses + :ref:`here `). + +Unless you want to change the name displayed, you *no longer* need to +include ``gt``, ``num_date``, ``clade_membership``, or ``augur +seqtraits`` output (like clade or drug resistance information) in your +config file - if that information is present, it will automatically be +included. To exclude it, don't pass in the corresponding file to +``--node-data``. + +.. note:: Remember that if you are using ``--metadata-color-by`` on the + command-line, only the traits given there will be color-by + options! To include everything in your config file, don't + use ``--metadata-color-by``, but include all traits you want + as coloring options in "colorings" in the config file. + + Put another way, if a trait is listed in + ``--metadata-color-by`` and not in the config, it will be + included. If a trait is in the config but not in + ``--metadata-color-by`` it will be excluded. If a trait is + in both, but has ``"title"`` and ``"type"`` information in + the config file, this information *will* be used by export v2. + +In short, if using a config file and the command line, ensure +everything you want as a coloring option is in ``--metadata-color-by``. +You only need to also include it ``"colorings"`` in the config file if +you want to set the ``"title"`` and/or ``"type"``. + +geo_resolutions +,,,,,,,,,,,,,,, + +This specifies the geographical traits you want Auspice to use. You +can pass this in the same way as in the v1 config file, or you can now +specify a title to be displayed by option, using a slightly different +structure. + +For example, for many users, these might be "country" and "region", +i.e. ``"geo_resolutions: ["country", "region"]``. If you want to give +them new titles, use the format ``"geo_resolutions": [{"key": +"country", "title": "Areas"}, {"key": "region", "title": "Global"}]``. + +You can also mix the two, if you just want a title for one location: +``"geo_resolutions": [ {"key": "country", "title": "Areas"}, "region"]`` + +.. _filters: + +filters +,,,,,,, + +This specifies which traits you can filter by in Auspice. E.g. +``"filters": ["country", "region", "symptom", "age"]``. If you don't +include this option in your config file, all non-continuous traits +that are coloring options will be included as filters. If you don't +want any filter options, include this option with an empty list, i.e. +``"filters": []``. This is the same as the "filters" field in previous +config files, but the behavior has changed slightly. + +.. _display-defaults: + +display_defaults +,,,,,,,,,,,,,,,, + +This allows you to specify the default view that users will see when +they visualise the data in Auspice. There are five options you can set +here -- note they are similar to those in the previous config files +but we have now standardised them to snake_case: + +* ``geo_resolution`` - Sets which of the "geo_resolutions" should be + shown. *Default: country* +* ``color_by`` - Sets what trait should be used for coloring. *Default: country* +* ``distance_measure`` - Sets whether tree branch lengths are in ``time`` + or ``divergence``. Options are ``num_date`` (time, default if available) + or ``div`` (divergence). +* ``layout`` - Sets how the tree is visualised. Options are ``rect`` + (rectangular, default), ``radial``, ``unrooted``, and ``clock``, + corresponding to the four options normally shown on the left in + Auspice. +* ``map_triplicate`` - Sets whether the map is extended / wrapped + around, which can be useful if transmissions are worldwide. Set to + ``true`` or ``false``. *Default: false* + +Config file examples +******************** + +Here is an example of how all of the above options would fit into a config file: + +.. code-block:: json + :linenos: + + { + "title": "Phylodynamics of my Pathogen", + "maintainers": [ + {"name": "Jane Doe", "url": "www.janedoe.com"}, + {"name": "Ravi Kupra", "url": "www.ravikupra.co.uk"} + ], + "build_url": "https://github.com/nextstrain/zika", + "colorings": [ + { + "key": "age", + "title": "Host age", + "type": "continuous" + }, + { + "key": "hospitalized", + "type": "boolean" + }, + { + "key": "country", + "type": "categorical" + }, + { + "key": "region", + "type": "categorical" + } + ], + "geo_resolutions": [ + {"key":"country", "title": "Areas"}, + "region" + ], + "panels": ["tree", "map"], + "filters": ["country","region","symptom","age"], + "display_defaults": { + "color_by": "symptom", + "geo_resolution": "region", + "distance_measure": "div", + "map_triplicate": "true" + } + } + +If you want some examples of the new config files used in practice, +you can see some in these builds: + +.. raw:: html + +
  • TO DO
+ + + +.. _updating-your-config-file: + +Updating your config file +************************* + +It's fairly easy to convert old export v1 config files to work with +export v2. + +Here's an export v1 config file on the left, and an export v2 config +file on the right. We've tried to line them up to highlight the +differences: + ++-----------------------------------------------+-----------------------------------------------------------------+ +| Export v1 config: | Export v2 config: | +| | | +| .. code-block:: json | .. code-block:: json | +| :linenos: | :linenos: | +| | | +| { | { | +| "title": "Phylodynamics of Virus A", | "title": "Phylodynamics of Virus A", | +| "color_options": { | "colorings": [ | +| "gt": { | | +| "menuItem": "genotype", | | +| "legendTitle": "Genotype", | | +| "type": "discrete", | | +| "key": "genotype" | | +| }, | | +| "num_date": { | | +| "menuItem": "date", | | +| "legendTitle": "Sampling date", | | +| "type": "continuous", | | +| "key": "num_date" | | +| }, | | +| "age": { | { | +| "menuItem": "Host age", | "key": "age", | +| "legendTitle": "Host Age (years)", | "title": "Host age", | +| "type": "continuous", | "type": "continuous" | +| "key": "age" | }, | +| }, | { | +| "host": { | "key": "host", | +| "menuItem": "Animal", | "title": "Animal", | +| "legendTitle": "Animal", | "type": "categorical" | +| "type": "discrete", | }, | +| "key": "host" | { | +| }, | "key": "country", | +| "country": { | "type": "categorical" | +| "type": "discrete" | }, | +| }, | { | +| "region": { | "key": "region", | +| "type": "discrete" | "type": "categorical" | +| }, | } | +| "clade_membership":{ | ], | +| "menuItem": "Clade", | | +| "legendTitle": "Clade", | | +| "type": "discrete", | | +| "key": "clade_membership" | | +| } | | +| }, | | +| "geo": [ | "geo_resolutions": [ | +| "country", | "country", | +| "region" | "region" | +| ], | ], | +| "maintainer": [ | "maintainers": [ | +| "Hanna Kukk, Mohammad Fahir", | {"name": "Hanna Kukk", "url": "http://vamuzlab.org"}, | +| "http://vamuzlab.org" | {"name": "Mohammad Fahir", "url": "http://mfahir.co.uk"} | +| ], | ], | +| "filters": [ | "filters": [ | +| "country", "region" | "country", "region" | +| ], | ], | +| "defaults": { | "display_defaults": { | +| "layout": "unrooted", | "layout": "unrooted", | +| "colorBy": "age" | "color_by": "age" | +| } | } | +| } | } | ++-----------------------------------------------+-----------------------------------------------------------------+ + +Vaccine choices +*************** + +In previous versions of augur, certain strains could be defined in the +config file as ``vaccine_choices`` (auspice would display this as a +cross over the tip in the tree). This functionality is now specified +via a node-data JSON (see the v6 release notes). + +.. _prettifying-metadata-fields: + +Prettifying metadata fields +--------------------------- + +In Auspice v1, we automatically 'prettified' many metadata values. For +example, a country value of 'new_zealand' would display as 'New +Zealand', and a metadata column called 'age_range' would display as +'Age Range'. + +This worked well most of the time, but meant that users couldn't +intentionally keep underscores or lower-case values. It also meant we +had to detect exception cases like turning 'usa' into 'USA' rather +than 'Usa'. + +In Auspice v2, all values are now displayed exactly as they arrive, +allowing users to ensure every gene and abbreviation displays just as +it should. However, this means that you should ensure your data looks +exactly how you'd like it to display - change any 'new_zealand's in +your metadata to 'New Zealand'! + +Don't forget to also change them in any custom lat-long and/or +coloring files you are using. We've also become stricter about the +format of the files that pass in color and lat-long information. +Previously, it didn't matter if columns were separated by spaces or +tabs - now, they must be separated by tabs. + +You can find out more about how to add +`custom coloring `__ +and `lat-long `__ +values. + +If you use the command ``parse`` to generate a metadata table from +fields in a fasta header, you can use the flag ``--prettify-fields`` to +apply some prettifying operations to specific metadata entries, +:doc:`see the documentation ` for ``parse``. From df6b8ab3937f4675f3dd1551eeee6b196cee9953 Mon Sep 17 00:00:00 2001 From: John SJ Anderson Date: Wed, 7 Aug 2024 17:34:08 -0700 Subject: [PATCH 3/5] Convert "v6" to RST [#1569] --- docs/releases/v6.md | 113 ------------------------------- docs/releases/v6.rst | 156 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 156 insertions(+), 113 deletions(-) delete mode 100644 docs/releases/v6.md create mode 100644 docs/releases/v6.rst diff --git a/docs/releases/v6.md b/docs/releases/v6.md deleted file mode 100644 index 9bdbcec9d..000000000 --- a/docs/releases/v6.md +++ /dev/null @@ -1,113 +0,0 @@ -# Augur v6 Release Notes - -Augur v6 was released on 2019-12-10. -This release contains a number of changes from Augur v5, including feature additions and bugfixes. -The biggest change is related to how Augur _exports_ files for visualisation by [Auspice](https://nextstrain.github.io/auspice/). -We've written [an extensive guide](migrating-v5-v6) explaining our motivations here, what has changed, how to upgrade, and how this interfaces with Auspice. - ---- - -* [Export JSONs for specific versions of Auspice](#export-jsons-for-specific-versions-of-auspice) -* [Reference sequence output](#reference-sequence-output) -* [Change in `augur ancestral`'s arguments](#change-in-augur-ancestrals-arguments) -* [Import BEAST MCC trees](#import-beast-mcc-trees) -* [Prettifying of strings](#prettifying-of-strings) -* [Whitespace in colors and lat-longs TSVs](#whitespace-in-colors-lat-longs-tsvs) -* [Move to GFF-style annotations](#move-to-gff-style-annotations) -* [Improvements and usage of JSON validation](#improvements-usage-of-json-validation) -* [Removal of non-modular Augur and old builds](#removal-of-non-modular-augur-old-builds) -* [Test builds](#test-builds) -* [Documentation improvements](#documentation-improvements) -* [Miscellaneous](#miscellaneous) - -## Export JSONs for specific versions of Auspice -Probably the biggest (breaking) change you'll encounter is that `augur export` no longer works! -See the [migration guide](migrating-v5-v6) for a detailed explanation of this. - -Breaking change: `augur export` no longer works, and now requires a further positional argument to define which version of Auspice you wish to target. -`augur export v1` should behave the same as previous versions' `augur export`. - - - -## Reference sequence output -The export command now accepts a flag to export the reference/root sequence relative to which mutations are called, see [here](migrating-v5-v6.html#different-outputs) for more detail. - -## Change in `augur ancestral`'s arguments - -`augur ancestral`, which reconstructs mutations across a tree, now supports two forms of output and the arguments have become more descriptive. - -1. JSON output, including mutations for each branch and (inferred) ancestral sequences. -This is specified by the `--output-node-data` argument. - -2. FASTA output of reconstructed ancestral sequences. -This had previously been available for VCF-inputs, but now works for any input. -Users can ask for this output and specify a file name using `--output-sequences`. - -Deprecation warning: The argument `--output` is now deprecated. Please use `--output-node-data` instead. - -## Import BEAST MCC trees -We now have instructions and functionality to import BEAST trees, see [here](https://docs.nextstrain.org/en/latest/guides/bioinformatics/import-beast.html). - -## Prettifying of strings -Previous auspice version "prettified" metadata strings (like changing 'north_america' to 'North America'). -Auspice v2 no longer does this, see [here](migrating-v5-v6.html#prettifying-metadata-fields) for more detail. -The `parse` command now accepts and argument to apply string prettifying operations to metadata parsed from fasta headers. - - -## Whitespace in colors and lat-longs TSVs -To allow whitespace in metadata, files specifying colors and geographic locations now need to be TAB delimited. - -## Move to GFF-style annotations - -Starting with Augur v6 we now use GFF coordinates: [one-origin, inclusive] as opposed to BED coordinates. -Strands are represented by `+` or `-` rather than `1` or `0`. -Additionally, we export the `seqid`, but don't use it in Auspice. - -## Improvements and usage of JSON validation -The export command will now validate the produced JSON against the schema. - -## Removal of non-modular Augur and old builds -Augur has been a dynamic, shapeshifting beast. -It started as scripts for Nextflu, took on more and more pathogens, was refactored into "prepare" and "process" steps, and refactored again into the "modular" Augur we now have. -Earlier incarnations of Augur have now been removed from the GitHub repo (`./base/*`). - -Paralleling the different incarnations of Augur was a move to "builds" being their own self-contained repos. -We think this has been remarkably successful, and de-couples the bioinformatics tooling from a pathogen build. -With this release of Augur we've now removed these builds from the Augur GitHub repo, and the only builds that remain are the test ones. - -## Test builds - -There have been a number of test builds in the Augur repo and we have leaned heavily on them while we developed this version of Augur as well as Auspice v2. -They are all self contained within `./tests/builds` and can all be run and examined in Auspice via - -```bash -cd tests/builds -bash runner.sh # creates output in ./auspice -auspice view --datasetDir auspice -``` - -(See the [Auspice docs](https://nextstrain.github.io/auspice) for Auspice-specific questions.) - -## Documentation improvements - -Documentation has always been a bit hit-or-miss with Nextstrain projects. -We've tried to make Augur's read-the-docs documentation more comprehensive, with better flow. -This entails new sections, with each Augur command having its own page. -We've tried to use redirects to ensure that all the old links continue to work. - -## Miscellaneous - -* `augur filter`: More interpretable output of how many sequences each stage has filtered out. - -* `augur filter`: Additional flag `--subsample-seed` to seed the random number generator and thereby make subsampling reproducible. - -* `augur sequence-traits`: Numerical output as originally intended, but required an Auspice bugfix. - -* `augur traits`: Explanation of what is considered missing data & how it is interpreted. - -* `augur traits`: GTR models are exported in the output JSON for better accountability & reproducibility. - -* Errors in formatting of input files (e.g. metadata files, Auspice config files) weren't handled nicely, often resulting in hard-to-interpret stack traces. -We now try to catch these and print an error indicating the offending file. - -* Tests using Python version 2 have now been removed. diff --git a/docs/releases/v6.rst b/docs/releases/v6.rst new file mode 100644 index 000000000..641e670f7 --- /dev/null +++ b/docs/releases/v6.rst @@ -0,0 +1,156 @@ +====================== +Augur v6 Release Notes +====================== + +Augur v6 was released on 2019-12-10. This release contains a number of +changes from Augur v5, including feature additions and bugfixes. The +biggest change is related to how Augur *exports* files for +visualisation by `Auspice +`__. We've +written :doc:`an extensive guide ` explaining our +motivations here, what has changed, how to upgrade, and how this +interfaces with Auspice. + +---- + +.. contents:: + :local: + :depth: 1 + +---- + +Export JSONs for specific versions of Auspice +--------------------------------------------- + +Probably the biggest (breaking) change you'll encounter is that +``augur export`` no longer works! See the :doc:`migration guide +<./migrating-v5-v6>` for a detailed explanation of this. + +.. warning:: **Breaking change:** ``augur export`` no longer works, + and now requires a further positional argument to define + which version of Auspice you wish to target. ``augur + export v1`` should behave the same as previous versions' + ``augur export``. + +Reference sequence output +------------------------- + +The export command now accepts a flag to export the reference/root +sequence relative to which mutations are called, see :ref:`here +` for more detail. + +Change in ``augur ancestral``'s arguments +----------------------------------------- + +``augur ancestral``, which reconstructs mutations across a tree, now +supports two forms of output and the arguments have become more +descriptive. + +1. JSON output, including mutations for each branch and (inferred) + ancestral sequences. This is specified by the + ``--output-node-data`` argument. + +2. FASTA output of reconstructed ancestral sequences. This had + previously been available for VCF-inputs, but now works for any + input. Users can ask for this output and specify a file name using + ``--output-sequences``. + +.. warning:: **Deprecation warning:** The argument ``--output`` is now + deprecated. Please use ``--output-node-data`` instead. + +Import BEAST MCC trees +---------------------- + +We now have instructions and functionality to import BEAST trees, see +`here `__. + +Prettifying of strings +---------------------- + +Previous auspice version "prettified" metadata strings (like changing +'north_america' to 'North America'). Auspice v2 no longer does this, +see :ref:`here ` for more detail. The +``parse`` command now accepts and argument to apply string prettifying +operations to metadata parsed from fasta headers. + +Whitespace in colors and lat-longs TSVs +--------------------------------------- + +To allow whitespace in metadata, files specifying colors and +geographic locations now need to be TAB delimited. + +Move to GFF-style annotations +----------------------------- + +Starting with Augur v6 we now use GFF coordinates: [one-origin, +inclusive] as opposed to BED coordinates. Strands are represented by +``+`` or ``-`` rather than ``1`` or ``0``. Additionally, we export the +``seqid``, but don't use it in Auspice. + +Improvements and usage of JSON validation +----------------------------------------- + +The export command will now validate the produced JSON against the +schema. + +Removal of non-modular Augur and old builds +------------------------------------------- + +Augur has been a dynamic, shapeshifting beast. It started as scripts +for Nextflu, took on more and more pathogens, was refactored into +"prepare" and "process" steps, and refactored again into the "modular" +Augur we now have. Earlier incarnations of Augur have now been removed +from the GitHub repo (``./base/*``). + +Paralleling the different incarnations of Augur was a move to "builds" +being their own self-contained repos. We think this has been +remarkably successful, and de-couples the bioinformatics tooling from +a pathogen build. With this release of Augur we've now removed these +builds from the Augur GitHub repo, and the only builds that remain are +the test ones. + +Test builds +----------- + +There have been a number of test builds in the Augur repo and we have +leaned heavily on them while we developed this version of Augur as +well as Auspice v2. They are all self contained within +``./tests/builds`` and can all be run and examined in Auspice via + +.. code-block:: bash + + cd tests/builds + bash runner.sh # creates output in ./auspice + auspice view --datasetDir auspice + +(See the `Auspice docs +`__ for +Auspice-specific questions.) + +Documentation improvements +-------------------------- + +Documentation has always been a bit hit-or-miss with Nextstrain +projects. We've tried to make Augur's read-the-docs documentation more +comprehensive, with better flow. This entails new sections, with each +Augur command having its own page. We've tried to use redirects to +ensure that all the old links continue to work. + +Miscellaneous +------------- + +* ``augur filter``: More interpretable output of how many sequences + each stage has filtered out. +* ``augur filter``: Additional flag ``--subsample-seed`` to seed the + random number generator and thereby make subsampling reproducible. +* ``augur sequence-traits``: Numerical output as originally intended, + but required an Auspice bugfix. +* ``augur traits``: Explanation of what is considered missing data & + how it is interpreted. +* ``augur traits``: GTR models are exported in the output JSON for + better accountability & reproducibility. +* Errors in formatting of input files (e.g. metadata files, Auspice + config files) weren't handled nicely, often resulting in + hard-to-interpret stack traces. We now try to catch these and print + an error indicating the offending file. +* Tests using Python version 2 have now been removed. From 4e50f24450d5ad480ecc362a313edd304a117506 Mon Sep 17 00:00:00 2001 From: John SJ Anderson Date: Wed, 7 Aug 2024 17:40:01 -0700 Subject: [PATCH 4/5] Fix various broken links [#1569] --- docs/faq/metadata.rst | 3 +-- docs/faq/what-is-a-build.md | 4 ++-- docs/usage/cli/export.rst | 3 +-- docs/usage/usage.rst | 2 +- 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/faq/metadata.rst b/docs/faq/metadata.rst index e904cf245..622c045da 100644 --- a/docs/faq/metadata.rst +++ b/docs/faq/metadata.rst @@ -81,8 +81,7 @@ locations ('new_zealand' became 'New Zealand'). Auspice will still do this if you are exporting 'V1' type JSON files (from augur v5 or augur v6 using ``export v1``), but will not do this if -you are using ``export v2`` (`read -more <../releases/migrating-v5-v6.html#prettifying-metadata-fields>`__). +you are using ``export v2`` (:ref:`read more `). Instead, you should update your metadata files so that traits look the same as you'd like them to display in Auspice (change 'new_zealand' to 'New Zealand' in your metadata, and in any additional latitude-longitude diff --git a/docs/faq/what-is-a-build.md b/docs/faq/what-is-a-build.md index 83c05f172..7aff97bf1 100644 --- a/docs/faq/what-is-a-build.md +++ b/docs/faq/what-is-a-build.md @@ -11,7 +11,7 @@ These steps, taken together, are what we refer to as a "build". ### Example build -The [Zika virus tutorial](https://docs.nextstrain.org/en/latest/tutorials/zika.html#build-steps) describes a build which contains the following steps: +The [Zika virus tutorial](https://docs.nextstrain.org/en/latest/tutorials/creating-a-phylogenetic-workflow.html#run-a-nextstrain-build) describes a build which contains the following steps: 1. Prepare pathogen sequences and metadata 2. Align sequences @@ -28,5 +28,5 @@ If you look at the [other tutorials](https://docs.nextstrain.org/en/latest/tutor While it is possible to run a build by running each of the individual steps, we typically group these together into a make-type file. [Snakemake](https://snakemake.readthedocs.io/en/stable/index.html) is "a tool to create reproducible and scalable data analyses... via a human-readable, Python-based language." -> Snakemake is installed as part of the [conda environment](https://docs.nextstrain.org/en/latest/guides/install/local-installation.html#install-augur-auspice-with-conda) or the [docker container](https://docs.nextstrain.org/en/latest/guides/install/cli-install.html). +> Snakemake is installed as part of the [conda environment](https://docs.nextstrain.org/en/latest/guides/install/local-installation.html) or the [docker container](https://docs.nextstrain.org/en/latest/guides/install/cli-install.html). If you ever see a build which has a "Snakefile" then you can run this by typing `snakemake --cores 1` or `nextstrain build --cpus 1 .`, respectively. diff --git a/docs/usage/cli/export.rst b/docs/usage/cli/export.rst index 9950c2ec2..8d0aeb9da 100644 --- a/docs/usage/cli/export.rst +++ b/docs/usage/cli/export.rst @@ -2,8 +2,7 @@ augur export ============ -.. note:: The behavior of ``augur export`` has changed in v6. Please see `here <../../upgrading/migrating-v5-v6.html>`__ for more details. - +.. note:: The behavior of ``augur export`` has changed in v6. Please see :doc:`here <../../releases/migrating-v5-v6>` for more details. .. argparse:: :module: augur diff --git a/docs/usage/usage.rst b/docs/usage/usage.rst index 079118f3b..468d2fb64 100644 --- a/docs/usage/usage.rst +++ b/docs/usage/usage.rst @@ -14,7 +14,7 @@ Each command requires at least one input file and will produce one or more outpu Most commands admit additional arguments to modify how the analysis is run. Each command is documented below, and we are continually adding more examples to each command and providing real-life examples of their usage. -For instance, the documentation for `augur filter <./cli/filter.html>`__ shows how we use this to subsample the `nextstrain zika analysis `__. +For instance, the documentation for :doc:`augur filter <./cli/filter>` shows how we use this to subsample the `nextstrain zika analysis `__. `Note that you can also run each command with the` ``--help`` `option, for example` ``augur tree --help``, `for more information at the command-line.` From 3e4a5a9a761a64a5edf46b74c425ed9e9fc9fe32 Mon Sep 17 00:00:00 2001 From: John SJ Anderson Date: Wed, 7 Aug 2024 17:43:44 -0700 Subject: [PATCH 5/5] Update a link that is permanently redirected [#1569] --- docs/installation/non-python-dependencies.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation/non-python-dependencies.rst b/docs/installation/non-python-dependencies.rst index 2dd567538..4af95315c 100644 --- a/docs/installation/non-python-dependencies.rst +++ b/docs/installation/non-python-dependencies.rst @@ -5,7 +5,7 @@ Augur uses some external bioinformatics programs that are not available on PyPI: - ``augur tree`` requires at least one of: - `IQ-TREE `__ (used by default) - - `RAxML `__ (optional alternative) + - `RAxML `__ (optional alternative) - `FastTree `__ (optional alternative) - Bacterial data (or any VCF usage) requires `vcftools `__