Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new DBs for MAPseq #6379

Merged
merged 7 commits into from
Sep 30, 2024
Merged

Add new DBs for MAPseq #6379

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5d9db30
add new DBs
RZ9082 Sep 27, 2024
7981599
flake8
RZ9082 Sep 27, 2024
dcd584f
Update mapseq_db_fetcher.xml
bgruening Sep 27, 2024
ff2e253
Merge branch 'galaxyproject:main' into mapseq-datamanager
RZ9082 Sep 30, 2024
91db4c4
update test
RZ9082 Sep 30, 2024
414458a
add test files
RZ9082 Sep 30, 2024
f5b416a
fix indentation
RZ9082 Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 18 additions & 8 deletions data_managers/data_manager_mapseq/data_manager/data_manager_fetch_mapseq_db.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,28 @@
import wget

DB_paths = {
"mgnify_lsu": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/silva_lsu-20200130.tar.gz",
"mgnify_ssu": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/silva_ssu-20200130.tar.gz",
"mgnify_its_unite": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/UNITE-20200214.tar.gz",
"mgnify_its_itsonedb": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/ITSoneDB-20200214.tar.gz",
"mgnify_v5_lsu": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/silva_lsu-20200130.tar.gz",
"mgnify_v5_ssu": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/silva_ssu-20200130.tar.gz",
"mgnify_v5_its_unite": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/UNITE-20200214.tar.gz",
"mgnify_v5_its_itsonedb": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipeline-5.0/ref-dbs/ITSoneDB-20200214.tar.gz",
"mgnify_v6_lsu": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipelines/tool-dbs/silva-lsu/silva-lsu_138.1.tar.gz",
"mgnify_v6_ssu": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipelines/tool-dbs/silva-ssu/silva-ssu_138.1.tar.gz",
"mgnify_v6_its_unite": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipelines/tool-dbs/unite/unite_9.0.tar.gz",
"mgnify_v6_its_itsonedb": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipelines/tool-dbs/itsonedb/itsonedb_1.141.tar.gz",
"mgnify_v6_pr2": "ftp://ftp.ebi.ac.uk/pub/databases/metagenomics/pipelines/tool-dbs/pr2/pr2_5.0.0.tar.gz",
"test_lsu": "https://zenodo.org/record/8205348/files/test_lsu.tar.gz",
}

DB_names = {
"mgnify_lsu": "MGnify LSU (v5.0.7) - silva_lsu-20200130",
"mgnify_ssu": "MGnify SSU (v5.0.7) - silva_ssu-20200130",
"mgnify_its_unite": "MGnify ITS UNITE (v5.0.7) - UNITE-20200214",
"mgnify_its_itsonedb": "MGnify ITS ITSonedb (v5.0.7) - ITSoneDB-20200214",
"mgnify_v5_lsu": "MGnify LSU (v5.0.7) - silva_lsu-20200130",
"mgnify_v5_ssu": "MGnify SSU (v5.0.7) - silva_ssu-20200130",
"mgnify_v5_its_unite": "MGnify ITS UNITE (v5.0.7) - UNITE-20200214",
"mgnify_v5_its_itsonedb": "MGnify ITS ITSonedb (v5.0.7) - ITSoneDB-20200214",
"mgnify_v6_lsu": "MGnify LSU (v6.0) - silva_lsu-20240702",
"mgnify_v6_ssu": "MGnify SSU (v6.0) - silva_ssu-20240701",
"mgnify_v6_its_unite": "MGnify ITS UNITE (v6.0) - UNITE-20240702",
"mgnify_v6_its_itsonedb": "MGnify ITS ITSonedb (v6.0) - ITSoneDB-20240702",
"mgnify_v6_pr2": "MGnify PR2 (v6.0) - PR2-20240702",
"test_lsu": "Trimmed LSU Test DB",
}

Expand Down
2 changes: 1 addition & 1 deletion data_managers/data_manager_mapseq/data_manager/macros.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0"?>
<macros>
<token name="@TOOL_VERSION@">1.1</token>
<token name="@VERSION_SUFFIX@">0</token>
<token name="@VERSION_SUFFIX@">1</token>
<token name="@PROFILE@">22.05</token>
<xml name="requirements">
<requirements>
Expand Down
59 changes: 38 additions & 21 deletions data_managers/data_manager_mapseq/data_manager/mapseq_db_fetcher.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
<?xml version="1.0"?>
<tool id="mapseq_db_fetcher" name="Mapseq DB fetcher" tool_type="manage_data" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="@PROFILE@">
<description>Fetches the DB required for mapseq</description>
<macros>
Expand All @@ -12,36 +11,54 @@
--out '${out_file}'
--version '${version}'
--database-type '${database_type}'
$test_data_manager
$test_data_manager_v5
$test_data_manager_v6
]]>
</command>
<inputs>
<!-- <param name="test_data_manager" type="hidden" /> -->
<param name="test_data_manager" type="boolean" truevalue="--test" falsevalue="" checked="False" label="Download minimal test DB and create mock data table entry." />

<param name="database_type" type="select" multiple="false" label="Database Type">
<option value="mgnify_lsu">MGnify LSU (v5.0.7)</option>
<option value="mgnify_ssu">MGnify SSU (v5.0.7)</option>
<option value="mgnify_its_itsonedb">MGnify ITS ITSonedb (v5.0.7)</option>
<option value="mgnify_its_unite">MGnify ITS UNITE (v5.0.7)</option>
</param>

<!-- <param name="test_data_manager" type="text" value=""/> -->
<param argument="--version" type="text" value="5.0.7" help="Check MGnify GitHub (https://github.com/EBI-Metagenomics/pipeline-v5/releases) for the version."/>
RZ9082 marked this conversation as resolved.
Show resolved Hide resolved
<!-- <param name="test_data_manager" type="hidden" /> -->
<param name="test_data_manager_v5" type="boolean" truevalue="--test" falsevalue="" checked="False" label="Download minimal test DB and create mock data table entry." />
<param name="test_data_manager_v6" type="boolean" truevalue="--test" falsevalue="" checked="False" label="Download minimal test DB and create mock data table entry." />

<param name="database_type" type="select" multiple="false" label="Database Type">
<option value="mgnify_v6_lsu">MGnify LSU (v6.0)</option>
<option value="mgnify_v6_ssu">MGnify SSU (v6.0)</option>
<option value="mgnify_v6_its_itsonedb">MGnify ITS ITSonedb (v6.0)</option>
<option value="mgnify_v6_its_unite">MGnify ITS UNITE (v6.0)</option>
<option value="mgnify_v6_pr2">MGnify PR2 (v6.0)</option>
<option value="mgnify_v5_lsu">MGnify LSU (v5.0.7)</option>
<option value="mgnify_v5_ssu">MGnify SSU (v5.0.7)</option>
<option value="mgnify_v5_its_itsonedb">MGnify ITS ITSonedb (v5.0.7)</option>
<option value="mgnify_v5_its_unite">MGnify ITS UNITE (v5.0.7)</option>
</param>

<!-- <param name="test_data_manager" type="text" value=""/> -->
<param argument="--version" type="text" value="6.0" help="Check MGnify GitHub (https://github.com/EBI-Metagenomics/amplicon-pipeline) for the version."/>
</inputs>
<outputs>
<data format="data_manager_json" name="out_file" />
</outputs>
<tests>
<test expect_num_outputs="1">
<param name="test_data_manager" value="--test"/>
<param name="test_data_manager_v5" value="--test"/>
<param name="version" value="5.0.7"/>
<param name="database_type" value="mgnify_lsu"/>
<param name="database_type" value="mgnify_v5_lsu"/>
<output name="out_file">
<assert_contents>
<has_text text="mgnify_lsu"/>
<has_text text="5.0.7"/>
</assert_contents>
<assert_contents>
<has_text text="mgnify_v5_lsu"/>
<has_text text="5.0.7"/>
</assert_contents>
</output>
</test>
<test expect_num_outputs="1">
<param name="test_data_manager_v6" value="--test"/>
<param name="version" value="6.0"/>
<param name="database_type" value="mgnify_v6_pr2"/>
<output name="out_file">
<assert_contents>
<has_text text="mgnify_v6_pr2"/>
<has_text text="6.0"/>
</assert_contents>
</output>
</test>
</tests>
Expand All @@ -51,4 +68,4 @@
</help>
<expand macro="citations" />
<expand macro="creator" />
</tool>
</tool>
0 ...ta/mgnify_lsu/LSU.fasta_trimmed.mscluster → ...mgnify_v5_lsu/LSU.fasta_trimmed.mscluster
View file
Open in desktop
File renamed without changes.
0 ...eq/test-data/mgnify_lsu/LSU_trimmed.fasta → ...test-data/mgnify_v5_lsu/LSU_trimmed.fasta
View file
Open in desktop
File renamed without changes.
0 ...seq/test-data/mgnify_lsu/lsu2_trimmed.otu → .../test-data/mgnify_v5_lsu/lsu2_trimmed.otu
View file
Open in desktop
File renamed without changes.
0 .../mgnify_lsu/slv_lsu_filtered2_trimmed.txt → ...nify_v5_lsu/slv_lsu_filtered2_trimmed.txt
View file
Open in desktop
File renamed without changes.
4 changes: 4 additions & 0 deletions data_managers/data_manager_mapseq/test-data/mgnify_v6_rp2/PR2-tax_trimmed.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#cutoff: 0.00:0.08 0.00:0.08 0.70:0.35 0.70:0.35 0.70:0.35 0.70:0.35 0.80:0.25 0.92:0.08 0.95:0.05
#levels: Domain Supergroup Division Subdivision Class Order Family Genus Species
#name: PR2
AB353770.1.1740_U d__Eukaryota;sg__TSAR;dv__Alveolata;sdv__Dinoflagellata;c__Dinophyceae;o__Peridiniales;f__Kryptoperidiniaceae;g__Unruhdinium;s__Unruhdinium_kevei
4 changes: 4 additions & 0 deletions data_managers/data_manager_mapseq/test-data/mgnify_v6_rp2/PR2.fasta_trimmed.mscluster
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
0 205465 754 1665 1981 2573 3170 3779 4152 4355 5272 5330 5671 5784 5995 7470 8458 9336 9837 10543 11522 11752 12319 12385 13481 13623 14541 14768 14900 14931 14980 15260 15638 15912 16277 17330 18346 20015 20711 20923 21192 21315 22226 22801 23067 23263 23279 24044 24657 24805 24900 25005 26884 27112 27561 28308 29356 29423 30139 30945 31003 31340 31521 31593 31806 32167 34340 34468 35422 36425 36880 37494 38725 38860 40286 40653 41017 41106 42490 43211 43978 44665 45224 46502 47029 48371 48595 49510 50779 50784 51352 51601 51725 52244 52603 52607 53194 53358 53577 54672 54827 55196 55365 56745 57336 57724 57846 59382 59741 60962 61637 62141 62297 62324 62659 62739 62821 64686 66855 67304 67441 67855 68091 68461 68742 70749 71830 72309 72709 73032 74387 74617 74624 77802 77930 78279 79272 80603 81428 82088 82315 83571 84447 85181 85823 86779 87089 89391 90298 91045 91093 93794 95095 96213 97223 98014 98191 98387 99054 99264 100023 100068 101545 101839 102733 103364 103377 103977 104183 104390 104785 106726 106834 107030 107388 108179 108270 108598 108630 109203 110648 110981 111132 111595 111976 111992 112242 112657 114625 118148 118635 119238 119586 120826 121873 122186 124026 124492 125746 126825 126862 126867 128172 128242 128840 129005 129056 129500 130486 130504 131255 131356 131858 132159 132779 133202 134013 134339 135658 135727 135859 137532 138968 139782 139925 141250 141654 141661 145922 147008 149592 150200 150201 150202 150203 150204 150205 150206 150207 151737 151767 151781 151812 151814 151842 151845 151850 151881 151885 151964 151970 151975 151989 151996 152019 152039 152055 152068 152094 152105 152193 152207 152216 152226 152296 152307 152314 152339 152340 152361 154570 158485 158489 181294 181295 181297 181300 181301 181303 181308 181309 181310 181311 181312 181313 181314 181315 181316 181317 181319 181320 181322 181324 181325 181326 181327 181328 181329 181330 181359 181360 181361 181362 181363 181364 181365 181366 181367 181368 181369 181370 181371 181372 181373 181374 181375 181376 181377 181378 181379 181380 181381 181382 181383 181392 181586 184844 185293 197488 198891 198894 205167 205168 205179 205184 205191 205192 216479 216510 216971 216990
1 6400 1130 2149 27364 44735 50750 58365 110841 204950 204951 204952 204953 204954 204955
2 111578 98334
3 30450 30452
4 changes: 4 additions & 0 deletions data_managers/data_manager_mapseq/test-data/mgnify_v6_rp2/PR2_trimmed.fasta
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
>AB353770.1.1740_U
ATGCTTGTCTCAAAGATTAAGCCATGCATGTCTCAGTATAAGCTTTTACATGGCGAAACTGCGAATGGCTCATTAAAACAGTTACAGTTTATTTGAAGGTCATTTTCTACATGGATAACTGTGGTAATTCTAGAGCTAATACATGCGCCCAAACCCGACTCCGTGGAAGGGTTGTATTTATTAGTTACAGAACCAACCCAGGTTCGCCTGGCCATTTGGTGATTCATAATAAACGAGCGAATTGCACAGCCTCAGCTGGCGATGTATCATTCAAGTTTCTGACCTATCAGCTTCCGACGGTAGGGTATTGGCCTACCGTGGCAATGACGGGTAACGGAGAATTAGGGTTCGATTCCGGAGAGGGAGCCTGAGAAACGGCTACCACATCTAAGGAAGGCAGCAGGCGCGCAAATTACCCAATCCTGACACAGGGAGGTAGTGACAAGAAATAACAATACAGGGCAACCATGTCTTGTAATTGGAATGAGTAGAATTTAAATCCCTTTACGAGTATCCATTGGAGGGCAAGTCTGGTGCCAGCAGCCGCGGTAATTCCAGCTCCAATAGCGTATATTAAAGTTGTTGCGGTTAAAAAGCTCGTAGTTGGATTTCTGTCGAAGGAGACCGGTCCGCCCTCTGGGTGAGTATCTGGATCTCTTTGGACATCTTCTTGGGGAACGTATCTGCACTTCATTGTGCGGTGCGGTACTCAAGACTTTTACTTTGAGGAAATTAGAGTGTTTCAAGCAGGCACACGCCTTGAATACATTAGCATGGAATAATAAGATAGGACCTCGGTTCTATTTTGTTGGTTTCTAGAGCTGAGGTAATGATTAATAGGGATAGTTGGGGGCATTCGTATTTAACTGTCAGAGGTGAAATTCTTGGATTTGTTAAAGACGGACTACTGCGAAAGCATTTGCCAAGGATGTTTTCATTGATCAAGAACGAAAGTTAGGGGATCGAAGACGATCAGATACCGTCGTAGTCTTAACCATAAACCATGCCGACTAGAGATTGGAGGTCGTTATCCGTACGACTCCTTCAGCACCTTATGAGAAATCAAAGTCTTTGGGTTCCGGGGGGAGTATGGTCGCAAGGCTGAAACTTAAAGGAATTGACGGAAGGGCACCACCAGGAGTGGAGCCTGCGGCTTAATTTGACTCAACACGGGGAAACTTACCAGGTCCAGACATAGTAAGGATTGACAGATTGATAGCTCTTTCTTGATTCTATGGGTGGTGGTGCATGGCCGTTCTTAGTTGGTGGAGTGATTTGTCTGGTTAATTCCGTTAACGAACGAGACCTTAACCTGCTAAATAGTGACACATTACCCCGGTAATGTGGGTTACTTCTTAGAGGGACTTTGCGTGTCTAACGCAAGGAAGTTTGAGGCAATAACAGGTCTGTGATGCCCTTAGATGTTCTGGGCTGCACGCGCGCTACACTGATGCGCTCAACGAGTTTATGACCTTGCCCGGAAGGGTTGGGTAATCTTGTTAAAACGCATCGTGATGGGGATAGATTATTGCAATTATTAATCTTCAACGAGGAATTCCTAGTAAGCGCGAGTCATCAGCTCGTGCTGATTAAGTCCCTGCCCTTTGTACACACCGCCCGTCGCTCCTACCGATTGAGTGATCCGGTGAATAATTCGGACCGCAGCATTTGTCAGTTCCTGACTCATGCCGTGGAAAGTCTAGTGAACCTTATCACTTAGAGGAAGGAGAAGTCGTAACA
>AB284159.1.1765_U
TGATCCTGCCAGTAGTCATATGCTTGTCTCAAAGATTAAGCCATGCATGTCTCAGTATAAGCTTCAACATGGCAAGACTGTGAATGGCTCATTAAAACAGTTGTAGTTTATTTGGTGGCCTCTTTACATGGATAGCCGTGGTAATTCTAGAACTAATACATGCGCTCAAGCCCGACTTCGCAGAAGGGCTGTGTTTATTTGTTACAGAACCATTTCAGGCTCTGCCTGGTTTTTGGTGAATCAAAATACCTTATGGATTGTGTGGCATCAGCTGGTGATGACTCATTCAAGCTTCTGACCTATCAGCTTCCGACGGTAGGGTATTGGCCTACCCTGGCAATGACGGGTAACGGAGAATTAGGGTTTGATACCGGAGAGGGAGCCTGAGAAACGGCTACCACATCTAAGGAAGGCAGCAGGCGCGCAAATTACCCAATCCTGACACAGAGAGGTAGTGACAAGAAATAACAATACAGGGCATCCATGTCTTGTAATTGGAATGAATAGAATGTATACCCTTTTATGAGTATCAATTGGAGGGCAAGTCTGGTGCCAGCAGCCGCGGTAATTCCAGCTCCAATAGTGTATATTAAAGTTGTTGCGGTTAAAAAGCTCGTAGTTGGATTTCTGCCAAGGCTGACCGGTCCGCCCTTTGGGTGAGTATCTGGCTTGGCTGGAGCATTTTCTTGGAGGGCATGGCTGCACTTAACTGTGTGGTATGCAACCCAAGACTTTTACTTTGAGGAAATTAGAGTGTCTCAAGCAGGCTCGCGCCTTGAATACATTAGCATGGAATAATAAGATAAGGCCTTGGTTGTATTTTGTTGGTTTCTACAGCTGAGGTAATGATGAACAGGGATACTTGGGGGCATACGAATTTAACTGTCAGAGGTGAAATTCTTGGATTTGTTAAAGACGGACTACTGTGAAAGCATTTGCCAAGAATGTTTTCATTGATCAAGAATGAAAGTTAGGGGGTCGAAGATGATCAGATACCGTCTTAGTCTTAACCATAAACTATGCCGACTAGAGATTGGAGGTTGCTACTTACATGACTCCTTCAGCACCTTATGGGAAATCAAAGTCTTTGGGTTCCGGGGGGAGTATGGTTGCAAGGCTGAAACTTAAAGGAATTGACGGAAGGGCACCACCAGTAGTGGAGCCTGCGGCTTAATTTGACTCAACACGGGGAAACTTACCAGGTCCAGACATAGTAAGGATTGACAGATTGATAGCTCCTTCTTGATTCTGTGGGTGGTGGTGCATGGCCGTTCTTAGTTGGTGGAGTGATTTGTCTGGTTAATTCCGTTAACGAACGAGACCTTATCCTGCTAAATAGATTCGCATAGCCATGGTTATGTGGGGAACTTCTTAGAGGAACTTTGTGTGTCTAATGCAAGGAAGTTTGAGGCAATAACAGGTCTGTGATGCCCTTAGATGTTCTGGGCTGCACGCGCGCTACACTGATGAGCTCAACAAGTGTGCAACTTTGCCTGAAAAGGTAGGGCAATCTTTTGAAAACGTATCATGATGGGGATCGATTATTGCAATTATTAGTCTTCAACAAGGAATTCCTAGTAAGTGTGAGTCATCAACTTATGCTGATTACGTCCCTGCCCTTTGTACACACCGCCCGTCGCTCCTACTGATTGAGTGATCCGGTGAATATTTTGGACTGCAGCATAGCTTAGTTCCTGAGTGTTGCAGCATAAAGTTTAGTGAACCTTATCACTTAGAGGAAGGAGAAGTCGTAACAAGGTTTCCG
4 changes: 4 additions & 0 deletions data_managers/data_manager_mapseq/test-data/mgnify_v6_rp2/PR2_trimmed.otu
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
1 Unclassified
2 d__Archaea
3 d__Archaea;sg__DPANN
4 d__Archaea;sg__DPANN;dv__Nanoarchaeaeota
1 change: 1 addition & 0 deletions data_managers/data_manager_mapseq/test-data/mgnify_v6_rp2/VERSION.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
5.0.0