Existing Workflow systems

Permalink: https://s.apache.org/existing-workflow-systems

Cite as (update dates):

Peter Amstutz, Maxim Mikheev, Michael R. Crusoe, Nebojša Tijanić, Samuel Lampa, et al. (2024):
Existing Workflow systems.
Common Workflow Language wiki, GitHub.
https://s.apache.org/existing-workflow-systems updated 2024-08-18, accessed 2024-08-18.

Computational Data Analysis Workflow Systems

An incomplete list

Please add new entries at the bottom. Try to include links to website and/or source code and ideally DOI for publications.

In addition to this list, actively developed free/open-source systems should be registered at https://workflows.community/systems

See also: https://github.com/pditommaso/awesome-pipeline

1. Arvados - CWL-based distributed computing platform for data analysis on massive data sets. https://arvados.org/ https://github.com/arvados/arvados
2. Apache Taverna http://www.taverna.org.uk/ https://taverna.incubator.apache.org/
3. Galaxy http://galaxyproject.org/
4. SHIWA https://www.shiwa-workflow.eu/
5. Apache Oozie https://oozie.apache.org/
6. DNANexus https://wiki.dnanexus.com/API-Specification-v1.0.0/IO-and-Run-Specifications https://documentation.dnanexus.com/developer/api/running-analyses/io-and-run-specifications https://wiki.dnanexus.com/API-Specification-v1.0.0/Workflows-and-Analyses https://documentation.dnanexus.com/developer/workflows
7. BioDT http://www.biodatomics.com/ archived at https://web.archive.org/web/20180609011656/http://www.biodatomics.com/
8. Agave http://agaveapi.co/live-docs/ https://docs.agaveapi.com/
9. DiscoveryEnvironment http://www.iplantcollaborative.org/ci/discovery-environment
10. Wings http://www.wings-workflows.org/
11. Knime https://www.knime.org/
12. make, rake, drake, ant, scons & many others. Software development relies heavily on tools to manage workflows related to compiling and packaging applications. For the most part these are file based and usually run on a single node, usually supporting parallel steps (make -j) and in some cases able to dispatch build steps to other machines https://code.google.com/p/distcc/ https://github.com/Factual/drake
13. Snakemake https://snakemake.github.io
14. BPipe http://bpipe.org http://docs.bpipe.org/ https://doi.org/10.1093/bioinformatics/bts167
15. Ruffus https://github.com/cgat-developers/ruffus
16. NextFlow http://nextflow.io
17. Luigi. Python package that helps you build complex pipelines of batch jobs http://github.com/spotify/luigi https://luigi.readthedocs.io
18. SciLuigi. Helper library built on top of Luigi to ease development of Scientific workflows in Luigi: http://github.com/pharmbio/sciluigi
19. Luigi Analysis Workflow (LAW) https://github.com/riga/law
20. GATK Queue https://www.broadinstitute.org/gatk/guide/topic?name=queue
21. Yabi https://ccg.murdoch.edu.au/yabi
22. seqware Workflows are written in Java and executed using the Oozie Workflow Engine on Hadoop or SGE clusters. Uses Zip64 files to group the workflow definition file, workflow itself, sample settings, and data dependencies in a single file that can be exchanged between SeqWare users or archived. https://seqware.github.io/ https://seqware.github.io/docs/6-pipeline/
23. Ketrew https://github.com/hammerlab/ketrew
24. Pegasus http://pegasus.isi.edu/
25. Apache Airflow https://github.com/apache/airflow
26. Couler https://github.com/couler-proj/couler - Unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
27. Cosmos https://cosmos.hms.harvard.edu/documentation/index.html https://doi.org/10.1093/bioinformatics/btu385 Cosmos2: https://github.com/LPM-HMS/COSMOS2 http://cosmos.hms.harvard.edu/COSMOS2/
28. Pinball https://github.com/pinterest/pinball
29. bcbio https://bcbio-nextgen.readthedocs.org/en/latest/
30. Chronos https://github.com/mesos/chronos
31. Azkaban https://azkaban.github.io/
32. Apache NiFi https://nifi.apache.org/docs/nifi-docs/html/overview.html
33. flowr (R-based) http://docs.flowr.space/ https://github.com/sahilseth/flowr
34. Mistral https://github.com/arteria-project https://wiki.openstack.org/wiki/Mistral#What_is_Mistral.3F https://docs.openstack.org/mistral/latest/user/wf_lang_v2.html
35. nipype http://nipy.org/nipype/
36. End of Day https://github.com/joestubbs/endofday
37. BioDSL https://github.com/maasha/BioDSL
38. bds (BigDataScript) https://pcingola.github.io/bds/ https://doi.org/10.1093/bioinformatics/btu595
39. Omics Pipe: uses Ruffus http://sulab.scripps.edu/omicspipe/
40. Ensembl Hive https://github.com/Ensembl/ensembl-hive
41. QuickNGS http://bifacility.uni-koeln.de/quickngs/web
42. GenePattern http://www.broadinstitute.org/cancer/software/genepattern/
43. Chipster http://chipster.csc.fi/
44. The Genome Modeling System https://github.com/genome/gms
45. Cuneiform, A Functional Workflow Language https://github.com/joergen7/cuneiform http://www.cuneiform-lang.org/ https://doi.org/10.1017/S0956796817000119
46. Anvaya https://doi.org/10.1142/s0219720012500060 http://webapp.cabgrid.res.in/biocomp/Anvaya/ANVAYA_Main.html#HOWTO_INSTALL_ANVAYA
47. Makeflow http://ccl.cse.nd.edu/software/makeflow/
48. Apache Airavata http://airavata.apache.org/
49. Pyflow https://github.com/Illumina/pyflow
50. Cluster Flow http://clusterflow.io
51. Unipro UGENE http://ugene.net/ https://doi.org/10.7717/peerj.644
52. CloudSlang http://www.cloudslang.io/
53. Stacks http://catchenlab.life.illinois.edu/stacks/
54. Leaf http://www.francesconapolitano.it/leaf/index.html
55. omictools http://omictools.com/
56. Job Description Language. The Job Description Language, JDL, is a high-level, user-oriented language based on Condor classified advertisements for describing jobs and aggregates of jobs such as Direct Acyclic Graphs and Collections. https://edms.cern.ch/ui/file/590869/1/WMS-JDL.pdf
57. YAWL yet another workflow language https://doi.org/10.1016/j.is.2004.02.002 http://www.yawlfoundation.org/
58. Triquetrum https://projects.eclipse.org/projects/technology.triquetrum https://github.com/eclipse/triquetrum/
59. Kronos https://github.com/jtaghiyar/kronos
60. qsubsec https://github.com/alastair-droop/qsubsec https://doi.org/10.1093/bioinformatics/btv698
61. YesWorkflow http://yesworkflow.org
62. gwf - Grid WorkFlow https://github.com/gwforg/gwf http://gwf.readthedocs.io/
63. Fireworks. https://github.com/materialsproject/fireworks https://github.com/materialsproject/fireworks https://doi.org/10.1002/cpe.3505
64. NGLess: NGS with less work http://ngless.rtfd.io
65. pypipegraph https://github.com/TyberiusPrime/pypipegraph
66. Cromwell https://github.com/broadinstitute/cromwell
67. Dagobah - Simple DAG-based job scheduler in Python. https://github.com/thieman/dagobah
68. sushi https://github.com/uzh/sushi
69. Clinical Trial Processor - A program for processing clinical trials data. http://mircwiki.rsna.org/index.php?title=MIRC_CTP
70. Noodles http://nlesc.github.io/noodles/
71. Swift & Swift/T http://swift-lang.org/main/ http://swift-lang.org/Swift-T/
72. Consonance (runs SeqWare & CWL) https://github.com/Consonance/consonance/wiki
73. Dog https://github.com/dogtools/dog
74. Produce https://github.com/texttheater/produce
75. LONI Pipeline http://pipeline.loni.usc.edu/
76. Cpipe https://github.com/MelbourneGenomics/cpipe
77. AWE https://github.com/MG-RAST/AWE
78. (Py)COMPSs https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/
79. KLIKO https://github.com/gijzelaerr/kliko
80. SoS Workflow https://github.com/vatlab/SoS https://vatlab.github.io/sos-docs/ https://doi.org/10.1093/bioinformatics/bty405 https://doi.org/10.1371/journal.pcbi.1006843
81. XNAT Pipeline Engine https://wiki.xnat.org/display/XNAT/Pipeline+Engine https://wiki.xnat.org/display/XNAT/XNAT+Pipeline+Development+Schema
82. Metapipe https://github.com/TorkamaniLab/metapipe
83. OCCAM (Open Curation for Computer Architecture Modeling) https://occam.cs.pitt.edu/
84. Copernicus http://www.copernicus-computing.org
85. iRODS Rule Language https://github.com/samuell/irods-cheatsheets/blob/master/irods-rule-lang-full-guide.md
86. VisTrails https://www.vistrails.org
87. Bionode Watermill https://github.com/bionode/bionode-watermill
88. BIOVIA Pipeline Pilot Overview http://accelrys.com/products/collaborative-science/biovia-pipeline-pilot/
89. Dagman A meta-scheduler for HTCondor https://research.cs.wisc.edu/htcondor/dagman/dagman.html
90. UNICORE https://www.unicore.eu/docstore/workflow-7.6.0/workflow-manual.html#wf_dialect
91. Toil (A scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python) https://github.com/BD2KGenomics/toil
92. Cylc (a workflow engine for cycling systems) https://cylc.github.io/
93. Autodesk Cloud Compute Canon https://github.com/Autodesk/cloud-compute-cannon
94. Civet https://github.com/TheJacksonLaboratory/civet
95. Cumulus https://github.com/Kitware/cumulus
96. High-performance integrated virtual environment (HIVE) https://hive.biochemistry.gwu.edu
97. Cloudgene http://cloudgene.uibk.ac.at/cloudgene-yaml
98. FASTR https://bitbucket.org/bigr_erasmusmc/fastr/ http://fastr.readthedocs.io/en/stable/
99. BioMake https://github.com/evoldoers/biomake https://doi.org/10.1101/093245
100. remake https://github.com/richfitz/remake
101. SciFloware http://www-sop.inria.fr/members/Didier.Parigot/pmwiki/Scifloware/
102. OpenAlea https://openalea.rtfd.io https://hal.archives-ouvertes.fr/hal-01166298/file/openalea-PradalCohen-Boulakia.pdf
103. COMBUSTI/O https://github.com/jarlebass/combustio http://hdl.handle.net/10037/9361
104. BioCloud https://github.com/ccwang002/biocloud-server-kai https://doi.org/10.6342/NTU201601295
105. Triana http://www.trianacode.org/ (website seems to have been taken over by SEO)
106. Kepler https://kepler-project.org/
107. Anduril http://anduril.org/site/
108. dgsh http://www.dmst.aueb.gr/dds/sw/dgsh/
109. EDGE bioinformatics: Empowering the Development of Genomics Expertise https://bioedge.lanl.gov/edge_ui/ http://edge.readthedocs.io/ https://lanl-bioinformatics.github.io/EDGE/
110. Pachyderm http://pachyderm.io/ http://pachyderm.readthedocs.io/en/stable/advanced/advanced.html
111. Digdag https://www.digdag.io/
112. Agua / Automated Genomics Utilities Agent http://aguadev.org
113. BioDepot Workflow Builder (BwB) https://github.com/BioDepot/BioDepot-workflow-builder https://doi.org/10.1101/099010
114. IMP: a pipeline for reproducible reference-independent integrated metagenomic and metatranscriptomic analyses http://r3lab.uni.lu/web/imp/ https://doi.org/10.1186/s13059-016-1116-8
115. Butler https://github.com/llevar/butler
116. adage / yadage https://github.com/diana-hep/adage https://github.com/diana-hep/yadage
117. HI-WAY: Execution of Scientific Workflows on Hadoop YARN https://github.com/marcbux/Hi-WAY https://openproceedings.org/2017/conf/edbt/paper-248.pdf
118. OpenMOLE https://github.com/openmole/openmole https://www.openmole.org/ https://doi.org/10.3389/fninf.2017.00021
119. Biopet https://github.com/biopet/biopet
120. Nephele https://nephele.niaid.nih.gov/
121. TOPPAS https://doi.org/10.1021/pr300187f
122. SBpipe https://pdp10.github.io/sbpipe/ https://github.com/pdp10/sbpipe https://doi.org/10.1186/s12918-017-0423-3
123. Dray http://dray.it/ (website seems to have been taken over by SEO)
124. GenomeVIP https://github.com/ding-lab/GenomeVIP https://doi.org/10.1101/gr.211656.116
125. GridSAM https://sourceforge.net/projects/gridsam/
126. Roddy https://github.com/eilslabs/Roddy
127. SciFlo (historical; doesn't seem to be maintained anymore) https://web.archive.org/web/20161118011409/https://sciflo.jpl.nasa.gov/SciFloWiki/FrontPage
128. GNU Guix Workflow Language https://git.roelj.com/guix/gwl.git#gnu-guix-workflow-language-extension https://github.com/UMCUGenetics/guix-workflows/blob/master/umcu/workflows/rnaseq.scm
129. Porcupine https://timvanmourik.github.io/Porcupine/
130. Parsl (a Parallel Scripting Library for Python) http://parsl-project.org https://doi.org/10.1145/3307681.3325400
131. ECFLOW (Workflow primarily for Meteorological Applications) https://software.ecmwf.int/wiki/display/ECFLOW/ecflow+home
132. Ophidia http://ophidia.cmcc.it/
133. WebLicht https://weblicht.sfs.uni-tuebingen.de/
134. GATE Cloud https://cloud.gate.ac.uk/
135. SCIPION http://scipion.cnb.csic.es/m/home/ https://github.com/I2PC/scipion/wiki/Creating-a-Protocol
136. Ergatis http://ergatis.sourceforge.net/
137. TIGR "Workflow" https://sourceforge.net/projects/tigr-workflow/ http://tigr-workflow.sourceforge.net/
138. Archivematica https://wiki.archivematica.org/Main_Page (A preservation workflow system that implements the ISO-OAIS standard using gearman/MCP)
139. Martian http://martian-lang.org/about/
140. BioMAJ http://genouest.github.io/biomaj/
141. Conveyor http://conveyor.cebitec.uni-bielefeld.de (retired). https://doi.org/10.1093/bioinformatics/btr040
142. Biopipe http://www.biopipe.org (appears to be defunct) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC403782/
143. Wildfire http://wildfire.bii.a-star.edu.sg/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-6-69
144. BioWBI https://bioinformatics.hsanmartino.it/bits_library/library/00079.pdf
145. BioWMS https://bioinformatics.hsanmartino.it/bits_library/library/00568.pdf
146. BioMoby http://biomoby.open-bio.org/ https://doi.org/10.1186/1471-2105-7-523
147. SIBIOS http://ieeexplore.ieee.org/document/1309094/
148. NGSANE https://github.com/BauerLab/ngsane https://doi.org/10.1093/bioinformatics/btu036
149. Pwrake https://github.com/misshie/Workflows https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3180464/
150. Nesoni https://github.com/Victorian-Bioinformatics-Consortium/nesoni
151. Skam http://skam.sourceforge.net/skam-intro.html
152. TREVA http://bioinformatics.petermac.org/treva/ https://doi.org/10.1371/journal.pone.0095217
153. EGene https://www.semanticscholar.org/paper/EGene-a-configurable-pipeline-generation-system-fo-Durham-Kashiwabara/4c0656195b5efcdd3aa7bdcb55fc95a957c150aa https://doi.org/10.1093/bioinformatics/btu366
154. WEP https://bioinformatics.cineca.it/wep/ https://doi.org/10.1186/1471-2105-14-S7-S11
155. Microbase http://www.microbasecloud.com/
156. e-Science Central http://www.esciencecentral.co.uk/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3538293/
157. Cyrille2 https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-96
158. PaPy https://code.google.com/archive/p/papy/ https://doi.org/10.1186/1471-2105-12-61
159. JobCenter https://github.com/yeastrc/jobcenter https://doi.org/10.1186/1751-0473-7-8
160. CoreFlow https://doi.org/10.1016/j.jprot.2014.01.023
161. dynamic-pipeline https://code.google.com/archive/p/dynamic-pipeline/
162. XiP http://xip.hgc.jp/wiki/en/Main_Page https://doi.org/10.1093/bioinformatics/bts630
163. Eoulsan http://www.outils.genomique.biologie.ens.fr/eoulsan/ https://doi.org/10.1093/bioinformatics/bts165
164. CloudDOE http://clouddoe.iis.sinica.edu.tw/
165. BioPig https://github.com/JGI-Bioinformatics/biopig https://doi.org/10.1093/bioinformatics/btt528
166. SeqPig https://github.com/HadoopGenomics/SeqPig https://doi.org/10.1093/bioinformatics/btt601
167. zymake http://www-personal.umich.edu/~ebreck/code/zymake/
168. JMS https://github.com/RUBi-ZA/JMS https://doi.org/10.1371/journal.pone.0134273
169. CLC Genomics Workbench https://www.qiagenbioinformatics.com/products/clc-genomics-workbench/
170. NG6 http://ng6.toulouse.inra.fr/ https://doi.org/10.1186/1471-2164-13-462
171. VIBE http://www.incogen.com/vibe/
172. WDL (Workflow Description Language) https://github.com/broadinstitute/wdl
173. SciFlow https://github.com/kaizhang/SciFlow (not to be confused with SciFloware and SciFlo).
174. Bioshake https://github.com/PapenfussLab/bioshake https://doi.org/10.7717/peerj.7223
175. SciPipe http://scipipe.org
176. Kapacitor / TICKscripts https://docs.influxdata.com/kapacitor/v1.3/tick/
177. AiiDA: Automated Interactive Infrastructure and Database for Computational Science http://www.aiida.net/ https://doi.org/10.1016/j.commatsci.2015.09.013
178. Reflow: a language and runtime for distributed, integrated data processing in the cloud https://github.com/grailbio/reflow
179. Resolwe: an open source dataflow package for Django framework https://github.com/genialis/resolwe
180. Yahoo! Pipes (historical) https://en.wikipedia.org/wiki/Yahoo!_Pipes
181. Walrus https://github.com/fjukstad/walrus
182. Apache Beam https://beam.apache.org/
183. CLOSHA https://closha.kobic.re.kr/ https://www.bioexpress.re.kr/go_tutorial http://docplayer.net/19700397-Closha-manual-ver1-1-kobic-korean-bioinformation-center-kogun82-kribb-re-kr-2016-05-08-bioinformatics-workflow-management-system-in-bio-express.html https://doi.org/10.1186/s12859-018-2019-3
184. WopMars https://github.com/aitgon/wopmars http://wopmars.readthedocs.io/
185. flowing-clj https://github.com/stain/flowing-clj
186. Plumbing and Graph https://github.com/plumatic/plumbing
187. LabView http://www.ni.com/en-us/shop/labview.html
188. MyOpenLab http://myopenlab.org/
189. Max/MSP https://cycling74.com/products/max/
190. NoFlo https://noflojs.org/
191. Flowstone http://www.dsprobotics.com/flowstone.html
192. HyperLoom https://code.it4i.cz/ADAS/loom https://code.it4i.cz/ADAS/loom
193. Dask http://dask.pydata.org/en/latest/ https://github.com/dask/dask
194. Stimela https://github.com/SpheMakh/Stimela https://github.com/SpheMakh/Stimela/wiki https://www.acru.ukzn.ac.za/~cosmosafari2017/wp-content/uploads/2017/02/makhathini.pdf
195. JTracker https://jtracker.io/ https://github.com/jtracker-io
196. PipelineDog http://pipeline.dog/ https://github.com/zhouanbo/pipelinedog https://doi.org/10.1093/bioinformatics/btx759
197. DALiuGE https://arxiv.org/abs/1702.07617 https://github.com/ICRAR/daliuge https://daliuge.readthedocs.io/
198. Overseer https://github.com/framed-data/overseer
199. Squonk https://squonk.it/
200. GC3Pie https://github.com/uzh/gc3pie
201. Fractalide https://github.com/fractalide/fractalide
202. TOGGLe http://toggle.southgreen.fr/ https://doi.org/10.1101/245480
203. Askalon http://www.askalon.org
204. Eclipse ICE (The Integrated Computational Environment) https://www.eclipse.org/ice
205. Sandia Analysis Workbench (SAW) http://www.sandia.gov/saw/
206. dispel4py https://github.com/dispel4py/dispel4py
207. Jobber https://pypi.python.org/pypi/Jobber/0.1.4
208. NeatSeq-Flow http://neatseq-flow.readthedocs.io/
209. S4M https://bitbucket.org/uqokorn/s4m_base/wiki/Home
210. Loom http://med.stanford.edu/gbsc/loom.html https://github.com/StanfordBioinformatics/loom http://loom.readthedocs.io/en/latest/templates.html
211. Watchdog - XML Schema-based bioinformatics workflow system https://www.bio.ifi.lmu.de/software/watchdog/ https://doi.org/10.1186/s12859-018-2107-4 https://github.com/klugem/watchdog
212. phpflo https://github.com/phpflo/phpflo
213. BASTet: Berkeley Analysis and Storage Toolkit https://openmsi.nersc.gov/openmsi/client/bastet.html https://biorack.github.io/BASTet/ https://doi.org/10.1109/TVCG.2017.2744479
214. Tavaxy: Pattern based workflow system for the bioinformatics domain http://www.tavaxy.org/
215. Ginflow: Decentralised adaptive workflow engine https://ginflow.inria.fr/
216. SciApps: A cloud-based platform for reproducible bioinformatics workflows https://doi.org/10.1093/bioinformatics/bty439 https://www.sciapps.org/
217. Stoa: Script Tracking for Observational Astronomy https://github.com/petehague/Stoa
218. Collective Knowledge (CK) framework http://cknowledge.org/
219. QosCosGrid (QCG) http://www.qoscosgrid.org/ http://www.qoscosgrid.org/trac/qcg-broker/wiki/qcg-advanced-client%20
220. High-Throughput Binding Affinity Calculator (HTBAC) https://htbac.readthedocs.io/en/latest/ https://github.com/radical-cybertools/htbac https://arxiv.org/abs/1801.01174
221. BioWorkbench (Swift-based) https://arxiv.org/abs/1801.03915 https://github.com/mmondelli/bioworkbench
222. ENVI Task Engine https://gbdxdocs.digitalglobe.com/docs/envi-task-engine https://www.harrisgeospatial.com/Learn/Whitepapers/TabId/2359/ArtMID/10212/ArticleID/17299/Workflow-Tools-in-ENVI.aspx https://envi-py-engine.readthedocs.io/en/latest/index.html
223. Pypeline https://github.com/cgarciae/pypeln
224. mpipe http://vmlaker.github.io/mpipe/
225. idseq-dag https://github.com/chanzuckerberg/idseq-dag
226. Piper (based upon GATK Queue) https://github.com/NationalGenomicsInfrastructure/piper
227. Apache Object Oriented Data Technology (OODT) http://oodt.apache.org/
228. JX Workflow (DSL for Makeflow) https://ccl.cse.nd.edu/software/manuals/jx-quick.html http://ccl.cse.nd.edu/research/papers/jx-escience-2018.pdf
229. The Adaptable IO System (ADIOS), ADIOS using applications can be the orchestrated into a workflow http://csmd.ornl.gov/adios
230. GenPipes https://bitbucket.org/mugqic/genpipes https://doi.org/10.1101/459552
231. Argo https://argoproj.github.io/ https://github.com/argoproj/argo https://github.com/argoproj/argo/blob/master/examples/README.md
232. Reana https://reana.readthedocs.io/en/latest/ https://github.com/reanahub/reana
233. Cuisine Framework https://www.astron.nl/~renting/cuisine.html
234. Niassa https://github.com/oicr-gsi/niassa https://oicr-gsi.github.io/niassa-docs/
235. pypeFLOW https://github.com/PacificBiosciences/pypeFLOW
236. Tiny Cloud Engine http://ka.cb.k.u-tokyo.ac.jp/tce/
237. Xbowflow https://github.com/ChrisSuess/Project-Xbow/tree/master/xbowflow
238. AdaptiveMd https://github.com/markovmodel/adaptivemd
239. Meshroom https://github.com/alicevision/meshroom
240. LSST Data Management https://github.com/lsst/pipe_base
241. CGAT-core https://github.com/cgat-developers/cgat-core
242. Prefect https://docs.prefect.io/
243. Apache SCXML engine https://commons.apache.org/proper/commons-scxml/guide/core-engine.html https://commons.apache.org/proper/commons-scxml/guide/scxml-documents.html
244. IceProd https://github.com/WIPACrepo/iceprod
245. AnADAMA2 http://huttenhower.sph.harvard.edu/anadama2
246. Luna https://luna-lang.org/
247. Passerelle https://code.google.com/archive/a/eclipselabs.org/p/passerelle
248. Kurator-Akka https://github.com/kurator-org/kurator-akka
249. Jug https://doi.org/10.5334/jors.161
250. Node-RED https://nodered.org/
251. Databolt Flow https://github.com/d6t/d6tflow
252. Frictionless Data Package Pipelines https://github.com/frictionlessdata/datapackage-pipelines
253. DataFlows https://github.com/datahq/dataflows
254. Volcano https://github.com/volcano-sh/volcano
255. DataJoint https://datajoint.io/
256. DIRAC3 (LHCb grid software) https://doi.org/10.1088/1742-6596/219/6/062029
257. Orange https://orange.biolab.si/ http://jmlr.org/papers/v14/demsar13a.html
258. Ensemble Toolkit (EnTK) https://radicalentk.readthedocs.io/en/latest/entk.html https://arxiv.org/abs/1602.00678v2
259. BioQueue http://www.bioqueue.org/ https://github.com/liyao001/BioQueue https://doi.org/10.1093/bioinformatics/btx403
260. mlr3pipelines https://mlr3pipelines.mlr-org.com/
261. Kedro - Production-Ready Data & ML Pipelines https://github.com/quantumblacklabs/kedro
262. DATAVIEW - DATAVIEW is a big data workflow management system. https://github.com/shiyonglu/DATAVIEW
263. SecDATAVIEW - SecDATAVIEW is a security oriented big data workflow management system compatible with the heterogeneous computing environments. https://github.com/shiyonglu/SecDATAVIEW
264. Giraffe Tools https://giraffe.tools/
265. CERAMICCA - Cloud Engine Resource for Accelerated Medical Image Computing for Clinical Applications: https://ceramicca.ensc.sfu.ca
266. uap - Universal Analysis Pipeline. A workflow management system dedicated to robust, consistent, and reproducible HTS data analysis. https://github.com/yigbt/uap
267. signac https://signac.io/ https://github.com/glotzerlab/signac-flow
268. cwltool - reference implementation of Common Workflow Language (CWL) https://github.com/common-workflow-language/cwltool/
269. CWLEXEC - CWL executor for IBM Spectrum LSF clusters https://github.com/IBMSpectrumComputing/cwlexec
270. drmr https://github.com/ParkerLab/drmr/ https://drmr.readthedocs.io/
271. Autosubmit - a versatile tool to manage Weather and Climate Experiments in diverse Supercomputing Environments. https://www.bsc.es/research-and-development/software-and-apps/software-list/autosubmit https://autosubmit.readthedocs.io/en/latest/
272. JUDI https://doi.org/10.1093/bioinformatics/btz956 https://github.com/ncbi/JUDI
273. Sumatra - a tool for managing and tracking projects based on numerical simulation or analysis, with the aim of supporting reproducible research https://neuralensemble.org/sumatra/
274. Netflix Conductor: https://netflix.github.io/conductor/
275. Pipengine https://doi.org/10.21105/joss.00341 https://github.com/fstrozzi/bioruby-pipengine
276. MyQueue https://doi.org/10.21105/joss.01844 https://gitlab.com/myqueue/myqueue
277. Drake R package https://doi.org/10.21105/joss.00550 https://github.com/ropensci/drake (superseded by [targets]{.title-ref})
278. MaDaTS https://10.21105/joss.00830 https://github.com/dghoshal-lbl/madats
279. Cadence https://cadenceworkflow.io/
280. Merlin https://merlin.readthedocs.io/
281. Janis https://janis.readthedocs.io/en/latest/index.html
282. AlphaSQL https://github.com/Matts966/alphasql
283. zeebe https://github.com/zeebe-io/zeebe
284. durabletask https://github.com/Azure/durabletask
285. Illumina State Language https://iap-docs.readme.io/docs/wes_illumina-states-language
286. Moteur http://www.i3s.unice.fr/~johan/publis/MOTEUR-poster-A4.pdf https://indico.cern.ch/event/286/contributions/654974/attachments/530521/731563/userForumEGEE_MOTEUR.pdf
287. SimStack http://www.simstack.de/
288. Maestro Workflow Conductor https://github.com/LLNL/maestrowf
289. HyWare https://doi.org/10.1007/s41060-020-00237-x
290. HyperFlow https://github.com/hyperflow-wms/
291. BRANE Framework https://onnovalkering.github.io/brane/
292. ApolloWF https://apollowf.github.io/
293. IS-EPOS Platform https://ieeexplore.ieee.org/document/9308147 https://tcs.ah-epos.eu/
294. pyinvoke: Pythonic task management & command execution, uses python as workflow language. http://www.pyinvoke.org/
295. targets R package https://cran.r-project.org/package=targets https://doi.org/10.21105/joss.02959
296. Compi https://doi.org/10.7717/peerj-cs.593 https://github.com/sing-group/compi https://www.sing-group.org/compi
297. TriggerFlow: Event-based Orchestration of Serverless Workflows https://github.com/triggerflow/triggerflow
298. Google Cloud Workflows: Orchestrate and automate Google Cloud and HTTP-based API services with serverless workflows. https://cloud.google.com/workflows/docs
299. PanDA Workflow Management System: https://doi.org/10.1051/epjconf/201921403050 https://doi.org/10.1051/epjconf/201610801003 https://github.com/PanDAWMS
300. Harvester: https://github.com/HSF/harvester
301. BD-Processor https://github.com/big-data-processor/bd-processor/
302. redun (yet another redundant workflow engine) https://github.com/insitro/redun
303. pyiron (The materials science IDE) https://pyiron.org/
304. looper (pipeline submitting engine) https://github.com/pepkit/looper
305. dagster (Python based data orchestration platform) https://dagster.io/
306. StackStorm (Devops automation engine) https://stackstorm.com/
307. Geoweaver (compose and execute full-stack deep learning workflows) https://esipfed.github.io/Geoweaver/
308. Popper: Container-native task automation engine https://github.com/getpopper/popper
309. Cloud Build: Build, test, and deploy on our serverless CI/CD platform https://cloud.google.com/build
310. Task/Taskfile: A task runner / simpler Make alternative written in Go https://github.com/go-task/task https://taskfile.dev/
311. pypyr: task runner for automation pipelines script sequential task workflow steps in yaml conditional execution, loops, error handling & retries https://pypyr.io/
312. SimTool/Sim2Ls: Jupyter notebook-based pipelines of Simulation Tools for the HUBzero platform lead by nanoHUB https://github.com/hubzero/simtool https://simtool.readthedocs.io/ https://doi.org/10.1371/journal.pone.0264492
313. SideIO: A Side I/O system framework for hybrid scientific workflow (no project/source code available) https://doi.org/10.1016/j.jpdc.2016.07.001
314. Flyte https://flyte.org/
315. StreamFlow https://streamflow.di.unito.it/
316. Jupyter Workflow https://jupyter-workflow.di.unito.it/
317. Nnodes: a simple workflow manager for Python functions and command line tools https://github.com/icui/nnodes
318. Orchest: A GUI for developing, running and managing container workflows https://github.com/orchest/orchest
319. Wasmflow: platform for building applications out of WebAssembly code containers https://github.com/wasmflow/wasmflow https://wasmflow.com/
320. HyperShell: cross-platform, high-performance computing utility for processing shell commands over a distributed, asynchronous queue. https://doi.org/10.1145/3491418.3535138 https://github.com/glentner/hyper-shell
321. Covalent: Pythonic distributed workflow tool used to prototype and run high performance classical and quantum software https://github.com/AgnostiqHQ/covalent
322. Icolos: workflow manager for structure-based workflows in computational chemistry https://github.com/MolecularAI/Icolos https://doi.org/10.26434/chemrxiv-2022-vqbxg
323. dwork: Task graph scheduler with a minimalistic API. https://github.com/frobnitzem/dwork
324. pmake: parallel make developed for use within batch jobs https://docs.olcf.ornl.gov/software/workflows/pmake.html#workflows-pmake
325. Texera: Collaborative Data Analytics Using Workflows https://github.com/Texera/texera
326. swif2: The Scientific Workflow Indefatigable Factotum https://scicomp.jlab.org/docs/swif2
327. Jobflow: jobflow is a library for writing computational workflows https://github.com/materialsproject/jobflow
328. Balsam: High throughput workflows and automation for HPC https://github.com/argonne-lcf/balsam
329. Task Vine: Dynamic workflows that run on HPC clusters, GPU clusters, and commercial clouds https://ccl.cse.nd.edu/software/taskvine/
330. Globus Compute: High Performance Function Serving for Science https://github.com/funcx-faas/funcX
331. mkite: Distributed computing for high-throughput materials simulation https://github.com/mkite-group
332. hpcflow: Automated simulate, process, archive workflows on HPC systems https://github.com/hpcflow/hpcflow-new, DOI: 10.12688/materialsopenres.17516.1
333. Data Version Control: DVC data pipelines as "Makefile" system for machine learning projects https://dvc.org/doc/start/data-management/data-pipelines
334. ZnFlow: General purpose framework for computational graphs https://github.com/zincware/ZnFlow
335. Globus Flows: Globus automation services provide secure and reliable task orchestration across a set of heterogeneous resources at scale. https://docs.globus.org/api/flows/
336. Kestra: Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines. https://github.com/kestra-io/kestra
337. Celery: Distributed Task Queue. https://github.com/celery/celery
338. dflow: Dflow is a Python framework for constructing scientific computing workflows (e.g. concurrent learning workflows) employing Argo Workflows as the workflow engine. https://github.com/deepmodeling/dflow
339. pydra: A simple dataflow engine with scalable semantics. https://github.com/nipype/pydra
340. funsies: A python library and execution engine to build reproducible, fault-tolerant, distributed and composable computational workflows. https://github.com/aspuru-guzik-group/funsies
341. Maize: A graph-based workflow manager for computational chemistry pipelines. https://github.com/MolecularAI/maize
342. doit: task management & automation tool. https://github.com/pydoit/doit
343. drama: asynchronous workflow executor engine based on Dramatiq and Apache Kafka https://github.com/KhaosResearch/drama
344. Titan: Semantic scientific workflow platform based on Drama, Apache Kafka, Avro and Spark https://titan.khaos.uma.es/
345. disBatch: Tool to distribute a list of computational tasks over a pool of compute resources. The pool can grow or shrink. https://github.com/flatironinstitute/disBatch
346. dawgz: Directed Acyclic Workflow Graph Scheduling. https://github.com/francois-rozet/dawgz
347. hummingqueue: Single tenant distributed scheduling. https://github.com/NablaChem/hummingqueue
348. Ewoks: Python-based meta workflow system for large-scale facilities, with multiple execution bindings https://ewoks.esrf.fr/ https://ewoks.readthedocs.io/
349. DAGonStar: simple Python-based workflow engine that runs jobs on everything from the local machine to distributed virtual HPC clusters hosted in private and public clouds. https://github.com/DagOnStar/dagonstar
350. MaRDIFlow: automation of abstracting meta-data embedded in an ontology of mathematical objects https://doi.org/10.48550/arXiv.2405.00028 https://doi.org/10.5281/zenodo.7863519
351. Kerblam: Rust-based pipeline system for scientific data analysis https://www.kerblam.dev/
352. PlantCV: Python-based imaging processing pipeline system for plant phenotyping https://plantcv.readthedocs.io/ https://doi.org/10.1016/j.molp.2015.06.005
353. Maestro: Netflix’s Workflow Orchestrator https://github.com/Netflix/maestro
354. BEE: Build and Execution Environment https://github.com/lanl/BEE
355. pipefunc: Lightweight function pipeline (DAG) creation in pure Python for scientific workflows with local and SLURM parallelization and QoL tools https://github.com/pipefunc/pipefunc
356. LEXIS: orchestration of complex scientific and industrial workflows which can be easily run on distributed HPC and Cloud resources https://lexis-project.eu/web/lexis-platform/
357. Perqueue: persistent queue. https://gitlab.com/asm-dtu/perqueue
358. Playbook Workflow Builder: Interactive Construction of Bioinformatics Workflows from a Network of Microservices https://playbook-workflow-builder.cloud/ https://github.com/MaayanLab/Playbook-Workflow-Builder https://doi.org/10.1101/2024.06.08.598037