Common Workflow Language

Support

The Common Workflow Language (CWL) is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry.

CWL is developed by an informal, multi-vendor working group consisting of organizations and individuals aiming to enable scientists to share data analysis workflows. The CWL project is on Github and we follow the Open-Stand.org principles for collaborative open standards development

CWL builds on technologies such as JSON-LD for data modeling and Docker for portable runtime environments.

User Guide

The CWL user guide (v1.0) provides a gentle introduction to learning how to write CWL command line tool and workflow descriptions.

Specification

For developers and advanced users, the current CWL specification (v1.0.2) provides authoritative documentation of the execution of CWL documents.

Citation

Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić (editors), Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes, Luka Stojanovic (2016): Common Workflow Language, v1.0. Specification, Common Workflow Language working group. https://w3id.org/cwl/v1.0/ doi:10.6084/m9.figshare.3115156.v2

Implementations

Software	Description	CWL support	Platform support
cwltool	Reference implementation of CWL		Linux, OS X, local execution only
Arvados	Distributed computing platform for data analysis on massive data sets. Using CWL on Arvados		AWS, GCP, Azure, Slurm
Toil	Toil is a workflow engine entirely written in Python.		AWS, Azure, GCP, Grid Engine, LSF, Mesos, OpenStack, Slurm
Rabix Bunny	An open-source, Java-based implementation of Common Workflow Language with support for multiple drafts/versions. See Rabix.io for details.		Linux, OS X, GA4GH TES (experimental)
cwl-tes	CWL engine backended by the GA4GH Task Execution API		Local, GCP, AWS, HTCondor, Grid Engine, PBS/Torque, Slurm
Airflow	Airflow updated with CWL-runner feature (supported by BioWardrobe Team, CCHMC)		Linux, OS X
Consonance	orchestration tool for running SeqWare workflows and CWL tools	pending	AWS, OpenStack, Azure
Apache Taverna	Domain-independent Workflow Management System	alpha	Java
Galaxy	Web-based platform for data intensive biomedical research.	alpha	Python
AWE	Workflow and resource management system for bioinformatics data analysis.	alpha	Go
Xenon	Run CWL workflows using Xenon	alpha	any Xenon backend: local, ssh, SLURM, Torque, Grid Engine

Repositories of CWL Tools and Workflows

Repository	Description
Workflows repository	Git repository of user contributed tools and workflows.
Dockstore tool registry	An open platform for sharing Docker-based tools described with the Common Workflow Language used by the GA4GH.

Software for working with CWL

Software	Description
cwltest	CWL testing framework, automated testing of tools and workflows written with CWL
cwl-upgrader	Upgrade CWL documents from draft-3 to v1.0
argparse2tool	Generate CWL CommandLineTool wrappers (and/or Galaxy tool descriptions) from Python programs that use argparse. Also supports the click argument parser.
cwl2argparse	Generate Python argparse code from CWL CommandLineTool description.
pypi2cwl	Automatically run argparse2cwl on any package in PyPi
cwlavro	Java classes for loading CWL documents
acd2cwl	CWL generator for ACD (EMBOSS) files
CWLviewer	A web application to view and share Common Workflow Language workflows
cwl2zshcomp	generates ZSH auto completions from CWL command line tool descriptions
scriptcwl	Create CWL workflows by writing a simple Python script
python-cwlgen	Generation of CWL programmatically from Python.

Projects the CWL community is participating in

Name	Details
GA4GH Task Execution API	a minimal common API for submitting a single job to a remote execution endpoint. Many contributions from CWL project participants.
GA4GH Workflow Execution API	a minimal common API for submitting workflow requests to workflow execution systems in a standardized way. Many contributions from CWL project participants.
Bio-compute objects	"a step towards evaluation and validation of bio-medical scientific computations", CWL and researchobject.org participants are cooperating with this effort

Support, Community and Contributing

The recommended place to ask a question about all things CWL is on Biostars.

If you are interested in learning more or contributing ideas or code, come chat with us on Gitter, check out #CommonWL on Twitter, join the mailing list common-workflow-language on Google Groups or fork the repository and send a pull request!

Code of Conduct

The CWL Project is dedicated to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, age, race, or religion. We do not tolerate harassment of participants in any form.

This code of conduct applies to all CWL Project spaces, including the Google Group, the Gitter chat room, the Google Hangouts chats, both online and off. Anyone who violates this code of conduct may be sanctioned or expelled from these spaces at the discretion of the leadership team.

For more details, see our Code of Conduct.

Participating Organizations

• Curoverse
• Seven Bridges Genomics
• Galaxy Project
• Apache Taverna
• Institut Pasteur
• Wellcome Trust Sanger Institute
• University of California Santa Cruz
• Harvard T.H. Chan School of Public Health
• Cincinnati Children's Hospital Medical Center
• Broad Institute
• University of Melbourne Center for Cancer Research
• Netherlands eScience Center
• Texas Advanced Computing Center Life Science Computing Group / Agave Platform
• CyVerse
• Institute for Systems Biology
• ELIXIR Europe
• BioExcel CoE
• BD2K

Individual Contributors

(Alphabetical)

• Peter Amstutz peter.amstutz@curoverse.com
• Robin Andeer robin.andeer@gmail.com
• Brad Chapman bchapman@hsph.harvard.edu
• John Chilton jmchilton@gmail.com
• Michael R. Crusoe michael.crusoe@gmail.com
• Roman Valls Guimerà brainstorm@nopcode.org
• Guillermo Carrasco Hernandez guille.ch.88@gmail.com
• Kenzo-Hugo Hillion kehillio@pasteur.fr
• Sinisa Ivkovic sinisa.ivkovic@sbgenomics.com
• Sehrish Kanwal skanwal@student.unimelb.edu.au
• Andrey Kartashov Andrey.Kartashov@cchmc.org
• John Kern kern3020@gmail.com
• Farah Zaib Khan farahk@student.unimelb.edu.au
• Dan Leehr dan.leehr@duke.edu
• Hervé Ménager herve.menager@gmail.com
• Maxim Mikheev mikhmv@biodatomics.com
• Michael Miller mmiller@systemsbiology.org
• Tim Pierce twp@unchi.org
• Josh Randall joshua.randall@sanger.ac.uk
• Mark Robinson mark.robinson-4@student.manchester.ac.uk
• Janko Simonović janko.simonovic@sbgenomics.com
• Stian Soiland-Reyes soiland-reyes@manchester.ac.uk
• Luka Stojanovic luka.stojanovic@sbgenomics.com
• Nebojša Tijanić nebojsa.tijanic@sbgenomics.com

The CWL Community Engineer, Michael R. Crusoe, publishes a blog about his work with updates at http://mrc.commonwl.org.