Snakemake workflow: MPRAsnakeflow

This pipeline processes sequencing data from Massively Parallel Reporter Assays (MPRA) to create count tables for candidate sequences tested in the experiment.

MPRAsnakeflow is built on top of Snakemake. Insert your code into the respective folders, i.e. scripts, rules, and envs. Define the entry point of the workflow in the Snakefile and the main configuration in a config.yaml file.

Authors

• Max Schubach (@visze), Berlin Institute of Health at Charité -- Universitätsklinikum Berlin, Computational Genome Biology Group

Documentation

You can find a extensive documentations here

Usage

If you use this workflow in a paper, don't forget to give credits to the authors by citing the URL of this (original) repository and, if available, its DOI (see above).

Step 1: Obtain a copy of this workflow

Clone this repository to your local system, into the place where you want to perform the data analysis.

Step 2: Configure workflow

Configure the workflow according to your needs via editing the files in the config/ folder. Create or adjust the config.yaml to configure the workflow execution. When running on a cluster environment there are drmaa.yaml for drmaa runs or cluster.yaml for SLURM environment which contain the resources required for each job.

Step 3: Install Snakemake

Install Snakemake using conda:

conda create -c bioconda -n snakemake snakemake

For installation details, see the instructions in the Snakemake documentation.

Step 4: Execute workflow

Activate the conda environment:

conda activate snakemake

Test your configuration by performing a dry-run via

snakemake --use-conda --configfile conf/config.yaml -n

Execute the workflow locally via

snakemake --use-conda --cores $N --configfile conf/config.yaml

using $N cores or run it in a cluster environment (SLURM using sbatch) via

snakemake --use-conda --configfile conf/config.yaml --cluster "sbatch --nodes=1 --ntasks={cluster.threads} --mem={cluster.mem} -t {cluster.time} -p {cluster.queue} -o {cluster.output}" --jobs 100 --cluster-config config/sbatch.yaml

or

snakemake --use-conda --configfile conf/config.yaml --drmaa "-n {cluster.threads} --mem={cluster.mem} -t {cluster.time} -p {cluster.queue} -o {cluster.output}" --jobs 100

using DRMAA.

Please note that the log folder of the cluster environment has to be generated first, e.g:

mkdir -p logs

For other cluster environments please check the Snakemake documentation and adapt accodingly.

If you not only want to fix the software stack but also the underlying OS, use

snakemake --use-conda --use-singularity --configfile conf/config.yaml

in combination with any of the modes above.

It is also possible to run the workflow in a different folder so that the results get stored not in the MPRAsnakeflow folder. Here you have to specify the snakefile path, like

snakemake --use-conda --configfile yourConfigFile.yaml --snakefile <path/to/MPRAsnakeflow>/MPRAsnakeflow/workflow/Snakefile --cores $N

See the Snakemake documentation for further details.

Step 5: Investigate results

After successful execution, you can create a self-contained interactive HTML report with all results via:

snakemake --report report.html --configfile conf/config.yaml

This report can, e.g., be forwarded to your collaborators.

Step 6: Commit changes

Whenever you change something, don't forget to commit the changes back to your github copy of the repository:

git commit -a
git push

Step 7: Obtain updates from upstream

Whenever you want to synchronize your workflow copy with new developments from upstream, do the following.

1. Once, register the upstream repository in your local copy: git remote add -f upstream git@github.com:snakemake-workflows/MPRAsnakeflow.git or git remote add -f upstream https://github.com/snakemake-workflows/MPRAsnakeflow.git if you do not have setup ssh keys.
2. Update the upstream version: git fetch upstream.
3. Create a diff with the current version: git diff HEAD upstream/master workflow > upstream-changes.diff.
4. Investigate the changes: vim upstream-changes.diff.
5. Apply the modified diff via: git apply upstream-changes.diff.
6. Carefully check whether you need to update the config files: git diff HEAD upstream/master config. If so, do it manually, and only where necessary, since you would otherwise likely overwrite your settings and samples.

Step 8: Contribute back

In case you have also changed or added steps, please consider contributing them back to the original repository:

1. Fork the original repo to a personal or lab account.
2. Clone the fork to your local system, to a different place than where you ran your analysis.
3. Copy the modified files from your analysis to the clone of your fork, e.g., cp -r workflow path/to/fork. Make sure to not accidentally copy config file contents or sample sheets. Instead, manually update the example config files if necessary.
4. Commit and push your changes to your fork.
5. Create a pull request against the original repository.

Testing

Test cases are in the subfolder .test. They are automatically executed via continuous integration with Github Actions.