Snakemake workflow #52

rnmitchell · 2023-03-09T14:47:11Z

This PR will create a snakemake workflow to simplify running the entire lusSTR pipeline on either a single file or a set of files.

standage · 2023-03-17T20:31:37Z

lusSTR/workflow/.DS_Store

standage · 2023-03-17T20:34:08Z

No concerns about the Snakefile at the moment. I'd be happy to chat about some streamlining ideas some time, but that would be icing on the cake.

rnmitchell · 2023-03-28T15:20:24Z

Alright so I've been trying to get lusSTR into a snakemake workflow so that you can just put in the command lusstr all and it runs the entire STR pipeline. I've stored the arguments in a config file that can be easily edited. The strs.smk snake file itself does run successfully, now I've been trying to incorporate it into the lusSTR cli. And here is where I have absolutely no idea what I'm doing. 😅 I've tried to refactor the code to do this (see the most recent commit), but continue to run into the error:

Traceback (most recent call last):
  File "/Users/rebecca.mitchell/mambaforge/envs/lusstr/bin/lusstr", line 33, in <module>
    sys.exit(load_entry_point('lusSTR', 'console_scripts', 'lusstr')())
  File "/Users/rebecca.mitchell/mambaforge/envs/lusstr/bin/lusstr", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/Users/rebecca.mitchell/mambaforge/envs/lusSTR/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 94, in load
    module = import_module(match.group('module'))
  File "/Users/rebecca.mitchell/mambaforge/envs/lusSTR/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'lusSTR.__main__'

I think I need your assistance with this @standage. No rush and we can definitely talk in person about what I'm trying to do. My goal is to have two workflows: one for STRs and one for SNPs. At this time, I just have a placeholder for the SNPs one and I figured once I can successfully run the STR side, I'll fill that one in. Just trying to learn how to create this atm 😃

standage · 2023-03-28T15:53:12Z

We've done thin CLI wrappers around Snakemake workflows for several projects now, and I really like it as a strategy. I'd love to take a look at this myself, but if I can't make enough time soon anyone else on my team has experience and should be qualified to help you hash it out!

rnmitchell · 2023-03-28T17:46:15Z

Sounds good!

… ci]

rnmitchell · 2023-04-19T13:41:21Z

lusSTR/cli/config.py

+def main(args):
+    Path(args.workdir).mkdir(parents=True, exist_ok=True)
+    final_dest = f"{args.workdir}/config.yaml"
+    config = resource_filename("lusSTR", "data/config.yaml")
+    final_config = edit_config(config, args)
+    with open(final_dest, "w") as file:
+        yaml.dump(final_config, file)
+


I created the command lusstr config to create the config file. Settings can be specified as command line arguments. If no arguments are provided, the default settings are used and the user can edit manually. If a working directory is specified, the config file is dumped there, otherwise to the cwd.

Sounds good.

rnmitchell · 2023-04-19T13:42:09Z

lusSTR/cli/snps.py

+def main(args):
+    snakefile = resource_filename("lusSTR", "workflows/snps.smk")
+    pretarget = "annotate" if args.filter else "all"
+    result = snakemake.snakemake(
+        snakefile, config=args.config, targets=pretarget,
+        workdir=args.work_dir
+    )
+    if result is not True:
+        raise SystemError('Snakemake failed')
+


The snps command is not ready yet. I figured I'd do another PR after this one (since this one is already quite a lot). This is just a placeholder.

Python has an Exception class dedicated for this scenario: I'd recommend clearing out—or commenting out—the current code in the main function and replacing it with a raise NotImplementedError('SNP workflow implementation pending') statement.

rnmitchell · 2023-04-19T13:46:42Z

lusSTR/workflows/snps.smk

+## placeholder until I update for snps
+
+configfile: "config.yaml"
+output_name = config["output"]
+input_name = config["samp_input"]
+software = config["output_type"]
+prof = config["profile_type"]
+data = config["data_type"]
+filter_sep = config["filter_sep"]


Also a placeholder for the snps snake file

rnmitchell · 2023-04-19T13:49:05Z

lusSTR/wrappers/annot.py

-        if args.separate:
-            indiv_files(autosomal_final_table, input_name, ".txt")
-        else:
-            autosomal_final_table.to_csv(args.out, sep="\t", index=False)


I removed the separate argument for the annotate step. This was really just used when lusSTR files were being fed into LLAMAS which required separate files for each sample. lusSTR however requires a single file with all samples to be fed into the next step, so it's unnecessary.

rnmitchell · 2023-04-19T13:54:54Z

This is ready for further review @standage. I added a new command (lusstr config) to create the config file to store all the settings for the STR workflow, allowing the user to specify settings via the command line or they could open the file once created and manually edit it (probably the preferred way for non-bioinformatics folks 😄 ). The entire workflow doesn't have to be run (can just run up to the annotate step or just the format step if wanted).

I also updated all tests and all are passing with the exception of test_snps.py, as I haven't updated any of the snp workflow at this time. Whenever you have some time to poke around and give any advice and try out the tests, that would be great!

rnmitchell · 2023-04-19T14:58:15Z

I do still need to update the README.

rnmitchell · 2023-04-19T17:55:48Z

I do still need to update the README.

Nevermind! updated!

standage

Overall, this looks pretty good. A few comments, but hopefully should be quick to resolve.

One thing you might consider is that the various lusstr subcommands give users the flexibility to run the STR analysis workflow step-wise, so having to specify format, annotate, or all seems a bit redundant. Why is the user going to run lusstr strs if they don't want to run all? There might be a good reason for this, and I could probably formulate the contours of that reason myself. But I'm curious what you think. If we cut out that argument, that could simplify the interface a bit (I'm not sure it would simplify the workflow implementation at all). If you decide that it's necessary and should stay, how would you feel about making it an option with all as the default?

standage · 2023-04-19T20:10:40Z

setup.py

            "lusSTR/tests/data/NGS_stutter_test/*",
+            "lusSTR/workflows/*",
+            "lusSTR/wrappers/*",
        ]
    },
    include_package_data=True,
    install_requires=["pandas>=1.0", "openpyxl>=3.0.6"],


New package dependencies need to be declared here at a minimum.

snakemake

pyyaml(?)

standage · 2023-04-19T20:16:20Z

lusSTR/cli/config.py

+        data["separate"] = True
+    if args.nocombine:
+        data["nocombine"] = True
+    if args.efm_sep:


I get an error when I run lusstr config with no arguments. It points to this line, which doesn't appear to be a perfect match for any option declared in the argument parser. I assume this should be if args.efm?

Nope, old argument that I forgot to remove from that function. There were two different separate pieces, one for the annotate step and one for the filter step. Now only the filter step argument remains.

standage · 2023-04-19T20:17:36Z

lusSTR/cli/config.py

+def main(args):
+    Path(args.workdir).mkdir(parents=True, exist_ok=True)
+    final_dest = f"{args.workdir}/config.yaml"
+    config = resource_filename("lusSTR", "data/config.yaml")
+    final_config = edit_config(config, args)
+    with open(final_dest, "w") as file:
+        yaml.dump(final_config, file)
+


Sounds good.

standage · 2023-04-19T20:19:43Z

lusSTR/cli/snps.py

+def main(args):
+    snakefile = resource_filename("lusSTR", "workflows/snps.smk")
+    pretarget = "annotate" if args.filter else "all"
+    result = snakemake.snakemake(
+        snakefile, config=args.config, targets=pretarget,
+        workdir=args.work_dir
+    )
+    if result is not True:
+        raise SystemError('Snakemake failed')
+


Python has an Exception class dedicated for this scenario: I'd recommend clearing out—or commenting out—the current code in the main function and replacing it with a raise NotImplementedError('SNP workflow implementation pending') statement.

standage · 2023-04-19T21:13:33Z

lusSTR/tests/test_suite.py

+def test_snakemake(command, output, format_out, annot_out, all_out, tmp_path):
+    config = str(tmp_path / "config.yaml")
+    inputfile = data_file("UAS_bulk_input/Positive Control Sample Details Report 2315.xlsx")
+    exp_output = data_file(output)


Running this data through lusstr strs all on the command line went seamlessly for me.

rnmitchell · 2023-04-20T12:45:37Z

So do you mean by running lusstr format or lusstr annotate? Those subcommands no longer work since I refactored all the code for snakemake.

standage · 2023-04-20T13:02:10Z

Ah, gotcha. I saw that lusSTR still had a subcommand interface, but glossed over the fact that the old subcommands aren't supported any more. Carry on!

rnmitchell · 2023-04-20T16:01:56Z

I believe this is ready now, @standage!

standage

LGTM!

standage · 2023-04-21T13:52:59Z

It looks like some filter tests are failing due to the DataFrame append method not being found. I noted that pandas 2.0 was released recently and probably has some breaking changes. We need to pin both minimum and maximum version numbers for pandas now, it looks like: pandas>=1.0,<2.0.

SNP tests failed as well. If you're saving the SNP workflow for another PR, you should probably disable those tests for now.

rnmitchell · 2023-04-21T14:20:40Z

Ok will look into this!

…to snakemake

rnmitchell · 2023-04-21T16:49:24Z

Ok looks like things are passing now! Everything good for you?

created snakefile

0dde64c

rnmitchell marked this pull request as draft March 9, 2023 14:47

rnmitchell added 4 commits March 9, 2023 12:38

cleaned up code and fixed rule running x2

b7d759c

fixed double run error

88eeb8c

updated code to accomodate EFM different outputs [skip ci]

85cadca

fixed failing tests

f114939

standage reviewed Mar 17, 2023

View reviewed changes

lusSTR/workflow/.DS_Store Outdated

Copy link

Member

standage Mar 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope.

reformatted snakemake

1eff1e2

rnmitchell added 18 commits March 29, 2023 09:56

fixed main error

835f352

fixing errors [skip ci]

c4c4986

fixed str scripts

5e34778

fixed STR cli issue [skip ci]

f638e21

added command for copy config file to working dir

656de31

updated manifest and setup.py [skip ci]

fbd4841

added target for cli [skip ci]

72ad24c

added option to change config file using cl arguments

9869f4d

updated clis [skip ci]

52ea09c

updating code to accomodate nocombine option [skip ci]

9c32166

updated test_repeat.py

09e6faa

updated format tests [skip ci]

5f87672

updated test_marker.py [skip ci]

f468e57

fixed all tests for test_suite.py [skip ci]

eeee192

fixed error in test file [skip ci]

913750a

updated snakefile to extract sampleIDs from format or annot output file

f78b560

updated snakefile to assign correct output name for filter step [skip…

6a59b6e

… ci]

update tests [skip ci]

ea3efa1

rnmitchell added 4 commits April 18, 2023 14:50

fixed format and removed separate from annot.py [skip ci]

ec99cd7

removed separate from config for annotate step [skip ci]

82bc6ae

updated config file [skip ci]

176672e

remove .DS_Store file [skip ci]

effe222

rnmitchell commented Apr 19, 2023

View reviewed changes

updated README [skip ci]

d5af9bd

standage reviewed Apr 19, 2023

View reviewed changes

rnmitchell added 2 commits April 20, 2023 07:12

add error message for snps workflow, update setup.py [skip ci]

7cb34b9

removed old argument from config.py [skip ci]

a0bbbba

rnmitchell added 3 commits April 20, 2023 09:51

changed all "annotate" to "convert" [skip ci]

ea86844

updated default config [skip ci]

26433e0

banishing annotate and anno in tests and test files [skip ci]

1fec516

rnmitchell marked this pull request as ready for review April 20, 2023 16:02

rnmitchell and others added 2 commits April 20, 2023 12:18

missed files [skip ci]

83ddecf

NotImplementedError (change to trigger a CI build)

bce9efc

standage approved these changes Apr 21, 2023

View reviewed changes

rnmitchell added 3 commits April 21, 2023 12:34

pinned pandas version and deselected snp tests

d483f8e

Merge branch 'snakemake' of https://github.com/bioforensics/lusSTR in…

93f9ed3

…to snakemake

added missing test files

abbfc30

standage merged commit 6730754 into master Apr 21, 2023

standage deleted the snakemake branch April 21, 2023 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snakemake workflow #52

Snakemake workflow #52

rnmitchell commented Mar 9, 2023

standage Mar 17, 2023

standage commented Mar 17, 2023

rnmitchell commented Mar 28, 2023

standage commented Mar 28, 2023

rnmitchell commented Mar 28, 2023

rnmitchell Apr 19, 2023

standage Apr 19, 2023

rnmitchell Apr 19, 2023

standage Apr 19, 2023

rnmitchell Apr 19, 2023

rnmitchell Apr 19, 2023

rnmitchell commented Apr 19, 2023

rnmitchell commented Apr 19, 2023

rnmitchell commented Apr 19, 2023

standage left a comment

standage Apr 19, 2023

standage Apr 19, 2023

rnmitchell Apr 20, 2023

standage Apr 19, 2023

standage Apr 19, 2023

standage Apr 19, 2023

rnmitchell commented Apr 20, 2023

standage commented Apr 20, 2023

rnmitchell commented Apr 20, 2023

standage left a comment

standage commented Apr 21, 2023

rnmitchell commented Apr 21, 2023

rnmitchell commented Apr 21, 2023

Snakemake workflow #52

Snakemake workflow #52

Conversation

rnmitchell commented Mar 9, 2023

Choose a reason for hiding this comment

standage commented Mar 17, 2023

rnmitchell commented Mar 28, 2023

standage commented Mar 28, 2023

rnmitchell commented Mar 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnmitchell commented Apr 19, 2023

rnmitchell commented Apr 19, 2023

rnmitchell commented Apr 19, 2023

standage left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnmitchell commented Apr 20, 2023

standage commented Apr 20, 2023

rnmitchell commented Apr 20, 2023

standage left a comment

Choose a reason for hiding this comment

standage commented Apr 21, 2023

rnmitchell commented Apr 21, 2023

rnmitchell commented Apr 21, 2023