Update/Revamp `focal-cn-file-preparation` module #452

cbethell · 2020-01-17T20:40:00Z

Purpose/implementation Section

Per #186 (comment), update and revamp the focal-cn-file-preparation module.

What GitHub issue does your pull request address?

This PR addresses issue #186.

Directions for reviewers. Tell potential reviewers what kind of feedback you are soliciting.

Which areas should receive a particularly close look?

Are there any steps that can be refactored?
The cytoband data was removed from the final results' tables in order to minimize memory usage, is this a decision that you would agree with?
Is there any additional information that should be added to the final tables in this PR?

Is there anything that you want to discuss further?

Note: I’m summarizing the GISTIC broad_values_by_arm.txt calls into loss, gain, and neutral, and amplification in the broad_status column, the same coding used for the values in the status column.

Is the analysis in a mature enough form that the resulting figure(s) and/or table(s) are ready for review?

Yes, this is ready for review.

Results

The results in this PR include the updated tsv files found in this module's results directory as follows:

analyses/focal-cn-file-preparation/results/cnvkit_annotated_cn_autosomes.tsv.gz
analyses/focal-cn-file-preparation/results/cnvkit_annotated_cn_x_and_y.tsv.gz
analyses/focal-cn-file-preparation/results/controlfreec_annotated_cn_autosomes.tsv.gz
analyses/focal-cn-file-preparation/results/controlfreec_annotated_cn_x_and_y.tsv.gz

Reproducibility Checklist

The dependencies required to run the code in this pull request have been added to the project Dockerfile.
This analysis has been added to continuous integration.

Documentation Checklist

This analysis module has a README and it is up to date.
This analysis is recorded in the table in analyses/README.md and the entry is up to date.
The analytical code is documented and contains comments.

…into speed_up_granges

- compare cytoband calls - update `README` - update comment in shell script and add `exon_file` option - Incorporate GISTIC calls in final cn tables - rerun `01-prepare-cn-file.R`

analyses/focal-cn-file-preparation/README.md

cansavvy

Cool beans! We are getting there!! 🚀

analyses/focal-cn-file-preparation/01-prepare-cn-file.R

cansavvy · 2020-01-17T20:44:23Z

analyses/focal-cn-file-preparation/01-prepare-cn-file.R

+    copy_number = cnv_gr$copy_number[overlaps_ucsc@from],
+    ploidy = cnv_gr$tumor_ploidy[overlaps_ucsc@from],
+    ensembl = cnv_gr$org_id[overlaps_ucsc@from],
+    cytoband = ucsc_cytoband_gr@elementMetadata@listData$cytoband[overlaps_ucsc@to],
    stringsAsFactors = FALSE
  ) %>%
    dplyr::distinct() %>%


What's this distinct doing?

Were you able to remove this distinct or determine what it was doing, @cbethell ?

analyses/focal-cn-file-preparation/01-prepare-cn-file.R

analyses/focal-cn-file-preparation/README.md

analyses/focal-cn-file-preparation/run-prepare-cn.sh

jaclyn-taroni · 2020-01-18T18:36:28Z

@cbethell @cansavvy - as you are working on this, keep in mind that there may be a period of time where we have a consensus SEG file that we want to annotate but no GISTIC file yet. (See #186 (comment) for more context.) Are there ways to make the GISTIC steps optional here? That change can be implemented in a later pull request, but it's a good to keep an eye on any design decisions here that may make that more difficult to implement later.

…into speed_up_granges

- rename instances of `exon` to `cds` - update comments - add large files (`gistic-results` and `results/cnvkit_annotated_cn_autosomes.tsv.gz` which is 101.2 MB) to `.gitignore` - add inclusion of gistic results as an option - update shell script to include gistic option - remove `cytoband_org` variable - update `README.md` - refactor section of script reading in and wrangling GISTIC data

- minor update to comment

…p_granges

cansavvy

@cbethell We are getting there! You did a good job trimming out some of the duplicate items! There's a few documentation adds I think we should make and I wanna take a look at your file that keeps changing sizes because I wanna get to the bottom of that before we call it a day.

analyses/focal-cn-file-preparation/01-prepare-cn-file.R

cansavvy · 2020-01-21T22:29:58Z

analyses/focal-cn-file-preparation/run-prepare-cn.sh

 #
 # Usage: bash run-prepare-cn.sh

 set -e
 set -o pipefail

 XYFLAG=${OPENPBTA_XY:-1}
+cds_file=../../scratch/gencode.v27.primary_assembly.annotation.bed


I've been meaning to do this where I have this in snv-callers but we should probably include an indicator in this file name that tells people it's been filtered to CDS sequences only. Because it is a scratch file and isn't meant to stick around, I haven't made it a priority to change this. My bad.

analyses/focal-cn-file-preparation/.gitignore

- update comments - rename gene id column - update comment in shell script - update column documentation in README - run styler

- remove code in `.gitignore` file re large `cnvkit_annotated_autosomes.tsv.gz` file that no longer exists - remove `gzip` step in `run-prepare-cn.sh` as this is handled in the R script

- update `run-oncoprint.sh`, `00-subset-files-for-ATRT.R`, `02-HGG-molecular-subtyping-subset-files.R`

cbethell · 2020-01-22T21:11:36Z

@cansavvy I made the above documentation changes and changed the format of the output compressed files to bz2.

Let me know if you find something that I may have missed or that may need some refactoring.

cansavvy · 2020-01-22T21:16:26Z

@cbethell has the file stopped changing sizes now? Were you able to get to the bottom of that?

cbethell · 2020-01-22T21:21:53Z

@cbethell has the file stopped changing sizes now? Were you able to get to the bottom of that?

@cansavvy the file was always the same size, it seems that one of the columns just compressed differently. I realized that the smaller file that I thought was a new file at the end of the day yesterday was really just the old compressed file so it all makes sense now.

cansavvy

Cool beans. This LGTM then! Assuming CircleCI passes.

Pre: AlexsLemonade#452

…SEG case (#479) * Back to exons; before GISTIC Pre: #452 * Have expression script use gzipped; skip for now * Renumber scripts and notebooks * Add notebook for preparing consensus SEG file Add ploidy! * Make the cnvkit option name more general * Rip out CDS steps * Rip out GISTIC option * Add back in the GTF option * Remove the CDS + cytoband files * Rename ID column in consensus prep * Add consensus annotation to shell script; rerun * Remove numbering from RNA expression step * Update module README * Update modules at a glance entry * Add section summarizing seg.mean by status * Fix downstream modules * Update documentation to go back to gzipped file * Get a bit smarter about paths and filenames Also rerun + use $XYFLAG for the consensus step * Use compressed consensus SEG file * Rerun with new gzipped consensus SEG * Skip consensus file annotation in CI * Apply suggestions from code review Co-Authored-By: Candace Savonen <cansav09@gmail.com> * Update documentation * Remove outdated comment * Flesh out example usage Co-authored-by: Candace Savonen <cansav09@gmail.com>

cansavvy and others added 3 commits January 15, 2020 14:50

Changed some things to speed up the gene_id retrieval

b3f86ce

Merge branch 'master' of https://github.com/cbethell/OpenPBTA-analysis …

9cfe031

…into speed_up_granges

Update and revamp focal-cn-file-preparation module

3dd4e9c

- compare cytoband calls - update `README` - update comment in shell script and add `exon_file` option - Incorporate GISTIC calls in final cn tables - rerun `01-prepare-cn-file.R`

cbethell requested a review from cansavvy January 17, 2020 20:40

jaclyn-taroni reviewed Jan 17, 2020

View reviewed changes

analyses/focal-cn-file-preparation/README.md Outdated Show resolved Hide resolved

cansavvy reviewed Jan 17, 2020

View reviewed changes

This was referenced Jan 18, 2020

Updated analysis: RNA expression of copy number losses #387

Open

Proposed Analysis: map from SEG file to genes (and broader segments) #186

Closed

cbethell and others added 5 commits January 21, 2020 11:17

Merge branch 'master' of https://github.com/cbethell/OpenPBTA-analysis …

4a5c8a1

…into speed_up_granges

Merge branch 'master' into speed_up_granges

06843cb

Fix syntax error in shell script

712d029

- minor update to comment

Merge remote-tracking branch 'cansavvy/speed_up_granges' into speed_u…

5759811

…p_granges

cansavvy reviewed Jan 21, 2020

View reviewed changes

cbethell and others added 9 commits January 22, 2020 12:14

Rename cds_object as cds_gr

d549d5c

- update comments - rename gene id column - update comment in shell script - update column documentation in README - run styler

Merge branch 'master' into speed_up_granges

eb8a3ba

Add new line to .gitignore

00c4b42

Replace gz compressed files with bz2

95e0a44

- remove code in `.gitignore` file re large `cnvkit_annotated_autosomes.tsv.gz` file that no longer exists - remove `gzip` step in `run-prepare-cn.sh` as this is handled in the R script

Fix file path to gistic file

5d62eee

update file name in 02-rna-expression-validation.R script

90fe685

Update cn filename in downstream modules

99974f5

- update `run-oncoprint.sh`, `00-subset-files-for-ATRT.R`, `02-HGG-molecular-subtyping-subset-files.R`

Merge branch 'master' into speed_up_granges

0c18229

Update analyses/README.md to reflect annotated cn files changed format

b295399

cansavvy approved these changes Jan 22, 2020

View reviewed changes

jaclyn-taroni merged commit b4b5230 into AlexsLemonade:master Jan 22, 2020

jaclyn-taroni mentioned this pull request Jan 23, 2020

Annotated CNVkit data no longer shows SMARCB1 deletions in ATRT #473

Closed

jaclyn-taroni added a commit to jaclyn-taroni/OpenPBTA-analysis that referenced this pull request Jan 28, 2020

Back to exons; before GISTIC

cbd7fca

Pre: AlexsLemonade#452

jaclyn-taroni mentioned this pull request Jan 28, 2020

Update focal CN file prep to use exons again and cover the consensus SEG case #479

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update/Revamp `focal-cn-file-preparation` module #452

Update/Revamp `focal-cn-file-preparation` module #452

cbethell commented Jan 17, 2020

cansavvy left a comment

cansavvy Jan 17, 2020

cansavvy Jan 21, 2020

jaclyn-taroni commented Jan 18, 2020 •

edited

Loading

cansavvy left a comment

cansavvy Jan 21, 2020

cbethell commented Jan 22, 2020 •

edited

Loading

cansavvy commented Jan 22, 2020

cbethell commented Jan 22, 2020 •

edited

Loading

cansavvy left a comment •

edited

Loading

Update/Revamp focal-cn-file-preparation module #452

Update/Revamp focal-cn-file-preparation module #452

Conversation

cbethell commented Jan 17, 2020

Purpose/implementation Section

What GitHub issue does your pull request address?

Directions for reviewers. Tell potential reviewers what kind of feedback you are soliciting.

Which areas should receive a particularly close look?

Is there anything that you want to discuss further?

Is the analysis in a mature enough form that the resulting figure(s) and/or table(s) are ready for review?

Results

Reproducibility Checklist

Documentation Checklist

cansavvy left a comment

Choose a reason for hiding this comment

cansavvy Jan 17, 2020

Choose a reason for hiding this comment

cansavvy Jan 21, 2020

Choose a reason for hiding this comment

jaclyn-taroni commented Jan 18, 2020 • edited Loading

cansavvy left a comment

Choose a reason for hiding this comment

cansavvy Jan 21, 2020

Choose a reason for hiding this comment

cbethell commented Jan 22, 2020 • edited Loading

cansavvy commented Jan 22, 2020

cbethell commented Jan 22, 2020 • edited Loading

cansavvy left a comment • edited Loading

Choose a reason for hiding this comment

Update/Revamp `focal-cn-file-preparation` module #452

Update/Revamp `focal-cn-file-preparation` module #452

jaclyn-taroni commented Jan 18, 2020 •

edited

Loading

cbethell commented Jan 22, 2020 •

edited

Loading

cbethell commented Jan 22, 2020 •

edited

Loading

cansavvy left a comment •

edited

Loading