GVCF "reblocking" code to compress homRef blocks #4940

ldgauthier · 2018-06-22T18:46:32Z

This is a port of the GATK3 version I actually ran. Some differences in the genotyping engines and the fixed median calculation in MathUtils between GATK3 and GATK4 make the results slightly different in some cases, but still within tolerances.

lbergelson

@ldgauthier I did a pass on this. Sorry for the deluge of stylistic comments. They're mostly very simple changes. Thanks for porting this to gakt4!

lbergelson · 2018-06-22T19:46:34Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+import java.util.*;
+
+/**
+ * Perform joint genotyping on gVCF files produced by HaplotypeCaller


This copied comment isn't right. Needs up to date javadoc.

lbergelson · 2018-06-22T19:49:50Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+@DocumentedFeature
+public class ReblockGVCF extends VariantWalker {
+
+    private static String GVCF_BLOCK = "GVCFBlock";


expose this as a constant in GVCFWriter and then use that constant here, and in GenotypeGVCFs as well as in rangeToVCFHeaderLine in GVCFWriter

lbergelson · 2018-06-22T19:49:59Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+public class ReblockGVCF extends VariantWalker {
+
+    private static String GVCF_BLOCK = "GVCFBlock";
+    private static int PLOIDY_TWO = 2;  //FIXME: hack to avoid extra arg collections


make it final in any case

lbergelson · 2018-06-22T19:51:27Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+        programGroup = OtherProgramGroup.class,
+        omitFromCommandLine = true)
+@DocumentedFeature
+public class ReblockGVCF extends VariantWalker {


this should probably be final

lbergelson · 2018-06-22T19:52:26Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+    private static String GVCF_BLOCK = "GVCFBlock";
+    private static int PLOIDY_TWO = 2;  //FIXME: hack to avoid extra arg collections
+
+    private VariantContextWriter vcfWriter;


I would put the mutable fields after the argument definitions

lbergelson · 2018-06-26T20:59:20Z

...ava/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCFIntegrationTest.java

+
+        final ArgumentsBuilder args = new ArgumentsBuilder();
+        args.addReference(new File(b37_reference_20_21))
+                .addArgument("V", getToolTestDataDir() + "NA12878.prod.chr20snippet.g.vcf")


The large g.vcf and expected result should be put into src/test/resources/large since they're pretty big. It would be good to bgzip them too.

lbergelson · 2018-06-26T21:00:33Z

...ava/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCFIntegrationTest.java

+import java.util.Collections;
+
+/**
+ * Created by gauthier on 10/2/17.


we typically don't include these since git captures it

Finally found where to turn that off (Preferences > Editor > File and Code Templates > Includes).

lbergelson · 2018-06-26T21:08:30Z

src/main/java/org/broadinstitute/hellbender/utils/variant/writers/GVCFWriter.java

-                return null;
+            //there's a use case here related to ReblockGVCFs for overlapping deletions on different haplotypes
+            if ( vc.getStart() <= nextAvailableStart && vc.getContig().equals(contigOfNextAvailableStart) ) {
+                if (vc.getEnd() <= nextAvailableStart)


Could you add a unit case that triggers the first condition but not the second one?

Yep, I have new tests in GVCFWriterUnitTests.

lbergelson · 2018-06-26T21:12:03Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+        // Remove GCVFBlocks
+        headerLines.removeIf(vcfHeaderLine -> vcfHeaderLine.getKey().startsWith(GVCF_BLOCK));
+
+        headerLines.remove(GATKVCFHeaderLines.getInfoLine(GATKVCFConstants.DOWNSAMPLED_KEY));


what happens if someone adds these in annotationsToKeep?

lbergelson · 2018-06-26T21:14:16Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+     * @param originalVC     the combined genomic VC
+     * @return a new VariantContext or null if the site turned monomorphic and we don't want such sites
+     */
+    protected VariantContext regenotypeVC(final ReferenceContext ref, final VariantContext originalVC) {


Is this method named / documented correctly? It seems more like reblocking them with some minimal stripping out of alleles rather than regenotyping.

ldgauthier · 2018-07-03T19:49:53Z

@lbergelson I addressed all your comments. I had some difficulty reconciling the list of Strings for annotation names to remove with the List for the getDefaultVariantAnnotations update, so there may be a few tests failing because the RAW_MQ header disappeared. I'll deal with those when Travis is done, but could you take a look at the other changes?

lbergelson

@ldgauthier A few more comments minor cleanup comments and then I think it's good to go. It is failing a bunch of tests due to the change to the RAW_MQ tag. See integration tests and Unit test reports.

It looks like there's 1 place where you meant to add a test and forgot.

lbergelson · 2018-07-06T19:50:31Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/GenotypeGVCFs.java

        //obvious cases
        if (vc == null || vc.getAlternateAlleles().isEmpty()) {
            return false;
        } else if (vc.isBiallelic()) {
-            return !(isSpanningDeletion(vc.getAlternateAllele(0)) || vc.isSymbolic());
+            return !(GATKVCFConstants.isSpanningDeletion(vc.getAlternateAllele(0)) || vc.isSymbolic());


Thank you for doing this cleanup.

lbergelson · 2018-07-06T19:53:12Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+import java.util.*;
+
+/**
+ * Perform joint genotyping on gVCF files produced by HaplotypeCaller


I think this line is still out of place?

lbergelson · 2018-07-06T20:18:31Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+ * <pre>
+ * gatk ReblockGVCF \
+ *   -R reference.fasta \
+ *   --variant sample1.g.vcf \


I'd just make this -V to match the others.

lbergelson · 2018-07-06T20:19:37Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+            doc="File to which variants should be written")
+    private File outputFile;
+
+    @Argument(fullName=GenotypeGVCFs.ALL_SITES_LONG_NAME, shortName=GenotypeGVCFs.ALL_SITES_SHORT_NAME, doc="Include loci found to be non-variant after genotyping")


It's a little weird to make a boolean argument default to true. Should the name be inverted so that you specify it to disable it?

lbergelson · 2018-07-06T20:19:52Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+    public GenotypeCalculationArgumentCollection genotypeArgs = new GenotypeCalculationArgumentCollection();
+
+    @Advanced
+    @Argument(fullName="GVCF-GQ-band", shortName="GQB", doc="Exclusive upper bounds for reference confidence GQ bands " +


this should use the new GQ_BAND_LONG_NAME constants

lbergelson · 2018-07-06T20:28:41Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCFUnitTest.java

+        Assert.assertTrue(!newG.hasAD());
+    }
+
+    //TODO: these are duplicated from PosteriorProbabilitiesUtilsUnitTest but PR #4947 modifies VariantContextTestUtils, so I'll do some refactoring before the second of the two is merged


We should remember to resolve this when #4947 is completed.

lbergelson · 2018-07-06T20:29:15Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCFUnitTest.java

+    }
+
+    @Test
+    public void testLowQualVariantToGQ0HomRef() {


forgotten test?

lbergelson · 2018-07-06T20:30:13Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCFUnitTest.java

+import java.util.Map;
+
+public class ReblockGVCFUnitTest {
+    private final ReblockGVCF reblocker = new ReblockGVCF();


I would just create this in each test that uses it, otherwise we potentially get weird stateful issues across multiple tests.

lbergelson · 2018-07-06T20:30:55Z

src/test/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCFUnitTest.java

+import java.util.List;
+import java.util.Map;
+
+public class ReblockGVCFUnitTest {


Thanks for adding these tests. Complicated functions like these have a tendency to lose bits of logic later on if there aren't direct tests for them.

lbergelson · 2018-07-06T20:32:02Z

src/test/java/org/broadinstitute/hellbender/utils/variant/writers/GVCFWriterUnitTest.java

+        final Allele alt1 = Allele.create("T", false);
+        final Allele ref2 = Allele.create("TACACACACACTACTA", true);
+        final Allele ref3 = Allele.create("C", true);
+        final VariantContext deletion1 = new VariantContextBuilder(null, "1", 10000, 10020, Arrays.asList(ref1, alt1,


setting these things up is always such a pain, thank you for writing it

codecov-io · 2018-07-09T16:07:31Z

Codecov Report

Merging #4940 into master will decrease coverage by 0.098%.
The diff coverage is 87.733%.

@@               Coverage Diff               @@
##              master     #4940       +/-   ##
===============================================
- Coverage     86.753%   86.655%   -0.098%     
+ Complexity     29767     29110      -657     
===============================================
  Files           1825      1811       -14     
  Lines         137744    135032     -2712     
  Branches       15181     14982      -199     
===============================================
- Hits          119497    117012     -2485     
+ Misses         12729     12576      -153     
+ Partials        5518      5444       -74

Impacted Files	Coverage Δ	Complexity Δ
...otypecaller/HaplotypeCallerArgumentCollection.java	`100% <ø> (ø)`	`2 <0> (ø)`	⬇️
...org/broadinstitute/hellbender/utils/MathUtils.java	`78.17% <ø> (ø)`	`209 <0> (ø)`	⬇️
...ute/hellbender/utils/variant/GATKVCFConstants.java	`80% <ø> (ø)`	`4 <0> (ø)`	⬇️
...der/tools/walkers/annotator/RMSMappingQuality.java	`93.814% <100%> (ø)`	`40 <2> (ø)`	⬇️
...e/hellbender/utils/variant/writers/GVCFWriter.java	`93.333% <100%> (+0.075%)`	`34 <1> (+1)`	⬆️
.../hellbender/utils/variant/writers/HomRefBlock.java	`93.243% <100%> (+0.188%)`	`36 <4> (+1)`	⬆️
...ellbender/tools/walkers/annotator/QualByDepth.java	`97.436% <100%> (+0.139%)`	`18 <2> (+1)`	⬆️
...plotypecaller/HaplotypeCallerGenotypingEngine.java	`78.014% <100%> (-1.856%)`	`32 <4> (-5)`
...ellbender/tools/walkers/GenotypeGVCFsUnitTest.java	`100% <100%> (ø)`	`21 <1> (ø)`	⬇️
...s/walkers/annotator/RMSMappingQualityUnitTest.java	`96.859% <100%> (ø)`	`26 <0> (ø)`	⬇️
... and 290 more

ldgauthier · 2018-08-22T15:31:43Z

@lbergelson this is good to go, unless my other PR gets merged first, in which case I'll need to update the expected test data.

lbergelson · 2018-08-22T16:35:41Z

That person might have been me...

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

cmnbroad · 2018-09-06T16:59:50Z

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

+ *
+ * <h3>Caveats</h3>
+ * <p>Only single-sample gVCF files produced by HaplotypeCaller can be used as input for this tool.</p>
+ * <p>By default this tool only passes through annotations used by VQSR.  A different set of annotations can be specified with the usual -A argument.  Annotation groups are not supported by this tool.</p>


I think annotation groups are automatically supported and propagated to the annotation engine via makeVariantAnnotations, unless the tool manually intervenes.

I agree -- that was probably just a leftover from the GATK3 version.

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java

lbergelson · 2018-10-03T15:34:52Z

@ldgauthier I think there's an newly introduce compile warning because of the method names that I deprecated in htsjdk.

lbergelson

This looks good to me with a quick re-review. There's a compilation warning that's failing things due to that deprecated Indeces function, quick fix to that and I think you're good to merge 👍

…GVCFs for smaller filesize

* adds ReblockGVCF a tool to merge reference blocks in single-sample GVCFs for smaller file size

lbergelson self-requested a review June 25, 2018 17:21

lbergelson requested changes Jun 26, 2018

View reviewed changes

lbergelson assigned ldgauthier Jun 26, 2018

ldgauthier assigned lbergelson and unassigned ldgauthier Jul 3, 2018

lbergelson reviewed Jul 6, 2018

View reviewed changes

lbergelson assigned ldgauthier and unassigned lbergelson Jul 6, 2018

ldgauthier assigned lbergelson and unassigned ldgauthier Aug 22, 2018

ldgauthier force-pushed the ldg_ReblockGVCF branch from d7ed84e to 0d1bb61 Compare August 22, 2018 15:31

droazen reviewed Aug 22, 2018

View reviewed changes

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java Show resolved Hide resolved

droazen reviewed Aug 22, 2018

View reviewed changes

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java Show resolved Hide resolved

cmnbroad reviewed Sep 6, 2018

View reviewed changes

src/main/java/org/broadinstitute/hellbender/tools/walkers/variantutils/ReblockGVCF.java Show resolved Hide resolved

ldgauthier force-pushed the ldg_ReblockGVCF branch from 551ae34 to 13db51e Compare October 1, 2018 18:25

lbergelson approved these changes Oct 3, 2018

View reviewed changes

lbergelson assigned ldgauthier and unassigned lbergelson Oct 3, 2018

Add ReblockGVCF -- a tool to merge reference blocks in single-sample …

22c4be1

…GVCFs for smaller filesize

ldgauthier force-pushed the ldg_ReblockGVCF branch from 13db51e to 22c4be1 Compare October 3, 2018 17:48

lbergelson merged commit 7368c62 into master Oct 3, 2018

lbergelson deleted the ldg_ReblockGVCF branch October 3, 2018 20:11

tomwhite mentioned this pull request Oct 16, 2018

Refactor GVCFWriter to allow push/pull iteration. #5311

Merged

EdwardDixon pushed a commit to EdwardDixon/gatk that referenced this pull request Nov 9, 2018

Add new tool ReblockGVCF (broadinstitute#4940)

545a333

* adds ReblockGVCF a tool to merge reference blocks in single-sample GVCFs for smaller file size

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GVCF "reblocking" code to compress homRef blocks #4940

GVCF "reblocking" code to compress homRef blocks #4940

ldgauthier commented Jun 22, 2018

lbergelson left a comment

lbergelson Jun 22, 2018

lbergelson Jun 22, 2018

lbergelson Jun 22, 2018

lbergelson Jun 22, 2018

lbergelson Jun 22, 2018

lbergelson Jun 22, 2018

lbergelson Jun 26, 2018

lbergelson Jun 26, 2018

ldgauthier Jun 29, 2018

lbergelson Jun 26, 2018

ldgauthier Jul 3, 2018

lbergelson Jun 26, 2018

lbergelson Jun 26, 2018

ldgauthier commented Jul 3, 2018

lbergelson left a comment

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

lbergelson Jul 6, 2018

codecov-io commented Jul 9, 2018 •

edited

Loading

ldgauthier commented Aug 22, 2018

lbergelson commented Aug 22, 2018

cmnbroad Sep 6, 2018

ldgauthier Oct 1, 2018

lbergelson commented Oct 3, 2018

lbergelson left a comment

GVCF "reblocking" code to compress homRef blocks #4940

GVCF "reblocking" code to compress homRef blocks #4940

Conversation

ldgauthier commented Jun 22, 2018

lbergelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldgauthier commented Jul 3, 2018

lbergelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jul 9, 2018 • edited Loading

Codecov Report

ldgauthier commented Aug 22, 2018

lbergelson commented Aug 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbergelson commented Oct 3, 2018

lbergelson left a comment

Choose a reason for hiding this comment

codecov-io commented Jul 9, 2018 •

edited

Loading