Upgrade to Spark 2.4.3

[tomwhite]

Fixes #5782

[codecov]

Codecov Report

Merging #5990 into master will decrease coverage by 22.542%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##              master     #5990        +/-   ##
================================================
- Coverage     86.927%   64.385%   -22.542%     
+ Complexity     32732     15800     -16932     
================================================
  Files           2014      1263       -751     
  Lines         151333     73286     -78047     
  Branches       16612     11785      -4827     
================================================
- Hits          131549     47185     -84364     
- Misses         13724     21227      +7503     
+ Partials        6060      4874      -1186

Impacted Files	Coverage Δ	Complexity Δ
...broadinstitute/hellbender/utils/svd/SimpleSVD.java	0% <0%> (-100%)	0% <0%> (-5%)
...groups/StructuralVariantDiscoveryProgramGroup.java	0% <0%> (-100%)	0% <0%> (-3%)
...ls/variant/writers/GVCFBlockCombiningIterator.java	0% <0%> (-100%)	0% <0%> (-1%)
...ender/utils/svd/ApacheSingularValueDecomposer.java	0% <0%> (-100%)	0% <0%> (-2%)
...otypecaller/RandomLikelihoodCalculationEngine.java	0% <0%> (-100%)	0% <0%> (-6%)
...sformers/MisencodedBaseQualityReadTransformer.java	0% <0%> (-100%)	0% <0%> (-4%)
...er/utils/smithwaterman/SWNativeAlignerWrapper.java	0% <0%> (-100%)	0% <0%> (-6%)
...ne/programgroups/CoverageAnalysisProgramGroup.java	0% <0%> (-100%)	0% <0%> (-3%)
...institute/hellbender/cmdline/TestProgramGroup.java	0% <0%> (-100%)	0% <0%> (-3%)
...ls/walkers/genotyper/HeterogeneousPloidyModel.java	0% <0%> (-100%)	0% <0%> (-14%)
... and 1364 more

[codecov]

Codecov Report

Merging #5990 into master will decrease coverage by 73.828%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             master     #5990        +/-   ##
===============================================
- Coverage     87.21%   13.381%   -73.828%     
+ Complexity    32711      6382     -26329     
===============================================
  Files          2011      2011                
  Lines        150912    150912                
  Branches      16131     16131                
===============================================
- Hits         131610     20194    -111416     
- Misses        13691    128454    +114763     
+ Partials       5611      2264      -3347

Impacted Files	Coverage Δ	Complexity Δ
...ls/variant/writers/GVCFBlockCombiningIterator.java	0% <0%> (-100%)	0% <0%> (-1%)
...ools/copynumber/models/AlleleFractionSamplers.java	0% <0%> (-100%)	0% <0%> (-2%)
...ark/AssemblyRegionReadShardArgumentCollection.java	0% <0%> (-100%)	0% <0%> (-1%)
...nder/tools/copynumber/utils/TagGermlineEvents.java	0% <0%> (-100%)	0% <0%> (-3%)
...r/tools/spark/pathseq/PSBwaArgumentCollection.java	0% <0%> (-100%)	0% <0%> (-1%)
...engine/spark/AssemblyRegionArgumentCollection.java	0% <0%> (-100%)	0% <0%> (-1%)
...titute/hellbender/utils/mcmc/DecileCollection.java	0% <0%> (-100%)	0% <0%> (-5%)
...ender/tools/readersplitters/ReadGroupSplitter.java	0% <0%> (-100%)	0% <0%> (-3%)
...nder/engine/spark/AssemblyRegionWalkerContext.java	0% <0%> (-100%)	0% <0%> (-4%)
...ynumber/models/AlleleFractionGlobalParameters.java	0% <0%> (-100%)	0% <0%> (-10%)
... and 1767 more

[tomwhite]

Upgrading to Spark 2.4 causes build errors that are fixed by also upgrading to Gradle 5. (Without upgrading Gradle, the shadow JAR could not be built, and I couldn't find a fix that didn't involve a Gradle upgrade.) The remaining failures are all due to the docker build with Gradle 5, see #6007 for a standalone PR tracking this.

[tomwhite]

Tests pass with #6007.

[tomwhite]

This can be merged now that #6007 is in. All tests are passing and I've also tested running a Spark job on a cluster with this change.