Benchmark ReadsPipelineSpark against running its component Spark tools individually

[droazen]

No description provided.

[tomwhite]

See https://github.com/broadinstitute/gatk/wiki/Spark-Evaluation-Results

[droazen]

These initial results suggest that the savings from a pure-Spark pipeline are in the 15-30% range. @tomwhite Do you attribute these savings mostly to avoiding writing/reading intermediate outputs?

[droazen]

Also, once we've confirmed these results, we'll want to compare the total core hours of the Spark pipeline against the core hours of an equivalent non-Spark pipeline, to see if the savings provided by a pure-Spark pipeline actually make it cheaper than a non-Spark pipeline.

[tomwhite]

Closing this as we have more Spark speed improvements now (see e.g. #5127). We can open new issues to track further Spark peformance improvements.