Replaced IntervalsSkipList with OverlapDetector

[davidbenjamin]

Closes #3541. Closes #3608.

@lbergelson May I assign you by virtue of your github activity on this and related issues?

Note that OverlapDetector::getOverlaps returns an unsorted Set so I sorted its output in a few places. I don't know if this was the right solution.

[droazen]

@davidbenjamin Have you actually profiled this change on a Spark/dataproc cluster?

[lbergelson]

@droazen Adam did profiling that showed that overlaps detector was strictly better. We also had some suspicion that there was some bug lurking in the skip list implementation because of weird non-deterministic results of something that relied on it.

[droazen]

@lbergelson Adam did the profiling? That must have been years ago. Someone definitely needs to profile this change on a Spark cluster using the latest master to see if his ancient findings still hold up.

[davidbenjamin]

@lbergelson @droazen I can do it but I would need a lot of hand-holding for both the profiling and the Spark.

[droazen]

We're happy to help -- I think it's worth doing even if this is a no-brainer substitution, as it's easy to break Spark functionality on a cluster by silly things like introducing a non-serializable class in the wrong place.

[davidbenjamin]

@droazen When we left off you showed me how to use JProfiler (which I now use regularly) and had pointed me to instructions for running GATK on a Spark cluster. I realize I'm going to need another round of hand-holding, because I'm not sure which Spark cluster to test on, and I'm not sure which command to test. And beyond measuring total runtime and looking for any new hotspots in JProfiler I don't know what else to measure. Could I get some more help from someone on the engine team?

[droazen]

@jamesemery Can you make a decision on what to do with this one after we merge #5292 ?

[codecov-io]

Codecov Report

Merging #4154 into master will increase coverage by 0.006%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##              master     #4154       +/-   ##
===============================================
+ Coverage     87.037%   87.043%   +0.006%     
+ Complexity     31728     31682       -46     
===============================================
  Files           1943      1938        -5     
  Lines         146193    146014      -179     
  Branches       16141     16117       -24     
===============================================
- Hits          127242    127095      -147     
+ Misses         13064     13035       -29     
+ Partials        5887      5884        -3

Impacted Files	Coverage Δ	Complexity Δ
...der/engine/spark/datasources/ReadsSparkSource.java	61.333% <ø> (ø)	17 <0> (ø)	⬇️
...nder/utils/runtime/StreamingProcessController.java	67.773% <0%> (+0.474%)	33% <0%> (ø)	⬇️

[lbergelson]

This is easily resolvable without performance profiling since the only remaining use of the IntervalSkipList was in ContextShard which is now also unused / untested. Removing context shard and rebasing.

[lbergelson]

@davidbenjamin I've stolen this branch from you. I hope you don't mind.

[davidbenjamin]

@lbergelson No objection. It's in better hands than mine now.

[lbergelson]

I approve this PR...

[lbergelson]

@davidbenjamin Thank you for this one!