Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

adding --sort-order option to SortSamSpark #4545

Merged
merged 7 commits into from
Jun 11, 2018
Merged

adding --sort-order option to SortSamSpark #4545

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a29ab53
adding --sort-order option to SortSamSpark
lbergelson Mar 20, 2018
d20579b
fix tests
lbergelson May 25, 2018
4d4265b
more test output to try to figure out what's going on in travis
lbergelson May 31, 2018
8855b4a
trying to sort splits
lbergelson Jun 5, 2018
2408189
trying again with better casts, but still bad
lbergelson Jun 6, 2018
558a13f
adding doc to SplitSortingSamInputFormat
lbergelson Jun 6, 2018
72997c5
responding to james
lbergelson Jun 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions src/main/java/org/broadinstitute/hellbender/engine/spark/datasources/ReadsSparkSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,7 @@
import org.broadinstitute.hellbender.utils.io.IOUtils;
import org.broadinstitute.hellbender.utils.read.*;
import org.broadinstitute.hellbender.utils.spark.SparkUtils;
import org.seqdoop.hadoop_bam.AnySAMInputFormat;
import org.seqdoop.hadoop_bam.BAMInputFormat;
import org.seqdoop.hadoop_bam.CRAMInputFormat;
import org.seqdoop.hadoop_bam.SAMRecordWritable;
import org.seqdoop.hadoop_bam.*;
import org.seqdoop.hadoop_bam.util.SAMHeaderReader;

import java.io.File;
Expand All @@ -40,6 +37,7 @@
import java.util.Comparator;
import java.util.List;
import java.util.Objects;
import java.util.stream.Stream;

/** Loads the reads from disk either serially (using samReaderFactory) or in parallel using Hadoop-BAM.
* The parallel code is a modified version of the example writing code from Hadoop-BAM.
Expand Down Expand Up @@ -76,14 +74,22 @@ public JavaRDD<GATKRead> getParallelReads(final String readFileName, final Strin


public static class SplitSortingSamInputFormat extends AnySAMInputFormat{

@SuppressWarnings("unchecked")
@Override
public List<InputSplit> getSplits(JobContext job) throws IOException {
final List<InputSplit> splits = super.getSplits(job);
splits.sort((a, b) -> {
FileSplit fa = (FileSplit) a, fb = (FileSplit) b;
return fa.getPath().compareTo(fb.getPath());
});


if( splits.stream().allMatch(split -> split instanceof FileVirtualSplit || split instanceof FileSplit)) {
splits.sort(Comparator.comparing(split -> {
if (split instanceof FileVirtualSplit) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mix the virtual and non-virtual splits here. This makes me uncomfortable as I don't know that its guaranteed either is going to return the same thing for getPath(). Could you use the filename instead? As that seems to be what is being sorted here anyway.

return ((FileVirtualSplit) split).getPath();
} else {
return ((FileSplit) split).getPath();
}
}));
}

return splits;
}
}
Expand Down