First Version of a weight-based splitter

[kcibul]

See Issue #7622 for more details

In addition to unit tests, here is how I validated:

##
# Split the WGS list
##
rm -rf test_split

./gatk --java-options "-Xmx4g $DEBUG" \
 WeightedSplitIntervals \
    --scatter-count 100 \
    --weight-bed-file gvs_vet_weights_1kb.bed \
    -R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta \
    --dont-mix-contigs true \
    -L wgs_calling_regions.hg38.noCentromeres.noTelomeres.interval_list \
    --output test_split

##
# merge all the intervals lists back into one
##
IL=""
for f in test_split/*-scattered.interval_list; do
    IL="${IL} -I $f "
done
./gatk IntervalListTools --ACTION UNION $IL -O test_split/merged.interval_list

#
# compare it to the original
##
./gatk CompareIntervalLists \
-R gs://gcp-public-data--broad-references/hg38/v0/Homo_sapiens_assembly38.fasta \
-L wgs_calling_regions.hg38.noCentromeres.noTelomeres.interval_list \
-L2 test_split/merged.interval_list

##
# A visual check to see that the ordering is the same, and that the only splits
# are across file boundaries
##
cat  test_split/*-scattered.interval_list | grep -v "@" | cut -f1-3 > test_split/combined.txt
cat  wgs_calling_regions.hg38.noCentromeres.noTelomeres.interval_list | grep -v "@" | cut -f1-3 > test_split/orig.txt
diff -y test_split/orig.txt test_split/combined.txt

[kcibul]

If we like this -- we also need to

• build a jar
• update the WDL to use this tool (and the Jar)
• Put the BED files someplace public/widely accessible (likely just the 1kb version)
• Run an E2E on QuickStart, merge the VCFs and compare (and see no differences)
• If we want to validate evenness we need to run with a lot of shards and enough data that they are interesting. Maybe Stroke 10k
• In parallel if we could turn some of the above script into integration tests that would be awesome

[lbergelson]

@kcibul Sorry for the extreme lateness, and the extreme number of comments. I ended up having a lot more to say than I expected. It really needs at least 1 integration test to show that it runs without exploding. If you're marking it as either a beta too or experimental we can get away with somewhat less responding to nitpicks and lower depth of testing, although if we're delivering things based on it we might at least want a test that asserts that a complex input interval list and weights file doesn't result in any LOST intervals anywhere.

Feel free to disregard what you think is over the top.

[lbergelson]

heh, recursive format strings.

[lbergelson]

Lets make this file name construction into a function since it's repeated. We could move it into the newly created OutputScatterArgumentCollection if we went that route.

[kcibul]

done, but this doesn't feel a whole lot cleaner 🤷