NPE When using GenomicsDB and GenotypeGVCFs with --max-alternates

[bbimber]

Hello,

I'm running GATK 4.2.5.0 with a command line this. Note --max-alternate-alleles and --genomicsdb-max-alternate-alleles.

/home/exacloud/gscratch/prime-seq/java/java8/bin/java \
	-Djava.io.tmpdir=<path> \
	-Xmx178g -Xms178g \
	-Xss2m \
	-jar GenomeAnalysisTK4.jar \
	GenotypeGVCFs \
	-R 128_Mmul_10.fasta \
	--variant gendb:///home/exacloud/gscratch/prime-seq/cachedData/16b9ede7-6db8-103a-9262-f8f3fc86a851/WGS_Feb22_1852.gdb \
	-O /home/exacloud/gscratch/prime-seq/workDir/1bb5295c-6ec5-103a-8692-f8f3fc86cd3f/Job1.work/WGS_pre-mGAPv2.3_1852.vcf.gz \
	--annotate-with-num-discovered-alleles \
	-stand-call-conf 30 \
	--max-alternate-alleles 6 \
	--genomicsdb-max-alternate-alleles 9 \
	--force-output-intervals mmul10.WGS-WXS.whitelist.v2.3.sort.merge.bed \
	-L 1:1-3714165 \
	--only-output-calls-starting-in-intervals \
	--genomicsdb-shared-posixfs-optimizations

I'm getting this NPE after it runs for about 40 minutes, meaning many sites were processed fine:

18 Feb 2022 11:32:59,897 DEBUG: 	java.lang.NullPointerException
18 Feb 2022 11:32:59,907 DEBUG: 		at org.broadinstitute.hellbender.utils.GenotypeUtils.computeDiploidGenotypeCounts(GenotypeUtils.java:85)
18 Feb 2022 11:32:59,919 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.annotator.ExcessHet.calculateEH(ExcessHet.java:96)
18 Feb 2022 11:32:59,943 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.annotator.ExcessHet.annotate(ExcessHet.java:84)
18 Feb 2022 11:32:59,955 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.annotator.VariantAnnotatorEngine.addInfoAnnotations(VariantAnnotatorEngine.java:355)
18 Feb 2022 11:32:59,971 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.annotator.VariantAnnotatorEngine.annotateContext(VariantAnnotatorEngine.java:334)
18 Feb 2022 11:32:59,985 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.annotator.VariantAnnotatorEngine.annotateContext(VariantAnnotatorEngine.java:306)
18 Feb 2022 11:32:59,995 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.GenotypeGVCFsEngine.regenotypeVC(GenotypeGVCFsEngine.java:185)
18 Feb 2022 11:33:00,007 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.GenotypeGVCFsEngine.callRegion(GenotypeGVCFsEngine.java:135)
18 Feb 2022 11:33:00,027 DEBUG: 		at org.broadinstitute.hellbender.tools.walkers.GenotypeGVCFs.apply(GenotypeGVCFs.java:283)
18 Feb 2022 11:33:00,035 DEBUG: 		at org.broadinstitute.hellbender.engine.VariantLocusWalker.lambda$null$1(VariantLocusWalker.java:161)
18 Feb 2022 11:33:00,042 DEBUG: 		at java.util.Iterator.forEachRemaining(Iterator.java:116)
18 Feb 2022 11:33:00,052 DEBUG: 		at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
18 Feb 2022 11:33:00,061 DEBUG: 		at java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:590)
18 Feb 2022 11:33:00,071 DEBUG: 		at org.broadinstitute.hellbender.engine.VariantLocusWalker.lambda$traverse$2(VariantLocusWalker.java:151)
18 Feb 2022 11:33:00,098 DEBUG: 		at java.util.Iterator.forEachRemaining(Iterator.java:116)
18 Feb 2022 11:33:00,120 DEBUG: 		at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
18 Feb 2022 11:33:00,135 DEBUG: 		at java.util.stream.ReferencePipeline$Head.forEachOrdered(ReferencePipeline.java:590)
18 Feb 2022 11:33:00,153 DEBUG: 		at org.broadinstitute.hellbender.engine.VariantLocusWalker.traverse(VariantLocusWalker.java:148)
18 Feb 2022 11:33:00,164 DEBUG: 		at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1085)
18 Feb 2022 11:33:00,171 DEBUG: 		at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
18 Feb 2022 11:33:00,176 DEBUG: 		at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
18 Feb 2022 11:33:00,182 DEBUG: 		at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
18 Feb 2022 11:33:00,188 DEBUG: 		at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
18 Feb 2022 11:33:00,194 DEBUG: 		at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
18 Feb 2022 11:33:00,200 DEBUG: 		at org.broadinstitute.hellbender.Main.main(Main.java:289)
18 Feb 2022 11:33:07,022 WARN : 	process exited with non-zero value: 3

which maps to:

gatk/src/main/java/org/broadinstitute/hellbender/utils/GenotypeUtils.java

Line 85 in da7cd83

final double[] normalizedLikelihoods = MathUtils.normalizeFromLog10ToLinearSpace(g.getLikelihoods().getAsVector());

final double[] normalizedLikelihoods = MathUtils.normalizeFromLog10ToLinearSpace(g.getLikelihoods().getAsVector());

Have you see issues like this before?

[droazen]

@ldgauthier Thoughts on this error? Is it related to the patch in #7670?

[droazen]

@bbimber We believe that this should be fixed by #7670, which will go out in the next GATK release. If you're able to test with that patch and give feedback on whether it resolves the error for you, that would be helpful!

[bbimber]

@droazen Is there a nightly or similar build I could take advantage of to get a JAR with this fix? I can built that branch locally if needed, but something automatic would save a little effort.

Any thoughts on how soon you'd consider making a minor release?

[droazen]

@bbimber Once that PR is merged, it will automatically go out as part of the next GATK nightly docker snapshot (https://hub.docker.com/r/broadinstitute/gatk-nightly). We are going to do a point release shortly after that PR goes in.

[bbimber]

ok, fantastic

[ldgauthier]

@bbimber that PR just went in and I still think it should solve your issue.

I introduced a bug in 4.2.5.0 in a conditional statement that let highly multi-allelic variants with no PLs get further than they should. I think that site had more than the GDB max alt count, so it didn’t get PLs, but with the bug I said that was okay if it was called. So it should get dropped with no PLs, but instead it snuck through and is throwing an NPE when ExcessHet goes to get the genotype counts. After the fix, the site will get dropped, which is consistent with the previous GenotypeGVCFs behavior.

[bbimber]

@ldgauthier Much appreciated. We're currently trying to figure out GenomicsDB memory issues (#7674), but this was going to be a blocking issue too. Any idea when you'll make a point release?

[droazen]

@bbimber Release is currently slated for tomorrow morning

[bbimber]

@droazen Have you given further thought to when you might make this release? Thanks.

[droazen]

Sorry for the delay @bbimber -- the release is currently being held up by an issue with the Google Cloud requester pays support that is affecting large numbers of GATK users (#7716). There is an open PR to fix this (#7730), but tests are not yet passing.

While you wait for the release to come out, you could use the latest nightly docker image (https://hub.docker.com/r/broadinstitute/gatk-nightly/) or build GATK from source to confirm that your NPE is indeed fixed as we believe it to be.