BGE settings for Reblocking and JointCalling pipelines #1076

meganshand · 2023-09-08T20:38:58Z

Description

This is a draft of how we'd like to run joint calling on BGE with a few changes to the current pipeline:

Use an updated variant calling interval list (bge v1.1 calling intervals)
Reblocking
- Use an updated version of ReblockGVCFs to work with DRAGEN input
- Added argument to strip “PRI” annotation: --format-annotations-to-remove PRI
- Add argument to move site level filters to genotype level instead: --add-site-filters-to-genotype
- No need to do extra blocking at this step since the block sizes are now set in the DRAGEN/single sample pipeline
Filtering
- Allow for the option to use the newer VETS pipeline rather than VQSR in JointGenotyping
- Hard filter sites that land outside of the targeted regions (but within the exome variant calling interval list)
- Train the VETS model only over the targeted regions of the BGE (where there's expected coverage)
- Provide the SCORE annotation calculated by the VETS model for all variants, but don't use it as an applied filter so that users can choose their own cutoff values

Making two paths in the JointCalling pipeline for two different filtering strategies was a bit messy, so I'm happy to hear if there are any recommendations on how to clean that up. I also tried to run the tests that already exist for these workflows, but ran into errors that seemed unrelated.

@jessicaway For now I left out any changes to the ByChromosome workflows and we can cross that bridge when we come to it.

Checklist

If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP documentation team by tagging either @ekiernan or @kayleemathews in a comment on this PR.

Did you add inputs, outputs, or tasks to a workflow?
Did you modify, delete or move: file paths, file names, input names, output names, or task names?
If you made a changelog update, did you update the pipeline version number?

github-actions · 2023-09-08T20:39:09Z

Remember to squash merge!

github-actions · 2023-09-12T18:49:29Z

Remember to squash merge!

github-actions · 2023-09-14T16:08:54Z

Remember to squash merge!

meganshand · 2023-09-18T18:25:27Z

@jessicaway @samuelklee This is ready for review. As discussed in the meeting this morning the two remaining issues are:

We'll need to update the version of GATK once it's been released
Test files still need to be copied into the appropriate bucket for all the tests to pass here

I don't think either of these should stop an initial review though. Thanks!

meganshand · 2023-09-18T18:26:20Z

@ekiernan This PR will need some documentation updates for the BGE pipeline. I also submitted a request through the portal. Happy to work with you on this.

samuelklee · 2023-09-18T18:39:46Z

Great, thanks @meganshand! Should be able to get an initial review back to you in the next couple days.

Would be great to also get broadinstitute/gatk#8131 in before the next GATK release. Let's chat about whether that necessitates any changes here---I don't think it should, but that PR has been on the back burner for a bit, so I'll have to double check.

samuelklee

Thanks @meganshand, just a few minor comments/questions from me!

samuelklee · 2023-09-20T16:05:50Z

pipelines/broad/dna_seq/germline/joint_genotyping/JointGenotyping.wdl

+
+  if (run_vets) {
+    String allele_specific_extra_args = if allele_specific_annotations then " --use-allele-specific-annotations " else ""
+    String resource_args = " --resource:hapmap,training=true,calibration=true " + hapmap_resource_vcf + 


Would be nice to expose this and demote the default resources from required arguments at some point, but this is fine for now.

samuelklee · 2023-09-20T16:08:07Z

pipelines/broad/dna_seq/germline/joint_genotyping/JointGenotyping.wdl

-  if (num_gvcfs > snps_variant_recalibration_threshold) {
-    call Tasks.SNPsVariantRecalibratorCreateModel {
+
+  if (run_vets) {


Ideally, we'd have separate VQSR and VETS subworkflows, and the latter would be a direct import of the JointVcfFiltering WDL in the GATK repo. Either we'll get there in the future or we'll remove VQSR, let's see which happens first!

Agreed, I can take a stab at pulling out VQSR into it's own subworkflow if that's helpful, but it sounds like now might not be the right time if we're close to deprecating VQSR completely.

samuelklee · 2023-09-20T16:10:06Z

pipelines/broad/dna_seq/germline/joint_genotyping/reblocking/ReblockGVCF.changelog.md

@@ -1,3 +1,9 @@
+# 2.1.6


Do you need to update some of these changelogs now that #1081 is in?

Yes, I did this to myself :(

I'll fix this when I have to resolve conflicts.

samuelklee · 2023-09-20T16:11:00Z

pipelines/broad/dna_seq/germline/joint_genotyping/test_inputs/Scientific/bge.json

+  "JointGenotyping.one_thousand_genomes_resource_vcf": "gs://gcp-public-data--broad-references/hg38/v0/1000G_phase1.snps.high_confidence.hg38.vcf.gz",
+  "JointGenotyping.one_thousand_genomes_resource_vcf_index": "gs://gcp-public-data--broad-references/hg38/v0/1000G_phase1.snps.high_confidence.hg38.vcf.gz.tbi",
+
+  "JointGenotyping.snp_recalibration_annotation_values": ["QD", "MQRankSum", "ReadPosRankSum", "FS", "MQ", "SOR", "DP"],


Why are these not allele-specific annotations, when use_allele_specific_annotations = true below?

Ooo good catch, I think dragen might not be able to generate allele specific annotations, or at the very least we're not currently set up to do that for BGE so we might have to run without allele specific annotations for now. I changed use_allele_specific_annotations to false for now, but this will also be a non-issue when we move to the new version of GATK/VETS (based on your other PR)

samuelklee · 2023-09-20T16:12:36Z

pipelines/broad/dna_seq/germline/joint_genotyping/test_inputs/Scientific/bge.json

+  "JointGenotyping.callset_name": "bge_joint_genotyping_plumbing",
+  "JointGenotyping.run_vets": true,
+
+  "JointGenotyping.unpadded_intervals_file": "",


Were unpadded_intervals_file, eval_interval_list, and targets_interval_list intentionally left empty for now?

Yeah, I need to work with @jessicaway on copying the input files (the Twist targets interval list and the sample files) into the test bucket get these two new tests to run.

samuelklee · 2023-09-20T16:14:27Z

tasks/broad/JointGenotypingTasks.wdl

@@ -273,6 +273,7 @@ task HardFilterAndMakeSitesOnlyVcf {

    String variant_filtered_vcf_filename
    String sites_only_vcf_filename
+    File? targets_interval_list # filters out all variants outside of the targets interval list for targetted sequencing


targetted -> targeted.

I'm also unclear about whether such changes to tasks need to be reflected in a changelog somewhere?

I put in a line in the JointGenotyping and UltimaJointGenotyping changelogs (which I think are the only pipelines that uses this task):

Added option to hard filter sites outside of provided interval list

Let me know if there's a clearer way to put that.

reblocking test docker fixing TDR index issue in reblock fix the rest of the bge changes adding bge inputs fixes comment renaming inputs fix

… map

github-actions · 2023-12-14T20:37:00Z

Remember to squash merge!

github-actions · 2023-12-14T21:00:32Z

Remember to squash merge!

meganshand · 2023-12-15T21:11:25Z

The failing smart tests here are due to a change in the VCF header when updating to GATK 4.5.0.0:

< ##INFO=<ID=RAW_GT_COUNT,Number=3,Type=Integer,Description="Counts of genotypes w.r.t. the reference allele in the following order: 0/0, 0/*, */*, i.e. all alts lumped together; for use in calculating excess heterozygosity">
---
> ##INFO=<ID=RAW_GT_COUNT,Number=3,Type=Integer,Description="Counts of genotypes w.r.t. the reference allele: 0/0, 0/*, */*, i.e. all alts lumped together; for use in calculating excess heterozygosity">

This change seems harmless to me, but I'm not sure who needs to sign off on this change. The PR with the rest of the GATK updates to 4.5.0.0 is still in progress, but should be coming soon.

github-actions · 2023-12-18T19:00:24Z

Remember to squash merge!

github-actions · 2023-12-19T18:21:46Z

Remember to squash merge!

meganshand · 2024-02-02T15:19:49Z

Failing Scientific Tests that can be updated:

JointGenotyping: BGE test is failing because the truth data is out of data. This can safely be updated as these differences are expected.
ReblockGVCFs: Tests are failing because documentation on a header line changed (see comment above). This can safely be updated.
VariantCalling: Same header line change as ReblockGVCFs. This can be updated.

github-actions · 2024-02-05T18:00:36Z

Remember to squash merge!

github-actions · 2024-02-05T20:28:57Z

Remember to squash merge!

nikellepetrillo · 2024-02-06T16:41:40Z

after merging in the latest develop, we re-ran JG, variant calling, and reblocking against Megan's truth (which was previously validated):

https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/65222/console
https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/65156/console/
https://gotc-jenkins.dsp-techops.broadinstitute.org/job/warp-workflow-tests/65155/console

pullapprove bot requested review from kachulis and samuelklee September 8, 2023 20:39

samuelklee requested changes Sep 20, 2023

View reviewed changes

samuelklee mentioned this pull request Sep 21, 2023

Performed a round of ablation on new annotation-based filtering tools. broadinstitute/gatk#8131

Merged

meganshand added 13 commits December 14, 2023 15:21

reblocking changes for BGE

2a21c73

reblocking test docker fixing TDR index issue in reblock fix the rest of the bge changes adding bge inputs fixes comment renaming inputs fix

changelogs

8e0ef79

trying to fix tests

bb5c6d3

moving test to scientific

ae22ff5

dragen gvcfs have a different file extension

d3ccab0

missed one

7a0a84b

not sure if this fixes the test because the input file is in a sample…

3ca7932

… map

addressing comments

35e65b2

fixing bge tests

e6d7ddf

fixing tests

6a68f7d

typo

baa4eec

one more

83c6945

updating to gatk 4.5.0.0

7beeae1

meganshand force-pushed the ms_bge branch from 27b2b66 to 7beeae1 Compare December 14, 2023 20:36

fixing vesion numbers

205eb2f

missed one update to 4.5.0.0

6eaabc9

update for new version of VETS

97588f3

jessicaway approved these changes Feb 2, 2024

View reviewed changes

Merge branch 'develop' into ms_bge

7906a91

nikellepetrillo approved these changes Feb 5, 2024

View reviewed changes

Merge branch 'develop' into ms_bge

d6bee0d

nikellepetrillo merged commit 45928e3 into develop Feb 6, 2024
4 of 6 checks passed

meganshand deleted the ms_bge branch March 26, 2024 12:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BGE settings for Reblocking and JointCalling pipelines #1076

BGE settings for Reblocking and JointCalling pipelines #1076

meganshand commented Sep 8, 2023

github-actions bot commented Sep 8, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 14, 2023

meganshand commented Sep 18, 2023

meganshand commented Sep 18, 2023

samuelklee commented Sep 18, 2023

samuelklee left a comment

samuelklee Sep 20, 2023

samuelklee Sep 20, 2023

meganshand Sep 20, 2023

samuelklee Sep 20, 2023

meganshand Sep 20, 2023 •

edited

Loading

samuelklee Sep 20, 2023

meganshand Sep 21, 2023

samuelklee Sep 20, 2023

meganshand Sep 20, 2023

samuelklee Sep 20, 2023

meganshand Sep 20, 2023

github-actions bot commented Dec 14, 2023

github-actions bot commented Dec 14, 2023

meganshand commented Dec 15, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 19, 2023

meganshand commented Feb 2, 2024

github-actions bot commented Feb 5, 2024

github-actions bot commented Feb 5, 2024

nikellepetrillo commented Feb 6, 2024

BGE settings for Reblocking and JointCalling pipelines #1076

BGE settings for Reblocking and JointCalling pipelines #1076

Conversation

meganshand commented Sep 8, 2023

Description

Checklist

github-actions bot commented Sep 8, 2023

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 14, 2023

meganshand commented Sep 18, 2023

meganshand commented Sep 18, 2023

samuelklee commented Sep 18, 2023

samuelklee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meganshand Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 14, 2023

github-actions bot commented Dec 14, 2023

meganshand commented Dec 15, 2023

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 19, 2023

meganshand commented Feb 2, 2024

github-actions bot commented Feb 5, 2024

github-actions bot commented Feb 5, 2024

nikellepetrillo commented Feb 6, 2024

meganshand Sep 20, 2023 •

edited

Loading