-
Notifications
You must be signed in to change notification settings - Fork 586
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[VS-693] Add support for VQSR Lite to GvsCreateFilterSet (#8157)
* Added a new suite of tools for variant filtering based on site-level annotations. (#7954) * Adds wdl that tests joint VCF filtering tools (#7932) * adding filtering wdl * renaming pipeline * addressing comments * added bash * renaming json * adding glob to extract for extra files * changing dollar signs * small comments * Added changes for specifying model backend and other tweaks to WDLs and environment. * Added classes for representing a collection of labeled variant annotations. * Added interfaces for modeling and scoring backends. * Added a new suite of tools for variant filtering based on site-level annotations. * Added integration tests. * Added test resources and expected results. * Miscellaneous changes. * Removed non-ASCII characters. * Added documentation for TrainVariantAnnotationsModel and addressed review comments. Co-authored-by: meganshand <[email protected]> * Added toggle for selecting resource-matching strategies and miscellaneous minor fixes to new annotation-based filtering tools. (#8049) * Adding use_allele_specific_annotation arg and fixing task with empty input in JointVcfFiltering WDL (#8027) * Small changes to JointVCFFiltering WDL * making default for use_allele_specific_annotations * addressing comments * first stab * wire through WDL changes * fixed typo * set model_backend input value * add gatk_override to JointVcfFiltering call * typo in indel_annotations * make model_backend optional * tabs and spaces * make all model_backends optional * use gatk 4.3.0 * no point in changing the table names as this is a POC * adding new branch to dockstore * adding in branching logic for classic VQSR vs VQSR-Lite * implementing the separate schemas for the VQSR vs VQSR-Lite branches, including Java changes necessary to produce the different tsv files * passing classic flag to indel run of CreateFilteringFiles * Update GvsCreateFilterSet.wdl cleaning up verbiage * Removed mapping error rate from estimate of denoised copy ratios output by gCNV and updated sklearn. (#7261) * cleanup up sloppy comment --------- Co-authored-by: samuelklee <[email protected]> Co-authored-by: meganshand <[email protected]> Co-authored-by: Rebecca Asch <[email protected]>
- Loading branch information
1 parent
053594d
commit cdb74b7
Showing
209 changed files
with
5,259 additions
and
129 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
# Filtering Automated Tests for WDL | ||
|
||
**This directory is for GATK devs only** | ||
|
||
This directory contains scripts for running Variant Site Level WDL tests in the automated travis build environment. | ||
|
||
Please note that this only tests whether the WDL will complete successfully. | ||
|
||
Test data is a "plumbing test" using a small portion of a 10 sample callset. |
38 changes: 38 additions & 0 deletions
38
scripts/vcf_site_level_filtering_cromwell_tests/run_vcf_site_level_filtering_wdl.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
#!/bin/bash -l | ||
set -e | ||
#cd in the directory of the script in order to use relative paths | ||
script_path=$( cd "$(dirname "${BASH_SOURCE}")" ; pwd -P ) | ||
cd "$script_path" | ||
|
||
WORKING_DIR=/home/runner/work/gatk | ||
|
||
set -e | ||
echo "Building docker image for VCF Site Level Filtering WDL tests (skipping unit tests)..." | ||
|
||
#assume Dockerfile is in root | ||
echo "Building docker without running unit tests... =========" | ||
cd $WORKING_DIR/gatk | ||
|
||
# IMPORTANT: This code is duplicated in the cnv and M2 WDL test. | ||
if [ ! -z "$CI_PULL_REQUEST" ]; then | ||
HASH_TO_USE=FETCH_HEAD | ||
sudo bash build_docker.sh -e ${HASH_TO_USE} -s -u -d $PWD/temp_staging/ -t ${CI_PULL_REQUEST}; | ||
echo "using fetch head:"$HASH_TO_USE | ||
else | ||
HASH_TO_USE=${CI_COMMIT} | ||
sudo bash build_docker.sh -e ${HASH_TO_USE} -s -u -d $PWD/temp_staging/; | ||
echo "using travis commit:"$HASH_TO_USE | ||
fi | ||
echo "Docker build done ==========" | ||
|
||
cd $WORKING_DIR/gatk/scripts/ | ||
sed -r "s/__GATK_DOCKER__/broadinstitute\/gatk\:$HASH_TO_USE/g" vcf_site_level_filtering_cromwell_tests/vcf_site_level_filtering_travis.json >$WORKING_DIR/vcf_site_level_filtering_travis.json | ||
echo "JSON FILES (modified) =======" | ||
cat $WORKING_DIR/vcf_site_level_filtering_travis.json | ||
echo "==================" | ||
|
||
|
||
echo "Running Filtering WDL through cromwell" | ||
ln -fs $WORKING_DIR/gatk/scripts/vcf_site_level_filtering_wdl/JointVcfFiltering.wdl | ||
cd $WORKING_DIR/gatk/scripts/vcf_site_level_filtering_wdl/ | ||
java -jar $CROMWELL_JAR run JointVcfFiltering.wdl -i $WORKING_DIR/vcf_site_level_filtering_travis.json |
14 changes: 14 additions & 0 deletions
14
scripts/vcf_site_level_filtering_cromwell_tests/vcf_site_level_filtering_travis.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
{ | ||
"JointVcfFiltering.gatk_docker": "__GATK_DOCKER__", | ||
"JointVcfFiltering.vcf": ["/home/runner/work/gatk/gatk/src/test/resources/large/filteringJointVcf/test_10_samples.22.avg.vcf.gz", | ||
"/home/runner/work/gatk/gatk/src/test/resources/large/filteringJointVcf/test_10_samples.23.avg.vcf.gz"], | ||
"JointVcfFiltering.vcf_index": ["/home/runner/work/gatk/gatk/src/test/resources/large/filteringJointVcf/test_10_samples.22.avg.vcf.gz.tbi", | ||
"/home/runner/work/gatk/gatk/src/test/resources/large/filteringJointVcf/test_10_samples.23.avg.vcf.gz.tbi"], | ||
"JointVcfFiltering.sites_only_vcf": "/home/runner/work/gatk/gatk/src/test/resources/large/filteringJointVcf/test_10_samples.sites_only.vcf.gz", | ||
"JointVcfFiltering.sites_only_vcf_index": "/home/runner/work/gatk/gatk/src/test/resources/large/filteringJointVcf/test_10_samples.sites_only.vcf.gz.tbi", | ||
"JointVcfFiltering.basename": "test_10_samples", | ||
"JointVcfFiltering.snp_annotations": "-A ReadPosRankSum -A FS -A SOR -A QD -A AVERAGE_TREE_SCORE -A AVERAGE_ASSEMBLED_HAPS -A AVERAGE_FILTERED_HAPS", | ||
"JointVcfFiltering.indel_annotations": "-A MQRankSum -A ReadPosRankSum -A FS -A SOR -A QD -A AVERAGE_TREE_SCORE", | ||
"JointVcfFiltering.model_backend": "PYTHON_IFOREST", | ||
"JointVcfFiltering.use_allele_specific_annotations": false | ||
} |
Oops, something went wrong.