Adding the subpopulation calculations to the VAT creation WDL #7399

RoriCremer · 2021-08-10T03:41:08Z

This pr adds the subpopulation AC/AN/AF calculations.
It does this by taking in the ancestry table and making sublists of each---then passing that list of samples into the SelectVariants GATK tool

Updated Lucid chart here: https://lucid.app/lucidchart/fee376a4-4b72-481e-a239-a027f7f6ab1f/edit?page=CsG3hy3S1zEH#

Design Doc for this work:
https://docs.google.com/document/d/1FnPu_Jkz2O9rElApAQld0v6iBEFGe22dKarVWcwNxGI/edit

misc:
how should I add the VAT validation to the VAT pipeline? Should it run automatically?

Anvil data version of this table: spec-ops-aou:anvil_100_for_testing.vat_aug19

scripts/variantstore/wdl/GvsSitesOnlyVCF.wdl

not all samples

rsasch · 2021-08-20T16:44:26Z

scripts/variantstore/wdl/GvsSitesOnlyVCF.wdl

 }

 ################################################################################
+
+task MakeSubpopulationFiles { ## TODO why am I seemingly able to get the ancestry file in Terra? is it cached? I should need the SA for it---so actually, but me worth trying the SA use sooner


Is this still an issue?

Not an issue--but I would feel more comfortable with the data in an AoU bucket just for completeness I suppose? I'm going to do a follow on pr where I move the ancestry file into an AoU protected bucket and add a SA json to this step

ahaessly · 2021-08-23T16:38:19Z

scripts/variantstore/variant_annotations_table/GvsValidateVAT.wdl

+        if [[ $NUMRESULTS != "0" ]]; then
+          echo "PASS: The VAT table ~{fq_vat_table} has a correct calculation for subpopulation" > validation_results.txt
+        else
+          echo "FAIL: The VAT table ~{fq_vat_table} has an incorrect calculation for subpopulation" > validation_results.txt


maybe add the vids that are returned from the query to make it easier to debug the issue.

yes! I think they are already returned in the validation_results.txt

ahaessly · 2021-08-23T16:43:16Z

scripts/variantstore/variant_annotations_table/GvsValidateVAT.wdl

+        FROM
+            ~{fq_vat_table}
+        WHERE
+            gvs_all_ac != 20


Is this before we substitute 20 for values less than 20? If so then we don't really need this check. And if not then there could still be a subpop that is lower then 20 even if the all_ac isn't so I'm not sure this check is really doing the correct thing.

This will all get ripped out during de-obfuscation pr -- it was really just so that there wouldn't be validation run at all on the rows where the AC had been obfuscated to 20 (because the math wouldn't check out there)

ahaessly · 2021-08-23T16:43:50Z

scripts/variantstore/variant_annotations_table/GvsValidateVAT.wdl

+        if [[ $NUMRESULTS != "0" ]]; then
+            echo "PASS: The VAT table ~{fq_vat_table} has a correct calculation for AC and the AC of subpopulations" > validation_results.txt
+            else
+            echo "FAIL: The VAT table ~{fq_vat_table} has an incorrect calculation for AC and the AC of subpopulations" > validation_results.txt


again maybe log the vids for easier debugging

Yes---the vids are logged in the output file

ahaessly · 2021-08-23T16:46:17Z

scripts/variantstore/wdl/GvsSitesOnlyVCF.wdl

@@ -17,9 +17,37 @@ workflow GvsSitesOnlyVCF {
        String? service_account_json_path
        File? gatk_override
        File AnAcAf_annotations_template
+        File ancestry_file


The name of this wdl seems relatively generic, but sense an ancestry_file is required it seem sub pop specific. Maybe we should rename this and move to the vat dir?

The plan was to rename the wdl when it gets split into the sites only and the vat creation wdl

it might make sense to do that as a ticket we tackle sooner rather than later

ahaessly · 2021-08-23T16:54:00Z

scripts/variantstore/wdl/GvsSitesOnlyVCF.wdl

@@ -403,6 +585,31 @@ task BigQueryLoadJson {
              v.gvs_all_ac,
              v.gvs_all_an,
              v.gvs_all_af,
+              v.gvs_max_af,


I think you need this line at the start of this task:
echo "project_id = ~{project_id}" > ~/.bigqueryrc

do I not need to add the project_id to all of my queries then?

no you shouldn't have to.

that line just prevents an interactive error message that we sometimes see

ahaessly · 2021-08-23T16:54:42Z

scripts/variantstore/wdl/ToyLoopOptional.example.inputs.json

@@ -0,0 +1,3 @@
+{
+  "ToyLoopOptional.string": "HALP"
+}


do you mean to check this in and the ToyLoop wdl?

ahaessly · 2021-08-23T17:01:06Z

scripts/variantstore/wdl/extract/extract_subpop.py

+          print(seen_subpopulations)
+      else:
+        print("WARNING: This list had an unexpected subpopulation", row_subpopulation)
+      ## TODO do I need to close these?


I think you do need to close them.
If you use a with clause then you don't, but I don't think that will work within this looping structure

* add subpop list * add hard coded steps for one set of subpop calculations * add sample list * update the WDL to have the subworkflows * parse the subpop ac/an/af in python * update schemas with the new subpop calculations * fix initial state bug for max calc * add gvs subpops * remove saved annotations used for debugging * short term testing in Terra * add ancestry file parsing * add validation step for subpops * adding service account to grab ancestry file * turn off localization optional for auth issues * add service account to subpop calcs * cleaner linux parsing tabs not all samples * add ancestry file * save annotation file for debugging * update subpop python script * add validation for the subpop AC * debugging req SA json insanity * cleaner to use expected file type * this version works--need to unhardcode a bit * #&^%@($&^ WDL!!! * pr review edits * close python file handles * remove from dockstore

RoriCremer changed the base branch from master to ah_var_store August 10, 2021 03:41

RoriCremer added 6 commits August 11, 2021 21:24

add subpop list

e5dc7a9

add hard coded steps for one set of subpop calculations

0f2b534

add sample list

69e1d5b

update the WDL to have the subworkflows

294807a

parse the subpop ac/an/af in python

800bd15

update schemas with the new subpop calculations

f001b4f

RoriCremer marked this pull request as ready for review August 12, 2021 14:00

RoriCremer added 2 commits August 12, 2021 17:10

fix initial state bug for max calc

d91bce4

add gvs subpops

03b3d53

RoriCremer force-pushed the rc-vs-164-subpop branch from aa3be2b to 03b3d53 Compare August 12, 2021 21:11

rsasch reviewed Aug 13, 2021

View reviewed changes

scripts/variantstore/wdl/GvsSitesOnlyVCF.wdl Outdated Show resolved Hide resolved

RoriCremer added 4 commits August 13, 2021 14:57

remove saved annotations used for debugging

335720b

short term testing in Terra

d4e7c8d

typo in python script

b7b71b0

add ancestry file parsing

2bc0c0f

RoriCremer force-pushed the rc-vs-164-subpop branch from bad75e2 to 2bc0c0f Compare August 16, 2021 02:46

RoriCremer added 7 commits August 15, 2021 23:02

add validation step for subpops

68ea4e9

adding service account to grab ancestry file

d3d6253

turn off localization optional for auth issues

d2bc19e

add service account to subpop calcs

8c40908

cleaner linux parsing tabs

fcacc78

not all samples

add ancestry file

cc7d1e2

save annotation file for debugging

8ff7985

RoriCremer force-pushed the rc-vs-164-subpop branch from 59289ba to 8ff7985 Compare August 17, 2021 15:55

RoriCremer added 5 commits August 17, 2021 22:36

update subpop python script

b841bd8

add validation for the subpop AC

1df7b9a

debugging req SA json insanity

0c6ce9e

cleaner to use expected file type

3c83520

this version works--need to unhardcode a bit

7427e0d

rsasch reviewed Aug 20, 2021

View reviewed changes

ahaessly reviewed Aug 23, 2021

View reviewed changes

RoriCremer added 2 commits August 23, 2021 22:53

#&^%@($&^ WDL!!!

6958412

pr review edits

c13e16a

RoriCremer force-pushed the rc-vs-164-subpop branch from 307c4a8 to c13e16a Compare August 24, 2021 02:54

RoriCremer added 2 commits August 23, 2021 23:00

close python file handles

26e409d

remove from dockstore

1a72f09

ahaessly approved these changes Aug 24, 2021

View reviewed changes

RoriCremer merged commit 79f23a5 into ah_var_store Aug 24, 2021

RoriCremer deleted the rc-vs-164-subpop branch August 24, 2021 17:16

This was referenced Mar 17, 2023

lb merge gvs branch #8248

Closed

testing something, please ignore #8251

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the subpopulation calculations to the VAT creation WDL #7399

Adding the subpopulation calculations to the VAT creation WDL #7399

RoriCremer commented Aug 10, 2021 •

edited

Loading

rsasch Aug 20, 2021

RoriCremer Aug 20, 2021

ahaessly Aug 23, 2021

RoriCremer Aug 24, 2021

ahaessly Aug 23, 2021

RoriCremer Aug 23, 2021

ahaessly Aug 23, 2021

RoriCremer Aug 23, 2021

ahaessly Aug 23, 2021

RoriCremer Aug 23, 2021

ahaessly Aug 23, 2021

RoriCremer Aug 24, 2021

ahaessly Aug 24, 2021

ahaessly Aug 24, 2021

ahaessly Aug 23, 2021

RoriCremer Aug 23, 2021

ahaessly Aug 23, 2021

Adding the subpopulation calculations to the VAT creation WDL #7399

Adding the subpopulation calculations to the VAT creation WDL #7399

Conversation

RoriCremer commented Aug 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RoriCremer commented Aug 10, 2021 •

edited

Loading