VS-775 vat validation shards

[RoriCremer]

Add additional validation around duplicated rows in the VAT

This has a successful run (except for one failure that is because it's being run on way less data)
https://job-manager.dsde-prod.broadinstitute.org/jobs/07ddde58-ac0d-4229-9f96-d093f5c11682
The failed test is:
SpotCheckForAAChangeAndExonNumberConsistency

Perhaps we want to update this to not run this test if there are less than 10k samples?
Yes we do:
Here's the ticket for that:
https://broadworkbench.atlassian.net/browse/VS-878

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (ah_var_store@8217073). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files

@@               Coverage Diff                @@
##             ah_var_store     #8175   +/-   ##
================================================
  Coverage                ?   85.880%           
  Complexity              ?     35515           
================================================
  Files                   ?      2194           
  Lines                   ?    167029           
  Branches                ?     18006           
================================================
  Hits                    ?    143444           
  Misses                  ?     17204           
  Partials                ?      6381           

[gbggrant]

The task 'DuplicateAnnotations' is failing for me (when run on my QuickStart) with:
/cromwell_root/script: line 76: syntax error near unexpected token )' /cromwell_root/script: line 76: ) > "$out94c3fb6c" 2> "$err94c3fb6c"'

Maybe you're not escaping the table string?

[mcovarr]

@gbggrant do you have a link to your failed run of DuplicateAnnotations? The query escapes look okay to me and nothing else jumps out as being wrong but something clearly is...

[gbggrant]

@mcovarr here's the failing workflow: https://app.terra.bio/#workspaces/gvs-dev/GVS%20Quickstart%20v3%20ggrant/job_history/399f4997-64a7-4e5a-9521-0dca132a8822

[mcovarr]

It should be possible to simplify this and the query below with a HAVING (i.e. without a nested query) as was done previously.

Suggested change

	bq query --nouse_legacy_sql --project_id=~{query_project_id} --format=csv 'SELECT * from
	(SELECT contig, position, gvs_all_an, COUNT(DISTINCT gvs_all_an) AS an_count FROM `~{fq_vat_table}`
	group by contig, position, gvs_all_an)
	where an_count >1' > bq_an_output.csv
	bq query --nouse_legacy_sql --project_id=~{query_project_id} --format=csv '
	SELECT contig, position, gvs_all_an, COUNT(DISTINCT gvs_all_an) AS an_count
	FROM `~{fq_vat_table}`
	GROUP BY contig, position, gvs_all_an
	HAVING an_count > 1
	' > bq_an_output.csv

[RoriCremer]

I'm trying to make sure that for multiple rows with the same VID, there aren't multiple values of AN (and below the AC!)

[mcovarr]

Suggested change

	bq query --nouse_legacy_sql --project_id=~{query_project_id} --format=csv 'SELECT * from
	(SELECT contig, position, vid, gvs_all_ac, COUNT(DISTINCT gvs_all_ac) AS ac_count FROM `~{fq_vat_table}`
	group by contig, position, vid, gvs_all_ac)
	where ac_count >1' > bq_ac_output.csv
	bq query --nouse_legacy_sql --project_id=~{query_project_id} --format=csv '
	SELECT contig, position, gvs_all_ac, COUNT(DISTINCT gvs_all_ac) AS ac_count
	FROM `~{fq_vat_table}`
	GROUP BY contig, position, gvs_all_ac
	HAVING ac_count > 1
	' > bq_ac_output.csv

[gbggrant]

Did you mean to replace ClinvarSignificance with DuplicateAnnotations? If so, you should get rid of the task itself.

[RoriCremer]

not yet---I want to check with Lee about how to handle this:

https://broadworkbench.atlassian.net/browse/VS-635

[gbggrant]

Please leave a comment explaining that ClinvarSignificance has been removed temporarily