* deleted VDS

* only one left

…tion of Delta (#8205)

* Lees name

* add vds validation script written by Tim

* fix rd tim typo

* make sure temp dir is set and not default for validate()

* swap to consistent kebab case

Co-authored-by: Miguel Covarrubias <[email protected]>

* clean up validation

* put init in the right place

* add proper example to notes

* update code formatting

---------

Co-authored-by: Miguel Covarrubias <[email protected]>

* Lees name

* add vds validation script written by Tim

* fix rd tim typo

* make sure temp dir is set and not default for validate()

* swap to consistent kebab case

Co-authored-by: Miguel Covarrubias <[email protected]>

* clean up validation

* put init in the right place

* add proper example to notes

* update code formatting

* update review

---------

Co-authored-by: Miguel Covarrubias <[email protected]>

* Don't run gatk tests when the only changes in a commit are in the scripts/variantstore directory.

* laying framework for FOFN bulk import code

* adding in terra notebook utils code

* updating wdl

* updating environment variables to make this work better

* quotey McBetterQuotes

* extra environment variables

* normalizing variable name with other wdls that require it

* gotta explicitly set WORKSPACE_NAMESPACE to the google project as well.  Apparently.

* typoooooooooooooooooo

* Didn't pipe the output files the entire way up

* whoopsie

* typo

* two updates after testing:
1. We do NOT want to assume that the sample ids we want are in the name field.  Pass that through as a parameter.
2. We want to explicitly pause every 500 samples, as that's our page size.  It slows our requests down enough to not spam the backend server and hit 503 errors, although it does slow down the rate at which we can write the files if the dataset is too big.  Which shouldn't be a concern, because as long as it doesn't cause errors it is still a hands off process.
3. We want to account to heterogenous data.  In AoU Delta, for instance, the control samples keep their vcf and vcf_index data in a different field.  This would cause the whole thing to fail if we weren't accounting for that explicitly, and now we generate an errors.txt file that will hold the row that we couldn't find the correct columns for so they can be examined later

* silly mistake copying the functioning code over from the workbook

* making script more robust against specifying imaginary columns in the data table and being slightly more informative in the output of the python script

* increasing the size of the disk this is running on for the sake of efficiency (and handling larger callsets)

* Passing errors up

* update params

* short term testing (rate lim)

* make it only 25 shards!

* add workspace id scraping

* add workspace id scraping fixup

* this is not functioning--need to curl in the wdl

* clean up vcfs so we dont run out of space

* add duplicates test to the shard loading

* clean up namespace prep
---------

Co-authored-by: Aaron Hatcher <[email protected]>

* Use the annotation 'AS_MQ' for indels.

…0] (#8274)

… table (#8278)

* Remove the unneeded SCORE field from the filter_set_info_vqsr table
* Updated the docker images.

* add queries for testing mismatched sites and variants across possible duplicates

* still need to wire these through

* plumb thru dup validation

* dockstore for testing

* update docker

* add xtrace

* better bool logic

* clean up bash

* okay  lets try ripping shit out to get this to work

* okay lets put a few lines back

* ok that worked, lets swap for better errors

* short term remove clinvar

* review changes

* update docker

* explain removal of clinvar test

* Adding tests for ExtractCohortLite.

* Simple fix to have the header of the VAT tsv to use tab characters.

* Updated to latest version of VQSR Lite (from Master)
* Ported tests and files for VQSR Lite over
* Refactored VQSR Classic code into its own WDL

)

* Add support for VQSR Lite to GvsExtractCohortFromSampleNames.wdl
* Remove obsolete gatk override jar

* add python script to our repo

* use the new python script!

* remove whl from integration test

* move script location for testing

* remove the damn wheel!

* add the replacement hail script

* proper renaming

* update docker

* Refactoring of ExtractCohortLite into ExtractCohort.

…ue. (#8312)

* Update override jar to fix support issue.

* Fix bug in VCF Integration test

* add Aarons changes

* put terra token in python

* id not bucket

* hardcode for testing

* do we need a new docker image?

* set workspace info

* pull in name from rawls

* pass output locations

* add back prepare

* add GvsImportGenomes back

* update python for grabbing cols

* split methods for easier testing

* set defaults, but allow optional overrides for sample table and id

* add unit test for python column guessing

* clean up python for testing

* add proper docker

* is this where the loop is coming from?

* better names

* remove testing artifact

* add back problem lines to the test

* throw out columns with values other than strings

* set defaults in the right place

* Update GvsExtractAvroFilesForHail.wdl to support VQSR Lite.
* Update gvs_avros_to_vds.py script to handle VQSR Lite differences.

…and Classic (#8336)

* Towards a better world where VQSR Lite and VQSR Classic coexist happily in filter_set_info table.
* Implemented auto detection of VQSR Lite vs Classic in GvsExtractCallset.wdl
* Implemented in GvsCalculatePrecisionAndSensitivity.wdl
* Implemented in GvsExtractAvroFilesForHail.wdl
* Fixed a minor but (in a failing integration test) that the VCFs we Extract when not filtering, contain a filter-specific line.
* Update dockers to fix minor bug noted during integration testing.
* Update truth path for integration test
* Make VQSR Classic the default in hail_gvs_import.py

* add tims script wholecloth

* add Tims script to integration test

* add Tims script to vds creation wrapper

* update docker image

* update dockstore

* Fix error in parsing monitoring log without a CPU
* Migrate VQSR Lite updates from gvs_avros_to_vds.py to import_gvs.py.
* Delete deprecated file.
* Update dockers

* generalizing the isGenotypeAllNonRef call to function when there aren't exactly 2 alleles

* First pass fixing ingest code for multiple ploidies.  Tested and succeeded against modified quickstart example

* modifying the alt_allele population queries to also put '1' and '2' into the "position1" group. This allows it to work for haploid entries, but NOT ploidies larger than 2.  We have to generalize to that later, if we need to support it

* updating alt allele docker image and dockstore stuff

* updating feature extraction for filter creation to work with "1" and "2"

* generalizing no-call detection during extraction

* Fixing how we write out the no call records so it doesn't assume it should write diploid data

* Bringing back in my changes to the populate_alt_allele.py script post-merge (by updating the docker image)

* code review feedback

* updating the image for the python scripts

* updating gatk docker image elsewhere

* testing branch-specific version of integraiton test

* Update Integration Test to support both VQSR Classic and VQSR Lite.
* Update path to latest truth.
* Increase tolerance on cost check.
* Add sort by alleles to ExtractFeaturesEngine to make inputs to VQSR consistent.
* Allow for different cost and table size truth files for VCF and Hail integration.
* Update GvsExtractAvroFilesForHail.wdl to support VQSR Lite.
* Update dockers
* Different truth directories for lite vs classic.

* wiring through the option for VQSR-Lite and also whether they have tigher gcp quotas (while I'm there)

* typos suck

Added additional information about cost and runtime for larger sets of samples. Copied from beta workspace.

…-941] (#8365)

* Update to latest docker.
(Couldn't find ExtractCohortToVcf in ExtractTask method in GvsExtractCallset.wdl)
Used the official docker tag

- Change coordinates for xgboost artifact as the JCenter version has gone missing.
- "Borrow" Chris Norman's work for freeing up space on the GHA runner.

* add sample sets as a pass thru

* where do we want to get sample sets

* add sample sets to wdl

* overzealous

* add sample sets to FOFN creation

* add to dockstore for testing

* wdl anger

* clean up python script defaults

* clean up wdl

* add latest python script

* okay what if its an older python

* add sample set set

* why cant i gsutil cp?

* okay why is the line issue not working

* make set

* better naming for set

* clean up todo

* clean up todo comments

* add list of remaining entity defining work

* add entity type method

* pass entitu info

* better guesses

* add entity set here

* okay these are not good, they should get gutted and improved

* better instructions to be mobbed on

* add metadata for a few params of note

* short term pass in entity types

* add validation for file length

* clean up unused var

* un-optional non-optionals

* better instructions

* merge dupe tasks together

* clean up fofn creation

* add entity type validation

* update docker image

* can i wdl?

* remove dead comments

* more wdl

* make sample_id optional

* dont pass optional param

* cmon bash

* cmon bash 2

* cmon bash hmmm test

* remove all the text files to see if thats whats breaking it

* add wc -l back

* pr review on documentation

* clean up testing

* this needs better error msgs

* remove silly debugging

* more cleanup

* moar documentation

* add better error msgs

* clean up whitespace

* clean up documentation

* write out the whole word!

* trailing new line!

* clean up FOFN creation

* entity set should be optional

* make sample sets optional

* update readme with pr review

* add more explicit failure mode to the python script when no vcf col is found

* rename  bq_project_id

* additional comments on the python code

* update error options

* update tests for python

* move notes to the top of the python file

* fix bug in python sets

* update tests

* only call tnu for the table list once

* add valueerror test

* new docker image

)

* Fix a bug where there were NO variants in a range.

* Migrated PopulateFilterSetTranches code from GvsCreateFilterSet.wdl to GvsVQSRClassic.wdl
* Migrated PopulateFilterSetInfo code
* Moved Warp tasks into GvsVQSRClassic.wdl

Co-authored-by: Jonn Smith <[email protected]>

Move BigQuery classes [VS-971]

* Update the dockers to handle new Google Storage Libray change
* Update jar in GvsJointVariantCalling.wdl

* Add developer's Exome documentation.
* Update documentation to reference (and explain) the interval list we are really using.

…t. (#8407)

* Remove some unused VCF header fields from ExtractFeatures
* Renaming VQS_SENS and related constants back to 'CALIBRATION_SENSITIVITY' (as it is named by VQSR Lite) to reduce confusion.
* Update dockers and override jar
* Updated truth path for integration tests

* Add the VQSR Lite field 'SCORE' to the data model
- Pull it from the VCFs exported from VQSR Lite and store it in the model
* Add code to put SCORE in the extracted VCF
* Pass the gatk_override to VQSRClassic wdl (oops)
* Update truth for integration test to include VQSR Lite VCFs with 'SCORE' field in them
* Update VQSR Lite avro file and test data to include SCORE field.
* Updated dockers and override jar

* pin hail version

* update dockstore for testing

* update correct file

* Updating beta docs to reflect bulk ingest

* Removing mentions of sample sets, making the bucket hash fake

* Apply suggestions from Miguel code review

Co-authored-by: Miguel Covarrubias <[email protected]>

Co-authored-by: Bec Asch <[email protected]>

---------

Co-authored-by: Kylee Degatano <[email protected]>
Co-authored-by: Miguel Covarrubias <[email protected]>
Co-authored-by: Kylee Degatano <[email protected]>
Co-authored-by: Bec Asch <[email protected]>

…2] (#8404)

…ows #8426 (#8429)

* VS-991. Wire interval list from Joint Calling through to other workflows
- Defaulting to WGS.
- Note that I had to thread the weighted bed file through (and a flag to use it). As this will also be needed to run exomes.

…ta sets. (#8446)

* Increasing max auto batch size to 25K to allow for larger beta sets.

* Update Override Jar in this here branch.

* Added Exome Integration Test on bulk_ingest_staging branch
* Set up to pull exome test data from data table.
* Pin the hail version to 0.2.119

* Allow VQSR Classic Memory overrides to be passed from GvsJointVariantCalling.wdl to GvsCreateFilterSet.wdl.
Increase memory overhead on a couple of tasks in GvsVQSRClassic.wdl

* A rough memory step function for SNPsVariantRecalibratorCreateModel

* Add branch name to the integration tests.

…7] (#8454)

* first draft of updated docs

* update list of tools used for filtering

* updating annotations list and adding pdf

* Updating diagram to remove reference to VQSR

* Realized there were some GATK docs I could reference for each tool

* updating workspace description as well

* adding blurb about change to vets

* update dockstore

* bump up mem size for extract

* add a lil step for disk size

* use gatk docker

* Miguels update to gate

Co-authored-by: Miguel Covarrubias <[email protected]>

---------

Co-authored-by: Miguel Covarrubias <[email protected]>