#7480)

…r re-scattered shards (#7493)

* add param to just print final query
* update print message

…tractCallset [VS-209] (#7495)

* export VAT to TSV

* call export vat task

* adding array values

* pass the normalized vcf

* clean up date spacing

* loop through vms rather than in the task

* remove partitioning because we are not going to do it for now

This reverts commit e6a9f77.

* add filtering on the FT tag level

* remove duplicates by 4 cols, not only 2

* make annotations machine bigger

* add duplicate count info

* dont rename things---use the names that already exist!

* drop all the excessive AS_QUALapprox info since it just adds weight

* swap gzip

* check with Lee about VAT - markings

* template update

* add in line counts for metrics

* get the count of high alt allele sites

* AC_hemi for all

* add new docker image

* get sample counts

* clean up files for memory preservation

* re-order errors

* pass dropped variants

* comments cleanup

* removed streaming

* removed streaming

* support for indexes as input

* move to same directory

* move to same directory

* move to same directory part two

* comments

* comments

* comments

* First implementation of working Reference Ranges-based Extract!

* added integration test

* PR cleanup

* parameter cleanup

* fix for upstream deletions

* comments from PR

* updated post rebase

* cherry pick

* Set gcloud config directory in travis (#7525)

* Set gcloud config directory in travis

* This fixes an issue while installing gcloud on travis due to permissions in the root directories

Co-authored-by: Louis Bergelson <[email protected]>

* fix bug copying files

* expose branch

* add reblocking wdl
* rename wdl
* make serivce account and site_id optional
* add requester pays option
* fix basename
* change all outputs to file

* fix variable assignment

* change outputs to external bucket path

* use pending BQ writer
* finalize stream before commit
* update create tables for ref ranges
* fix flushing and closing
* use python to update data model
* return error if response is not 200
* add bq table creation to assign ids, cleanup
* update import with ref_ranges as default and 40 as drop state. update docs
* fix tables for duplicate check
* create separate readme for AoU Gvs workspace
* handle null vet and pet creator
* add initial test for write api
* added sample_load_status table creation
* log load status and verify sample is not loaded
* used _default stream instead (#7588)
* safety checks for restarts via preemptibles (#7590)
* add is_loaded check back to GvsCreateFilterSet
* 3 retries and bigger machine for writeAPI ingest (#7592)

* read api bytes logging, upgrade bigquery client versions
* removed extra whitespace
* merge in changes from kc_gradle6 going to master

* pass through the drop state in the WDL

* dockstore for testing

* fixed bug introduced with new BQ libraries
* updated WDLs also
* now with tests

* clean up files for memory preservation

* update docker

* add gnomad failed filter col

* add clarifying comments and zip to retain space

* zip

* note that in the future we can remove N errors

* split up the filtering for less io strain

* what if I drop even more super-multi allelic sites?

* Add sample count to the schema
clean up annotations
comment out ones that will be used in the future

remove annotations I dont want

* lets try more mojo

* update the schema

* presorted avro file, fix performance issue
* updated WDL; JAR
* PR feedback

* WIP extract for ranges

* fixed signature in test

* updated WDLs

* updated name

* type

* bash typos

* stop converting long -> string -> long

* WDL bump

* update jars

* dockstore for testing

* make a sample name table to join to

* make it temp

* replace the second list with the join

* delete temp table after done

* throw an error if the table exists already

* remove from dockstore

* First Version

* wire into extract WDL

* type

* String -> File type

* Line continuation

* memory bump

* parameterize splitIntervals memory

* streamline implementation

* more memory friendly

* Addressing PR comments

Co-authored-by: Rebecca Asch <[email protected]>

Updated to public path for weights BED

* fixed quoting
* use named parameters

* analysis of shard runtimes

* WIP

* Update get_workflow_metadata.sh

* Update summarize_intervals.sh

… [VS-283] (#7654)

* gsutil cp them up

* creation of new files and splitting of code

* store in between states

* add to dockstore!

* validate wdl

* add service account before gsutil

* wait until subpipeline is over!

* make it cheaper!

* remove cruft from dockstore

* remove in-case-of-fire step since subworkflow works much better

* add readme and examples

* stylize readme for readability

* add examples to dockstore

* remove gatk override param

* clean up params

* tsv note

* add prepare step running instructions

* table prefix example

* keeping this backwards compatible---why tho?

* wire through datasetid---do we still need fq names?

* add dataset id

* pull thru dataset id

* update wdl too

* add to dockstore for testing

* unneeded edit

* dont need this method

* update unit tests

* add dataset id

* clean up vars

* wdl typo

* update WDL

* new name dummy!

* update jar

…7715)

* notes in ingest

* default drop state

* remove cost info

* gaussian notes

* add query for missing samples

Co-authored-by: Rori Cremer <[email protected]>

…S dataset [VS-335] (#7734)

* used constants; implemented non-AS transformation

* per Laura, using zero for ref MQ in fallback case

* added support to force non-as loading path

* Calculate SUM(AD) and make available

* fixed tests, PR comments

* bump JAR

* make sure dataset id is passed

* update dockstore

* allow bucket access for interval list

* add to dockstore

* allow for gatk to be overridden, update with known good jar

* dockstore

Co-authored-by: Rebecca Asch <[email protected]>

Created the GvsWithdrawSamples wdl.

* presorted avro files, fix performance issue

* PR feedback

* remove unnecessary annotations, change seen samples data structure

* performance improvements (see spreadsheet)

* fixed casting bug

* new jar; remove threading

* removed annotation optimization

* remove unused imports

* simplify

* refactoring

* PR comments

* typo

* PR feedback

* add call_AD to prepare py

* add callAD to the extract record

* add callAD to engine

* update schema for PLs and AD

* add callAD to extract

* still need to update the Avro file

* update dockstore

* add updated jar

* add emit AD to wdl

* pass wdl vars

* add docker image

* add AD to rescattering workflow in dockstore

* update naming

* update wdl with newest jar

* remove R scripts from filtering

* add to dockstore for testing

* temp_tables dataset doesnt exist anymore

* add to dockstore to test

* note usage

…7773)

* remove unwanted header

* missed the extra NAY

* update dockstore

* use new headers jar

* remove header lines from test too

* correct auth in split intervals

* I dont like it, but at least we know it works

Also (optionally) change the naming of the intervals file to match that of the VCFs

…ment. (#7802)

* Pinned typing_extensions python package to 4.1.1 to fix conda environment.

* Added documentation comment to conda environment.

* Switched over to using GithubActions for CI services and re-implemented our existing tests as github an actions workflow. Removed old Travis infrastructure and scripts.

Co-authored-by: Louis Bergelson <[email protected]>, James Emery <[email protected]>

* Docker build now only downloads the main large resources from git-lfs.  Previously it pulled everything.
* This should make building the docker substantially faster locally and possibly speed up the test suite slightly.

…the extract (#7788)

* remove old jar

* these should pass the override

* this fixes a typo that caused one of our docker steps to be broken.  It was masked by the build process which retries the same step if it needs to.

* fix typos

* create new WDL for second half of workflow only

* clean up deleted file

* catch any unexpected clinvar classification values

* clean up optional files while here

* add test and clean up vat validation

* update dockstore

* better VAT creation gating

* update docker along with python scripts

* update default VAT pipeline docker too

Add a wdl script to calculate the precision and sensitivity for joint calling sets.

* update docker for last half of pipeline wdl

* manual python test--can i add to docker

* add json files for expected and test data

* update test files

* update python test

* add vat expected for testing

* make sure docker does not build if the test fails

* add data for another test

* short term add test that needs to be refactored for clinvar correction

* spacing

* rename python test file

* fix up on rename

* update docker

* Update scripts/variantstore/wdl/extract/test_create_variant_annotation_table.py

Co-authored-by: Miguel Covarrubias <[email protected]>

* swap to nulls

* use with

* run tests inside docker container

* spacing

Co-authored-by: Miguel Covarrubias <[email protected]>

* add gate to keep clinvar vals approp limited

* update expected to not have greedy clinvar values

* more specific clinvar test

* move files for additional tests

* add additional clinvar specific tests to make sure behavior is not greedy

* Update scripts/variantstore/wdl/extract/create_variant_annotation_table.py

Co-authored-by: Miguel Covarrubias <[email protected]>

* pr review updates

* spacing

Co-authored-by: Miguel Covarrubias <[email protected]>