Consolidate various docs for AoU callset generation into one to rule them all [VS-553] #7971
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## ah_var_store #7971 +/- ##
================================================
Coverage ? 86.247%
Complexity ? 35200
================================================
Files ? 2173
Lines ? 165016
Branches ? 17793
================================================
Hits ? 142321
Misses ? 16368
Partials ? 6327 |
gbggrant
approved these changes
Aug 2, 2022
| - To optimize the GVS internal queries, each sample must have a unique and consecutive integer ID assigned. Running the `GvsAssignIds` will create a unique GVS ID for each sample (`sample_id`) and update the BQ `sample_info` table (creating it if it doesn't exist). This workflow takes care of creating the BQ `vet_*`, `ref_ranges_*` and `cost_observability` tables needed for the sample IDs generated. | ||
| - Run at the `sample set` level ("Step 1" in workflow submission) with a sample set of all the new samples to be included in the callset (created by the "Fetch WGS metadata for samples from list" notebook mentioned above). | ||
| - You will want to set the `external_sample_names` input based on the column in the workspace Data table, e.g. "this.samples.research_id". | ||
| - If new controls are being added, they need to be done in a separate run, with the `samples_are_controls` input set to "true" (the referenced Data columns may also be different, e.g. "this.control_samples.control_sample_id" instead of "this.samples.research_id"). |
There was a problem hiding this comment.
Slightly confused as what this means - the external_sample_name may be pulled from the control_sample_id?
There was a problem hiding this comment.
The external_sample_name workflow input is an array of strings, usually grabbed from the Terra Data store in the workspace. The columns in that store can be called anything, and in the past, control samples and participant samples were in different tables with different column names.
mcovarr
reviewed
Aug 2, 2022
| - **TBD VDS Extract WDL/notebook/??** | ||
| - Run the "Fetch WGS metadata for samples from list" notebook after you have placed the file with the list of the new samples to ingest in a GCS location the notebook (running with your @pmi-ops account) will have access to. This will grab the samples from the workspace where they were reblocked and bring them into this callset workspace. | ||
| - Set the `sample_list_file_path` variable in that notebook to the path of the file | ||
| - Run the "now that the data have been copied, you can make sample sets if you wish" step if you want to automatically break up the new samples into smaller sample sets. Set the `SUBSET_SIZE` and `set_name` variables to customize. |
There was a problem hiding this comment.
Under what circumstances would we want to do this?
There was a problem hiding this comment.
If you don't want to throw all 100K (or whatever number) samples at GvsImportGenomes at once.
| - To optimize the GVS internal queries, each sample must have a unique and consecutive integer ID assigned. Running the `GvsAssignIds` will create a unique GVS ID for each sample (`sample_id`) and update the BQ `sample_info` table (creating it if it doesn't exist). This workflow takes care of creating the BQ `vet_*`, `ref_ranges_*` and `cost_observability` tables needed for the sample IDs generated. | ||
| - Run at the `sample set` level ("Step 1" in workflow submission) with a sample set of all the new samples to be included in the callset (created by the "Fetch WGS metadata for samples from list" notebook mentioned above). | ||
| - You will want to set the `external_sample_names` input based on the column in the workspace Data table, e.g. "this.samples.research_id". | ||
| - If new controls are being added, they need to be done in a separate run, with the `samples_are_controls` input set to "true" (the referenced Data columns may also be different, e.g. "this.control_samples.control_sample_id" instead of "this.samples.research_id"). |
There was a problem hiding this comment.
How will we know if new controls are being added?
There was a problem hiding this comment.
In the past, Lee/AoU has let us know that we should add additional (or different) controls to a callset.
| - Run at the `sample set` level ("Step 1" in workflow submission). You can either run this on a sample_set of all the samples and rely on the workflow logic to break it up into batches (or manually set the `load_data_batch_size` input) or run it on smaller sample_sets created by the "Fetch WGS metadata for samples from list" notebook mentioned above. | ||
| - You will want to set the `external_sample_names`, `input_vcfs` and `input_vcf_indexes` inputs based on the columns in the workspace Data table, e.g. "this.samples.research_id", "this.samples.reblocked_gvcf_v2" and "this.samples.reblocked_gvcf_index_v2". | ||
| 3. `GvsWithdrawSamples` workflow | ||
| - Run if there are any samples to withdraw from the last callset. |
There was a problem hiding this comment.
How do we know if there are samples to withdraw?
There was a problem hiding this comment.
We compare the sample list for the callset we are creating (which we get from Lee/Aou) with the samples already in the database.
mcovarr
approved these changes
Aug 3, 2022
This was referenced Mar 17, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.