Populate cost metadata for GATK tasks. #7919

gbggrant · 2022-06-27T19:50:33Z

Populates cost_metadata table for two gatk tasks

codecov · 2022-06-27T20:17:44Z

Codecov Report

❗ No coverage uploaded for pull request base (ah_var_store@4b2cf4b). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head cb38cde differs from pull request most recent head 4710dfe. Consider uploading reports for the commit 4710dfe to get more accurate results

@@               Coverage Diff                @@
##             ah_var_store     #7919   +/-   ##
================================================
  Coverage                ?   16.934%           
  Complexity              ?      4702           
================================================
  Files                   ?      1375           
  Lines                   ?     82064           
  Branches                ?     13014           
================================================
  Hits                    ?     13897           
  Misses                  ?     66106           
  Partials                ?      2061

…tCallset.wdl

…k_cost_to_metadata

gbggrant · 2022-06-29T16:28:24Z

Passing integration test here: https://job-manager.dsde-prod.broadinstitute.org/jobs/2b9f4dc6-d058-4803-a583-5ab76fbd71e8

RoriCremer · 2022-06-29T19:08:00Z

scripts/variantstore/wdl/GvsCreateFilterSet.wdl

+      --call-set-identifier ~{call_set_identifier} \
+      --wdl-step GvsCreateFilterSet \
+      --wdl-call ExtractFilterTask \
+      --shard-identifier ~{intervals_name}
  >>>


fully out of the scope of this pr, but we have so many params! I wonder if we could cut down on a lot of the fq ones--like would we ever not know what the alt-allele-table was called if we have the proj and dataset?

RoriCremer · 2022-06-29T19:09:21Z

scripts/variantstore/wdl/GvsQuickstartIntegration.wdl

@@ -61,6 +61,7 @@ workflow GvsQuickstartIntegration {

    call Unified.GvsUnified {
        input:
+            call_set_identifier = branch_name,


RoriCremer · 2022-06-29T19:43:07Z

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/CostObservability.java

+        builder.addFields(
+                TableFieldSchema.newBuilder().setName("call_set_identifier").setType(TableFieldSchema.Type.STRING).setMode(TableFieldSchema.Mode.REQUIRED).build()
+        );
+        builder.addFields(


does the newBuilder need to be instantiated for each col? Or can you make the builder with all the cols?

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/CostObservability.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/extract/ExtractCohort.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/filtering/ExtractFeatures.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/filtering/ExtractFeaturesEngine.java

src/main/java/org/broadinstitute/hellbender/tools/gvs/extract/ExtractCohort.java

rsasch · 2022-06-29T23:51:10Z

Is there a way you can use/update the GvsQuickstartIntegration.AssertCostIsTrackedAndExpected task to check for these additional tracked costs?

…k_cost_to_metadata

And track cost for SampleList here too.

…eatability. Update truth path.

…k_cost_to_metadata

rsasch

LGTM 👍🏻

…k_cost_to_metadata

Working on ExtractFeatures now.

1896bf3

gbggrant added 7 commits June 27, 2022 16:40

Run it from the wdls

55398f5

Update to record costs from ExtractCohort.java as called by GvsExtrac…

1980d69

…tCallset.wdl

Oops

3391b13

See if we can test something

114ed37

Merge remote-tracking branch 'origin/ah_var_store' into gg_VS-479_gat…

a590c45

…k_cost_to_metadata

Undo test change

5148fe9

Updated override jars

78d6694

gbggrant marked this pull request as ready for review June 29, 2022 16:27

gbggrant requested review from mcovarr, rsasch and RoriCremer June 29, 2022 16:27

RoriCremer reviewed Jun 29, 2022

View reviewed changes

RoriCremer approved these changes Jun 29, 2022

View reviewed changes

RoriCremer reviewed Jun 29, 2022

View reviewed changes

src/main/java/org/broadinstitute/hellbender/tools/gvs/common/CostObservability.java Outdated Show resolved Hide resolved

mcovarr approved these changes Jun 29, 2022

View reviewed changes

RoriCremer reviewed Jun 29, 2022

View reviewed changes

src/main/java/org/broadinstitute/hellbender/tools/gvs/extract/ExtractCohort.java Outdated Show resolved Hide resolved

gbggrant added 8 commits June 30, 2022 14:01

Merge remote-tracking branch 'origin/ah_var_store' into gg_VS-479_gat…

4a5b655

…k_cost_to_metadata

Response to PR feedback.

6cfc4c3

Not sure about this.

2055635

Revert that last change

f0b438a

And track cost for SampleList here too.

Update the override jars

3dc0b2d

Fix cost observability query to include more rows and to sort for rep…

82d600e

…eatability. Update truth path.

Merge remote-tracking branch 'origin/ah_var_store' into gg_VS-479_gat…

a20e431

…k_cost_to_metadata

Remove a branch from .dockstore.yml

6a21731

rsasch approved these changes Jul 6, 2022

View reviewed changes

gbggrant added 16 commits July 6, 2022 16:15

Temporarily hack the GvsQuickstartIntegration.wdl to run a single task.

92a64c0

Stupid formatting issues.

5ef5695

The stupidity was me.

d0f991c

again

35fec5f

Let's get the basic query right huh?

98cbad8

Testing some approaches

8fc561e

Testing some approaches

583aea4

Testing some approaches

1ebb061

bash works on prem.

b20c583

bash works on prem.

a46907b

Test that it fails properly.

d296ce9

Reverted changes made for simple testing.

988f219

Merge remote-tracking branch 'origin/ah_var_store' into gg_VS-479_gat…

5d8fe37

…k_cost_to_metadata

Add another tolerance range.

e139199

Added a comment

cb38cde

Last cleanups

4710dfe

gbggrant merged commit faf3347 into ah_var_store Jul 8, 2022

gbggrant deleted the gg_VS-479_gatk_cost_to_metadata branch July 8, 2022 02:35

This was referenced Mar 17, 2023

lb merge gvs branch #8248

Closed

testing something, please ignore #8251

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Populate cost metadata for GATK tasks. #7919

Populate cost metadata for GATK tasks. #7919

gbggrant commented Jun 27, 2022 •

edited

Loading

codecov bot commented Jun 27, 2022 •

edited

Loading

gbggrant commented Jun 29, 2022

RoriCremer Jun 29, 2022

RoriCremer Jun 29, 2022

RoriCremer Jun 29, 2022

rsasch commented Jun 29, 2022 •

edited

Loading

rsasch left a comment

Populate cost metadata for GATK tasks. #7919

Populate cost metadata for GATK tasks. #7919

Conversation

gbggrant commented Jun 27, 2022 • edited Loading

codecov bot commented Jun 27, 2022 • edited Loading

Codecov Report

gbggrant commented Jun 29, 2022

RoriCremer Jun 29, 2022

Choose a reason for hiding this comment

RoriCremer Jun 29, 2022

Choose a reason for hiding this comment

RoriCremer Jun 29, 2022

Choose a reason for hiding this comment

rsasch commented Jun 29, 2022 • edited Loading

rsasch left a comment

Choose a reason for hiding this comment

gbggrant commented Jun 27, 2022 •

edited

Loading

codecov bot commented Jun 27, 2022 •

edited

Loading

rsasch commented Jun 29, 2022 •

edited

Loading