Track BigQuery costs of GVS python VS-480 #7915
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## ah_var_store #7915 +/- ##
================================================
Coverage ? 86.288%
Complexity ? 35194
================================================
Files ? 2170
Lines ? 164888
Branches ? 17785
================================================
Hits ? 142278
Misses ? 16288
Partials ? 6322 |
mcovarr
reviewed
Jun 23, 2022
| @@ -60,7 +61,7 @@ workflow GvsUnified { | |||
|
|
|||
| File interval_weights_bed = "gs://broad-public-datasets/gvs/weights/gvs_vet_weights_1kb.bed" | |||
|
|
|||
| String extract_output_file_base_name = filter_set_name | |||
| String extract_output_file_base_name = sub(filter_set_name, " ", "-") | |||
There was a problem hiding this comment.
are there any other characters we should be concerned about?
There was a problem hiding this comment.
at one point we were worried about underscores, but I think we ruled that out?
|
|
||
| # populate cost_observability data | ||
| sql = f"""INSERT INTO `{fq_dataset}.cost_observability` | ||
| (call_set_identifier, step, call, shard_identifier, event_key, call_start_timestamp, event_timestamp, event_bytes) |
There was a problem hiding this comment.
just realizing now that this is writing bytes and the datatype for event_bytes is INTEGER... happily BQ INTEGERS are unusually large at +/- 8 EiBs; if we overflow that we have bigger problems
mcovarr
approved these changes
Jun 23, 2022
gbggrant
approved these changes
Jun 27, 2022
mcovarr
reviewed
Jun 27, 2022
| # populate cost_observability data | ||
| sql = f"""INSERT INTO `{fq_dataset}.cost_observability` | ||
| (call_set_identifier, step, call, shard_identifier, event_key, call_start_timestamp, event_timestamp, event_bytes) | ||
| VALUES('{call_set_identifier}', '{step}', '{call}', '{shard_identifier}', 'BigQuery Query Billed', |
There was a problem hiding this comment.
is this supposed to be 'BigQuery Query Scanned' like in George's PR?
RoriCremer
reviewed
Jun 28, 2022
This was referenced Mar 17, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integration test run: https://app.terra.bio/#workspaces/gvs-dev/GVS%20Integration/job_history/30a2d8ee-13dd-4829-b3a8-4e6a67409705
Closes https://broadworkbench.atlassian.net/browse/VS-480