-
Notifications
You must be signed in to change notification settings - Fork 587
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
serial inserts for scaling prepare, factored out sample name #7288
Conversation
`{fq_sample_mapping_table}` s ON (new_pet.sample_id = s.sample_id)) | ||
""" | ||
( | ||
location INT64, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fix formatting
@@ -297,7 +327,7 @@ def make_extract_table(fq_pet_vet_dataset, | |||
#Default QueryJobConfig will be merged into job configs passed in | |||
#but if a specific default config is being updated (eg labels), new config must be added | |||
#to the client._default_query_job_config that already exists | |||
default_config = QueryJobConfig(labels=query_labels_map, priority="INTERACTIVE", use_query_cache=False) | |||
default_config = QueryJobConfig(labels=query_labels_map, priority="INTERACTIVE", use_query_cache=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know why this was false before, probably for testing/benchmarking
|
||
cohort_extract_final_query_job.result() | ||
JOB_IDS.add((f"insert final cohort table {fq_destination_table_data}", cohort_extract_final_query_job.job_id)) | ||
sql = f""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it more or less error prone in python to reuse the same variable name? sql
should we instead use create_table_sql
, etc?
Two primary sets of changes
Testing
Tested on the GVS tieout set. As expected the only difference in the cohort extract tables is that we are no longer seeing mis-joined VET information at
*
sites (which is a nice side benefits). Otherwise tables tie out exactly in SQL.In addition, I ran a full GIAB tieout before and after and the results are identical