WIP extract for ranges

[kcibul]

Some notes from the 10k tieout:

Prepare Step

• ~20 min per full ref_ranges table to insert
• ~7 min per full vet table to insert
• "bytes scanned" are same as data table size

Extract

Original Run - 293 min

• 103 minutes pulling down data, scanning 237 GB

  • 43 min on 20m vet records (20:26 - > 21:09)
  • 60 min on 291m vet records (21:09 -> 22:10)

• 190 minutes writing the VCF

Prepare Extract with minor tuning of sorting - 134 min

• 25 minutes pulling down data ( faster), scanning 10 GB (50x reduction)

  • 4 min on 20m vet records(02:43 -> 02:47) - NOTE 103s of that was sorting (44s) and spilling to disk (59 s)
  • 21 min on 291m vet records (02:47 -> 03:08) - NOTE 9 min of that was sorting (6 min) and spilling to disk (3 min)

• 109 minutes writing the VCF (this is the change to pre-sort the sample set merged to ah_var_store on 1/12/22)

Tieout is identical

kcibul@kc-specops-tiny:~/stroke_tieout$ md5sum gold.jointcallset_0.vcf.gz
496178eae4afe63c4391d8eba64a9947  gold.jointcallset_0.vcf.gz

kcibul@kc-specops-tiny:~/stroke_tieout$ md5sum trial.full.jointcallset_0.vcf.gz
496178eae4afe63c4391d8eba64a9947  trial.full.jointcallset_0.vcf.gz

[RoriCremer]

(for my edu only) Is it better to not convert to a string in the first place?

[kcibul]

it's expensive… to convert a number to a string and then parse the string to get back another number. the result from get is already a Long we just have to cast it as such. BigQuery doesn't return int, but we know it is an int and want it as such so we call intValue() on it.

[RoriCremer]

nit: spacing looks odd here

[rsasch]

LGTM 👍🏻