Ingest Error Handling Fixes [VS-261]

[mcovarr]

The main issue was that the StatusRuntimeExceptions that the baseline error handling code was trying to catch in practice always seem to be wrapped in at least one layer of exception of a different type. There was no catch handing for these wrapper exception types so the CreateVariantIngestFiles tool would simply crash.

The changes here also more generally try to follow the recommendations in the BQ Write API documentation, in particular closeing the JsonStreamWriter before retrying error codes not explicitly called out by the documentation.

EDIT: actually closing the writer didn't work out too well as we use the writer in PENDING mode and closing it seems to lose all pending writes. 😬 So in this circumstance we just throw and let WDL-level maxRetries start the data loading over from the beginning.

An exponential backoff was also added before retry attempts.

Parallel logic was also added to load status writing which should reduce (but not eliminate) the possibility of inconsistent sample status writes that require manual intervention. There is still the possibility of an inopportunely timed preemption, which is why VS-262 exists.

All of the WDL changes here are in support of a 2000-sample tieout, a large enough set that intermittent BigQuery errors are almost always observed. The tieout confirms that errors of the two major classes are seen (retryable and non-retryable) and that the number of rows per sample in the tieout dataset matches those in a reference dataset.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (ah_var_store@17a5e5e). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             ah_var_store     #7841   +/-   ##
================================================
  Coverage                ?   86.293%           
  Complexity              ?     35189           
================================================
  Files                   ?      2170           
  Lines                   ?    164876           
  Branches                ?     17784           
================================================
  Hits                    ?    142277           
  Misses                  ?     16276           
  Partials                ?      6323           

[mcovarr]

This is almost completely a cut/paste of a task from the quickstart integration test to GvsUtils.wdl so it could be reused.

[kcibul]

Nit: this method doesn't actually handle anything, maybe rename to extract instead of handle or something… but only if you're making another commit to this PR anyways