execute_with_retry() error handling improvements [VS-159]

[rsasch]

Closes https://broadworkbench.atlassian.net/browse/VS-159

• move execute_with_retry to utils so that only defined in one place
• add more specific error handling

NOTE for reviewers: you probably want to check "Hide whitespace changes" under settings because IntelliJ did some "helpful" cleanup when I made some code changes to files.

[kcibul]

Overall the refactoring looks good and makes sense… but I'm not seeing how this fixes the problem of eating exceptions we saw during a recent run. Can you explain what was happening before, and how the new code addresses it?

[rsasch]

Overall the refactoring looks good and makes sense… but I'm not seeing how this fixes the problem of eating exceptions we saw during a recent run. Can you explain what was happening before, and how the new code addresses it?

Sure! This code (besides refactoring so that it was only in one place) aims to fix two issues:

1. if query results in an error, it gets run three more times and then, because of while len(retry_delay) > 0, it doesn’t run again and the raise err line never gets executed, so no error is ever raised
2. if the query fails for a reason that has no chance of being fixed by a retry (eg. 404), it will still run three more times

I probably missed some errors that should be "retry-able" (maybe Aborted ? BadGateway? Cancelled ? full list here), but I still think it makes sense to not treat all errors the same.

[kcibul]

lgtm!