Retry remote contract calls for unsupported block number errors

[yondonfu]

What does this pull request do? Explain your changes. (required)

We've occasionally observed an unsupported block number error for RPC requests which can cause problems for node operation (see #2560).

According to an Offchain Labs team member, an Arb node can return this error if the node is not synced. This situation could occur if:

• A single Arb node has fallen behind the tip of the chain
• If requests are load balanced across a set of nodes and request 1 fetches block number N from node 1, but request 2 that references block number N is routed to node 2 which has only seen block number N - 1

The unsupported block number error appears to originate from the getSnapshot() and getSnapshotForNumberForHash() functions in offchainlabs/arbitrum/packages/arb-rpc-node/web3 which are called by many of the functions implementing the ETH JSON-RPC API [1].

From reviewing Discord conversations, it appears that this error is more likely to happen when sending concurrent RPC requests.

The solution in this PR is to add the unsupported block number error to the list of errors that will be retried when sending requests for remote contract calls. Note: This change only applies to contract calls and not other RPC requests which could be affected by this error as well. Addressing this issue with all RPC requests requires additional code changes, but since this error is primarily affecting contract calls right now, I've opted to just address this for contract calls.

Specific updates (required)

See commit history.

How did you test each of these updates (required)

• Added unit tests for helper functions
• Rolled this change to a broadcaster and observed retries for unsupported block number errors

Does this pull request close any open issues?

Addresses the root cause of #2560 although it could still make sense to update the behavior of the node following the suggestion in the issue.

Checklist:

• Read the contribution guide
• make runs successfully
• All tests in ./test.sh pass
• README and other documentation updated
• Pending changelog updated

[codecov]

Codecov Report

Merging #2563 (fa007c8) into master (f6948b8) will increase coverage by 0.09914%.
The diff coverage is 100.00000%.

@@                 Coverage Diff                 @@
##              master       #2563         +/-   ##
===================================================
+ Coverage   54.71489%   54.81403%   +0.09914%     
===================================================
  Files             95          95                 
  Lines          19852       19869         +17     
===================================================
+ Hits           10862       10891         +29     
+ Misses          8385        8373         -12     
  Partials         605         605                 

Impacted Files	Coverage Δ
eth/backend.go	50.35461% <100.00000%> (+16.48364%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6948b8...fa007c8. Read the comment docs.

Impacted Files	Coverage Δ
eth/backend.go	50.35461% <100.00000%> (+16.48364%)	⬆️

[leszko]

Added one comment. Other than that LGTM 👍

[yondonfu]

Rebased to re-order commits.

[thomshutt]

LGTM