[FEATURE] Support secondary storage when primary storage fails

[imyousuf]

Is your feature request related to a problem? Please describe.
In an attempt to increase high availability of the cluster, we need to plan for situations where RDBMS is not available either due to a network issue or DB server crashing etc. In such cases we want to at least receive the message so that rest of the processing can happen once the DB is recovered. For this supporting a secondary storage for "buffering" messages would come in handy.

Describe the solution you'd like
We would like the type of storage to be different from RDBMS so that when used in context of a cloud provider the failure scenario can be non-overlapping. We would also want composability of secondary storage such that multiple can be transparently overlaid. Given our initial infrastructure support is for AWS we would like compose DynamoDB over S3 as the method for highly available Webhook Broker; in this way, we could either just use S3 or just use DynamoDB or use a composition of the 2 such that both are used and depending on what is available the secondary worker picks the message.

Here is how the pseudo code might look -

1. Write to primary storage
2. If write fails write to secondary storage synchronously
3. If write to primary succeeds then write to secondary storage asynchronously with flag that it was written to primary

In a worker query secondary storage for all messages that has not yet been written to primary; write them to primary and keep note of it.

Additional context
One key question to consider is, whether we should write the message to the secondary storage regardless of whether DB is available. The use would be, if RDBMS is down along with the read replica and we need to recreate the DB from scratch, we can at least use the secondary storage to reinstate messages from last known healthy state of the DB that way ensuring no message left undelivered.