Leverage advanced RabbitMQ features to implement cheap webhook retry mechanism

[YoannAubineau]

Currently, webhook messages whose HTTP call fails are consumed and requeued from/to RabbitMQ continuously. Doing so wastes a full CPU core for nothing. A less costly approach would be to use the dead-letter mechanism RabbitMQ provides.

Here is the general idea:

1. After each ballot, a webhook message is enqueued into the webhook queue.
2. The webhook worker consumes these messages in order and tries to call the associated remote server.
3. It the call fails, the worker rejects the webhook message.
4. Rejected messages are automatically routed to an isolation queue with no consumer.
5. After a certain amount of time, isolated messages are automatically sent back to the webhook queue.
6. The message can be rejected and delayed this way many times.
7. When the webhook call succeeds, the worker acknowledges the message. Done.
8. If the message becomes too old, the worker can silently acknowledge it to make it disappear from the queue.

This mechanism requires subtle configuration that I will describe here once I manage to have working POC.