Manage outstanding tasks after server restart

[aarontp]

We have an issue filed to consider keeping local state in the server if there is a crash or restart, but even without this we can handle outstanding tasks better as currently they will remain open indefinitely even if the timeout has passed. Here are a couple options to investigate:

• Maybe celery has a better way to recover the state after crash/restart. Currently we don't have the task "stubs" on the server, so we can't update the metadata even if we get responses from workers after a crash.
• Considering "closing" all outstanding tasks when the server starts up. If the task is actually still running, then it can update its metadata from the worker (though we should verify what would happen in this case). We could potentially only close tasks where the timeouts have been exceeded as well.

[aarontp]

FYI @jleaniz : As discussed.