Slow response on RPC endpoints at high block height

[kantai]

Last week, during the day before the testnet reset, the Stacks argon master node was very slow to respond (or completely unresponsive) to RPC queries.

/v2/info began to take ~5s to respond on Aug 18, and by Aug. 19, it took up to a minute to respond (or didn't respond at all).

I grabbed a core dump from the machine before the testnet restarted, however, it did not prove to be terribly useful -- I couldn't produce a corresponding stacks-node binary with a matching symbols table, all threads did appear to have been running, though. I cannot locate this core dump anymore, though :(

I do have logging data for the period. I've attached the captured logs for the ~15 minutes before reset.

stacks-node-argon-aug-19.txt

[jcnelson]

Well, the good news is that unlike before, the relayer thread is still alive and doing stuff. You can see it with lines like:

"Aug 19, 2020 @ 14:08:14.000","[testnet/stacks-node/src/neon_node.rs:505] [ThreadId(6)] Relayer: Run tenure"
...
"Aug 19, 2020 @ 14:08:16.000","[testnet/stacks-node/src/neon_node.rs:377] [ThreadId(6)] Relayer: Try process attacheable blocks"
...
"Aug 19, 2020 @ 14:08:42.000","[testnet/stacks-node/src/neon_node.rs:421] [ThreadId(6)] Relayer: Process tenure"

However, I don't see any cases of Relayer: Handle network result, which suggests to me that the p2p thread has stalled out or stopped somehow.

[jcnelson]

We should try and reproduce this locally until the testnet block height can get this high again.

[jcnelson]

Is this still happening?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed. Please reopen if needed.