pkg/runtest: TestCover keeps timing out #4954
Comments
|
One more case: https://github.com/google/syzkaller/actions/runs/9758033237/job/26931740035 I was not the case up to this week, so I guess it's caused by some recent changes. |
|
It happens not just under That's here: syzkaller/pkg/runtest/run_test.go Line 390 in 780d1bc That's here: syzkaller/pkg/vminfo/features.go Line 69 in 780d1bc So some syz-executors just stop communicating and we wait forever for their responses? That's bad by itself, we probably somehow not react to their errors. Regarding the reason why they fail -- could it be that the Docker container we use for testing e.g. somehow limits the number of simultaneous processes? After the recent change that made TestCover subtests create their own PRC servers, the load must have increased significantly. |
a-nogikh
changed the title
|
https://github.com/google/syzkaller/actions/runs/9779999561/job/27000577263 Probably related? |
|
Memory consumption doesn't look crazy: In any case, I've tried to reproduce it many times locally and never managed to. |
|
@a-nogikh what commit do you think triggered or scaled this problem? |
|
I think we've begun to observe it after |
|
We may try to remove t.Parallel() from there, maybe there is some contention... |
Have you tried with |
|
The test VMs should be actually quite big (as told by @tarasmadan) -- up to 32 cores. If it were just getting slower, I think the test would have still passed, just slower. But here something definitely goes very wrong, and in a weird way. I wonder if we're really staying careful w.r.t. the syzkaller/pkg/rpcserver/local.go Line 87 in ecfab6a syzkaller/pkg/rpcserver/local.go Lines 96 to 97 in ecfab6a And then eventually close the flatrpc server. But do the runner loops ever notice that? If we don't send new requests, we're just sleeping and not even interacting with the socket. syzkaller/pkg/rpcserver/runner.go Lines 166 to 170 in ecfab6a (If yes, we have a separate problem -- no new requests mean we never read from the socket, so we never get results for the previous one(s)). Ideally, |
|
Good point. There are sleeping goroutines in connectionLoop in the dump. |
On a recent after submit testing: https://github.com/google/syzkaller/actions/runs/9742478582/job/26883877667
The text was updated successfully, but these errors were encountered: