tidb panic on br after inject pd leader io hang

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run pitr
2、run workload
3、inject pd leader io hang

2. What did you expect to see? (Required)

no panic

3. What did you see instead (Required)

tidb panic

2023-10-11 06:57:28 log="\n"
2023-10-11 06:57:28 log="/tidb-server --store=tikv --advertise-address=tc-tidb-1.tc-tidb-peer.endless-ha-test-ticdc-tps-3000240-1-651.svc --host=0.0.0.0 --path=tc-pd:2379 --config=/etc/tidb/tidb.toml\n"
2023-10-11 06:57:28 log="start tidb-server ...\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:150 +0x78\n"
2023-10-11 06:57:27 log="created by github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable in goroutine 777\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:152 +0x4d\n"
2023-10-11 06:57:27 log="github.com/pingcap/tidb/br/pkg/streamhelper.spawnJoinable.func1()\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:234 +0x1f\n"
2023-10-11 06:57:27 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).doConnect.func1()\n"
2023-10-11 06:57:27 log="\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/streamhelper/flush_subscriber.go:277 +0xbe5\n"
2023-10-11 06:57:27 log="github.com/pingcap/tidb/br/pkg/streamhelper.(*subscription).listenOver(0xc020210b00, {0x60d80a0, 0xc04930c3b0})\n"
2023-10-11 06:57:27 log="goroutine 246752358 [running]:\n"
2023-10-11 06:57:27 log="\n"
2023-10-11 06:57:27 log="panic: send on closed channel\n"
2023-10-11 06:57:27 log="[flush_subscriber.go:144] ["join handle timed out."]"
2023-10-11 06:57:01 log="[tikv_handler.go:2019] ["update server labels"] [labels="{\"host\":\"172.16.6.13\",\"topology.kubernetes.io/zone\":\"zone1\",\"zone\":\"zone1\"}"]"
2023-10-11 06:57:01 log="[tikv_handler.go:2019] ["update server labels"] [labels="{\"host\":\"172.16.6.13\",\"topology.kubernetes.io/zone\":\"zone1\",\"zone\":\"zone1\"}"]"
2023-10-11 06:56:31 log="[tikv_handler.go:2019] ["update server labels"] [labels="{\"host\":\"172.16.6.13\",\"topology.kubernetes.io/zone\":\"zone1\",\"zone\":\"zone1\"}"]"
2023-10-11 06:56:27 log="[flush_subscriber.go:297] ["Removing subscription."] [category="log backup subscription manager"] [store=4]"
2023-10-11 06:56:27 log="[flush_subscriber.go:144] ["join handle timed out."]"

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.5.0-alpha
Edition: Community
Git Commit Hash: 97c7741
Git Branch: heads/refs/tags/v7.5.0-alpha
UTC Build Time: 2023-10-10 11:47:18
GoVersion: go1.21.1
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-10-11T06:39:11.534+0800

[Lily2025]

/type bug
/severity critical
/assign YuJuncen