(pdms)tpcc continues to report a lot of Error 8027 after disable/enable ms mode

[mayjiang0203]

Bug Report

What did you do?

Enable and disable pd ms mode by edit tc.

What did you expect to see?

No error report by workload.

What did you see instead?

root@benchtoolset-0:/go# go-tpc tpcc run -D tpcc --host tc-tidb.testbed-oltp-hm-bm8rb -P4000 --warehouses 10 -T 10 --time 36000m --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062' --user root --password '' --interval '10s'
[2024-02-22 09:32:28] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:28] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:32] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[Current] DELIVERY - Takes(s): 0.9, Count: 1, TPM: 69.2, Sum(ms): 118.8, Avg(ms): 119.5, 50th(ms): 121.6, 90th(ms): 121.6, 95th(ms): 121.6, 99th(ms): 121.6, 99.9th(ms): 121.6, Max(ms): 121.6
[Current] NEW_ORDER - Takes(s): 9.9, Count: 22, TPM: 133.2, Sum(ms): 13133.6, Avg(ms): 595.5, 50th(ms): 21.0, 90th(ms): 31.5, 95th(ms): 3892.3, 99th(ms): 9126.8, 99.9th(ms): 9126.8, Max(ms): 9126.8
[Current] NEW_ORDER_ERR - Takes(s): 9.9, Count: 2, TPM: 12.1, Sum(ms): 13516.4, Avg(ms): 6710.9, 50th(ms): 5100.3, 90th(ms): 8589.9, 95th(ms): 8589.9, 99th(ms): 8589.9, 99.9th(ms): 8589.9, Max(ms): 8589.9
[Current] ORDER_STATUS - Takes(s): 9.9, Count: 4, TPM: 24.2, Sum(ms): 39.8, Avg(ms): 10.0, 50th(ms): 10.0, 90th(ms): 12.6, 95th(ms): 12.6, 99th(ms): 12.6, 99.9th(ms): 12.6, Max(ms): 12.6
[Current] PAYMENT - Takes(s): 9.9, Count: 18, TPM: 108.8, Sum(ms): 18912.1, Avg(ms): 1049.2, 50th(ms): 10.5, 90th(ms): 5100.3, 95th(ms): 5100.3, 99th(ms): 5100.3, 99.9th(ms): 5100.3, Max(ms): 5100.3
[Current] PAYMENT_ERR - Takes(s): 9.9, Count: 1, TPM: 6.0, Sum(ms): 5018.8, Avg(ms): 4966.1, 50th(ms): 5100.3, 90th(ms): 5100.3, 95th(ms): 5100.3, 99th(ms): 5100.3, 99.9th(ms): 5100.3, Max(ms): 5100.3
[Current] STOCK_LEVEL - Takes(s): 9.9, Count: 3, TPM: 18.2, Sum(ms): 45.8, Avg(ms): 15.2, 50th(ms): 15.7, 90th(ms): 17.8, 95th(ms): 17.8, 99th(ms): 17.8, 99.9th(ms): 17.8, Max(ms): 17.8
[2024-02-22 09:32:33] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:33] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:37] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:37] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:38] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:38] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:42] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[2024-02-22 09:32:42] execute run failed, err Error 8027: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV
[Current] DELIVERY - Takes(s): 9.7, Count: 3, TPM: 18.5, Sum(ms): 313.5, Avg(ms): 104.2, 50th(ms): 113.2, 90th(ms): 121.6, 95th(ms): 121.6, 99th(ms): 121.6, 99.9th(ms): 121.6, Max(ms): 121.6
[Current] NEW_ORDER - Takes(s): 9.9, Count: 28, TPM: 169.4, Sum(ms): 5450.2, Avg(ms): 190.5, 50th(ms): 24.1, 90th(ms): 32.5, 95th(ms): 35.7, 99th(ms): 4831.8, 99.9th(ms): 4831.8, Max(ms): 4831.8
[Current] NEW_ORDER_ERR - Takes(s): 9.9, Count: 5, TPM: 30.3, Sum(ms): 25099.7, Avg(ms): 4966.1, 50th(ms): 5100.3, 90th(ms): 5100.3, 95th(ms): 5100.3, 99th(ms): 5100.3, 99.9th(ms): 5100.3, Max(ms): 5100.3
[Current] ORDER_STATUS - Takes(s): 9.8, Count: 1, TPM: 6.2, Sum(ms): 10.4, Avg(ms): 10.2, 50th(ms): 10.5, 90th(ms): 10.5, 95th(ms): 10.5, 99th(ms): 10.5, 99.9th(ms): 10.5, Max(ms): 10.5
[Current] PAYMENT - Takes(s): 9.9, Count: 23, TPM: 139.2, Sum(ms): 56492.2, Avg(ms): 2461.4, 50th(ms): 19.9, 90th(ms): 8589.9, 95th(ms): 13958.6, 99th(ms): 14495.5, 99.9th(ms): 14495.5, Max(ms): 14495.5
[Current] PAYMENT_ERR - Takes(s): 9.9, Count: 3, TPM: 18.2, Sum(ms): 23998.3, Avg(ms): 8053.1, 50th(ms): 5100.3, 90th(ms): 14495.5, 95th(ms): 14495.5, 99th(ms): 14495.5, 99.9th(ms): 14495.5, Max(ms): 14495.5

What version of PD are you using (pd-server -V)?

sh-5.1# /pd-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: d6d9feab3e2a5180acbcc7095723d43e97798686
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time:  2024-02-21 11:38:00

[mayjiang0203]

/severity major
/assign @rleungx

[mayjiang0203]

w/a is restart all tidb nodes.