Test clientv3 balancer under network partitions, other failures

[gyuho]

Related: #8660 and #8828.

Servers have been extensively tested under network partitions, through functional-tester. And we haven't seen any failures for the last >5 months.

Client behavior under partitioned cluster needs more integration tests (though it's at least manually tested and we know that it works as expected). Add more tests to network_partition_test.go, once the new health balancer passes all current CIs.

3-node cluster and client is configured with all 3 endpoints (A, B, C):

• client balancer pins A, and member A fails (status: manually tested)
  • Watch without WithRequireLeader(context.Context), without HTTP/2 keepalive ping
    • balancer automatically switches to either B or C with a new connection
    • expect no error
    • done via clientv3/integration: add TestBalancerUnderServerShutdownWatch #8758
  • Put/Delete/Txn
    • current request returns an error to the client
    • balancer switches to either B or C with a new connection
    • manual retry is needed, next request succeeds with the new endpoint (B or C)
    • done via clientv3/integration: add TestBalancerUnderServerShutdownMutable* #8772
  • Linearizable Get
    • balancer switches to either B or C with a new connection
      • expect internal retrial logic handles server errors
    • no manual retry is needed, request should succeed with automatic retry
    • done via clientv3/integration: add TestBalancerUnderServerShutdownImmutable #8779
  • Serializable Get
    • balancer switches to either B or C with a new connection
      • expect internal retrial logic handles server errors
    • no manual retry is needed, request should succeed with automatic retry
    • done via clientv3/integration: add TestBalancerUnderServerShutdownImmutable #8779
• client balancer pins A, and member A is partitioned from member B and C, while client can still talk to all 3 nodes (status: manually tested)

  iptables -A OUTPUT -p tcp --destination-port 2380 -j DROP
  iptables -A INPUT -p tcp --destination-port 2380 -j DROP
  

  • Watch with WithRequireLeader(context.Context), without HTTP/2 keepalive ping
    • expect etcdserver.ErrNoLeader (watch channel should be closed)
    • done via clientv3/integration: add TestBalancerUnderNetworkPartitionWatch #8762
  • Put/Delete/Txn
    • current request returns an error to the client
    • balancer switches to either B or C with a new connection
    • manual retry is needed, next request succeeds with the new endpoint (B or C)
    • Put done via clientv3/balancer: handle network partition in health check #8669
    • Delete, Txn done via clientv3/integration: finish isolated node test cases #8785
  • Linearizable Get
    • balancer switches to either B or C with a new connection
      • expect internal retrial logic handles server errors
        • e.g. etcdserver: request timed out when context timeout > request timeout
    • no manual retry is needed, request should succeed with automatic retry
    • done via clientv3/integration: finish isolated node test cases #8785
  • Serializable Get
    • no manual retry is needed, request should succeed with automatic retry
    • done via clientv3/integration: finish isolated node test cases #8785
• client balancer pins A, and member A is blackholed (status: manually tested)

  iptables -A INPUT -p tcp --destination-port 2379 -j DROP
  

  • Watch without WithRequireLeader(context.Context), with HTTP/2 keepalive ping
    • balancer automatically switches to either B or C with a new connection
    • expect no error
    • done via *: add watch with client keepalive test #8626 and clientv3/integration: move to TestBalancerUnderBlackholeKeepAliveWatch #8792
  • Put/Delete/Txn
    • current request returns an error to the client
    • balancer switches to either B or C with a new connection
    • manual retry is needed, next request succeeds with the new endpoint (B or C)
    • Put done via clientv3/integration: add put blackhole test #8769
    • Delete, Txn done via clientv3/integration: add blackhole tests on mutable operations #8789
  • Linearizable Get
    • current request returns an error to the client (context.DeadlineExceeded)
    • manual retry is needed, next request succeeds with the new endpoint (B or C)
    • done via clientv3/integration: add blackhole tests for range RPCs #8790
  • Serializable Get
    • current request returns an error to the client (context.DeadlineExceeded)
    • manual retry is needed, next request succeeds with the new endpoint (B or C)
    • done via clientv3/integration: add blackhole tests for range RPCs #8790