Annotation consul.hashicorp.com/auto-join-port breaks connection between servers and clients

[ndhanushkodi]

The annotation consul.hashicorp.com/auto-join-port seems to break the connection between the consul servers and clients.

Reproduction steps:

1. Deploy the Consul Helm chart with the default values
2. Wait until you see all the clients and servers 'Running'
3. kubectl edit statefulset <helm-releasename>-consul-server and edit the pod template's annotations to include consul.hashicorp.com/auto-join-port: 8301. Since this is the default gossip port anyways, the expected behavior is that this would work.
4. The consul server pods should restart
5. Observe that the consul client pods stop running and start logging:

    2021-01-07T23:17:50.828Z [ERROR] agent.http: Request error: method=GET url=/v1/status/leader from=127.0.0.1:34840 error="No known Consul servers"
    2021-01-07T23:18:00.627Z [ERROR] agent: Coordinate update error: error="No known Consul servers"
    2021-01-07T23:18:00.829Z [ERROR] agent.http: Request error: method=GET url=/v1/status/leader from=127.0.0.1:34884 error="No known Consul servers"
    2021-01-07T23:18:03.309Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
    2021-01-07T23:18:10.830Z [ERROR] agent.http: Request error: method=GET url=/v1/status/leader from=127.0.0.1:34926 error="No known Consul servers"

Use case:

When trying to join a consul cluster in Kubernetes with external client agents running on an external VM, you might want to connect on the Kubernetes node IP to the consul servers gossip and RPC ports and to the consul clients gossip ports. To do this, the consul servers in Kubernetes would have to configure their gossip port to be different from the client gossip ports for example, to 9301. For the external client agent to auto-join using the k8s provider, I'd want to add this annotation to the consul server pods to say consul.hashicorp.com/auto-join-port: "9301".