Infinispan connectivity lost when cluster Service changes IP address

[dougbreaux]

We're trying to address a situation where our OpenShift OpenLibertyApplication using Infinispan (as Red Hat Data Grid) loses connection to the Infinispan cluster, apparently whenever the Infinispan cluster changes its IP address.

It's unclear when that is, but appears it might be whenever the RHDG Operator updates version. This case is on ROSA. The Infinispan IP, etc. appear to be injected into the OLA pods as environment variables, but they aren't automatically restarted when this changes.

The instructions also clearly tell us to configure the Service name, which we've done, so I'm a bit surprised the IP address is used. Our guess is that maybe for efficiency of not doing DNS lookups all the time, it's looked up once at pod/Liberty start?

[dougbreaux]

Looking for an update on this, please.

[dougbreaux]

From the Red Hat case:

I have checked with RHDG support team, the internal IP addresss of the INFINISPAN cluster is subject to change and there is no way to use a fixed address.

But to connect internally within ROSA, you can use the SERVICE - maybe you can replace all the IPs as service DNS instead as a test to see if OL pods can work that way.

If Hot Rod [1] is in use and there is no Intelligence set, it will use HASH_AWARE by default, now the first connection will go to the service. After connected the server (cluster) sends back a list of all the servers and the Hash. From that point the client will act the same way as if you connect to the internal IP. So this would be the way you are independent from the IP changes.

[1] https://access.redhat.com/documentation/fr-fr/red_hat_data_grid/6/html/developer_guide/chap-the_hot_rod_interface#:~:text=Hot%20Rod%20enables%20clients%20to,server%20that%20has%20the%20key.

Would you please also let us know the purpose of this approach and what is the idea behind it? Eg. Reach a special instance in the cluster or LoadBalancing?