Possibility to handle IP address conflicts with internal OpenShift CIDR

[mirogl]

Hi,
we have now run into the issue, that a client ip, which accesses the keepalived ip, is in the same CIDR IP range, like the internal OpenShift CIDR and can't access the services exposed by Keepalived. Requests from other Client IP ranges, whch are not in the CIDR of the OpenShift cluster work without any problem.
Our assumption is, that the packets are dropped by the OpenShift firewall, since the rules think, the packets come from the internal CIDR as SRC.

Current situation:
Client IP (10.131.3.x)->Keepalived IP(10.25.178.x) -> OpenShift internal CIDR (10.128.0.0/14)

IPTables:
Chain OPENSHIFT-FIREWALL-FORWARD (1 references)
target prot opt source destination
DROP all -- 10.128.0.0/14 anywhere /* attempted resend after connection close / ctstate INVALID
ACCEPT all -- anywhere 10.128.0.0/14 / forward traffic from SDN

Question:
Is the keepalived operator working in NAT mode for its ip, so the 10.25.178.x should be the src ip of the packets in the cluster or is it really the problem, that the src ip 10.131.3.x is beeing routed directly ?

Since i din't find any infos about the mode (NAT/DR) in this documentation and only found an annotation in another documentation
https://github.com/munnerz/keepalived-cloud-provider

The kube-keepalived-vip service supports both the NAT and DR methods of IPVS forwarding for the service traffic. The default forwarding method is NAT. Depending on your network topology, you may need to change that to DR (direct routing). To change this globally, you can set the environment variable KEEPALIVED_DEFAULT_FORWARD_METHOD to NAT or DR. To change it on a per service basis, then specify the method via the k8s.co/keepalived-forward-method annotation on the service as shown below:

Can you please give me the info, how the routing works and if it can be changed by a annotation ?

Thanks
Miro

[raffaelespazzoli]

The operator will use the keepalived default, which, I believe, is NAT, unless you change the behavior with a verbatim configuration. Either way (NAT or DR) I don't think your use case will work as long as you have conflicting CIDRs.

…

On Thu, Nov 25, 2021 at 4:37 AM mirogl ***@***.***> wrote: Hi, we have now run into the issue, that a client ip, which accesses the keepalived ip, is in the same CIDR IP range, like the internal OpenShift CIDR and can't access the services exposed by Keepalived. Requests from other Client IP ranges, whch are not in the CIDR of the OpenShift cluster work without any problem. Our assumption is, that the packets are dropped by the OpenShift firewall, since the rules think, the packets come from the internal CIDR as SRC. Current situation: Client IP (10.131.3.x)->Keepalived IP(10.25.178.x) -> OpenShift internal CIDR (10.128.0.0/14) IPTables: Chain OPENSHIFT-FIREWALL-FORWARD (1 references) target prot opt source destination DROP all -- 10.128.0.0/14 anywhere /* attempted resend after connection close */ ctstate INVALID ACCEPT all -- anywhere 10.128.0.0/14 <http://10.128.0.0/14> /* forward traffic from SDN Question: Is the keepalived operator working in NAT mode for its ip, so the 10.25.178.x should be the src ip of the packets in the cluster or is it really the problem, that the src ip 10.131.3.x is beeing routed directly ? Since i din't find any infos about the mode (NAT/DR) in this documentation and only found an annotation in another documentation https://github.com/munnerz/keepalived-cloud-provider The kube-keepalived-vip service supports both the NAT and DR methods of IPVS forwarding for the service traffic. The default forwarding method is NAT. Depending on your network topology, you may need to change that to DR (direct routing). To change this globally, you can set the environment variable KEEPALIVED_DEFAULT_FORWARD_METHOD to NAT or DR. To change it on a per service basis, then specify the method via the k8s.co/keepalived-forward-method annotation on the service as shown below: Can you please give me the info, how the routing works and if it can be changed by a annotation ? Thanks Miro — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#82>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPERXHUHGG4GYGRSHLOAUTUNX7VNANCNFSM5IYB24GQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- ciao/bye Raffaele

[mirogl]

Hi @raffaelespazzoli
Thanks - Is there some query, how i can read out the current active routing mode ?
Do you know, where i can find the documentation for the possible verbatim configuration. Under https://keepalived.readthedocs.io/en/latest/introduction.html I don't see any documentation.

Thanks
Miro

[raffaelespazzoli]

you can find the current configuration in the configmap mounted by the keepalived pods. It should not be difficult to discover. That configmap is managed by the operator. By verbatim configuration I meant this: https://github.com/redhat-cop/keepalived-operator#verbatim-configurations . it looks like you could probably control the `lb_algo` ( https://keepalived.readthedocs.io/en/latest/configuration_synopsis.html#virtual-server-definitions-synopsis) via verbatim config and try different combinations.

…

On Thu, Nov 25, 2021 at 7:30 AM mirogl ***@***.***> wrote: Hi @raffaelespazzoli <https://github.com/raffaelespazzoli> Thanks - Is there some query, how i can read out the current active routing mode ? Do you know, where i can find the documentation for the possible verbatim configuration. Under https://keepalived.readthedocs.io/en/latest/introduction.html I don't see any documentation. Thanks Miro — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPERXA7CPOKVLHF6ZSYL7LUNYT7LANCNFSM5IYB24GQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- ciao/bye Raffaele

[mirogl]

Hi @raffaelespazzoli
Many Thanks - than I will take a look into the documentation.
Thanks
Miro

[mirogl]

Ciao @raffaelespazzoli
I checked the documentation and i have one open question - you write in your documentation:

To account for that there is a way to pass verbatim options both at the keepalived group level (which maps to the keepalived config global_defs section) and at the service level (which maps to the keepalived config vrrp_instance section).

This is clear to me. But the documentation at https://keepalived.readthedocs.io/en/latest/configuration_synopsis.html#virtual-server-definitions-synopsis has
keepalived config Global Definitions
keepalived config Virtual Server Definitions
keepalived config VRRP Instance Definitions

So from your documentation it is clear, how to modify "Global Definitions" and "VRRP Instance Definitions", but not clear, how to modify the "Virtual Server Definitions" - should i just add them after the "Global Definitions" with the instructions for modifiying "Global Definitions" ?

Also unclear:
In the /etc/keepalived/keepalived.conf i found the lb_xxx setting only in the virtual server section, for e.G.:

virtual_server 192.168.200.100 443 { delay_loop 6 lb_algo rr lb_kind NAT persistence_timeout 50 protocol TCP
but this is the default configuration for some example virtual servers - but not my IP (expected)

my configuration has only the following values and no virtual server definition, where i could add the lb_xxx values:

`global_defs {
router_id keepalivedgroup-workers

}

vrrp_instance external/external {
interface ens192

virtual_router_id 3

virtual_ipaddress {

  10.25.179.x

}

}

vrrp_instance internal/internal {
interface ens192

virtual_router_id 4

virtual_ipaddress {

  10.25.179.y

}

}
`

So I'm still not clear, where to set the lb_xxx setting, so that it is used also with my vrrp_instances. Can I just add it to global config - but in the documentation it is not a valid key / value for global configuration, only for virtual server.

Thanks
Miro