Nodus is Network controller in Kubernetes that address multiple network use case as below
- Multi ovn network support
- Multi-interface ovn support
- Multi-IP address support
- Dynamic creation of virtual networks
- Route management across virtual networks and external networks
- Service Function chaining(SFC) support in Kubernetes
- OVN ACL based Network Policy
- Secure Nodus and OVN Network Traffic(WIP)
- SRIOV Overlay networking (WIP)
- OVN load balancer (WIP)
Nodus is latin word for "knot". Nodus converge multiple kubernetes networking use cases in a single network controller.
Nodus consist of 4 major components
- OVN control plane
- OVN controller
- Network Function Network(NFN) k8s operator/controller
- Network Function Network(NFN) agent
OVN control plane and OVN controller take care of OVN configuration and installation in each node in Kubernetes. NFN operator runs in the Kubernetes master and NFN agent run as a daemonset in each node.
- Exposes virtual, provider, chaining CRDs to external world
- Programs OVN to create L2 switches
- Watches for PODs being coming up
- Assigns IP addresses for every network of the deployment
- Looks for replicas and auto create routes for chaining to work
- Create LBs for distributing the load across CNF replicas
- Performs CNI operations.
- Configures VLAN and Routes in Linux kernel (in case of routes, it could do it in both root and network namespaces)
- Communicates with OVSDB to inform of provider interfaces. (creates ovs bridge and creates external-ids:ovn-bridge-mappings)
ovn4nfv-default-nw is the default logic switch create for the default networking in kubernetes pod network for cidr 10.233.64.0/18. Both node and pod in the kubernetes cluster share the same ipam information.
In general production env, we have multiple Network function such as SLB, NGFW and SDWAN CNFs.
There are general 3 sfc flows are there:
- Packets from the pod to reach internet: Ingress (L7 LB) -> SLB -> NGFW -> SDWAN CNF -> External router -> Internet
- Packets from the pod to internal server in the corp network: Ingress (L7 LB) -> SLB -> M3 server
- Packets from the internal server M3 to reach internet: M3 -> SLB -> NGFW -> SDWAN CNF -> External router -> Internet
Nodus SFC currently support all 3 flows.
Install the docker in the Kubernetes cluster node.
Follow the steps in create cluster kubeadm to create kubernetes cluster in master
In the master node run the kubeadm init
as below. The Nodus uses pod network cidr 10.210.0.0/16
$ kubeadm init --kubernetes-version=1.23.3 --pod-network-cidr=10.210.0.0/16 --apiserver-advertise-address=<master_eth0_ip_address>
Ensure the master node taint for no schedule is removed and labelled with ovn4nfv-k8s-plugin=ovn-control-plane
nodename=$(kubectl get node -o jsonpath='{.items[0].metadata.name}')
kubectl taint node $nodename node-role.kubernetes.io/master:NoSchedule-
kubectl label --overwrite node $nodename ovn4nfv-k8s-plugin=ovn-control-plane
Kustomize and deploy cert-manager:
$ curl -Ls https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml -o deploy/cert-manager/cert-manager.yaml && kubectl apply -k deploy/cert-manager/
Deploy the Nodus Pod network to the cluster.
$ kubectl apply -f deploy/ovn-daemonset.yaml
$ kubectl apply -f deploy/ovn4nfv-k8s-plugin.yaml
Ensure the configmap ovn-controller-network
data OVN_SUBNET
matches the pod network cidr as well in deploy/ovn4nfv-k8s-plugin.yaml
Join worker node by running the kubeadm join
on each node as root as mentioned in create cluster kubeadm
Kubespray support the Nodus as the network plugin- please follow the steps in kubernetes-sigs/kubespray/docs/ovn4nfv.md
The etcd store data such as cluster state and k8s secrets. The best approach is to set up a firewall between the Kubernetes API server in the control plane node and etcd in a different node, the access to the etcd is limited by the firewall for the API server in the control plane only. The user must always ensure the mutual auth via TLS client certificate. More information to setup can be found in the etcd documentation
By default k8s secret data stored in the etcd are not encrypted. The user must use the KMS encryption provider for strong encryption - link. The user should not use other encryption provider mechanisms such as secretbox(XSalsa20 and Poly1305 encryption), aesgcm(AES-GCM with random nonce), aescbc(AES-CBC with PKCS#7 padding) as they are show vulnerability/not recommended in the Kubernetes documentations
Add /var/log/openvswitch/ovn4k8s.log
in the audit.rules to monitor the log files and ensure the logs are not tampered - link
- How to use
- Configuration
- Development
- Validation & testcase
- Akraino ICN Recommended Operating system security tools
For any questions about Nodus k8s , feel free to ask a question in #general in the ICN slack, or open up a https://jira.akraino.org/projects/ICN/issues.