Cluster (Node) Networking

Each node must have at least one interface connected to a common network. Each interface must have an IP address configured. Every node must have a unique hostname as well as a unique MAC address.

Various ports need to be opened (firewall and security group settings must be configured) on the master and worker nodes as shown in the diagram. The worker nodes expose services for external access on ports 30000 - 32767 (for NodePort services).

In a multi-master setup, the ETCD clients need to communicate on port 2380, so that needs to be open as well.

Pod Networking

Now that we have configured networking at the cluster level as explained in the previous section, all the nodes can communicate with each other each other. But we also need to establish networking at the pod level. K8s doesn’t come with a native pod networking solution. Instead it requires us to set it up ourselves.

We can use a CNI compatible networking solution (plugin) to achieve this. Basically, the CNI plugin ensures that every pod on the cluster (irrespective of the node it is running on), gets a unique IP address.

After setting up the networking solution:

• Every pod in the cluster should have a unique IP Address.
• Every pod should be able to communicate with every other POD on the same node by using IP addresses.
• Every pod should be able to communicate with every other POD on other nodes by directly using their IP addresses, without using a NAT.

Networking solutions create a bridge network on each node with a different subnet mask and attach every pod to the bridge network on its node. This way, every pod in the cluster gets a unique IP address. Also, pods on the same node can now communicate with each other using IP addresses.

At this stage, pods cannot communicate across nodes. If the blue pod (10.244.1.2) tries to ping the purple pod (10.244.2.2), it won’t be able to since it is on a different subnet. It then routes the request to NODE1’s IP (192.168.1.11) as it is the default gateway for the bridge network on NODE1. Even NODE1 has no idea where the subnet 10.244.2.0/24 is as it is a private network on NODE2. So, we need a router configured as the default gateway on each node to tell them where to forward the requests going to pods on various bridge networks.

This way, the bridge networks running on each node coalesce together to form a large network with CIDR 10.244.0.0/16.