23/04/2019 How Kubernetes Networking Works - Under the Hood

(https://neuvector.com)
Products (https://neuvector.com/run- Solutions Learn About
time-container-security/) (https://neuvector.com/learn-
container-security/)
Kubernetes Security Company & Tea
Full Lifecycle Container Security (https://neuvector.com/kubernetes- (https://neuvect
(https://neuvector.com/container- security-solutions/)
security-platform/)
Contact Us
Network Security (https://neuvect
Complete Run-Time Security (https://neuvector.com/container-
(https://neuvector.com/run-time- network-security-solutions/)
Technology Par
container-security/)
(https://neuvect
Red Hat OpenShift
Unique Container Network Security (https://neuvector.com/redhat-
Solution Partne
(https://neuvector.com/container- openshift/)
(https://neuvect
network-security/)
partners/)
Service Mesh Security
Fast Vulnerability Scanning (https://neuvector.com/service-
Events (https://
(https://neuvector.com/vulnerability- mesh-security/)
scanning/)
Webinars
AWS, EKS and ECS
(https://neuvect
(https://neuvector.com/aws-
webinars/)
security-solutions/)

Press (https://n
Azure, Google, IBM, Alibaba Public
Cloud
(https://neuvector.com/public- Join Our Team
cloud-container-solutions/) descriptions/)

Scanning and Compliance

(https://neuvector.com/container-
compliance-auditing-solutions/)

How Kubernetes Networking Works – Under the Hood

NeuVector (https://neuvector.com)  Network Security (https://neuvector.com/category/network-security/)  How Kubernetes Networking Works – Under the Hood

NETWORK SECURITY (HTTPS://NEUVECTOR.COM/CATEGORY/NETWORK-SECURITY/)

How Kubernetes Networking Works – Under the Hood (https://neuvector.com/network-

security/advanced-kubernetes-networking/)

How to Understand and Set Up Kubernetes Networking, Including Multiple

Networks
 

By Tobias Gurtzick

Kubernetes networking is a complex topic, if not even the most complicated topic. This post will give you insight on how kubernetes
actually creates networks and also how to setup a network for a kubernetes cluster yourself.
https://neuvector.com/network-security/advanced-kubernetes-networking/ 1/14
 This article doesn’t cover how to setup a kubernetes cluster
23/04/2019 itself, you could
How Kubernetes use minikube
Networking to quickly
Works - Under spin up a test cluster. All the examples
the Hood
in this post will use a rancher 2.0 cluster, but apply everywhere else too. Even if you are planning to use any of the new public cloud
managed kubernetes services such as EKS, AKS, GKE or IBM cloud, you should understand how kubernetes networking works.

For a basic introduction to kubernetes networking, please see this post How Kubernetes Networking Works – The Basics
(https://neuvector.com/network-security/kubernetes-networking/).

How to Utilize Kubernetes Networking

Many kubernetes (k8s) deployment guides provide instructions for deploying a kubernetes networking CNI as part of the k8s deployment.
But if your k8s cluster is already running, and no network is yet deployed, deploying the network is as simple as running their provided
con g le against k8s (for most networks and basic use cases). For example, to deploy annel:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

With this k8s from a networking perspective is ready to go. To test everything is working we create 2 pods.

cat <<EOF | kubectl create -f -

---
apiVersion: v1
kind: Pod
metadata:
name: pod1
spec:
containers:
- name: pod1
image: docker.io/centos/tools:latest
command:
- /sbin/init
---
apiVersion: v1
kind: Pod
metadata:
name: pod2
spec:
containers:
- name: pod2
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF

This will create two pods, which are already utilizing our driver. Looking at one of the containers we nd the network with the ip range
10.42.0.0/24 attached.

$: kubectl exec -it pod1 ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 2a:70:97:76:a0:48 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.2.33/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::2870:97ff:fe76:a048/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever

A quick ping test from the other pod shows us, that the network is working properly.
https://neuvector.com/network-security/advanced-kubernetes-networking/ 2/14
23/04/2019 How Kubernetes Networking Works - Under the Hood

$: kubectl exec -it pod2 ping 10.42.2.33

PING 10.42.2.33 (10.42.2.33) 56(84) bytes of data.
64 bytes from 10.42.2.33: icmp_seq=1 ttl=62 time=0.818 ms
--- 10.42.2.33 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.818/0.818/0.818/0.000 ms

How Does Kubernetes Networking Compared to Docker Networking?

Kubernetes manages networking through CNI’s on top of docker and just attaches devices to docker. While docker with docker swarm also
has its own networking capabilities, such as overlay, macvlan, bridging, etc, the CNI’s provide similar types of functions.

It’s also important to mention that k8s does not use docker0, which is docker’ default bridge, but creates its own bridge named cbr0, which
was chosen to differentiate from the docker0 bridge.

Why Do We Need Overlay Networks?

Overlay networks such as vxlan or ipsec (which you might be familiar with from setting up secure VPNs) encapsulate the packet into
another packet. This makes entities addressable which are outside of the scope of another machine. Alternatives to overlay networks
include L3 solutions such as macvtap(lan) or even L2 solutions such as ivtap(lan), but these come with limitations or even unwanted side
effects.

Any solution on L2 or L3 makes a pod addressable on the network. This means a pod is reachable not just within the docker network, but is
directly addressable from outside the docker network. These could be public or private ip addresses.

However, communication on L2 is cumbersome and your experience will vary depending on your network equipment. Some switches need
some time to register your MAC Address, before it actually gets reachable to the rest of the network. You could also run into trouble because
the neighbor (ARP) table of the other hosts in the system still runs on an outdated cache, and you always need to run with dhcp instead of
host-local to avoid con icting ips between hosts. The mac address and neighbor table problems are the reasons solutions such as ipvlan
exist. These do not register new mac addresses but route traf c over the existing one, but these also have their own issues.

The conclusion and my recommendation is that for most users the overlay network is the default solution and should be adequate. However
as soon as workloads get more advanced with more speci c requirements you will want to consider other solutions such as BGP and direct
routing instead of overlay networks.

How Does Kubernetes Networking Work Under the Hood?

The rst thing to understand in kubernetes is that a pod is not actually the equivalent to a container, but is a collection of containers. And all
these containers of the same collection share a network stack. Kubernetes manages that by setting up the network itself on the pause
container, which you will nd for every pod you create. All other pods attach to the network of the pause container which itself does nothing
but provide the network. Therefore, it is also possible for one container to talk to a service in a different container, which is in the same
de nition of the same pod, via localhost.

Apart from that local communication, the communication between pods looks pretty much the same as container to container
communication in docker networks.

Kubernetes Tra c Routing

There are two scenarios which I’ll go into more detail to explain how traf c gets routed between pods.

1. Routing traf c on the same host:

There are two scenarios where the traf c does not leave the host.This is either when the service called is running on the same node, or it is
in the same container collection within a single pod.
In the case of calling localhost:80 from container 1 in our rst pod and having the service running in the container 2, the traf c will pass the
network device and forward the packet to its destination. In this case the route the traf c travels is quite short.

It gets a bit longer when we communicate to a different pod. The traf c will pass over to cbr0, which next will notice that we communicate
on the same subnet and therefore directly forwards the traf c to its destination pod, as shown below.

https://neuvector.com/network-security/advanced-kubernetes-networking/ 3/14
23/04/2019 How Kubernetes Networking Works - Under the Hood

2. Routing traf c across hosts:

This gets a bit more complicated when we leave the node. cbr0 will now pass the traf c to the next node, whose con guration is managed by
the CNI. These are basically just routes of the subnets with the destination host as a gateway. The destination host can then just proceed on
its own cbr0 and forward the traf c to the destination pod, as shown below.

https://neuvector.com/network-security/advanced-kubernetes-networking/ 4/14
23/04/2019 How Kubernetes Networking Works - Under the Hood

What Exactly is a CNI?

A CNI, which is short for Container Networking Interface, is basically an external module with a well de ned interface that can be called by
kubernetes to execute actions to provide networking functionality.

You can nd the maintained reference plugins which include most of the important ones in the of cial repo of containernetworking here
https://github.com/containernetworking/plugins (https://github.com/containernetworking/plugins).

A CNI version 3.1 is basically not very complicated. It consists of 3 required functions, ADD, DEL and VERSION, which pretty much should do
what they sound like for managing the network. For a more detailed description of what each function is expected to return and gets passed,
you can read the spec at https://github.com/containernetworking/cni/blob/master/SPEC.md
(https://github.com/containernetworking/cni/blob/master/SPEC.md).

The Di erent CNI’s

To give you a bit of orientation we will look at some of the most popular CNI’s.

Flannel
Flannel is a simple network and is the easiest setup option for an overlay network. It’s capabilities include native networking but has
limitations when using it on multiple networks. Flannel is for most users, beneath Canal, the default network to choose, it is a simple option
to deploy, and even provides some native networking capabilities such as host gateways. Flannel has some limitations including lack of
support for network security policies and the capability to have multiple networks.

Calico
Calico takes a different approach than annel. It is technically not an overlay network, but rather a system to con gure routing between all
systems involved. To accomplish this, Calico leverages the BorderGatewayProtocol (BGP) which is used for the Internet in a process named
peering, were every peering party exchanges traf c and participates in the bgp network. The BGP protocol itself propagates routes under its
ASN, with the difference that these are private and there isn’t a need to register them with RIPE.

However, for some scenarios also Calico works with an overlay network, in this case IPINIP, which is always used when a node is sitting on a
different network, in order to enable the exchange of traf c between those two hosts.
https://neuvector.com/network-security/advanced-kubernetes-networking/ 5/14
 Canal
23/04/2019 How Kubernetes Networking Works - Under the Hood

Canal is is based on Flannel but with some Calico components such as felix, the host agent, which allows you to utilize network security
policies. These are normally missing in Flannel. So it basically extends Flannel with the addition of security policies.

Multus
Multus is a CNI that actually is not a network interface itself. It orchestrates multiple interfaces and without an actual network con gured,
pods couldn’t communicate with multus alone. So multus is an enabler for multi device and multi subnet networks. The graphic below
shows how this works, multus itself basically calls the real CNI instead of the kubelet and communicates back to the kubelet the results.

Kube-Router
Also worth mentioning is kube-router, which like Calico works with BGP and routing instead of an overlay network and like Calico utilizes
IPINIP where needed. It also utilizes ipvs for loadbalancing.

Setting up a Multi-Network k8s Cluster

In the cases when you need to utilize multiple networks, you’ll likely be required to use multus. While multus is quite mature and works
without too many issues, you should know that there are currently some limitations.

One of that limitations is that portmapping does not work, which is documented and tracked on the following issue on github
(https://github.com/intel/multus-cni/issues/29 (https://github.com/intel/multus-cni/issues/29)). This is going to be xed in the future, but
currently should you be in need of mapping ports, either nodePort con gs or hostPort con gs, you won’t be able to do that due to the bug
referenced.

Setting Up Multus
The rst thing we need to do is to setup multus itself. This is pretty much the con g from the multus repositories examples, but with some
important adjustments. See the link below to the sample.

The rst thing was to adjust the con g map. Because we plan to have a default network with Flannel, we de ne the con guration in the
delegates array of the multus con g. Some important settings here marked in red are “masterplugin”: true and to de ne the bridge for the
annel network itself. You’ll see why we need this in the next steps. Other than that there isn’t much else to adjust except adding the
mounting de nition of the con g map, because for some reason this was not done in their example.

Another important thing about this con g map is that everything de ned in this con g map are the default networks that are automatically
mounted to the containers without further speci cation. Also should you want to edit this le, please note you either need to kill and rerun
the containers of the daemonset or reboot your node to have the changes take effect.

You can view the sample yaml le HERE .

https://neuvector.com/network-security/advanced-kubernetes-networking/ 6/14
 Setting up the Primary Flannel Overlay
23/04/2019
Network
How Kubernetes Networking Works - Under the Hood

For the primary Flannel network, things are pretty much very easy. We can take the example from the multus repository for this and just
deploy it. The only adjustments that have been made here are the CNI mount, adjustment of tolerations and some adjustments made for the
CNI settings of Flannel. For example adding “forceAddress”:true and removing “hairpinMode”: true.
This was tested on a cluster that was setup with RKE, but should work on other clusters as well as long as you mount the CNIs from your
host correctly, in our case /opt/cni/bin.

The multus team themself did not really change much; they only commented out the initcontainer con g, which you could just safely
delete. This is made since multus will setup its delegates and will act as the primary “CNI.”

You can view the modi ed Flannel daemonset HERE .

With these samples deployed we are pretty much done and our pods should now be assigned an ip address. Let’s test it:

$: cat <<EOF | kubectl create -f -

---
apiVersion: v1
kind: Pod
metadata:
name: overlay1pod
spec:
containers:
- name: overlay1pod
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF

pod/overlay1pod created

$: kubectl exec overlay1pod ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 1e:92:1f:ac:b9:68 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.2.43/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::1c92:1fff:feac:b968/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever

As you can see we have successfully deployed a pod and were assigned the ip 10.42.2.43 on the eth0 interface, which is the default interface.
All extra interfaces will appear as netX, i.e. net1.

Setting up the Secondary Network

The secondary network needs a few more adjustments and these are all made on the assumption that you want to deploy vxlan. To actually
serve a secondary overlay we need to change the VXLAN Identi er “VIN”, which by default is set to 1, and which is now already taken by our
rst overlay network. So we can change this by con guring the network on our etcd server. We use the clusters own etcd, here marked in
green (and we assume that the job runs on the host running the etcd client) and mount in our keys, here marked in red, from the local host
which in our case are stored in the /etc/kubernetes/ssl folder.

You can view the entire sample yaml le HERE .

Next we can actually deploy the secondary network. The setup of this pretty much looks the same as the primary one, but with some key
differences. The most obvious is that we changed the subnet, but we also need to change a few other things.

https://neuvector.com/network-security/advanced-kubernetes-networking/ 7/14
 First of all we need to set a different dataDir, i.e. /var/lib/cni/
23/04/2019 annel2, and
How Kubernetes a different
Networking subnetFile,
Works - Under i.e.
the /run/
Hood annel/ annel2.env. This is needed
because they are otherwise occupied and already used by our primary network. Next we need to adjust the bridge because kbr0 is already
used by the primary Flannel overlay network.

The remaining con guration required includes changing it to actually target our etcd server which we con gured before. In the primary
network this was done by connecting to the k8s api directly, which is done via the “–kube-subnet-mgr” ag. But we can’t do that because we
also need to modify the pre x from which we want to read. You can see this below marked in orange and settings for our cluster etcd
connection in red. The last setting is to specify the subnet le again, marked in green in the sample. Last but not least we add a network
de nition. The rest of the sample is identically to our main networks con g.

See the sample con g le for the above steps HERE .

Once this is done we have our secondary network ready.

Assigning Extra Networks

Now that we have a secondary network ready we also need to assign this. To do this we also need to rst de ne a
NetworkAttachmentDe nition, which we can use afterwards to assign this network to the container. This is basically the dynamic
alternative to the con gmap we setup before when initializing multus. This way we can mount the networks we need on demand. In this
de nition we need to specify the network type, in our case annel and also necessary con gurations. This includes the before mentioned
subnetFile, dataDir and bridge name.

The last thing we need to decide for is the name for the network, we name ours annel.2.

$: cat <<EOF | kubectl create -f -

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: flannel.2
spec:
config: '{
"cniVersion": "0.3.0",
"type": "flannel",
"subnetFile": "/run/flannel/flannel2.env",
"dataDir": "/var/lib/cni/flannel2",
"delegate": {
"bridge": "kbr1"
}
}'
EOF

Now we’re nally ready to spawn our rst pod with our secondary network.

$: cat <<EOF | kubectl create -f -

---
apiVersion: v1
kind: Pod
metadata:
name: overlay2pod
annotations:
k8s.v1.cni.cncf.io/networks: '[
{ "name": "flannel.2" }
]'
spec:
containers:
- name: overlay2pod
image: docker.io/centos/tools:latest
command:
- /sbin/init
EOF

https://neuvector.com/network-security/advanced-kubernetes-networking/ 8/14
 This should create your new pod with your secondary network,
23/04/2019 and weNetworking
How Kubernetes should seeWorks
those attached asHood
- Under the additional network interfaces now.

$: kubectl ~/setup/test/config exec overlay2pod ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 66:c5:7f:01:40:c8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.0.90/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::64c5:7fff:fe01:40c8/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
5: net1@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 46:a6:d5:33:54:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.5.22.4/24 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::44a6:d5ff:fe33:5404/64 scope link
valid_lft forever preferred_lft forever

Success, we see the secondary network which was assigned 10.5.22.4 as its ip address.

Troubleshooting
Should this example not work for you, you will need to look in the logs of your kubelet.
One common issue is missing CNIs. In my rst tests I was missing the bridge CNI since this was not deployed by RKE, xing this is as easy
as downloading them from the containernetworking repo.

External Connectivity and Load Balancing

Now that we have a network up and running, the next thing we want is to make our app reachable and con gure them to be highly available
and scalable. While HA and scalability are not solely achieved by load balancing it is the key component we need to have in place.

Kubernetes has basically four concepts to make an app externally available.

Using Load Balancers

Ingress
An Ingress is basically a loadbalancer with Layer 7 capabilities, speci cally HTTP(s). The most commonly used implementation of an
ingress controller is the nginx ingress. But this highly depends on your needs and your use case and when in doubt you can default to
whichever solution you want to use. For example, trae k or HA Proxy for which ingress controllers already exists. See the trae k guide for
an example on how to setup a different ingress controller:
https://docs.trae k.io/user-guide/kubernetes/ (https://docs.trae k.io/user-guide/kubernetes/)

Con guring an ingress is quite easy. In the following example you see an example of a linked service. In blue you will nd the basic
con guration which in this example points to a service. In green you nd the con guration needed to link your SSL certi cate unless you do
not employ SSL (you will need this certi cate installed before though). And last but not least in brown you will nd an example to adjust
some of the detailed settings of the nginx ingress. The latter ones you can look up over here: https://github.com/kubernetes/ingress-
nginx/blob/master/docs/user-guide/nginx-con guration/annotations.md (https://github.com/kubernetes/ingress-
nginx/blob/master/docs/user-guide/nginx-con guration/annotations.md).

https://neuvector.com/network-security/advanced-kubernetes-networking/ 9/14
23/04/2019 How Kubernetes Networking Works - Under the Hood

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 12m
name: my-ingress
namespace: default
spec:
rules:
- host: my.domain.com
http:
paths:
- backend:
serviceName: api
servicePort: 5000
path: /api
- backend:
serviceName: websockets
servicePort: 5000
path: /ws
tls:
- hosts:
- my.domain.com

Layer 4 Load Balancer

The layer 4 load balancer, which is de ned in kubernetes with type: LoadBalancer, is a service provider dependent load balancing solution.
For an on premise machine this will be most probably HA Proxy or a routing solution. Cloud providers may use either their own solution,
have special hardware in place or resort to an HA Proxy or a routing solution as well. Should you manage a bare metal installation of a k8s
cluster, you might want to give https://metallb.universe.tf/ (https://metallb.universe.tf/) a look.
The big difference is that the layer 4 load balancer does not understand high level application protocols (layer 7) and is only capable of
forwarding traf c. Most of the load balancers on this level also support SSL termination. This typically needs to be con gured through
annotations and is not standardized. So look this up in the docs of your cloud provider accordingly.

Using {host,node}Ports
A {host,node}Port is basically the equivalent to docker -p port:port, especially the hostPort. The nodePort, unlike the hostPort, is available on
all nodes instead of only on the nodes running the pod. For nodePort kubernetes creates a clusterIP rst and then load balances traf c over
this port. The nodePort itself is just an iptable rule to forward traf c on the port to the clusterIP.
A nodePort is rarely used except in quick testing and only really needed in production if you want every node to expose the port, i.e. for
monitoring. Most of the time you will want to use a layer 4 load balancer instead. And hostPort is mostly only really used for testing
purposes or very rarely to stick a pod to a speci c node and publish under a speci c ip address pointing to this node.

To give you an example, a hostPort is de ned in the container spec, like the following:

spec:
containers:
- args:
- sleep
- infinity
image: ubuntu:xenial
imagePullPolicy: Always
name: test
ports:
- containerPort: 1111
hostPort: 1111
name: 1111tcp11110
protocol: TCP

However a nodePort is de ned as a service instead and not in the container spec. An example for a nodePort would look like this:

https://neuvector.com/network-security/advanced-kubernetes-networking/ 10/14
23/04/2019 How Kubernetes Networking Works - Under the Hood

apiVersion: v1
kind: Service
spec:
externalTrafficPolicy: Cluster
ports:
- name: http
port: 7777
protocol: TCP
targetPort: 80
selector:
app: MyApp
sessionAffinity: None
type: NodePort

What is a ClusterIp?
A clusterIP is an internally reachable IP for the kubernetes cluster and all services within it. This ip itself load balances traf c to all pods that
matches its selector rules. A clusterIP also automatically gets generated in a lot of cases, for example when specifying a type: LoadBalancer
service or setting up nodePort. The reason behind this is that all the load balancing happens through the clusterIP.

The clusterIP itself as a concept was created to solve the problem of multiple addressable hosts and the effective renewal of those. It is much
easier to have single IP that does not change than to refetch data all the time via service discoveries all the time for all natures of services.
Although there are times when it is appropriate to use service discovery instead if you want explicit control, for example in some
microservice environments.

Common Troubleshooting
If you use a public cloud environment and setup the hosts manually your cluster might be missing rewall rules. For example in AWS you
will want to adjust your security groups to allow inter-cluster communication as well as ingress and egress. Otherwise this will lead to an
inoperable cluster. Make sure you always open the required ports between master and worker nodes. The same goes for ports that you open
on the hosts directly, i.e. hostPort or nodePort.

Network Security
Now that we have setup all of our kubernetes networking, we also need to make sure that we have some security in place. A simple rule in
security is to give applications the least access they need. This ensures to a certain degree that even in the case of a security breach
attackers will have a hard time to dig deeper into your network. While it does not completely secures you, it makes it harder and more time
consuming. This is important because it gives you more time to react and prevent further damage and can often prevent the damage itself.
A prominent example is the combination of different exploits/vulnerabilities of different applications, which the attacker is only able to
pursue, if there is actually any attack surface reachable from multiple vectors (e.g. network, container, host).

[Download this Container Segmentation Guide to Learn Strategies to Protect Workloads]

 

The options here are either we utilize network policies, or we look to 3rd party security solutions for container network security. With
network policy we have a solid base to ensure that traf c only ows where it should ow, but this only works for a few CNI’s. They work for
example with Calico and Kube-router. Flannel does not support it but luckily as of today you can move to canal, which makes the network
policy feature from Calico usable by Flannel. For most other CNI’s there is no support and also no support planned.

But this is not the only issue. The problem is that a network policy rule is only a very simple rewall rule targeting a certain port. This means
you can’t apply any advanced settings. For example, you can’t block just a single container on demand should you detect something is
suspicious with this container. Further network rules do not understand the traf c, so you don’t have any visibility of the traf c owing and
you’re purely limited to create rules on the Layer 3 and 4 level. And lastly there is no detection of network based threats or attacks such as
DDoS, DNS, SQL injection and other damaging network attacks that can occur even over trusted ip addresses and ports.

This is where specialized container network security solutions like NeuVector (https://neuvector.com/run-time-container-security/) provide
the security needed for critical applications such as nancial or compliance driven ones. The NeuVector container rewall
(https://neuvector.com/docker-security/how-to-deploy-a-docker-container- rewall/) is a solution that I had experience with deploying at
Arvato/Bertelsmann and it provided the layer 7 visibility and protection we needed.

It should be noted that any network security solution must be cloud-native and auto-scaling and adapting. You can’t be checking on iptable
rules or having to update anything when you deploy new applications or scale your pods. Perhaps for a simple application stack on a couple
nodes you could manage this all manually, but for any enterprise deployment security can’t be slowing down the CI/CD pipeline.

https://neuvector.com/network-security/advanced-kubernetes-networking/ 11/14
 In addition to the security and visibility, I also found that
23/04/2019 Howhaving connection
Kubernetes and packet
Networking Workslevel container
- Under network tools helped debug
the Hood
applications during test and staging. With a kubernetes network you’re never really certain where all the packets are going and which pods
are being routed to unless you can see the traf c.

Some Advice on Choosing the Network CNI

Now that kubernetes networking and CNI’s have been covered, one big question always comes up. Which CNI solution should choose? I will
try to at least give you advice about how to go about making this decision.

First, De ne the Problem

The rst thing for every project is to de ne the problem you want to solve rst in as much detail as possible. You will want to know what
kind of applications you want to deploy and what kind of load they generate. Some of the questions you could ask yourself are:

Is my application:

Heavy on the network?

Sensitive to latency?

A monolith?

A microservice architected service?

On multiple networks?

Can I Withstand Major Downtime? Or Even Minor?

This is an important question because that you should decide up front, because if you choose one solution now and later you want to switch
you will need to re-setup the network and redeploy all your containers. Unless you already have something in place like multus and work
with multiple networks, this will mean a downtime for your service. Most of the time it will be ne if you have a planned maintenance
window, but as you grow, zero downtime becomes more important.

My Application is on Multiple Networks

This scenario is quite common in on-premise installations. In fact, if you only want to separate the traf c going over the private network
and the public network this will require you to either setup multiple networks or have clever routing.

Is There a Certain Feature You Need from the CNI’s?

Another thing in uencing your decision can be that you want some features of the CNI’s not available to other CNI’s. For example you want
to utilize weave or you want more mature loadbalancing through ipvs.

What Network Performance is Required?

If you answered that your application is sensitive to latency or heavy on the network you might want to avoid any overlay network. Overlays
can can be expensive on performance and even more so at scale. If this is the case the only way to improve performance on the network is
to avoid overlays and utilize networking utilities like routing instead. When you look for network performance you have a few choices, for
example:

Ipvlan
Ipvlan could be an option for you, it has a good performance, but has it caveats, i.e. you can’t use macv{tap,lan} at the same time on the same
host.

Calico
Calico is not the most user friendly CNI, but it provides you with much better performance compared to vxlan and can
be scaled without issues.
Kube-Router

Kube-router will give you better performance like Calico, since they both use BGP and routing, plus support for LVS/IPVS. But it might not be
as battle tested as Calico.

Cloud Provider Solutions

Last but not least, some of the cloud providers do provide own kubernetes networking solutions that may or may not perform better.
Amazon for example has their aws-vpc which is supported on annel. Aws-vpc performs in most scenarios about as good as ipvlan.

But I Just Want Something That Works!

https://neuvector.com/network-security/advanced-kubernetes-networking/ 12/14
 Yes, that is understandable, and this is the case for most
23/04/2019 Howusers. In this case
Kubernetes probably
Networking canal
Works or Flannel
- Under with vxlan will be the way to go
the Hood
because it is a no-brainer and just works. However as I said before vxlan is slow and it will cost you signi cantly more resources as you
grow. But this is de nitely the easiest way to start.

Just Make a Decision

It really is a matter of making a decision rather than making none at all. If you don’t have speci c features requirements it is ne to start
with Flannel and vxlan. It will involve some work to migrate later if you’ve deployed to production but it is better to make decision that is
wrong in the long term than making no decision at all.

With all this information, I hope that you will have some relevant background and a good understanding of

how kubernetes networking works.

To learn more about the attack surface of Kubernetes workloads and how to protect the CI/CD pipeline, Kubernetes system services, and
containers at run-time, Download this Ultimate Guide to Kubernetes Security

About the Author

Tobias Gurtzick
Tobias is a passionate Technology, OSS and DevOps Evangelist and is engineering and architecting solutions and landscapes at WizardTales for
various customers. Previously, Tobias worked at Arvato Bertelsmann, where he redesigned and rebuilt the entire infrastructure and development
process to be a plug and play microservices architecture for bare metal hardware. This included fully automatic provisioning utilizing PXE
infrastructure, Rancher, Kubernetes and ci/cd pipelines with integrated security.

He has been programming for more than a decade and is crazy about Security, Networks, and Linux. When not working on new projects and OSS,
he loves to play music.

By Tobias Gurtzick
Tags: Container Networking (https://neuvector.com/tag/container-networking/), Kubernetes (https://neuvector.com/tag/kubernetes/)
No Comments

Share
 (https://www.facebook.com/sharer/sharer.php?u=https://neuvector.com/network-security/advanced-kubernetes-networking/) 
(http://twitter.com/intent/tweet?text=How Kubernetes Networking Works – Under the Hood%20https://neuvector.com/network-security/advanced-
kubernetes-networking/)  (https://www.linkedin.com/shareArticle?mini=true&url=https://neuvector.com/network-security/advanced-kubernetes-
networking/&title=How Kubernetes Networking Works – Under the Hood)  (http://reddit.com/submit?url=https://neuvector.com/network-
security/advanced-kubernetes-networking/&title=How Kubernetes Networking Works – Under the Hood)  (http://www.tumblr.com/share/link?
url=https://neuvector.com/network-security/advanced-kubernetes-networking/&name=How Kubernetes Networking Works – Under the Hood)
(https://plus.google.com/share?url=https://neuvector.com/network-security/advanced-kubernetes-networking/)  (mailto:?&subject=How
Kubernetes Networking Works – Under the Hood&body=Check%20out%20this%20article%20https://neuvector.com/network-security/advanced-
kubernetes-networking/)

https://neuvector.com/network-security/advanced-kubernetes-networking/ 13/14
 23/04/2019 How Kubernetes Networking Works - Under the Hood

NeuVector, the leader in Container Network Security,

delivers highly integrated, automated security for
Kubernetes and OpenShift, and is the only next
generation container rewall with packet-level
interrogation and enforcement.

Try NeuVector
(https://neuvector.com/try-
neuvector)

FOLLOW US

 (https://www.facebook.com/NeuVector/)  (https://twitter.com/neuvector?lang=en)  (https://www.linkedin.com/company/neuvector/)

Copyright © 2019 NeuVector Inc. | All Rights Reserved

()
() ()

https://neuvector.com/network-security/advanced-kubernetes-networking/ 14/14