<p>I am very new to kubernetes and have just got a stock kubernetes v.1.3.5 cluster up on AWS using kube-up. So far, I have been playing around with kubernetes in understanding it's mechanics (nodes, pods, svc and stuff). Based on my initial (or maybe crude) understanding , I had few questions:</p>

<p>1) How does routing to cluster IP work here (i.e in kube-aws) ? I see that the services have IPs in the range 10.0.0.0/16. I did a deployment with rc=3 of stock nginx and then attached a service to it with Node Port exposed. All works great! I can connect to the service from my dev machine. This nginx service has a cluster IP of 10.0.33.71:1321. Now, if I ssh into one of the minions(or nodes or VMS) and do a "telnet 10.0.33.71 1321", it connects as expected. But I am clueless how this works, I couldn't find any routes related to 10.0.0.0/16 in the VPC setup by kubernetes. What exactly happens under the hood here that results in a successful connection for app like telnet? However, If I ssh into the master node and do "telnet 10.0.33.71 1321", it does not connect. Why does it fail to connect from master?</p>

<p>2) There is a cbr0 interface inside each node. Each minion node has cbr0 configured as 10.244.x.0/24 and master has cbr0 as 10.246.0.0/24.
I can ping to any of the 10.244.x.x pods from any of the nodes(including master). But I am not able to ping 10.246.0.1 (cbr0 inside master node) from any of the minion nodes. What could be happening here? </p>

<p>Here's the routes set up by kubernetes in aws. VPC.</p>

<pre><code>Destination     Target
172.20.0.0/16   local
0.0.0.0/0       igw-&lt;hex value&gt;
10.244.0.0/24   eni-&lt;hex value&gt; / i-&lt;hex value&gt;
10.244.1.0/24   eni-&lt;hex value&gt; / i-&lt;hex value&gt;
10.244.2.0/24   eni-&lt;hex value&gt; / i-&lt;hex value&gt;
10.244.3.0/24   eni-&lt;hex value&gt; / i-&lt;hex value&gt;
10.244.4.0/24   eni-&lt;hex value&gt; / i-&lt;hex value&gt;
10.246.0.0/24   eni-&lt;hex value&gt; / i-&lt;hex value&gt;
</code></pre>

<p><strong><a href="http://www.markbetz.net/" rel="noreferrer">Mark Betz</a></strong> (<a href="https://twitter.com/markbetz" rel="noreferrer">SRE at Olark</a>) presents Kubernetes networking in three articles:</p>

<ul>
<li><strong><a href="https://medium.com/google-cloud/understanding-kubernetes-networking-pods-7117dd28727" rel="noreferrer">pods</a></strong></li>
<li><strong><a href="https://medium.com/google-cloud/understanding-kubernetes-networking-services-f0cb48e4cc82" rel="noreferrer">services</a></strong>: </li>
<li><strong><a href="https://medium.com/google-cloud/understanding-kubernetes-networking-ingress-1bc341c84078" rel="noreferrer">ingress</a></strong></li>
</ul>

<p>For a pod, you are looking at:</p>

<p><a href="https://i.stack.imgur.com/L88G7.png" rel="noreferrer"><img src="https://i.stack.imgur.com/L88G7.png" alt="pod network"></a></p>

<p>You find:</p>

<ul>
<li><strong>etho0</strong>: a "physical network interface"</li>
<li><strong>docker0</strong>/<strong>cbr0</strong>: a <a href="https://wiki.linuxfoundation.org/networking/bridge" rel="noreferrer">bridge</a> for connecting two <a href="https://en.wikipedia.org/wiki/Ethernet" rel="noreferrer">ethernet</a> segments no matter their protocol.</li>
<li><code>veth0</code>, <code>1</code>, <code>2</code>: Virtual Network Interface, one per container.<br>
<strong>docker0</strong> is the <a href="https://en.wikipedia.org/wiki/Default_gateway" rel="noreferrer">default Gateway</a> of <strong>veth0</strong>. It is named <strong>cbr0</strong> for "custom bridge".<br>
Kubernetes starts containers by sharing the <a href="https://docs.docker.com/engine/reference/run/#network-settings" rel="noreferrer">same <strong>veth0</strong></a>, which means each container must expose different ports.</li>
<li><strong>pause</strong>: a special container started in "<code>pause</code>", to detect SIGTERM sent to a pod, and forward it to the containers.</li>
<li><strong>node</strong>: an host</li>
<li><strong>cluster</strong>: a group of nodes</li>
<li><strong><a href="https://en.wikipedia.org/wiki/Routing_table" rel="noreferrer">router/gateway</a></strong> </li>
</ul>

<p>The last element is where things start to be more complex:</p>

<blockquote>
  <p>Kubernetes assigns an overall address space for the bridges on each node, and then assigns the bridges addresses within that space, based on the node the bridge is built on.<br>
  Secondly, it adds routing rules to the gateway at 10.100.0.1 telling it how packets destined for each bridge should be routed, i.e. which node’s <code>eth0</code> the bridge can be reached through. </p>
  
  <p>Such a combination of virtual network interfaces, bridges, and routing rules is usually called an <strong><a href="https://en.wikipedia.org/wiki/Overlay_network" rel="noreferrer">overlay network</a></strong>. </p>
</blockquote>

<p>When a pod contacts another pod, it goes through a <strong><a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="noreferrer">service</a></strong>.<br>
Why?</p>

<blockquote>
  <p>Pod networking in a cluster is neat stuff, but by itself it is insufficient to enable the creation of durable systems. That’s because <strong>pods in Kubernetes are ephemeral</strong>.<br>
  You can use a pod IP address as an endpoint but <strong>there is no guarantee that the address won’t change the next time the pod is recreated</strong>, which might happen for any number of reasons.</p>
</blockquote>

<p>That means: you need a reverse-proxy/dynamic load-balancer. And it better be resilient.</p>

<blockquote>
  <p><strong>A service is a type of kubernetes resource that causes a proxy to be configured to forward requests to a set of pods</strong>.<br>
  The set of pods that will receive traffic is determined by the selector, which matches labels assigned to the pods when they were created</p>
</blockquote>

<p>That service uses its own network. By default, its type is "<strong>ClusterIP</strong>"; it has its own IP.</p>

<p>Here is the communication path between two pods:</p>

<p><a href="https://i.stack.imgur.com/h0F5B.png" rel="noreferrer"><img src="https://i.stack.imgur.com/h0F5B.png" alt="two pods network"></a></p>

<p>It uses a <strong><a href="https://kubernetes.io/docs/reference/generated/kube-proxy/" rel="noreferrer">kube-proxy</a></strong>.<br>
This proxy uses itself a <strong><a href="http://www.netfilter.org/" rel="noreferrer">netfilter</a></strong>.</p>

<blockquote>
  <p><strong>netfilter is a rules-based packet processing engine</strong>.<br>
  It runs in kernel space and gets a look at every packet at various points in its life cycle.<br>
  It matches packets against rules and when it finds a rule that matches it takes the specified action.<br>
  Among the many actions it can take is redirecting the packet to another destination. </p>
</blockquote>

<p><a href="https://i.stack.imgur.com/0nHJP.png" rel="noreferrer"><img src="https://i.stack.imgur.com/0nHJP.png" alt="kube-proxy and netfilter"></a></p>

<blockquote>
  <p>In this mode, kube-proxy:</p>
  
  <ul>
  <li><strong>opens a port</strong> (10400 in the example above) on the local host interface to listen for requests to the test-service,</li>
  <li><strong>inserts netfilter rules to reroute packets destined for the service IP to its own port</strong>, and</li>
  <li><strong>forwards those requests</strong> to a pod on port 8080.</li>
  </ul>
  
  <p>That is how a request to <code>10.3.241.152:80</code> magically becomes a request to <code>10.0.2.2:8080</code>.<br>
  Given the capabilities of netfilter all that’s required to make this all work for any service is for <strong>kube-proxy to open a port and insert the correct netfilter rules for that service</strong>, which it does in response to notifications from the master api server of changes in the cluster.</p>
</blockquote>

<p>But:</p>

<blockquote>
  <p>There’s one more little twist to the tale.<br>
  I mentioned above that <strong>user space proxying is expensive due to marshaling packets</strong>. 
  In kubernetes 1.2, <strong>kube-proxy gained the ability to run in iptables mode</strong>. </p>
  
  <p>In this mode, kube-proxy mostly ceases to be a proxy for inter-cluster connections, and instead delegates to netfilter the work of detecting packets bound for service IPs and redirecting them to pods, all of which happens in kernel space.<br>
  In this mode kube-proxy’s job is more or less limited to keeping netfilter rules in sync.</p>
</blockquote>

<p>The network schema becomes:</p>

<p><a href="https://i.stack.imgur.com/aGxtU.png" rel="noreferrer"><img src="https://i.stack.imgur.com/aGxtU.png" alt="netfilter in action"></a></p>

<p>However, this is not a good fit for <em>external</em> (public facing) communication, which needs an external fixed IP.</p>

<p>You have dedicated services for that: <a href="https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport" rel="noreferrer"><strong>nodePort</strong> and <strong>LoadBalancer</strong></a>: </p>

<blockquote>
  <p>A service of type <strong>NodePort</strong> is a ClusterIP service with an additional capability: it is reachable at the IP address of the node as well as at the assigned cluster IP on the services network.<br>
  The way this is accomplished is pretty straightforward:</p>
  
  <p>When kubernetes creates a NodePort service, <strong>kube-proxy allocates a port in the range 30000–32767 and opens this port on the <code>eth0</code> interface of every node (thus the name “NodePort”)</strong>.</p>
  
  <p>Connections to this port are forwarded to the service’s cluster IP.</p>
</blockquote>

<p>You get:</p>

<p><a href="https://i.stack.imgur.com/APpSf.png" rel="noreferrer"><img src="https://i.stack.imgur.com/APpSf.png" alt="load-balancer / nodeport"></a></p>

<p>A Loadalancer is more advancer, and allows to expose services using stand ports.<br>
See the mapping here:</p>

<pre><code>$ kubectl get svc service-test
NAME      CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
openvpn   10.3.241.52     35.184.97.156   80:32213/TCP     5m
</code></pre>

<p>However:</p>

<blockquote>
  <p>Services of type LoadBalancer have some limitations. </p>
  
  <ul>
  <li>You cannot configure the lb to terminate https traffic.</li>
  <li>You can’t do virtual hosts or path-based routing, <strong>so you can’t use a single load balancer to proxy to multiple services in any practically useful way</strong>. </li>
  </ul>
  
  <p>These limitations led to the addition in version 1.2 of a separate kubernetes resource for configuring load balancers, called an <strong><a href="https://kubernetes.io/docs/concepts/services-networking/ingress/" rel="noreferrer">Ingress</a></strong>.</p>
  
  <p><strong>The Ingress API supports TLS termination, virtual hosts, and path-based routing</strong>. It can easily set up a load balancer to handle multiple backend services.<br>
  The implementation follows a basic kubernetes pattern: a resource type and a controller to manage that type.<br>
  The resource in this case is an Ingress, which comprises a request for networking resources</p>
</blockquote>

<p>For instance:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
    - secretName: my-ssl-secret
  rules:
  - host: testhost.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: service-test
          servicePort: 80
</code></pre>

<blockquote>
  <p><strong>The ingress controller is responsible for satisfying this request by driving resources in the environment to the necessary state.<br>
  When using an Ingress you create your services as type NodePort and let the ingress controller figure out how to get traffic to the nodes.</strong> </p>
  
  <p>There are ingress controller implementations for GCE load balancers, AWS elastic load balancers, and for popular proxies such as NGiNX and HAproxy. </p>
</blockquote>
