<p>I'm reading the <a href="http://kubernetes.io/v1.0/docs/getting-started-guides/scratch.html" rel="nofollow">Kubernetes "<em>Getting Started from Scratch</em>" Guide</a> and have arrived at the dreaded <a href="http://kubernetes.io/v1.0/docs/admin/networking.html#kubernetes-model" rel="nofollow"><strong>Network Section</strong></a>, where they state:</p>

<pre><code>Kubernetes imposes the following fundamental requirements on any networking implementation
(barring any intentional network segmentation policies):
* all containers can communicate with all other containers without NAT
* all nodes can communicate with all containers (and vice-versa) without NAT
* the IP that a container sees itself as is the same IP that others see it as
</code></pre>

<p>My first source of confusion is: <strong>How is this <em>different</em> than the "standard" Docker model?</strong> How is Docker different w.r.t. those 3 Kubernetes requirements?</p>

<p>The article then goes on to summarize how GCE achieves these requirements:</p>

<blockquote>
  <p>For the Google Compute Engine cluster configuration scripts, we use advanced routing to assign each VM a subnet (default is /24 - 254 IPs). Any traffic bound for that subnet will be routed directly to the VM by the GCE network fabric. This is in addition to the "main" IP address assigned to the VM, which is NAT'ed for outbound internet access. A linux bridge (called cbr0) is configured to exist on that subnet, and is passed to docker's --bridge flag.</p>
</blockquote>

<p>My question here is: <strong>Which requirement(s) from the 3 above does this paragraph address? More importantly, <em>how</em> does it achieve the requirement(s)?</strong> I guess I just don't understand how 1-subnet-per-VM accomplishes: container-container communication, node-container communication, and static IP.</p>

<hr>

<p>And, as a bonus/stretch concern: Why doesn't Marathon suffer from the same networking concerns as what Kubernetes is addressing here?</p>

<p>Docker's standard <a href="https://docs.docker.com/articles/networking/" rel="noreferrer">networking configuration</a> picks a container subnet for you out of its <a href="https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/drivers/bridge/setup_ipv4.go#L25" rel="noreferrer">chosen defaults</a>. As long as it doesn't conflict with any interfaces on your host, Docker is okay with it.</p>

<p>Then, Docker inserts an iptables MASQUERADE rule that allows containers to talk to the external world using the host's default interface.</p>

<p>Kubernetes' 3 requirements are violated by the fact that subnets are chosen only based on addresses in use on the host, which forces the requirement to NAT all container traffic using the MASQUERADE rule.</p>

<p>Consider the following 3-host Docker setup (a little contrived to highlight things):</p>

<h2>Host 1:</h2>

<p><strong>eth0</strong>: 10.1.2.3</p>

<p><strong>docker0</strong>: 172.17.42.1/16</p>

<p><strong>container-A</strong>: 172.17.42.2</p>

<h2>Host 2:</h2>

<p><strong>eth0</strong>: 10.1.2.4</p>

<p><strong>docker0</strong>: 172.17.42.1/16</p>

<p><strong>container-B</strong>: 172.17.42.2</p>

<h2>Host 3:</h2>

<p><strong>eth0</strong>: 172.17.42.2</p>

<p><strong>docker0</strong>: 172.18.42.1</p>

<p>Let's say <strong>container-B</strong> wants to access an HTTP service on port 80 of <strong>container-A</strong>. You can get docker to expose <strong>container-A</strong>'s port 80 somewhere on <strong>Host 1</strong>. Then <strong>container-B</strong> might make a request to 10.1.2.3:43210. This will be received on <strong>container-A</strong>'s port 80, but will look like it came from some random port on 10.1.2.4 because of the NAT on the way out of <strong>Host 2</strong>. This violates the <em>all containers communicate without NAT</em> and the <em>container sees same IP as others</em> requirements. Try to access <strong>container-A</strong>'s service directly from <strong>Host 2</strong> and you get your <em>nodes can communicate with containers without NAT</em> violation.</p>

<p>Now if either of those containers want to talk to <strong>Host 3</strong>, they're SOL (just a general argument for being careful with the auto-assigned docker0 subnets).</p>

<p>Kubernetes approach on GCE/AWS/Flannel/... is to <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go#L364" rel="noreferrer">assign</a> each host VM a subnet carved out of a flat private network. No subnets overlap with VM addresses or with each other. This lets containers and VMs communicate NATlessly.</p>
