<p>I'm in learning mode here, so please forgive me if this is a stupid question...</p>
<p>I have just installed microk8s on ubuntu following the instructions at <a href="https://ubuntu.com/tutorials/install-a-local-kubernetes-with-microk8s" rel="nofollow noreferrer">https://ubuntu.com/tutorials/install-a-local-kubernetes-with-microk8s</a></p>
<p>Everything works. The &quot;microbot&quot; application gets deployed and exposed and creates a simple web server.  But what surprised me is that after I stop microk8s (with &quot;microk8s stop&quot;), the web server is still apparently up and running.  It continues to respond to curl with its simple page content.</p>
<p>Is this expected behavior?  Do the pods continue to run after the orchestrator has stopped?</p>
<p>Also, I was trying to figure out what microk8s is doing with the network.  It fires up its dashboard on 10.152.183.203, but when I look at the interfaces and routing tables on my host, I can't figure out how traffic is being routed to that destination.  And if I run tcpdump I can't seem to capture any of the traffic being sent to that address.</p>
<p>Any explanation of what's going on here would be much appreciated!</p>
<ul>
<li>Duncan</li>
</ul>

Duncan
<blockquote>
<p>But what surprised me is that after I stop microk8s (with &quot;microk8s stop&quot;), the web server is still apparently up and running. It continues to respond to curl with its simple page content.</p>
<p>Is this expected behavior? Do the pods continue to run after the orchestrator has stopped?</p>
</blockquote>
<p>That's not expected behavior, and I can't reproduce it. What I see is that the service is available for a period of several seconds after running <code>microk8s stop</code>, but eventually everything gets shut down.</p>
<blockquote>
<p>Also, I was trying to figure out what microk8s is doing with the network. It fires up its dashboard on 10.152.183.203, but when I look at the interfaces and routing tables on my host, I can't figure out how traffic is being routed to that destination.</p>
</blockquote>
<p>I've deployed Microk8s locally, and the dashboard Service looks like this:</p>
<pre><code>root@ubuntu:~# kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes-dashboard   ClusterIP   10.152.183.151   &lt;none&gt;        443/TCP   14m
</code></pre>
<p>As you note in your question, I can access this at <code>http://10.152.183.151</code>, but there are no local interfaces on that network:</p>
<pre><code>root@ubuntu:~# ip addr |grep 10.152
&lt;no results&gt;
</code></pre>
<p>And there are no meaningful routes to that network. E.g., this shows that access to that ip would be via the default gateway, which doesn't make any sense:</p>
<pre><code>root@ubuntu:~# ip route get 10.152.183.151
10.152.183.151 via 192.168.122.1 dev enp1s0 src 192.168.122.72 uid 0
    cache
</code></pre>
<p>What's going on? It turns out that microk8s sets up a bunch of NAT rules in your local firewall configuration. If we look for the dashboard address in the NAT table, we find:</p>
<pre><code>root@ubuntu:~# iptables-legacy -t nat -S | grep 10.152.183.151
-A KUBE-SERVICES -d 10.152.183.151/32 -p tcp -m comment --comment &quot;kube-system/kubernetes-dashboard cluster IP&quot; -m tcp --dport 443 -j KUBE-SVC-4HQ2X6RJ753IMQ2F
-A KUBE-SVC-4HQ2X6RJ753IMQ2F ! -s 10.1.0.0/16 -d 10.152.183.151/32 -p tcp -m comment --comment &quot;kube-system/kubernetes-dashboard cluster IP&quot; -m tcp --dport 443 -j KUBE-MARK-MASQ
</code></pre>
<p>If we follow the chain, we find that:</p>
<ol>
<li><p>Packets going to <code>10.152.183.151</code> enter the NAT <code>PREROUTING</code> chain, which sends them to the <code>KUBE-SERVICES</code> chain.</p>
</li>
<li><p>In the <code>KUBE-SERVICES</code> chain, packets to the dashboard (for tcp port 443) are sent to the <code>KUBE-SVC-4HQ2X6RJ753IMQ2F</code> chain.</p>
</li>
<li><p>In the <code>KUBE-SVC-4HQ2X6RJ753IMQ2F</code>, packets are first sent to the <code>KUBE-MARK-MASQ</code> chain, which sets a mark on the packet (which is used elsewhere in the configuration), and then gets sent to the <code>KUBE-SEP-SZAWMA3BPGJYVHOD</code> chain:</p>
<pre><code>root@ubuntu:~# iptables-legacy -t nat -S KUBE-SVC-4HQ2X6RJ753IMQ2F
-A KUBE-SVC-4HQ2X6RJ753IMQ2F ! -s 10.1.0.0/16 -d 10.152.183.151/32 -p tcp -m comment --comment &quot;kube-system/kubernetes-dashboard cluster IP&quot; -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SVC-4HQ2X6RJ753IMQ2F -m comment --comment &quot;kube-system/kubernetes-dashboard -&gt; 10.1.243.198:8443&quot; -j KUBE-SEP-SZAWMA3BPGJYVHOD
</code></pre>
</li>
<li><p>In the <code>KUBE-SEP-SZAWMA3BPGJYVHOD</code> chain, the packets finally hit a <code>DNAT</code> rule that maps the connection to the IP of the pod:</p>
<pre><code>root@ubuntu:~# iptables-legacy -t nat -S KUBE-SEP-SZAWMA3BPGJYVHOD
-N KUBE-SEP-SZAWMA3BPGJYVHOD
-A KUBE-SEP-SZAWMA3BPGJYVHOD -s 10.1.243.198/32 -m comment --comment &quot;kube-system/kubernetes-dashboard&quot; -j KUBE-MARK-MASQ
-A KUBE-SEP-SZAWMA3BPGJYVHOD -p tcp -m comment --comment &quot;kube-system/kubernetes-dashboard&quot; -m tcp -j DNAT --to-destination 10.1.243.198:8443
</code></pre>
<p>We know that <code>10.1.243.198</code> is the Pod IP because we can see it like this:</p>
<pre><code>kubectl -n kube-system get pod kubernetes-dashboard-74b66d7f9c-plj8f -o jsonpath='{.status.podIP}'
</code></pre>
</li>
</ol>
<p>So, we can reach the dashboard at <code>10.152.183.151</code> because the <code>PREROUTING</code> chain ultimately hits a <code>DNAT</code> rule that maps the &quot;clusterip&quot; of the service to the pod ip.</p>
<blockquote>
<p>And if I run tcpdump I can't seem to capture any of the traffic being sent to that address.</p>
</blockquote>
<p>Based on the above discussion, if we use the pod ip instead, we'll see the traffic we expect. The following shows the result of me running <code>curl -k https://10.152.183.151</code> in another window:</p>
<pre><code>root@ubuntu:~# tcpdump -n -i any -c10 host 10.1.243.198
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
19:20:33.441481 cali7ef1137a66d Out IP 192.168.122.72.33034 &gt; 10.1.243.198.8443: Flags [S], seq 228813760, win 64240, options [mss 1460,sackOK,TS val 3747829344 ecr 0,nop,wscale 7], length 0
19:20:33.441494 cali7ef1137a66d In  IP 10.1.243.198.8443 &gt; 192.168.122.72.33034: Flags [S.], seq 3905988324, ack 228813761, win 65160, options [mss 1460,sackOK,TS val 1344719721 ecr 3747829344,nop,wscale 7], length 0
19:20:33.441506 cali7ef1137a66d Out IP 192.168.122.72.33034 &gt; 10.1.243.198.8443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3747829344 ecr 1344719721], length 0
19:20:33.442754 cali7ef1137a66d Out IP 192.168.122.72.33034 &gt; 10.1.243.198.8443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3747829345 ecr 1344719721], length 517
19:20:33.442763 cali7ef1137a66d In  IP 10.1.243.198.8443 &gt; 192.168.122.72.33034: Flags [.], ack 518, win 506, options [nop,nop,TS val 1344719722 ecr 3747829345], length 0
19:20:33.443004 cali7ef1137a66d In  IP 10.1.243.198.8443 &gt; 192.168.122.72.33034: Flags [P.], seq 1:772, ack 518, win 506, options [nop,nop,TS val 1344719722 ecr 3747829345], length 771
19:20:33.443017 cali7ef1137a66d Out IP 192.168.122.72.33034 &gt; 10.1.243.198.8443: Flags [.], ack 772, win 501, options [nop,nop,TS val 3747829345 ecr 1344719722], length 0
19:20:33.443677 cali7ef1137a66d Out IP 192.168.122.72.33034 &gt; 10.1.243.198.8443: Flags [P.], seq 518:582, ack 772, win 501, options [nop,nop,TS val 3747829346 ecr 1344719722], length 64
19:20:33.443680 cali7ef1137a66d In  IP 10.1.243.198.8443 &gt; 192.168.122.72.33034: Flags [.], ack 582, win 506, options [nop,nop,TS val 1344719723 ecr 3747829346], length 0
19:20:33.443749 cali7ef1137a66d In  IP 10.1.243.198.8443 &gt; 192.168.122.72.33034: Flags [P.], seq 772:827, ack 582, win 506, options [nop,nop,TS val 1344719723 ecr 3747829346], length 55
10 packets captured
38 packets received by filter
0 packets dropped by kernel
</code></pre>
