<p>Assume that there are some pods from Deployments/StatefulSet/DaemonSet, etc. running on a Kubernetes node.</p>
<p>Then I restarted the node directly, and then start docker, start kubelet with the same parameters.</p>
<p>What would happen to those pods?</p>
<ol>
<li>Are they recreated with metadata saved locally from kubelet? Or use info retrieved from api-server? Or recovered from OCI runtime and behaves like nothing happened?</li>
<li>Is it that only stateless pod(no --local-data) can be recovered normally? If any of them has a local PV/dir, would they be connected back normally?</li>
<li>What if I did not restart the node for a long time? Would api-server assign other nodes to create those pods? What is the default timeout value? How can I configure this?</li>
</ol>
<p>As far as I know:</p>
<pre><code> apiserver
    ^
    |(sync)
    V
  kubelet
    ^
    |(sync)
    V
-------------
| CRI plugin |(like api)
| containerd |(like api-server)
|    runc    |(low-level binary which manages container)
| c' runtime |(container runtime where containers run)
-------------
</code></pre>
<p>When kubelet received a PodSpec from kube-api-server, it calls CRI like a remote service, the steps be like:</p>
<ol>
<li>create PodSandbox(a.k.a 'pause' image, always 'stopped')</li>
<li>create container(s)</li>
<li>run container(s)</li>
</ol>
<p>So I <strong>guess</strong> that as the node and docker being restarted, steps 1 and 2 are already done, containers are at 'stopped' status; Then as kubelet being restarted, it pulls latest info from kube-api-server, found out that container(s) are not in 'running' state, so it calls CRI to run container(s), then everything are back to normal.</p>
<p>Please help me confirm.</p>
<p>Thank you in advance~</p>

<p>Good questions. A few things first; a Pod is not pinned to a certain node. The nodes is mostly seen as a &quot;server farm&quot; that Kubernetes can use to run its workload. E.g. you give Kubernetes a set of nodes and you also give a set of e.g. <code>Deployment</code> - that is desired state of applications that should run on your servers. Kubernetes is responsible for scheduling these Pods and also keep them running when something in the cluster is changed.</p>
<p>Standalone pods is not managed by anything, so if a Pod crashes it is not recovered. You typically want to deploy your stateless apps as <code>Deployments</code>, that then initiates <code>ReplicaSets</code> that manage a set of Pods - e.g. 4 Pods - instances of your app.</p>
<p>Your desired state; a <code>Deployment</code> with e.g. <code>replicas: 4</code> is saved in the <strong>etcd</strong> database within the Kubernetes control plane.</p>
<p>Then a set of controllers for <code>Deployment</code> and <code>ReplicaSet</code> is responsible for keeping 4 replicas of your app alive. E.g. if a node becomes unresponsible (or dies), new pods will be created on other Nodes, if they are managed by the controllers for <code>ReplicaSet</code>.</p>
<p>A <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="nofollow noreferrer">Kubelet</a> receives a PodSpecs that are scheduled to the node, and then keep these pods alive by regularly health checks.</p>
<blockquote>
<p>Is it that only stateless pod(no --local-data) can be recovered normally?</p>
</blockquote>
<p>Pods should be seen as emphemeral - e.g. can disappear - but is recovered by a controller that manages them - unless deployed as standalone Pod. So don't store local data within the pod.</p>
<p>There is also <code>StatefulSet</code> pods, those are meant for <em>stateful</em> workload - but <em>distributed stateful workload</em>, typically e.g. 3 pods, that use <a href="https://raft.github.io/" rel="nofollow noreferrer">Raft</a> to replicate data. The etcd database is an example of distributed database that uses Raft.</p>
