<p>Pod controlled by StatefulSet is stuck in <code>ContainerCreating</code> state</p>

<p>kubectl get pods</p>

<pre><code>md-0                    1/1       Running             0          4h        10.242.208.59   node-5
md-1                    1/1       Running             0          4h        10.242.160.36   node-6
md-2                    0/1       ContainerCreating   0          4h        &lt;none&gt;          node-6
</code></pre>

<p>kubectl describe pod md-2</p>

<pre><code>Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
...
Events:
  Type     Reason                  Age               From                                 Message
  ----     ------                  ----              ----                                 -------
  Warning  FailedCreatePodSandBox  2m (x68 over 4h)  kubelet, node-6  Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
</code></pre>

<p>kubectl describe statefulset md</p>

<pre><code>Replicas:           3 desired | 3 total
Pods Status:        2 Running / 1 Waiting / 0 Succeeded / 0 Failed
...
Events:                         &lt;none&gt;
</code></pre>

<p>kubelet log from node-6</p>

<pre><code>RunPodSandbox from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
CreatePodSandbox for pod "md-2_exc(a995dd3d-158d-11e9-967b-6cb311235088)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
createPodSandbox for pod "md-2_exc(a995dd3d-158d-11e9-967b-6cb311235088)" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Error syncing pod a995dd3d-158d-11e9-967b-6cb311235088 ("md-2_exc(a995dd3d-158d-11e9-967b-6cb311235088)"), skipping: failed to "CreatePodSandbox" for "md-2_exc(a995dd3d-158d-11e9-967b-6cb311235088)" with CreatePodSandboxError: "CreatePodSandbox for pod \"md-2_exc(a995dd3d-158d-11e9-967b-6cb311235088)\" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded"
</code></pre>

<p>I have two other StatefulSets and they works as expected. For some reason this one is broken.
Also direct <code>kubectl run</code> and <code>docker run</code> works fine.</p>

<p><strong>update 2019-01-18</strong></p>

<p>After restoration of change's timeline I see that this specific pod was deleted with docker command bypassing kubernetes.</p>

<p>Probably this somehow corrupted kubernetes' state or something.</p>

<p>After numerous searching, asking and troubleshooting I still could not find what's exactly wrong. So I had to restart kubelet (<code>systemctl restart kubelet</code>) on node where pod was assigned. And the issue is gone.</p>

<p>I hoped to understand how to check what exactly wrong with kubernetes (or kubelet?) but could not find any clues. And kubelet behavior remains black box in this case.</p>

<p>As <a href="https://stackoverflow.com/users/10901442/alexar">alexar</a> mentioned in update:</p>

<p>After restoration of change's timeline I see that this specific pod was deleted with docker command bypassing kubernetes.</p>

<p>Probably this somehow corrupted kubernetes' state or something.</p>

<p>After numerous searching, asking and troubleshooting I still could not find what's exactly wrong. So I had to restart kubelet (systemctl restart kubelet) on node where pod was assigned. And the issue is gone.</p>
