<p>Sometimes pod creation fails with the 500 error on our GKE cluster:</p>

<pre><code>1m        1m        1         installer-u57ab1f7707b03   Pod                 Normal    Scheduled    {default-scheduler }                                       Successfully assigned installer-u57ab1f7707b03 to gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l
1m        1m        1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container ff8573fbf0b90a25b5565b1feb36671f13367115dde74e581cf249be772d8e4e: [8] System error: read parent: connection reset by peer\n"
1m        1m        1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container fbd7151d4489ed3ac9b21ef9ee3268039374fe3aee1f5933dc27d003f5388e7d: [8] System error: read parent: connection reset by peer\n"
1m        1m        1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container c6b7969fd036fd187f8b5b815106887d718780b290b81e6dde12162d15c22728: [8] System error: read parent: connection reset by peer\n"
49s       49s       1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container 5b0d78ee31759a3472f15fe375ef4f2542dcc65518023a1bd06593fe7d28a448: [8] System error: read parent: connection reset by peer\n"
32s       32s       1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container 7ff5941a30ce432aa1b1382e4b20d272a08a7113f79f7f1ff2f8898a00ca8f06: [8] System error: read parent: connection reset by peer\n"
18s       18s       1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container a91ae7d6dc9dee5196e73457d817bc46f8009c26147cc81727920aebfa52cc38: [8] System error: read parent: connection reset by peer\n"
2s        2s        1         installer-u57ab1f7707b03   Pod                 Warning   FailedSync   {kubelet gke-oro-cloud-v1-1445426963-ffbcc283-node-bo1l}   Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container ad8b7bbe72410232d7fe6197e057d15e9003e24f6d8aad15bc7068430cfea508: [8] System error: read parent: connection reset by peer\n"
</code></pre>

<p>In docker.log I found:</p>

<pre><code>time="2016-08-10T12:37:24.458097892Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/ad8b7bbe72410232d7fe6197e057d15e9003e24f6d8aad15bc7068430cfea508/shm: invalid argument\nfailed to umount /var/lib/docker/containers/ad8b7bbe72410232d7fe6197e057d15e9003e24f6d8aad15bc7068430cfea508/mqueue: invalid argument"
time="2016-08-10T12:37:24.458280187Z" level=error msg="Handler for POST /containers/ad8b7bbe72410232d7fe6197e057d15e9003e24f6d8aad15bc7068430cfea508/start returned error: Cannot start container ad8b7bbe72410232d7fe6197e057d15e9003e24f6d8aad15bc7068430cfea508: [8] System error: read parent: connection reset by peer"
time="2016-08-10T12:37:24.458315257Z" level=error msg="HTTP Error" err="Cannot start container ad8b7bbe72410232d7fe6197e057d15e9003e24f6d8aad15bc7068430cfea508: [8] System error: read parent: connection reset by peer" statusCode=500
time="2016-08-10T12:37:40.151776337Z" level=warning msg="signal: killed" 
</code></pre>

<p>Kubernetes version v1.2.5<br>
Docker version 1.9.1</p>

<p>Any ideas how to fix it?</p>

<p>This is probably due to the <a href="https://github.com/docker/docker/issues/14203" rel="nofollow">runc bug</a> in Docker 1.9 where the container reads its config, but closes the read pipe before the the parent is done writing.</p>

<p>A fixed runc is included in Docker 1.10. Kubernetes 1.3 uses Docker 1.11.2, but until you upgrade, you may be able to work around the issue by <a href="https://github.com/docker/docker/issues/14203#issuecomment-204976854" rel="nofollow">adding extra characters</a> to your container's command line.</p>
