<h1>Initial Post</h1>
<p>I have the same docker image running on two different CoreOS servers. (They're in a Kubernetes cluster, but I think that is irrelevant to the current problem.)</p>
<p>They both are running image hash <code>01e95e0a93af</code>. They both should have curl. One does not. This seems... impossible.</p>
<p><em>Good Server</em></p>
<pre><code>core@ip-10-0-0-61 ~ $ docker pull gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a
Digest: sha256:5d8bf456ad2d08ce3cd15f05b62fddc07fda3955267ee0d3ef73ee1a96b98e68
[cut]
Status: Image is up to date for gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a

core@ip-10-0-0-61 ~ $ docker run -it --rm gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a /bin/bash
root@d29cb8783830:/app/bundle# curl
curl: try 'curl --help' or 'curl --manual' for more information
root@d29cb8783830:/app/bundle# 
</code></pre>
<p><em>Bad Server</em></p>
<pre><code>core@ip-10-0-0-212 ~ $ docker pull gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a
[cut]
Digest: sha256:5d8bf456ad2d08ce3cd15f05b62fddc07fda3955267ee0d3ef73ee1a96b98e68
Status: Image is up to date for gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a

core@ip-10-0-0-212 ~ $ docker run -it --rm gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a /bin/bash
root@fe6a536393f8:/app/bundle# curl
bash: curl: command not found
root@fe6a536393f8:/app/bundle# 
</code></pre>
<p><a href="https://gist.github.com/iameli/72506721b70dcba6d5b2" rel="nofollow noreferrer">Full logs available on this gist</a>. I took the bad server out of our production cluster but still have it running if anyone wants me to do any other research.</p>
<h1>Added 2015-12-04</h1>
<p>I've run <code>docker tag gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a weird-image</code> on both servers to make everything more readable.</p>
<h2>which curl</h2>
<blockquote>
<p>Can you do a which curl in the first component to check where it finds its curl? And see if that file exists in the second component. – VonC</p>
</blockquote>
<p>Seems to not exist at all on the bad server.</p>
<p><em>Good Server</em></p>
<pre><code>core@ip-10-0-0-61 ~ $ docker run -it --rm weird-image /bin/bash 
root@529b8f20a610:/app/bundle# which curl
/usr/bin/curl
</code></pre>
<p><em>Bad Server</em></p>
<pre><code>core@ip-10-0-0-212 ~ $ docker run -it --rm weird-image /bin/bash 
root@ff98c850dbaa:/app/bundle# ls /usr/bin/curl
ls: cannot access /usr/bin/curl: No such file or directory
root@ff98c850dbaa:/app/bundle# 
</code></pre>
<h2>alias docker</h2>
<blockquote>
<p>Any chance you have set up an alias on the bad box? Run alias docker to check – morloch</p>
</blockquote>
<p>Nope.</p>
<p><em>Good Server</em></p>
<pre><code>core@ip-10-0-0-61 ~ $ alias docker
-bash: alias: docker: not found
</code></pre>
<p><em>Bad Server</em></p>
<pre><code>core@ip-10-0-0-212 ~ $ alias docker
-bash: alias: docker: not found
</code></pre>
<h2>time</h2>
<p>More weirdness: it takes a lot longer to run the container on the bad server.</p>
<p><em>Good Server</em></p>
<pre><code>core@ip-10-0-0-61 ~ $ time docker run weird-image echo &quot;Done&quot;
Done

real  0m0.422s
user  0m0.015s
sys 0m0.015s
</code></pre>
<p><em>Bad Server</em></p>
<pre><code>core@ip-10-0-0-212 ~ $ time docker run weird-image echo &quot;Done&quot;
Done

real  0m4.602s
user  0m0.010s
sys 0m0.010s
</code></pre>

<p>I've seen lots of cases where Docker images on-disk get random bits of corruption (causing weird inconsistencies like the one you describe here), and deleting and re-pulling the image "fixes" the problem.</p>

<p>To test this, you'll want to make sure you not only <code>docker rmi gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a</code> (which will minimally output <code>Untagged: gcr.io/surveyadmin-001/wolfgang:commit_e78e07eb6ce5727af6ffeb4ca3e903907e3ab83a</code>), but also delete the individual layers (and any other tags they may have) so that they're forced to be re-pulled.</p>
