title: Docker docker-compose not picking up relevant cached image
body: <p>I ran <code>docker-compose build celery</code> and (after hours of attempts on my poor connection) it succeeded. The first 80% of the <code>app</code> Dockerfile is identical, yet it doesn't reuse the cache. From what I can tell browsing around, Docker compares the base image and the instructions in Dockerfile and reuses if possible. </p>
  
  <hr>
  
  <p>UPDATE: The problem this question refers to disappeared, no idea why. Notes below.</p>
  
  <hr>
  
  <p>Yet I'm getting:</p>
  
  <pre><code>docker-compose build celery
  Building celery
  Step 1 : FROM python:2.7
   ---&gt; eb867117097c
  Step 2 : RUN apt-get update &amp;&amp; apt-get install -y vim gdal-bin libgdal-dev postgresql-client
   ---&gt; Using cache
   ---&gt; 2966946ca235
  
  . . . identical steps . . .
  
  Step 9 : RUN pip install --no-cache-dir -r requirements/production.txt
   ---&gt; Running in 02b42f721a34
  Collecting git+https://github.com . . .
  . . .
   ---&gt; f70ecc01cada
  Removing intermediate container 02b42f721a34
  Step 10 : RUN apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
   ---&gt; Running in 3575383edcef
  </code></pre>
  
  <p>When I type <code>docker images</code> I can see <code>f70ecc01cada</code>, but it is not found in the following:</p>
  
  <pre><code>docker-compose build app
  Building app
  Step 1 : FROM python:2.7
   ---&gt; eb867117097c
  Step 2 : RUN apt-get update &amp;&amp; apt-get install -y vim gdal-bin libgdal-dev postgresql-client
   ---&gt; Using cache
   ---&gt; 03e5040df047
  
  . . . identical steps . . .
  
  Step 9 : RUN pip install --no-cache-dir -r requirements/production.txt
   ---&gt; Running in bb26fab28548
  Collecting git+https://github.com . . .
  . . .
  </code></pre>
  
  <p>Am I right in thinking the latter should reuse the f70ecc01cada cached image? [Edit: I now realize 03e5040df047 is just a container thanks @Ohmen, so no point deleting it] </p>
  
  <p>I have tried editing docker-compose to be identical memory etc but it didn't help. Can I force reuse?</p>
  
  <hr>
  
  <p>UPDATE: The <code>app</code> build all of a sudden started picking up the cache from the <code>celery</code> build. I don't know what fixed it. Between it failing and working, I did connect to the image on the <code>celery</code> hierarchy just before the Dockerfiles diverged, manually ran the remaining commands on the <code>app</code> image (<code>docker run -it ... --entry=bash</code>, <code>adduser</code> and <code>chown app/</code>), and <code>docker committed</code> the resulting container. I then re-ran <code>docker-compose build app</code> and for some reason it picked up the cache from the <code>celery</code> hierarchy correctly and ran the remaining commands (create user and chown) successfully. I have no idea what made it work all of a sudden.</p>
  
  <hr>
  
  <p>EDIT: Docker history of the two image trees, showing that the history diverged on the line <code>RUN apt-get update &amp;&amp; apt-get install -y vim gdal-bin libgdal-dev postgresql-client</code>. Maybe I ran both builds simultaneously? I was pretty sure I didn't, as on my connection they take an hour or more, but perhaps that is the boring and embarrassing explanation. Perhaps Docker was randomly selecting the tree that subsequently failed, then randomly (after 56 attempts) selected the one that didn't.</p>
  
  <p>Initial successful build - layers sequenced from bottom up:</p>
  
  <pre><code>Chriss-MacBook-Pro:~ technicaltitch$ docker history fdw_celery
  IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
  d71bf8f2b902        28 hours ago        /bin/sh -c #(nop)  ENTRYPOINT ["/bin/sh" "-c"   0 B                 
  336bdb2b9ca9        28 hours ago        /bin/sh -c #(nop)  USER [celery]                0 B                 
  a1259f89bc1f        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   15.39 MB            
  57dd9a330337        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   335.2 kB            
  3accad8aa55c        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
  b4ff0c1d71fb        28 hours ago        /bin/sh -c #(nop) COPY dir:a8f922c5264fe2275a   15.39 MB            
  bc65bc84abbc        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
  f70ecc01cada        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   214.1 MB            
  85eb413bd5da        28 hours ago        /bin/sh -c #(nop)  ARG PIP_INDEX_URL=https://   0 B                 
  3cee97bf3ce1        28 hours ago        /bin/sh -c #(nop)  ARG PIP_TRUSTED_HOST=127.0   0 B                 
  cb090c4b1886        28 hours ago        /bin/sh -c #(nop)  WORKDIR /usr/src/app         0 B                 
  d0a1d3d15f94        28 hours ago        /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d   3.145 kB            
  e27ecd707562        28 hours ago        |1 http_proxy= /bin/sh -c mkdir -p /usr/src/a   0 B                 
  0b6e2857206c        28 hours ago        |1 http_proxy= /bin/sh -c VERSION=$(gdal-conf   5.903 MB            
  2966946ca235        29 hours ago        |1 http_proxy= /bin/sh -c apt-get update &amp;&amp; a   241.4 MB            
  eb867117097c        42 hours ago        /bin/sh -c #(nop) CMD ["python2"]               0 B                 
  </code></pre>
  
  <p>Second, failing build, with same script:</p>
  
  <pre><code>Chriss-MacBook-Pro:~ technicaltitch$ docker history 5ee
  IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
  5ee29875a771        29 hours ago        /bin/sh -c #(nop)  ARG PIP_INDEX_URL=https://   0 B                 
  daad37fd701b        29 hours ago        /bin/sh -c #(nop)  ARG PIP_TRUSTED_HOST=127.0   0 B                 
  f891ee9277da        29 hours ago        /bin/sh -c #(nop)  WORKDIR /usr/src/app         0 B                 
  2cce4acf9f2f        29 hours ago        /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d   3.145 kB            
  9484e6b7fa51        29 hours ago        |1 http_proxy=None /bin/sh -c mkdir -p /usr/s   0 B                 
  0031cdd56926        29 hours ago        |1 http_proxy=None /bin/sh -c VERSION=$(gdal-   5.903 MB            
  03e5040df047 &lt;-- [already diverged]      |1 http_proxy=None /bin/sh -c apt-get update    243.1 MB            
  eb867117097c        42 hours ago        /bin/sh -c #(nop) CMD ["python2"]               0 B                 
  </code></pre>
  
  <p>Finally, this is the image history after it magically succeeded. I had not deleted the other images when this worked (and still haven't):</p>
  
  <pre><code>Chriss-MacBook-Pro:~ technicaltitch$ docker history fdw_app
  IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
  edc615da4d6f        26 hours ago        /bin/sh -c #(nop)  ENTRYPOINT ["/usr/src/app/   0 B                 
  747c8744f6a7        26 hours ago        /bin/sh -c #(nop)  EXPOSE 8000/tcp              0 B                 
  8c543333e10b        26 hours ago        /bin/sh -c #(nop)  USER [django]                0 B                 
  b08f02d80d29        26 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   15.39 MB            
  ba42415ad78b        26 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   335.2 kB            
  027e0c8e39a9        26 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
  d56a78c02d18        26 hours ago        /bin/sh -c #(nop) COPY dir:9274abe4540edd1e86   15.39 MB            
  bc65bc84abbc        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   0 B                 
  f70ecc01cada        28 hours ago        |3 PIP_INDEX_URL=https://pypi.python.org/simp   214.1 MB            
  85eb413bd5da        28 hours ago        /bin/sh -c #(nop)  ARG PIP_INDEX_URL=https://   0 B                 
  3cee97bf3ce1        28 hours ago        /bin/sh -c #(nop)  ARG PIP_TRUSTED_HOST=127.0   0 B                 
  cb090c4b1886        28 hours ago        /bin/sh -c #(nop)  WORKDIR /usr/src/app         0 B                 
  d0a1d3d15f94        28 hours ago        /bin/sh -c #(nop) COPY dir:ec1503af2cbef5220d   3.145 kB            
  e27ecd707562        28 hours ago        |1 http_proxy= /bin/sh -c mkdir -p /usr/src/a   0 B                 
  0b6e2857206c        28 hours ago        |1 http_proxy= /bin/sh -c VERSION=$(gdal-conf   5.903 MB            
  2966946ca235 &lt;-- [correct img, suddenly..why?!]  ..xy= /bin/sh -c apt-get update &amp;&amp; a   241.4 MB            
  eb867117097c        42 hours ago        /bin/sh -c #(nop) CMD ["python2"]               0 B                 
  </code></pre>
  
  <p>The following is the start of both Dockerfiles. I have tried making the machine definitions in docker-compose.yml identical but to no avail (possibly after the 03e image was created).</p>
  
  <pre><code>FROM python:2.7
  
  RUN apt-get update &amp;&amp; apt-get install -y vim gdal-bin libgdal-dev postgresql-client
  
  # On Jessie, $(gdal-config --version) gives 1.10.1 and Pypi only has 1.10.0
  # so we need to strip the last part of the version number
  RUN VERSION=$(gdal-config --version); CFLAGS=$(gdal-config --cflags) easy_install GDAL==${VERSION%.*}
  
  RUN mkdir -p /usr/src/app/requirements
  COPY requirements /usr/src/app/requirements
  
  WORKDIR /usr/src/app
  
  ARG PIP_TRUSTED_HOST=127.0.0.1
  ARG PIP_INDEX_URL=https://pypi.python.org/simple/
  RUN pip install --no-cache-dir -r requirements/production.txt
  
  # Clean up APT when done.
  RUN apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
  
  COPY . /usr/src/app
  
  RUN mkdir -p /usr/src/app/log
  </code></pre>