{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/23684","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/23684/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/23684/comments","events_url":"https://api.github.com/repos/docker/docker/issues/23684/events","html_url":"https://github.com/docker/docker/issues/23684","id":160931996,"number":23684,"title":"Docker pull/push won't recognize images from docker load.. sometimes?","user":{"login":"ellerycrane","id":784493,"avatar_url":"https://avatars.githubusercontent.com/u/784493?v=3","gravatar_id":"","url":"https://api.github.com/users/ellerycrane","html_url":"https://github.com/ellerycrane","followers_url":"https://api.github.com/users/ellerycrane/followers","following_url":"https://api.github.com/users/ellerycrane/following{/other_user}","gists_url":"https://api.github.com/users/ellerycrane/gists{/gist_id}","starred_url":"https://api.github.com/users/ellerycrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ellerycrane/subscriptions","organizations_url":"https://api.github.com/users/ellerycrane/orgs","repos_url":"https://api.github.com/users/ellerycrane/repos","events_url":"https://api.github.com/users/ellerycrane/events{/privacy}","received_events_url":"https://api.github.com/users/ellerycrane/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/group/distribution","name":"group/distribution","color":"F6FBFF"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.11","name":"version/1.11","color":"ededed"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2016-06-17T17:00:02Z","updated_at":"2016-06-27T16:59:26Z","closed_at":null,"body":" If you do a `docker load` of an image/repo exported with `docker save <repo>:<tag>`, you'll have the images and layers available for building. If you, however, attempt to do a `docker push <repo>:<tag>`, it will push up ​**all**​ of the layers, even if they are identical to the ones in the remote registry for that `<repo>:<tag>`. \r\n\r\nIf, instead, you do a `docker pull <repo>:<tag>` before doing a `docker push`, it will download ​**all**​ of the layers (even though you've already got them via your previous `docker load`), but then subsequently be smart about only pulling/pushing new layers. \r\n\r\nWhat's even more mystifying is that if you've done a `docker pull`, then used `docker rmi` to remove all of the images you pulled, *then* doing `docker load` followed by `docker push` will properly track 'already present' layers. \r\n\r\nThere's something a `docker pull` does to the local docker cache that establishes a link to the remote registry that I don't fully understand. Is there some way this link can be established _without_ downloading all of the (already present via `docker load`) remote images/layers? \r\n\r\n<!--\r\nIf you are reporting a new issue, make sure that we do not have any duplicates\r\nalready open. You can ensure this by searching the issue list for this\r\nrepository. If there is a duplicate, please close your issue and add a comment\r\nto the existing issue instead.\r\n\r\nIf you suspect your issue is a bug, please edit your issue description to\r\ninclude the BUG REPORT INFORMATION shown below. If you fail to provide this\r\ninformation within 7 days, we cannot debug your issue and will close it. We\r\nwill, however, reopen it if you later provide the information.\r\n\r\nFor more information about reporting issues, see\r\nhttps://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues\r\n\r\n---------------------------------------------------\r\nBUG REPORT INFORMATION\r\n---------------------------------------------------\r\nUse the commands below to provide key information from your environment:\r\nYou do NOT have to include this information if this is a FEATURE REQUEST\r\n-->\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.11.2\r\n API version:  1.23\r\n Go version:   go1.5.4\r\n Git commit:   b9f10c9\r\n Built:        Wed Jun  1 21:20:08 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.11.2\r\n API version:  1.23\r\n Go version:   go1.5.4\r\n Git commit:   b9f10c9\r\n Built:        Wed Jun  1 21:20:08 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 1\r\nServer Version: 1.11.2\r\nStorage Driver: btrfs\r\n Build Version: Btrfs v3.17\r\n Library Version: 101\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: null host bridge\r\nKernel Version: 4.4.12-boot2docker\r\nOperating System: Buildroot 2016.02 (containerized)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 1.955 GiB\r\nName: c79f2d0a-a4b0-4248-4d41-bab905aad64c\r\nID: WJMK:7VEO:PFBY:NSII:JD6N:AIHE:M3QK:YGS4:EBA7:DHJN:JRTO:WLQN\r\nDocker Root Dir: /var/lib/docker\r\nDebug mode (client): false\r\nDebug mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: bridge-nf-call-iptables is disabled\r\nWARNING: bridge-nf-call-ip6tables is disabled\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nRunning inside of a runC-based container, but I've duplicated the behavior using OSX docker-machine/VirtualBox set up.\r\n\r\n**Steps to reproduce the issue:**\r\nFirst: \r\n1. Do a `docker pull <repo>:<tag>`.\r\n2. Do a `docker save -o image.tar <repo>:<tag>`\r\n3. In fresh environment, run `docker load -i image.tar` on file created from step 2.\r\n\r\nFollow steps 1-2, then do each scenario below after doing step 3 in a fresh environment:\r\n* A. Do `docker push <repo>:<tag>` to try to push the loaded image. It will push **all** of the layers, even though they're already present in the remote registry.\r\n* B. Do `docker pull <repo>:<tag>`. It will pull down **all** of the layers, even though they are already present from doing `docker load`.\r\n* C.\r\n  * a. Do `docker pull <repo>:<tag>`. (Everything is downloaded)\r\n  * b. Do `docker rmi <repo>:<tag>`. (All of the images are removed; another pull will download everything again)\r\n  * c. Do `docker load -i image.tar`. (Images will be back again)\r\n  * d. Do `docker pull <repo>:<tag>`. **(Layers will _not_ be downloaded, saying they already exist!)**\r\n  * e. Instead of d, do `docker push <repo>:<tag>`. **(Layers will _not_ be pushed, saying they already exist!)**\r\n\r\n**Describe the results you received:**\r\nA.  It will push **all** of the layers, even though they're already present in the remote registry.\r\nB.  It will pull down **all** of the layers, even though they are already present from doing `docker load`.\r\nC. After doing a `docker pull` followed by a `docker rmi`, the images/layers loaded via `docker load` *will* be recognized as 'already exists' when pushing to/pulling from the remote repo, even though they weren't before.\r\n\r\n**Describe the results you expected:**\r\nA. It should say \"Layer already exists\" when trying to push each layer, since they were loaded already. \r\nB. It should say \"Already exists\" for all of the layers loaded in 3, and not download them.\r\nC. The images and layers loaded via `docker load` should be treated as 'already exists' when doing `docker pull` or `docker push` regardless of whether or not a `docker pull` has happened before. Downloading everything via `docker pull` then deleting everything via `docker rmi` and then loading everything again via `docker load` should not be different from just doing a `docker load` in the first place.\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\nThis is a huge impediment for an automated CI build process.  Having no way to 'prime the cache' via `docker load` in a fresh build environment (which may be spun up dynamically!) and having to pay the price of doing a full `docker pull` of a potentially multi-hundred megabyte docker image just to 'sync up' with the remote repository when you've already got all of the data loaded is unacceptable. \r\n\r\nBuilding and pushing only the new layers should be a very quick process (in some cases, 30 seconds or less), but the forced download of the entire image can turn this into a _minimum_ of **5-10 minutes**. Using a local registry configured for pull-through caching could help, but as per the documentation and the [roadmap](https://github.com/docker/distribution/blob/master/ROADMAP.md#proxying-to-other-registries), this cannot be used for private registries. Because using Dockerhub to host repos (even private repos) isn't an acceptable solution for many (including my organization), the local registry mirror isn't a solution. Even were it possible, when all of the data has already been loaded via `docker load` needing to download it again for any reason is silly. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/228807203","html_url":"https://github.com/docker/docker/issues/23684#issuecomment-228807203","issue_url":"https://api.github.com/repos/docker/docker/issues/23684","id":228807203,"user":{"login":"ellerycrane","id":784493,"avatar_url":"https://avatars.githubusercontent.com/u/784493?v=3","gravatar_id":"","url":"https://api.github.com/users/ellerycrane","html_url":"https://github.com/ellerycrane","followers_url":"https://api.github.com/users/ellerycrane/followers","following_url":"https://api.github.com/users/ellerycrane/following{/other_user}","gists_url":"https://api.github.com/users/ellerycrane/gists{/gist_id}","starred_url":"https://api.github.com/users/ellerycrane/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ellerycrane/subscriptions","organizations_url":"https://api.github.com/users/ellerycrane/orgs","repos_url":"https://api.github.com/users/ellerycrane/repos","events_url":"https://api.github.com/users/ellerycrane/events{/privacy}","received_events_url":"https://api.github.com/users/ellerycrane/received_events","type":"User","site_admin":false},"created_at":"2016-06-27T16:59:26Z","updated_at":"2016-06-27T16:59:26Z","body":"@tonistiigi Would it be possible add an option to `docker save` to include registry checksums, and an option to `docker pull` that just downloads the checksums and not the data? The key point here is that needing to `docker pull` and `push` large amounts of data when it already exists on both sides is *bad*, and there should be a way to avoid that- even if it involves some custom flags on the commands."}}
{"actor":{"display_login":"ellerycrane"}}