{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/21147","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/21147/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/21147/comments","events_url":"https://api.github.com/repos/docker/docker/issues/21147/events","html_url":"https://github.com/docker/docker/issues/21147","id":140346647,"number":21147,"title":"Odd Behavior for 1.10.x Registry v1 Pushes","user":{"login":"ekristen","id":48329,"avatar_url":"https://avatars.githubusercontent.com/u/48329?v=3","gravatar_id":"","url":"https://api.github.com/users/ekristen","html_url":"https://github.com/ekristen","followers_url":"https://api.github.com/users/ekristen/followers","following_url":"https://api.github.com/users/ekristen/following{/other_user}","gists_url":"https://api.github.com/users/ekristen/gists{/gist_id}","starred_url":"https://api.github.com/users/ekristen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ekristen/subscriptions","organizations_url":"https://api.github.com/users/ekristen/orgs","repos_url":"https://api.github.com/users/ekristen/repos","events_url":"https://api.github.com/users/ekristen/events{/privacy}","received_events_url":"https://api.github.com/users/ekristen/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/group/distribution","name":"group/distribution","color":"F6FBFF"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-03-12T04:40:06Z","updated_at":"2016-03-12T16:34:22Z","closed_at":null,"body":"I know things changed with 1.10, but registry v1 is still supported, but using a registry v1 is painful now. Takes over 70 seconds before it attempts to push an image, then takes much longer per image then docker 1.9.1 and below.\r\n\r\n**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.10.2\r\n API version:  1.22\r\n Go version:   go1.6\r\n Git commit:   c3959b1\r\n Built:        Tue Feb 23 21:12:32 UTC 2016\r\n OS/Arch:      darwin/amd64\r\n\r\nServer:\r\n Version:      1.10.3\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   20f81dd\r\n Built:        Thu Mar 10 15:54:52 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 23\r\n Running: 0\r\n Paused: 0\r\n Stopped: 23\r\nImages: 2222\r\nServer Version: 1.10.3\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 2276\r\n Dirperm1 Supported: false\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nPlugins:\r\n Volume: local\r\n Network: bridge null host\r\nKernel Version: 3.13.0-39-generic\r\nOperating System: Ubuntu 14.04.3 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 8\r\nTotal Memory: 7.797 GiB\r\nName: random\r\nID: J5MX:3AOW:4ASK:PT4U:Z4JX:XZWF:VAKF:YBUI:TYMY:Z4KV:OX3R:SFJT\r\nDebug mode (server): true\r\n File Descriptors: 13\r\n Goroutines: 34\r\n System Time: 2016-03-11T23:21:28.202728025-05:00\r\n EventsListeners: 0\r\n Init SHA1: 14bbd54b64d2a737269118ce4f1613787e2d0ce8\r\n Init Path: /usr/lib/docker/dockerinit\r\n Docker Root Dir: /var/lib/docker\r\nUsername: random\r\nRegistry: https://index.docker.io/v1/\r\nWARNING: No swap limit support\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\n\r\n\r\n**Steps to reproduce the issue:**\r\n1. Build an image\r\n2. Push to a v1 registry\r\n\r\n\r\n**Describe the results you received:**\r\nAll the layers go into **waiting immediately**\r\n\r\n```\r\nf14109b5884e: Waiting\r\nbd389813facf: Waiting\r\n0dbcdbf526ce: Waiting\r\n187d53198f19: Waiting\r\n6384a5c74e87: Waiting\r\n3612bd767d5f: Waiting\r\ndd72ba050b45: Waiting\r\n309b7466071b: Waiting\r\n4353bfbc20c0: Waiting\r\n5f70bf18a086: Waiting\r\n90f774439683: Waiting\r\ndfd83ed44976: Waiting\r\n217e6c75dcfc: Waiting\r\nc0b5f9221fac: Waiting\r\n```\r\n\r\nThen the first 5 layers immediately go into **Retrying**\r\n\r\n```\r\nf14109b5884e: Retrying in 6 seconds\r\nbd389813facf: Retrying in 6 seconds\r\n0dbcdbf526ce: Retrying in 6 seconds\r\n187d53198f19: Retrying in 6 seconds\r\n6384a5c74e87: Retrying in 6 seconds\r\n3612bd767d5f: Waiting\r\ndd72ba050b45: Waiting\r\n309b7466071b: Waiting\r\n4353bfbc20c0: Waiting\r\n5f70bf18a086: Waiting\r\n90f774439683: Waiting\r\ndfd83ed44976: Waiting\r\n217e6c75dcfc: Waiting\r\nc0b5f9221fac: Waiting\r\n```\r\n\r\nThen after about 5 retries (70+ seconds), docker starts pushing from the bottom up.\r\n\r\n```\r\nf14109b5884e: Waiting\r\nbd389813facf: Waiting\r\n0dbcdbf526ce: Waiting\r\n187d53198f19: Waiting\r\n6384a5c74e87: Waiting\r\n3612bd767d5f: Waiting\r\ndd72ba050b45: Waiting\r\n309b7466071b: Waiting\r\n4353bfbc20c0: Waiting\r\n5f70bf18a086: Image successfully pushed\r\n90f774439683: Pushing [==>                                                ] 10.31 MB/178 MB\r\ndfd83ed44976: Image successfully pushed\r\n217e6c75dcfc: Already exists\r\nc0b5f9221fac: Already exists\r\n```\r\n\r\nIt finally pushes.\r\n\r\n**Describe the results you expected:**\r\n\r\nTough to say, with the change to content addressable layers and sha256 hashes, I'm not sure how much really changed in the registry pushing process, but since the v1 registry hasn't been deprecated, prior to 1.10.0 release it would sequentially go through the layers and identify the ones that were already pushed and push the new one, one at a time. \r\n\r\nIt would also push immediately, none of this retry for 70+ seconds before doing anything.\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/195769796","html_url":"https://github.com/docker/docker/issues/21147#issuecomment-195769796","issue_url":"https://api.github.com/repos/docker/docker/issues/21147","id":195769796,"user":{"login":"ekristen","id":48329,"avatar_url":"https://avatars.githubusercontent.com/u/48329?v=3","gravatar_id":"","url":"https://api.github.com/users/ekristen","html_url":"https://github.com/ekristen","followers_url":"https://api.github.com/users/ekristen/followers","following_url":"https://api.github.com/users/ekristen/following{/other_user}","gists_url":"https://api.github.com/users/ekristen/gists{/gist_id}","starred_url":"https://api.github.com/users/ekristen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ekristen/subscriptions","organizations_url":"https://api.github.com/users/ekristen/orgs","repos_url":"https://api.github.com/users/ekristen/repos","events_url":"https://api.github.com/users/ekristen/events{/privacy}","received_events_url":"https://api.github.com/users/ekristen/received_events","type":"User","site_admin":false},"created_at":"2016-03-12T16:34:22Z","updated_at":"2016-03-12T16:34:22Z","body":"I think it's very possible that it is detecting things incorrectly, this isn't the first time registry version detection has been a problem, IIRC.\r\n\r\nI'm using docker's v1 registry, however I am using my own open source index implementation (which has worked flawlessly up until 1.10.x). https://github.com/ekristen/docker-index\r\n\r\nIt's very possible that my index is returning something that is causing incorrect detection.\r\n\r\nCould you point me to where the detection code is?\r\n\r\nIn the mean time, when I get some time today (hopefully) I'll grab some debug logs for you."}}