{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/22822","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/22822/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/22822/comments","events_url":"https://api.github.com/repos/docker/docker/issues/22822/events","html_url":"https://github.com/docker/docker/issues/22822","id":155596811,"number":22822,"title":"Docker pull failing to download blobs from private registry (configured as a proxy)","user":{"login":"jonnymbgood","id":485900,"avatar_url":"https://avatars.githubusercontent.com/u/485900?v=3","gravatar_id":"","url":"https://api.github.com/users/jonnymbgood","html_url":"https://github.com/jonnymbgood","followers_url":"https://api.github.com/users/jonnymbgood/followers","following_url":"https://api.github.com/users/jonnymbgood/following{/other_user}","gists_url":"https://api.github.com/users/jonnymbgood/gists{/gist_id}","starred_url":"https://api.github.com/users/jonnymbgood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonnymbgood/subscriptions","organizations_url":"https://api.github.com/users/jonnymbgood/orgs","repos_url":"https://api.github.com/users/jonnymbgood/repos","events_url":"https://api.github.com/users/jonnymbgood/events{/privacy}","received_events_url":"https://api.github.com/users/jonnymbgood/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/group/distribution","name":"group/distribution","color":"F6FBFF"},{"url":"https://api.github.com/repos/docker/docker/labels/version/1.10","name":"version/1.10","color":"ededed"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-05-18T20:43:05Z","updated_at":"2016-05-19T17:14:13Z","closed_at":null,"body":"**Output of `docker version`:**\r\n\r\n```\r\nClient:\r\n Version:      1.10.3\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   8acee1b\r\n Built:\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.10.3\r\n API version:  1.22\r\n Go version:   go1.5.3\r\n Git commit:   8acee1b\r\n Built:\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\nContainers: 0\r\n Running: 0\r\n Paused: 0\r\n Stopped: 0\r\nImages: 1\r\nServer Version: 1.10.3\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nPlugins:\r\n Volume: local\r\n Network: host bridge null\r\nKernel Version: 4.5.0-coreos-r1\r\nOperating System: CoreOS 1010.3.0 (MoreOS)\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 1\r\nTotal Memory: 3.864 GiB\r\nName: worker-1\r\nID: E4I5:TQHH:XFX2:AXZH:WY5R:KFAB:4QT6:WTVD:XNCP:BDFK:XPQB:ZS6W\r\nDebug mode (server): true\r\n File Descriptors: 13\r\n Goroutines: 28\r\n System Time: 2016-05-18T20:16:52.543996087Z\r\n EventsListeners: 0\r\n Init SHA1: abe9d229a8fee27dcfb8ac8ab741b4497058fed4\r\n Init Path: /usr/libexec/docker/dockerinit\r\n Docker Root Dir: /var/lib/docker\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\n\r\nCoreOS 1010.3.0 running in Vagrant (using the officially provided vagrant box `Linux worker-1 4.5.0-coreos-r1 #2 SMP Thu May 5 07:27:26 UTC 2016 x86_64 Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz GenuineIntel GNU/Linux`)\r\n\r\n**Steps to reproduce the issue:**\r\n1. Configure the docker daemon with the following opts: `docker daemon --host=fd:// -D --insecure-registry 192.168.1.62:5000 --registry-mirror http://192.168.1.62:5000 --exec-opt native.cgroupdriver=systemd --bip=10.2.82.1/24 --mtu=1472 --ip-masq=false --selinux-enabled`\r\n2. Setup a private registry (see below for details)\r\n3. run `docker pull busybox`\r\n\r\n**Docker Registry Details**\r\nRunning registry:2 (2.4.0 as of this issue) with `/var/lib/registry` and `/etc/docker/registry` mounted\r\n\r\n**config.yaml:**\r\n````yaml\r\nversion: 0.1\r\nlog:\r\n  fields:\r\n    service: registry\r\nstorage:\r\n    cache:\r\n        blobdescriptor: inmemory\r\n    filesystem:\r\n        rootdirectory: /var/lib/registry\r\nhttp:\r\n    addr: :5000\r\n    headers:\r\n        X-Content-Type-Options: [nosniff]\r\nhealth:\r\n  storagedriver:\r\n    enabled: true\r\n    interval: 10s\r\n    threshold: 3\r\nproxy:\r\n  remoteurl: https://registry-1.docker.io\r\n  username: [redacted]\r\n  password: [redacted]\r\n````\r\n\r\n\r\n\r\n**Describe the results you received:**\r\n\r\nIt tries and fails to pull the image from my private registry/cache, and falls back to pulling from docker hub\r\n\r\n````\r\n$ docker pull busybox\r\nUsing default tag: latest\r\nlatest: Pulling from library/busybox\r\n385e281300cc: Downloading [==================================================>]   676 kB/676 kB\r\na3ed95caeb02: Download complete\r\nlatest: Pulling from library/busybox\r\n385e281300cc: Pull complete\r\na3ed95caeb02: Pull complete\r\nDigest: sha256:4a887a2326ec9e0fa90cce7b4764b0e627b5d6afcb81a3f73c85dc29cea00048\r\nStatus: Downloaded newer image for busybox:latest\r\n````\r\n\r\nThe output from docker (running with -D):\r\n````\r\nMay 18 20:14:22 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:22.282308829Z\" level=debug msg=\"Calling GET /v1.22/info\"\r\nMay 18 20:14:22 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:22.282334143Z\" level=debug msg=\"GET /v1.22/info\"\r\nMay 18 20:14:22 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:22.369936803Z\" level=debug msg=\"Calling POST /v1.22/images/create\"\r\nMay 18 20:14:22 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:22.370181704Z\" level=debug msg=\"POST /v1.22/images/create?fromImage=busybox%3Alatest\"\r\nMay 18 20:14:22 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:22.370433003Z\" level=debug msg=\"Trying to pull busybox from http://192.168.1.62:5000/ v2\"\r\nMay 18 20:14:28 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:28.554019116Z\" level=debug msg=\"Pulling ref from V2 registry: busybox:latest\"\r\nMay 18 20:14:28 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:28.557525620Z\" level=debug msg=\"pulling blob \\\"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\\\"\"\r\nMay 18 20:14:28 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:28.557823911Z\" level=debug msg=\"pulling blob \\\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\\\"\"\r\nMay 18 20:14:28 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:28.602032284Z\" level=debug msg=\"Downloaded a3ed95caeb02 to tempfile /var/lib/docker/tmp/GetImageBlob953663924\"\r\nMay 18 20:14:28 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:28.671871793Z\" level=error msg=\"Download failed, retrying: unexpected EOF\"\r\nMay 18 20:14:33 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:33.673627057Z\" level=debug msg=\"pulling blob \\\"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\\\"\"\r\nMay 18 20:14:33 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:33.808571022Z\" level=error msg=\"Download failed, retrying: unexpected EOF\"\r\nMay 18 20:14:43 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:43.810067351Z\" level=debug msg=\"pulling blob \\\"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\\\"\"\r\nMay 18 20:14:43 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:43.914003406Z\" level=error msg=\"Download failed, retrying: unexpected EOF\"\r\nMay 18 20:14:58 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:58.914610114Z\" level=debug msg=\"pulling blob \\\"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\\\"\"\r\nMay 18 20:14:58 worker-1 dockerd[1691]: time=\"2016-05-18T20:14:58.990499995Z\" level=error msg=\"Download failed, retrying: unexpected EOF\"\r\nMay 18 20:15:18 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:18.992090538Z\" level=debug msg=\"pulling blob \\\"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\\\"\"\r\nMay 18 20:15:19 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:19.086256971Z\" level=error msg=\"Download failed: unexpected EOF\"\r\nMay 18 20:15:19 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:19.087336822Z\" level=debug msg=\"Error trying v2 registry: unexpected EOF\"\r\nMay 18 20:15:19 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:19.087508540Z\" level=debug msg=\"Trying to pull busybox from https://registry-1.docker.io v2\"\r\nMay 18 20:15:20 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:20.246987196Z\" level=debug msg=\"Increasing token expiration to: 0 seconds\"\r\nMay 18 20:15:20 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:20.834621924Z\" level=debug msg=\"Pulling ref from V2 registry: busybox:latest\"\r\nMay 18 20:15:20 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:20.836611928Z\" level=debug msg=\"pulling blob \\\"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\\\"\"\r\nMay 18 20:15:20 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:20.837918372Z\" level=debug msg=\"pulling blob \\\"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\\\"\"\r\nMay 18 20:15:22 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:22.309309801Z\" level=debug msg=\"Downloaded a3ed95caeb02 to tempfile /var/lib/docker/tmp/GetImageBlob211073895\"\r\nMay 18 20:15:24 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:24.706191486Z\" level=debug msg=\"Downloaded 385e281300cc to tempfile /var/lib/docker/tmp/GetImageBlob622299546\"\r\nMay 18 20:15:24 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:24.707247549Z\" level=debug msg=\"Start untar layer\"\r\nMay 18 20:15:24 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:24.806273720Z\" level=debug msg=\"Untar time: 0.09902161400000001s\"\r\nMay 18 20:15:24 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:24.806617334Z\" level=debug msg=\"Applied tar sha256:1834950e52ce4d5a88a1bbd131c537f4d0e56d10ff0dd69e66be3b7dfa9df7e6 to 9922b3c0dab64fa183644287c6642786e0c13293afdb65ec62d05b94b591df29, size: 1112820\"\r\nMay 18 20:15:24 worker-1 dockerd[1691]: time=\"2016-05-18T20:15:24.867609516Z\" level=debug msg=\"Applied tar sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef to d2c414aa0e3d8a512730127c765b57d9da17712737a844c7ec3da91dd3858029, size: 0\"\r\n````\r\n\r\nThe output from the registry is:\r\n````\r\n172.17.0.1 - - [18/May/2016:20:14:58 +0000] \"GET /v2/library/busybox/blobs/sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66 HTTP/1.1\" 200 675992 \"\" \"docker/1.10.3 go/go1.5.3 git-commit/8acee1b kernel/4.5.0-coreos-r1 os/linux arch/amd64\" time=\"2016-05-18T20:15:18Z\" level=info msg=\"response completed\" go.version=go1.6.1 http.request.host=\"192.168.1.62:5000\" http.request.id=2b57139d-ae50-497f-930f-7000a90217a1 http.request.method=GET http.request.remoteaddr=\"172.17.0.1:58590\" http.request.uri=\"/v2/library/busybox/blobs/sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\" http.request.useragent=\"docker/1.10.3 go/go1.5.3 git-commit/8acee1b kernel/4.5.0-coreos-r1 os/linux arch/amd64\" http.response.contenttype=\"application/octet-stream\" http.response.duration=14.837144ms http.response.status=200 http.response.written=675992 instance.id=fc3e81fe-0073-42e2-8bc0-1517bd5684ef version=v2.4.0 \r\n\r\n172.17.0.1 - - [18/May/2016:20:15:18 +0000] \"GET /v2/library/busybox/blobs/sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66 HTTP/1.1\" 200 675992 \"\" \"docker/1.10.3 go/go1.5.3 git-commit/8acee1b kernel/4.5.0-coreos-r1 os/linux arch/amd64\"\r\n````\r\n\r\nA manual curl (HEAD) to the registry for the blob results in:\r\n````\r\n$ curl -I 192.168.1.62:5000/v2/library/busybox/blobs/sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\r\nHTTP/1.1 200 OK\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=31536000\r\nContent-Length: 675992\r\nContent-Type: application/octet-stream\r\nDocker-Content-Digest: sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\r\nDocker-Distribution-Api-Version: registry/2.0\r\nEtag: \"sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66\"\r\nX-Content-Type-Options: nosniff\r\nDate: Wed, 18 May 2016 20:10:31 GMT\r\n````\r\n\r\nA curl call to get the contents results in a file with the correct hash:\r\n````\r\n$ curl 192.168.1.62:5000/v2/library/busybox/blobs/sha256:385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66 > blob\r\n$ sha256sum blob\r\n385e281300cc6d88bdd155e0931fbdfbb1801c2b0265340a40481ee2b733ae66  blob\r\n````\r\n\r\nI have verified that the registry does include the image \r\n````\r\n$ curl 192.168.1.62:5000/v2/_catalog\r\n{\"repositories\":[\"library/busybox\"]}\r\n````\r\n\r\n**Describe the results you expected:**\r\n\r\nFor it to pull from my private registry mirror successfully"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/220391338","html_url":"https://github.com/docker/docker/issues/22822#issuecomment-220391338","issue_url":"https://api.github.com/repos/docker/docker/issues/22822","id":220391338,"user":{"login":"aaronlehmann","id":10601430,"avatar_url":"https://avatars.githubusercontent.com/u/10601430?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronlehmann","html_url":"https://github.com/aaronlehmann","followers_url":"https://api.github.com/users/aaronlehmann/followers","following_url":"https://api.github.com/users/aaronlehmann/following{/other_user}","gists_url":"https://api.github.com/users/aaronlehmann/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronlehmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronlehmann/subscriptions","organizations_url":"https://api.github.com/users/aaronlehmann/orgs","repos_url":"https://api.github.com/users/aaronlehmann/repos","events_url":"https://api.github.com/users/aaronlehmann/events{/privacy}","received_events_url":"https://api.github.com/users/aaronlehmann/received_events","type":"User","site_admin":false},"created_at":"2016-05-19T17:14:13Z","updated_at":"2016-05-19T17:14:13Z","body":"Thanks, I did not notice that.\r\n\r\nIt seems like the main mystery here is why the connection closes before the download completes. Presumably that has something to do with the registry acting as a proxy, but beyond that I'm not sure."}}