{"action":"created","issue":{"url":"https://api.github.com/repos/docker/distribution/issues/1328","labels_url":"https://api.github.com/repos/docker/distribution/issues/1328/labels{/name}","comments_url":"https://api.github.com/repos/docker/distribution/issues/1328/comments","events_url":"https://api.github.com/repos/docker/distribution/issues/1328/events","html_url":"https://github.com/docker/distribution/issues/1328","id":125445045,"number":1328,"title":"Ceph RADOS storage driver: infernalis (v9.1.0)","user":{"login":"h0tbird","id":417440,"avatar_url":"https://avatars.githubusercontent.com/u/417440?v=3","gravatar_id":"","url":"https://api.github.com/users/h0tbird","html_url":"https://github.com/h0tbird","followers_url":"https://api.github.com/users/h0tbird/followers","following_url":"https://api.github.com/users/h0tbird/following{/other_user}","gists_url":"https://api.github.com/users/h0tbird/gists{/gist_id}","starred_url":"https://api.github.com/users/h0tbird/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/h0tbird/subscriptions","organizations_url":"https://api.github.com/users/h0tbird/orgs","repos_url":"https://api.github.com/users/h0tbird/repos","events_url":"https://api.github.com/users/h0tbird/events{/privacy}","received_events_url":"https://api.github.com/users/h0tbird/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-01-07T17:25:32Z","updated_at":"2016-01-07T19:19:21Z","closed_at":null,"body":"I am attempting to setup RADOS as my storage back-end.\r\nCeph cluster is running v9.1.0.\r\nWhen I attempt to connect to `/v2` all I get is:\r\n```\r\n{\"errors\":[{\"code\":\"UNAVAILABLE\",\"message\":\"service unavailable\",\"detail\":\"health check failed: please see /debug/health\"}]}\r\n```\r\nRegistry logs looks like this:\r\n```\r\ntime=\"2016-01-07T16:58:01.437072765Z\" level=debug msg=\"using \\\"text\\\" logging formatter\" \r\ntime=\"2016-01-07T16:58:01.43859602Z\" level=info msg=\"Opening connection to pool registry-ext\" \r\ntime=\"2016-01-07T16:58:01.445975478Z\" level=info msg=Connected \r\ntime=\"2016-01-07T16:58:01.446791078Z\" level=info msg=\"Connected to pool registry-ext\" \r\ntime=\"2016-01-07T16:58:01.446847045Z\" level=warning msg=\"No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.446905196Z\" level=info msg=\"redis not configured\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.446931855Z\" level=info msg=\"using inmemory blob descriptor cache\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.446925561Z\" level=info msg=\"Starting upload purge in 28m0s\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.450180027Z\" level=debug msg=\"rados.Stat(\\\"/scheduler-state.json\\\")\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry trace.duration=3.193602ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).Stat\" trace.id=0ba86df3-136a-4cd8-9e63-a5b67aee13a6 trace.line=141 version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.450224712Z\" level=info msg=\"Starting cached object TTL expiration scheduler...\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.897549175Z\" level=info msg=\"Registry configured as a proxy cache to https://registry-1.docker.io\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:01.897692105Z\" level=info msg=\"listening on [::]:5000\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry version=v2.2.1 \r\ntime=\"2016-01-07T16:58:11.900675199Z\" level=debug msg=\"rados.List(\\\"/\\\")\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry trace.duration=2.494498ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).List\" trace.id=522b7f6c-17f1-49d0-a40a-bab2a2fd8337 trace.line=154 version=v2.2.1 \r\ntime=\"2016-01-07T16:58:21.900473574Z\" level=debug msg=\"rados.List(\\\"/\\\")\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry trace.duration=2.389597ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).List\" trace.id=ef32005a-c7c9-4a94-8f24-dc933189a2c7 trace.line=154 version=v2.2.1 \r\ntime=\"2016-01-07T16:58:31.900460147Z\" level=debug msg=\"rados.List(\\\"/\\\")\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry trace.duration=2.386736ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).List\" trace.id=07d5b4ca-2df7-4a6b-b9fa-ffd92a481b7a trace.line=154 version=v2.2.1 \r\ntime=\"2016-01-07T16:58:41.900459557Z\" level=debug msg=\"rados.List(\\\"/\\\")\" go.version=go1.5.2 instance.id=4f717ca1-f6b2-4d7b-998b-3b508964f913 service=registry trace.duration=2.352806ms trace.file=\"/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go\" trace.func=\"github.com/docker/distribution/registry/storage/driver/base.(*Base).List\" trace.id=f597d459-017f-480b-88c1-45c933c90813 trace.line=154 version=v2.2.1\r\n10.128.2.161 - - [07/Jan/2016:17:00:25 +0000] \"GET /v2 HTTP/1.1\" 503 125 \"\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36\"\r\n10.128.2.161 - - [07/Jan/2016:17:00:25 +0000] \"GET /favicon.ico HTTP/1.1\" 503 125 \"http://registry-ext-sys.marathon.cell-1.mad.mesos:5000/v2\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36\"\r\n10.128.2.161 - - [07/Jan/2016:17:00:25 +0000] \"GET /favicon.ico HTTP/1.1\" 503 125 \"http://registry-ext-sys.marathon.cell-1.mad.mesos:5000/v2\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36\"\r\n```\r\nAnd this is my config:\r\n```\r\nversion: 0.1\r\nlog:\r\n  level: debug\r\n  formatter: text\r\n  fields:\r\n    service: registry\r\nstorage:\r\n    cache:\r\n        blobdescriptor: inmemory\r\n    rados:\r\n        poolname: registry-ext\r\nhttp:\r\n    addr: :5000\r\n    headers:\r\n        X-Content-Type-Options: [nosniff]\r\nhealth:\r\n  storagedriver:\r\n    enabled: true\r\n    interval: 10s\r\n    threshold: 3\r\nproxy:\r\n  remoteurl: https://registry-1.docker.io\r\n```\r\nI don't see errors in `ceph status -w`\r\n\r\nDocker registry is using:\r\n```\r\nroot@7f3ec20b39aa:/# dpkg -s librados2                 \r\nPackage: librados2\r\nStatus: install ok installed\r\nPriority: optional\r\nSection: libs\r\nInstalled-Size: 5696\r\nMaintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>\r\nArchitecture: amd64\r\nSource: ceph\r\nVersion: 0.80.10-0ubuntu1.14.04.3\r\nReplaces: libcrush, libcrush1, librados, librados1\r\nDepends: libboost-system1.54.0, libboost-thread1.54.0, libc6 (>= 2.14), libgcc1 (>= 1:4.1.1), libnspr4 (>= 2:4.9-2~) | libnspr4-0d (>= 1.8.0.10), libnss3 (>= 2:3.13.4-2~) | libnss3-1d (>= 3.12.0~1.9b1), libstdc++6 (>= 4.6), libuuid1 (>= 2.16)\r\nPre-Depends: multiarch-support\r\nConflicts: libcrush, libcrush1, librados, librados1\r\nDescription: RADOS distributed object store client library\r\n RADOS is a reliable, autonomic distributed object storage cluster\r\n developed as part of the Ceph distributed storage system.  This is a\r\n shared library allowing applications to access the distributed object\r\n store using a simple file-like interface.\r\nHomepage: http://ceph.com/\r\nOriginal-Maintainer: Ceph Maintainers <ceph-maintainers@lists.ceph.com>\r\n```\r\n\r\nwhereas Ceph mon and osd are using:\r\n```\r\nroot@core-1:/# dpkg -s librados2\r\nPackage: librados2\r\nStatus: install ok installed\r\nPriority: optional\r\nSection: libs\r\nInstalled-Size: 6669\r\nMaintainer: Ceph Maintainers <ceph-maintainers@lists.ceph.com>\r\nArchitecture: amd64\r\nSource: ceph\r\nVersion: 9.2.0-1trusty\r\nReplaces: librados, librados1\r\nDepends: libblkid1 (>= 2.16), libboost-system1.54.0, libboost-thread1.54.0, libc6 (>= 2.15), libgcc1 (>= 1:4.1.1), liblttng-ust0, libnspr4 (>= 2:4.9-2~) | libnspr4-0d (>= 1.8.0.10), libnss3 (>= 2:3.13.4-2~) | libnss3-1d (>= 3.12.9~beta2), libstdc++6 (>= 4.8)\r\nConflicts: librados, librados1\r\nDescription: RADOS distributed object store client library\r\n RADOS is a reliable, autonomic distributed object storage cluster\r\n developed as part of the Ceph distributed storage system.  This is a\r\n shared library allowing applications to access the distributed object\r\n store using a simple file-like interface.\r\nHomepage: http://ceph.com/\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/distribution/issues/comments/169778923","html_url":"https://github.com/docker/distribution/issues/1328#issuecomment-169778923","issue_url":"https://api.github.com/repos/docker/distribution/issues/1328","id":169778923,"user":{"login":"RichardScothern","id":922018,"avatar_url":"https://avatars.githubusercontent.com/u/922018?v=3","gravatar_id":"","url":"https://api.github.com/users/RichardScothern","html_url":"https://github.com/RichardScothern","followers_url":"https://api.github.com/users/RichardScothern/followers","following_url":"https://api.github.com/users/RichardScothern/following{/other_user}","gists_url":"https://api.github.com/users/RichardScothern/gists{/gist_id}","starred_url":"https://api.github.com/users/RichardScothern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/RichardScothern/subscriptions","organizations_url":"https://api.github.com/users/RichardScothern/orgs","repos_url":"https://api.github.com/users/RichardScothern/repos","events_url":"https://api.github.com/users/RichardScothern/events{/privacy}","received_events_url":"https://api.github.com/users/RichardScothern/received_events","type":"User","site_admin":false},"created_at":"2016-01-07T19:19:20Z","updated_at":"2016-01-07T19:19:20Z","body":"That message indicates a misconfiguration with your registry setup.  Can you add a debug endpoint (config below) to you configuration and check the debug endpoint:\r\n`http://<host>:5001/debug/health`\r\n\r\n```\r\nhttp:\r\n    debug:\r\n        addr: localhost:5001\r\n```\r\n\r\n"}}