{"action":"closed","issue":{"url":"https://api.github.com/repos/poldracklab/mriqc/issues/499","repository_url":"https://api.github.com/repos/poldracklab/mriqc","labels_url":"https://api.github.com/repos/poldracklab/mriqc/issues/499/labels{/name}","comments_url":"https://api.github.com/repos/poldracklab/mriqc/issues/499/comments","events_url":"https://api.github.com/repos/poldracklab/mriqc/issues/499/events","html_url":"https://github.com/poldracklab/mriqc/issues/499","id":228443784,"number":499,"title":"[something to look into] Update Docker to 1.10 to update manifest version to 2","user":{"login":"vsoch","id":814322,"avatar_url":"https://avatars0.githubusercontent.com/u/814322?v=4","gravatar_id":"","url":"https://api.github.com/users/vsoch","html_url":"https://github.com/vsoch","followers_url":"https://api.github.com/users/vsoch/followers","following_url":"https://api.github.com/users/vsoch/following{/other_user}","gists_url":"https://api.github.com/users/vsoch/gists{/gist_id}","starred_url":"https://api.github.com/users/vsoch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vsoch/subscriptions","organizations_url":"https://api.github.com/users/vsoch/orgs","repos_url":"https://api.github.com/users/vsoch/repos","events_url":"https://api.github.com/users/vsoch/events{/privacy}","received_events_url":"https://api.github.com/users/vsoch/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/poldracklab/mriqc/milestones/8","html_url":"https://github.com/poldracklab/mriqc/milestone/8","labels_url":"https://api.github.com/repos/poldracklab/mriqc/milestones/8/labels","id":2119546,"number":8,"title":"Wishlist","description":"","creator":{"login":"oesteban","id":598470,"avatar_url":"https://avatars2.githubusercontent.com/u/598470?v=4","gravatar_id":"","url":"https://api.github.com/users/oesteban","html_url":"https://github.com/oesteban","followers_url":"https://api.github.com/users/oesteban/followers","following_url":"https://api.github.com/users/oesteban/following{/other_user}","gists_url":"https://api.github.com/users/oesteban/gists{/gist_id}","starred_url":"https://api.github.com/users/oesteban/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oesteban/subscriptions","organizations_url":"https://api.github.com/users/oesteban/orgs","repos_url":"https://api.github.com/users/oesteban/repos","events_url":"https://api.github.com/users/oesteban/events{/privacy}","received_events_url":"https://api.github.com/users/oesteban/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":2,"state":"open","created_at":"2016-11-07T16:55:54Z","updated_at":"2018-05-04T03:00:59Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2017-05-13T02:42:14Z","updated_at":"2018-05-04T03:00:59Z","closed_at":"2018-05-04T03:00:59Z","author_association":"OWNER","body":"hey @poldracklab! Here is a fun little docker hub api adventure, mostly for fun, but it will make integration with Singularity better for you down the line. I noticed recently that your image returns a version 1 of the Docker manifest, versus [version 2](https://docs.docker.com/registry/spec/manifest-v2-2/).  For example:\r\n\r\n```\r\nclient=DockerApiConnection(image='poldracklab/mriqc')\r\nDEBUG Headers found: Accept,Content-Type\r\nVERBOSE Registry: index.docker.io\r\nVERBOSE Namespace: poldracklab\r\nVERBOSE Repo Name: mriqc\r\nVERBOSE Repo Tag: latest\r\nVERBOSE Version: None\r\nVERBOSE Obtaining tags: https://index.docker.io/v2/poldracklab/mriqc/tags/list\r\nDEBUG GET https://index.docker.io/v2/poldracklab/mriqc/tags/list\r\nDEBUG Http Error with code 401\r\nDEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:poldracklab/mriqc:pull\r\nDEBUG Headers found: Accept,Authorization,Content-Type\r\n\r\nmanifest = client.get_manifest()\r\nVERBOSE Obtaining manifest: https://index.docker.io/v2/poldracklab/mriqc/manifests/latest\r\nDEBUG GET https://index.docker.io/v2/poldracklab/mriqc/manifests/latest\r\n```\r\nYours has the old version 1 fields, which have `fsLayers` instead of `layers` like:\r\n\r\n```\r\n{'architecture': 'amd64',\r\n 'fsLayers': [{'blobSum': 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'},\r\n...\r\n```\r\nand we can verify version 1 of the schema:\r\n\r\n```\r\nmanifest['schemaVersion']\r\n1\r\n```\r\nversus what version 2 looks like, [like this](https://docs.docker.com/registry/spec/manifest-v2-2/) and was added [here](https://github.com/moby/moby/commit/9f482a66ab37ec396ac61ed0c00d59122ac07453).\r\n\r\nThe reason this is important is because the newer schema returns information about the layer size. It should look something like this (this is for ubuntu):\r\n\r\n```\r\nIn [3]: client.get_manifest()\r\nVERBOSE Obtaining manifest: https://index.docker.io/v2/library/ubuntu/manifests/latest\r\nDEBUG GET https://index.docker.io/v2/library/ubuntu/manifests/latest\r\nOut[3]: \r\n{'config': {'digest': 'sha256:f7b3f317ec734a73deca91b34c2b1e3dd7454650da9c8ef3047d29a873865178',\r\n  'mediaType': 'application/vnd.docker.container.image.v1+json',\r\n  'size': 3624},\r\n 'layers': [{'digest': 'sha256:aafe6b5e13de557451e1781fe7276620275625f970015cbd10036ab7d8ae27c0',\r\n   'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',\r\n   'size': 46792902},\r\n  {'digest': 'sha256:0a2b43a726608d3835aa027bbe181624789130c212eb191baa481f1d788a0676',\r\n   'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',\r\n   'size': 815},\r\n  {'digest': 'sha256:18bdd1e546d21240bd4c3e4e29032a38f18102d780a78686508d40fe01c60158',\r\n   'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',\r\n   'size': 514},\r\n  {'digest': 'sha256:8198342c3e05101bb56b0a585dc34e39346ece39ee5d39287cda24b1fb445fed',\r\n   'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',\r\n   'size': 850},\r\n  {'digest': 'sha256:f56970a44fd409805e3872cac236d29f5eed1a45e189bb765bcd2163c915f384',\r\n   'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip',\r\n   'size': 161}],\r\n 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json',\r\n 'schemaVersion': 2}\r\n```\r\n\r\nI am trying to get a feature added to Singularity 2.3 so you can do:\r\n\r\n```\r\nsingularity pull docker://poldracklab/mriqc\r\n```\r\n\r\nand it will both `create` and `import` docker layers into a `poldracklab-mriqc-latest.img` (or one that you name specifically):\r\n\r\n```\r\nsingularity pull --name myprecious.img docker://poldracklab/mriqc\r\n```\r\n\r\nGiven that you have the version 2 manifest, I can get the size automatically, and the container is generated without needing it. Given that I can't get this manifest, we can only return the default Singularity image size, or you would need to specify it:\r\n\r\n\r\n```\r\nsingularity pull --size 4000 docker://poldracklab/mriqc\r\n```\r\n\r\nwhich is more annoying of the two use cases, as it would vary per image! We could also get the size of the compressed layers from the content header (while it's stream downloading to the system), but then this would only work given that you don't already have the layers cached. We also can't get the uncompressed size from the last 4 bytes in the file because this is 32 bit, so it [doesn't work](http://stackoverflow.com/questions/2712173/check-the-total-content-size-of-a-tar-gz-file) for files greater than 4GB. We could also just estimate with some other hack on the fly, but the higher powers above me (and I agree) would like this to come from a reliable (single) place and not rely on the user's python, tar, etc.\r\n\r\nSo my next logical question was - why is this image delivering a Manifest Schema 1?  I first checked out the [builds](https://hub.docker.com/r/poldracklab/mriqc/builds/) on Docker Hub, and they are over 6 months old! But duh, this is because you've been building and pushing from circle (makes sense, for testing right!) and those are [pushed during CI here](https://github.com/poldracklab/mriqc/blob/master/circle.yml#L69).\r\n\r\nI then poked into the old manifest being served, and from the history, it looks like the v1Compatibility json (history) stops at April of this year:\r\n\r\n```\r\nfor layer in manifest['history']: print(json.loads(layer['v1Compatibili\r\n    ...: ty'])['created'])\r\n\r\n2017-04-23T18:22:38.555180752Z\r\n2017-04-23T18:22:28.997826306Z\r\n2017-04-23T18:22:19.759039782Z\r\n2017-04-23T18:22:10.947418041Z\r\n2017-04-23T18:22:01.07861345Z\r\n2017-04-23T18:21:54.31756029Z\r\n2017-04-23T18:21:30.692066584Z\r\n2017-04-23T18:21:24.244138728Z\r\n2017-04-23T18:21:17.426181353Z\r\n2017-04-23T18:20:44.055114417Z\r\n2017-04-23T18:20:37.553078338Z\r\n2017-04-23T18:19:39.267013266Z\r\n2017-04-23T18:19:33.433588271Z\r\n2017-04-23T18:19:21.2173366Z\r\n2017-04-23T18:19:10.912841181Z\r\n2017-04-23T18:16:10.119395228Z\r\n2017-04-23T18:16:07.072211516Z\r\n2017-04-23T18:15:52.52128285Z\r\n2017-04-23T18:15:46.009745722Z\r\n2017-04-23T18:15:36.445782669Z\r\n2017-04-23T18:15:21.588705635Z\r\n2017-04-23T18:15:18.094309164Z\r\n2017-04-23T18:15:15.196973388Z\r\n2017-04-23T18:15:11.113514794Z\r\n2017-04-23T18:12:23.393168016Z\r\n2017-04-23T18:12:20.789563151Z\r\n2017-04-23T18:11:25.65503471Z\r\n2017-04-23T18:10:49.379840044Z\r\n2016-12-15T17:45:24.24960419Z\r\n2016-12-15T17:45:23.840105252Z\r\n2016-12-15T17:45:22.900373296Z\r\n2016-12-15T17:45:22.024921767Z\r\n2016-12-15T17:45:21.123443146Z\r\n2016-12-15T17:45:19.987827436Z\r\n```\r\n\r\nThat was largely just the wrong rabbit hole **jumps out**. My next thought was that it must be something to do with the build process - if the version of Docker on circle doesn't push a thing that can generate a version schema 2 manifest, then it would fall back to 1. You are installing [1.9.1](https://github.com/poldracklab/mriqc/blob/master/circle.yml#L3). And after digging a bit in the abysmal hole of blue sadness that is the Docker documentation base, I found [this tidbit](https://docs.docker.com/registry/compatibility/#registry-v23):\r\n\r\n>> Manifest Push with Docker 1.9 and Older\r\n>> The Docker Engine will construct a Schema 1 manifest which the registry will persist to disk.\r\n>> When the manifest is pulled by digest or tag with any docker version, a Schema 1 manifest will be returned.\r\n\r\nso **TLDR**: if you are able to update your Docker on Circle (1.10 or up), this would be really great, because images will be served from Docker Hub that can tell users (the Singularity client being one of them!) how fat they are (let's be honest, Poldracklab images are VERY fat) and then with a **TBD** pull command, it will just create the right image size, instead of failing and then trying to shove a really fat cat into a tiny box, which only results in kittastrophe."}}
{"actor":{"display_login":"oesteban"}}