{"action":"created","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360/events","html_url":"https://github.com/cBio/cbio-cluster/issues/360","id":125702509,"number":360,"title":"Docker image storage policy on Hal","user":{"login":"lzamparo","id":1569186,"avatar_url":"https://avatars.githubusercontent.com/u/1569186?v=3","gravatar_id":"","url":"https://api.github.com/users/lzamparo","html_url":"https://github.com/lzamparo","followers_url":"https://api.github.com/users/lzamparo/followers","following_url":"https://api.github.com/users/lzamparo/following{/other_user}","gists_url":"https://api.github.com/users/lzamparo/gists{/gist_id}","starred_url":"https://api.github.com/users/lzamparo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lzamparo/subscriptions","organizations_url":"https://api.github.com/users/lzamparo/orgs","repos_url":"https://api.github.com/users/lzamparo/repos","events_url":"https://api.github.com/users/lzamparo/events{/privacy}","received_events_url":"https://api.github.com/users/lzamparo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-01-08T21:54:19Z","updated_at":"2016-01-17T15:54:31Z","closed_at":null,"body":"(migrated discussion from [msk-hpc github discussion](https://github.com/juanperin/mskcc-hpc/issues/2))\r\n\r\nPrevious docker-centric cbio-cluster issues (#288, #358) suggest that we need to come up with a new policy for GPFS-wide docker image storage.  It is a critical part of @tatarsky's proposed temporary fix for the `docker pull` problem (#358). If you want the full story, please see the linked issues.  \r\n\r\n#### Short summary:\r\n\r\n[Docker hub](https://forums.docker.com/t/docker-pull-deprecation-for-for-v1-5-clients/5071) has deprecated the version of docker that is installed on Hal.  So, our docker client can no longer interact with docker hub. To install a newer version, we need to upgrade the kernel, and possibly the OS on participating nodes.  This is a very complex solution to a simpler problem.  \r\n\r\nAs an interim solution to preserve docker functionality while we debate the best policy for upgrading the OS on existing nodes, we should set up a centralized \"root area\" for docker images that is shared across all nodes via GPFS.  \r\n\r\n#### What's involved in the proposed solution:\r\n\r\nSetting up a GPFS-wide docker root area is not simple.  Quoting @tatarsky:\r\n\r\n> That root area consists of more than just images. It is the area where I believe much of the container state is also kept so I don't think you can just toss the entire region into a shared GPFS directory. Or at least I would have to experiment with that and I can sort of see there is going to be collisions if I do.\r\n\r\n>My brief reading seems to point to the use of a local repository if a large collection of images and their local modifications is required. With its backend storage then on some shared media. This requires a daemon be run and there appear to be two projects with implementations.\r\n\r\n>All this would be considered an enhancement but I said I would look into it and so here is the start of my trying to understand the requirements to move forward.\r\n\r\n@jchodera supports trying out a centralized root area: \r\n\r\n>I like the idea of an interim solution, but I think it might be easier to just designate a GPFS shared directory (/cbio/shared/docker?).\r\n\r\n@tatarsky agrees, and has specific suggestions for how this should be attempted:\r\n\r\n>However I agree generically with the points above and asked in cBio/cbio-cluster#358 if people were ready to give the shared GPFS docker dir a try. I'd rather not turn a compute node into a file server. If that was truly needed because GPFS didn't work I would suggest a head node be used for that role. (Several are coming).\r\n\r\n>Also it would I think be wiser to do initial tests of that GPFS docker spool on a system not actively running other peoples jobs (as cpu-6-1 and cpu-6-2 are busy busy machines) and I will need to prep the new kernel on a system out of torque at the moment to test if it works. I do not have time for that from the road but can attempt it if I get some cycles waiting on installs at JHU. (gpu-3-8 and gpu-3-9 are good places to test this as they are connected for now to the new Torque/Moab head and not in the main scheduler path)\r\n\r\nIf any of the docker users would like to comment or have suggestions/concerns about how to roll out a centralized root area, please leave them here.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/comments/172343517","html_url":"https://github.com/cBio/cbio-cluster/issues/360#issuecomment-172343517","issue_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360","id":172343517,"user":{"login":"tatarsky","id":6439722,"avatar_url":"https://avatars.githubusercontent.com/u/6439722?v=3","gravatar_id":"","url":"https://api.github.com/users/tatarsky","html_url":"https://github.com/tatarsky","followers_url":"https://api.github.com/users/tatarsky/followers","following_url":"https://api.github.com/users/tatarsky/following{/other_user}","gists_url":"https://api.github.com/users/tatarsky/gists{/gist_id}","starred_url":"https://api.github.com/users/tatarsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatarsky/subscriptions","organizations_url":"https://api.github.com/users/tatarsky/orgs","repos_url":"https://api.github.com/users/tatarsky/repos","events_url":"https://api.github.com/users/tatarsky/events{/privacy}","received_events_url":"https://api.github.com/users/tatarsky/received_events","type":"User","site_admin":false},"created_at":"2016-01-17T15:54:31Z","updated_at":"2016-01-17T15:54:31Z","body":"Yep. Basically I don't think docker supports this concept. I've read a fair number of items about this before and its why I placed the items locally. There are some references to various other services (I'm looking at flocker at the moment) but I think the \"simply share the backend dirs\" isn't an option. \r\n\r\nI will continue to schedule the kernel update however so we can return to \"what we had before they deprecated the v1 repo\" even if you do have to re-pull your image. \r\n\r\nWe could consider a local repo if the speed of that was important from my other reading."}}