{"action":"created","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360/events","html_url":"https://github.com/cBio/cbio-cluster/issues/360","id":125702509,"number":360,"title":"Docker image storage policy on Hal","user":{"login":"lzamparo","id":1569186,"avatar_url":"https://avatars.githubusercontent.com/u/1569186?v=3","gravatar_id":"","url":"https://api.github.com/users/lzamparo","html_url":"https://github.com/lzamparo","followers_url":"https://api.github.com/users/lzamparo/followers","following_url":"https://api.github.com/users/lzamparo/following{/other_user}","gists_url":"https://api.github.com/users/lzamparo/gists{/gist_id}","starred_url":"https://api.github.com/users/lzamparo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lzamparo/subscriptions","organizations_url":"https://api.github.com/users/lzamparo/orgs","repos_url":"https://api.github.com/users/lzamparo/repos","events_url":"https://api.github.com/users/lzamparo/events{/privacy}","received_events_url":"https://api.github.com/users/lzamparo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":32,"created_at":"2016-01-08T21:54:19Z","updated_at":"2016-01-18T15:58:35Z","closed_at":null,"body":"(migrated discussion from [msk-hpc github discussion](https://github.com/juanperin/mskcc-hpc/issues/2))\r\n\r\nPrevious docker-centric cbio-cluster issues (#288, #358) suggest that we need to come up with a new policy for GPFS-wide docker image storage.  It is a critical part of @tatarsky's proposed temporary fix for the `docker pull` problem (#358). If you want the full story, please see the linked issues.  \r\n\r\n#### Short summary:\r\n\r\n[Docker hub](https://forums.docker.com/t/docker-pull-deprecation-for-for-v1-5-clients/5071) has deprecated the version of docker that is installed on Hal.  So, our docker client can no longer interact with docker hub. To install a newer version, we need to upgrade the kernel, and possibly the OS on participating nodes.  This is a very complex solution to a simpler problem.  \r\n\r\nAs an interim solution to preserve docker functionality while we debate the best policy for upgrading the OS on existing nodes, we should set up a centralized \"root area\" for docker images that is shared across all nodes via GPFS.  \r\n\r\n#### What's involved in the proposed solution:\r\n\r\nSetting up a GPFS-wide docker root area is not simple.  Quoting @tatarsky:\r\n\r\n> That root area consists of more than just images. It is the area where I believe much of the container state is also kept so I don't think you can just toss the entire region into a shared GPFS directory. Or at least I would have to experiment with that and I can sort of see there is going to be collisions if I do.\r\n\r\n>My brief reading seems to point to the use of a local repository if a large collection of images and their local modifications is required. With its backend storage then on some shared media. This requires a daemon be run and there appear to be two projects with implementations.\r\n\r\n>All this would be considered an enhancement but I said I would look into it and so here is the start of my trying to understand the requirements to move forward.\r\n\r\n@jchodera supports trying out a centralized root area: \r\n\r\n>I like the idea of an interim solution, but I think it might be easier to just designate a GPFS shared directory (/cbio/shared/docker?).\r\n\r\n@tatarsky agrees, and has specific suggestions for how this should be attempted:\r\n\r\n>However I agree generically with the points above and asked in cBio/cbio-cluster#358 if people were ready to give the shared GPFS docker dir a try. I'd rather not turn a compute node into a file server. If that was truly needed because GPFS didn't work I would suggest a head node be used for that role. (Several are coming).\r\n\r\n>Also it would I think be wiser to do initial tests of that GPFS docker spool on a system not actively running other peoples jobs (as cpu-6-1 and cpu-6-2 are busy busy machines) and I will need to prep the new kernel on a system out of torque at the moment to test if it works. I do not have time for that from the road but can attempt it if I get some cycles waiting on installs at JHU. (gpu-3-8 and gpu-3-9 are good places to test this as they are connected for now to the new Torque/Moab head and not in the main scheduler path)\r\n\r\nIf any of the docker users would like to comment or have suggestions/concerns about how to roll out a centralized root area, please leave them here.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/comments/172569421","html_url":"https://github.com/cBio/cbio-cluster/issues/360#issuecomment-172569421","issue_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/360","id":172569421,"user":{"login":"tatarsky","id":6439722,"avatar_url":"https://avatars.githubusercontent.com/u/6439722?v=3","gravatar_id":"","url":"https://api.github.com/users/tatarsky","html_url":"https://github.com/tatarsky","followers_url":"https://api.github.com/users/tatarsky/followers","following_url":"https://api.github.com/users/tatarsky/following{/other_user}","gists_url":"https://api.github.com/users/tatarsky/gists{/gist_id}","starred_url":"https://api.github.com/users/tatarsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatarsky/subscriptions","organizations_url":"https://api.github.com/users/tatarsky/orgs","repos_url":"https://api.github.com/users/tatarsky/repos","events_url":"https://api.github.com/users/tatarsky/events{/privacy}","received_events_url":"https://api.github.com/users/tatarsky/received_events","type":"User","site_admin":false},"created_at":"2016-01-18T15:58:35Z","updated_at":"2016-01-18T15:58:35Z","body":"Three more nodes upgraded this morning following drain: `gpu-2-9 gpu-2-12 gpu-2-14`. \r\nThe `docker` attribute added to their node resource list. \r\nCasting about for some more to offline. "}}