{"action":"opened","issue":{"url":"https://api.github.com/repos/ngageoint/scale/issues/196","repository_url":"https://api.github.com/repos/ngageoint/scale","labels_url":"https://api.github.com/repos/ngageoint/scale/issues/196/labels{/name}","comments_url":"https://api.github.com/repos/ngageoint/scale/issues/196/comments","events_url":"https://api.github.com/repos/ngageoint/scale/issues/196/events","html_url":"https://github.com/ngageoint/scale/issues/196","id":151397356,"number":196,"title":"DR: NFS is easily overwhelmed by large clusters with many jobs with many mounts","user":{"login":"cshamis","id":15897164,"avatar_url":"https://avatars.githubusercontent.com/u/15897164?v=3","gravatar_id":"","url":"https://api.github.com/users/cshamis","html_url":"https://github.com/cshamis","followers_url":"https://api.github.com/users/cshamis/followers","following_url":"https://api.github.com/users/cshamis/following{/other_user}","gists_url":"https://api.github.com/users/cshamis/gists{/gist_id}","starred_url":"https://api.github.com/users/cshamis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cshamis/subscriptions","organizations_url":"https://api.github.com/users/cshamis/orgs","repos_url":"https://api.github.com/users/cshamis/repos","events_url":"https://api.github.com/users/cshamis/events{/privacy}","received_events_url":"https://api.github.com/users/cshamis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-27T14:16:54Z","updated_at":"2016-04-27T14:16:54Z","closed_at":null,"body":"When there are a great many nodes running many jobs, each with a live connection to a NFS workspace the NFS server can become overwhelmed and start timing out or hanging.  Increasing the number of NFS daemons helps, but is not sustainable.  Other, more scalable, approaches will be necessary in the future.  Off the top of my head, these options may include (but are not limited to) Lustre, HDFS, pNFS, WebDAV, S3, and Ceph.  Hybrid solutions involving FTP, SCP, HTTP, BBCP, Docker DataContainers, or a just a more \"restrained\" use of NFS may be required.\r\n\r\nI would like to start the discussion on finding a suitable replacement for our heavy reliance of NFS, and so we can have an implementation plan ready by the start of next sprint.\r\n"}}