{"action":"created","issue":{"url":"https://api.github.com/repos/TIGRLab/admin/issues/766","repository_url":"https://api.github.com/repos/TIGRLab/admin","labels_url":"https://api.github.com/repos/TIGRLab/admin/issues/766/labels{/name}","comments_url":"https://api.github.com/repos/TIGRLab/admin/issues/766/comments","events_url":"https://api.github.com/repos/TIGRLab/admin/issues/766/events","html_url":"https://github.com/TIGRLab/admin/issues/766","id":179580994,"number":766,"title":"queue jobs that write logs over NFS fail (sometimes)","user":{"login":"josephdviviano","id":4142570,"avatar_url":"https://avatars.githubusercontent.com/u/4142570?v=3","gravatar_id":"","url":"https://api.github.com/users/josephdviviano","html_url":"https://github.com/josephdviviano","followers_url":"https://api.github.com/users/josephdviviano/followers","following_url":"https://api.github.com/users/josephdviviano/following{/other_user}","gists_url":"https://api.github.com/users/josephdviviano/gists{/gist_id}","starred_url":"https://api.github.com/users/josephdviviano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josephdviviano/subscriptions","organizations_url":"https://api.github.com/users/josephdviviano/orgs","repos_url":"https://api.github.com/users/josephdviviano/repos","events_url":"https://api.github.com/users/josephdviviano/events{/privacy}","received_events_url":"https://api.github.com/users/josephdviviano/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"josephdviviano","id":4142570,"avatar_url":"https://avatars.githubusercontent.com/u/4142570?v=3","gravatar_id":"","url":"https://api.github.com/users/josephdviviano","html_url":"https://github.com/josephdviviano","followers_url":"https://api.github.com/users/josephdviviano/followers","following_url":"https://api.github.com/users/josephdviviano/following{/other_user}","gists_url":"https://api.github.com/users/josephdviviano/gists{/gist_id}","starred_url":"https://api.github.com/users/josephdviviano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josephdviviano/subscriptions","organizations_url":"https://api.github.com/users/josephdviviano/orgs","repos_url":"https://api.github.com/users/josephdviviano/repos","events_url":"https://api.github.com/users/josephdviviano/events{/privacy}","received_events_url":"https://api.github.com/users/josephdviviano/received_events","type":"User","site_admin":false},"assignees":[{"login":"josephdviviano","id":4142570,"avatar_url":"https://avatars.githubusercontent.com/u/4142570?v=3","gravatar_id":"","url":"https://api.github.com/users/josephdviviano","html_url":"https://github.com/josephdviviano","followers_url":"https://api.github.com/users/josephdviviano/followers","following_url":"https://api.github.com/users/josephdviviano/following{/other_user}","gists_url":"https://api.github.com/users/josephdviviano/gists{/gist_id}","starred_url":"https://api.github.com/users/josephdviviano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josephdviviano/subscriptions","organizations_url":"https://api.github.com/users/josephdviviano/orgs","repos_url":"https://api.github.com/users/josephdviviano/repos","events_url":"https://api.github.com/users/josephdviviano/events{/privacy}","received_events_url":"https://api.github.com/users/josephdviviano/received_events","type":"User","site_admin":false},{"login":"tomwright01","id":7861803,"avatar_url":"https://avatars.githubusercontent.com/u/7861803?v=3","gravatar_id":"","url":"https://api.github.com/users/tomwright01","html_url":"https://github.com/tomwright01","followers_url":"https://api.github.com/users/tomwright01/followers","following_url":"https://api.github.com/users/tomwright01/following{/other_user}","gists_url":"https://api.github.com/users/tomwright01/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwright01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwright01/subscriptions","organizations_url":"https://api.github.com/users/tomwright01/orgs","repos_url":"https://api.github.com/users/tomwright01/repos","events_url":"https://api.github.com/users/tomwright01/events{/privacy}","received_events_url":"https://api.github.com/users/tomwright01/received_events","type":"User","site_admin":false},{"login":"DESm1th","id":10541496,"avatar_url":"https://avatars.githubusercontent.com/u/10541496?v=3","gravatar_id":"","url":"https://api.github.com/users/DESm1th","html_url":"https://github.com/DESm1th","followers_url":"https://api.github.com/users/DESm1th/followers","following_url":"https://api.github.com/users/DESm1th/following{/other_user}","gists_url":"https://api.github.com/users/DESm1th/gists{/gist_id}","starred_url":"https://api.github.com/users/DESm1th/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DESm1th/subscriptions","organizations_url":"https://api.github.com/users/DESm1th/orgs","repos_url":"https://api.github.com/users/DESm1th/repos","events_url":"https://api.github.com/users/DESm1th/events{/privacy}","received_events_url":"https://api.github.com/users/DESm1th/received_events","type":"User","site_admin":false},{"login":"djoverton","id":20860851,"avatar_url":"https://avatars.githubusercontent.com/u/20860851?v=3","gravatar_id":"","url":"https://api.github.com/users/djoverton","html_url":"https://github.com/djoverton","followers_url":"https://api.github.com/users/djoverton/followers","following_url":"https://api.github.com/users/djoverton/following{/other_user}","gists_url":"https://api.github.com/users/djoverton/gists{/gist_id}","starred_url":"https://api.github.com/users/djoverton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djoverton/subscriptions","organizations_url":"https://api.github.com/users/djoverton/orgs","repos_url":"https://api.github.com/users/djoverton/repos","events_url":"https://api.github.com/users/djoverton/events{/privacy}","received_events_url":"https://api.github.com/users/djoverton/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2016-09-27T18:53:01Z","updated_at":"2016-09-28T15:58:15Z","closed_at":null,"body":"Looks like, if you ask SGE to write your logs out on an NFS mount, a good deal of your jobs will randomly fail.\r\n\r\nI dont think this is because the NFS mount is inaccessible, because all of the data the pipelines run on also require NFS, but moving the logs to a local file (e.g., `/tmp`) resolves the issue.\r\n\r\nWhat's likely happening is some sort of overloading of the network when 100s of jobs are deployed simultaneously, where NFS is rendered unresponsive and SGE complains that the directory is inaccessible.\r\n\r\nFor now, the best solution is to ensure your grid jobs (particularly those in `datman`) are only writing logs to local folders."},"comment":{"url":"https://api.github.com/repos/TIGRLab/admin/issues/comments/250211803","html_url":"https://github.com/TIGRLab/admin/issues/766#issuecomment-250211803","issue_url":"https://api.github.com/repos/TIGRLab/admin/issues/766","id":250211803,"user":{"login":"josephdviviano","id":4142570,"avatar_url":"https://avatars.githubusercontent.com/u/4142570?v=3","gravatar_id":"","url":"https://api.github.com/users/josephdviviano","html_url":"https://github.com/josephdviviano","followers_url":"https://api.github.com/users/josephdviviano/followers","following_url":"https://api.github.com/users/josephdviviano/following{/other_user}","gists_url":"https://api.github.com/users/josephdviviano/gists{/gist_id}","starred_url":"https://api.github.com/users/josephdviviano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josephdviviano/subscriptions","organizations_url":"https://api.github.com/users/josephdviviano/orgs","repos_url":"https://api.github.com/users/josephdviviano/repos","events_url":"https://api.github.com/users/josephdviviano/events{/privacy}","received_events_url":"https://api.github.com/users/josephdviviano/received_events","type":"User","site_admin":false},"created_at":"2016-09-28T15:58:15Z","updated_at":"2016-09-28T15:58:15Z","body":"Solutions:\r\n\r\n+ Redirect standard out and standard error to either `/dev/null` or to `/localscratch`. \r\n+ Send your .o and .e files to /localscratch while the job is running, then copy them back to your home directory once the job has finished. \r\n+ If you are reading or writing to a large number of files during your high throughput jobs, add code to your script to first copy them under /localscratch, work with them there, and then copy any results back to your home directory upon job completion.  This will be job-dependent.  If you need help, let us know.\r\n\r\nI think the key is to reduce a large number of requests at the exact same time, which is most likely to happen when deploying a very large number of jobs to the queue and asking them all to open log files over NFS."}}
{"actor":{"display_login":"josephdviviano"}}