{"action":"created","issue":{"url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles/issues/163","repository_url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles","labels_url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles/issues/163/labels{/name}","comments_url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles/issues/163/comments","events_url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles/issues/163/events","html_url":"https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/163","id":138516808,"number":163,"title":"problems running DM jobs at scale at SLAC","user":{"login":"tony-johnson","id":11215747,"avatar_url":"https://avatars.githubusercontent.com/u/11215747?v=3","gravatar_id":"","url":"https://api.github.com/users/tony-johnson","html_url":"https://github.com/tony-johnson","followers_url":"https://api.github.com/users/tony-johnson/followers","following_url":"https://api.github.com/users/tony-johnson/following{/other_user}","gists_url":"https://api.github.com/users/tony-johnson/gists{/gist_id}","starred_url":"https://api.github.com/users/tony-johnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tony-johnson/subscriptions","organizations_url":"https://api.github.com/users/tony-johnson/orgs","repos_url":"https://api.github.com/users/tony-johnson/repos","events_url":"https://api.github.com/users/tony-johnson/events{/privacy}","received_events_url":"https://api.github.com/users/tony-johnson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"SimonKrughoff","id":945715,"avatar_url":"https://avatars.githubusercontent.com/u/945715?v=3","gravatar_id":"","url":"https://api.github.com/users/SimonKrughoff","html_url":"https://github.com/SimonKrughoff","followers_url":"https://api.github.com/users/SimonKrughoff/followers","following_url":"https://api.github.com/users/SimonKrughoff/following{/other_user}","gists_url":"https://api.github.com/users/SimonKrughoff/gists{/gist_id}","starred_url":"https://api.github.com/users/SimonKrughoff/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SimonKrughoff/subscriptions","organizations_url":"https://api.github.com/users/SimonKrughoff/orgs","repos_url":"https://api.github.com/users/SimonKrughoff/repos","events_url":"https://api.github.com/users/SimonKrughoff/events{/privacy}","received_events_url":"https://api.github.com/users/SimonKrughoff/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2016-03-04T16:26:23Z","updated_at":"2016-03-04T22:30:38Z","closed_at":null,"body":"We are currently writing the output from DM jobs to NFS space (specifically I am using /nfs/farm/g/lsst/u1/users/tonyj/). When running hundreds of jobs at once I am seeing a significant fraction (maybe 5%) failing with:\r\n\r\nIOError: [Errno 116] Stale file handle\r\n\r\nMy guess is that this is caused by multiple jobs updating the registry.sqlite3 file. It is known that file locking is not reliable on NFS 3 mounted disks which may be causing this.\r\n\r\nA example output file can be found here:\r\n\r\nhttp://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/log.jsp?pi=40767767\r\n\r\nOur normal workaround for this kind of problem is to have jobs write their output to local scratch space, and copy the output to the final location at the end of the job. I don't think DM jobs are readily amenable to this kind of fix. Another fix might be to try finding a different output area (lucene or gpfs)?\r\n\r\nThis is hopefully not fatal, since the failed jobs can be rerun (assuming that the jobs that appear to run are not in fact corrupting the registry) but it does significantly slow down processing."},"comment":{"url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles/issues/comments/192500992","html_url":"https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/163#issuecomment-192500992","issue_url":"https://api.github.com/repos/DarkEnergyScienceCollaboration/Twinkles/issues/163","id":192500992,"user":{"login":"tony-johnson","id":11215747,"avatar_url":"https://avatars.githubusercontent.com/u/11215747?v=3","gravatar_id":"","url":"https://api.github.com/users/tony-johnson","html_url":"https://github.com/tony-johnson","followers_url":"https://api.github.com/users/tony-johnson/followers","following_url":"https://api.github.com/users/tony-johnson/following{/other_user}","gists_url":"https://api.github.com/users/tony-johnson/gists{/gist_id}","starred_url":"https://api.github.com/users/tony-johnson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tony-johnson/subscriptions","organizations_url":"https://api.github.com/users/tony-johnson/orgs","repos_url":"https://api.github.com/users/tony-johnson/repos","events_url":"https://api.github.com/users/tony-johnson/events{/privacy}","received_events_url":"https://api.github.com/users/tony-johnson/received_events","type":"User","site_admin":false},"created_at":"2016-03-04T22:30:38Z","updated_at":"2016-03-04T22:30:38Z","body":"I may have been mistaken in thinking that the issue was related to the registry.sqlite file, it was just that the stack dump immiediately followed the message about opening the registry.\r\n\r\nCameraMapper: Loading registry registry from /nfs/farm/g/lsst/u1/users/tonyj/work/0/output/_parent/registry.sqlite3\r\nTraceback (most recent call last):\r\n  File \"/nfs/farm/g/desc/u1/Twinkles/obs_lsstSim/bin/processEimage.py\", line 25, in <module>\r\n    ProcessEimageTask.parseAndRun()\r\n\r\nI think I will try using lustre as an experiment anyway, to see if it fixes the issue whatever it is."}}