{"action":"created","issue":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/612","repository_url":"https://api.github.com/repos/flux-framework/flux-core","labels_url":"https://api.github.com/repos/flux-framework/flux-core/issues/612/labels{/name}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/612/comments","events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/612/events","html_url":"https://github.com/flux-framework/flux-core/issues/612","id":142809809,"number":612,"title":"Intermittent failure of waitfile.lua","user":{"login":"lipari","id":740218,"avatar_url":"https://avatars.githubusercontent.com/u/740218?v=3","gravatar_id":"","url":"https://api.github.com/users/lipari","html_url":"https://github.com/lipari","followers_url":"https://api.github.com/users/lipari/followers","following_url":"https://api.github.com/users/lipari/following{/other_user}","gists_url":"https://api.github.com/users/lipari/gists{/gist_id}","starred_url":"https://api.github.com/users/lipari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lipari/subscriptions","organizations_url":"https://api.github.com/users/lipari/orgs","repos_url":"https://api.github.com/users/lipari/repos","events_url":"https://api.github.com/users/lipari/events{/privacy}","received_events_url":"https://api.github.com/users/lipari/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2016-03-22T23:25:03Z","updated_at":"2016-03-23T21:56:17Z","closed_at":null,"body":"I have been tracking down the intermittent failures of two sched tests:  https://github.com/flux-framework/flux-sched/issues/134 and https://github.com/flux-framework/flux-sched/issues/142.  I believe they have the same ultimate cause:  the waitfile.lua script times out detecting a valid file.  First off, these failures are very rare - maybe once in 50 times - at least when running on a local machine.  But when they occur, they follow a similar pattern for both of the above tests:\r\n\r\n* flux-waitjob writes its wo.st.<id> file - just before the job is submitted.\r\n* waitfile.lua is kicked off with a 5 second timeout statusing that file every 0.25 seconds\r\n* The file is not initially detected and waitfile.lua emits, `No file yet`\r\n* the 5 second timeout is breached.\r\n\r\nThe only telltale artifact is that when  waitfile.lua successfully detects a the presence of the file after the first 0.25 second cycle, it emits, `wakeup`.  In the failure mode, this statement is missing.  Instead, there is nothing written until the 5 second timeout elapses:\r\n```\r\nwaitfile: 0.001s: No file yet\r\nwaitfile: 5.001s: Timeout on wo.st.1 after 5s\r\n-rw------- 1 lipari lipari 0 Mar 22 22:54 wo.st.1\r\n\r\n```\r\nAs you know, waitfile.lua runs `flux_stat_watcher` under the covers and it could be the ultimate cause.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments/200559381","html_url":"https://github.com/flux-framework/flux-core/issues/612#issuecomment-200559381","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/612","id":200559381,"user":{"login":"dongahn","id":862123,"avatar_url":"https://avatars.githubusercontent.com/u/862123?v=3","gravatar_id":"","url":"https://api.github.com/users/dongahn","html_url":"https://github.com/dongahn","followers_url":"https://api.github.com/users/dongahn/followers","following_url":"https://api.github.com/users/dongahn/following{/other_user}","gists_url":"https://api.github.com/users/dongahn/gists{/gist_id}","starred_url":"https://api.github.com/users/dongahn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dongahn/subscriptions","organizations_url":"https://api.github.com/users/dongahn/orgs","repos_url":"https://api.github.com/users/dongahn/repos","events_url":"https://api.github.com/users/dongahn/events{/privacy}","received_events_url":"https://api.github.com/users/dongahn/received_events","type":"User","site_admin":false},"created_at":"2016-03-23T21:56:17Z","updated_at":"2016-03-23T21:56:17Z","body":"+1"}}