{"action":"created","issue":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/772","repository_url":"https://api.github.com/repos/flux-framework/flux-core","labels_url":"https://api.github.com/repos/flux-framework/flux-core/issues/772/labels{/name}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/772/comments","events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/772/events","html_url":"https://github.com/flux-framework/flux-core/issues/772","id":171536918,"number":772,"title":"flux-wreckrun loses lwj.state callbacks","user":{"login":"grondo","id":741970,"avatar_url":"https://avatars.githubusercontent.com/u/741970?v=3","gravatar_id":"","url":"https://api.github.com/users/grondo","html_url":"https://github.com/grondo","followers_url":"https://api.github.com/users/grondo/followers","following_url":"https://api.github.com/users/grondo/following{/other_user}","gists_url":"https://api.github.com/users/grondo/gists{/gist_id}","starred_url":"https://api.github.com/users/grondo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grondo/subscriptions","organizations_url":"https://api.github.com/users/grondo/orgs","repos_url":"https://api.github.com/users/grondo/repos","events_url":"https://api.github.com/users/grondo/events{/privacy}","received_events_url":"https://api.github.com/users/grondo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-08-16T22:54:12Z","updated_at":"2016-08-17T03:28:35Z","closed_at":null,"body":"Running verbose, flux-wreckrun should print job state update for each lwj.state transition `reserved->starting->running->complete` -- however, some of these are missing in larger runs:\r\n\r\nA working 512 task case:\r\n```\r\n$ time flux wreckrun -v -n512 /bin/true\r\nwreckrun: 0.005s: Sending LWJ request for 512 tasks (cmdline \"/bin/true\")\r\nwreckrun: 0.012s: Registered jobid 1\r\nwreckrun: Allocating 512 tasks across 2304 available nodes..\r\nwreckrun: tasks per node: node[0-511]: 1\r\nwreckrun: 0.109s: Sending run event\r\nwreckrun: 0.302s: State = reserved\r\nwreckrun: 5.084s: State = starting\r\nwreckrun: 5.637s: State = running\r\nwreckrun: 6.300s: State = complete\r\nwreckrun: tasks [0-511]: exited with exit code 0\r\nwreckrun: All tasks completed successfully.\r\n```\r\n\r\nvs a suspicious run of the same size:\r\n```\r\n$ time flux wreckrun -v -n512 /bin/true\r\nwreckrun: 0.004s: Sending LWJ request for 512 tasks (cmdline \"/bin/true\")\r\nwreckrun: 0.009s: Registered jobid 2\r\nwreckrun: Allocating 512 tasks across 2304 available nodes..\r\nwreckrun: tasks per node: node[0-511]: 1\r\nwreckrun: 0.109s: Sending run event\r\nwreckrun: 2.697s: State = complete\r\nwreckrun: tasks [0-511]: exited with exit code 0\r\nwreckrun: All tasks completed successfully.\r\n```"},"comment":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments/240304200","html_url":"https://github.com/flux-framework/flux-core/issues/772#issuecomment-240304200","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/772","id":240304200,"user":{"login":"grondo","id":741970,"avatar_url":"https://avatars.githubusercontent.com/u/741970?v=3","gravatar_id":"","url":"https://api.github.com/users/grondo","html_url":"https://github.com/grondo","followers_url":"https://api.github.com/users/grondo/followers","following_url":"https://api.github.com/users/grondo/following{/other_user}","gists_url":"https://api.github.com/users/grondo/gists{/gist_id}","starred_url":"https://api.github.com/users/grondo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grondo/subscriptions","organizations_url":"https://api.github.com/users/grondo/orgs","repos_url":"https://api.github.com/users/grondo/repos","events_url":"https://api.github.com/users/grondo/events{/privacy}","received_events_url":"https://api.github.com/users/grondo/received_events","type":"User","site_admin":false},"created_at":"2016-08-17T03:28:35Z","updated_at":"2016-08-17T03:28:35Z","body":"Sorry guys, I didn't mean to leave this vague issue for you to speculate on! I just wanted to capture what I was seeing before heading out for first week of school shenanigans.\r\n\r\n@garlick, I didn't mean to imply that the reactor was dropping the state changes, but more generally flux-wreckrun or lua bindings is losing them, sorry for the confusion.\r\n\r\nI actually think @garlick was onto something when he asked if lua bindings use `kvs_watch_once` vs `kvs_watch`, and I think they may. That is especially bad when the reactor is busy because the callback may be sitting for awhile without being handled and this is when state transitions could be missed.\r\n\r\nRegardless, moving the `kvs_watch` call up in `flux-wreckrun` is definitely required, but probably also a band-aid to the flux-lua bindings"}}
{"actor":{"display_login":"grondo"}}