{"action":"closed","number":1472,"pull_request":{"url":"https://api.github.com/repos/flux-framework/flux-core/pulls/1472","id":182341461,"html_url":"https://github.com/flux-framework/flux-core/pull/1472","diff_url":"https://github.com/flux-framework/flux-core/pull/1472.diff","patch_url":"https://github.com/flux-framework/flux-core/pull/1472.patch","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/1472","number":1472,"state":"closed","locked":false,"title":"wreck: rework job.submit, job.create, wrexecd.run handlers with continuations","user":{"login":"garlick","id":169947,"avatar_url":"https://avatars1.githubusercontent.com/u/169947?v=4","gravatar_id":"","url":"https://api.github.com/users/garlick","html_url":"https://github.com/garlick","followers_url":"https://api.github.com/users/garlick/followers","following_url":"https://api.github.com/users/garlick/following{/other_user}","gists_url":"https://api.github.com/users/garlick/gists{/gist_id}","starred_url":"https://api.github.com/users/garlick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garlick/subscriptions","organizations_url":"https://api.github.com/users/garlick/orgs","repos_url":"https://api.github.com/users/garlick/repos","events_url":"https://api.github.com/users/garlick/events{/privacy}","received_events_url":"https://api.github.com/users/garlick/received_events","type":"User","site_admin":false},"body":"This is a hopefully somewhat better structured version of PR #1471.\r\n\r\nThis one doesn't add a hash of active jobs and doesn't limt the job module's ability to ingest jobs on ranks other than 0.\r\n\r\nIt adds a `struct wjob` data structure (with unit tests), and reworks the `job.submit`, `job.create`, and `wrexecd.run` request handlers to manipulate the KVS etc using continuations  rather than synchronous RPC's, to improve job module responsiveness.\r\n\r\nIt also satisfies a long standing feature request for an ability to synchronously publish events; that is, get a response indicating that the event has received a sequence number, as discussed in #337 and #342.  That was necessary to remove the final synchronous step in job.create and job.submit, where the handlers block listening for an event to be published before sending the response to the user.\r\n\r\nFWIW I reran a little script inspired by @trws that that submits 50K jobs (wtih the sched module check disabled), running 200 submits in parallel:\r\n```sh\r\n#!/bin/bash\r\nulimit 65536\r\nseq 1 50000 | xargs -n 1 -P 200  \\\r\n    flux submit -n 1 sleep 0\r\n```\r\nBefore this PR I got 134 job per second.  After I get 442.\r\n\r\nI haven't tried to probe the effect on job launch, but it should improve latency under load - for example if there are many small jobs being launched, the overhead of checking a new job _not_ targetting the local rank should not slow down (as much) the launch of one that _does_ target the local rank, since new events can start to be handled before the last one has completed.  E.g. there could be many fetches of `R_lite` for different jobs occurring simultaneously on each rank.\r\n\r\nSorry the \"rework\" commits are basically unreadable.  They each effectively rewrite a handler and its helper functions as a chain of continuations.  It might be better to look at the final result.","created_at":"2018-04-18T00:57:01Z","updated_at":"2018-04-18T19:45:39Z","closed_at":"2018-04-18T19:45:39Z","merged_at":"2018-04-18T19:45:39Z","merge_commit_sha":"cb2f8db1e5be51ff4df3fa6c123c92a0ca6861d6","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/flux-framework/flux-core/pulls/1472/commits","review_comments_url":"https://api.github.com/repos/flux-framework/flux-core/pulls/1472/comments","review_comment_url":"https://api.github.com/repos/flux-framework/flux-core/pulls/comments{/number}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/1472/comments","statuses_url":"https://api.github.com/repos/flux-framework/flux-core/statuses/4914855d4f0fa75a4c312c9c33c465000e04efca","head":{"label":"garlick:job_state2","ref":"job_state2","sha":"4914855d4f0fa75a4c312c9c33c465000e04efca","user":{"login":"garlick","id":169947,"avatar_url":"https://avatars1.githubusercontent.com/u/169947?v=4","gravatar_id":"","url":"https://api.github.com/users/garlick","html_url":"https://github.com/garlick","followers_url":"https://api.github.com/users/garlick/followers","following_url":"https://api.github.com/users/garlick/following{/other_user}","gists_url":"https://api.github.com/users/garlick/gists{/gist_id}","starred_url":"https://api.github.com/users/garlick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garlick/subscriptions","organizations_url":"https://api.github.com/users/garlick/orgs","repos_url":"https://api.github.com/users/garlick/repos","events_url":"https://api.github.com/users/garlick/events{/privacy}","received_events_url":"https://api.github.com/users/garlick/received_events","type":"User","site_admin":false},"repo":{"id":24518180,"name":"flux-core","full_name":"garlick/flux-core","owner":{"login":"garlick","id":169947,"avatar_url":"https://avatars1.githubusercontent.com/u/169947?v=4","gravatar_id":"","url":"https://api.github.com/users/garlick","html_url":"https://github.com/garlick","followers_url":"https://api.github.com/users/garlick/followers","following_url":"https://api.github.com/users/garlick/following{/other_user}","gists_url":"https://api.github.com/users/garlick/gists{/gist_id}","starred_url":"https://api.github.com/users/garlick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/garlick/subscriptions","organizations_url":"https://api.github.com/users/garlick/orgs","repos_url":"https://api.github.com/users/garlick/repos","events_url":"https://api.github.com/users/garlick/events{/privacy}","received_events_url":"https://api.github.com/users/garlick/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/garlick/flux-core","description":"core  services for the Flux resource management framework","fork":true,"url":"https://api.github.com/repos/garlick/flux-core","forks_url":"https://api.github.com/repos/garlick/flux-core/forks","keys_url":"https://api.github.com/repos/garlick/flux-core/keys{/key_id}","collaborators_url":"https://api.github.com/repos/garlick/flux-core/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/garlick/flux-core/teams","hooks_url":"https://api.github.com/repos/garlick/flux-core/hooks","issue_events_url":"https://api.github.com/repos/garlick/flux-core/issues/events{/number}","events_url":"https://api.github.com/repos/garlick/flux-core/events","assignees_url":"https://api.github.com/repos/garlick/flux-core/assignees{/user}","branches_url":"https://api.github.com/repos/garlick/flux-core/branches{/branch}","tags_url":"https://api.github.com/repos/garlick/flux-core/tags","blobs_url":"https://api.github.com/repos/garlick/flux-core/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/garlick/flux-core/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/garlick/flux-core/git/refs{/sha}","trees_url":"https://api.github.com/repos/garlick/flux-core/git/trees{/sha}","statuses_url":"https://api.github.com/repos/garlick/flux-core/statuses/{sha}","languages_url":"https://api.github.com/repos/garlick/flux-core/languages","stargazers_url":"https://api.github.com/repos/garlick/flux-core/stargazers","contributors_url":"https://api.github.com/repos/garlick/flux-core/contributors","subscribers_url":"https://api.github.com/repos/garlick/flux-core/subscribers","subscription_url":"https://api.github.com/repos/garlick/flux-core/subscription","commits_url":"https://api.github.com/repos/garlick/flux-core/commits{/sha}","git_commits_url":"https://api.github.com/repos/garlick/flux-core/git/commits{/sha}","comments_url":"https://api.github.com/repos/garlick/flux-core/comments{/number}","issue_comment_url":"https://api.github.com/repos/garlick/flux-core/issues/comments{/number}","contents_url":"https://api.github.com/repos/garlick/flux-core/contents/{+path}","compare_url":"https://api.github.com/repos/garlick/flux-core/compare/{base}...{head}","merges_url":"https://api.github.com/repos/garlick/flux-core/merges","archive_url":"https://api.github.com/repos/garlick/flux-core/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/garlick/flux-core/downloads","issues_url":"https://api.github.com/repos/garlick/flux-core/issues{/number}","pulls_url":"https://api.github.com/repos/garlick/flux-core/pulls{/number}","milestones_url":"https://api.github.com/repos/garlick/flux-core/milestones{/number}","notifications_url":"https://api.github.com/repos/garlick/flux-core/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/garlick/flux-core/labels{/name}","releases_url":"https://api.github.com/repos/garlick/flux-core/releases{/id}","deployments_url":"https://api.github.com/repos/garlick/flux-core/deployments","created_at":"2014-09-26T23:15:54Z","updated_at":"2018-04-07T14:23:12Z","pushed_at":"2018-04-18T18:17:14Z","git_url":"git://github.com/garlick/flux-core.git","ssh_url":"git@github.com:garlick/flux-core.git","clone_url":"https://github.com/garlick/flux-core.git","svn_url":"https://github.com/garlick/flux-core","homepage":"","size":28204,"stargazers_count":0,"watchers_count":0,"language":"C","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"gpl-2.0","name":"GNU General Public License v2.0","spdx_id":"GPL-2.0","url":"https://api.github.com/licenses/gpl-2.0"},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"flux-framework:master","ref":"master","sha":"c644b7f7d3cf6fa41a676cd2b21afbcbdaff2c61","user":{"login":"flux-framework","id":7545483,"avatar_url":"https://avatars2.githubusercontent.com/u/7545483?v=4","gravatar_id":"","url":"https://api.github.com/users/flux-framework","html_url":"https://github.com/flux-framework","followers_url":"https://api.github.com/users/flux-framework/followers","following_url":"https://api.github.com/users/flux-framework/following{/other_user}","gists_url":"https://api.github.com/users/flux-framework/gists{/gist_id}","starred_url":"https://api.github.com/users/flux-framework/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flux-framework/subscriptions","organizations_url":"https://api.github.com/users/flux-framework/orgs","repos_url":"https://api.github.com/users/flux-framework/repos","events_url":"https://api.github.com/users/flux-framework/events{/privacy}","received_events_url":"https://api.github.com/users/flux-framework/received_events","type":"Organization","site_admin":false},"repo":{"id":24517284,"name":"flux-core","full_name":"flux-framework/flux-core","owner":{"login":"flux-framework","id":7545483,"avatar_url":"https://avatars2.githubusercontent.com/u/7545483?v=4","gravatar_id":"","url":"https://api.github.com/users/flux-framework","html_url":"https://github.com/flux-framework","followers_url":"https://api.github.com/users/flux-framework/followers","following_url":"https://api.github.com/users/flux-framework/following{/other_user}","gists_url":"https://api.github.com/users/flux-framework/gists{/gist_id}","starred_url":"https://api.github.com/users/flux-framework/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flux-framework/subscriptions","organizations_url":"https://api.github.com/users/flux-framework/orgs","repos_url":"https://api.github.com/users/flux-framework/repos","events_url":"https://api.github.com/users/flux-framework/events{/privacy}","received_events_url":"https://api.github.com/users/flux-framework/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/flux-framework/flux-core","description":"core  services for the Flux resource management framework","fork":false,"url":"https://api.github.com/repos/flux-framework/flux-core","forks_url":"https://api.github.com/repos/flux-framework/flux-core/forks","keys_url":"https://api.github.com/repos/flux-framework/flux-core/keys{/key_id}","collaborators_url":"https://api.github.com/repos/flux-framework/flux-core/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/flux-framework/flux-core/teams","hooks_url":"https://api.github.com/repos/flux-framework/flux-core/hooks","issue_events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/events{/number}","events_url":"https://api.github.com/repos/flux-framework/flux-core/events","assignees_url":"https://api.github.com/repos/flux-framework/flux-core/assignees{/user}","branches_url":"https://api.github.com/repos/flux-framework/flux-core/branches{/branch}","tags_url":"https://api.github.com/repos/flux-framework/flux-core/tags","blobs_url":"https://api.github.com/repos/flux-framework/flux-core/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/flux-framework/flux-core/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/flux-framework/flux-core/git/refs{/sha}","trees_url":"https://api.github.com/repos/flux-framework/flux-core/git/trees{/sha}","statuses_url":"https://api.github.com/repos/flux-framework/flux-core/statuses/{sha}","languages_url":"https://api.github.com/repos/flux-framework/flux-core/languages","stargazers_url":"https://api.github.com/repos/flux-framework/flux-core/stargazers","contributors_url":"https://api.github.com/repos/flux-framework/flux-core/contributors","subscribers_url":"https://api.github.com/repos/flux-framework/flux-core/subscribers","subscription_url":"https://api.github.com/repos/flux-framework/flux-core/subscription","commits_url":"https://api.github.com/repos/flux-framework/flux-core/commits{/sha}","git_commits_url":"https://api.github.com/repos/flux-framework/flux-core/git/commits{/sha}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/comments{/number}","issue_comment_url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments{/number}","contents_url":"https://api.github.com/repos/flux-framework/flux-core/contents/{+path}","compare_url":"https://api.github.com/repos/flux-framework/flux-core/compare/{base}...{head}","merges_url":"https://api.github.com/repos/flux-framework/flux-core/merges","archive_url":"https://api.github.com/repos/flux-framework/flux-core/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/flux-framework/flux-core/downloads","issues_url":"https://api.github.com/repos/flux-framework/flux-core/issues{/number}","pulls_url":"https://api.github.com/repos/flux-framework/flux-core/pulls{/number}","milestones_url":"https://api.github.com/repos/flux-framework/flux-core/milestones{/number}","notifications_url":"https://api.github.com/repos/flux-framework/flux-core/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/flux-framework/flux-core/labels{/name}","releases_url":"https://api.github.com/repos/flux-framework/flux-core/releases{/id}","deployments_url":"https://api.github.com/repos/flux-framework/flux-core/deployments","created_at":"2014-09-26T22:27:19Z","updated_at":"2018-04-16T16:43:52Z","pushed_at":"2018-04-18T19:45:39Z","git_url":"git://github.com/flux-framework/flux-core.git","ssh_url":"git@github.com:flux-framework/flux-core.git","clone_url":"https://github.com/flux-framework/flux-core.git","svn_url":"https://github.com/flux-framework/flux-core","homepage":"","size":27991,"stargazers_count":32,"watchers_count":32,"language":"C","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":16,"mirror_url":null,"archived":false,"open_issues_count":182,"license":{"key":"gpl-2.0","name":"GNU General Public License v2.0","spdx_id":"GPL-2.0","url":"https://api.github.com/licenses/gpl-2.0"},"forks":16,"open_issues":182,"watchers":32,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/flux-framework/flux-core/pulls/1472"},"html":{"href":"https://github.com/flux-framework/flux-core/pull/1472"},"issue":{"href":"https://api.github.com/repos/flux-framework/flux-core/issues/1472"},"comments":{"href":"https://api.github.com/repos/flux-framework/flux-core/issues/1472/comments"},"review_comments":{"href":"https://api.github.com/repos/flux-framework/flux-core/pulls/1472/comments"},"review_comment":{"href":"https://api.github.com/repos/flux-framework/flux-core/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/flux-framework/flux-core/pulls/1472/commits"},"statuses":{"href":"https://api.github.com/repos/flux-framework/flux-core/statuses/4914855d4f0fa75a4c312c9c33c465000e04efca"}},"author_association":"OWNER","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"grondo","id":741970,"avatar_url":"https://avatars3.githubusercontent.com/u/741970?v=4","gravatar_id":"","url":"https://api.github.com/users/grondo","html_url":"https://github.com/grondo","followers_url":"https://api.github.com/users/grondo/followers","following_url":"https://api.github.com/users/grondo/following{/other_user}","gists_url":"https://api.github.com/users/grondo/gists{/gist_id}","starred_url":"https://api.github.com/users/grondo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grondo/subscriptions","organizations_url":"https://api.github.com/users/grondo/orgs","repos_url":"https://api.github.com/users/grondo/repos","events_url":"https://api.github.com/users/grondo/events{/privacy}","received_events_url":"https://api.github.com/users/grondo/received_events","type":"User","site_admin":false},"comments":12,"review_comments":22,"maintainer_can_modify":false,"commits":6,"additions":646,"deletions":250,"changed_files":6}}
{"actor":{"display_login":"grondo"}}