{"action":"created","comment":{"url":"https://api.github.com/repos/xapi-project/xenopsd/pulls/comments/175778376","pull_request_review_id":105362958,"id":175778376,"diff_hunk":"@@ -0,0 +1,44 @@\n+(* -*- tuareg -*- *)\n+#require \"unix\"\n+\n+let flags = function\n+  | [] -> \"\"\n+  | pkgs ->\n+    let cmd = \"ocamlfind ocamlc -verbose\" ^ (\n+        List.fold_left (fun acc pkg -> acc ^ \" -package \" ^ pkg) \"\" pkgs\n+      ) in\n+    let ic = Unix.open_process_in\n+        (cmd ^ \" | grep -oEe '-ppx (\\\"([^\\\"\\\\]|\\\\.)+\\\"|\\\\w+)'\")\n+    in\n+    let rec go ic acc =\n+      try go ic (acc ^ \" \" ^ input_line ic) with End_of_file -> close_in ic; acc\n+    in\n+    go ic \"\"\n+\n+let flags = flags [\"ppx_deriving_rpc\"]\n+\n+let coverage_rewriter =\n+  let is_coverage = try Unix.getenv \"BISECT_ENABLE\" = \"YES\" with Not_found -> false in\n+  if is_coverage then\n+    \"(preprocess (pps (bisect_ppx -conditional)))\"\n+  else\n+    \"\"\n+\n+let () = Printf.ksprintf Jbuild_plugin.V1.send {|\n+(jbuild_version 1)\n+\n+(executable (\n+  (name main)\n+  (libraries (alcotest\n+              xapi-stdext-pervasives","path":"xapi-work-queues/test/jbuild","position":33,"original_position":33,"commit_id":"47eb128ef88b04fb49aa5e6dd69ab3108486b770","original_commit_id":"47eb128ef88b04fb49aa5e6dd69ab3108486b770","user":{"login":"mseri","id":4040623,"avatar_url":"https://avatars0.githubusercontent.com/u/4040623?v=4","gravatar_id":"","url":"https://api.github.com/users/mseri","html_url":"https://github.com/mseri","followers_url":"https://api.github.com/users/mseri/followers","following_url":"https://api.github.com/users/mseri/following{/other_user}","gists_url":"https://api.github.com/users/mseri/gists{/gist_id}","starred_url":"https://api.github.com/users/mseri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mseri/subscriptions","organizations_url":"https://api.github.com/users/mseri/orgs","repos_url":"https://api.github.com/users/mseri/repos","events_url":"https://api.github.com/users/mseri/events{/privacy}","received_events_url":"https://api.github.com/users/mseri/received_events","type":"User","site_admin":false},"body":"This did also need `xapi-stdext-threads`","created_at":"2018-03-20T14:10:10Z","updated_at":"2018-03-20T14:10:10Z","html_url":"https://github.com/xapi-project/xenopsd/pull/337#discussion_r175778376","pull_request_url":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/xapi-project/xenopsd/pulls/comments/175778376"},"html":{"href":"https://github.com/xapi-project/xenopsd/pull/337#discussion_r175778376"},"pull_request":{"href":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337"}}},"pull_request":{"url":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337","id":120601056,"html_url":"https://github.com/xapi-project/xenopsd/pull/337","diff_url":"https://github.com/xapi-project/xenopsd/pull/337.diff","patch_url":"https://github.com/xapi-project/xenopsd/pull/337.patch","issue_url":"https://api.github.com/repos/xapi-project/xenopsd/issues/337","number":337,"state":"closed","locked":false,"title":"Refactor work queues","user":{"login":"edwintorok","id":721894,"avatar_url":"https://avatars1.githubusercontent.com/u/721894?v=4","gravatar_id":"","url":"https://api.github.com/users/edwintorok","html_url":"https://github.com/edwintorok","followers_url":"https://api.github.com/users/edwintorok/followers","following_url":"https://api.github.com/users/edwintorok/following{/other_user}","gists_url":"https://api.github.com/users/edwintorok/gists{/gist_id}","starred_url":"https://api.github.com/users/edwintorok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edwintorok/subscriptions","organizations_url":"https://api.github.com/users/edwintorok/orgs","repos_url":"https://api.github.com/users/edwintorok/repos","events_url":"https://api.github.com/users/edwintorok/events{/privacy}","received_events_url":"https://api.github.com/users/edwintorok/received_events","type":"User","site_admin":false},"body":"Currently XAPI has a single thread that processes all events from xenopsd. This can lead to a significant delay between the VM running, and XAPI reporting its state as running, e.g. you can receive a UDP packet from the VM, but XAPI still doesn't report it as running for minutes (CA-87377).\r\n\r\nThe [fix for that](\r\nhttps://github.com/edwintorok/xen-api/commit/b367bf86d3af4f773db9bf5d1500a4dec0f99bfa#diff-344dc1d17c4663add7fe5500813feef2) parallelizes event processing on a per-VM basis.\r\n\r\nTo avoid duplicating code the `WorkerPool` from `xenopsd` should be shared between `xenopsd` and `xapi`. In order to do that we should remove any `xenopsd` specific details from it.\r\nInitially I thought of doing that with an `'a Item.t` , but there were multiple operations that had to be performed on it, essentialy ending up with a record threaded through everything, like a home grown functor. So I turned this into into a proper functor (see [its documentation here](https://github.com/xapi-project/xenopsd/pull/337/files#diff-08ea6b223a2a8da3d28bac7c572e32a2)):\r\n * the work items submitted should be abstract, the WorkerPool shouldn't have any knowledge of its internals (`Item.t`)\r\n * the item to be executed should be as generic as possible, i.e. it should be able to pass just a `unit -> unit` to be executed by the workers, all the [`xenopsd` specific task execution](https://github.com/xapi-project/xenopsd/pull/337/commits/54ae6d6f5c8d171bdb438bbc9b21f977cc81e22a#diff-469ce4709f9a70be7fae148c604342d1L612) is moved to `execute` and `finally`.\r\n* the semantics of `execute` and `finally` are documented in the interface (in short: `execute` and `finally` are always called on an `Item.t` to run them, exceptions are logged and ignored)\r\n* `should_keep` is specific to xenopsd, used for removing certain tasks if they are already present in the queue\r\n* there are some functions used only by debugging/diagnostics, e.g. `describe_item`, `describe_task`.\r\n\r\nMost of the internal functions of the redirector and worker pool are hidden, except:\r\n * setting the size of the worker pool (which you must do in order to have the jobs execute at all)\r\n * submitting jobs to the pool\r\n\r\nThere are 2 global queues in xenopsd (`default` and `parallel`), which would've been nice to avoid, however there is more global state in the redirector (the `overrides` map), which would've made this PR even larger.\r\n\r\nThe scheduling algorithm is briefly explained [here](https://github.com/xapi-project/xenopsd/pull/337/files#diff-08ea6b223a2a8da3d28bac7c572e32a2R68), in the case of `xapi` the  tags will be the VM ids for example.\r\n\r\nThis PR refactors xenopsd to extract the worker pool as a separate file, independent of `xenopsd`-specific implementation details, i.e. as a functor. The functionality and scheduling algorithm should be unaffected by the refactoring, this PR should be a no-op in terms of runtime visible change, except tiny differences in the debug output.\r\nI've also added a new test to exercise the new worker pool code paths.\r\n\r\nA second PR will move the code (`xapi_work_queues.ml{,i}` and `xwq_test.ml`) to a separate repository `xapi-work-queues` and update `xenopsd` and `xen-api` to use that.\r\n\r\nThe `xen-api` instance of the worker pool functor will look like this:\r\nhttps://github.com/edwintorok/xen-api/commit/8efe9e0e136b9e3b10738668eb89c0c0295c9184#diff-344dc1d17c4663add7fe5500813feef2R1931\r\n\r\nI tried to keep commits that functorize (or refactor in preparation for that) separate from the commits that just move code around to make it easier to follow.","created_at":"2017-05-15T13:26:40Z","updated_at":"2018-03-20T14:10:10Z","closed_at":"2018-03-19T16:24:42Z","merged_at":"2018-03-19T16:24:42Z","merge_commit_sha":"0e59785a262de58687b13e38a5672504e81654a8","assignee":null,"assignees":[],"requested_reviewers":[{"login":"jonludlam","id":210963,"avatar_url":"https://avatars3.githubusercontent.com/u/210963?v=4","gravatar_id":"","url":"https://api.github.com/users/jonludlam","html_url":"https://github.com/jonludlam","followers_url":"https://api.github.com/users/jonludlam/followers","following_url":"https://api.github.com/users/jonludlam/following{/other_user}","gists_url":"https://api.github.com/users/jonludlam/gists{/gist_id}","starred_url":"https://api.github.com/users/jonludlam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonludlam/subscriptions","organizations_url":"https://api.github.com/users/jonludlam/orgs","repos_url":"https://api.github.com/users/jonludlam/repos","events_url":"https://api.github.com/users/jonludlam/events{/privacy}","received_events_url":"https://api.github.com/users/jonludlam/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":792908493,"url":"https://api.github.com/repos/xapi-project/xenopsd/labels/needs%20review","name":"needs review","color":"fbca04","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337/commits","review_comments_url":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337/comments","review_comment_url":"https://api.github.com/repos/xapi-project/xenopsd/pulls/comments{/number}","comments_url":"https://api.github.com/repos/xapi-project/xenopsd/issues/337/comments","statuses_url":"https://api.github.com/repos/xapi-project/xenopsd/statuses/47eb128ef88b04fb49aa5e6dd69ab3108486b770","head":{"label":"edwintorok:refactor-work-queues","ref":"refactor-work-queues","sha":"47eb128ef88b04fb49aa5e6dd69ab3108486b770","user":{"login":"edwintorok","id":721894,"avatar_url":"https://avatars1.githubusercontent.com/u/721894?v=4","gravatar_id":"","url":"https://api.github.com/users/edwintorok","html_url":"https://github.com/edwintorok","followers_url":"https://api.github.com/users/edwintorok/followers","following_url":"https://api.github.com/users/edwintorok/following{/other_user}","gists_url":"https://api.github.com/users/edwintorok/gists{/gist_id}","starred_url":"https://api.github.com/users/edwintorok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edwintorok/subscriptions","organizations_url":"https://api.github.com/users/edwintorok/orgs","repos_url":"https://api.github.com/users/edwintorok/repos","events_url":"https://api.github.com/users/edwintorok/events{/privacy}","received_events_url":"https://api.github.com/users/edwintorok/received_events","type":"User","site_admin":false},"repo":{"id":88852402,"name":"xenopsd","full_name":"edwintorok/xenopsd","owner":{"login":"edwintorok","id":721894,"avatar_url":"https://avatars1.githubusercontent.com/u/721894?v=4","gravatar_id":"","url":"https://api.github.com/users/edwintorok","html_url":"https://github.com/edwintorok","followers_url":"https://api.github.com/users/edwintorok/followers","following_url":"https://api.github.com/users/edwintorok/following{/other_user}","gists_url":"https://api.github.com/users/edwintorok/gists{/gist_id}","starred_url":"https://api.github.com/users/edwintorok/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edwintorok/subscriptions","organizations_url":"https://api.github.com/users/edwintorok/orgs","repos_url":"https://api.github.com/users/edwintorok/repos","events_url":"https://api.github.com/users/edwintorok/events{/privacy}","received_events_url":"https://api.github.com/users/edwintorok/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/edwintorok/xenopsd","description":"XCP toolstack domain manager","fork":true,"url":"https://api.github.com/repos/edwintorok/xenopsd","forks_url":"https://api.github.com/repos/edwintorok/xenopsd/forks","keys_url":"https://api.github.com/repos/edwintorok/xenopsd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/edwintorok/xenopsd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/edwintorok/xenopsd/teams","hooks_url":"https://api.github.com/repos/edwintorok/xenopsd/hooks","issue_events_url":"https://api.github.com/repos/edwintorok/xenopsd/issues/events{/number}","events_url":"https://api.github.com/repos/edwintorok/xenopsd/events","assignees_url":"https://api.github.com/repos/edwintorok/xenopsd/assignees{/user}","branches_url":"https://api.github.com/repos/edwintorok/xenopsd/branches{/branch}","tags_url":"https://api.github.com/repos/edwintorok/xenopsd/tags","blobs_url":"https://api.github.com/repos/edwintorok/xenopsd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/edwintorok/xenopsd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/edwintorok/xenopsd/git/refs{/sha}","trees_url":"https://api.github.com/repos/edwintorok/xenopsd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/edwintorok/xenopsd/statuses/{sha}","languages_url":"https://api.github.com/repos/edwintorok/xenopsd/languages","stargazers_url":"https://api.github.com/repos/edwintorok/xenopsd/stargazers","contributors_url":"https://api.github.com/repos/edwintorok/xenopsd/contributors","subscribers_url":"https://api.github.com/repos/edwintorok/xenopsd/subscribers","subscription_url":"https://api.github.com/repos/edwintorok/xenopsd/subscription","commits_url":"https://api.github.com/repos/edwintorok/xenopsd/commits{/sha}","git_commits_url":"https://api.github.com/repos/edwintorok/xenopsd/git/commits{/sha}","comments_url":"https://api.github.com/repos/edwintorok/xenopsd/comments{/number}","issue_comment_url":"https://api.github.com/repos/edwintorok/xenopsd/issues/comments{/number}","contents_url":"https://api.github.com/repos/edwintorok/xenopsd/contents/{+path}","compare_url":"https://api.github.com/repos/edwintorok/xenopsd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/edwintorok/xenopsd/merges","archive_url":"https://api.github.com/repos/edwintorok/xenopsd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/edwintorok/xenopsd/downloads","issues_url":"https://api.github.com/repos/edwintorok/xenopsd/issues{/number}","pulls_url":"https://api.github.com/repos/edwintorok/xenopsd/pulls{/number}","milestones_url":"https://api.github.com/repos/edwintorok/xenopsd/milestones{/number}","notifications_url":"https://api.github.com/repos/edwintorok/xenopsd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/edwintorok/xenopsd/labels{/name}","releases_url":"https://api.github.com/repos/edwintorok/xenopsd/releases{/id}","deployments_url":"https://api.github.com/repos/edwintorok/xenopsd/deployments","created_at":"2017-04-20T10:18:25Z","updated_at":"2017-04-20T10:18:27Z","pushed_at":"2018-03-19T15:09:16Z","git_url":"git://github.com/edwintorok/xenopsd.git","ssh_url":"git@github.com:edwintorok/xenopsd.git","clone_url":"https://github.com/edwintorok/xenopsd.git","svn_url":"https://github.com/edwintorok/xenopsd","homepage":null,"size":32783,"stargazers_count":0,"watchers_count":0,"language":"OCaml","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"xapi-project:master","ref":"master","sha":"38cbe76425e13d7b2b42ac94a7eddf6482368bec","user":{"login":"xapi-project","id":317268,"avatar_url":"https://avatars3.githubusercontent.com/u/317268?v=4","gravatar_id":"","url":"https://api.github.com/users/xapi-project","html_url":"https://github.com/xapi-project","followers_url":"https://api.github.com/users/xapi-project/followers","following_url":"https://api.github.com/users/xapi-project/following{/other_user}","gists_url":"https://api.github.com/users/xapi-project/gists{/gist_id}","starred_url":"https://api.github.com/users/xapi-project/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xapi-project/subscriptions","organizations_url":"https://api.github.com/users/xapi-project/orgs","repos_url":"https://api.github.com/users/xapi-project/repos","events_url":"https://api.github.com/users/xapi-project/events{/privacy}","received_events_url":"https://api.github.com/users/xapi-project/received_events","type":"Organization","site_admin":false},"repo":{"id":7563740,"name":"xenopsd","full_name":"xapi-project/xenopsd","owner":{"login":"xapi-project","id":317268,"avatar_url":"https://avatars3.githubusercontent.com/u/317268?v=4","gravatar_id":"","url":"https://api.github.com/users/xapi-project","html_url":"https://github.com/xapi-project","followers_url":"https://api.github.com/users/xapi-project/followers","following_url":"https://api.github.com/users/xapi-project/following{/other_user}","gists_url":"https://api.github.com/users/xapi-project/gists{/gist_id}","starred_url":"https://api.github.com/users/xapi-project/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xapi-project/subscriptions","organizations_url":"https://api.github.com/users/xapi-project/orgs","repos_url":"https://api.github.com/users/xapi-project/repos","events_url":"https://api.github.com/users/xapi-project/events{/privacy}","received_events_url":"https://api.github.com/users/xapi-project/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/xapi-project/xenopsd","description":"XCP toolstack domain manager","fork":false,"url":"https://api.github.com/repos/xapi-project/xenopsd","forks_url":"https://api.github.com/repos/xapi-project/xenopsd/forks","keys_url":"https://api.github.com/repos/xapi-project/xenopsd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/xapi-project/xenopsd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/xapi-project/xenopsd/teams","hooks_url":"https://api.github.com/repos/xapi-project/xenopsd/hooks","issue_events_url":"https://api.github.com/repos/xapi-project/xenopsd/issues/events{/number}","events_url":"https://api.github.com/repos/xapi-project/xenopsd/events","assignees_url":"https://api.github.com/repos/xapi-project/xenopsd/assignees{/user}","branches_url":"https://api.github.com/repos/xapi-project/xenopsd/branches{/branch}","tags_url":"https://api.github.com/repos/xapi-project/xenopsd/tags","blobs_url":"https://api.github.com/repos/xapi-project/xenopsd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/xapi-project/xenopsd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/xapi-project/xenopsd/git/refs{/sha}","trees_url":"https://api.github.com/repos/xapi-project/xenopsd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/xapi-project/xenopsd/statuses/{sha}","languages_url":"https://api.github.com/repos/xapi-project/xenopsd/languages","stargazers_url":"https://api.github.com/repos/xapi-project/xenopsd/stargazers","contributors_url":"https://api.github.com/repos/xapi-project/xenopsd/contributors","subscribers_url":"https://api.github.com/repos/xapi-project/xenopsd/subscribers","subscription_url":"https://api.github.com/repos/xapi-project/xenopsd/subscription","commits_url":"https://api.github.com/repos/xapi-project/xenopsd/commits{/sha}","git_commits_url":"https://api.github.com/repos/xapi-project/xenopsd/git/commits{/sha}","comments_url":"https://api.github.com/repos/xapi-project/xenopsd/comments{/number}","issue_comment_url":"https://api.github.com/repos/xapi-project/xenopsd/issues/comments{/number}","contents_url":"https://api.github.com/repos/xapi-project/xenopsd/contents/{+path}","compare_url":"https://api.github.com/repos/xapi-project/xenopsd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/xapi-project/xenopsd/merges","archive_url":"https://api.github.com/repos/xapi-project/xenopsd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/xapi-project/xenopsd/downloads","issues_url":"https://api.github.com/repos/xapi-project/xenopsd/issues{/number}","pulls_url":"https://api.github.com/repos/xapi-project/xenopsd/pulls{/number}","milestones_url":"https://api.github.com/repos/xapi-project/xenopsd/milestones{/number}","notifications_url":"https://api.github.com/repos/xapi-project/xenopsd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/xapi-project/xenopsd/labels{/name}","releases_url":"https://api.github.com/repos/xapi-project/xenopsd/releases{/id}","deployments_url":"https://api.github.com/repos/xapi-project/xenopsd/deployments","created_at":"2013-01-11T17:19:31Z","updated_at":"2018-03-20T10:55:26Z","pushed_at":"2018-03-20T10:55:24Z","git_url":"git://github.com/xapi-project/xenopsd.git","ssh_url":"git@github.com:xapi-project/xenopsd.git","clone_url":"https://github.com/xapi-project/xenopsd.git","svn_url":"https://github.com/xapi-project/xenopsd","homepage":null,"size":33288,"stargazers_count":7,"watchers_count":7,"language":"OCaml","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":56,"mirror_url":null,"archived":false,"open_issues_count":10,"license":{"key":"other","name":"Other","spdx_id":null,"url":null},"forks":56,"open_issues":10,"watchers":7,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337"},"html":{"href":"https://github.com/xapi-project/xenopsd/pull/337"},"issue":{"href":"https://api.github.com/repos/xapi-project/xenopsd/issues/337"},"comments":{"href":"https://api.github.com/repos/xapi-project/xenopsd/issues/337/comments"},"review_comments":{"href":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337/comments"},"review_comment":{"href":"https://api.github.com/repos/xapi-project/xenopsd/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/xapi-project/xenopsd/pulls/337/commits"},"statuses":{"href":"https://api.github.com/repos/xapi-project/xenopsd/statuses/47eb128ef88b04fb49aa5e6dd69ab3108486b770"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"mseri"}}