{"action":"created","comment":{"url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/comments/196379039","pull_request_review_id":129917787,"id":196379039,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5NjM3OTAzOQ==","diff_hunk":"@@ -0,0 +1,23 @@\n+# Evaluation of the KafkaStreamer for ESS\n+## Motivation\n+The KafkaStreamer has been previously developed to support live streaming of event data from a kafka broker into Mantid. The purpose of this document is to present an evaluation of the this streamer in the context of the ESS. The ESS will make use of a Kafka architecture which produces messages at a rate fixed to the frequency of the neutron source, 14Hz (frames per second). Therefore, any live streaming mechanism in Mantid should be able to support the consumption of kafka messages at or above 14Hz for sensible event detection rates. ESS instruments are expected to produce detection rates in the 10<sup>7</sup>Hz range or ~714286 events per kafka message.\n+\n+## Live Consumption Tests\n+The first stage of testing involved the scenario where the instrument data is inspected in the absence of any data reduction. A single workstation was used to run a docker image which contained the configuration for a Kafka broker. Data was published to the broker using the `main_nexusPublisher` which can be found [here](https://github.com/ess-dmsc/NeXus-Streamer). The `main_nexusPublisher` accepts event nexus files and publishes events to a kafka stream. This application also supports producing random events at user-defined rates, based on an input nexus file, which facilitated stress testing Mantid for our purposes. All instructions for the setup of the broker and using the `main_nexusPulisher` can be found in the link above.\n+\n+Three instruments with varying numbers of detectors were chosen for this investigation. Due, to the limitations of the current implementation of the `main_nexusPublisher` we were restricted to using ISIS nexus files. The selected instruments were:\n+\n+Instrument|Number of Detectors\n+---|---\n+SANS2D|122'888\n+MERLIN|286'729\n+WISH (10 panel)|778'245\n+\n+### Results\n+![Trend1](figure_1.png)\n+\n+The figure above shows the trend in event rate versus message consumption rate. The live listener GUI timeout was set to one second. SANS2D was able to maintain a 23Hz consumption rate at 10<sup>7</sup> event rate with a 1 second refresh rate of the live listener. Enabling the instrument view for inspection did not seem to affect the message consumption rate for any of the instruments. Both MERLIN and WISH were unable to support rates of 10<sup>7</sup> with consumption rates of 7Hz and 3hz respectively. At 10<sup>6</sup>, MERLIN consumed Kafka messages at a rate of 93Hz and WISH at 38Hz.\n+\n+![Trend2](figure_2.png)\n+\n+The above figure shows that increasing the gui timeout can result in a slight improvement in performance. This is however very limited and efforts to improve the listener design for performance should not depend too heavily on this.","path":"investigations/Live Reduction/LiveReductionInvestigation.md","position":23,"original_position":23,"commit_id":"a62dbb8d2682041f1c742cc54ad80522cebd19c2","original_commit_id":"a62dbb8d2682041f1c742cc54ad80522cebd19c2","user":{"login":"OwenArnold","id":1109536,"node_id":"MDQ6VXNlcjExMDk1MzY=","avatar_url":"https://avatars3.githubusercontent.com/u/1109536?v=4","gravatar_id":"","url":"https://api.github.com/users/OwenArnold","html_url":"https://github.com/OwenArnold","followers_url":"https://api.github.com/users/OwenArnold/followers","following_url":"https://api.github.com/users/OwenArnold/following{/other_user}","gists_url":"https://api.github.com/users/OwenArnold/gists{/gist_id}","starred_url":"https://api.github.com/users/OwenArnold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/OwenArnold/subscriptions","organizations_url":"https://api.github.com/users/OwenArnold/orgs","repos_url":"https://api.github.com/users/OwenArnold/repos","events_url":"https://api.github.com/users/OwenArnold/events{/privacy}","received_events_url":"https://api.github.com/users/OwenArnold/received_events","type":"User","site_admin":false},"body":"I think by \"GUI timeout\" you mean the `StarLiveData`/`MonitorLiveData` update frequency? Would be useful to clarify.","created_at":"2018-06-19T10:57:26Z","updated_at":"2018-06-19T11:02:20Z","html_url":"https://github.com/DMSC-Instrument-Data/documents/pull/24#discussion_r196379039","pull_request_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/comments/196379039"},"html":{"href":"https://github.com/DMSC-Instrument-Data/documents/pull/24#discussion_r196379039"},"pull_request":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24"}}},"pull_request":{"url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24","id":195754447,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk1NzU0NDQ3","html_url":"https://github.com/DMSC-Instrument-Data/documents/pull/24","diff_url":"https://github.com/DMSC-Instrument-Data/documents/pull/24.diff","patch_url":"https://github.com/DMSC-Instrument-Data/documents/pull/24.patch","issue_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/24","number":24,"state":"open","locked":false,"title":"Live Reduction Survey Notes","user":{"login":"LamarMoore","id":15360272,"node_id":"MDQ6VXNlcjE1MzYwMjcy","avatar_url":"https://avatars1.githubusercontent.com/u/15360272?v=4","gravatar_id":"","url":"https://api.github.com/users/LamarMoore","html_url":"https://github.com/LamarMoore","followers_url":"https://api.github.com/users/LamarMoore/followers","following_url":"https://api.github.com/users/LamarMoore/following{/other_user}","gists_url":"https://api.github.com/users/LamarMoore/gists{/gist_id}","starred_url":"https://api.github.com/users/LamarMoore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LamarMoore/subscriptions","organizations_url":"https://api.github.com/users/LamarMoore/orgs","repos_url":"https://api.github.com/users/LamarMoore/repos","events_url":"https://api.github.com/users/LamarMoore/events{/privacy}","received_events_url":"https://api.github.com/users/LamarMoore/received_events","type":"User","site_admin":false},"body":"Here are my notes on the survey of the Kafka Streamer in Mantid. This will be updated as I carry out more tests.","created_at":"2018-06-19T08:53:49Z","updated_at":"2018-06-19T11:02:20Z","closed_at":null,"merged_at":null,"merge_commit_sha":"05e4de6a2f885aca38908399ac59168e1552d098","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24/commits","review_comments_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24/comments","review_comment_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/comments{/number}","comments_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/24/comments","statuses_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/statuses/a62dbb8d2682041f1c742cc54ad80522cebd19c2","head":{"label":"DMSC-Instrument-Data:live_reduction_survey","ref":"live_reduction_survey","sha":"a62dbb8d2682041f1c742cc54ad80522cebd19c2","user":{"login":"DMSC-Instrument-Data","id":15102351,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTAyMzUx","avatar_url":"https://avatars0.githubusercontent.com/u/15102351?v=4","gravatar_id":"","url":"https://api.github.com/users/DMSC-Instrument-Data","html_url":"https://github.com/DMSC-Instrument-Data","followers_url":"https://api.github.com/users/DMSC-Instrument-Data/followers","following_url":"https://api.github.com/users/DMSC-Instrument-Data/following{/other_user}","gists_url":"https://api.github.com/users/DMSC-Instrument-Data/gists{/gist_id}","starred_url":"https://api.github.com/users/DMSC-Instrument-Data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DMSC-Instrument-Data/subscriptions","organizations_url":"https://api.github.com/users/DMSC-Instrument-Data/orgs","repos_url":"https://api.github.com/users/DMSC-Instrument-Data/repos","events_url":"https://api.github.com/users/DMSC-Instrument-Data/events{/privacy}","received_events_url":"https://api.github.com/users/DMSC-Instrument-Data/received_events","type":"Organization","site_admin":false},"repo":{"id":44163828,"node_id":"MDEwOlJlcG9zaXRvcnk0NDE2MzgyOA==","name":"documents","full_name":"DMSC-Instrument-Data/documents","owner":{"login":"DMSC-Instrument-Data","id":15102351,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTAyMzUx","avatar_url":"https://avatars0.githubusercontent.com/u/15102351?v=4","gravatar_id":"","url":"https://api.github.com/users/DMSC-Instrument-Data","html_url":"https://github.com/DMSC-Instrument-Data","followers_url":"https://api.github.com/users/DMSC-Instrument-Data/followers","following_url":"https://api.github.com/users/DMSC-Instrument-Data/following{/other_user}","gists_url":"https://api.github.com/users/DMSC-Instrument-Data/gists{/gist_id}","starred_url":"https://api.github.com/users/DMSC-Instrument-Data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DMSC-Instrument-Data/subscriptions","organizations_url":"https://api.github.com/users/DMSC-Instrument-Data/orgs","repos_url":"https://api.github.com/users/DMSC-Instrument-Data/repos","events_url":"https://api.github.com/users/DMSC-Instrument-Data/events{/privacy}","received_events_url":"https://api.github.com/users/DMSC-Instrument-Data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DMSC-Instrument-Data/documents","description":null,"fork":false,"url":"https://api.github.com/repos/DMSC-Instrument-Data/documents","forks_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/forks","keys_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/teams","hooks_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/hooks","issue_events_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/events{/number}","events_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/events","assignees_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/assignees{/user}","branches_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/branches{/branch}","tags_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/tags","blobs_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/refs{/sha}","trees_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/statuses/{sha}","languages_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/languages","stargazers_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/stargazers","contributors_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/contributors","subscribers_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/subscribers","subscription_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/subscription","commits_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/commits{/sha}","git_commits_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/commits{/sha}","comments_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/comments{/number}","issue_comment_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/comments{/number}","contents_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/contents/{+path}","compare_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/merges","archive_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/downloads","issues_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues{/number}","pulls_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls{/number}","milestones_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/milestones{/number}","notifications_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/labels{/name}","releases_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/releases{/id}","deployments_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/deployments","created_at":"2015-10-13T08:50:04Z","updated_at":"2018-06-14T08:29:24Z","pushed_at":"2018-06-19T08:53:50Z","git_url":"git://github.com/DMSC-Instrument-Data/documents.git","ssh_url":"git@github.com:DMSC-Instrument-Data/documents.git","clone_url":"https://github.com/DMSC-Instrument-Data/documents.git","svn_url":"https://github.com/DMSC-Instrument-Data/documents","homepage":null,"size":101588,"stargazers_count":1,"watchers_count":1,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":3,"license":null,"forks":3,"open_issues":3,"watchers":1,"default_branch":"master"}},"base":{"label":"DMSC-Instrument-Data:master","ref":"master","sha":"76d19fb1c16b7f0136c2989cf3fc80713120fe46","user":{"login":"DMSC-Instrument-Data","id":15102351,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTAyMzUx","avatar_url":"https://avatars0.githubusercontent.com/u/15102351?v=4","gravatar_id":"","url":"https://api.github.com/users/DMSC-Instrument-Data","html_url":"https://github.com/DMSC-Instrument-Data","followers_url":"https://api.github.com/users/DMSC-Instrument-Data/followers","following_url":"https://api.github.com/users/DMSC-Instrument-Data/following{/other_user}","gists_url":"https://api.github.com/users/DMSC-Instrument-Data/gists{/gist_id}","starred_url":"https://api.github.com/users/DMSC-Instrument-Data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DMSC-Instrument-Data/subscriptions","organizations_url":"https://api.github.com/users/DMSC-Instrument-Data/orgs","repos_url":"https://api.github.com/users/DMSC-Instrument-Data/repos","events_url":"https://api.github.com/users/DMSC-Instrument-Data/events{/privacy}","received_events_url":"https://api.github.com/users/DMSC-Instrument-Data/received_events","type":"Organization","site_admin":false},"repo":{"id":44163828,"node_id":"MDEwOlJlcG9zaXRvcnk0NDE2MzgyOA==","name":"documents","full_name":"DMSC-Instrument-Data/documents","owner":{"login":"DMSC-Instrument-Data","id":15102351,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTAyMzUx","avatar_url":"https://avatars0.githubusercontent.com/u/15102351?v=4","gravatar_id":"","url":"https://api.github.com/users/DMSC-Instrument-Data","html_url":"https://github.com/DMSC-Instrument-Data","followers_url":"https://api.github.com/users/DMSC-Instrument-Data/followers","following_url":"https://api.github.com/users/DMSC-Instrument-Data/following{/other_user}","gists_url":"https://api.github.com/users/DMSC-Instrument-Data/gists{/gist_id}","starred_url":"https://api.github.com/users/DMSC-Instrument-Data/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DMSC-Instrument-Data/subscriptions","organizations_url":"https://api.github.com/users/DMSC-Instrument-Data/orgs","repos_url":"https://api.github.com/users/DMSC-Instrument-Data/repos","events_url":"https://api.github.com/users/DMSC-Instrument-Data/events{/privacy}","received_events_url":"https://api.github.com/users/DMSC-Instrument-Data/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/DMSC-Instrument-Data/documents","description":null,"fork":false,"url":"https://api.github.com/repos/DMSC-Instrument-Data/documents","forks_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/forks","keys_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/keys{/key_id}","collaborators_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/teams","hooks_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/hooks","issue_events_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/events{/number}","events_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/events","assignees_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/assignees{/user}","branches_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/branches{/branch}","tags_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/tags","blobs_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/refs{/sha}","trees_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/trees{/sha}","statuses_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/statuses/{sha}","languages_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/languages","stargazers_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/stargazers","contributors_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/contributors","subscribers_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/subscribers","subscription_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/subscription","commits_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/commits{/sha}","git_commits_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/git/commits{/sha}","comments_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/comments{/number}","issue_comment_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/comments{/number}","contents_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/contents/{+path}","compare_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/compare/{base}...{head}","merges_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/merges","archive_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/downloads","issues_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues{/number}","pulls_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls{/number}","milestones_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/milestones{/number}","notifications_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/labels{/name}","releases_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/releases{/id}","deployments_url":"https://api.github.com/repos/DMSC-Instrument-Data/documents/deployments","created_at":"2015-10-13T08:50:04Z","updated_at":"2018-06-14T08:29:24Z","pushed_at":"2018-06-19T08:53:50Z","git_url":"git://github.com/DMSC-Instrument-Data/documents.git","ssh_url":"git@github.com:DMSC-Instrument-Data/documents.git","clone_url":"https://github.com/DMSC-Instrument-Data/documents.git","svn_url":"https://github.com/DMSC-Instrument-Data/documents","homepage":null,"size":101588,"stargazers_count":1,"watchers_count":1,"language":"JavaScript","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"open_issues_count":3,"license":null,"forks":3,"open_issues":3,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24"},"html":{"href":"https://github.com/DMSC-Instrument-Data/documents/pull/24"},"issue":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/24"},"comments":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/issues/24/comments"},"review_comments":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24/comments"},"review_comment":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/pulls/24/commits"},"statuses":{"href":"https://api.github.com/repos/DMSC-Instrument-Data/documents/statuses/a62dbb8d2682041f1c742cc54ad80522cebd19c2"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"OwenArnold"}}