{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/orleans/issues/2060","repository_url":"https://api.github.com/repos/dotnet/orleans","labels_url":"https://api.github.com/repos/dotnet/orleans/issues/2060/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/orleans/issues/2060/comments","events_url":"https://api.github.com/repos/dotnet/orleans/issues/2060/events","html_url":"https://github.com/dotnet/orleans/pull/2060","id":171237886,"number":2060,"title":"Threading model rework","user":{"login":"dVakulen","id":5787619,"avatar_url":"https://avatars.githubusercontent.com/u/5787619?v=3","gravatar_id":"","url":"https://api.github.com/users/dVakulen","html_url":"https://github.com/dVakulen","followers_url":"https://api.github.com/users/dVakulen/followers","following_url":"https://api.github.com/users/dVakulen/following{/other_user}","gists_url":"https://api.github.com/users/dVakulen/gists{/gist_id}","starred_url":"https://api.github.com/users/dVakulen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dVakulen/subscriptions","organizations_url":"https://api.github.com/users/dVakulen/orgs","repos_url":"https://api.github.com/users/dVakulen/repos","events_url":"https://api.github.com/users/dVakulen/events{/privacy}","received_events_url":"https://api.github.com/users/dVakulen/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dotnet/orleans/labels/cla-already-signed","name":"cla-already-signed","color":"009800"},{"url":"https://api.github.com/repos/dotnet/orleans/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2016-08-15T19:03:31Z","updated_at":"2016-09-09T20:40:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dotnet/orleans/pulls/2060","html_url":"https://github.com/dotnet/orleans/pull/2060","diff_url":"https://github.com/dotnet/orleans/pull/2060.diff","patch_url":"https://github.com/dotnet/orleans/pull/2060.patch"},"body":"Currently Orleans uses SEDA (Staged Execution Design Architecture) implemented with fixed dedicated threads per stage and <code>BlockingCollection</code>'s as queues. However, the performance is highly sensitive to the number of threads in each SEDA stage and queuing delay reportedly dominates the end-to-end latency, and by far exceeds the network latency, as well as processing time in each stage, so in order to achieve lowest possible latency dynamic threads reallocation among the stages is needed. \r\n\r\nOn the other hand using <code>BlockingCollection</code> doesn't seems as optimal for Orleans use case: it's synchronization algorithms consumes quite big part of CPU time. \r\n\r\nResults of profiling: \r\n\r\n      Including system methods:\r\n![timeline64_2016-08-15_21-29-55](https://cloud.githubusercontent.com/assets/5787619/17674725/6f621da0-632f-11e6-95f8-7e69d793a9c3.png)\r\n\r\n- top five methods by CPU usage are system ones, and caused by internals of the <code>BlockingCollection</code>\r\n\r\n\r\n         Excluding system methods:\r\n\r\n![timeline64_2016-08-15_21-35-16](https://cloud.githubusercontent.com/assets/5787619/17674909/3f6ca65a-6330-11e6-9cf0-323235bcb5b5.png)\r\n\r\n- Top 3 again related to the queue usage, and fourth one - <code>AsynchThreadProc</code> is caused by semaphore usage in fixed threads processor.\r\n\r\nAs solution each stage was implemented as dataflow block, which allows to handle dynamic thread reallocation automatically, plus requires very little synchronization.\r\n\r\nProfile results with dataflow stages:\r\n\r\n       With system methods:\r\n\r\n![timeline64_2016-08-15_21-50-35](https://cloud.githubusercontent.com/assets/5787619/17675402/4e9e03e2-6332-11e6-8bd4-f988f74c1728.png)\r\n   \r\n       And without:\r\n\r\n![timeline64_2016-08-15_21-49-59](https://cloud.githubusercontent.com/assets/5787619/17675375/37cc9fac-6332-11e6-829b-15665834e6f4.png)\r\n\r\nAll thread synchronization hot spots are gone except Monitor.Enter, which does not caused by usage of Dataflow library. \r\n\r\nActual benchmark from https://github.com/dotnet/orleans/pull/2003 shows 34 - 50 % (depends on amount of data for processing)  perfomance improvement while consuming 4 - 8 % less CPU. \r\n\r\nAll OrleansTaskScheduler tests are passing, except few ones can randomly fail, but same fails can be observed  even more frequently when running against current master. \r\n\r\nCurrent implementation have some minor todo's, which will be addressed after Orleans internal performance test run, which will also show whether it worthy reviewing. Also in case of need it's possible to implement swappable execution models, but it will be not an easy task. "},"comment":{"url":"https://api.github.com/repos/dotnet/orleans/issues/comments/246033432","html_url":"https://github.com/dotnet/orleans/pull/2060#issuecomment-246033432","issue_url":"https://api.github.com/repos/dotnet/orleans/issues/2060","id":246033432,"user":{"login":"dVakulen","id":5787619,"avatar_url":"https://avatars.githubusercontent.com/u/5787619?v=3","gravatar_id":"","url":"https://api.github.com/users/dVakulen","html_url":"https://github.com/dVakulen","followers_url":"https://api.github.com/users/dVakulen/followers","following_url":"https://api.github.com/users/dVakulen/following{/other_user}","gists_url":"https://api.github.com/users/dVakulen/gists{/gist_id}","starred_url":"https://api.github.com/users/dVakulen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dVakulen/subscriptions","organizations_url":"https://api.github.com/users/dVakulen/orgs","repos_url":"https://api.github.com/users/dVakulen/repos","events_url":"https://api.github.com/users/dVakulen/events{/privacy}","received_events_url":"https://api.github.com/users/dVakulen/received_events","type":"User","site_admin":false},"created_at":"2016-09-09T20:40:30Z","updated_at":"2016-09-09T20:40:30Z","body":"There's indeed is point of contention., so now I'm working on solution (again :). The approach I'm currently taking is the following: one global work queue with fixed number of executors into which all stages submits work items, the queue itself implements work stealing pattern, so that transition between stages should happen without touching global queue. "}}
{"actor":{"display_login":"dVakulen"}}