{"action":"created","issue":{"url":"https://api.github.com/repos/pmix/pmix/issues/665","repository_url":"https://api.github.com/repos/pmix/pmix","labels_url":"https://api.github.com/repos/pmix/pmix/issues/665/labels{/name}","comments_url":"https://api.github.com/repos/pmix/pmix/issues/665/comments","events_url":"https://api.github.com/repos/pmix/pmix/issues/665/events","html_url":"https://github.com/pmix/pmix/issues/665","id":294175076,"number":665,"title":"PMIx/dstore performance issues","user":{"login":"artpol84","id":7534910,"avatar_url":"https://avatars1.githubusercontent.com/u/7534910?v=4","gravatar_id":"","url":"https://api.github.com/users/artpol84","html_url":"https://github.com/artpol84","followers_url":"https://api.github.com/users/artpol84/followers","following_url":"https://api.github.com/users/artpol84/following{/other_user}","gists_url":"https://api.github.com/users/artpol84/gists{/gist_id}","starred_url":"https://api.github.com/users/artpol84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artpol84/subscriptions","organizations_url":"https://api.github.com/users/artpol84/orgs","repos_url":"https://api.github.com/users/artpol84/repos","events_url":"https://api.github.com/users/artpol84/events{/privacy}","received_events_url":"https://api.github.com/users/artpol84/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2018-02-04T06:43:58Z","updated_at":"2018-04-16T21:40:33Z","closed_at":null,"author_association":"MEMBER","body":"I'm creating this issue as the reference for the dstore locking performance issue.\r\nPOC for evaluation: https://github.com/artpol84/poc/tree/master/benchmarks/shmem_locking\r\n\r\nCurrently dstore is using global lock to arbitrate between PMIx server writing and PMIx clients reading at the same time (this is mainly for direct modex mode).\r\n\r\n* Originally dstore was using flock but it was found to perform very poor on high core count systems like PPC and KNL: https://github.com/pmix/pmix/pull/273#issuecomment-275944934\r\n* To partially address that issue pthread-based locking was introduced\r\n* However detailed analysis showed that performance of atomic operations is not scalable:\r\n![image](https://user-images.githubusercontent.com/7534910/35774780-a23d2f70-092e-11e8-9345-c52f2ef256bc.png)\r\n![image](https://user-images.githubusercontent.com/7534910/35774788-0d701c76-092f-11e8-9399-3e235b20c392.png)\r\nThose results was obtained with the following POC benchmarks:\r\n  * PPC, POWER8: https://github.com/artpol84/poc/tree/master/atomics/power8\r\n  * For KNL corresponding asm primitives derived from OPAL was used,\r\n* On the Telecon (Jan, 22 2018) we agreed to try the locking scheme where server will maintain independent locks for each client. I've updated my POC code to support independent locking scheme (https://github.com/artpol84/poc/commit/8396509d4cb4f810b40ab8c6e41b2b1a72d9c146, https://github.com/artpol84/poc/commit/fe31bbd3b94e7ab6cb28c06fee985620b2f81da9).\r\nHere are the first results on Power:\r\n\r\n`pthread_prio` test corresponds to the solution currently used in PMIx:\r\n```\r\n$ mpirun -np 160 -mca coll ^hcoll -mca pml ob1 -mca btl self,vader --use-hwthread-cpus \\\r\n        -map-by hwthread --bind-to hwthread ./test_pthread_prio\r\nreaders WO:ovh  WO:lk/s RO:ovh[avg/min/max]     RO:lk/s RW:wovh RW:wtm  RW:rlk/s[avg/min/max]\r\n159     5e-03   2120645 4e+00/3e+00/4e+00       2611    1e-01   1e-01   1517/950/3028\r\n```\r\n`pthread_nlocks` is using 2x159 independent pthread rwlock's (2 per each client) to ensure correctness\r\n```\r\n$ mpirun -np 160 -mca coll ^hcoll -mca pml ob1 -mca btl self,vader --use-hwthread-cpus \\\r\n        -map-by hwthread --bind-to hwthread ./test_pthread_nlocks\r\nreaders WO:ovh  WO:lk/s RO:ovh[avg/min/max]     RO:lk/s RW:wovh RW:wtm  RW:rlk/s[avg/min/max]\r\n159     6e-01   15743   1e-02/9e-03/1e-02       941169  1e-01   1e-01   239722/130145/355652\r\n```\r\n`pthread_nlocks2` is using 2x159 independent pthread mutexes (2 per each client) to ensure correctness\r\n```\r\n$ mpirun -np 160 -mca coll ^hcoll -mca pml ob1 -mca btl self,vader --use-hwthread-cpus \\\r\n-map-by hwthread --bind-to hwthread ./test_pthread_nlocks2\r\nreaders WO:ovh  WO:lk/s RO:ovh[avg/min/max]     RO:lk/s RW:wovh RW:wtm  RW:rlk/s[avg/min/max]\r\n159     5e-01   21062   1e-02/8e-03/1e-02       983763  1e-01   1e-01   300232/128769/511602\r\n```\r\nAlthough the new scheme is not ideal (relatively low write locking rate) it seems to me that mutex-based approach might be good step forward. It is obvious that Read lock performance is significantly better (two orders of magnitude).\r\nAs it looks promising I'm going to implement this scheme as POC in PMIx to see how it will affect the overall performance. I will also provide the parameter that will allow select number of locks that are used. This may provide some reasonable tradeoff and will allow for further experiments."},"comment":{"url":"https://api.github.com/repos/pmix/pmix/issues/comments/381758996","html_url":"https://github.com/pmix/pmix/issues/665#issuecomment-381758996","issue_url":"https://api.github.com/repos/pmix/pmix/issues/665","id":381758996,"user":{"login":"artpol84","id":7534910,"avatar_url":"https://avatars1.githubusercontent.com/u/7534910?v=4","gravatar_id":"","url":"https://api.github.com/users/artpol84","html_url":"https://github.com/artpol84","followers_url":"https://api.github.com/users/artpol84/followers","following_url":"https://api.github.com/users/artpol84/following{/other_user}","gists_url":"https://api.github.com/users/artpol84/gists{/gist_id}","starred_url":"https://api.github.com/users/artpol84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artpol84/subscriptions","organizations_url":"https://api.github.com/users/artpol84/orgs","repos_url":"https://api.github.com/users/artpol84/repos","events_url":"https://api.github.com/users/artpol84/events{/privacy}","received_events_url":"https://api.github.com/users/artpol84/received_events","type":"User","site_admin":false},"created_at":"2018-04-16T21:40:33Z","updated_at":"2018-04-16T21:40:33Z","author_association":"MEMBER","body":"x86_64 results:\r\n![image](https://user-images.githubusercontent.com/7534910/38836720-14556e1e-4184-11e8-88f0-52cfaa73c034.png)\r\n"}}
{"actor":{"display_login":"artpol84"}}