{"action":"created","issue":{"url":"https://api.github.com/repos/chapel-lang/chapel/issues/10996","repository_url":"https://api.github.com/repos/chapel-lang/chapel","labels_url":"https://api.github.com/repos/chapel-lang/chapel/issues/10996/labels{/name}","comments_url":"https://api.github.com/repos/chapel-lang/chapel/issues/10996/comments","events_url":"https://api.github.com/repos/chapel-lang/chapel/issues/10996/events","html_url":"https://github.com/chapel-lang/chapel/issues/10996","id":357470324,"node_id":"MDU6SXNzdWUzNTc0NzAzMjQ=","number":10996,"title":"Concurrent reads on atomic variable results in degrading performance with CHPL_ATOMICS=intrinsics, 20x slower than CHPL_ATOMICS=cstdlib","user":{"login":"LouisJenkinsCS","id":4269990,"node_id":"MDQ6VXNlcjQyNjk5OTA=","avatar_url":"https://avatars2.githubusercontent.com/u/4269990?v=4","gravatar_id":"","url":"https://api.github.com/users/LouisJenkinsCS","html_url":"https://github.com/LouisJenkinsCS","followers_url":"https://api.github.com/users/LouisJenkinsCS/followers","following_url":"https://api.github.com/users/LouisJenkinsCS/following{/other_user}","gists_url":"https://api.github.com/users/LouisJenkinsCS/gists{/gist_id}","starred_url":"https://api.github.com/users/LouisJenkinsCS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LouisJenkinsCS/subscriptions","organizations_url":"https://api.github.com/users/LouisJenkinsCS/orgs","repos_url":"https://api.github.com/users/LouisJenkinsCS/repos","events_url":"https://api.github.com/users/LouisJenkinsCS/events{/privacy}","received_events_url":"https://api.github.com/users/LouisJenkinsCS/received_events","type":"User","site_admin":false},"labels":[{"id":496591153,"node_id":"MDU6TGFiZWw0OTY1OTExNTM=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/area:%20Runtime","name":"area: Runtime","color":"d4c5f9","default":false},{"id":496603405,"node_id":"MDU6TGFiZWw0OTY2MDM0MDU=","url":"https://api.github.com/repos/chapel-lang/chapel/labels/type:%20Performance","name":"type: Performance","color":"e99695","default":false}],"state":"open","locked":false,"assignee":{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false},"assignees":[{"login":"ronawho","id":1588337,"node_id":"MDQ6VXNlcjE1ODgzMzc=","avatar_url":"https://avatars1.githubusercontent.com/u/1588337?v=4","gravatar_id":"","url":"https://api.github.com/users/ronawho","html_url":"https://github.com/ronawho","followers_url":"https://api.github.com/users/ronawho/followers","following_url":"https://api.github.com/users/ronawho/following{/other_user}","gists_url":"https://api.github.com/users/ronawho/gists{/gist_id}","starred_url":"https://api.github.com/users/ronawho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronawho/subscriptions","organizations_url":"https://api.github.com/users/ronawho/orgs","repos_url":"https://api.github.com/users/ronawho/repos","events_url":"https://api.github.com/users/ronawho/events{/privacy}","received_events_url":"https://api.github.com/users/ronawho/received_events","type":"User","site_admin":false},{"login":"dmk42","id":20049827,"node_id":"MDQ6VXNlcjIwMDQ5ODI3","avatar_url":"https://avatars0.githubusercontent.com/u/20049827?v=4","gravatar_id":"","url":"https://api.github.com/users/dmk42","html_url":"https://github.com/dmk42","followers_url":"https://api.github.com/users/dmk42/followers","following_url":"https://api.github.com/users/dmk42/following{/other_user}","gists_url":"https://api.github.com/users/dmk42/gists{/gist_id}","starred_url":"https://api.github.com/users/dmk42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmk42/subscriptions","organizations_url":"https://api.github.com/users/dmk42/orgs","repos_url":"https://api.github.com/users/dmk42/repos","events_url":"https://api.github.com/users/dmk42/events{/privacy}","received_events_url":"https://api.github.com/users/dmk42/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2018-09-06T02:11:56Z","updated_at":"2018-09-17T22:39:10Z","closed_at":null,"author_association":"MEMBER","body":"@ronawho \r\n\r\n`CHPL_ATOMICS=intrinsics`  currently performs `load` and `store` operations using expensive compare-and-swap primitives, which add a significant overhead to `loads` and `stores` to shared locations, and also ignores the memory ordering specified by the user.\r\n\r\nhttps://github.com/chapel-lang/chapel/blob/f6112453b43b5e31f6175f80b8ed4fc271335391/runtime/include/atomics/intrinsics/chpl-atomics.h#L110-L126\r\n\r\n**MWE**\r\n\r\n```chpl\r\nproc load(ref ptr : atomic int) : int {\r\n        return ptr.read();\r\n}\r\nuse Time;\r\n\r\nconst n = 1024 * 1024 * 1024;\r\nvar timer = new Timer();\r\ntimer.start();\r\nvar y : atomic int;\r\nfor 1..n do load(y);\r\nwriteln(\"Serial Atomics: \", timer.elapsed());\r\ntimer.clear();\r\n\r\nforall 1..n do load(y);\r\nwriteln(\"Parallel Atomics: \", timer.elapsed());\r\n```\r\n\r\n**Output - CHPL_ATOMICS=intrinsics**\r\n\r\n```\r\nSerial Atomics: 6.36694\r\nParallel Atomics: 23.2151\r\n```\r\n\r\n**Output - CHPL_ATOMICS=cstdlib**\r\n\r\n```\r\nSerial Atomics: 0.350935\r\nParallel Atomics: 0.120661\r\n```\r\n\r\nNote as well that since atomic loads are implemented using a Compare-and-Swap operation, a load is as expensive as any other read-modify-write operation.\r\n\r\n**MWE - RMW vs Read**\r\n\r\n```chpl\r\nproc load(ref ptr : atomic int) : int {\r\n        return ptr.read();\r\n}\r\nproc add(ref ptr : atomic int) {\r\n        ptr.add(1);\r\n}\r\nuse Time;\r\n\r\nconst n = 1024 * 1024 * 1024;\r\nvar timer = new Timer();\r\ntimer.start();\r\nvar y : atomic int;\r\nfor 1..n do load(y);\r\nwriteln(\"Read Atomics: \", timer.elapsed());\r\ntimer.clear();\r\n\r\nfor 1..n do add(y);\r\nwriteln(\"Increment Atomics: \", timer.elapsed());\r\n```\r\n\r\n**Output - CHPL_ATOMICS=intrinsics**\r\n\r\n```\r\nRead Atomics: 6.51129\r\nIncrement Atomics: 6.49777\r\n```\r\n\r\n**Output - CHPL_ATOMICS=cstdlib**\r\n\r\n```\r\nRead Atomics: 0.351615\r\nIncrement Atomics: 6.23698\r\n```\r\n\r\nNote that the performance of 'read' is actually 20x slower with `intrinsics` than `cstdlib`, but not from an increment. \r\n\r\n**I will make note that this is a gating issue for developing future non-blocking algorithms, and that it will impact the performance of certain spinlock implementations, such as the [Test&Test&Set Lock](https://geidav.wordpress.com/tag/test-and-test-and-set/) that achieves significant performance improvements of a normal Test&Set Lock due to being able to avoid cache-line invalidation, in which is impossible with `intrinsics` as a `read` will invalidate the cache-line.**"},"comment":{"url":"https://api.github.com/repos/chapel-lang/chapel/issues/comments/422194883","html_url":"https://github.com/chapel-lang/chapel/issues/10996#issuecomment-422194883","issue_url":"https://api.github.com/repos/chapel-lang/chapel/issues/10996","id":422194883,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjE5NDg4Mw==","user":{"login":"bradcray","id":7536222,"node_id":"MDQ6VXNlcjc1MzYyMjI=","avatar_url":"https://avatars0.githubusercontent.com/u/7536222?v=4","gravatar_id":"","url":"https://api.github.com/users/bradcray","html_url":"https://github.com/bradcray","followers_url":"https://api.github.com/users/bradcray/followers","following_url":"https://api.github.com/users/bradcray/following{/other_user}","gists_url":"https://api.github.com/users/bradcray/gists{/gist_id}","starred_url":"https://api.github.com/users/bradcray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradcray/subscriptions","organizations_url":"https://api.github.com/users/bradcray/orgs","repos_url":"https://api.github.com/users/bradcray/repos","events_url":"https://api.github.com/users/bradcray/events{/privacy}","received_events_url":"https://api.github.com/users/bradcray/received_events","type":"User","site_admin":false},"created_at":"2018-09-17T22:39:10Z","updated_at":"2018-09-17T22:39:10Z","author_association":"MEMBER","body":"Offline, I just caught up with @ronaho on this issue and agree that we shouldn't close it before understanding better whether his SIZE_ALIGN_TYPE approach would help improve the performance of reads/writes on intrinsics.  Putting a performance warning in the documentation seems like a reasonable alternative to me if we can't come up with a plan that will resolve the performance difference."}}
{"actor":{"display_login":"bradcray"}}