{"action":"created","issue":{"url":"https://api.github.com/repos/kokkos/kokkos/issues/285","repository_url":"https://api.github.com/repos/kokkos/kokkos","labels_url":"https://api.github.com/repos/kokkos/kokkos/issues/285/labels{/name}","comments_url":"https://api.github.com/repos/kokkos/kokkos/issues/285/comments","events_url":"https://api.github.com/repos/kokkos/kokkos/issues/285/events","html_url":"https://github.com/kokkos/kokkos/issues/285","id":155335523,"number":285,"title":"Wide variance in OpenMP wall time","user":{"login":"mcarilli","id":7799218,"avatar_url":"https://avatars.githubusercontent.com/u/7799218?v=3","gravatar_id":"","url":"https://api.github.com/users/mcarilli","html_url":"https://github.com/mcarilli","followers_url":"https://api.github.com/users/mcarilli/followers","following_url":"https://api.github.com/users/mcarilli/following{/other_user}","gists_url":"https://api.github.com/users/mcarilli/gists{/gist_id}","starred_url":"https://api.github.com/users/mcarilli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcarilli/subscriptions","organizations_url":"https://api.github.com/users/mcarilli/orgs","repos_url":"https://api.github.com/users/mcarilli/repos","events_url":"https://api.github.com/users/mcarilli/events{/privacy}","received_events_url":"https://api.github.com/users/mcarilli/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-17T18:59:02Z","updated_at":"2016-05-17T19:48:04Z","closed_at":null,"body":"Hello Kokkos Team,\r\n\r\nI am testing code (consisting purely of parallel_for loops) on a node with two 10-core CPUs.  \r\n\r\nThe single-core serial runtime of my application is 7.13 seconds.\r\nWhen I export OMP_NUM_THREADS=20 and run the application multiple times, sometimes it runs in 0.38 seconds.  7.13/0.38 = 18.7X, so I am achieving almost linear speedup across 20 cores, which is good.\r\n\r\nHowever, often (more than half the time) the application takes 0.68 sec, or 0.58, or 0.5, or some other value that is greater than 0.38 seconds.  My only guess is that it has something to do with how threads are dynamically assigned to cores by openmp.  Naturally I'd like to ensure that it runs in 0.38 sec every time.  Do you have any thoughts?\r\n\r\nMuch appreciated,\r\nMichael"},"comment":{"url":"https://api.github.com/repos/kokkos/kokkos/issues/comments/219832373","html_url":"https://github.com/kokkos/kokkos/issues/285#issuecomment-219832373","issue_url":"https://api.github.com/repos/kokkos/kokkos/issues/285","id":219832373,"user":{"login":"crtrott","id":9490481,"avatar_url":"https://avatars.githubusercontent.com/u/9490481?v=3","gravatar_id":"","url":"https://api.github.com/users/crtrott","html_url":"https://github.com/crtrott","followers_url":"https://api.github.com/users/crtrott/followers","following_url":"https://api.github.com/users/crtrott/following{/other_user}","gists_url":"https://api.github.com/users/crtrott/gists{/gist_id}","starred_url":"https://api.github.com/users/crtrott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crtrott/subscriptions","organizations_url":"https://api.github.com/users/crtrott/orgs","repos_url":"https://api.github.com/users/crtrott/repos","events_url":"https://api.github.com/users/crtrott/events{/privacy}","received_events_url":"https://api.github.com/users/crtrott/received_events","type":"User","site_admin":false},"created_at":"2016-05-17T19:48:04Z","updated_at":"2016-05-17T19:48:04Z","body":"You need to apply binding. Assuming you don't do MPI this goes through the environment variable:\r\nGOMP_CPU_AFFINITY=0-19 for a GNU compatible runtime (works with GCC and Intel for example). \r\nWith Intel you could also use KMP_AFFINITY=compact or KMP_AFFINITY=scatter. The former would put threads on hyper threads in a dense fashion. So if you have hyper threading enabled you would only get a single socket, and you would need to increase number of OpenMP threads to 40. "}}