{"action":"created","issue":{"url":"https://api.github.com/repos/xianyi/OpenBLAS/issues/923","repository_url":"https://api.github.com/repos/xianyi/OpenBLAS","labels_url":"https://api.github.com/repos/xianyi/OpenBLAS/issues/923/labels{/name}","comments_url":"https://api.github.com/repos/xianyi/OpenBLAS/issues/923/comments","events_url":"https://api.github.com/repos/xianyi/OpenBLAS/issues/923/events","html_url":"https://github.com/xianyi/OpenBLAS/issues/923","id":165354236,"number":923,"title":"caffe + openblas not using cpu to the max","user":{"login":"fenrus75","id":1302037,"avatar_url":"https://avatars.githubusercontent.com/u/1302037?v=3","gravatar_id":"","url":"https://api.github.com/users/fenrus75","html_url":"https://github.com/fenrus75","followers_url":"https://api.github.com/users/fenrus75/followers","following_url":"https://api.github.com/users/fenrus75/following{/other_user}","gists_url":"https://api.github.com/users/fenrus75/gists{/gist_id}","starred_url":"https://api.github.com/users/fenrus75/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fenrus75/subscriptions","organizations_url":"https://api.github.com/users/fenrus75/orgs","repos_url":"https://api.github.com/users/fenrus75/repos","events_url":"https://api.github.com/users/fenrus75/events{/privacy}","received_events_url":"https://api.github.com/users/fenrus75/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-07-13T15:44:00Z","updated_at":"2016-07-14T13:50:04Z","closed_at":null,"body":"When using the combination of caffe + openblas, it turns out that all cpus cores except cpu 0 are only used about 50% (AlexNet benchmark), the rest is idle time.\r\n\r\n(this is sort of hidden if you're not careful due to the polling nature of the default openblas config, but it shows up if you fix that, or if you use the openmp setup. In the default polling setup you can see it by how many cycles are spent in the polling loop)\r\n\r\nprofile below:\r\n\r\n  63.28%  libopenblas_haswellp-r0.2.16.so  [.] sgemm_kernel\r\n   8.78%  libopenblas_haswellp-r0.2.16.so  [.] sgemm_incopy\r\n   6.10%  libcaffe.so.1.0.0-rc3            [.] caffe::im2col_cpu<float>\r\n   4.55%  libopenblas_haswellp-r0.2.16.so  [.] sgemm_itcopy\r\n   2.59%  libopenblas_haswellp-r0.2.16.so  [.] sgemm_oncopy\r\n   1.32%  libcaffe.so.1.0.0-rc3            [.] caffe::PoolingLayer<float>::Forward_cpu\r\n   1.27%  libgomp.so.1.0.0                 [.] gomp_barrier_wait_end\r\n   1.22%  libm-2.23.so                     [.] __powf\r\n\r\n(and then a tail of very small things)\r\n\r\nI'm kind of lost in terms of how to diagnose this further; increasing the number of worker threads does nothing; each thread does proportionally less work but the total cpu use does not change significant. \r\n\r\nIt feels (subjectively) that something is either not feeding the worker threads enough, or that there is a sequential step going on that is not threaded but that takes about 50% of wallclock time.\r\n\r\nIs this a known behavior? Any suggestions on how I can diagnose this further?\r\n"},"comment":{"url":"https://api.github.com/repos/xianyi/OpenBLAS/issues/comments/232670866","html_url":"https://github.com/xianyi/OpenBLAS/issues/923#issuecomment-232670866","issue_url":"https://api.github.com/repos/xianyi/OpenBLAS/issues/923","id":232670866,"user":{"login":"fenrus75","id":1302037,"avatar_url":"https://avatars.githubusercontent.com/u/1302037?v=3","gravatar_id":"","url":"https://api.github.com/users/fenrus75","html_url":"https://github.com/fenrus75","followers_url":"https://api.github.com/users/fenrus75/followers","following_url":"https://api.github.com/users/fenrus75/following{/other_user}","gists_url":"https://api.github.com/users/fenrus75/gists{/gist_id}","starred_url":"https://api.github.com/users/fenrus75/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fenrus75/subscriptions","organizations_url":"https://api.github.com/users/fenrus75/orgs","repos_url":"https://api.github.com/users/fenrus75/repos","events_url":"https://api.github.com/users/fenrus75/events{/privacy}","received_events_url":"https://api.github.com/users/fenrus75/received_events","type":"User","site_admin":false},"created_at":"2016-07-14T13:50:04Z","updated_at":"2016-07-14T13:50:04Z","body":"(interesting that you use PTS, I added the openblas support to that)\r\n\r\nthe problem you are seeing is that openblas in the worker threads will just spin the cpu if there is no useful work for the thread to do, so the time command will show them as busy;\r\nyou can either look at \"perf top\" during the run and see that the bulk of the time is spent in this spinner, or you can apply a small patch (below) that turns the spin into a short sleep which then accounts better\r\n\r\n(even if one does not like usleep(), yield() is sort of a worst case thing to do, asm(\"rep nop\") would be nicer for both the kernel and a hyperthreading scenario)\r\n\r\n\r\n\r\n--- OpenBLAS-0.2.16/common.h~   2016-03-15 18:49:10.000000000 +0000\r\n+++ OpenBLAS-0.2.16/common.h    2016-06-12 16:12:41.864694505 +0000\r\n@@ -349,7 +349,7 @@\r\n */\r\n\r\n #ifndef YIELDING\r\n-#define YIELDING       sched_yield()\r\n+#define YIELDING       usleep(1)\r\n #endif\r\n\r\n /***\r\n"}}
{"actor":{"display_login":"fenrus75"}}