{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/26948","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/26948/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/26948/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/26948/events","html_url":"https://github.com/pytorch/pytorch/issues/26948","id":499212897,"node_id":"MDU6SXNzdWU0OTkyMTI4OTc=","number":26948,"title":"[libtorch] Is there any global lock in process?","user":{"login":"wizardk","id":20595052,"node_id":"MDQ6VXNlcjIwNTk1MDUy","avatar_url":"https://avatars3.githubusercontent.com/u/20595052?v=4","gravatar_id":"","url":"https://api.github.com/users/wizardk","html_url":"https://github.com/wizardk","followers_url":"https://api.github.com/users/wizardk/followers","following_url":"https://api.github.com/users/wizardk/following{/other_user}","gists_url":"https://api.github.com/users/wizardk/gists{/gist_id}","starred_url":"https://api.github.com/users/wizardk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wizardk/subscriptions","organizations_url":"https://api.github.com/users/wizardk/orgs","repos_url":"https://api.github.com/users/wizardk/repos","events_url":"https://api.github.com/users/wizardk/events{/privacy}","received_events_url":"https://api.github.com/users/wizardk/received_events","type":"User","site_admin":false},"labels":[{"id":1294499454,"node_id":"MDU6TGFiZWwxMjk0NDk5NDU0","url":"https://api.github.com/repos/pytorch/pytorch/labels/module:%20cpu","name":"module: cpu","color":"f7e101","default":false,"description":"CPU specific problem (e.g., perf, algorithm)"},{"id":679952992,"node_id":"MDU6TGFiZWw2Nzk5NTI5OTI=","url":"https://api.github.com/repos/pytorch/pytorch/labels/topic:%20performance","name":"topic: performance","color":"0e8a16","default":false,"description":"Issues related to performance, either of kernel code or framework glue"},{"id":1301347167,"node_id":"MDU6TGFiZWwxMzAxMzQ3MTY3","url":"https://api.github.com/repos/pytorch/pytorch/labels/triaged","name":"triaged","color":"006b75","default":false,"description":"This issue has been looked at a team member, and triaged and prioritized into an appropriate module"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2019-09-27T03:19:02Z","updated_at":"2019-11-26T13:00:07Z","closed_at":null,"author_association":"NONE","body":"## 🐛 Bug\r\n\r\nCPU usage is not increasing by increasing the number of threads when inferring with CPU only libtorch in one process and multiple threads.\r\n\r\n## To Reproduce\r\n\r\n```\r\nconst int THREAD_CNT = 10\r\n\r\ntorch::jit::script::Module load()\r\n{\r\n    torch::NoGradGuard no_guard;\r\n    torch::jit::script::Module module = torch::jit::load(\"xxx.jit\");\r\n    module.to(torch::kCPU);\r\n    module.eval();\r\n    return module;\r\n}\r\n\r\ntorch::TensorOptions options = torch::TensorOptions().dtype(torch::kFloat32).requires_grad(false).device(torch::kCPU);\r\n\r\nvoid * working_thread(void * param)\r\n{\r\n    torch::init_num_threads();\r\n    torch::jit::script::Module * module = (torch::jit::script::Module *)param;\r\n    torch::NoGradGuard no_guard;\r\n    while (1) {\r\n        torch::Tensor x = torch::randn(1, 10, options);\r\n        torch::Tensor y = module->forward({x}).toTensor();\r\n    }\r\n    return 0;\r\n}\r\n\r\nvoid main()\r\n{\r\n    // load module\r\n    torch::jit::script::Module module = load();\r\n\r\n    // muti-threading\r\n    pthread_t thread_handle[THREAD_CNT];\r\n    for (int thread_idx = 0; thread_idx < THREAD_CNT; ++thread_idx) {\r\n        pthread_create(thread_handle + thread_idx, 0, working_thread, &module);\r\n    }\r\n\r\n    // finished\r\n    for (int thread_idx = 0; thread_idx < THREAD_CNT; ++thread_idx) {\r\n        pthread_join(thread_handle[thread_idx], 0);\r\n    }\r\n}\r\n```\r\n\r\n## Expected behavior\r\n\r\nOne process, more threads, higher CPU Usage.\r\n\r\n## Environment\r\n\r\n - LibTorch Version : 1.2\r\n - OS : Linux\r\n - How you installed LibTorch : download from pytorch.org\r\n - CPU : 96 cores\r\n\r\n## Additional context\r\n\r\nHere is my experiment result:\r\nI can get 33% CPU usage by 1 process and 1 thread per process.\r\nI can get 66% CPU usage by 1 process and 2 threads per process.\r\nI can get 46% CPU usage by 1 process and 3 threads per process.\r\nI can get 74% CPU usage by 2 processes and 1 thread per process.\r\nI can get 92% CPU usage by 2 processes and 2 threads per process.\r\n\r\nI guess there should be a global lock in process range to limit the performance of multi-threading inference.\r\n\n\ncc @suo"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/558617531","html_url":"https://github.com/pytorch/pytorch/issues/26948#issuecomment-558617531","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/26948","id":558617531,"node_id":"MDEyOklzc3VlQ29tbWVudDU1ODYxNzUzMQ==","user":{"login":"wizard-baller","id":58059682,"node_id":"MDQ6VXNlcjU4MDU5Njgy","avatar_url":"https://avatars2.githubusercontent.com/u/58059682?v=4","gravatar_id":"","url":"https://api.github.com/users/wizard-baller","html_url":"https://github.com/wizard-baller","followers_url":"https://api.github.com/users/wizard-baller/followers","following_url":"https://api.github.com/users/wizard-baller/following{/other_user}","gists_url":"https://api.github.com/users/wizard-baller/gists{/gist_id}","starred_url":"https://api.github.com/users/wizard-baller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wizard-baller/subscriptions","organizations_url":"https://api.github.com/users/wizard-baller/orgs","repos_url":"https://api.github.com/users/wizard-baller/repos","events_url":"https://api.github.com/users/wizard-baller/events{/privacy}","received_events_url":"https://api.github.com/users/wizard-baller/received_events","type":"User","site_admin":false},"created_at":"2019-11-26T13:00:07Z","updated_at":"2019-11-26T13:00:07Z","author_association":"NONE","body":"I  cared for CPU usage in the last time. Now, I did another speed test on libtorch 1.3.0.\r\nThe test code is almost same with the previous. I only add some code to compute elapsed time by std::chrono::system_clock. And there are 48 CPU cores.\r\n\r\nThe test result are:\r\n\r\nset_num_interop_threads(1),  set_num_threads(1), 1 process, 1 working thread, 100 times inference in each thread:\r\naverage inference time is 536 ms, CPU usage is 100.\r\n\r\nset_num_interop_threads(1),  set_num_threads(1), 1 process, 10 working threads, 100 times inference in each thread:\r\naverage inference time is 749 ms, CPU usage is 1000.\r\n\r\nset_num_interop_threads(1),  set_num_threads(1), 1 process, 20 working threads, 100 times inference in each thread:\r\naverage inference time is 1363 ms, CPU usage is 2000.\r\n\r\nset_num_interop_threads(1),  set_num_threads(1), 1 process, 40 working threads, 100 times inference in each thread:\r\naverage inference time is 7223 ms, CPU usage is 4000.\r\n\r\nYou can find out the CPU usage looks normal but inference speed is abnormal.\r\n@suo Any progress on the CPU usage issue? There is another speed issue has nothing to do with OpenMP I guess."}}
{"actor":{"display_login":"wizard-baller"}}