{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/nccl/issues/36","repository_url":"https://api.github.com/repos/NVIDIA/nccl","labels_url":"https://api.github.com/repos/NVIDIA/nccl/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/nccl/issues/36/comments","events_url":"https://api.github.com/repos/NVIDIA/nccl/issues/36/events","html_url":"https://github.com/NVIDIA/nccl/issues/36","id":166198735,"number":36,"title":"NCCL All reduce on M40","user":{"login":"sharannarang","id":7253756,"avatar_url":"https://avatars.githubusercontent.com/u/7253756?v=3","gravatar_id":"","url":"https://api.github.com/users/sharannarang","html_url":"https://github.com/sharannarang","followers_url":"https://api.github.com/users/sharannarang/followers","following_url":"https://api.github.com/users/sharannarang/following{/other_user}","gists_url":"https://api.github.com/users/sharannarang/gists{/gist_id}","starred_url":"https://api.github.com/users/sharannarang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sharannarang/subscriptions","organizations_url":"https://api.github.com/users/sharannarang/orgs","repos_url":"https://api.github.com/users/sharannarang/repos","events_url":"https://api.github.com/users/sharannarang/events{/privacy}","received_events_url":"https://api.github.com/users/sharannarang/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-07-18T21:32:53Z","updated_at":"2016-08-03T16:37:28Z","closed_at":"2016-08-03T16:37:27Z","body":"Hello,\r\n\r\nI tried running `test/single/all_reduce` test on M40 nodes and the test just hangs. The same test works fine on TitanX nodes. I'm running the test on 2 GPUs with cuda 7.5. The driver version for the TitanX node is 352.79. Here is the output from the TitanX nodes:\r\n\r\n```\r\n~/nccl$ ./build/test/single/all_reduce_test 100# Using devices\r\n#   Rank  0 uses device  0 [0x04] GeForce GTX TITAN X\r\n#   Rank  1 uses device  1 [0x05] GeForce GTX TITAN X\r\n\r\n#                                                 out-of-place                    in-place\r\n#      bytes             N    type      op     time  algbw  busbw      res     time  algbw  busbw      res\r\n         100           100    char     sum    0.038   0.00   0.00    0e+00    0.046   0.00   0.00    0e+00\r\n         100           100    char    prod    0.032   0.00   0.00    0e+00    0.046   0.00   0.00    0e+00\r\n         100           100    char     max    0.032   0.00   0.00    0e+00    0.045   0.00   0.00    0e+00\r\n         100           100    char     min    0.037   0.00   0.00    0e+00    0.044   0.00   0.00    0e+00\r\n         100            25     int     sum    0.033   0.00   0.00    0e+00    0.045   0.00   0.00    0e+00\r\n         100            25     int    prod    0.032   0.00   0.00    0e+00    0.044   0.00   0.00    0e+00\r\n         100            25     int     max    0.032   0.00   0.00    0e+00    0.044   0.00   0.00    0e+00\r\n         100            25     int     min    0.032   0.00   0.00    0e+00    0.043   0.00   0.00    0e+00\r\n         100            50    half     sum    0.033   0.00   0.00    0e+00    0.044   0.00   0.00    0e+00\r\n         100            50    half    prod    0.048   0.00   0.00    0e+00    0.069   0.00   0.00    0e+00\r\n         100            50    half     max    0.035   0.00   0.00    0e+00    0.053   0.00   0.00    0e+00\r\n         100            50    half     min    0.036   0.00   0.00    0e+00    0.048   0.00   0.00    0e+00\r\n         100            25   float     sum    0.034   0.00   0.00    0e+00    0.052   0.00   0.00    0e+00\r\n         100            25   float    prod    0.035   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n         100            25   float     max    0.035   0.00   0.00    0e+00    0.050   0.00   0.00    0e+00\r\n         100            25   float     min    0.036   0.00   0.00    0e+00    0.050   0.00   0.00    0e+00\r\n          96            12  double     sum    0.033   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12  double    prod    0.034   0.00   0.00    0e+00    0.050   0.00   0.00    0e+00\r\n          96            12  double     max    0.033   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12  double     min    0.046   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12   int64     sum    0.034   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12   int64    prod    0.033   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12   int64     max    0.034   0.00   0.00    0e+00    0.052   0.00   0.00    0e+00\r\n          96            12   int64     min    0.034   0.00   0.00    0e+00    0.048   0.00   0.00    0e+00\r\n          96            12  uint64     sum    0.046   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12  uint64    prod    0.033   0.00   0.00    0e+00    0.049   0.00   0.00    0e+00\r\n          96            12  uint64     max    0.034   0.00   0.00    0e+00    0.052   0.00   0.00    0e+00\r\n          96            12  uint64     min    0.034   0.00   0.00    0e+00    0.053   0.00   0.00    0e+00\r\n```\r\n\r\nI tried running the same binary on M40 nodes with drivers 352.79 and 352.93. However, the test just stalls:\r\n\r\n```\r\n~/nccl$ ./build/test/single/all_reduce_test 100\r\n# Using devices\r\n#   Rank  0 uses device  0 [0x04] Tesla M40\r\n#   Rank  1 uses device  1 [0x05] Tesla M40\r\n\r\n#                                                 out-of-place                    in-place\r\n#      bytes             N    type      op     time  algbw  busbw      res     time  algbw  busbw      res\r\n```\r\n\r\nCan you advice regarding this problem? Have you tried running these tests on M40s? \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/nccl/issues/comments/237292027","html_url":"https://github.com/NVIDIA/nccl/issues/36#issuecomment-237292027","issue_url":"https://api.github.com/repos/NVIDIA/nccl/issues/36","id":237292027,"user":{"login":"sharannarang","id":7253756,"avatar_url":"https://avatars.githubusercontent.com/u/7253756?v=3","gravatar_id":"","url":"https://api.github.com/users/sharannarang","html_url":"https://github.com/sharannarang","followers_url":"https://api.github.com/users/sharannarang/followers","following_url":"https://api.github.com/users/sharannarang/following{/other_user}","gists_url":"https://api.github.com/users/sharannarang/gists{/gist_id}","starred_url":"https://api.github.com/users/sharannarang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sharannarang/subscriptions","organizations_url":"https://api.github.com/users/sharannarang/orgs","repos_url":"https://api.github.com/users/sharannarang/repos","events_url":"https://api.github.com/users/sharannarang/events{/privacy}","received_events_url":"https://api.github.com/users/sharannarang/received_events","type":"User","site_admin":false},"created_at":"2016-08-03T16:37:27Z","updated_at":"2016-08-03T16:37:27Z","body":"Thanks, I see that the issue is fixed with the updated driver. "}}
{"actor":{"display_login":"sharannarang"}}