{"action":"created","issue":{"url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/93","repository_url":"https://api.github.com/repos/soumith/convnet-benchmarks","labels_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/93/comments","events_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/93/events","html_url":"https://github.com/soumith/convnet-benchmarks/issues/93","id":138356929,"number":93,"title":"Nervana's Neon and Winograd","user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-03-04T02:23:11Z","updated_at":"2016-03-04T04:21:45Z","closed_at":null,"body":"After serious perf improvements by NVIDIA's CUDNN R4 across board, I suppose Nervana weren't too happy to be left behind. \r\nThey've just released (as part of Neon) their Winograd-based kernels which have a non-trivial improvement in performance. \r\n[Their blog post can be found here](http://www.nervanasys.com/winograd/), where Scott went into full detail about the technical implementation, challenges as well as data points showing no side-effects of using these kernels in terms of convergence. The implementation seems very sophisticated, and quite a challenge.\r\n\r\nI've benchmarked them independently, and here are the numbers:\r\n\r\nFP-16\r\n--------------------------------------------------------------------------\r\n| Network Type   | Nervana Neon  | CuDNN R4 (Torch) | Speedup |\r\n|:-------------------:|:-------------------:|:-------------------:|:----------------------|\r\n| AlexNet           |  78 | **71** | 0.91x |\r\n| Overfeat          |  **176** | 242 |  1.37x |\r\n| VGG-A            |  **254** | 471 |  1.85x |\r\n| Googlenet-1       |  **230** | 462 | 2.00x |\r\n\r\nFP-32\r\n--------------------------------------------------------------------------\r\n| Network Type   | Nervana Neon  | CuDNN R4 (Torch) | Speedup |\r\n|:-------------------:|:-------------------:|:-------------------:|:---------------------:|\r\n| AlexNet           |  87 | **81** | 0.93x |\r\n| Overfeat          |  **211** | 268 | 1.27x |\r\n| VGG-A            |  **320** | 529 | 1.65x |\r\n| Googlenet-1       | **270** | 470 | 1.74x |\r\n\r\nIt's really cool that they're still squeezing out performance of this generation of hardware.\r\nThey seem to have real wins when the network uses 3x3 convolutions.\r\n\r\nAt this point, I expect that this is the last round of software optimizations for small convolutions, considering that they're hitting peak limits of the GPU, but happy to be surprised :)\r\n\r\nFull logs are checked in to the nervana/ folder.\r\n"},"comment":{"url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/comments/192093844","html_url":"https://github.com/soumith/convnet-benchmarks/issues/93#issuecomment-192093844","issue_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/93","id":192093844,"user":{"login":"scott-gray","id":9326960,"avatar_url":"https://avatars.githubusercontent.com/u/9326960?v=3","gravatar_id":"","url":"https://api.github.com/users/scott-gray","html_url":"https://github.com/scott-gray","followers_url":"https://api.github.com/users/scott-gray/followers","following_url":"https://api.github.com/users/scott-gray/following{/other_user}","gists_url":"https://api.github.com/users/scott-gray/gists{/gist_id}","starred_url":"https://api.github.com/users/scott-gray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scott-gray/subscriptions","organizations_url":"https://api.github.com/users/scott-gray/orgs","repos_url":"https://api.github.com/users/scott-gray/repos","events_url":"https://api.github.com/users/scott-gray/events{/privacy}","received_events_url":"https://api.github.com/users/scott-gray/received_events","type":"User","site_admin":false},"created_at":"2016-03-04T04:21:45Z","updated_at":"2016-03-04T04:21:45Z","body":"I should point out that Andrew here not only worked through all the math for Winograd but our long discussions were pretty integral to the successful design of these kernels.\r\n\r\nOh and while we're giving credit we were just discussing that these two guys probably deserve as much as Shmuel Winograd for working out the original math:\r\n\r\nhttps://en.wikipedia.org/wiki/Andrei_Toom\r\nhttps://en.wikipedia.org/wiki/Stephen_Cook\r\n\r\nI'll go into a bit more detail on this in the full blog."}}