{"action":"created","issue":{"url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/101","repository_url":"https://api.github.com/repos/soumith/convnet-benchmarks","labels_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/101/labels{/name}","comments_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/101/comments","events_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/101/events","html_url":"https://github.com/soumith/convnet-benchmarks/issues/101","id":148512920,"number":101,"title":"DeepMark","user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":42,"created_at":"2016-04-14T23:41:06Z","updated_at":"2016-04-19T02:43:37Z","closed_at":null,"body":"Hi all,\r\n\r\nThe reason I've been slow on convnet-benchmarks these days is because i've been working on the side on DeepMark.\r\n\r\nI initially wrote *convnet-benchmarks* to increase competition among frameworks so that we can work towards faster ConvNets, and they served their purpose well. After the release of convnet-benchmarks, multiple frameworks pulled up their socks to speedup convnets, with a deep sense of prestige for being on top of these benchmarks. In these two years, we as a community accelerated GPU ConvNets across all frameworks between 4x to 10x, efficiently implementing tricks such as FFT, Winograd, and powered by faster hardware. Alex Khrizevsky, Yangqing Jia, Scott Gray, Nicolas Vasilache, Sander Dieleman, Michael Mathieu, Julien Denmouth and many other human compilers helped make this a reality -- looking at the diversity in terms of where each of us work(ed) professionally shows that this kind of acceleration was truly a community effort with a ton of openness, something that is plain awesome! :)\r\nI've also enjoyed reading the deeply technical discussions that take place on convnet-benchmarks (my favorites: https://github.com/soumith/convnet-benchmarks/issues/93 , https://github.com/soumith/convnet-benchmarks/issues/66 , https://github.com/soumith/convnet-benchmarks/issues/59 in recent times ).\r\n\r\nMoving on,  *convnet-benchmarks* do not accurately capture everything we think of when we say *deep learning*. We don't have Recurrent Nets, we don't have video use-cases, speech, NLP etc. There is a need for such comprehensive benchmarks, especially as the space is getting ready for dedicated hardware chips, multi-GPU and multi-machine frameworks and more complex use-cases.\r\n\r\nI've sat down with a few of you at NIPS and GTC to discuss and freeze the initial round of benchmarks for what I am calling **DeepMark**. My initial plan was to work on the initial set of benchmark scripts by myself and cover the most popular frameworks, and then let the direction and maintenance of the benchmarks be community-driven. But the breadth of this effort has been overwhelming to say the least. After careful thought, I've decided that I'll just ask everyone to pitch in for their part of the benchmarks with making scripts etc., especially as many of you were very receptive to the idea offline.\r\n\r\nHere are the initial set of use-cases we want to cover:\r\n\r\n## Networks\r\n### Images\r\n- InceptionV3-batchnorm (http://arxiv.org/abs/1512.00567 , https://github.com/Moodstocks/inception-v3.torch)\r\n- Alexnet-OWT\r\n- VGG\r\n- ResNet-50 ( http://arxiv.org/abs/1512.03385 , https://github.com/facebook/fb.resnet.torch )\r\n\r\n### Video\r\n- C3D - A vgg-style 3D net ( http://vlg.cs.dartmouth.edu/c3d/ )\r\n\r\n### Audio\r\n- DeepSpeech2 - Convnet + RNN + FC ( http://arxiv.org/abs/1512.02595 )\r\n- MSR's 5 layer FC net ( https://github.com/Alexey-Kamenev/Benchmarks )\r\n\r\n### Text\r\n- Small RNN LSTM ( https://github.com/karpathy/char-rnn/blob/master/train.lua#L38-L48 )\r\n- Large RNN LSTM ( BIG-LSTM in http://arxiv.org/abs/1602.02410 )\r\n\r\n\r\n### Platform\r\n- Initially multi-GPU with (1 to 4 titan-X cards)\r\n- However, multi-machine, custom hardware, other GPU cards such as AMD, CPUs etc. can and should be accommodated, we will work this out after the initial push.\r\n\r\n## Metrics\r\n- Round-trip time for 1 epoch of training (will define an epoch size separately for each network)\r\n- Maximum batch-size that fits (to show and focus on the extra memory consumption that the framework uses)\r\n\r\n## Frameworks\r\nEveryone who wants to join-in, but I thought an initial set that is important to cover would be:\r\n- Caffe\r\n- Chainer\r\n- MXNet\r\n- Neon\r\n- Theano\r\n- TensorFlow\r\n- Torch\r\n\r\n## Scripts format\r\n- Emit JSON output (so that the README -- or jekyll website can be auto-generated, similar to http://autumnai.com/deep-learning-benchmarks )\r\n\r\n## Guarantees\r\n- I will personally to the best of my abilities make sure that the benchmarking is fair and unbiased. The hope is that the community at large will watch these and point-out / fix mistakes.\r\n\r\n## Governance\r\n- The benchmarks will be placed at https://github.com/DeepMark/deepmark and other key community members / organizations who want ownership will be welcome to join in proposing new benchmarks that get relevant as the field progresses.\r\n\r\n## Timeframe\r\n- Initial Release: June 15th, 2016\r\n\r\nMy hope is that these new set of benchmarks will not only increase competition but will also be beneficial in other ways to the community, serving as common examples to get started, etc.\r\n\r\nLet me know what you think :)\r\nSoumith\r\n\r\ncc: @hughperkins @f0k @scott-gray @rajatmonga @vrv @benanne @nouiz @Yangqing @tqchen @unnonouno"},"comment":{"url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/comments/211702398","html_url":"https://github.com/soumith/convnet-benchmarks/issues/101#issuecomment-211702398","issue_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/101","id":211702398,"user":{"login":"ajtulloch","id":1121581,"avatar_url":"https://avatars.githubusercontent.com/u/1121581?v=3","gravatar_id":"","url":"https://api.github.com/users/ajtulloch","html_url":"https://github.com/ajtulloch","followers_url":"https://api.github.com/users/ajtulloch/followers","following_url":"https://api.github.com/users/ajtulloch/following{/other_user}","gists_url":"https://api.github.com/users/ajtulloch/gists{/gist_id}","starred_url":"https://api.github.com/users/ajtulloch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ajtulloch/subscriptions","organizations_url":"https://api.github.com/users/ajtulloch/orgs","repos_url":"https://api.github.com/users/ajtulloch/repos","events_url":"https://api.github.com/users/ajtulloch/events{/privacy}","received_events_url":"https://api.github.com/users/ajtulloch/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T02:43:37Z","updated_at":"2016-04-19T02:43:37Z","body":" AFAIK it's just a model for benchmarking, probably derived from some of their fully-connected speech recognition nets. It's similar to the net MSR used in the 1-bit SGD paper for speech recognition (http://research.microsoft.com/pubs/230137/IS140694.PDF), but not identical."}}