{"action":"created","issue":{"url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/issues/277","repository_url":"https://api.github.com/repos/tiny-dnn/tiny-dnn","labels_url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/issues/277/labels{/name}","comments_url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/issues/277/comments","events_url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/issues/277/events","html_url":"https://github.com/tiny-dnn/tiny-dnn/issues/277","id":171154813,"number":277,"title":"Performance optimization","user":{"login":"Abai","id":592671,"avatar_url":"https://avatars.githubusercontent.com/u/592671?v=3","gravatar_id":"","url":"https://api.github.com/users/Abai","html_url":"https://github.com/Abai","followers_url":"https://api.github.com/users/Abai/followers","following_url":"https://api.github.com/users/Abai/following{/other_user}","gists_url":"https://api.github.com/users/Abai/gists{/gist_id}","starred_url":"https://api.github.com/users/Abai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Abai/subscriptions","organizations_url":"https://api.github.com/users/Abai/orgs","repos_url":"https://api.github.com/users/Abai/repos","events_url":"https://api.github.com/users/Abai/events{/privacy}","received_events_url":"https://api.github.com/users/Abai/received_events","type":"User","site_admin":false},"labels":[{"id":399800909,"url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/labels/performance","name":"performance","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-08-15T11:40:08Z","updated_at":"2016-12-06T16:30:14Z","closed_at":null,"body":"Hi all,\n\nI have benchmarked tiny-cnn vs caffe on CPU using pretrained bvlc_reference_caffenet model provided by caffe. \n\nMy findings are that the convolution and fully connected layer implementation are significantly slower in tiny-cnn. Do you have thoughts on why this is and how we can improve this? Does tiny-cnn use some kind of im2col routine already? \n\nA bit about the benchmark. I tested a somewhat old commit from March 15 2016 a9566139154a666bc073209e12d6f124052067e9. I need to update my benchmarking code to work with a newer version as the library changed significantly since than. I'll do this as soon as I can. Meanwhile I am hoping that these results could still be useful for initial insights.\n\nEach benchmark was run 1000 iterations and averaged. Below are the cmake settings for both tiny-cnn and caffe:\n\n**Settings tiny-cnn:**\n\n```\n-- -------------------------------------------------------------------------------\n-- General configuration for tiny_cnn 0.0.1\n-- -------------------------------------------------------------------------------\n-- BUILD_EXAMPLES=OFF BUILD_TESTS=OFF USE_TBB=ON USE_OMP=OFF USE_SSE=ON USE_AVX=ON\n-- -------------------------------------------------------------------------------\n--\n    Built as dynamic libs?:ON\n    Compiler:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++\n-- C++ flags (Release):           -msse3  -mavx  -std=c++11 -Wall -Wpedantic  -O3\n-- C++ flags (Debug):             -msse3  -mavx  -std=c++11 -Wall -Wpedantic  -g3\n-- CMAKE_CXX_FLAGS:             -msse3  -mavx  -std=c++11 -Wall -Wpedantic\n-- OpenCV libs:              opencv_core;opencv_highgui;opencv_imgproc;opencv_objdetect;opencv_imgcodecs\n--\n-- CMAKE_SYSTEM_PROCESSOR = x86_64\n-- REQUIRED_LIBRARIES = /usr/local/lib/libtbb.dylib;/usr/local/lib/libtbbmalloc.dylib\n-- CMAKE_BUILD_TYPE = Release\n```\n\n**Settings caffe:**\n\n```\n-- ******************* Caffe Configuration Summary *******************\n-- General:\n--   Version           :   1.0.0-rc3\n--   Git               :   rc3-82-gbe163be\n--   System            :   Darwin\n--   C++ compiler      :   /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++\n--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized\n--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized\n--   Build type        :   Release\n--\n--   BUILD_SHARED_LIBS :   ON\n--   BUILD_python      :   OFF\n--   BUILD_matlab      :   OFF\n--   BUILD_docs        :   OFF\n--   CPU_ONLY          :   ON\n--   USE_OPENCV        :   ON\n--   USE_LEVELDB       :   ON\n--   USE_LMDB          :   ON\n--   ALLOW_LMDB_NOLOCK :   OFF\n--\n-- Dependencies:\n--   BLAS              :   Yes (vecLib)\n--   Boost             :   Yes (ver. 1.60)\n--   glog              :   Yes\n--   gflags            :   Yes\n--   protobuf          :   Yes (ver. 2.6.1)\n--   lmdb              :   Yes (ver. 0.9.14)\n--   LevelDB           :   Yes (ver. 1.18)\n--   Snappy            :   Yes (ver. 1.1.3)\n--   OpenCV            :   Yes (ver. 3.1.0)\n--   CUDA              :   No\n```\n\n**Performance**\n\n```\ncaffe    bvlc_reference_caffenet forward    48.0482 ms\ntiny-cnn bvlc_reference_caffenet forward    486.034 ms\n\ncaffe    dropout forward                    0.000481 ms\ntiny-cnn dropout forward                    0.002832 ms\n\ncaffe    relu forward                       0.066614 ms\ntiny-cnn relu forward                       0.596546 ms\n\ncaffe    lrn forward                        1.52823 ms\ntiny-cnn lrn forward                        1.23587 ms\n\ncaffe    fully connected forward            16.4621 ms\ntiny-cnn fully connected forward            192.811 ms\n\ncaffe    pool forward                       2.38592 ms\ntiny-cnn pool forward                       1.07456 ms\n\ncaffe    conv forward                       1.78803 ms\ntiny-cnn conv forward                       15.6862 ms\n```\n\nWould be glad to hear your thoughts on this :)\n"},"comment":{"url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/issues/comments/265197496","html_url":"https://github.com/tiny-dnn/tiny-dnn/issues/277#issuecomment-265197496","issue_url":"https://api.github.com/repos/tiny-dnn/tiny-dnn/issues/277","id":265197496,"user":{"login":"Abai","id":592671,"avatar_url":"https://avatars.githubusercontent.com/u/592671?v=3","gravatar_id":"","url":"https://api.github.com/users/Abai","html_url":"https://github.com/Abai","followers_url":"https://api.github.com/users/Abai/followers","following_url":"https://api.github.com/users/Abai/following{/other_user}","gists_url":"https://api.github.com/users/Abai/gists{/gist_id}","starred_url":"https://api.github.com/users/Abai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Abai/subscriptions","organizations_url":"https://api.github.com/users/Abai/orgs","repos_url":"https://api.github.com/users/Abai/repos","events_url":"https://api.github.com/users/Abai/events{/privacy}","received_events_url":"https://api.github.com/users/Abai/received_events","type":"User","site_admin":false},"created_at":"2016-12-06T16:30:14Z","updated_at":"2016-12-06T16:30:14Z","body":"@Randl @edgarriba \r\n\r\n[Here](https://gist.github.com/Abai/273fe51faadb77807b79879507fe945a) is a more clear Markdown overview of a test I ran today in.\r\n\r\nI have a feeling that something is wrong with the CPU time measurements, but the real time (last two columns) measurements are correct as far as I can see. \r\n\r\nTo clarify, the loading of weights and input data is not included in the measured time. So this is purely `layer->forward()` calls on each layer. I know that forward() retrieves some data internally. If there is a better way to test just the forward propagation, let me know and I will change the benchmark."}}
{"actor":{"display_login":"Abai"}}