{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/768","repository_url":"https://api.github.com/repos/deeplearning4j/nd4j","labels_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/768/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/768/comments","events_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/768/events","html_url":"https://github.com/deeplearning4j/nd4j/issues/768","id":145082505,"number":768,"title":"dl4j examples performance: 3.8 vs. 3.9-native","user":{"login":"AlexDBlack","id":2360237,"avatar_url":"https://avatars.githubusercontent.com/u/2360237?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deeplearning4j/nd4j/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2016-04-01T04:19:16Z","updated_at":"2016-04-20T02:55:53Z","closed_at":null,"body":"Summary of example issues and times for 3.8 vs 3.9.\r\nTest system:\r\n- Windows 8.1\r\n- i7-3930k (6 core)\r\n- OpenBLAS 0.2.14 for both 3.8 and 3.9\r\n\r\n***Last updated: 4am, Tuesday 12th April PDT***\r\n\r\n**Feed forward examples**\r\n\r\n**Name** | **Time (3.8) (ms), 3 run avg** | **Time (3.9) (ms), 3 run avg** | Notes\r\n--- | --- | --- | ----\r\nMLPClassifierMoon | **5100** | 11178 | (time just for fit)\r\nMLPMnistSingleLayerExample | 65628 | **27598** | *1 epoch only, fit only* (before bettertoflattened: 54891ms; as of 3 April: 41789ms)\r\n\r\nExact file I'm running for this: https://gist.github.com/AlexDBlack/a0ec053b86e29f67327365d47a213931\r\n\r\nVisualVM: MLPMnistSingleLayerExample:\r\n![image](https://cloud.githubusercontent.com/assets/2360237/14457477/14b8b218-00ee-11e6-895d-978474db2a65.png)\r\n\r\n\r\nOf note here:\r\n* putScalar is from the MnistDataSetIterator\r\n\r\n\r\n---\r\n---\r\n---\r\n\r\n**Convolutional Examples**\r\n\r\n(Note: Times not yet updated post bettertoflattened, but this probably won't help much here).\r\n\r\n**Name** | **Time (3.8) (ms)** | **Time (3.9) (ms)** | Notes\r\n--- | --- | --- | ----\r\n~~LenetMnistExample~~ | ~~**863291**~~ | ~~1031003~~ | ~~*1 epoch only*. nd4j-native: poor CPU utilization. Bottleneck: **tensorMmul (reshape + assign > mmul part)**; also [**this**](https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-core/src/main/java/org/deeplearning4j/nn/layers/convolution/subsampling/SubsamplingLayer.java#L101) is approx 60% of total runtime~~\r\n\r\n**Note**: This needs to be updated to the new LenetMnistExample. Will do so once we have subsampling optimization fixed\r\n\r\n---\r\n---\r\n---\r\n\r\n**RNN Examples**\r\n\r\n**Name** | **Time (3.8) (ms)** (avg, 2 runs) | **Time (3.9) (ms)** (avg, 2 runs) | Notes\r\n--- | --- | --- | ----\r\nGravesLSTMCharModellingExample | 130345 | **123306** | Bottlenecks: see below. Also note: some performance optimizations on the DL4J side (smaller number of larger ops, etc) should be possible.\r\n\r\nExact code I'm running: https://gist.github.com/AlexDBlack/819d9e89c9164b678e6d0cba03d89346\r\n\r\n![image](https://cloud.githubusercontent.com/assets/2360237/14457834/c5f9f134-00f0-11e6-9f46-c63a25397acf.png)\r\n\r\n[OLD visualvm, for reference]\r\n![image](https://cloud.githubusercontent.com/assets/2360237/14237750/83e21640-fa6b-11e5-9d11-237bf076d35a.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/comments/212225853","html_url":"https://github.com/deeplearning4j/nd4j/issues/768#issuecomment-212225853","issue_url":"https://api.github.com/repos/deeplearning4j/nd4j/issues/768","id":212225853,"user":{"login":"saudet","id":3328023,"avatar_url":"https://avatars.githubusercontent.com/u/3328023?v=3","gravatar_id":"","url":"https://api.github.com/users/saudet","html_url":"https://github.com/saudet","followers_url":"https://api.github.com/users/saudet/followers","following_url":"https://api.github.com/users/saudet/following{/other_user}","gists_url":"https://api.github.com/users/saudet/gists{/gist_id}","starred_url":"https://api.github.com/users/saudet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saudet/subscriptions","organizations_url":"https://api.github.com/users/saudet/orgs","repos_url":"https://api.github.com/users/saudet/repos","events_url":"https://api.github.com/users/saudet/events{/privacy}","received_events_url":"https://api.github.com/users/saudet/received_events","type":"User","site_admin":false},"created_at":"2016-04-20T02:55:53Z","updated_at":"2016-04-20T02:55:53Z","body":"@treo That particular issue I spotted is apparently caused by inefficient code used to subsample for CNNs here: https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j-core/src/main/java/org/deeplearning4j/nn/layers/convolution/subsampling/SubsamplingLayer.java#L101\r\nThis causes over 235000 allocations per iteration in LenetMnistExample, but @AlexDBlack knows how to fix this, now that IsMax is working, right?\r\n\r\nBTW, it doesn't change anything on Fedora 23 if we link with jemalloc explicitly, so I'm guessing recent versions of glibc do something very similar to jemalloc."}}