{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6718","repository_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6718/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6718/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6718/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/6718","id":381281979,"node_id":"MDU6SXNzdWUzODEyODE5Nzk=","number":6718,"title":"performance issues on gpu box","user":{"login":"fmorbinitc","id":22603687,"node_id":"MDQ6VXNlcjIyNjAzNjg3","avatar_url":"https://avatars1.githubusercontent.com/u/22603687?v=4","gravatar_id":"","url":"https://api.github.com/users/fmorbinitc","html_url":"https://github.com/fmorbinitc","followers_url":"https://api.github.com/users/fmorbinitc/followers","following_url":"https://api.github.com/users/fmorbinitc/following{/other_user}","gists_url":"https://api.github.com/users/fmorbinitc/gists{/gist_id}","starred_url":"https://api.github.com/users/fmorbinitc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmorbinitc/subscriptions","organizations_url":"https://api.github.com/users/fmorbinitc/orgs","repos_url":"https://api.github.com/users/fmorbinitc/repos","events_url":"https://api.github.com/users/fmorbinitc/events{/privacy}","received_events_url":"https://api.github.com/users/fmorbinitc/received_events","type":"User","site_admin":false},"labels":[{"id":67900999,"node_id":"MDU6TGFiZWw2NzkwMDk5OQ==","url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/labels/Bug","name":"Bug","color":"d93f0b","default":false},{"id":372474058,"node_id":"MDU6TGFiZWwzNzI0NzQwNTg=","url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/labels/Performance","name":"Performance","color":"f9d0c4","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2018-11-15T18:17:01Z","updated_at":"2018-11-19T16:33:21Z","closed_at":null,"author_association":"NONE","body":"two machines:\r\ncpu box is an apple laptop with 8 cores\r\ngpu box is a p2.xlarge aws box which has 4 cpu cores and a tesla k80 gpu\r\n\r\nreporting PerformanceListener for first 4 iterations, batch size always equal to 1000.\r\n\r\nno particular mem configuration for the jvm on all machines.\r\n\r\n**gpu machine (using gpu for both nd4j and cudnn, gc setup using: Nd4j.getMemoryManager().setOccasionalGcFrequency(10000))**\r\n\r\n81148 [main] INFO  o.d.o.l.PerformanceListener - ETL: 15068 ms; iteration 0; iteration time: 30044 ms; samples/sec: 33.285; batches/sec: 0.033; score: 18.329551218376967;\r\n98099 [main] INFO  o.d.o.l.PerformanceListener - ETL: 11029 ms; iteration 1; iteration time: 16951 ms; samples/sec: 58.994; batches/sec: 0.059; score: 61.64040843607799;\r\n110852 [main] INFO  o.d.o.l.PerformanceListener - ETL: 6976 ms; iteration 2; iteration time: 12753 ms; samples/sec: 78.413; batches/sec: 0.078; score: 131.75989663147402;\r\n123493 [main] INFO  o.d.o.l.PerformanceListener - ETL: 7690 ms; iteration 3; iteration time: 12641 ms; samples/sec: 79.108; batches/sec: 0.079; score: 139.66879515652315;\r\n\r\n**gpu machine, cpu mode:**\r\n38761 [main] INFO  o.d.o.l.PerformanceListener - ETL: 751 ms; iteration 0; iteration time: 3375 ms; samples/sec: 296.296; batches/sec: 0.296; score: 17.973563904870705;\r\n44856 [main] INFO  o.d.o.l.PerformanceListener - ETL: 674 ms; iteration 1; iteration time: 6094 ms; samples/sec: 164.096; batches/sec: 0.164; score: 60.711884561402954;\r\n49045 [main] INFO  o.d.o.l.PerformanceListener - ETL: 465 ms; iteration 2; iteration time: 4189 ms; samples/sec: 238.720; batches/sec: 0.239; score: 129.93905362345566;\r\n52998 [main] INFO  o.d.o.l.PerformanceListener - ETL: 670 ms; iteration 3; iteration time: 3953 ms; samples/sec: 252.972; batches/sec: 0.253; score: 151.48013304498966;\r\n\r\n\r\n**cpu machine (laptop, cpu only)**\r\nINFO  09:16:54.501 [main           ] [PerformanceListener      ] ETL: 703 ms; iteration 0; iteration time: 2492 ms; samples/sec: 401.284; batches/sec: 0.401; score: 18.235179144737707;\r\nINFO  09:16:56.990 [main           ] [PerformanceListener      ] ETL: 461 ms; iteration 1; iteration time: 2489 ms; samples/sec: 401.768; batches/sec: 0.402; score: 61.1129676358959;\r\nINFO  09:16:58.967 [main           ] [PerformanceListener      ] ETL: 420 ms; iteration 2; iteration time: 1977 ms; samples/sec: 505.817; batches/sec: 0.506; score: 136.98393463715627;\r\nINFO  09:17:00.830 [main           ] [PerformanceListener      ] ETL: 425 ms; iteration 3; iteration time: 1863 ms; samples/sec: 536.769; batches/sec: 0.537; score: 146.0492742728752;\r\n\r\ngpu machine and cpu machine perform similarly when running in cpu mode, of course the gpu machine is slower as it has half the cores.\r\nhowever when run in gpu mode performance is dismal.\r\n\r\ncan you suggest setup we could try? network configuration is similar to what is described in https://gist.github.com/fmorbini/936a7f1c9b0aa905e3ecf71dc096cb30"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/439956194","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/6718#issuecomment-439956194","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/6718","id":439956194,"node_id":"MDEyOklzc3VlQ29tbWVudDQzOTk1NjE5NA==","user":{"login":"fmorbinitc","id":22603687,"node_id":"MDQ6VXNlcjIyNjAzNjg3","avatar_url":"https://avatars1.githubusercontent.com/u/22603687?v=4","gravatar_id":"","url":"https://api.github.com/users/fmorbinitc","html_url":"https://github.com/fmorbinitc","followers_url":"https://api.github.com/users/fmorbinitc/followers","following_url":"https://api.github.com/users/fmorbinitc/following{/other_user}","gists_url":"https://api.github.com/users/fmorbinitc/gists{/gist_id}","starred_url":"https://api.github.com/users/fmorbinitc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fmorbinitc/subscriptions","organizations_url":"https://api.github.com/users/fmorbinitc/orgs","repos_url":"https://api.github.com/users/fmorbinitc/repos","events_url":"https://api.github.com/users/fmorbinitc/events{/privacy}","received_events_url":"https://api.github.com/users/fmorbinitc/received_events","type":"User","site_admin":false},"created_at":"2018-11-19T16:33:21Z","updated_at":"2018-11-19T16:33:21Z","author_association":"NONE","body":"@raver119 here is the ETL info for 256 batch size case, first 20 iterations:\r\n\r\n3395475 [main] INFO  o.d.o.l.PerformanceListener - ETL: 1405 ms; iteration 0; iteration time: 9843 ms; samples/sec: 26.008; batches/sec: 0.102; score: 18.15232703624806;\r\n3399812 [main] INFO  o.d.o.l.PerformanceListener - ETL: 1321 ms; iteration 1; iteration time: 4337 ms; samples/sec: 59.027; batches/sec: 0.231; score: 17.983460623292984;\r\n3402915 [main] INFO  o.d.o.l.PerformanceListener - ETL: 1034 ms; iteration 2; iteration time: 3103 ms; samples/sec: 82.501; batches/sec: 0.322; score: 18.995852123519533;\r\n3407018 [main] INFO  o.d.o.l.PerformanceListener - ETL: 1085 ms; iteration 3; iteration time: 4103 ms; samples/sec: 62.393; batches/sec: 0.244; score: 19.05778558468622;\r\n3411049 [main] INFO  o.d.o.l.PerformanceListener - ETL: 78 ms; iteration 4; iteration time: 4031 ms; samples/sec: 63.508; batches/sec: 0.248; score: 17.762127788516647;\r\n3413107 [main] INFO  o.d.o.l.PerformanceListener - ETL: 65 ms; iteration 5; iteration time: 2058 ms; samples/sec: 124.393; batches/sec: 0.486; score: 18.31833226202926;\r\n3416049 [main] INFO  o.d.o.l.PerformanceListener - ETL: 960 ms; iteration 6; iteration time: 2942 ms; samples/sec: 87.016; batches/sec: 0.340; score: 16.736513300771;\r\n3419010 [main] INFO  o.d.o.l.PerformanceListener - ETL: 60 ms; iteration 7; iteration time: 2961 ms; samples/sec: 86.457; batches/sec: 0.338; score: 16.235420508595805;\r\n3422088 [main] INFO  o.d.o.l.PerformanceListener - ETL: 71 ms; iteration 8; iteration time: 3078 ms; samples/sec: 83.171; batches/sec: 0.325; score: 16.521826175915162;\r\n3425092 [main] INFO  o.d.o.l.PerformanceListener - ETL: 60 ms; iteration 9; iteration time: 3004 ms; samples/sec: 85.220; batches/sec: 0.333; score: 17.382304153042043;\r\n3428064 [main] INFO  o.d.o.l.PerformanceListener - ETL: 972 ms; iteration 10; iteration time: 2972 ms; samples/sec: 86.137; batches/sec: 0.336; score: 16.33120707225146;\r\n3431057 [main] INFO  o.d.o.l.PerformanceListener - ETL: 67 ms; iteration 11; iteration time: 2993 ms; samples/sec: 85.533; batches/sec: 0.334; score: 15.971231363672402;\r\n3434041 [main] INFO  o.d.o.l.PerformanceListener - ETL: 73 ms; iteration 12; iteration time: 2984 ms; samples/sec: 85.791; batches/sec: 0.335; score: 16.769584922012893;\r\n3437020 [main] INFO  o.d.o.l.PerformanceListener - ETL: 68 ms; iteration 13; iteration time: 2978 ms; samples/sec: 85.964; batches/sec: 0.336; score: 15.55782221419269;\r\n3439063 [main] INFO  o.d.o.l.PerformanceListener - ETL: 62 ms; iteration 14; iteration time: 2043 ms; samples/sec: 125.306; batches/sec: 0.489; score: 15.986110289506193;\r\n3442049 [main] INFO  o.d.o.l.PerformanceListener - ETL: 977 ms; iteration 15; iteration time: 2986 ms; samples/sec: 85.733; batches/sec: 0.335; score: 15.293667938423795;\r\n3445046 [main] INFO  o.d.o.l.PerformanceListener - ETL: 970 ms; iteration 16; iteration time: 2997 ms; samples/sec: 85.419; batches/sec: 0.334; score: 15.342508540370146;\r\n3449128 [main] INFO  o.d.o.l.PerformanceListener - ETL: 1016 ms; iteration 17; iteration time: 4082 ms; samples/sec: 62.714; batches/sec: 0.245; score: 13.99101006679934;\r\n3452188 [main] INFO  o.d.o.l.PerformanceListener - ETL: 999 ms; iteration 18; iteration time: 3060 ms; samples/sec: 83.660; batches/sec: 0.327; score: 13.962916481107207;\r\n3455170 [main] INFO  o.d.o.l.PerformanceListener - ETL: 72 ms; iteration 19; iteration time: 2982 ms; samples/sec: 85.848; batches/sec: 0.335; score: 13.61915423678386;\r\n\r\nprofiling is similar, 70% solver, 22% reading, 7% GC. hotspots are:\r\n![screen shot 2018-11-19 at 8 32 24 am](https://user-images.githubusercontent.com/22603687/48720959-bf02e700-ebd5-11e8-8370-1b8f06cb7323.png)\r\n"}}
{"actor":{"display_login":"fmorbinitc"}}