{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/2156","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/2156/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/2156/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/2156/events","html_url":"https://github.com/fchollet/keras/issues/2156","id":145048445,"number":2156,"title":"Tensorflow backend is 4x slower than Theano","user":{"login":"elanmart","id":10772830,"avatar_url":"https://avatars.githubusercontent.com/u/10772830?v=3","gravatar_id":"","url":"https://api.github.com/users/elanmart","html_url":"https://github.com/elanmart","followers_url":"https://api.github.com/users/elanmart/followers","following_url":"https://api.github.com/users/elanmart/following{/other_user}","gists_url":"https://api.github.com/users/elanmart/gists{/gist_id}","starred_url":"https://api.github.com/users/elanmart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elanmart/subscriptions","organizations_url":"https://api.github.com/users/elanmart/orgs","repos_url":"https://api.github.com/users/elanmart/repos","events_url":"https://api.github.com/users/elanmart/events{/privacy}","received_events_url":"https://api.github.com/users/elanmart/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-31T23:53:19Z","updated_at":"2016-04-01T10:33:32Z","closed_at":null,"body":"@fchollet this should interest you.\r\n\r\nBelow you can find attached timings for 3 runs of the model with different backends. \r\nIt's a simple FFN, 3 layers + dropouts + batchnorm (see the attached code for details)\r\n\r\nIt seems like TensorFlow backend is extremely inefficient and should not be used at the moment. \r\nEven with native TF code, Theano still comes on top. Perhaps this should be mentioned in the docs.\r\n\r\nPerhaps 1.0 release will make it easier to work with TF itself and make the tf backend more useful?\r\n\r\n`TensorFlow` is a standard `K.backend`\r\n`TensorFlowNative` is a quick hack to replace everything from softmax up (softmax, optimizer, updates) with TF code\r\n`Theano` is benched with and without `allow_gc`\r\n\r\nGPU: `GTX 980` + `cuda-7.5` + `cudnn-4.0.7`\r\n\r\nBackend | Timing\r\n--- | ---\r\nTensorFlow | `[29.6s, 29.8s, 30.6s]`\r\nTensorFlowNative | `[11.6s, 11.4s, 11.5s]`\r\nTheano noGC | `[7.4s, 7.6s, 7.4s]`\r\nTheano GC | `[7.7s, 7.66s, 7.66s]`\r\n\r\nHere's the script to reproduce those results.\r\n\r\n```python\r\nimport numpy as np\r\nimport numpy.random as nr\r\n\r\nfrom keras.utils import np_utils\r\nfrom keras.layers.core import Layer, Dense, Dropout\r\nfrom keras.layers.normalization import BatchNormalization\r\nfrom keras.models import Sequential\r\n\r\nX = np.random.randn(10000, 500)\r\ny = np.random.randint(0, 64, size=(10000, 1))\r\ny = np_utils.to_categorical(y)\r\n\r\ntest = Sequential()\r\ntest.add(Layer(input_shape=(500, )))\r\n\r\ntest.add(Dense(1024, activation='relu'))\r\ntest.add(Dropout(0.5))\r\ntest.add(BatchNormalization())\r\n\r\ntest.add(Dense(1024, activation='relu'))\r\ntest.add(Dropout(0.5))\r\ntest.add(BatchNormalization())\r\n\r\ntest.add(Dense(1024, activation='relu'))\r\ntest.add(Dropout(0.5))\r\ntest.add(BatchNormalization())\r\n\r\ntest.add(Dense(64, activation='softmax'))\r\n\r\ntest.compile('adam', 'categorical_crossentropy')\r\n\r\n%%time \r\ntest.fit(X, y, nb_epoch=25, verbose=0, )\r\n```"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/204342820","html_url":"https://github.com/fchollet/keras/issues/2156#issuecomment-204342820","issue_url":"https://api.github.com/repos/fchollet/keras/issues/2156","id":204342820,"user":{"login":"meanmee","id":8026086,"avatar_url":"https://avatars.githubusercontent.com/u/8026086?v=3","gravatar_id":"","url":"https://api.github.com/users/meanmee","html_url":"https://github.com/meanmee","followers_url":"https://api.github.com/users/meanmee/followers","following_url":"https://api.github.com/users/meanmee/following{/other_user}","gists_url":"https://api.github.com/users/meanmee/gists{/gist_id}","starred_url":"https://api.github.com/users/meanmee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/meanmee/subscriptions","organizations_url":"https://api.github.com/users/meanmee/orgs","repos_url":"https://api.github.com/users/meanmee/repos","events_url":"https://api.github.com/users/meanmee/events{/privacy}","received_events_url":"https://api.github.com/users/meanmee/received_events","type":"User","site_admin":false},"created_at":"2016-04-01T10:33:32Z","updated_at":"2016-04-01T10:33:32Z","body":"in my computer they are almost the same runtime"}}