{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/1309","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/1309/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/1309/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/1309/events","html_url":"https://github.com/fchollet/keras/issues/1309","id":123025090,"number":1309,"title":"Slow BatchNormalization layers (CPU tested)","user":{"login":"Zebreu","id":1616150,"avatar_url":"https://avatars.githubusercontent.com/u/1616150?v=3","gravatar_id":"","url":"https://api.github.com/users/Zebreu","html_url":"https://github.com/Zebreu","followers_url":"https://api.github.com/users/Zebreu/followers","following_url":"https://api.github.com/users/Zebreu/following{/other_user}","gists_url":"https://api.github.com/users/Zebreu/gists{/gist_id}","starred_url":"https://api.github.com/users/Zebreu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zebreu/subscriptions","organizations_url":"https://api.github.com/users/Zebreu/orgs","repos_url":"https://api.github.com/users/Zebreu/repos","events_url":"https://api.github.com/users/Zebreu/events{/privacy}","received_events_url":"https://api.github.com/users/Zebreu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-18T20:59:17Z","updated_at":"2016-05-31T12:30:54Z","closed_at":null,"body":"Hi,\r\n\r\nI added BatchNormalization layers to my model and it suddenly took much more time to train. \r\n\r\nIt takes 561 seconds for one epoch with them:\r\nEpoch 1/1\r\n4096/4096 [==============================] - 561s - loss: 0.0946 - acc: 0.9006\r\n\r\nThis is if I comment out the BatchNorm layers (186 seconds):\r\nEpoch 1/1\r\n4096/4096 [==============================] - 186s - loss: 4.5043 - acc: 0.5933\r\n\r\nI wrote on the user group and someone informed me that adding Dropout slows it down further, but even without Dropout this is much slower (above 500 seconds too).\r\n\r\nIs it a CPU-related issue? Or is such a slowdown expected? Below is my model.\r\n\r\n```\r\n    model = Sequential()\r\n    model.add(Convolution2D(32, 11, 11, subsample=(4,4),input_shape=(3,227,227)))\r\n    model.add(PReLU())\r\n    model.add(BatchNormalization())\r\n\r\n    model.add(Convolution2D(64, 5, 5, subsample=(2,2)))\r\n    model.add(PReLU())\r\n    model.add(BatchNormalization())\r\n\r\n    model.add(Convolution2D(64, 3, 3))\r\n    model.add(PReLU())\r\n    #model.add(BatchNormalization())\r\n    model.add(MaxPooling2D((2,2), strides=(2,2)))\r\n    \r\n    model.add(Flatten())\r\n    model.add(BatchNormalization())\r\n    model.add(Dense(400))\r\n    model.add(PReLU())\r\n    model.add(BatchNormalization())\r\n    model.add(Dropout(0.25))\r\n\r\n    model.add(Dense(400))\r\n    model.add(PReLU())\r\n    model.add(BatchNormalization())\r\n    model.add(Dropout(0.25))\r\n    \r\n    model.add(Dense(3, activation='softmax'))\r\n    optimizer = keras.optimizers.RMSprop(lr=0.0005, rho=0.9, epsilon=1e-6)\r\n        \r\n    model.compile(loss='categorical_crossentropy',optimizer=optimizer)\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/222674207","html_url":"https://github.com/fchollet/keras/issues/1309#issuecomment-222674207","issue_url":"https://api.github.com/repos/fchollet/keras/issues/1309","id":222674207,"user":{"login":"kmul00","id":6665222,"avatar_url":"https://avatars.githubusercontent.com/u/6665222?v=3","gravatar_id":"","url":"https://api.github.com/users/kmul00","html_url":"https://github.com/kmul00","followers_url":"https://api.github.com/users/kmul00/followers","following_url":"https://api.github.com/users/kmul00/following{/other_user}","gists_url":"https://api.github.com/users/kmul00/gists{/gist_id}","starred_url":"https://api.github.com/users/kmul00/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmul00/subscriptions","organizations_url":"https://api.github.com/users/kmul00/orgs","repos_url":"https://api.github.com/users/kmul00/repos","events_url":"https://api.github.com/users/kmul00/events{/privacy}","received_events_url":"https://api.github.com/users/kmul00/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T12:30:53Z","updated_at":"2016-05-31T12:30:53Z","body":"Hi,\r\n\r\nAny update on this ?\r\n\r\nThe `BatchNorm` is Keras is till tremendously slow. Is there any way to alleviate the problem ?\r\n\r\nAnd it increases linearly with batch size. So even if my GPU is under utilized, I cannot increase my batch size, since that would make it even more slower."}}
{"actor":{"display_login":"kmul00"}}