{"action":"created","issue":{"url":"https://api.github.com/repos/keras-team/keras/issues/5935","repository_url":"https://api.github.com/repos/keras-team/keras","labels_url":"https://api.github.com/repos/keras-team/keras/issues/5935/labels{/name}","comments_url":"https://api.github.com/repos/keras-team/keras/issues/5935/comments","events_url":"https://api.github.com/repos/keras-team/keras/issues/5935/events","html_url":"https://github.com/keras-team/keras/issues/5935","id":216246120,"node_id":"MDU6SXNzdWUyMTYyNDYxMjA=","number":5935,"title":"Memory leak during model.fit()","user":{"login":"HareeshBahuleyan","id":13996331,"node_id":"MDQ6VXNlcjEzOTk2MzMx","avatar_url":"https://avatars3.githubusercontent.com/u/13996331?v=4","gravatar_id":"","url":"https://api.github.com/users/HareeshBahuleyan","html_url":"https://github.com/HareeshBahuleyan","followers_url":"https://api.github.com/users/HareeshBahuleyan/followers","following_url":"https://api.github.com/users/HareeshBahuleyan/following{/other_user}","gists_url":"https://api.github.com/users/HareeshBahuleyan/gists{/gist_id}","starred_url":"https://api.github.com/users/HareeshBahuleyan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HareeshBahuleyan/subscriptions","organizations_url":"https://api.github.com/users/HareeshBahuleyan/orgs","repos_url":"https://api.github.com/users/HareeshBahuleyan/repos","events_url":"https://api.github.com/users/HareeshBahuleyan/events{/privacy}","received_events_url":"https://api.github.com/users/HareeshBahuleyan/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":35,"created_at":"2017-03-23T00:14:01Z","updated_at":"2018-07-31T07:46:44Z","closed_at":"2017-04-12T13:44:12Z","author_association":"NONE","body":"Hi,\r\n\r\nI am trying to train a simple CNN in keras. During training via model.fit(), the  system free memory keeps reducing and eventually it runs out of memory with a \"Killed\" error. When I train it one epoch at a time, I can clearly see the reduction in free memory after each epoch. Is this normal?\r\n\r\n```\r\nmodel = Sequential()\r\nmodel.add(Conv2D(input_shape=(1,30,300), filters=10, kernel_size=(3, 300), padding='valid', \r\n                        data_format=\"channels_first\", activation='relu'))\r\nmodel.add(Reshape((10, 28)))\r\nmodel.add(MaxPooling1D(pool_size=10))\r\n\r\nmodel.add(Flatten())\r\nmodel.add(BatchNormalization())\r\nmodel.add(Dense(20, activation='relu'))\r\nmodel.add(Dropout(0.2))\r\nmodel.add(BatchNormalization())\r\nmodel.add(Dense(10, activation='relu'))\r\nmodel.add(Dropout(0.2))\r\nmodel.add(BatchNormalization())\r\nmodel.add(Dense(5, activation='relu'))\r\nmodel.add(Dropout(0.2))\r\nmodel.add(Dense(1, activation='sigmoid')) \r\n\r\nmodel.compile(loss='binary_crossentropy',\r\n              optimizer='adam',\r\n              metrics=['accuracy'])\r\n\r\nmodel.fit(X_train, Y_train, batch_size=32, epochs=1, validation_data=(X_test, Y_test), verbose=0)\r\n```\r\n\r\nHowever, when I set **batch_size = 1**, I see that it works fine with no memory leaks. \r\n\r\nI am running keras version 2.02 with theano backend on CPU.\r\n\r\nThanks,\r\nHareesh\r\n\r\nEdit: I was not facing this issue with Keras version 1.2.\r\n\r\n- [x] Check that you are up-to-date with the master branch of Keras. You can update with:\r\npip install git+git://github.com/fchollet/keras.git --upgrade --no-deps\r\n\r\n- [x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found [here](https://www.tensorflow.org/get_started/os_setup).\r\n\r\n- [x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:\r\npip install git+git://github.com/Theano/Theano.git --upgrade --no-deps\r\n\r\n- [x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).\r\n"},"comment":{"url":"https://api.github.com/repos/keras-team/keras/issues/comments/409127979","html_url":"https://github.com/keras-team/keras/issues/5935#issuecomment-409127979","issue_url":"https://api.github.com/repos/keras-team/keras/issues/5935","id":409127979,"node_id":"MDEyOklzc3VlQ29tbWVudDQwOTEyNzk3OQ==","user":{"login":"Tixierae","id":13351315,"node_id":"MDQ6VXNlcjEzMzUxMzE1","avatar_url":"https://avatars0.githubusercontent.com/u/13351315?v=4","gravatar_id":"","url":"https://api.github.com/users/Tixierae","html_url":"https://github.com/Tixierae","followers_url":"https://api.github.com/users/Tixierae/followers","following_url":"https://api.github.com/users/Tixierae/following{/other_user}","gists_url":"https://api.github.com/users/Tixierae/gists{/gist_id}","starred_url":"https://api.github.com/users/Tixierae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tixierae/subscriptions","organizations_url":"https://api.github.com/users/Tixierae/orgs","repos_url":"https://api.github.com/users/Tixierae/repos","events_url":"https://api.github.com/users/Tixierae/events{/privacy}","received_events_url":"https://api.github.com/users/Tixierae/received_events","type":"User","site_admin":false},"created_at":"2018-07-31T07:46:44Z","updated_at":"2018-07-31T07:46:44Z","author_association":"NONE","body":"@joelthchao\r\nI'm having the same problem with tensorflow backend (like @BIGBALLON), on Ubuntu 16.04 with keras 2.2.0 and tensorflow-gpu 1.5.0. The RAM gets eat up after each epoch, so if the model is trained for too many epochs I'm out of RAM. I use a custom generator with `use_multiprocessing=False` and `workers=1`."}}
{"actor":{"display_login":"Tixierae"}}