{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30773","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30773/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30773/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30773/events","html_url":"https://github.com/tensorflow/tensorflow/issues/30773","id":468848824,"node_id":"MDU6SXNzdWU0Njg4NDg4MjQ=","number":30773,"title":"[TF 2.0] [Jupyter] Can not compile model more than once without running out of memory","user":{"login":"jbgh2","id":140016,"node_id":"MDQ6VXNlcjE0MDAxNg==","avatar_url":"https://avatars0.githubusercontent.com/u/140016?v=4","gravatar_id":"","url":"https://api.github.com/users/jbgh2","html_url":"https://github.com/jbgh2","followers_url":"https://api.github.com/users/jbgh2/followers","following_url":"https://api.github.com/users/jbgh2/following{/other_user}","gists_url":"https://api.github.com/users/jbgh2/gists{/gist_id}","starred_url":"https://api.github.com/users/jbgh2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbgh2/subscriptions","organizations_url":"https://api.github.com/users/jbgh2/orgs","repos_url":"https://api.github.com/users/jbgh2/repos","events_url":"https://api.github.com/users/jbgh2/events{/privacy}","received_events_url":"https://api.github.com/users/jbgh2/received_events","type":"User","site_admin":false},"labels":[{"id":1160540362,"node_id":"MDU6TGFiZWwxMTYwNTQwMzYy","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/2.0","name":"2.0","color":"e79b38","default":false},{"id":1097547538,"node_id":"MDU6TGFiZWwxMDk3NTQ3NTM4","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/comp:gpu","name":"comp:gpu","color":"0052cc","default":false},{"id":473184161,"node_id":"MDU6TGFiZWw0NzMxODQxNjE=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:support","name":"type:support","color":"159b2e","default":false}],"state":"open","locked":false,"assignee":{"login":"ymodak","id":42785357,"node_id":"MDQ6VXNlcjQyNzg1MzU3","avatar_url":"https://avatars1.githubusercontent.com/u/42785357?v=4","gravatar_id":"","url":"https://api.github.com/users/ymodak","html_url":"https://github.com/ymodak","followers_url":"https://api.github.com/users/ymodak/followers","following_url":"https://api.github.com/users/ymodak/following{/other_user}","gists_url":"https://api.github.com/users/ymodak/gists{/gist_id}","starred_url":"https://api.github.com/users/ymodak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymodak/subscriptions","organizations_url":"https://api.github.com/users/ymodak/orgs","repos_url":"https://api.github.com/users/ymodak/repos","events_url":"https://api.github.com/users/ymodak/events{/privacy}","received_events_url":"https://api.github.com/users/ymodak/received_events","type":"User","site_admin":false},"assignees":[{"login":"ymodak","id":42785357,"node_id":"MDQ6VXNlcjQyNzg1MzU3","avatar_url":"https://avatars1.githubusercontent.com/u/42785357?v=4","gravatar_id":"","url":"https://api.github.com/users/ymodak","html_url":"https://github.com/ymodak","followers_url":"https://api.github.com/users/ymodak/followers","following_url":"https://api.github.com/users/ymodak/following{/other_user}","gists_url":"https://api.github.com/users/ymodak/gists{/gist_id}","starred_url":"https://api.github.com/users/ymodak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymodak/subscriptions","organizations_url":"https://api.github.com/users/ymodak/orgs","repos_url":"https://api.github.com/users/ymodak/repos","events_url":"https://api.github.com/users/ymodak/events{/privacy}","received_events_url":"https://api.github.com/users/ymodak/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2019-07-16T20:29:31Z","updated_at":"2019-07-25T20:46:19Z","closed_at":null,"author_association":"NONE","body":"**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10\r\n- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:\r\n- TensorFlow installed from (source or binary): binary\r\n- TensorFlow version (use command below): 2.0.0-dev20190702\r\n- Python version: 3.6.7\r\n- Bazel version (if compiling from source): None\r\n- GCC/Compiler version (if compiling from source): None\r\n- CUDA/cuDNN version: V10.0.130, 7.3.1\r\n- GPU model and memory: Surface Book 1 Nvidia GPU\r\n\r\n**Describe the current behavior**\r\n\r\nBuild a Keras model, compile it, run it.\r\nTry and rebuild model with new parameters.\r\nResult: OOM on GPU. Memory has not been freed or re-used.\r\n\r\n**Describe the expected behavior**\r\n\r\nCompile a model more than once without the GPU running out of memory.\r\nMore specifically, be able to hyper-parameter tuning without restarting the Jupyter kernel.\r\n\r\nI've tried:\r\n-  set_memory_growth on the GPU\r\n- del model + gc.collect\r\n- clear_session\r\nnone of them help.\r\n\r\n**Code to reproduce the issue**\r\nProvide a reproducible test case that is the bare minimum necessary to generate the problem.\r\n\r\n```\r\n#Import basics and check everything works\r\nimport tensorflow as tf\r\nfrom tensorflow import keras\r\n\r\nAUTOTUNE = tf.data.experimental.AUTOTUNE\r\n\r\nprint(\"Versions:\", tf.version.VERSION, tf.version.GIT_VERSION)\r\nprint(\"GPU availablilty:\", tf.test.is_gpu_available())\r\nprint(\"Eager execution:\", tf.executing_eagerly())\r\n\r\n#Quick test\r\nx = [[2.]]\r\nm = tf.matmul(x, x)\r\nprint(\"hello, {}\".format(m))\r\n\r\ndef make_model(input_shape, n_hidden1=2049, n_hidden2=500, n_hidden3=180, batch_n_mom=0.99, dropout_rate=0.1):\r\n\r\n    from tensorflow.keras.initializers import he_normal\r\n    \r\n    stacked_ae = keras.models.Sequential([\r\n        keras.layers.Flatten(input_shape=input_shape),\r\n        keras.layers.BatchNormalization(axis=1, momentum=batch_n_mom),\r\n        \r\n        keras.layers.Dense(n_hidden1, activation=\"selu\", name=\"he1\", kernel_initializer=he_normal(seed=27)),\r\n        keras.layers.BatchNormalization(axis=1, momentum=batch_n_mom),\r\n        keras.layers.Dropout(dropout_rate),\r\n        \r\n        keras.layers.Dense(n_hidden2, activation=\"selu\", name=\"he2\", kernel_initializer=he_normal(seed=42)),\r\n        keras.layers.BatchNormalization(axis=1, momentum=batch_n_mom),\r\n        \r\n        keras.layers.Dense(n_hidden3, activation=\"selu\", name=\"he3\", kernel_initializer=he_normal(seed=65)),\r\n        keras.layers.BatchNormalization(axis=1, momentum=batch_n_mom),\r\n        \r\n        keras.layers.Dense(n_hidden2, activation=\"selu\", name=\"hd2\", kernel_initializer=he_normal(seed=42)),\r\n        keras.layers.BatchNormalization(axis=1, momentum=batch_n_mom),\r\n        \r\n        keras.layers.Dense(n_hidden1, activation=\"selu\", name=\"hd1\", kernel_initializer=he_normal(seed=27)),\r\n        keras.layers.BatchNormalization(axis=1, momentum=batch_n_mom),\r\n        keras.layers.Dropout(dropout_rate),\r\n        \r\n        keras.layers.Dense(input_shape[0] * input_shape[1], name=\"output\", kernel_initializer=he_normal(seed=62)),\r\n        keras.layers.Reshape(input_shape)\r\n    ])\r\n    \r\n    return stacked_ae\r\n\r\nimport numpy as np\r\n\r\n#Data doesn't matter\r\nx_train = np.ones((32,60,80))\r\ny_train = np.ones((32,60,80))\r\n\r\n#Once runs ok\r\ninput_shape = [60,80]\r\nae_model = make_model(input_shape)\r\nae_model.compile(loss=\"mse\",\r\n                 optimizer=keras.optimizers.Adam(learning_rate=0.001, decay=1e-6),\r\n                metrics=['accuracy'])\r\nprint(ae_model.summary())\r\n\r\n#Do something with the model\r\nhistory = ae_model.fit(x=x_train, y=y_train,  epochs=1, steps_per_epoch=1)\r\n\r\n#Second run, new model\r\nae_model = make_model(input_shape, n_hidden1=2150)\r\nae_model.compile(loss=\"mse\",\r\n                 optimizer=keras.optimizers.Adam(learning_rate=0.001, decay=1e-6),\r\n                metrics=['accuracy'])\r\nprint(ae_model.summary())\r\n\r\n#Run again. GPU OOM.\r\nhistory = ae_model.fit(x=x_train, y=y_train,  epochs=1, steps_per_epoch=1)\r\n\r\n```\r\n\r\n\r\n**Other info / logs**\r\nInclude any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.\r\n\r\n2019-07-16 16:24:06.019147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\r\n2019-07-16 16:24:12.775543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2019-07-16 16:24:12.789530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0\r\n2019-07-16 16:24:12.799822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N\r\n2019-07-16 16:24:12.813163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 517 MB memory) -> physical GPU (device: 0, name: GeForce GPU, pci bus id: 0000:01:00.0, compute capability: 5.0)\r\n2019-07-16 16:24:12.847183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:\r\nname: GeForce GPU major: 5 minor: 0 memoryClockRate(GHz): 0.993\r\npciBusID: 0000:01:00.0\r\n2019-07-16 16:24:12.868383: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.\r\n2019-07-16 16:24:12.887076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\r\n2019-07-16 16:24:12.902106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:\r\nname: GeForce GPU major: 5 minor: 0 memoryClockRate(GHz): 0.993\r\npciBusID: 0000:01:00.0\r\n2019-07-16 16:24:12.925257: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.\r\n2019-07-16 16:24:12.946163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\r\n2019-07-16 16:24:12.958309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2019-07-16 16:24:12.977399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0\r\n2019-07-16 16:24:12.988725: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N\r\n2019-07-16 16:24:13.001442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 517 MB memory) -> physical GPU (device: 0, name: GeForce GPU, pci bus id: 0000:01:00.0, compute capability: 5.0)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/515208087","html_url":"https://github.com/tensorflow/tensorflow/issues/30773#issuecomment-515208087","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30773","id":515208087,"node_id":"MDEyOklzc3VlQ29tbWVudDUxNTIwODA4Nw==","user":{"login":"ymodak","id":42785357,"node_id":"MDQ6VXNlcjQyNzg1MzU3","avatar_url":"https://avatars1.githubusercontent.com/u/42785357?v=4","gravatar_id":"","url":"https://api.github.com/users/ymodak","html_url":"https://github.com/ymodak","followers_url":"https://api.github.com/users/ymodak/followers","following_url":"https://api.github.com/users/ymodak/following{/other_user}","gists_url":"https://api.github.com/users/ymodak/gists{/gist_id}","starred_url":"https://api.github.com/users/ymodak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymodak/subscriptions","organizations_url":"https://api.github.com/users/ymodak/orgs","repos_url":"https://api.github.com/users/ymodak/repos","events_url":"https://api.github.com/users/ymodak/events{/privacy}","received_events_url":"https://api.github.com/users/ymodak/received_events","type":"User","site_admin":false},"created_at":"2019-07-25T20:46:19Z","updated_at":"2019-07-25T20:46:19Z","author_association":"CONTRIBUTOR","body":"Did you try restarting your kernel before compiling your model again? Also\r\nCan you try to execute your model from terminal may be a python script?"}}
{"actor":{"display_login":"ymodak"}}