{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6111","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6111/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6111/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6111/events","html_url":"https://github.com/tensorflow/tensorflow/issues/6111","id":193677245,"node_id":"MDU6SXNzdWUxOTM2NzcyNDU=","number":6111,"title":"Flawed memory management: allow_growth=True consumes more memory, causing out-of-memory","user":{"login":"xuancong84","id":10172392,"node_id":"MDQ6VXNlcjEwMTcyMzky","avatar_url":"https://avatars0.githubusercontent.com/u/10172392?v=4","gravatar_id":"","url":"https://api.github.com/users/xuancong84","html_url":"https://github.com/xuancong84","followers_url":"https://api.github.com/users/xuancong84/followers","following_url":"https://api.github.com/users/xuancong84/following{/other_user}","gists_url":"https://api.github.com/users/xuancong84/gists{/gist_id}","starred_url":"https://api.github.com/users/xuancong84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xuancong84/subscriptions","organizations_url":"https://api.github.com/users/xuancong84/orgs","repos_url":"https://api.github.com/users/xuancong84/repos","events_url":"https://api.github.com/users/xuancong84/events{/privacy}","received_events_url":"https://api.github.com/users/xuancong84/received_events","type":"User","site_admin":false},"labels":[{"id":473172988,"node_id":"MDU6TGFiZWw0NzMxNzI5ODg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":{"login":"zheng-xq","id":15736910,"node_id":"MDQ6VXNlcjE1NzM2OTEw","avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"assignees":[{"login":"zheng-xq","id":15736910,"node_id":"MDQ6VXNlcjE1NzM2OTEw","avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2016-12-06T03:56:26Z","updated_at":"2018-10-01T15:40:22Z","closed_at":"2017-06-16T23:41:48Z","author_association":"NONE","body":"To prevent tensorflow (TF) from allocating the totality of graphic memory, I always use the following options when creating sessions:\r\n\r\n```\r\nconfig = tf.ConfigProto()\r\nconfig.gpu_options.allow_growth = True\r\nsess = tf.Session(config=config)\r\n```\r\n\r\nHowever, doing so causes some experiments to run out of memory while not doing so will not cause memory overflow. For example, when running experiments involving RNN, such as translate.py or ptb_word_lm.py in the sample code, if I specify allow_growth=True, I always encounter the following:\r\n\r\n```\r\nTraining Epoch 0 ; learning_rate= 0.002 :                                                                                                \r\nI tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 2631 get requests, put_count=2230 evicted_count=1000 eviction_rate=0.44843 and unsatisfied allocation rate=0.570506                                                                              \r\nI tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 100 to 110                                     \r\nI tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 2631 get requests, put_count=2546 evicted_count=1000 eviction_rate=0.392773 and unsatisfied allocation rate=0.421133                                                                             \r\nI tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 256 to 281                                     \r\nI tensorflow/core/common_runtime/gpu/pool_allocator.cc:245] PoolAllocator: After 9384 get requests, put_count=9389 evicted_count=1000 eviction_rate=0.106508 and unsatisfied allocation rate=0.112319                                                                             \r\nI tensorflow/core/common_runtime/gpu/pool_allocator.cc:257] Raising pool_size_limit_ from 655 to 720                                     \r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 2.00G (2147483648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY  \r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 1.80G (1932735232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY  \r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 1.62G (1739461632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY  \r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 1.46G (1565515520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY  \r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 1.31G (1408964096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY  \r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate 1.18G (1268067840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY  \r\n```\r\nHowever, without specifying allow_growth=True, I can run it successfully. Moreover, the OOM occurs only after going through some epoches in the training data, not right from the beginning.\r\n\r\nIn principle, for an ideal memory manager, whether OOM will occur should not depends on whether memory is pre-allocated in one go or allocated step-by-step dynamically. Thus, Tensorflow's low-level memory management code must be flawed in one way or another.\r\n\r\n\r\nBelow are my system info:\r\n```\r\nOperating System:\r\nUbuntu 14.04.5 LTS\r\n\r\nInstalled version of CUDA and cuDNN:\r\n/usr/local/cuda-8.0\r\ncudnn-8.0-linux-x64-v5.1.tgz\r\n\r\nIt is installed from binary pip package\r\n\r\nA link to the pip package you installed:\r\nhttps://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.11.0-cp27-none-linux_x86_64.whl\r\n\r\nThe output from python -c \"import tensorflow; print(tensorflow.version)\"\r\nxuancong@wxc-i2r:~/projects/tf-rnnlm$ python -c \"import tensorflow; print(tensorflow.version)\"\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally\r\n0.11.0\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/425955092","html_url":"https://github.com/tensorflow/tensorflow/issues/6111#issuecomment-425955092","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6111","id":425955092,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNTk1NTA5Mg==","user":{"login":"Netroman","id":20833124,"node_id":"MDQ6VXNlcjIwODMzMTI0","avatar_url":"https://avatars2.githubusercontent.com/u/20833124?v=4","gravatar_id":"","url":"https://api.github.com/users/Netroman","html_url":"https://github.com/Netroman","followers_url":"https://api.github.com/users/Netroman/followers","following_url":"https://api.github.com/users/Netroman/following{/other_user}","gists_url":"https://api.github.com/users/Netroman/gists{/gist_id}","starred_url":"https://api.github.com/users/Netroman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Netroman/subscriptions","organizations_url":"https://api.github.com/users/Netroman/orgs","repos_url":"https://api.github.com/users/Netroman/repos","events_url":"https://api.github.com/users/Netroman/events{/privacy}","received_events_url":"https://api.github.com/users/Netroman/received_events","type":"User","site_admin":false},"created_at":"2018-10-01T15:40:22Z","updated_at":"2018-10-01T15:40:22Z","author_association":"NONE","body":"Does anyone have a solution for this? \r\n\r\nI'm trying to run a Yolo example using tensorflow-gpu:\r\nhttps://github.com/CornerStone12/yolov2-tensorflow\r\n\r\nThis works fine on my GeForce 1050Ti, as well as on the CPU, but trying to run on a Tesla V100 it allocates all the GPU memory and  seem to spawn a lot of processes just grabbing more and more memory until it is killed. \r\n\r\nThe values of `gpu_options.allow_growth` and `gpu_options.per_process_gpu_memory_fraction` doesn't seem to make a difference.\r\n\r\n```\r\nPID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command \r\n 6559 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6560 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6561 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6562 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6563 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6564 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6565 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n 6566 root       20   0 49.5G 2761M  606M S  0.0  8.6  0:00.00 python demo_yolo_v2.py\r\n...\r\n```"}}
{"actor":{"display_login":"Netroman"}}