{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3373","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3373/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3373/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3373/events","html_url":"https://github.com/tensorflow/tensorflow/issues/3373","id":166191997,"number":3373,"title":"Cannot allocate 6GB variable on 12GB GPU","user":{"login":"lightcatcher","id":287200,"avatar_url":"https://avatars.githubusercontent.com/u/287200?v=3","gravatar_id":"","url":"https://api.github.com/users/lightcatcher","html_url":"https://github.com/lightcatcher","followers_url":"https://api.github.com/users/lightcatcher/followers","following_url":"https://api.github.com/users/lightcatcher/following{/other_user}","gists_url":"https://api.github.com/users/lightcatcher/gists{/gist_id}","starred_url":"https://api.github.com/users/lightcatcher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lightcatcher/subscriptions","organizations_url":"https://api.github.com/users/lightcatcher/orgs","repos_url":"https://api.github.com/users/lightcatcher/repos","events_url":"https://api.github.com/users/lightcatcher/events{/privacy}","received_events_url":"https://api.github.com/users/lightcatcher/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/awaiting%20googler","name":"awaiting googler","color":"b60205"}],"state":"open","locked":false,"assignee":{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars.githubusercontent.com/u/15736910?v=3","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"assignees":[{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars.githubusercontent.com/u/15736910?v=3","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-07-18T20:58:28Z","updated_at":"2016-07-19T23:44:51Z","closed_at":null,"body":"I'm running Tensorflow 0.9.0 on a K40m (with 12GB of memory) with CUDA 7.5.0.\r\n\r\nI'm attempting to preload data as described here: https://www.tensorflow.org/versions/r0.9/how_tos/reading_data/index.html#preloaded-data\r\n\r\nTreating the data as a constant doesn't work because constants are limited to 2GB, and a minibatch approach does not make sense for my application.\r\n\r\nPerhaps Tensorflow wants to allocate space for the placeholder on GPU, allocate space for the variable, copy from host to placeholder (on GPU), and then copy from placeholder to variable?\r\n\r\nHere's a minimal failing test case. \r\n\r\nThe following program:\r\n```python\r\nimport numpy as np\r\nimport tensorflow as tf\r\n\r\n# allocate 6GB of zeros                                                                                                                                                                                                                                                                   \r\n_data = np.zeros(1536 * (1 << 20), dtype=np.float32)\r\n\r\ndata_init = tf.placeholder(tf.float32, shape=_data.shape)\r\ndata = tf.Variable(data_init, trainable=False, collections=[])\r\n\r\nwith tf.Session() as sess:\r\n    sess.run(data.initializer, feed_dict={data_init: _data})\r\n```\r\n\r\nproduces the following output:\r\n```\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: \r\nname: Tesla K40m\r\nmajor: 3 minor: 5 memoryClockRate (GHz) 0.745\r\npciBusID 0000:04:00.0\r\nTotal memory: 11.25GiB\r\nFree memory: 11.12GiB\r\nW tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x16034f0\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties: \r\nname: Tesla K40m\r\nmajor: 3 minor: 5 memoryClockRate (GHz) 0.745\r\npciBusID 0000:42:00.0\r\nTotal memory: 11.25GiB\r\nFree memory: 11.12GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:59] cannot enable peer access from device ordinal 0 to device ordinal 1\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:59] cannot enable peer access from device ordinal 1 to device ordinal 0\r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 1 \r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y N \r\nI tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 1:   N Y \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:04:00.0)\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40m, pci bus id: 0000:42:00.0)\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (256):       Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (512):       Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1024):      Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2048):      Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4096):      Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8192):      Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16384):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (32768):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (65536):     Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (131072):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (262144):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (524288):    Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1048576):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2097152):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4194304):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8388608):   Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16777216):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (33554432):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (67108864):  Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (134217728):         Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (268435456):         Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:656] Bin for 6.00GiB was 256.00MiB, Chunk State: \r\nI tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x4208f40000 of size 11344840448\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:689]      Summary of in-use Chunks by size: \r\nI tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 11344840448 totalling 10.57GiB\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:696] Sum Total of in-use chunks: 10.57GiB\r\nI tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats: \r\nLimit:                 11344840295\r\nInUse:                 11344840448\r\nMaxInUse:              11344840448\r\nNumAllocs:                       1\r\nMaxAllocSize:          11344840448\r\n\r\nW tensorflow/core/common_runtime/bfc_allocator.cc:270] *********************************************************xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\r\nW tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 6.00GiB.  See logs for memory state.\r\nW tensorflow/core/framework/op_kernel.cc:909] Resource exhausted: OOM when allocating tensor with shape[1610612736]\r\nTraceback (most recent call last):\r\n  File \"tf_fail.py\", line 11, in <module>\r\n    sess.run(data.initializer, feed_dict={data_init: _data})\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/client/session.py\", line 372, in run\r\n    run_metadata_ptr)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/client/session.py\", line 636, in _run\r\n    feed_dict_string, options, run_metadata)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/client/session.py\", line 708, in _do_run\r\n    target_list, options, run_metadata)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/client/session.py\", line 728, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[1610612736]\r\n         [[Node: Variable/Assign = Assign[T=DT_FLOAT, _class=[\"loc:@Variable\"], use_locking=true, validate_shape=true, _device=\"/job:localhost/replica:0/task:0/gpu:0\"](Variable, _recv_Placeholder_0/_2)]]\r\nCaused by op u'Variable/Assign', defined at:\r\n  File \"tf_fail.py\", line 8, in <module>\r\n    data = tf.Variable(data_init, trainable=False, collections=[])\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/ops/variables.py\", line 211, in __init__\r\n    dtype=dtype)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/ops/variables.py\", line 309, in _init_from_args\r\n    validate_shape=validate_shape).op\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/ops/gen_state_ops.py\", line 45, in assign\r\n    use_locking=use_locking, name=name)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/ops/op_def_library.py\", line 704, in apply_op\r\n    op_def=op_def)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/framework/ops.py\", line 2260, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/software/rhel7/tensorflow-0.9.0/lib/tensorflow/python/framework/ops.py\", line 1230, in __init__\r\n    self._traceback = _extract_stack()\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/233799526","html_url":"https://github.com/tensorflow/tensorflow/issues/3373#issuecomment-233799526","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/3373","id":233799526,"user":{"login":"zheng-xq","id":15736910,"avatar_url":"https://avatars.githubusercontent.com/u/15736910?v=3","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"created_at":"2016-07-19T23:44:51Z","updated_at":"2016-07-19T23:44:51Z","body":"@concretevitamin, the initializer is fundamentally a tf.Assign. So there will be at least two copies of the tensors in that operation, lhs is the variable, and rhs is the initial value. Since TensorFlow didn't get all the GPU memory, it only allocates about 11GB memory. So half of that is about 5.5GB. If you change the size of your variable to (1400 << (1 <<20)), it should work.\r\n\r\nShort of an in-place initialization kernel, there is no current way to save the second copy of tensor for initial values. In most cases, this is not a problem, since the memory is only temporary, and will be released immediately after the assignment. \r\n"}}
{"actor":{"display_login":"zheng-xq"}}