{"action":"opened","issue":{"url":"https://api.github.com/repos/dennybritz/chatbot-retrieval/issues/11","repository_url":"https://api.github.com/repos/dennybritz/chatbot-retrieval","labels_url":"https://api.github.com/repos/dennybritz/chatbot-retrieval/issues/11/labels{/name}","comments_url":"https://api.github.com/repos/dennybritz/chatbot-retrieval/issues/11/comments","events_url":"https://api.github.com/repos/dennybritz/chatbot-retrieval/issues/11/events","html_url":"https://github.com/dennybritz/chatbot-retrieval/issues/11","id":171634103,"number":11,"title":"OOM on Ubuntu 14.04 with GTX670","user":{"login":"JoeyTeng","id":12524680,"avatar_url":"https://avatars.githubusercontent.com/u/12524680?v=3","gravatar_id":"","url":"https://api.github.com/users/JoeyTeng","html_url":"https://github.com/JoeyTeng","followers_url":"https://api.github.com/users/JoeyTeng/followers","following_url":"https://api.github.com/users/JoeyTeng/following{/other_user}","gists_url":"https://api.github.com/users/JoeyTeng/gists{/gist_id}","starred_url":"https://api.github.com/users/JoeyTeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoeyTeng/subscriptions","organizations_url":"https://api.github.com/users/JoeyTeng/orgs","repos_url":"https://api.github.com/users/JoeyTeng/repos","events_url":"https://api.github.com/users/JoeyTeng/events{/privacy}","received_events_url":"https://api.github.com/users/JoeyTeng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-17T11:23:27Z","updated_at":"2016-08-17T11:23:27Z","closed_at":null,"body":"Hi,\r\n\r\nI tried to run the demo with GPU, and it raised an error at Step 68001 suddenly. \r\n\r\nI tried to run this on my Macbook Pro with 'CPU only' version, and there was no error over 70000 steps. I believe this error is due to the small memory of the GPU. Is there any way to fix this problem?\r\n\r\nLast part of the Log (excluding the memory part):\r\n```\r\nInUse:                  1890070528\r\nMaxInUse:               1890070528\r\nNumAllocs:              1307072597\r\nMaxAllocSize:             73287680\r\n\r\nW tensorflow/core/common_runtime/bfc_allocator.cc:270] ****************************************************************************************************\r\nW tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 1.00MiB.  See logs for memory state.\r\nW tensorflow/core/framework/op_kernel.cc:909] Resource exhausted: OOM when allocating tensor with shape[256,1024]\r\nINFO:tensorflow:Saving checkpoint for step 68001 to checkpoint: /home/harmanjohll/SaveTheDepressed/chatbot-retrieval/runs/1471341127/model.ckpt.\r\nTraceback (most recent call last):\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/client/session.py\", line 715, in _do_call\r\n    return fn(*args)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/client/session.py\", line 697, in _run_fn\r\n    status, run_metadata)\r\n  File \"/usr/lib/python3.4/contextlib.py\", line 66, in __exit__\r\n    next(self.gen)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/framework/errors.py\", line 450, in raise_exception_on_not_ok_status\r\n    pywrap_tensorflow.TF_GetCode(status))\r\ntensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[256,1024]\r\n\t [[Node: rnn/RNN/while/LSTMCell/BiasAdd = BiasAdd[T=DT_FLOAT, data_format=\"NHWC\", _device=\"/job:localhost/replica:0/task:0/gpu:0\"](rnn/RNN/while/LSTMCell/MatMul, rnn/RNN/while/LSTMCell/BiasAdd/Enter)]]\r\n\t [[Node: OptimizeLoss/train/update/_201717 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/cpu:0\", send_device=\"/job:localhost/replica:0/task:0/gpu:0\", send_device_incarnation=1, tensor_name=\"edge_1043_OptimizeLoss/train/update\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"udc_train.py\", line 70, in <module>\r\n    tf.app.run()\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"udc_train.py\", line 67, in main\r\n    estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py\", line 182, in fit\r\n    monitors=monitors)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py\", line 484, in _train_model\r\n    monitors=monitors)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py\", line 328, in train\r\n    reraise(*excinfo)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/six.py\", line 686, in reraise\r\n    raise value\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py\", line 257, in train\r\n    session, last_step + 1, [train_op, loss_op], feed_dict, monitors)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py\", line 111, in _run_with_monitors\r\n    outputs = session.run(tensors, feed_dict=feed_dict)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/client/session.py\", line 372, in run\r\n    run_metadata_ptr)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/client/session.py\", line 636, in _run\r\n    feed_dict_string, options, run_metadata)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/client/session.py\", line 708, in _do_run\r\n    target_list, options, run_metadata)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/client/session.py\", line 728, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[256,1024]\r\n\t [[Node: rnn/RNN/while/LSTMCell/BiasAdd = BiasAdd[T=DT_FLOAT, data_format=\"NHWC\", _device=\"/job:localhost/replica:0/task:0/gpu:0\"](rnn/RNN/while/LSTMCell/MatMul, rnn/RNN/while/LSTMCell/BiasAdd/Enter)]]\r\n\t [[Node: OptimizeLoss/train/update/_201717 = _Recv[client_terminated=false, recv_device=\"/job:localhost/replica:0/task:0/cpu:0\", send_device=\"/job:localhost/replica:0/task:0/gpu:0\", send_device_incarnation=1, tensor_name=\"edge_1043_OptimizeLoss/train/update\", tensor_type=DT_FLOAT, _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\r\nCaused by op 'rnn/RNN/while/LSTMCell/BiasAdd', defined at:\r\n  File \"udc_train.py\", line 70, in <module>\r\n    tf.app.run()\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/platform/app.py\", line 30, in run\r\n    sys.exit(main(sys.argv))\r\n  File \"udc_train.py\", line 67, in main\r\n    estimator.fit(input_fn=input_fn_train, steps=None, monitors=[eval_monitor])\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py\", line 182, in fit\r\n    monitors=monitors)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py\", line 449, in _train_model\r\n    train_op, loss_op = self._get_train_ops(features, targets)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py\", line 673, in _get_train_ops\r\n    _, loss, train_op = self._call_model_fn(features, targets, ModeKeys.TRAIN)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py\", line 656, in _call_model_fn\r\n    features, targets, mode=mode)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/udc_model.py\", line 40, in model_fn\r\n    targets)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/models/dual_encoder.py\", line 56, in dual_encoder_model\r\n    dtype=tf.float32)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py\", line 580, in dynamic_rnn\r\n    swap_memory=swap_memory, sequence_length=sequence_length)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py\", line 709, in _dynamic_rnn_loop\r\n    swap_memory=swap_memory)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/control_flow_ops.py\", line 1873, in while_loop\r\n    result = context.BuildLoop(cond, body, loop_vars)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/control_flow_ops.py\", line 1749, in BuildLoop\r\n    body_result = body(*vars_for_body_with_tensor_arrays)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py\", line 692, in _time_step\r\n    skip_conditionals=True)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py\", line 326, in _rnn_step\r\n    new_output, new_state = call_cell()\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/rnn.py\", line 680, in <lambda>\r\n    call_cell = lambda: cell(input_t, state)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/rnn_cell.py\", line 552, in __call__\r\n    lstm_matrix = nn_ops.bias_add(math_ops.matmul(cell_inputs, concat_w), b)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/nn_ops.py\", line 313, in bias_add\r\n    return gen_nn_ops._bias_add(value, bias, data_format=data_format, name=name)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/gen_nn_ops.py\", line 279, in _bias_add\r\n    data_format=data_format, name=name)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/ops/op_def_library.py\", line 704, in apply_op\r\n    op_def=op_def)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/framework/ops.py\", line 2260, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/home/harmanjohll/SaveTheDepressed/chatbot-retrieval/venv/lib/python3.4/site-packages/tensorflow/python/framework/ops.py\", line 1230, in __init__\r\n    self._traceback = _extract_stack()\r\n\r\n```\r\n\r\nBy the way, will udc_train.py automatically resume training from the checkpoint? And do I need to process all the training data?\r\n"}}
{"actor":{"display_login":"JoeyTeng"}}