{"action":"closed","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6046","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6046/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6046/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/6046/events","html_url":"https://github.com/tensorflow/tensorflow/issues/6046","id":193218676,"number":6046,"title":"ResourceExhaustedError when save model. Both memory and disk are enough","user":{"login":"machi2074","id":24324035,"avatar_url":"https://avatars.githubusercontent.com/u/24324035?v=3","gravatar_id":"","url":"https://api.github.com/users/machi2074","html_url":"https://github.com/machi2074","followers_url":"https://api.github.com/users/machi2074/followers","following_url":"https://api.github.com/users/machi2074/following{/other_user}","gists_url":"https://api.github.com/users/machi2074/gists{/gist_id}","starred_url":"https://api.github.com/users/machi2074/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/machi2074/subscriptions","organizations_url":"https://api.github.com/users/machi2074/orgs","repos_url":"https://api.github.com/users/machi2074/repos","events_url":"https://api.github.com/users/machi2074/events{/privacy}","received_events_url":"https://api.github.com/users/machi2074/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-12-02T21:11:05Z","updated_at":"2016-12-02T21:51:11Z","closed_at":"2016-12-02T21:51:11Z","body":"Hi,\r\n  I was running tensorflow 0.12.0rc0 on ubuntu 14.04. The training is on CPU only. The crash happened when saver saves the model.\r\n\r\n>W tensorflow/core/framework/op_kernel.cc:975] Resource exhausted:/home/code/2016_11_28_90d_strict_win10.emb-354830.data-00000-of-00001\r\nTraceback (most recent call last):\r\n  File \"/home/code/tf_train.py\", line 474, in <module>\r\n    save_model_and_log_step(tf_saver=saver, tf_session=session, global_step=step)\r\n  File \"/home/code/tf_train.py\", line 199, in save_model_and_log_step\r\n    tf_saver.save(tf_session, FLAGS.out_model_path, global_step=global_step)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py\", line 1323, in save\r\n    {self.saver_def.filename_tensor_name: checkpoint_file})\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 766, in run\r\n    run_metadata_ptr)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 964, in _run\r\n    feed_dict_string, options, run_metadata)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 1014, in _do_run\r\n    target_list, options, run_metadata)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py\", line 1034, in _do_call\r\n    raise type(e)(node_def, op, message)\r\ntensorflow.python.framework.errors_impl.ResourceExhaustedError: ~/2016_11_28_90d_strict_win10.emb-354830.data-00000-of-00001\r\n         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_r$\r\ncv_save/Const_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, embeddings, embeddings/Adagrad, sm_b, sm_b/Adagrad, sm_w_t, sm_w_t/Adagrad)]]\r\n>Caused by op u'save/SaveV2', defined at:\r\n  File \"/home/code/train.py\", line 430, in <module>\r\n    saver = tf.train.Saver(max_to_keep=FLAGS.max_to_keep)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py\", line 1000, in __init__\r\n    self.build()\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py\", line 1030, in build\r\n    restore_sequentially=self._restore_sequentially)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py\", line 622, in build\r\n    save_tensor = self._AddSaveOps(filename_tensor, saveables)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py\", line 229, in _AddSaveOps\r\n    save = self.save_op(filename_tensor, saveables)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py\", line 172, in save_op\r\n    tensors)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py\", line 552, in save_v2\r\n    tensors=tensors, name=name)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py\", line 759, in apply_op\r\n    op_def=op_def)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 2240, in create_op\r\n    original_op=self._default_original_op, op_def=op_def)\r\n  File \"/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py\", line 1128, in __init__\r\n    self._traceback = _extract_stack()\r\n>ResourceExhaustedError (see above for traceback): ~/2016_11_28_90d_strict_win10.emb-354830.data-00000-of-00001\r\n         [[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=\"/job:localhost/replica:0/task:0/cpu:0\"](_re\r\ncv_save/Const_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, embeddings, embeddings/Adagrad, sm_b, sm_b/Adagrad, sm_w_t, sm_w_t/Adagrad)]]\r\n\r\nNo OOM or OOD happened on the machine. I remember this was a bug in 0.9.0 when saving a large tensor it has tensor size too large (>2G) problem because of a proto field is defined as int32. I am not sure if it's related?\r\n\r\n"}}
{"actor":{"display_login":"machi2074"}}