{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/1817","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/1817/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/1817/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/1817/events","html_url":"https://github.com/tensorflow/models/issues/1817","id":239571552,"number":1817,"title":"Out Of Memory when training on Big Images","user":{"login":"pjeambrun","id":26221332,"avatar_url":"https://avatars2.githubusercontent.com/u/26221332?v=4","gravatar_id":"","url":"https://api.github.com/users/pjeambrun","html_url":"https://github.com/pjeambrun","followers_url":"https://api.github.com/users/pjeambrun/followers","following_url":"https://api.github.com/users/pjeambrun/following{/other_user}","gists_url":"https://api.github.com/users/pjeambrun/gists{/gist_id}","starred_url":"https://api.github.com/users/pjeambrun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pjeambrun/subscriptions","organizations_url":"https://api.github.com/users/pjeambrun/orgs","repos_url":"https://api.github.com/users/pjeambrun/repos","events_url":"https://api.github.com/users/pjeambrun/events{/privacy}","received_events_url":"https://api.github.com/users/pjeambrun/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"derekjchow","id":1195088,"avatar_url":"https://avatars2.githubusercontent.com/u/1195088?v=4","gravatar_id":"","url":"https://api.github.com/users/derekjchow","html_url":"https://github.com/derekjchow","followers_url":"https://api.github.com/users/derekjchow/followers","following_url":"https://api.github.com/users/derekjchow/following{/other_user}","gists_url":"https://api.github.com/users/derekjchow/gists{/gist_id}","starred_url":"https://api.github.com/users/derekjchow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekjchow/subscriptions","organizations_url":"https://api.github.com/users/derekjchow/orgs","repos_url":"https://api.github.com/users/derekjchow/repos","events_url":"https://api.github.com/users/derekjchow/events{/privacy}","received_events_url":"https://api.github.com/users/derekjchow/received_events","type":"User","site_admin":false},"assignees":[{"login":"derekjchow","id":1195088,"avatar_url":"https://avatars2.githubusercontent.com/u/1195088?v=4","gravatar_id":"","url":"https://api.github.com/users/derekjchow","html_url":"https://github.com/derekjchow","followers_url":"https://api.github.com/users/derekjchow/followers","following_url":"https://api.github.com/users/derekjchow/following{/other_user}","gists_url":"https://api.github.com/users/derekjchow/gists{/gist_id}","starred_url":"https://api.github.com/users/derekjchow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekjchow/subscriptions","organizations_url":"https://api.github.com/users/derekjchow/orgs","repos_url":"https://api.github.com/users/derekjchow/repos","events_url":"https://api.github.com/users/derekjchow/events{/privacy}","received_events_url":"https://api.github.com/users/derekjchow/received_events","type":"User","site_admin":false}],"milestone":null,"comments":32,"created_at":"2017-06-29T18:29:39Z","updated_at":"2018-02-26T17:53:17Z","closed_at":"2017-09-14T14:34:41Z","author_association":"NONE","body":"# Out Of Memory when training on Big Images\r\n\r\n### Systeme Information\r\n- **OS Platform and Distribution**: Linux Ubuntu 16.04 LTS\r\n- **TensorFlow installed from**: pip tensorflow-gpu\r\n- **TensorFlow version**: ('v1.2.0-rc2-21-g12f033d', '1.2.0')\r\n- **CUDA/cuDNN version**: Cuda 8.0, Cudnn 5.1\r\n- **GPU model and memory**: 8 Tesla K-80, Cores 2, Memory 7.5 G\r\n\r\n\r\n### Describe the Problem\r\nI have successfully run the pets tutorial on this Google Compute Instance\r\nWhen I train a fasterrcnn resnet 101 on my dataset (VOC format, 47 classes, image_size: 1000/2000) with:\r\n```sh\r\npython object_detection/train.py --train_dir=data_xxxx --pipeline_config_path=data_xxxx/faster_rcnn_resnet101_pets.config\r\n```\r\nI get the following error at the beginning of the training:\r\n```\r\n2017-06-29 17:24:13.193833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:7) -> (device: 7, name: Tesla K80, pci bus id: 0000:00:0b.0)\r\n2017-06-29 17:24:15.414228: I tensorflow/core/common_runtime/simple_placer.cc:675] Ignoring device specification /device:GPU:0 for node 'prefetch_queue_Dequeue' because the input edge from 'prefetch_queue' is a reference connection and already has a device field set to /device:CPU:0\r\nINFO:tensorflow:Restoring parameters from /home/ubuntu/models/data_xxxx/model.ckpt\r\nINFO:tensorflow:Starting Session.\r\nINFO:tensorflow:Saving checkpoint to path data_doliprane/model.ckpt\r\nINFO:tensorflow:Starting Queues.\r\nINFO:tensorflow:global_step/sec: 0\r\n[1]    4359 killed     python object_detection/train.py --train_dir=data_xxxx \r\n``` \r\n\r\n__I managed to avoid the oom on this dataset by resizing all the images and annotation files (divided the dimensions by 4).__\r\n\r\n__I didn't modify the config file(just the number of class and the paths), therefore they should be resized at 600 1024, and the bug should not occur with the big images__\r\n\r\nIs there a way I can train on my images without having to shrink them? Are there some parameters I can tune to avoid this problem? \r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/models/issues/comments/368589113","html_url":"https://github.com/tensorflow/models/issues/1817#issuecomment-368589113","issue_url":"https://api.github.com/repos/tensorflow/models/issues/1817","id":368589113,"user":{"login":"zeynali","id":18167711,"avatar_url":"https://avatars0.githubusercontent.com/u/18167711?v=4","gravatar_id":"","url":"https://api.github.com/users/zeynali","html_url":"https://github.com/zeynali","followers_url":"https://api.github.com/users/zeynali/followers","following_url":"https://api.github.com/users/zeynali/following{/other_user}","gists_url":"https://api.github.com/users/zeynali/gists{/gist_id}","starred_url":"https://api.github.com/users/zeynali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeynali/subscriptions","organizations_url":"https://api.github.com/users/zeynali/orgs","repos_url":"https://api.github.com/users/zeynali/repos","events_url":"https://api.github.com/users/zeynali/events{/privacy}","received_events_url":"https://api.github.com/users/zeynali/received_events","type":"User","site_admin":false},"created_at":"2018-02-26T17:53:17Z","updated_at":"2018-02-26T17:53:17Z","author_association":"NONE","body":"@derekjchow ,I am a new in Tensorflow , i saw these queue_capacity and min_after_dequeue at the input_reader.proto in the object_detection/protos (object detection api) path. First, I wanted to know the meaning of these two, and what could the role and effect of the two be? in the CPU/GPU ? why ? What are the high and low the effect of these? please more explain about these.\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"zeynali"}}