title: Tensorflow Object Detection API for custom dataset - Killed during training
body: <p>Training for object detection on custom dataset (images of shop fronts), for a single class (285 images in total), running locally on CPU, with 8GB RAM gets <strong>killed</strong> after a few steps.</p>
  
  <p>I'm following this <a href="https://medium.com/towards-data-science/how-to-train-your-own-object-detector-with-tensorflows-object-detector-api-bec72ecfe1d9" rel="nofollow noreferrer">blog</a> as a reference.</p>
  
  <p>Here's the console log</p>
  
  <pre><code>(tensorflow) rajaram@rajaram-Lenovo-ideapad-110-15ISK:~/tensorflow/models$ python object_detection/train.py \
  &gt;     --logtostderr \
  &gt;     --pipeline_config_path=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/ssd_mobilenet_v1_sf_train.config \
  &gt;     --train_dir=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train
  INFO:tensorflow:Summary name Learning Rate is illegal; using Learning_Rate instead.
  WARNING:tensorflow:From /home/rajaram/tensorflow/models/object_detection/meta_architectures/ssd_meta_arch.py:607: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
  Instructions for updating:
  Please use tf.global_variables instead.
  INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
  2017-09-26 22:15:08.121785: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
  2017-09-26 22:15:08.122313: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
  2017-09-26 22:15:08.123308: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
  2017-09-26 22:15:08.124144: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
  2017-09-26 22:15:08.124658: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
  2017-09-26 22:15:08.953929: I tensorflow/core/common_runtime/simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'prefetch_queue_Dequeue' because the input edge from 'prefetch_queue' is a reference connection and already has a device field set to /device:CPU:0
  INFO:tensorflow:Restoring parameters from /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/ssd_mobilenet_v1_coco_11_06_2017/model.ckpt
  INFO:tensorflow:Starting Session.
  INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
  INFO:tensorflow:Starting Queues.
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:Recording summary at step 0.
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:Recording summary at step 0.
  INFO:tensorflow:Recording summary at step 0.
  INFO:tensorflow:Recording summary at step 0.
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:Recording summary at step 1.
  INFO:tensorflow:Recording summary at step 1.
  INFO:tensorflow:global_step/sec: 0.00238991
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:Recording summary at step 1.
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:global step 1: loss = 14.4365 (801.196 sec/step)
  INFO:tensorflow:Recording summary at step 1.
  INFO:tensorflow:Recording summary at step 1.
  INFO:tensorflow:global_step/sec: 0
  INFO:tensorflow:Recording summary at step 1.
  INFO:tensorflow:global step 2: loss = 12.9940 (173.981 sec/step)
  INFO:tensorflow:Recording summary at step 2.
  INFO:tensorflow:Recording summary at step 3.
  INFO:tensorflow:global step 3: loss = 12.4866 (166.656 sec/step)
  INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
  INFO:tensorflow:Recording summary at step 3.
  INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
  INFO:tensorflow:global step 4: loss = 11.2386 (162.260 sec/step)
  INFO:tensorflow:Recording summary at step 4.
  INFO:tensorflow:Recording summary at step 4.
  INFO:tensorflow:Recording summary at step 5.
  INFO:tensorflow:global step 5: loss = 10.8210 (416.903 sec/step)
  INFO:tensorflow:Recording summary at step 5.
  Killed
  (tensorflow) rajaram@rajaram-Lenovo-ideapad-110-15ISK:~/tensorflow/models$ 
  </code></pre>
  
  <h2>My thoughts and questions</h2>
  
  <p>1) <strong>Are image sizes an issue</strong>? - Distribution of my images are as follows:
  &lt;= 400x300 (5%), between 400x300 &amp; 640x480 (22%), between 640x480 &amp; 800x600 (63%) and > 800x600 (22%). Although images with dimensions around 400x300 would be good enough for recognition of shopboards, there's a skew towards larger resolutions in my dataset because the next step is text recognition on those boards. </p>
  
  <ul>
  <li>Is this thinking right? </li>
  <li>Should I resize the images to a smaller dimension (if yes - what dimensions are good) and re-do the annotation before starting the whole process again?</li>
  </ul>
  
  <p><em>I could train the Oxford-IIIT Pet data (~7.9k images - which took ~13 hours) for 2000 steps (num_steps = 2000 in train_config section of config file), without crashing or getting killed. So, I figure a mere 285 images should be able to run on the CPU itself.</em></p>
  
  <p>2) <strong>Is swap memory an issue?</strong> - I've also checked the other posts (<a href="https://stackoverflow.com/questions/45150773/tensorflow-object-detection-training-killed-resource-starvation">increase swap space suggestion</a>, <a href="https://stackoverflow.com/questions/44833085/tensorflow-object-detection-killed-before-starting">no follow-up</a> &amp; <a href="https://stackoverflow.com/questions/45247292/ram-error-running-the-tensorflow-object-detection-api">another increase swap memory suggestion</a>) which are on similar lines but since I could train the Oxford-IIIT pet dataset on my current system settings, training as little as 285 images shouldn't kill the process.</p>
  
  <ul>
  <li>Is my thinking correct? </li>
  <li>If not and this is indeed a solution, then I need pointers and clear steps to do this.</li>
  </ul>
  
  <hr>
  
  <p>I would like to know what's going wrong and get it running locally. I hope I have given sufficient information to get help. If not, please let me know what's needed.</p>
  
  <pre><code>                         ---------------------------
  </code></pre>
  
  <h2><strong>System information</strong></h2>
  
  <ol>
  <li><strong>What is the top-level directory of the model you are using:</strong> tensorflow/models (haven't updated to the new folder structure yet)</li>
  <li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow):</strong> Yes - minimal changes (following <a href="https://github.com/datitran/raccoon_dataset" rel="nofollow noreferrer">Dat Trans's
  template for my own data set - Github</a>)</li>
  <li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):</strong> Ubuntu 16.04.3 LTS</li>
  <li><strong>TensorFlow installed from (source or binary):</strong> binary (into a virtual environment)</li>
  <li><strong>TensorFlow version (use command below):</strong> 1.3.0</li>
  <li><strong>Bazel version (if compiling from source):</strong> NA</li>
  <li><strong>CUDA/cuDNN version:</strong> NA</li>
  <li><strong>GPU model and memory:</strong> NA</li>
  <li><strong>Exact command to reproduce:</strong> python object_detection/train.py --logtostderr --pipeline_config_path=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/ssd_mobilenet_v1_sf_train.config
  --train_dir=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train</li>
  </ol>