title: Time Consuming Tensorflow C++ Session->Run - Images for Real-time Inference
body: <p>[Tensorflow (TF) on CPU]
  I am using the skeleton code provided for C++ TF inference from GitHub [label_image/main.cc] in order to run a frozen model I have created in Python. This model is an FC NN with two hidden layers.</p>
  
  <p>In my current project's C++ code, I run the NN's frozen classifier for each single image (8x8 pixels). For each sample, a Session->Run call takes about 0.02 seconds, which is expensive in my application, since I can have 64000 samples that I have to run.  </p>
  
  <p>When I send a batch of 1560 samples, the Session->Run call takes about 0.03 seconds.</p>
  
  <p>Are these time measurements normal for the Session->Run Call? From the C++ end, should I send my frozen model batches of images and not single samples? From the Python end, are there optimisation tricks to alleviate that bottleneck? Is there a way to concurrently do Session-Run calls in C++?</p>
  
  <p>Environment info</p>
  
  <p>Operating System: Linux
  Installed version of CUDA and cuDNN: N/A</p>
  
  <p>What other attempted solutions have you tried?</p>
  
  <ol>
  <li>I installed TF using the optimised instruction set for the CPU, but it does not seem to give me the huge time saving mentioned in StackOverflow</li>
  <li>Unified the session for the Graph I created.</li>
  </ol>
  
  <p><strong>EDIT</strong><br>
  <em>It seems that MatMul is the bottleneck -- Any suggestions how to improve that?</em>  </p>
  
  <p><em>Should I use 'optimize_for_inference.py' script for my frozen graph?</em>  </p>
  
  <p><em>How can you measure the time in Python with high precision?</em></p>
  
  <p><a href="https://i.stack.imgur.com/3E9jU.png" rel="nofollow noreferrer">Timeline for feeding an 8x8 sample and getting the result in Python</a> </p>
  
  <p><a href="https://i.stack.imgur.com/jEAKx.png" rel="nofollow noreferrer">Timeline for feeding an 8x8 batch and getting the result in Python</a></p>