title: ONNX Runtime Inference | session.run() multiprocessing
body: <p><strong>Goal: run Inference in parallel on multiple CPU cores</strong></p>
  <p>I'm experimenting with Inference using <a href="https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/simple_onnxruntime_inference.ipynb" rel="nofollow noreferrer">simple_onnxruntime_inference.ipynb</a>.</p>
  <p>Individually:</p>
  <pre class="lang-py prettyprint-override"><code>outputs = session.run([output_name], {input_name: x})
  </code></pre>
  <p>Many:</p>
  <pre class="lang-py prettyprint-override"><code>outputs = session.run([&quot;output1&quot;, &quot;output2&quot;], {&quot;input1&quot;: indata1, &quot;input2&quot;: indata2})
  </code></pre>
  <p>Sequentially:</p>
  <pre class="lang-py prettyprint-override"><code>%%time
  outputs = [session.run([output_name], {input_name: inputs[i]})[0] for i in range(test_data_num)]
  </code></pre>
  <hr />
  <p>This Multiprocessing <a href="https://www.machinelearningplus.com/python/parallel-processing-python/" rel="nofollow noreferrer">tutorial</a> offers many approaches for parallelising any tasks.</p>
  <p>However, I want to know which approach would be best for <code>session.run()</code>, with or without <code>outputs</code> being passed.</p>
  <p><strong>How do I Inference all outputs and inputs together, in parallel?</strong></p>
  <p><strong>Code:</strong></p>
  <pre class="lang-py prettyprint-override"><code>import onnxruntime
  import multiprocessing as mp
  
  session = onnxruntime.InferenceSession('bert.opt.quant.onnx')
  
  i = 0
  # First Input
  input_name = session.get_inputs()[i].name
  print(&quot;Input Name  :&quot;, input_name)
  
  # First Output
  output_name = session.get_outputs()[i].name
  print(&quot;Output Name  :&quot;, output_name)  
  
  pool = mp.Pool(mp.cpu_count())
  
  # PARALLELISE THIS LINE
  outputs = [session.run([], {input_name: inputs[i]})[0] for i in range(test_data_num)]
  # outputs = pool.starmap(func, zip(iter_1, iter_2))
  
  pool.close()
  
  print(results)
  </code></pre>
  <hr />
  <p><strong>Update</strong>: this <a href="https://stackoverflow.com/a/49026821/17840900">solution</a> suggests using <code>starmap()</code> and <code>zip()</code> in order to pass a function name and 2 separate iterables.</p>
  <p>Replacing line with this:</p>
  <pre class="lang-py prettyprint-override"><code>outputs = pool.starmap(session.run, zip([output_name], [ {input_name: inputs[i]}[0] for i in range(test_data_num) ]))
  </code></pre>
  <p>Traceback:</p>
  <pre><code>---------------------------------------------------------------------------
  KeyError                                  Traceback (most recent call last)
  &lt;ipython-input-45-0aab302a55eb&gt; in &lt;module&gt;
       25 #%%time
       26 #outputs = [session.run([output_name], {input_name: inputs[i]})[0] for i in range(test_data_num)]
  ---&gt; 27 outputs = pool.starmap(session.run, zip([output_name], [ {input_name: inputs[i]}[0] for i in range(test_data_num) ]))
       28 
       29 pool.close()
  
  &lt;ipython-input-45-0aab302a55eb&gt; in &lt;listcomp&gt;(.0)
       25 #%%time
       26 #outputs = [session.run([output_name], {input_name: inputs[i]})[0] for i in range(test_data_num)]
  ---&gt; 27 outputs = pool.starmap(session.run, zip([output_name], [ {input_name: inputs[i]}[0] for i in range(test_data_num) ]))
       28 
       29 pool.close()
  
  KeyError: 0
  </code></pre>