title: PyTorch: How to do inference in batches (inference in parallel)
body: <p>How to do inference in batches in PyTorch? How to do inference in parallel to speed up that part of the code.</p>
  <p>I've started with the standard way of doing inference:</p>
  <pre><code>with torch.no_grad():
      for inputs, labels in dataloader['predict']:
          inputs = inputs.to(device)
          output = model(inputs)
          output = output.to(device)
  </code></pre>
  <p>And I've researched and the only mention of doing inference in parallel (in the same machine) seems to be with the library Dask: <a href="https://examples.dask.org/machine-learning/torch-prediction.html" rel="nofollow noreferrer">https://examples.dask.org/machine-learning/torch-prediction.html</a></p>
  <p>Currently attempting to understand that library and create a working example. In the meanwhile do you know of a better way?</p>