title: TensorFlow Lite model on Coral Dev Board not running on TPU
body: <p>I have a TensorFlow Lite model and a Coral Dev Board, and I want to perform inference on the Dev Board's TPU.</p>
  
  <p>When initialising the TensorFlow Lite interpreter in my Python inference script, I add "libedgetpu.so.1" as an experimental delegate, following the example in <a href="https://github.com/google-coral/tflite/blob/master/python/examples/classification/classify_image.py" rel="nofollow noreferrer">the Google Coral TFLite Python example</a> (linked to in the <a href="https://coral.ai/docs/dev-board/get-started/#6-run-a-model-using-the-tensorflow-lite-api" rel="nofollow noreferrer">getting started guide for the Coral Dev Board</a>), however inference is exactly the same speed as when I don't specify the TPU experimental delegate, so I'm assuming that inference is still running on the Dev Board's CPU. Inference time on the Dev Board (with and without the experimental delegate) is 32s; on my desktop PC, inference time for the same test-set is 10s if I run the TFLite model on CPU, and 1.3s if I run the same model in Keras before converting to TFLite (I assume this is faster than TFLite because it utilises multiple cores).</p>
  
  <p>My question: How can I make inference run on the Dev Board's TPU instead of the CPU?</p>
  
  <p>I wonder if this is something I need to specify while building the Keras model on my PC before converting to TFLite format (EG using a <code>with tf.device</code> context manager or something which makes the resulting TFLite model use the TPU), but I can't see anything about this in the <a href="https://www.tensorflow.org/lite/convert/python_api" rel="nofollow noreferrer">TensorFlow Lite Converter Python API documentation</a>.</p>
  
  <p>The Dev Board is running Mendel version 2.0, Python version 3.5.3, tflite-runtime version 2.1.0.post1 (I know I should update the Mendel version, however I'm currently using a Windows PC, and it will be a pain to get access to a Linux machine, or to try and update the Dev Board from Windows using Putty, VirtualBox or WSL. If only Coral supported Windows, like Raspberry Pi does...).</p>
  
  <p>Below is my inference script (I can also upload training script and model if necessary; dataset is MNIST, converted to NumPy float data as described in <a href="https://gist.github.com/jakelevi1996/5c532463d59016f42d2bbdcfedd3372a" rel="nofollow noreferrer">this Gist</a>):</p>
  
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  from time import perf_counter
  try:
      # Try importing the small tflite_runtime module (this runs on the Dev Board)
      print("Trying to import tensorflow lite runtime...")
      from tflite_runtime.interpreter import Interpreter, load_delegate
      experimental_delegates=[load_delegate('libedgetpu.so.1.0')]
  except ModuleNotFoundError:
      # Try importing the full tensorflow module (this runs on PC)
      try:
          print("TFLite runtime not found; trying to import full tensorflow...")
          import tensorflow as tf
          Interpreter = tf.lite.Interpreter
          experimental_delegates = None
      except ModuleNotFoundError:
          # Couldn't import either module
          raise RuntimeError("Could not import Tensorflow or Tensorflow Lite")
  
  # Load data
  mnist_file = np.load("data/mnist.npz")
  x_test = mnist_file["x_test"]
  y_test = mnist_file["y_test"]
  x_test = x_test.astype(np.float32)
  
  # Initialise the interpreter
  tfl_filename = "lstm_mnist_model_b10000.tflite"
  interpreter = Interpreter(model_path=tfl_filename,
      experimental_delegates=experimental_delegates)
  interpreter.allocate_tensors()
  
  print("Starting evaluation...")
  for _ in range(3):
      input_index = (interpreter.get_input_details()[0]['index'])
      output_index = (interpreter.get_output_details()[0]['index'])
      # Perform inference
      t0 = perf_counter()
      interpreter.set_tensor(input_index, x_test)
      interpreter.invoke()
      result = interpreter.get_tensor(output_index)
      t1 = perf_counter()
      # Print accuracy and speed
      num_correct = (result.argmax(axis=1) == y_test).sum()
      print("Time taken (TFLite) = {:.4f} s".format(t1 - t0))
      print('TensorFlow Lite Evaluation accuracy = {} %'.format(
          100 * num_correct / len(x_test)))
      # Reset interpreter state (I don't know why this should be necessary, but
      # accuracy suffers without it)
      interpreter.reset_all_variables()
  </code></pre>