<p>Unfortunately, Tensorflow does not provide the tooling for post-training per layer quantization in flatbuffer (tflite) yet, but only in protobuf. The only available way now is to introduce <a href="https://www.tensorflow.org/api_docs/python/tf/quantization/fake_quant_with_min_max_vars" rel="nofollow noreferrer">fakeQuantization</a> layers in your graph and re-train / fine-tune your model on the train or a calibration set. This is called "<a href="https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/quantize" rel="nofollow noreferrer">Quantization-aware training</a>".</p>

<p>Once the fakeQuant layers are introduced, then you can feed the training set and TF is going to use them on Feed-Forward as simulated quantisation layers (fp-32 datatypes that represent 8-bit values) and back-propagate using full precision values. This way, you can get back the accuracy loss that caused by quantization.</p>

<p>In addition, the fakeQuant layers are going to capture the ranges per layer or per channel through moving average and store them in min / max variables. </p>

<p>Later, you can extract the graph definition and get rid of the fakeQuant nodes through <code>freeze_graph</code> tool.</p>

<p>Finally, the model can be fed into tf_lite_converter (cross-fingers it won't brake) and extract the u8_tflite with captured ranges.</p>

<p>A very good white-paper, explaining all these is provided by Google here : <a href="https://arxiv.org/pdf/1806.08342.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1806.08342.pdf</a></p>

<p>Hope that helps.</p>
Convert Keras MobileNet model to TFLite with 8-bit quantization
<p>I have used Keras to finetune MobileNet v1. Now I have <code>model.h5</code> and I need to convert it to TensorFlow Lite to use it in Android app. </p>

<p>I use TFLite conversion <a href="https://www.tensorflow.org/lite/convert/cmdline_examples" rel="nofollow noreferrer">script</a> <code>tflite_convert</code>. I can convert it without quantization but I need more performance so I need to make quantization.</p>

<p>If I run this script:</p>

<pre><code>tflite_convert --output_file=model_quant.tflite \
 --keras_model_file=model.h5 \
 --inference_type=QUANTIZED_UINT8 \
 --input_arrays=input_1 \
 --output_arrays=predictions/Softmax \
 --mean_values=128 \
 --std_dev_values=127 \
 --input_shape="1,224,224,3" 
</code></pre>

<p>It fails:</p>

<blockquote>
  <p>F tensorflow/contrib/lite/toco/tooling_util.cc:1634] Array
  conv1_relu/Relu6, which is an input to the DepthwiseConv operator
  producing the output array conv_dw_1_relu/Relu6, is lacking min/max
  data, which is necessary for quantization. If accuracy matters, either
  target a non-quantized output format, or run quantized training with
  your model from a floating point checkpoint to change the input graph
  to contain min/max information. If you don't care about accuracy, you
  can pass --default_ranges_min= and --default_ranges_max= for easy
  experimentation.\nAborted (core dumped)\n"</p>
</blockquote>

<p>If I use <code>default_ranges_min</code> and <code>default_ranges_max</code> (called as "dummy-quantization"), it works but it is only for debugging performance without accuracy as it is described in error log. </p>

<p>So what I need to do to make Keras model correctly quantizable? Do I need to find best <code>default_ranges_min</code> and <code>default_ranges_max</code>? How? Or it is about changes in Keras training phase?</p>

<p>Library versions:</p>

<pre><code>Python 3.6.4
TensorFlow 1.12.0
Keras 2.2.4
</code></pre>