<p>You might want to check how quantization works in TensorFlow - maybe you missed handling zero-point and scale?</p>

<ul>
<li><a href="https://arxiv.org/pdf/1712.05877.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1712.05877.pdf</a></li>
<li><a href="https://www.tensorflow.org/lite/performance/quantization_spec" rel="nofollow noreferrer">https://www.tensorflow.org/lite/performance/quantization_spec</a></li>
</ul>
Getting erroneous results while running inference with quantized TFlite weights
<p>I want to export a quantized model onto FPGA</p>

<p>I adopted the Quantization Aware Training flow as per <a href="https://www.tensorflow.org/model_optimization/guide/quantization/training_example" rel="nofollow noreferrer">https://www.tensorflow.org/model_optimization/guide/quantization/training_example</a> to get a tflite model with uint8 quantization.</p>

<p>Dataset :MNIST</p>

<p>Model used:</p>

<pre><code>  model = keras.Sequential([
  keras.layers.InputLayer(input_shape=(28, 28)),   
  keras.layers.Flatten(),                    
  keras.layers.Dense(32, activation=tf.nn.relu),
  keras.layers.Dense(16, activation=tf.nn.relu),
  keras.layers.Dense(10, activation=tf.nn.softmax)
])

</code></pre>

<p>Accuracy Results of tflite model obtained</p>

<pre><code>Quant TFLite test_accuracy: 0.9478
Quant TF test accuracy: 0.9527000188827515
</code></pre>

<p>Now, I want to use this quantized model on FPGA. So, I export the tflite model and export the weights as numpy array using netron (<a href="https://github.com/lutzroeder/netron" rel="nofollow noreferrer">https://github.com/lutzroeder/netron</a>) .</p>

<p>Using these weights, I run an inference on python by handcoding the forward path as shown . I get accuracy = 32 % as against the accuarcy of 95% obtained using tflite interpreter .</p>

<p>Here is the handcoded forward path </p>

<pre><code>w_32x784=np.load("w_32x784.npy")
b_32 = np.load("b_32.npy")
w_16x32 = np.load("w_16x32.npy")
b_16 = np.load("b_16.npy")
w_10x16 = np.load("w_10x16.npy")
b_10 =np.load("b_10.npy")

def eval_q(x_inp):
    layer1_op = relu(np.squeeze(np.matmul(w_32x784,x_inp))+b_32)
    layer2_op = relu(np.squeeze(np.matmul(w_16x32,layer1_op))+b_16)
    layer3_op = np.squeeze(np.matmul(w_10x16,layer2_op))+b_10
    predict = np.argmax(layer3_op)
    return predict

def relu(w):
    op =[x if x &gt; 0 else 0 for x in w]
    return op 

# Evaluate Model
predictions=[]
for img in test_images:
    x_inp = img.reshape(784,1)
    predictions.append(eval_q(x_inp))
print((predictions==test_labels).mean())
</code></pre>

<pre><code>Accuracy =0.3248
</code></pre>

<p>Please help me out with finding where I'm going wrong.</p>