<p>There is mixed precision in training as in the link you mentioned.
Nvidia has a little bit more in depth information what that means here:
<a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html" rel="nofollow noreferrer">https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html</a></p>
<p>Here is the paper describing the actual process, as well as how and why there is a copy of the FP16 weights in FP32 as master weights (therefore a mix of precisions).
<a href="https://arxiv.org/pdf/1710.03740.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1710.03740.pdf</a></p>
<p>But there are also mixed precision operations in hardware and these can speedup your inference when your data is FP32 and your weights/biases are in FP16, with hardware that supports mixed precision operations this can accelerate inference a lot.</p>
<p>For example with a Nvidia T4 i had a a speedup of ~2 on YOLO3, but no speedup on a Nvidia 1080.</p>
Does mixed tf.keras.mixed_precision work for inference?
<p>I am not sure if I understand the idea of tensorflow keras <a href="https://www.tensorflow.org/guide/mixed_precision" rel="nofollow noreferrer">mixed precision</a>. My goal is to run a <code>tf.keras</code> model with floating point 16 precision to improve inference speed. Can this be done with mixed precision?</p>
<p>I am setting this policy before training my model:</p>
<pre class="lang-py prettyprint-override"><code>from tensorflow.keras.mixed_precision import experimental as mixed_precision
policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)
</code></pre>
<p>Or this is just to speed-up training. If this is the case, how could I achieve weights/activations of my tf.keras model to have FP16 precision?</p>
<p><em>Note: I am using <code>tensorflow==2.3.0</code></em></p>