<p>float16 training is tricky: your model might not converge when using standard float16, but float16 does save memory, and is also faster if you are using the latest Volta GPUs. Nvidia recommends "Mixed Precision Training" in the latest <a href="http://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#mptrain" rel="nofollow noreferrer">doc</a> and <a href="https://arxiv.org/abs/1710.03740" rel="nofollow noreferrer">paper</a>.</p>

<p>To better use float16, you need to manually and carefully choose the <strong>loss_scale</strong>. If loss_scale is too large, you may get NANs and INFs; if loss_scale is too small, the model might not converge. Unfortunately, there is no common loss_scale for all models, so you have to choose it carefully for your specific model.</p>

<p>If you just want to reduce the memory usage, you could also try tf. to_bfloat16, which might converge better. </p>
float16 vs float32 for convolutional neural networks
<p>The standard is float32 but I'm wondering under what conditions it's ok to use float16?</p>

<p>I've compared running the same covnet with both datatypes and haven't noticed any issues. With large dataset I prefer float16 because I can worry less about memory issues..</p>