<p>According to this study: </p>

<blockquote>
  <p>Gupta, S., Agrawal, A., Gopalakrishnan, K., &amp; Narayanan, P. (2015,
  June). Deep learning with limited numerical precision. In
  International Conference on Machine Learning (pp. 1737-1746). At:
  <a href="https://arxiv.org/pdf/1502.02551.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1502.02551.pdf</a></p>
</blockquote>

<p>stochastic rounding was required to obtain convergence when using half-point floating precision (float16); however, when that rounding technique was used, they claimed to get very good results.</p>

<p>Here's a relevant quotation from that paper:</p>

<p>"A recent work (Chen et al., 2014) presents a hardware accelerator
for deep neural network training that employs
fixed-point computation units, but finds it necessary to
use 32-bit fixed-point representation to achieve convergence
while training a convolutional neural network on
the MNIST dataset. In contrast, our results show that
it is possible to train these networks using only 16-bit
fixed-point numbers, so long as stochastic rounding is used
during fixed-point computations."</p>

<p>For reference, here's the citation for Chen at al., 2014:</p>

<blockquote>
  <p>Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., ... &amp; Temam,
  O. (2014, December). Dadiannao: A machine-learning supercomputer. In
  Proceedings of the 47th Annual IEEE/ACM International Symposium on
  Microarchitecture (pp. 609-622). IEEE Computer Society. At:
  <a href="http://ieeexplore.ieee.org/document/7011421/?part=1" rel="nofollow noreferrer">http://ieeexplore.ieee.org/document/7011421/?part=1</a></p>
</blockquote>
float16 vs float32 for convolutional neural networks
<p>The standard is float32 but I'm wondering under what conditions it's ok to use float16?</p>

<p>I've compared running the same covnet with both datatypes and haven't noticed any issues. With large dataset I prefer float16 because I can worry less about memory issues..</p>