<p>Surprisingly, it's totally OK to use 16 bits, even not just for fun, but in production as well. For example, in <a href="https://www.youtube.com/watch?v=sUzQpd-Ku4o" rel="noreferrer">this video</a> Jeff Dean talks about <strong>16-bit</strong> calculations at Google, around 52:00. A quote from the slides:</p>

<blockquote>
  <p>Neural net training very tolerant of reduced precision</p>
</blockquote>

<p>Since GPU memory is the main bottleneck in ML computation, there has been a lot of research on precision reduction. E.g. </p>

<ul>
<li><p><a href="https://arxiv.org/abs/1502.02551" rel="noreferrer">Gupta at al paper</a> "Deep Learning with Limited Numerical Precision" about <strong>fixed</strong> (not floating) <strong>16-bit</strong> training but with <em>stochastic rounding</em>. </p></li>
<li><p><a href="https://arxiv.org/abs/1412.7024" rel="noreferrer">Courbariaux at al</a> "Training Deep Neural Networks with Low Precision Multiplications" about <strong>10-bit</strong> activations and <strong>12-bit</strong> parameter updates. </p></li>
<li><p>And this is not the limit. <a href="https://arxiv.org/abs/1602.02830" rel="noreferrer">Courbariaux et al</a>, "BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1". Here they discuss <strong>1-bit</strong> activations and weights (though higher precision for the gradients), which makes the forward pass super fast.</p></li>
</ul>

<p>Of course, I can imagine some networks may require high precision for training, but I would recommend at least to try 16 bits for training a big network and switch to 32 bits if it proves to work worse.</p>
float16 vs float32 for convolutional neural networks
<p>The standard is float32 but I'm wondering under what conditions it's ok to use float16?</p>

<p>I've compared running the same covnet with both datatypes and haven't noticed any issues. With large dataset I prefer float16 because I can worry less about memory issues..</p>