<p>From <a href="https://stats.stackexchange.com/q/232032/12359">least amount of bits needed for single neuron</a>:</p>

<p>The following papers have studied this question (descending chronological order):</p>

<ul>
<li>Accelerating Deep Convolutional Networks using low-precision and sparsity. Ganesh Venkatesh, Eriko Nurvitadhi, Debbie Marr. 2016-10-02. <a href="https://arxiv.org/abs/1610.00324" rel="nofollow noreferrer">https://arxiv.org/abs/1610.00324</a></li>
<li>Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio
arxiv: <a href="http://arxiv.org/abs/1602.02830" rel="nofollow noreferrer">http://arxiv.org/abs/1602.02830</a></li>
<li>Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan
Deep Learning with Limited Numerical Precision <a href="https://arxiv.org/abs/1502.02551" rel="nofollow noreferrer">https://arxiv.org/abs/1502.02551</a></li>
<li>Courbariaux, Matthieu, Jean-Pierre David, and Yoshua Bengio. "Training deep neural networks with low precision multiplications." arXiv preprint arXiv:1412.7024 (2014). <a href="https://arxiv.org/abs/1412.7024" rel="nofollow noreferrer">https://arxiv.org/abs/1412.7024</a></li>
<li>Vanhoucke, Vincent, Andrew Senior, and Mark Z. Mao. "Improving the speed of neural networks on CPUs." (2011). <a href="https://scholar.google.com/scholar?cluster=14667574137314459294&amp;hl=en&amp;as_sdt=0,22" rel="nofollow noreferrer">https://scholar.google.com/scholar?cluster=14667574137314459294&amp;hl=en&amp;as_sdt=0,22</a> ; <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf" rel="nofollow noreferrer">https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37631.pdf</a></li>
</ul>

<p>Example from <em>Deep Learning with Limited Numerical Precision</em>:</p>

<p><a href="https://i.stack.imgur.com/jv6Kr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jv6Kr.png" alt="enter image description here"></a></p>
deep neural network's precision for image recognition, float or double?
<p>Neural networks for image recognition can be really big.
There can be thousands of inputs/hidden neurons, millions of connections what
can take up a lot of computer resources.</p>

<p>While <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format" rel="nofollow noreferrer">float</a> being commonly 32bit and <a href="https://en.wikipedia.org/wiki/Double-precision_floating-point_format" rel="nofollow noreferrer">double</a> 64bit in c++, they don't have much performance difference in speed yet using floats can save up some memory.</p>

<p>Having a neural network what is using <a href="https://en.wikipedia.org/wiki/Logistic_function" rel="nofollow noreferrer">sigmoid</a> as an activation function,
if we could choose of which variables in neural network can be float or double
which could be float to save up memory without making neural network unable to perform?</p>

<p>While inputs and outputs for training/test data can definitely be floats
because they do not require double precision since colors in image can
only be in range of 0-255 and when normalized 0.0-1.0 scale, unit value would be
1 / 255 = 0.0039~ </p>

<p><strong>1. what about hidden neurons output precision,
would it be safe to make them float too?</strong></p>

<p>hidden neuron's output gets it value from the sum of previous layer neuron's output * its connection weight to currently being calculating neuron and then sum being passed into activation function(currently sigmoid) to get the new output. Sum variable itself could be double since it could become a really large number when network is big.</p>

<p><a href="https://i.stack.imgur.com/gzrsx.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gzrsx.png" alt="enter image description here"></a></p>

<p><strong>2. what about connection weights, could they be floats?</strong></p>

<p>while inputs and neuron's outputs are at the range of 0-1.0 because of sigmoid,
weights are allowed to be bigger than that.</p>

<hr>

<p><a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="nofollow noreferrer">Stochastic gradient descent</a> <a href="https://en.wikipedia.org/wiki/Backpropagation" rel="nofollow noreferrer">backpropagation</a> suffers on <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem" rel="nofollow noreferrer">vanishing gradient problem</a> because of the activation function's derivative, I decided not to put this out as an a question of what precision should gradient variable be, feeling that float will simply not be precise enough, specially when network is deep.</p>