<p>A neural network can be seen as a function approximation tool. The quality of the approximation is defined by its' error, i.e. how far the prediction is from the underlying ground truth. If we leave the practicioner approach (trial &amp; error) aside, there's two theories through which we can investigate the effect of number of nodes (aka width) on the network's quality; one is theory of computation, and the other is algebraic topology. Neither has yet provided results immediately translatable to "if you add another node then this happens", but both have some very nice insights to offer. I am not sure if this is the kind of answer you are expecting, but I will try to very briefly walk you through the major points of what the latter field offers in terms of explanations.</p>

<p><strong>Algebraic Topology / Control Theory</strong></p>

<ol>
<li>A "shallow" network (i.e. a single dense layer) can approximate with arbitrary low error any continuous function under the assumption of <strong>no constraints</strong> on number of nodes. What this says is that your network can learn (almost) perfectly whatever you throw at it, no matter how complex it is, provided you can let it use as many nodes as it wants (potentially countably infinite). Practically, even though we know that there exists a shallow network that approximates with error <em>ε</em>→ 0 a continuous function <em>f</em>, we do not know what that network is or how to estimate its parameters. Generally, the more complex <em>f</em> is and the lower we want <em>ε</em> to be, the more nodes we would need, up to the point where training becomes unfeasible due to the curse of dimensionality. In very applied terms, this means that the wider your layer, the richer your representation, the more accurate your prediction. As a side effect you will also have more parameters that need to be trained, thus more data requirements, and measures against overfitting will become necessary.</li>
<li>A high-rank tensor, such as the ones usually used as objective functions for neural networks, can be decomposed into a series of potentially lower rank tensors. This effectively reduces degrees of freedom and makes numeric representation easier with far lesser parameters. However, determining the rank of the decomposition (the number of summands) is NP-Hard, as is determining the coefficients themselves. The <strong>shallow network</strong> corresponds to the canonical decomposition, so due to it being NP-Hard, no claims can be made regarding the number of nodes necessary to construct a perfect approximation. What we do know, however, is that <strong>recurrent networks</strong> correspond to another sort of decomposition, the Tensor-Train decomposition, which is far more memory efficient and stable; therefore we know that <em>a shallow network would need exponentially more width to mimic a recurrent network of the same width</em>. Similarly, we know that a <strong>convolutional network</strong> corresponds the Hierarchical-Tucker decomposition, which is also more efficient than the shallow network; therefore a <em>conv layer can compute in polynomial size what would require super-polynomial size for a shallow layer</em>. </li>
</ol>

<p>Refs: </p>

<ol>
<li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Approximation by Superpositions of a Sigmoidal Function</a></li>
<li><a href="http://proceedings.mlr.press/v48/cohenb16.pdf" rel="nofollow noreferrer">Convolutional Rectifier Networks as Generalized Tensor Decomposition</a></li>
<li><a href="https://arxiv.org/abs/1711.00811" rel="nofollow noreferrer">Expressive Power of Recurrent Neural Networks</a></li>
</ol>

<p><strong>TL;DR</strong>: We don't know much about how much width is necessary for some approximation, but we can compare width-efficiency between different network types. We know a shallow network (fully-connected layer) can approximate anything if we let it grow with no constraints. We also know that an exponential increase in its size is equivalent to a linear increase in a recurrent layer size, and that a super-polynomial increase in its size is equivalent to a polynomial increase in a convolutional layer. So if you're adding width, it better be on an RNN cell :)</p>

<p>The computational theory perspective follows a different route; that is, translating various network types to computation theoretic machines and inspecting their Turing degree. There are claims made about the number of nodes necessary to simulate a Turing machine using shallow nets, and how various networks relate to one-another in terms of size complexity, but I'm not sure if this is anywhere close to what you're asking so I'll skip this part.</p>

<p>I did not go into the comparison between width and depth efficiency either, as this is not something you're asking, but there are many more experimental results on that topic (and many SO answers far better than I could ever write myself). </p>
Intuition for varying number of nodes in deep learning model
<p>Other than just trial and error, what impact does varying the number of nodes in a deep learning model achieve?</p>

<p>How I interpret it is this: each learned representation of a layer is a dense vector if the number of nodes is low and inversely each representation is a sparse vector if the number of nodes is high. How does this contribute to more or less accurate training accuracy?</p>