<p>Continuous neural networks are not known to be universal approximators (in the sense of density in $L^p$ or $C(\mathbb{R})$ for the topology of uniform convergence on compacts, i.e.: as in <a href="https://www.sciencedirect.com/science/article/pii/0893608089900208" rel="nofollow noreferrer">the universal approximation theorem</a>) but only universal interpolators in the sense of this paper:
<a href="https://arxiv.org/abs/1908.07838" rel="nofollow noreferrer">https://arxiv.org/abs/1908.07838</a></p>
Continuous vs Discrete artificial neural networks
<p>I realize that this is probably a very niche question, but has anyone had experience with working with continuous neural networks? I'm specifically interested in what a continuous neural network may be useful for vs what you normally use discrete neural networks for.</p>
<p>For clarity I will clear up what I mean by continuous neural network as I suppose it can be interpreted to mean different things. I do <strong>not</strong> mean that the activation function is continuous. Rather I allude to the idea of a increasing the number of  neurons in the hidden layer to an infinite amount.</p>
<p>So for clarity, here is the architecture of your typical discreet NN:
<a href="https://i.stack.imgur.com/XU442.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XU442.png" alt="alt text" /></a><br />
<sub>(source: <a href="https://sites.google.com/site/garamatt/nn.png" rel="nofollow noreferrer">garamatt at sites.google.com</a>)</sub></p>
<p>The <code>x</code> are the input, the <code>g</code> is the activation of the hidden layer, the <code>v</code> are the weights of the hidden layer, the <code>w</code> are the weights of the output layer, the <code>b</code> is the bias and apparently the output layer has a linear activation (namely none.)</p>
<p>The difference between a discrete NN and a continuous NN is depicted by this figure:
<a href="https://i.stack.imgur.com/kl3uA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/kl3uA.png" alt="alt text" /></a><br />
<sub>(source: <a href="https://sites.google.com/site/garamatt/nn2.png" rel="nofollow noreferrer">garamatt at sites.google.com</a>)</sub></p>
<p>That is you let the number of hidden neurons become infinite so that your final output is an integral. In practice this means that instead of computing a deterministic sum you instead must approximate the corresponding integral with quadrature.</p>
<p>Apparently its a common misconception with neural networks that too many hidden neurons produces over-fitting.</p>
<p>My question is specifically, given this definition of discrete and continuous neural networks, I was wondering if anyone had experience working with the latter and what sort of things they used them for.</p>
<p>Further description on the topic can be found here:
<a href="http://www.iro.umontreal.ca/%7Elisa/seminaires/18-04-2006.pdf" rel="nofollow noreferrer">http://www.iro.umontreal.ca/~lisa/seminaires/18-04-2006.pdf</a></p>