<p><strong>TL;DR</strong>: 1. Yes 2. Yes 3. No
<hr>
<strong>TS;WM</strong>:</p>

<ol>
<li><a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">Batch normalization</a> was a great invention by Sergey Ioffe and Christian Szegedy early 2015. Back in those days, battling vanishing or exploding gradients was an everyday problem. Read that article if you want to gain a deep understanding. but basically this quote from the abstract should give you some idea:</li>
</ol>

<blockquote>
  <p>Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs.</p>
</blockquote>

<p>They did in fact first use batch normalization for DCNNs, which allowed them to beat human performance in the top-5 ImageNet classification, but any network where there are nonlinearities can benefit from batch normalization. Including a network consisting of fully-connected layers.</p>

<ol start="2">
<li><p>Yes, it is used for shallow CNN-s too. Any network with more than one layer can benefit from it, albeit it is true that more benefit comes to deeper networks.</p></li>
<li><p>First of all, one-hot vectors should <em>never</em> be normalized. Normalization means you subtract the mean and divide by the variance, thus creating a dataset with 0 mean and 1 variance. If you do this to a one-hot vector, then the cross-entropy loss calculation will be completely off. Second, there is no point in normalizing a concat layer separately, since it does not change the values, just concatenates them. Batch normalization is done on the <em>input</em> of a layer, so the one after the concat, that will get the concatenated values, can do it if necessary.</p></li>
</ol>
Is batchnorm used in neural networks that are not CNN?
<p>1.) Batchnorm is always used in deep convolutional neural networks. But is it also used in not-CNN. In NN. In networks with just fully-connected layers?</p>

<p>2.) Is batchnorm used in shallow CNNs?</p>

<p>3.) If I have a CNN with an input image and an input array IN_array, the output is an array after the last fully-connected layer. I call this array FC_array. If I want to concat that FC_array with the IN_array.</p>

<pre><code>CONCAT_array = tf.concat(values=[FC_array, IN_array])
</code></pre>

<p>Is it useful to have a bachnorm after the concat layer? Or should that batchnorm be just after the FC_array before the concat layer?</p>

<p>For information, the IN_array is a tf.one_hot() vector.</p>

<p>Thank you</p>