<p>This sounds related to the learning rate.</p>

<p>Scaling the inputs probably does not have too much effect on the network. Yes, normalizing the inputs e.g. with a z-score is standard and will help with convergence. But I think this is not what is driving your results.</p>

<p>For the VAE case I think it is the fact that your 10x or 100x factor will also scale the <em>outputs</em>. That will cause larger losses, which will send a stronger signal back through the rest of the network. You'll get faster convergence if your original learning rate was too low. Which, by the sound of things, it was. A very long period with nothing happening, followed by eventual convergence, often means you can train faster.</p>

<p>So, instead of scaling your inputs by 10x or 100x, try increasing your learning rate by 10x or 100x. The <a href="https://arxiv.org/abs/1708.07120" rel="nofollow noreferrer">Super-Convergence paper</a> by Leslie N. Smith has some great insights on high learning rates and a good LR schedule for fast convergence. If you want to go even deeper into hyperparameter tuning, he also has an excellent paper on <a href="http://%20https://arxiv.org/abs/1803.09820" rel="nofollow noreferrer">a disciplined approach to neural network hyper-parameters</a>.</p>
Link between range of input values and loss convergence
<p>I am training a variational autoencoder with tensorflow 2.0 using the Keras high-end API. The aim is to resonctruct images, which consist of a <em>shape</em> with homogenous intensity <code>int_shape</code> not equal to zero on a zero background, see following image with <code>int_shape = -0.25</code>:</p>

<p><a href="https://i.stack.imgur.com/8aYfJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8aYfJ.png" alt="enter image description here"></a></p>

<p>The variational autoencoder has the following architecture, w<em>ith the latent space being of 50 dimensions, opposed to the 16 stated in the image</em>:</p>

<p><a href="https://i.stack.imgur.com/7dJs4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7dJs4.png" alt="enter image description here"></a></p>

<p>The lambda layer uses a function to sample from a normal distribution which looks like that:</p>

<pre><code>def sampling(args):
    z_mean, z_log_var = args
    epsilon = K.random_normal(shape =(1,1,latent_dim))
    return z_mean + K.exp(0.5 * z_log_var) * epsilon
</code></pre>

<p>The loss is a combination of the KL divergence and an MSE loss:</p>

<pre><code>    def vae_loss(y_pred, y_gt):
        mse_loss = mse(y_pred, y_gt)
        original_dim = GLOBAL.input_res**2
        mse_loss *= original_dim
        z_mean = model.get_layer('z_mean_layer').output
        z_log_var = model.get_layer('z_log_var_layer').output
        kl_loss = 1 + z_log_var - K.square(z_mean) - K.exp(z_log_var)
        kl_loss = K.sum(kl_loss, axis=-1)
        kl_loss *= -0.5
        return K.mean(mse_loss + GLOBAL.beta*kl_loss)
</code></pre>

<p>Currently, <code>int_shape</code> takes values in the range (-0.5, 0.5), however of different magnitude (1e-1, 1e-2 and rarely 1e-3). The aim of the VAE is to take the image as input and reconstruct the same image. 
For the range (-0.5, 0.5) the convergence of the loss is not deterministic: Sometimes it stays constant the whole time, sometimes it stays constant for 5 / 50/180 epochs and then drops and converges. As a result, reconstructions are not consistently good, some are messy, sometimes only noise comes out. In my opinion, the convergence depends strongly on the initial starting point for optimization.</p>

<p>However, if I scale the data with 10, i.e. range (-5, 5) or even 100, i.e. (-50, 50), the loss converges and I <em>constantly</em> get good reconstructions. In the range (-5,5) higher absolute values tend to have better reconstructions, whereas in the range (-50,50) virtually every input is reconstructed correctly.</p>

<p><strong>My question is: Is there a link between the range of the input data and the convergence of the loss in the case of a VAE?</strong> </p>

<p>From my experiments, shapes with a higher <code>int_shape</code> tend to have better reconstructions. I would attribute this to higher gradients and thus bigger steps in the hyperparameter space, since the value of the gradients are dependant on the value of the input. Bigger steps lead to to point where the loss finds a minimum and converges. In this sense, smaller values would produce smaller gradients and thus smaller optimization steps will be taken.</p>

<p>Another theory that I have is that for a small range of (-0.5, 0.5) the points in the latent space representing the input are closer to each other and during random sampling they are confused and the "wrong" point is picked for the decoder. I have verified by plotting histograms of the latent space components. The higher the range of the input values, the higher the variance of the distribution in the latent space.</p>

<p>Update:
I tried various learning rates, ranging from 1e-4 to 5e-2, with and without learning rate decay (decrease by 10 % every 10 epochs). I also trained for a suficient amount of epochs (300), batch size 64, training set 3800 images, val on 100. The loss did not improve and reconstruction is not good. Both 32-bit and 64-bit floating point precision were utilized without any success. For learning rates bigger than 2e-2, loss ocasionally turns to Nan. The highest learning rate providing stable training is 1e-2. Also, latent space is 50 (should not have any effect on the problem though, IMO).</p>