<p>I don't know details of <a href="https://arxiv.org/pdf/1511.06434.pdf" rel="nofollow noreferrer">DCGAN thesis</a>, but if I look into it I can find the below guidelines to make stable DCGAN. Why did you use <code>LeakyReLU</code> in Generator instead of <code>ReLU</code>?</p>

<blockquote>
  <p><strong>Architecture guidelines for stable Deep Convolutional GANs</strong></p>
  
  <ul>
  <li>Replace any pooling layers with strided convolutions (discriminator) and fractional-strided convolutions (generator).</li>
  <li>Use batchnorm in both the generator and the discriminator.</li>
  <li>Remove fully connected hidden layers for deeper architectures.</li>
  <li><strong>Use ReLU activation in generator</strong> for all layers except for the output, which uses Tanh.</li>
  <li>Use LeakyReLU activation in the discriminator for all layers</li>
  </ul>
</blockquote>
Poor Result with BatchNormalization
<p>I have been trying to implement the <a href="https://arxiv.org/abs/1511.06434" rel="nofollow noreferrer">DCGan</a>, the face book's paper, and blocked by below two issues almost for 2 weeks. Any suggestions would be appreciated. Thanks.</p>

<p><strong>Issue 1:</strong></p>

<p>DCGAN paper suggest to use BN(Batch Normalization) both the generator and discriminator. But, I couldn't get better result with BN rather than w/out BN. </p>

<p>I copied DCGAN model I used which is exactly the same with a DCGAN paper. I don't think it is due to overfitting. Because (1) It keeps showing the noise the same with an initial noise picture and seems never been trained.
(2) The Loss value is very stable that gan and discriminator both are not really changed. (It's staying about 0.6 ~ 0.7 and never felt down or bumped up like when both models are collapsed.) If I check only loss function, seems like it is getting trained well.</p>

<p><strong>Issue 2:</strong> </p>

<p>When I used float16, it always gives me Nan with the model below.
I have changed epsilon as 1e-4 1e-3 both but failed.
And here is one more question.
If I don't use the BatchNormalization, it can be Nan. it enough makes sense, I can get it. But, if I use BatchNormalization, it normalizes in every layer. Even if the result becomes very big number or very small number it will be batch normalized in every layer that the result will be almost centered and the fade-out shouldn't happen. isn't it? it's actually my thought but I don't know what I am thinking wrong..
please, somebody, help me.</p>

<p>===== Generator =====</p>

<p>Input  # (None, 128) &lt;= latent</p>

<p>Dense  # (None, 16384)<br>
BatchNormalization<br>
LeakyReLU                                                              </p>

<p>Reshape  # (None, 4, 4, 1024)  </p>

<p>Conv2DTranspose  # (None, 4, 4, 512)</p>

<p>BatchNormalization<br>
LeakyReLU</p>

<p>Conv2DTranspose  # (None, 8, 8, 256)</p>

<p>BatchNormalization<br>
LeakyReLU</p>

<p>Conv2DTranspose  # (None, 16, 16, 128)</p>

<p>BatchNormalization<br>
LeakyReLU      </p>

<p>Conv2DTranspose  # (None, 32, 32, 64)</p>

<p>BatchNormalization<br>
LeakyReLU </p>

<p>Conv2DTranspose  # (None, 64, 64, 32)</p>

<p>BatchNormalization<br>
LeakyReLU</p>

<p>Conv2DTranspose  # (None, 128, 128, 16)</p>

<p>BatchNormalization<br>
LeakyReLU</p>

<p>Conv2D  # (None, 128, 128, 3)            </p>

<p>===== Discriminator =====</p>

<p>Conv2D  # (None, 128, 128, 3)
LeakyReLU  </p>

<p>Conv2D  # (None, 64, 64, 16)
BatchNormalization<br>
Dropout<br>
LeakyReLU</p>

<p>Conv2D  # (None, 32, 32, 32)<br>
BatchNormalization<br>
Dropout<br>
LeakyReLU</p>

<p>Conv2D  # (None, 16, 16, 64)<br>
BatchNormalization<br>
Dropout<br>
LeakyReLU</p>

<p>Conv2D  # (None, 8, 8, 128)<br>
BatchNormalization<br>
Dropout<br>
LeakyReLU</p>

<p>Conv2D  # (None, 4, 4, 256)<br>
BatchNormalization<br>
Dropout<br>
LeakyReLU</p>

<p>Conv2D  # (None, 2, 2, 512)<br>
BatchNormalization<br>
Dropout<br>
LeakyReLU</p>

<p>Flatten<br>
Dropout<br>
Dense</p>

<p>and the last hyperparameters I have tried are as below and I didn't forget to add the gaussian noise to training pictures.</p>

<pre><code>image_shape =&gt; (128, 128, 3)
latent_dim =&gt; 128
channels =&gt; 3
iterations =&gt; 10000
batch_size =&gt; 128
epsilon =&gt; 0.005
weight_init_stddev =&gt; 0.02
beta_1 =&gt; 0.5
discriminator_lr =&gt; 0.0002
gan_lr =&gt; 0.0002
</code></pre>