<p>You better leave the gradients intact and make your optimizer so that it will count the effects you need.</p>

<p>Gradients will in most cases be deleted before a new forward anyway.</p>

<p>Some newer algorithms such as <a href="https://arxiv.org/abs/1904.00962" rel="nofollow noreferrer">Lamb</a> do the trick of parameter-wise (layer-wise) gradient normalization (MSE in this case) like you plan.</p>

<p>Also, check the <a href="https://arxiv.org/abs/1805.08318" rel="nofollow noreferrer">SAGAN</a> paper and why they used <code>nn.utils.spectral_norm</code> since you mantioned GANs.</p>
PyTorch normalize two sets of gradients during training
<p>In this <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noreferrer">GAN tutorial</a>, if you scroll down to the training loop you can see they combine the gradients 
 <code>errD = errD_real + errD_fake</code> like this. Where <code>errD_real = criterion(output, label)</code> and <code>errD_fake = criterion(output, label)</code> and <code>criterion = nn.BCELoss()</code>. I want to do the same thing but before doing a backward pass I want to normalize both gradients to the lower Euclidean norm of the two. How would I do that?</p>

<p>I know I can access the gradients of each weight individually on netD by printing out <code>netD.weight.grad</code>, but is there some way to batchnorm them to the lower Euclidean norm of the two?</p>

<p>Here's the part of the training loop I'm talking about:</p>

<pre><code>for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):

        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Classify all fake batch with D
        output = netD(fake.detach()).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()
        ...
</code></pre>