<p>As I understand the mapping network is not trained separately. It it part of generator network and adjusts weights based on gradients just like other parts of the network.</p>
<p>In their <a href="https://github.com/NVlabs/stylegan/blob/1e0d5c781384ef12b50ef20a62fee5d78b38e88f/training/networks_stylegan.py#L300" rel="nofollow noreferrer">stylegan generator code implementation</a> it written the Generator is composed of two sub networks one mapping and another synthesis. In <a href="https://github.com/NVlabs/stylegan3/blob/a5a69f58294509598714d1e88c9646c3d7c6ec94/training/networks_stylegan3.py#L511" rel="nofollow noreferrer">stylegan3 generator source</a> it is much easier to see. The output of mapping is passed to synthesis network which generates image.</p>
<pre><code>class Generator(torch.nn.Module):
    ...
    def forward(self, z, ...):
        ws = self.mapping(z, ...)
        img = self.synthesis(ws, ...)
        return img
</code></pre>
<p>The diagram below shows mapping network from <a href="https://arxiv.org/abs/1812.04948" rel="nofollow noreferrer">stylegan 2019 paper</a>. Section 2 describes about mapping network.</p>
<h2>Generator Diagram with Mapping Layer</h2>
<p><a href="https://i.stack.imgur.com/6lCrE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6lCrE.png" alt="enter image description here" /></a></p>
<p>Mapping layer is represented with <code>f</code> in the paper that takes noise vector <code>z</code> initialized from normal distribution and maps to intermediate latent representation <code>w</code>. It is implemented with 8 layer MLP. <a href="https://github.com/NVlabs/stylegan/blob/1e0d5c781384ef12b50ef20a62fee5d78b38e88f/training/networks_stylegan.py#L384" rel="nofollow noreferrer">Stylegan mapping network implementation</a> has MLP layers set to 8.</p>
<p>In section 4 they mention,</p>
<blockquote>
<p>a common goal is a latent space that consists of linear subspaces, each of which controls one factor of variation. However, the sampling probability of each combination of factors in <code>Z</code> needs to match the corresponding density in the training data.</p>
</blockquote>
<blockquote>
<p>A major benefit of our generator architecture is that the intermediate latent space <code>W</code> does not have to support sampling according to any fixed distribution.</p>
</blockquote>
<p>So, <code>z</code> and <code>w</code> have same dimensions but <code>w</code> is more disentangled than <code>z</code>. Finding a <code>w</code> from intermediate latent space <code>W</code> for an image allows specific image editing.</p>
<p>From <a href="https://arxiv.org/pdf/2102.02766.pdf" rel="nofollow noreferrer">Encoder for Editing</a> paper,</p>
<p><a href="https://i.stack.imgur.com/mn74n.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mn74n.png" alt="enter image description here" /></a></p>
<p>In stylegan2-ada paper with other changes they found mapping network depth of 2 better than 8. In <a href="https://github.com/NVlabs/stylegan3/blob/a5a69f58294509598714d1e88c9646c3d7c6ec94/training/networks_stylegan3.py#L115" rel="nofollow noreferrer">stylegan3 mapping layer code implementation</a> default number of layers in mapping is set to 2.</p>
<h2>References</h2>
<ul>
<li><a href="https://arxiv.org/abs/1812.04948" rel="nofollow noreferrer">https://arxiv.org/abs/1812.04948</a></li>
<li><a href="https://arxiv.org/abs/2102.02766" rel="nofollow noreferrer">https://arxiv.org/abs/2102.02766</a></li>
<li><a href="https://github.com/NVlabs/stylegan3" rel="nofollow noreferrer">https://github.com/NVlabs/stylegan3</a></li>
<li><a href="https://github.com/NVlabs/stylegan" rel="nofollow noreferrer">https://github.com/NVlabs/stylegan</a></li>
</ul>
How does Mapping Network in StyleGAN work?
<p>I am learning StyleGAN architecture and I got confused about the purpose of the Mapping Network. In the original paper it says:</p>
<blockquote>
<p>Our mapping network consists of 8 fully-connected layers, and the
dimensionality of all input and output activations— including z and w
— is 512.</p>
</blockquote>
<p>And there is no information about this network being trained in any way.</p>
<p>Like, wouldn’t it just generate some nonsense values?</p>
<p>I've tried creating a network like that (but with a smaller shape <code>(16,)</code>):</p>
<pre class="lang-py prettyprint-override"><code>import tensorflow as tf
import numpy as np

model = tf.keras.models.Sequential()
model.add(tf.keras.Input(shape=(16)))

for i in range(7):
  model.add(tf.keras.layers.Dense(16, activation='relu'))

model.compile()
</code></pre>
<p>and then evaluated it on some random values:</p>
<pre class="lang-py prettyprint-override"><code>g = tf.random.Generator.from_seed(34)
model(
    g.normal(shape=(16, 16))
)
</code></pre>
<p>And I am getting some random outputs like:</p>
<pre><code>array([[0.        , 0.01045225, 0.        , 0.        , 0.02217731,
        0.00940356, 0.02321716, 0.00556996, 0.        , 0.        ,
        0.        , 0.03117323, 0.        , 0.        , 0.00734158,
        0.        ],
       [0.03159791, 0.05680077, 0.        , 0.        , 0.        ,
        0.        , 0.05907414, 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.03110216, 0.04647615, 0.        ,
        0.04566741],
       .
       .  # More similar vectors goes there
       .   
       [0.        , 0.01229661, 0.00056016, 0.        , 0.03534952,
        0.02654905, 0.03212402, 0.        , 0.        , 0.        ,
        0.        , 0.0913604 , 0.        , 0.        , 0.        ,
        0.        ]], dtype=float32)&gt;
</code></pre>
<p>What am I missing? Is there any information on the Internet about training Mapping Network? Any math explanation? Got really confused :(</p>