<p><strong>Is the implementation and understanding of the dense synthesizer correct?</strong></p>
<p>Not exactly, <code>linear1 = nn.Linear(d,d)</code> according to the paper and not <code>(d,l)</code>.
Of course this does not work if <code>X.shape = (l,d)</code> according to matrix multiplication rules.</p>
<p>This is because :</p>
<p><a href="https://i.stack.imgur.com/jNuyj.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jNuyj.png" alt="enter image description here" /></a>
<a href="https://i.stack.imgur.com/HSMW7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HSMW7.png" alt="enter image description here" /></a></p>
<p>So <code>F</code> is applied to each  <code>Xi</code> in X for i in <code>[1,l]</code></p>
<p>The resulting matrix <code>B</code> is then passed to the softmax function and multiplied by <code>G(x)</code>.
So you'd have to modify your code to sequentially process the input then use the returned matrix to compute <code>Y</code>.</p>
<p><strong>how is that different from a multi-layered perceptron that takes in two different inputs and makes uses of it at different point in the forward propagation?</strong></p>
<p>To understand, we need to put things into context, the idea of introducing attention mechanism was first described here in the context of Encoder - Decoder : <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1409.0473.pdf</a></p>
<p>The core idea is to allow the model to have control over how the context vector from the encoder is retrieved using a neural network instead of relying solely on the last encoded state :</p>
<p><a href="https://i.stack.imgur.com/LQYTm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/LQYTm.png" alt="enter image description here" /></a></p>
<p>see this <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms">post</a> for more detail.</p>
<p>The Transformers introduced the idea of using &quot;Multi-Head Attention&quot; (see graph below) to reduce the computational burden and focus solely on the attention mechanism itself. <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms">post</a></p>
<p><a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1706.03762.pdf</a></p>
<p><a href="https://i.stack.imgur.com/tgulR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/tgulR.png" alt="enter image description here" /></a></p>
<p><a href="https://i.stack.imgur.com/uxrkF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uxrkF.png" alt="enter image description here" /></a></p>
<p><em><strong>So where does the Dense synthesizer fits into all of that ?</strong></em></p>
<p>It simply replaces the Dot product (as illustrated in the first pictures in your post) by <code>F(.)</code>. If you replace what's inside the <code>softmax</code> by <code>F</code> you get the equation for <code>Y</code></p>
<p><a href="https://i.stack.imgur.com/YSpTL.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YSpTL.png" alt="enter image description here" /></a></p>
<p><strong>Conclusion</strong></p>
<p>This is an MLP but applied step wise to the input in the context of sequence processing.</p>
<p>Thank you</p>
Implementation of the Dense Synthesizer
<p>I’m trying to understand the Synthesizer paper (<a href="https://arxiv.org/pdf/2005.00743.pdf" rel="noreferrer">https://arxiv.org/pdf/2005.00743.pdf</a> 1) and there’s a description of the dense synthesizer mechanism that should replace the traditional attention model as described in the Transformer architecture.</p>

<p><a href="https://i.stack.imgur.com/DF09Y.png" rel="noreferrer"><img src="https://i.stack.imgur.com/DF09Y.png" alt="enter image description here"></a></p>

<p>The <strong>Dense Synthesizer</strong> is described as such:</p>

<p><a href="https://i.stack.imgur.com/EK7I3.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EK7I3.png" alt="enter image description here"></a></p>

<p>So I tried to implement the layer and it looks like this but I’m not sure whether I’m getting it right:</p>

<pre><code>class DenseSynthesizer(nn.Module):
    def __init__(self, l, d):
        super(DenseSynthesizer, self).__init__()
        self.linear1 = nn.Linear(d, l)
        self.linear2 = nn.Linear(l, l)

    def forward(self, x, v):
        # Equation (1) and (2)
        # Shape: l x l
        b = self.linear2(F.relu(self.linear1(x)))   
        # Equation (3)
        # [l x l] x [l x d] -&gt; [l x d]
        return torch.matmul(F.softmax(b), v) 
</code></pre>

<p>Usage:</p>

<pre><code>l, d = 4, 5

x, v =  torch.rand(l, d), torch.rand(l, d)

synthesis = DenseSynthesizer(l, d)
synthesis(x, v) 
</code></pre>

<p>Example:</p>

<p>x and v are tensors:</p>

<pre><code>x = tensor([[0.0844, 0.2683, 0.4299, 0.1827, 0.1188],
         [0.2793, 0.0389, 0.3834, 0.9897, 0.4197],
         [0.1420, 0.8051, 0.1601, 0.3299, 0.3340],
         [0.8908, 0.1066, 0.1140, 0.7145, 0.3619]])

v = tensor([[0.3806, 0.1775, 0.5457, 0.6746, 0.4505],
         [0.6309, 0.2790, 0.7215, 0.4283, 0.5853],
         [0.7548, 0.6887, 0.0426, 0.1057, 0.7895],
         [0.1881, 0.5334, 0.6834, 0.4845, 0.1960]])
</code></pre>

<p>And passing through a forward pass through the dense synthesis, it returns:</p>

<pre><code>&gt;&gt;&gt; synthesis = DenseSynthesizer(l, d)
&gt;&gt;&gt; synthesis(x, v) 

tensor([[0.5371, 0.4528, 0.4560, 0.3735, 0.5492],
        [0.5426, 0.4434, 0.4625, 0.3770, 0.5536],
        [0.5362, 0.4477, 0.4658, 0.3769, 0.5468],
        [0.5430, 0.4461, 0.4559, 0.3755, 0.5551]], grad_fn=&lt;MmBackward&gt;)
</code></pre>

<p><strong>Is the implementation and understanding of the dense synthesizer correct?</strong></p>

<p>Theoretically, <strong>how is that different from a multi-layered perceptron that takes in two different inputs and makes uses of it at different point in the forward propagation?</strong></p>