<p>I must say PyTorch implementations are a bit confusing as it contains too many mask parameters. But I can shed light on the two mask parameters that you are referring to. Both <code>src_mask</code> and <code>src_key_padding_mask</code> is used in the <code>MultiheadAttention</code> mechanism. According to the documentation of <a href="https://pytorch.org/docs/master/generated/torch.nn.MultiheadAttention.html#torch.nn.MultiheadAttention" rel="noreferrer">MultiheadAttention</a>:</p>
<blockquote>
<p>key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention.</p>
<p>attn_mask – 2D or 3D mask that prevents attention to certain positions.</p>
</blockquote>
<p>As you know from the paper, <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">Attention is all you need</a>, MultiheadAttention is used in both Encoder and Decoder. However, in Decoder, there are two types of MultiheadAttention. One is called <code>Masked MultiheadAttention</code> and another one is the regular <code>MultiheadAttention</code>. To accommodate both these techniques, PyTorch uses the above mentioned two parameters in their MultiheadAttention implementation.</p>
<p>So, long story short-</p>
<ul>
<li><code>attn_mask</code> and <code>key_padding_mask</code> is used in Encoder's <code>MultiheadAttention</code> and Decoder's <code>Masked MultiheadAttention</code>.</li>
<li><code>memory_mask </code> is used in Decoder's <code>MultiheadAttention</code> mechanism as pointed out <a href="https://github.com/pytorch/pytorch/blob/ec5d579929b2c56418aacaec0874b92937d095a4/torch/nn/modules/transformer.py#L124-L127" rel="noreferrer">here</a>.</li>
</ul>
<p>Looking into the implementation of <a href="https://github.com/pytorch/pytorch/blob/11f1014c05b902d3eef0fe01a7c432f818c2bdfe/torch/nn/functional.py#L3854" rel="noreferrer">MultiheadAttention</a> might help you.</p>
<p>As you can see from <a href="https://github.com/pytorch/pytorch/blob/11f1014c05b902d3eef0fe01a7c432f818c2bdfe/torch/nn/functional.py#L4110" rel="noreferrer">here</a> and <a href="https://github.com/pytorch/pytorch/blob/11f1014c05b902d3eef0fe01a7c432f818c2bdfe/torch/nn/functional.py#L4117" rel="noreferrer">here</a>, first <code>src_mask</code> is used to block specific positions from attending and then <code>key_padding_mask</code> is used to block attending to pad tokens.</p>
<p><strong>Note.</strong> Answer updated based on @michael-jungo's comment.</p>
Difference between src_mask and src_key_padding_mask
<p>I am having a difficult time in understanding transformers. Everything is getting clear bit by bit but one thing that makes my head scratch is
what is the difference between src_mask and src_key_padding_mask which is passed as an argument in forward function in both encoder layer and decoder layer.</p>

<p><a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer" rel="noreferrer">https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer</a></p>