title: What are the inputs to the transformer encoder and decoder in BERT?
body: <p>I was reading the <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">BERT paper</a> and was not clear regarding the inputs to the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noreferrer">transformer</a> encoder and decoder. </p>
  
  <p>For learning masked language model (Cloze task), the paper says that 15% of the tokens are masked and the network is trained to predict the masked tokens. Since this is the case, what are the inputs to the transformer encoder and decoder?</p>
  
  <p><a href="https://i.stack.imgur.com/jIIuo.png" rel="noreferrer"><img src="https://i.stack.imgur.com/jIIuo.png" alt="BERT input representation (from the paper)"></a></p>
  
  <p>Is the input to the transformer encoder this input representation (see image above). If so, what is the decoder input?</p>
  
  <p>Further, how is the output loss computed? Is it a softmax for only the masked locations? For this, the same linear layer is used for all masked tokens?</p>