title: Use Bert to predict multiple tokens
body: <p>I'm looking for suggestions on using Bert and Bert's masked language model to predict multiple tokens.</p>
  
  <p>My data looks like:</p>
  
  <p>context: <code>some very long context paragraph</code></p>
  
  <p>question: <code>rainy days lead to @placeholder</code> and the answer for this <code>@placeholder</code> is <code>wet weather</code>. In the model, <code>wet environment</code> is the answer to predict. </p>
  
  <p>So at the pre-processing stage, should I change the text into <code>rainy days lead to [MASK]</code> or something like <code>rainy days lead to [MASK] [MASK]</code>? I know that the masked LM works well on the single token prediction, do you think the masked LM can work well on the multiple tokens prediction? If no, do you have any suggestions on how to pre-process and train this kind of data?</p>
  
  <p>Thanks so much!</p>