<p>I'm not sure if there is standard practice, but what I saw the others have done is to simply take the average of the sub-tokens embeddings. example: <a href="https://arxiv.org/abs/2006.01346" rel="nofollow noreferrer">https://arxiv.org/abs/2006.01346</a>, Section 2.3 line 4</p>
How do you get single embedding vector for each word (token) from RoBERTa?
<p>As you may know, <code>RoBERTa (BERT, etc.)</code> has its own tokenizer and sometimes you get pieces of given word as tokens, e.g. embeddings » embed, #dings</p>
<p>Since the nature of the task I am working on, I need a single representation for each word. How do I get it?</p>
<p><strong>CLEARANCE:</strong></p>
<blockquote>
<p>sentence: &quot;embeddings are good&quot; --&gt; 3 word tokens given<br />
output: [embed,#dings,are,good] --&gt; 4 tokens are out</p>
</blockquote>
<p>When I give <em>sentence</em> to pre-trained RoBERTa, I get encoded tokens. At the end I need representation for each token. Whats the solution? <strong>Summing embed + #dings tokens point-wise?</strong></p>