<p>The use of the <code>[CLS]</code> token to represent the entire sentence comes from the <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="noreferrer">original BERT paper</a>, section 3:</p>
<blockquote>
<p>The first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks.</p>
</blockquote>
<p>Your intuition is correct that averaging the vectors of all the tokens may produce superior results. In fact, that is exactly what is mentioned in the Huggingface documentation for <a href="https://huggingface.co/transformers/v3.0.2/model_doc/bert.html#bertmodel" rel="noreferrer">BertModel</a>:</p>
<blockquote>
<p><strong>Returns</strong></p>
<p>pooler_output (<code>torch.FloatTensor</code>: of shape <code>(batch_size, hidden_size)</code>):</p>
<p>Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pre-training.</p>
<p>This output is usually not a good summary of the semantic content of the input, <em>you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence</em>.</p>
</blockquote>
<p><strong>Update</strong>: Huggingface removed that statement (&quot;This output is usually not a good summary of the semantic content ...&quot;) in v3.1.0. You'll have to ask them why.</p>
Why Bert transformer uses [CLS] token for classification instead of average over all tokens?
<p>I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task.</p>
<p>Bert's last layer looks like this :</p>
<p><a href="https://i.stack.imgur.com/m0jrg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/m0jrg.png" alt="enter image description here" /></a></p>
<p>Where we take the [CLS] token of each sentence :</p>
<p><a href="https://i.stack.imgur.com/1OklZ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/1OklZ.png" alt="enter image description here" /></a></p>
<p><a href="https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/" rel="noreferrer">Image source</a></p>
<p>I went through many discussion on this <a href="https://github.com/huggingface/transformers/issues/1950" rel="noreferrer">huggingface issue</a>,  <a href="https://datascience.stackexchange.com/questions/66207/what-is-purpose-of-the-cls-token-and-why-its-encoding-output-is-important">datascience forum question</a>,  <a href="https://github.com/google-research/bert/issues/196" rel="noreferrer">github issue</a> Most of the data scientist gives this explanation :</p>
<blockquote>
<p>BERT is bidirectional, the [CLS] is encoded including all
representative information of all tokens through the multi-layer
encoding procedure. The representation of [CLS] is individual in
different sentences.</p>
</blockquote>
<p>My question is, Why the author ignored the other information ( each token's vector ) and taking the average, max_pool or other methods to make use of all information rather than using [CLS] token for classification?</p>
<p>How does this [CLS] token help compare to the average of all token vectors?</p>