title: Why Bert transformer uses [CLS] token for classification instead of average over all tokens?
body: <p>I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task.</p>
  <p>Bert's last layer looks like this :</p>
  <p><a href="https://i.stack.imgur.com/m0jrg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/m0jrg.png" alt="enter image description here" /></a></p>
  <p>Where we take the [CLS] token of each sentence :</p>
  <p><a href="https://i.stack.imgur.com/1OklZ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/1OklZ.png" alt="enter image description here" /></a></p>
  <p><a href="https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/" rel="noreferrer">Image source</a></p>
  <p>I went through many discussion on this <a href="https://github.com/huggingface/transformers/issues/1950" rel="noreferrer">huggingface issue</a>,  <a href="https://datascience.stackexchange.com/questions/66207/what-is-purpose-of-the-cls-token-and-why-its-encoding-output-is-important">datascience forum question</a>,  <a href="https://github.com/google-research/bert/issues/196" rel="noreferrer">github issue</a> Most of the data scientist gives this explanation :</p>
  <blockquote>
  <p>BERT is bidirectional, the [CLS] is encoded including all
  representative information of all tokens through the multi-layer
  encoding procedure. The representation of [CLS] is individual in
  different sentences.</p>
  </blockquote>
  <p>My question is, Why the author ignored the other information ( each token's vector ) and taking the average, max_pool or other methods to make use of all information rather than using [CLS] token for classification?</p>
  <p>How does this [CLS] token help compare to the average of all token vectors?</p>