title: In CBOW model, do we need to take Average at Hidden layer?
body: <p>I search and read some articles about CBOW. But seem to have difference between these articles.</p>
  <p>As I understand:</p>
  <ul>
  <li>Input is a batch vector. And we will feed it to Hidden layer. So that we will get another <strong>batch vector H</strong> at Hidden layer.</li>
  <li>In an <a href="https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/" rel="nofollow noreferrer">article (part 2.2.1)</a>, they say that we will not use any Activation Function at Hidden layer, but we will <strong>take average on <em>batch vector H</em></strong> to get a single vector (not a batch anymore). Then we will feed this average vector to Output layer and apply Softmax on it.</li>
  </ul>
  <p><a href="https://i.stack.imgur.com/x1Gsb.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/x1Gsb.png" alt="enter image description here" /></a></p>
  <ul>
  <li>However, in this <a href="https://www.coursera.org/lecture/probabilistic-models-in-nlp/training-a-cbow-model-forward-propagation-Vphwi" rel="nofollow noreferrer">Coursera's video</a>, they <em>don't</em> take average on batch vector H. They just feed this batch vector H to Output layer and apply Softmax on batch Output vector. And then calculate Cost function on it.</li>
  <li>And, in <a href="https://www.coursera.org/lecture/probabilistic-models-in-nlp/training-a-cbow-model-forward-propagation-Vphwi" rel="nofollow noreferrer">Coursera's video</a>, they say that we can use RelU as Activation function at Hidden layer. Is this a new method? Because I read many articles, but they always say that there is no Activation function at Hidden layer.</li>
  </ul>
  <p>Can you please help me to answer it?</p>