title: scikit-learn TfidfVectorizer ignoring certain words
body: <p>I'm trying TfidfVectorizer on a sentence taken from wikipedia page about the History of Portugal. However i noticed that the <code>TfidfVec.fit_transform</code> method is ignoring certain words. Here's the sentence i tried with:</p>
  
  <pre><code>sentence = "The oldest human fossil is the skull discovered in the Cave of Aroeira in Almonda."
  
  TfidfVec = TfidfVectorizer()
  tfidf = TfidfVec.fit_transform([sentence])
  
  cols = [words[idx] for idx in tfidf.indices]
  matrix = tfidf.todense()
  pd.DataFrame(matrix,columns = cols,index=["Tf-Idf"])
  </code></pre>
  
  <p>output of the dataframe:</p>
  
  <p><a href="https://i.stack.imgur.com/iEKnc.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iEKnc.png" alt="enter image description here"></a></p>
  
  <p>Essentially, it is ignoring the words "Aroeira" and "Almonda".</p>
  
  <p>But i don't want it to ignore those words so what should i do? I can't find anywhere on the documentation where they talk about this.</p>
  
  <p>Another question is why is the word "the" repeated? should the algorithm consider just one "the" and compute its tf-idf?</p>