Take the 2-minute tour ×

It is common practice to not index so called stop words when analyzing documents for a search engine. Stop words are common words, such as a, the, and this, that appear frequently in language. The idea is that if stop words are indexed, they take up too much space in the index and add little to the quality of the search results.

I would like to know if this is always the case.

In modern search engines, does indexing stop words make the index size explode? Or is it just a marginal increase.

Also, how does removing stop words affect phrase searches? Searching for "beatles" and "the beatles" seem to be two very different things.

I am building an app with elasticsearch, but this question is equally applicable to Solr, direct lucene, or any other variant.

share|improve this question

closed as off topic by Mitch Wheat, Burkhard, Firo, hims056, j0k Nov 28 '12 at 11:47


With google "beatles" and "the beatles" (without the "") gives the same result, so I guess that the "the" is indeed ignored. –  Burkhard Nov 28 '12 at 8:06
When i test the same thing it get 382.000.000 and 232.000.000 results. So there is a difference. But, can't tell what it's worth. –  TysHTTP Nov 28 '12 at 8:14
@TysHTTP: As a general note: the number of "hits" is not reliable (due to search engines optimizations, when it is sure what the top K should be - it stops searching and stop midifying the "results found"). Though I have no idea if this is the case, and I assume it is not. –  amit Nov 28 '12 at 8:27

2 Answers 2

up vote 3 down vote accepted
  1. The main problem with stop words is not the index size - but the quality of the answer. They tend to be dominant (have very high tf value and thus might make the results returned wrong), and not the size of the index.
    In any case, indexing stop words does not increase the size of the index significantly (and it definetly does not "explode")

  2. One way to overcome it is to use the stop words (and not omit them completely) when indexing n-grams. I don't know if it actually being done, but it definitely can help improve the returned results.

Also: stop words are not always* omitted. In sarcasm detectors, for example - it seems (empirically) stop words are very significant to the answer.

share|improve this answer

I think all search engines handle this differently. You can read about these things at: http://searchenginewatch.com

But if you are just one guy who's building a (small) app, i don't think you should focus on these minor details and just leave out these words and focus on the more relevant words instead.

share|improve this answer
