title: Why do I get and how to remove duplicates from solr results regarding my unique field?
body: <p>I am using Solr 3.6.0 for full-text search.
  I have the following fields defined in my schema.xml:</p>
  
  <pre><code>&lt;field name="productNumber" type="ngramtext" indexed="true" stored="true" required="true" /&gt;  
  &lt;field name="additionalTextData" type="text_en" indexed="true" stored="true"/&gt;  
  </code></pre>
  
  <p>I have also the following field type defined:</p>
  
  <pre><code>&lt;fieldType name="ngramtext" class="solr.TextField" positionIncrementGap="100"&gt;
      &lt;analyzer type="index"&gt;
          &lt;tokenizer class="solr.KeywordTokenizerFactory"/&gt;
          &lt;filter class="solr.LowerCaseFilterFactory"/&gt;
          &lt;filter class="solr.NGramFilterFactory" minGramSize="2" maxGramSize="3" /&gt;
      &lt;/analyzer&gt;
      &lt;analyzer type="query"&gt;
          &lt;tokenizer class="solr.KeywordTokenizerFactory"/&gt;
          &lt;filter class="solr.LowerCaseFilterFactory"/&gt;
      &lt;/analyzer&gt;
  &lt;/fieldType&gt;
  </code></pre>
  
  <p>This creates a tokens with size 2 and 3 and places them in the index. Great for searching <a href="http://en.wikipedia.org/wiki/Stock-keeping_unit" rel="nofollow">SKUs</a>. For now they go in the same index, and I know that this might create performance issues. But I decided to deal with that when the performance issue appears. For the data set that I have now, the speed is great (subjectively speaking :))</p>
  
  <p>I also have the following:</p>
  
  <pre><code>&lt;uniqueKey&gt;productNumber&lt;/uniqueKey&gt;
  </code></pre>
  
  <p>In the solrconfig.xml I have placed the additionalTextData for every "df" part:</p>
  
  <pre><code>="df"&gt;="df"&gt;additionalTextData&lt;
  </code></pre>
  
  <p>When I do the following query:</p>
  
  <p>select?indent=on&amp;version=2.2&amp;q=productNumber:12&amp;fq=&amp;start=0&amp;rows=10000&amp;fl=productNumber,additionalTextData&amp;wt=json&amp;explainOther&amp;hl.fl</p>
  
  <p>or </p>
  
  <p>select?indent=on&amp;version=2.2&amp;q=productNumber:12&amp;fq=additionalTextData:*&amp;start=0&amp;rows=10000&amp;fl=productNumber,additionalTextData&amp;wt=json&amp;explainOther&amp;hl.fl</p>
  
  <p><strong>I get 2345 results with some of them repeating. I search by some productNumber and I can see it twice or 3 times. Why does this happen? And how can I overcome this?</strong> </p>
  
  <p><strong>EDIT1:</strong><br>
  I found out also that adding:</p>
  
  <pre><code>&lt;filter class="solr.RemoveDuplicatesTokenFilterFactory" /&gt;
  </code></pre>
  
  <p>in ngramtext fieldType after all other filters, doesn't solve the issue.</p>
  
  <p><strong>SOLUTION1 according to mbonaci:</strong><br>
  instead of:</p>
  
  <pre><code>&lt;field name="productNumber" type="ngramtext" indexed="true" stored="true" required="true" /&gt;
  </code></pre>
  
  <p>place:</p>
  
  <pre><code>&lt;field name="productNumber" type="string" indexed="true" stored="true" required="true" /&gt;
  &lt;field name="productNumberCopyField" type="ngramtext" indexed="true" stored="true" required="true" /&gt;
  </code></pre>
  
  <p>After the fields tag add:</p>
  
  <pre><code>&lt;copyField source="productNumber" dest="productNumberCopyField" /&gt;
  </code></pre>
  
  <p>And, specify the following query:</p>
  
  <p>select?indent=on&amp;version=2.2&amp;q=<strong>productNumberCopyField</strong>:12&amp;fq=additionalTextData:*&amp;start=0&amp;rows=10000&amp;fl=productNumber,additionalTextData&amp;wt=json&amp;explainOther&amp;hl.fl</p>