<h1 dir="auto">Summary</h1>
<p dir="auto">There seems to be an unexplained inconsistency between the results of <code class="notranslate">_percolate</code> and <code class="notranslate">_search</code> when using the same document and query set.</p>
<h1 dir="auto">Description</h1>
<p dir="auto">I have an index defined with a custom analyzer set as my default and a few string fields. The analyzer tokenizes on commas, and nothing else. Curling the mapping returns the following output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;news_documents_20140729&quot;: {
        &quot;mappings&quot;: {
            &quot;0000000022-nid&quot;: {
                &quot;_all&quot;: {
                    &quot;enabled&quot;: false
                },
                &quot;_timestamp&quot;: {
                    &quot;enabled&quot;: true,
                    &quot;store&quot;: true
                },
                &quot;_ttl&quot;: {
                    &quot;enabled&quot;: true
                },
                &quot;dynamic&quot;: &quot;false&quot;,
                &quot;analyzer&quot;: &quot;csv_lowercase_commas&quot;,
                &quot;properties&quot;: {
                     ...
                     &quot;subjects&quot;: {
                        &quot;type&quot;: &quot;string&quot;
                     },
                     &quot;product&quot;: {
                        &quot;type&quot;: &quot;string&quot;,
                        &quot;null_value&quot;: &quot;DJGM&quot;
                     },
                     ..."><pre class="notranslate"><code class="notranslate">{
    "news_documents_20140729": {
        "mappings": {
            "0000000022-nid": {
                "_all": {
                    "enabled": false
                },
                "_timestamp": {
                    "enabled": true,
                    "store": true
                },
                "_ttl": {
                    "enabled": true
                },
                "dynamic": "false",
                "analyzer": "csv_lowercase_commas",
                "properties": {
                     ...
                     "subjects": {
                        "type": "string"
                     },
                     "product": {
                        "type": "string",
                        "null_value": "DJGM"
                     },
                     ...
</code></pre></div>
<p dir="auto">The analyzer referenced above is defined as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&quot;csv_lowercase_commas&quot; : {
  &quot;type&quot; : &quot;custom&quot;,
  &quot;filter&quot; : [ &quot;lowercase&quot; ],
  &quot;tokenizer&quot; : &quot;commas&quot;
}"><pre class="notranslate"><code class="notranslate">"csv_lowercase_commas" : {
  "type" : "custom",
  "filter" : [ "lowercase" ],
  "tokenizer" : "commas"
}
</code></pre></div>
<p dir="auto">And the <code class="notranslate">commas</code> tokenizer is defined as follows:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&quot;commas&quot; : {
    &quot;pattern&quot; : &quot;,+&quot;,
    &quot;type&quot; : &quot;pattern&quot;
}"><pre class="notranslate"><code class="notranslate">"commas" : {
    "pattern" : ",+",
    "type" : "pattern"
}
</code></pre></div>
<p dir="auto">I've indexed a very simple document to the above index, as the shown type..</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{ &quot;subjects&quot;:&quot;N/DJMT&quot;, &quot;product&quot;:&quot;DJGM&quot;}"><pre class="notranslate"><code class="notranslate">{ "subjects":"N/DJMT", "product":"DJGM"}
</code></pre></div>
<p dir="auto">I search for the above document using the following three queries, and the document is returned each time, as expected.</p>
<h2 dir="auto">Match All Query</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{&quot;query&quot;: { &quot;match_all&quot;:{}}}"><pre class="notranslate"><code class="notranslate">{"query": { "match_all":{}}}
</code></pre></div>
<h2 dir="auto">Term Query</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;query&quot;: {
        &quot;filtered&quot;: {
            &quot;filter&quot;: {
                &quot;and&quot;: [
                    {
                        &quot;term&quot;: {
                            &quot;product&quot;: &quot;djgm&quot;
                        }
                    },
                    {
                        &quot;in&quot;: {
                            &quot;subjects&quot;: [
                                &quot;n/djmt&quot;
                            ]
                        }
                    }
                ]
            }
        }
    }
}"><pre class="notranslate"><code class="notranslate">{
    "query": {
        "filtered": {
            "filter": {
                "and": [
                    {
                        "term": {
                            "product": "djgm"
                        }
                    },
                    {
                        "in": {
                            "subjects": [
                                "n/djmt"
                            ]
                        }
                    }
                ]
            }
        }
    }
}
</code></pre></div>
<h2 dir="auto">Phrase Query</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;query&quot;: {
        &quot;filtered&quot;: {
            &quot;filter&quot;: {
                &quot;term&quot;: {
                    &quot;product&quot;: &quot;djgm&quot;
                }
            },
            &quot;query&quot;: {
                &quot;match_phrase&quot; : {
                    &quot;subjects&quot; : &quot;n/djmt&quot;
                }
            }
        }
    }
}"><pre class="notranslate"><code class="notranslate">{
    "query": {
        "filtered": {
            "filter": {
                "term": {
                    "product": "djgm"
                }
            },
            "query": {
                "match_phrase" : {
                    "subjects" : "n/djmt"
                }
            }
        }
    }
}
</code></pre></div>
<h2 dir="auto">Here's the problem</h2>
<p dir="auto">I add each of the above queries to the .percolator type in the above described index, and then percolate <code class="notranslate">{"doc":{ "subjects":"N/DJMT", "product":"DJGM"}}</code> -- i.e., the exact same document content I indexed and searched on previously. <em>However, the result contains only the <code class="notranslate">Match All Query</code> and the <code class="notranslate">Phrase Query</code></em>.</p>
<p dir="auto">To be more concise...</p>
<p dir="auto"><strong>Searching for the document using the above queries returns the document independent of which query is used to perform the search. Percolating the same document, with the above three queries indexed in the same index's .percolator type, returns <em>only the first two queries</em>.</strong></p>
<p dir="auto"><strong>The phrase query is inexplicably missing from the percolate results.</strong></p>
<h1 dir="auto">More Information</h1>
<p dir="auto">This was verified on a 1.1.0 instance, within a clean index. It was originally discovered in a production index through a slightly more complicated, albeit technically synonymous (by my assessment), example.</p>
<p dir="auto"><del>Sorry, I don't have access to a 1.3 cluster at the moment, so I haven't verified it there.</del><br>
Also verified on a 1.3.4 test cluster I just spooled up for testing.</p>
<p dir="auto">A quick grep through the issue tracker using <code class="notranslate">percolate is:open label:bug</code> didn't show anything similar.</p>
<h1 dir="auto">(Naive) Partially-substantiated Insights</h1>
<p dir="auto">Running the same experiment, except removing the '/' in the string 'n/djmt' in both all of the queries and the document, produces the expected, correct results. In other words, <code class="notranslate">_percolate</code> returns all three queries, and <code class="notranslate">_search</code> returns the document in response to all three queries. <em>(Did this on a 1.3.4 cluster with the same mappings as described above.)</em></p>
<p dir="auto">The combination of (1) the punctuation being a variable in the problem and (2) the fact that analysis is applied to phrase queries and not to term queries, has me believing the issue has to do with analysis, specifically it not being applied properly when percolating.</p>
<p dir="auto"><em>Lastly, if I'm missing something trivial, then I apologize in advance for wasting time. It's very likely I'm ignorant of some crucial fact here.</em></p>
<p dir="auto">Recently, we moved from version <code class="notranslate">0.19.11</code> to last stable <code class="notranslate">0.90.0</code> and found one very strange behavior that looks like an issue.</p>
<p dir="auto">If on the item root level we have specified our custom <code class="notranslate">index_analyzer</code>, <code class="notranslate">search_analyzer</code> (or just <code class="notranslate">analyzer</code>), then <code class="notranslate">index_analyzer</code> works well, but not the <code class="notranslate">search_analyzer</code>. Also, fif we update existing mapping with explicitly specifying <code class="notranslate">search_analyzer</code> on the field level, then it still doesn't seem to work and ES uses standard one.</p>
<h3 dir="auto">To reproduce:</h3>
<p dir="auto">Create a new index and define our custom analyzer <code class="notranslate">de_stem</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPUT 'http://localhost:9200/issue/' -d '{&quot;index&quot;: {&quot;number_of_shards&quot;: 1,&quot;analysis&quot;: {&quot;filter&quot;: {&quot;de_snowball&quot;: {&quot;type&quot;: &quot;snowball&quot;,&quot;language&quot;: &quot;German&quot;}},&quot;analyzer&quot;: {&quot;de_stem&quot;: {&quot;type&quot;: &quot;custom&quot;,&quot;tokenizer&quot;: &quot;standard&quot;,&quot;filter&quot;: [&quot;lowercase&quot;, &quot;de_snowball&quot;]}}}}},&quot;number_of_replicas&quot;: 0}}'"><pre class="notranslate"><code class="notranslate">curl -XPUT 'http://localhost:9200/issue/' -d '{"index": {"number_of_shards": 1,"analysis": {"filter": {"de_snowball": {"type": "snowball","language": "German"}},"analyzer": {"de_stem": {"type": "custom","tokenizer": "standard","filter": ["lowercase", "de_snowball"]}}}}},"number_of_replicas": 0}}'
</code></pre></div>
<p dir="auto">Put mapping with specified <code class="notranslate">index_analyzer</code> and <code class="notranslate">search_analyzer</code> :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPUT 'http://localhost:9200/issue/item/_mapping' -d '{&quot;item&quot;: {&quot;index_analyzer&quot; : &quot;de_stem&quot;,&quot;search_analyzer&quot; : &quot;de_stem&quot;,&quot;properties&quot;: {&quot;content&quot;: {&quot;dynamic&quot;: false,&quot;properties&quot;: {&quot;body&quot;: {&quot;type&quot;: &quot;string&quot;}}}}}}}'"><pre class="notranslate"><code class="notranslate">curl -XPUT 'http://localhost:9200/issue/item/_mapping' -d '{"item": {"index_analyzer" : "de_stem","search_analyzer" : "de_stem","properties": {"content": {"dynamic": false,"properties": {"body": {"type": "string"}}}}}}}'
</code></pre></div>
<p dir="auto">Try <code class="notranslate">search_analyzer</code> for the field <code class="notranslate">content.body</code> with Analyze API</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET 'localhost:9200/issue/_analyze?pretty=true&amp;field=content.body' -d 'Apple'"><pre class="notranslate"><code class="notranslate">curl -XGET 'localhost:9200/issue/_analyze?pretty=true&amp;field=content.body' -d 'Apple'
</code></pre></div>
<h4 dir="auto">Actual result:</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;tokens&quot; : [ {
    &quot;token&quot; : &quot;apple&quot;,
    &quot;start_offset&quot; : 0,
    &quot;end_offset&quot; : 5,
    &quot;type&quot; : &quot;&lt;ALPHANUM&gt;&quot;,
    &quot;position&quot; : 1
  } ]
}"><pre class="notranslate"><code class="notranslate">{
  "tokens" : [ {
    "token" : "apple",
    "start_offset" : 0,
    "end_offset" : 5,
    "type" : "&lt;ALPHANUM&gt;",
    "position" : 1
  } ]
}
</code></pre></div>
<h4 dir="auto">Expected result:</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;tokens&quot; : [ {
    &quot;token&quot; : &quot;appl&quot;,
    &quot;start_offset&quot; : 0,
    &quot;end_offset&quot; : 5,
    &quot;type&quot; : &quot;&lt;ALPHANUM&gt;&quot;,
    &quot;position&quot; : 1
  } ]
}"><pre class="notranslate"><code class="notranslate">{
  "tokens" : [ {
    "token" : "appl",
    "start_offset" : 0,
    "end_offset" : 5,
    "type" : "&lt;ALPHANUM&gt;",
    "position" : 1
  } ]
}
</code></pre></div>
<p dir="auto">The right (expected) result still possible to get, but with explicitly specified <code class="notranslate">search_analyzer</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET 'localhost:9200/issue/_analyze?pretty=true&amp;field=content.body&amp;analyzer=de_stem' -d 'Apple'"><pre class="notranslate"><code class="notranslate">curl -XGET 'localhost:9200/issue/_analyze?pretty=true&amp;field=content.body&amp;analyzer=de_stem' -d 'Apple'
</code></pre></div>
<h3 dir="auto">Index Analyzer is set well</h3>
<p dir="auto">As we see above, <code class="notranslate">search_analyzer</code> seems wasn't set, but <code class="notranslate">index_analyzer</code>  works well.</p>
<p dir="auto">Let's index a document:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -PUT 'http://localhost:9200/issue/item/1' -d '{&quot;content&quot; : {&quot;body&quot;: &quot;10 Things We Hate About Apple&quot;}}'"><pre class="notranslate"><code class="notranslate">curl -PUT 'http://localhost:9200/issue/item/1' -d '{"content" : {"body": "10 Things We Hate About Apple"}}'
</code></pre></div>
<p dir="auto">If <code class="notranslate">index_analyzer</code> was set well to <code class="notranslate">de_stem</code> the word <code class="notranslate">Apple</code> should be indexed as <code class="notranslate">appl</code>, but not <code class="notranslate">apple</code> (as <code class="notranslate">standard</code> analyzer does).</p>
<p dir="auto">Let's search for <code class="notranslate">appl</code> first:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET 'http://localhost:9200/issue/_search?search_type=count&amp;pretty=true' -d '{&quot;query&quot;:{&quot;query_string&quot;:{&quot;fields&quot;:[&quot;content.body&quot;],&quot;query&quot;:&quot;appl&quot;}}}'"><pre class="notranslate"><code class="notranslate">curl -XGET 'http://localhost:9200/issue/_search?search_type=count&amp;pretty=true' -d '{"query":{"query_string":{"fields":["content.body"],"query":"appl"}}}'
</code></pre></div>
<p dir="auto">It works! We get back 1 result:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  &quot;hits&quot; : {
    &quot;total&quot; : 1,
    &quot;max_score&quot; : 0.0,
    &quot;hits&quot; : [ ]
  }"><pre class="notranslate"><code class="notranslate">  "hits" : {
    "total" : 1,
    "max_score" : 0.0,
    "hits" : [ ]
  }
</code></pre></div>
<p dir="auto">For the word <code class="notranslate">apple</code>, as expected, it doesn't work since <code class="notranslate">search_analyzer</code> is standard, but  <code class="notranslate">index_analyzer</code> is <code class="notranslate">de_stem</code> (so, actual search term will stay <code class="notranslate">apple</code>, but indexed is <code class="notranslate">appl</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET 'http://localhost:9200/issue/_search?search_type=count&amp;pretty=true' -d '{&quot;query&quot;:{&quot;query_string&quot;:{&quot;fields&quot;:[&quot;content.body&quot;],&quot;query&quot;:&quot;apple&quot;}}}'"><pre class="notranslate"><code class="notranslate">curl -XGET 'http://localhost:9200/issue/_search?search_type=count&amp;pretty=true' -d '{"query":{"query_string":{"fields":["content.body"],"query":"apple"}}}'
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  &quot;hits&quot; : {
    &quot;total&quot; : 0,
    &quot;max_score&quot; : 0.0,
    &quot;hits&quot; : [ ]
  }"><pre class="notranslate"><code class="notranslate">  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
</code></pre></div>
<h3 dir="auto">Specifying search analyzer with Put Mapping API doesn't help</h3>
<p dir="auto">Ok, i try to update mapping and specify explicitly <code class="notranslate">search analyzer</code> for <code class="notranslate">content.body</code> field on the existing index we created above:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -PUT 'http://localhost:9200/issue/item/_mapping' -d '{&quot;item&quot;: {&quot;properties&quot;: {&quot;content&quot;: {&quot;dynamic&quot;: false,&quot;properties&quot;: {&quot;body&quot;: {&quot;type&quot;: &quot;string&quot;, &quot;search_analyzer&quot;: &quot;de_stem&quot;}}}}}}'"><pre class="notranslate"><code class="notranslate">curl -PUT 'http://localhost:9200/issue/item/_mapping' -d '{"item": {"properties": {"content": {"dynamic": false,"properties": {"body": {"type": "string", "search_analyzer": "de_stem"}}}}}}'
</code></pre></div>
<p dir="auto">Response is <code class="notranslate">ok</code>, but the all problems described above stay the same. So, it seems the <code class="notranslate">search_analyzer</code> for the field <code class="notranslate">content.body</code> is still <code class="notranslate">standard</code>.</p>