title: Elasticsearch query cyclically returns different results
body: <h3>Background:</h3>
  
  <p>I've inherited an Elasticsearch project that is returning some very odd results, and I can't really determine what I need to do to properly fix this.</p>
  
  <p>Based on my reading of the code, it appears that 4 queries are run against the index based on search terms - the first one being exact match, the second and subsequent searches allowing more "slop" and "fuzziness". The search results with the highest scores are then combined into a single return; duplicate matches with lower scores are discarded.</p>
  
  <h3>Problem:</h3>
  
  <p>The queries with "slop" and "fuzziness" seem to cycle through results every 3 times I run the query. I've determined this by looking for a specific unique item in the query result and it only shows up 1 out of every 3 times I run the query. This cycle happens for all 3 of the non-exact match queries.</p>
  
  <h3>Additional information:</h3>
  
  <p>Based on the results of <code>_cat/segments?v&amp;index=[MY_INDEX_NAME]</code>, it appears that we have the index spread across 3 computers but only one shard. This gives me some hope that this is the reason we're getting the correct result only 1 out of every 3 times, but it's still very confusing as to why this would happen.</p>
  
  <h3>Band-aid fix:</h3>
  
  <p>I've been able to get consistent results for these problematic queries by increasing the "size" parameter from 50 to 150. This does slow the query down by a small amount, but at least it works for now. I am pretty sure that this isn't the correct solution.</p>
  
  <h3>Topology:</h3>
  
  <p><code>/_cat/nodes</code>:</p>
  
  <pre><code>ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
  xx.x.xx.x             2          58   0    0.00    0.00     0.00 i         -      elastic-ingest-001
  xx.x.xx.x            55          86   9    0.59    0.47     0.33 md        -      elastic-data-002
  xx.x.xx.x             1          57   0    0.03    0.02     0.00 i         -      elastic-ingest-000
  xx.x.xx.x            21          94   9    1.05    0.96     0.64 md        -      elastic-data-001
  xx.x.xx.x            18          84   7    0.22    0.21     0.19 md        *      elastic-data-000
  xx.x.xx.xx            7          58   0    0.00    0.00     0.00 i         -      elastic-ingest-002
  </code></pre>
  
  <p><code>/_cat/indices?v</code>:</p>
  
  <pre><code>health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
  green  open   .watcher-history-7-2018.06.05 aCfGd37MT5W2fJfK6HZjsQ   1   1          7            0      203kb        101.5kb
  green  open   .watcher-history-7-2018.06.29 WpTLI_WUSVeDUblRh59uvg   1   1      12114            0     25.2mb         12.7mb
  green  open   .watcher-history-7-2018.06.22 vt-LYb9NRSaZ46eReuixXg   1   1      11953            0     23.7mb         11.9mb
  green  open   .monitoring-es-6-2018.06.30   dnNVGu7pQ1GriAZLfIAQaA   1   1     458763          672    587.3mb          292mb
  green  open   .watcher-history-7-2018.06.02 8zM5yosrQIGJiSfzvMEC_A   1   1          0            0       460b           230b
  green  open   .triggered_watches            J9SWF-w8R2yd0aYtPBigAg   1   1          2        61157     53.7mb         24.4mb
  green  open   .watcher-history-7-2018.06.07 x0aT6E71RNCdXjIFEhHOPw   1   1      12094            0     24.3mb         12.3mb
  green  open   .watcher-history-7-2018.06.28 1nhqH54JQJOj9g63ov_NPw   1   1       8909            0       21mb         10.5mb
  green  open   .watcher-history-7-2018.06.26 _rpVOWKkS1mERWgFA5Myag   1   1       9144            0     22.1mb         11.1mb
  green  open   .watcher-history-7-2018.06.17 8zK45nMcR8WmGda4wGW82Q   1   1      12219            0     24.7mb         12.3mb
  green  open   [DIFFERENT_INDEX01]           0GCz0zu3R6SaRjNHiWOa6g   1   2    1818246            0      1.3gb        470.9mb
  green  open   .watcher-history-7-2018.06.20 FGhBth4OTJW-xusT7gplaw   1   1      12180            0     24.2mb         12.2mb
  green  open   .watcher-history-7-2018.06.27 -lK0pwYiTvi3a08dO7AoyQ   1   1       8955            0     20.9mb         10.4mb
  green  open   .watcher-history-7-2018.07.03 JmTpXIY7SXqoVodSpKRtMA   1   1      11896            0     24.3mb           12mb
  green  open   .watcher-history-7-2018.07.05 GMCpHn7MTc-D1HEtDa-Ydw   1   1       7853            0     16.5mb          8.3mb
  green  open   .watcher-history-7-2018.06.04 GXgFHhDdS9GJDou4sBd6RA   1   1          0            0       460b           230b
  green  open   .watches                      a3dbI5smSauUB7nSc8alTw   1   1          6            0    221.2kb        110.5kb
  green  open   .watcher-history-7-2018.06.19 aCzHvUa5SJ6n6wzKoXBJwA   1   1      12026            0       24mb         12.1mb
  green  open   .watcher-history-7-2018.06.09 56pGfAiWQmeNog8JVZtICw   1   1      11983            0     23.9mb         11.9mb
  green  open   .watcher-history-7-2018.06.01 MRqAmVqmThaIF_6KK5AlRQ   1   1          0            0       460b           230b
  green  open   .watcher-history-7-2018.07.02 Ij_8wgk4T-aJ6-PYAf9gqg   1   1      12015            0     24.4mb         12.2mb
  green  open   .watcher-history-7-2018.06.18 oZViVas5SoWd1D2_naVr3w   1   1      11996            0     23.9mb         11.9mb
  green  open   .watcher-history-7-2018.06.03 2_V6x656RCKGTe0IZyCkqA   1   1          0            0       460b           230b
  green  open   .watcher-history-7-2018.06.11 F4STy7gFS9a7e8qOV81AOA   1   1      11780            0     23.8mb         11.9mb
  green  open   .watcher-history-7-2018.06.10 MjxPItf4SOKtk4l0bPH7Tg   1   1      11909            0     23.7mb           12mb
  green  open   .monitoring-es-6-2018.07.04   3FPHjJFfTvuZrb71X3hcZA   1   1     501436          212      608mb        306.2mb
  green  open   .watcher-history-7-2018.06.12 STvls1wbSvCOU_kRerqckg   1   1      11897            0     24.1mb         12.1mb
  green  open   .monitoring-es-6-2018.07.05   k0wjXw5tR2KaBqrmvJAgCg   1   1     336928            0    488.2mb        242.3mb
  green  open   .security-6                   ldkFJ1TkRVScBdJIpA0Aeg   1   2          1            0     10.4kb          3.4kb
  green  open   [DIFFERENT_INDEX02]           RAcmKwl3RuiXMgGiRlX1HQ   2   2   46436060            0     60.8gb         20.2gb
  green  open   .monitoring-es-6-2018.07.03   nmBQmnnoTL2wZuF0O_pt1w   1   1     484715          306    593.1mb        305.2mb
  green  open   .monitoring-es-6-2018.06.28   lZR6SssRRx-yPQXk_vfBsw   1   1      97451          192    124.2mb           62mb
  green  open   .watcher-history-7-2018.07.04 8nDY3NoORYmWLOGpX5hb_g   1   1      12082            0     24.9mb         12.4mb
  green  open   .watcher-history-7-2018.07.01 _hmho-_zSu-D9H90gCKzWg   1   1      12072            0     24.9mb         12.5mb
  green  open   .watcher-history-7-2018.06.15 PGXkh70YTjOYhFLjK9a8pA   1   1      11946            0     24.3mb         12.1mb
  green  open   .watcher-history-7-2018.06.21 BEPkxD46TKm2y3yEaGgHNQ   1   1      12077            0       24mb         12.1mb
  green  open   .watcher-history-7-2018.06.14 Y74e7fY4SKS1aT8PK-S2vg   1   1      11907            0     23.9mb           12mb
  green  open   .watcher-history-7-2018.06.06 7opzBsl1SF-mQ_O8Y_5sJg   1   1       1424            0      3.1mb          1.5mb
  green  open   .monitoring-es-6-2018.07.01   AOG4_pk8RB-UanCjMM6GHg   1   1     467312          294    583.3mb        284.9mb
  green  open   .watcher-history-7-2018.06.24 pYKR7RG3RuGdgw7naxn-5Q   1   1      11955            0     23.8mb         11.8mb
  green  open   .watcher-history-7-2018.06.30 j4GdW5xhSNKeqT_c1376AQ   1   1      12125            0     25.1mb         12.7mb
  green  open   [DIFFERENT_INDEX03]           CDDpop1nTv6E3466IIhzCg   1   2    4591962       766253      9.4gb          2.6gb
  green  open   .watcher-history-7-2018.06.08 5eP2tPteTwGnoGJhQ37HoA   1   1      11848            0     23.8mb         12.1mb
  green  open   .watcher-history-7-2018.06.25 7xbkQaObSQWJhg93_PmFQw   1   1      12041            0     24.8mb         12.4mb
  green  open   .monitoring-es-6-2018.07.02   HBRphDn_TcSEXIiFn0ZtQg   1   1     475272          300    593.9mb          295mb
  green  open   .watcher-history-7-2018.06.13 CWOQnBuKTNa-DLGvo8XlMQ   1   1      11909            0     23.7mb         11.9mb
  green  open   [MY_INDEX]                    NdA3qJ16RGa5hpxvKpsDsg   1   2   10171359      1260206     24.1gb          6.4gb
  green  open   .monitoring-alerts-6          5HGKo73hQqa0dakVhdon6w   1   1         48            3    127.1kb         52.1kb
  green  open   .watcher-history-7-2018.06.16 7xyor_rvTemap3DWx6vkqg   1   1      12015            0     24.2mb         12.1mb
  green  open   .monitoring-es-6-2018.06.29   UfXjNo-ATjKKA0Hv5jZw-A   1   1     450751            0    580.1mb        287.3mb
  green  open   .watcher-history-7-2018.06.23 MyZMWHeYSm65MDen6WSGkw   1   1      11919            0     23.8mb         11.9mb
  </code></pre>
  
  <p><code>/MY_INDEX/_search_shards</code></p>
  
  <pre><code>{
      "nodes": {
          "9cP8Z9B8SFqq9Plszz7-HQ": {
              "name": "elastic-data-000",
              "ephemeral_id": "5Mm87T8lR5CFoIjoFGLQGg",
              "transport_address": "removed",
              "attributes": {}
          },
          "gg6rbEX8QdqujYjuAu9kvw": {
              "name": "elastic-data-002",
              "ephemeral_id": "I6ZpdVLgTyigh-2f7gtMFQ",
              "transport_address": "removed",
              "attributes": {}
          },
          "JDakz0EGT6aib0m87CfiCg": {
              "name": "elastic-data-001",
              "ephemeral_id": "c-Z3VRmtTsubCbXiSfsyOg",
              "transport_address": "removed",
              "attributes": {}
          }
      },
      "indices": {
          "MY_INDEX": {}
      },
      "shards": [
          [
              {
                  "state": "STARTED",
                  "primary": true,
                  "node": "9cP8Z9B8SFqq9Plszz7-HQ",
                  "relocating_node": null,
                  "shard": 0,
                  "index": "MY_INDEX",
                  "allocation_id": {
                      "id": "IQzbKGCMR9O0BobnePiKpg"
                  }
              },
              {
                  "state": "STARTED",
                  "primary": false,
                  "node": "gg6rbEX8QdqujYjuAu9kvw",
                  "relocating_node": null,
                  "shard": 0,
                  "index": "MY_INDEX",
                  "allocation_id": {
                      "id": "3fvXIyXGTa2NgAsb_uv78A"
                  }
              },
              {
                  "state": "STARTED",
                  "primary": false,
                  "node": "JDakz0EGT6aib0m87CfiCg",
                  "relocating_node": null,
                  "shard": 0,
                  "index": "MY_INDEX",
                  "allocation_id": {
                      "id": "whHOsuxfTdSnQDi-9RAuKw"
                  }
              }
          ]
      ]
  }
  </code></pre>