title: Mongoengine / MongoDB model filtering on basis of key presence in Dictionary field
body: <p>I'm building a data-model using Mongoengine to store some metada for, in this case, email files. The very stripped down model, with only the field relevant for my issue, is:</p>
  
  <pre><code>class Email(Document):
      headers = DictField()
  </code></pre>
  
  <p>The headers dictionary will contain extracted header-data from emails which works in a key-value relationship.</p>
  
  <p>In this header-data I know, at times, there will be a header present with the name (and dictionary key): <code>x-mailer</code> (all header keys are auto lower-cased). I've build a simple query to see if a header contains this key like this:</p>
  
  <pre><code>xmailer_emails = Email.objects.filter(headers__exists='x-mailer')
  </code></pre>
  
  <p>However, the results of this don't contain <em>all</em> the entries in the <code>email</code> collection but it contains quite a few that don't have <code>x-mailer</code> as a key in the <code>headers</code> dictionary. Here is some code I used to check the resulting data:</p>
  
  <pre><code>xmailer_emails = Email.objects.filter(headers__exists='x-mailer'))
  log(xmailer_emails._query)
  
  log('Total email count: ' + str(Email.objects.count()))
  log('X-Mailer email acount: ' + str(xmailer_emails.count()))
  
  no_xmailer = len([e for e in xmailer_emails if 'x-mailer' not in e.headers])
  log('Filtered no x-mailer count: ' + str(no_xmailer))
  
  has_xmailer = len([e for e in xmailer_emails if 'x-mailer' in e.headers])
  log('Filtered has x-mailer count: ' + str(has_xmailer))
  </code></pre>
  
  <p>Here is the output I get:</p>
  
  <pre><code>[21:32:54 05/27/19] {'headers': {'$exists': 'x-mailer'}}
  [21:32:54 05/27/19] Total email count: 86
  [21:32:54 05/27/19] X-Mailer email count: 79
  [21:32:54 05/27/19] Filtered no x-mailer count: 55
  [21:32:54 05/27/19] Filtered has x-mailer count: 24
  </code></pre>
  
  <p>So while there are 86 entries in the collection it pulls back 79 of them on a filter for <code>x-mailer</code> in <code>headers</code>. Filtering out of those only 24 actually have that key-value in the dictionary. I seem to be doing something wrong but I don't know what.</p>
  
  <p>Here's a short dump of a few of the items that it also pulls back that do not contain the <code>x-mailer</code> key, these are the keys:</p>
  
  <pre><code>dict_keys(['x-receiver', 'to', 'mime-version', 'received', 'x-priority', 'x-sender', 'date', 'content-type', 'message-id', 'subject', 'x-riferimento-message-id', 'from'])
  dict_keys(['content-type', 'mime-version'])
  dict_keys(['x-wum-to', 'date', 'x-uidl', 'message-id', 'in-reply-to', 'x-wum-replyto', 'x-message-delivery', 'mime-version', 'received', 'x-savecopy', 'authentication-results', 'x-wum-nature', 'x-account-key', 'x-wum-from', 'message-context', 'x-originalarrivaltime', 'x-me-spamrating', 'x-me-spamlevel', 'to', 'x-dkim-result', 'x-mozilla-status', 'x-mozilla-status2', 'references', 'x-auth-result', 'x-store-info', 'return-path', 'x-message-info', 'x-sid-pra', 'x-message-status', 'sender', 'x-wum-cci', 'content-type', 'reply-to', 'subject', 'from'])
  dict_keys(['x-ms-has-attach', 'to', 'mime-version', 'received', 'thread-topic', 'x-auto-response-suppress', 'x-ms-exchange-organization-authsource', 'content-language', 'content-type', 'date', 'subject', 'message-id', 'thread-index', 'reply-to', 'from'])
  </code></pre>
  
  <p>I can't figure out what is going on, while the <code>exists</code> operator works it also includes non-matching documents but not every one in the collection.</p>
  
  <p>I've also raised this question as an issue on the mongoengine repository with no reply (besides a linked SO issue for the <code>exists</code> operator): <a href="https://github.com/MongoEngine/mongoengine/issues/2059" rel="nofollow noreferrer">https://github.com/MongoEngine/mongoengine/issues/2059</a></p>