title: mongodb query data where the key is variable and aggregate results using python
body: <p>i'm currently working with a saltstack implementation that is using a mongodb returner and i'm trying to query the results that are being stored in it for reporting purposes. unfortunately the way that the returner is writing information to the database looks like the following:</p>
  
  <blockquote>
    <p>db.hostname21.find({"20140421035007474763" : {$exists : 1}}, {_id: 0}).pretty()</p>
  </blockquote>
  
  <pre><code>{
       "fun" : "state.sls",
       "20140421035007474763" : {
              "file_|-Get-Logins-Stats_|-/scripts/server_logins_stats_|-managed" : {
                      "comment" : "File /scripts/server_logins_stats is in the correct state",
                      "__run_num__" : 2,
                      "changes" : {
  
                      },
                      "name" : "/scripts/server_logins_stats",
                      "result" : true
              },
              "service_|-Logger_|-rsyslog_|-running" : {
                      "comment" : "Service rsyslog is already enabled, and is in the desired state",
                      "__run_num__" : 1,
                      "changes" : {
  
                      },
                      "name" : "rsyslog",
                      "result" : true
              },
              "cmd_|-Run_Script_|-/scripts/server_logins_stats_|-run" : {
                      "comment" : "Command \"/scripts/server_logins_stats\" run",
                      "__run_num__" : 4,
                      "changes" : {
                              "pid" : 20899,
                              "retcode" : 0,
                              "stderr" : "",
                              "stdout" : "0"
                      },
                      "name" : "/scripts/server_logins_stats",
                      "result" : true
              },
              "cron_|-Schedule_Run_|-/scripts/server_logins_stats_|-present" : {
                      "comment" : "Cron /scripts/server_logins_stats already present",
                      "__run_num__" : 3,
                      "changes" : {
  
                      },
                      "name" : "/scripts/server_logins_stats",
                      "result" : true
              },
              "pkg_|-nc_|-nc_|-installed" : {
                      "comment" : "Package nc is already installed",
                      "__run_num__" : 0,
                      "changes" : {
  
                      },
                      "name" : "nc",
                      "result" : true
              }
      }
  }
  </code></pre>
  
  <p>as you can see here the keys are constantly changing instead of having a key that identifies each script. however, i have found that the failures are formatted pretty consistently, except that they don't have any keys identifying that it is a failure just an array of strings:</p>
  
  <pre><code>       "20140421041507478163" : [
              "Pillar failed to render with the following messages:",
              "Specified SLS 'globals' in environment 'Production' is not available on the salt master"
         ],
         "fun" : "state.sls"
  </code></pre>
  
  <p>so ultimately what i want to do is to be able to report on each failure by identifying the job, the host and the nature of the failure as well as the aggregate number of successes and failures over time. if you notice each host (minion) has its own collection created. so i've got a python script that will iterate over the collections to determine if the job was actually executed on that host:</p>
  
  <pre><code>import datetime
  import pymongo
  
  
  #hosts = ["mongoDBStaging", "mongoDBUAT"]
  hosts = ["mongodbuat"]
  
  for host in hosts:
      conn = pymongo.Connection(host)
      mdb = conn['salt']
      collections = set(mdb.collection_names())
  
  
      hosts = []
      jids = []
  
      # for every collection, identify whether it is a host or a job
      for c in collections:   
          # if the collection is a host add it to the host array
          if not (len(c) == 20 and int(c)):
              #print "{0} is a host".format(c)
              hosts.append(c)
          # other  wise add it to the job array
          else:
              #print "{0} is a jid".format(c)
              jids.append(c)
  
  
      for h in hosts:
          # for every job in a host connect to that collection 
          # and search for the job id to see if it exists 
          # and what its return was so we can report on that info
          for j in jids:
              coll = conn['salt'][h]
              #print "%s collection, %s jid" % (coll, j)
              for doc in coll.find({j: {'$exists': True}, "fun": "state.sls"}, {"_id": 0}):
                  print "{0}".format(coll)
                  print "{0} is a doc".format(doc)
  </code></pre>
  
  <p>but i'm struggling to be able to query into the results to actually determine whether it was successful or not. by being able to pull out elements of the document that is returned to read into the results from each document.</p>
  
  <p>if anyone has any suggestions on how i could consistently query into results where the keys are constantly changing to get to the results that would be really helpful.</p>