title: High performance impression tracking
body: <p>Basically, one part of some metrics that I would like to track is the amount of impressions that certain objects receive on our marketing platform.
  If you imagine that we display lots of objects, we would like to track each time an object is served up.</p>
  
  <p>Every object is returned to the client through a single gateway/interface. So if you imagine that a request comes in for a page with some search criteria, and then the search request is proxied to our Solr index.</p>
  
  <p>We then get 10 results back.</p>
  
  <p>Each of these 10 results should be regarded as an impression.</p>
  
  <p>I'm struggling to find an incredibly fast and accurate implementation.</p>
  
  <p>Any suggestions on how you might do this? You can throw in any number of technologies. We currently use, Gearman, PHP, Ruby, Solr, Redis, Mysql, APC and Memcache.</p>
  
  <p>Ultimately all impressions should eventually be persisted to mysql, which I could do every hour. But I'm not sure how to store the impressions in memory fast without effecting the load time of the actual search request.</p>
  
  <h1>Ideas  (I just added option 4 and 5)</h1>
  
  <ol>
  <li><p>Once the results are returned to the client, the client then requests a base64 encoded URI on our platform which contains the ID's of all of the objects that they have been served. This object is then passed to gearman, which then saves the count to redis. Once an hour, redis is flushed and the count is increments for each object in mysql.</p></li>
  <li><p>After the results have been returned from Solr, loop over, and save directly to Redis. (Haven't benchmarked this for speed). Repeat the flushing to mysql every hour.</p></li>
  <li><p>Once the items are returned from Solr, send all the ID's in a single job to gearman, which will then submit to Redis..</p></li>
  <li><p><strong>new idea</strong> Since the most number of items returned  will be around 20, I could set a X-Application-Objects header with a base64 header of the ID's returned. These ID's (in the header) could then be stripped out by nginx, and using a custom LUA nginx module, I could write the ID's directly to Redis from nginx. This might be overkill though. The benefit of this though is that I can tell nginx to return the response object immediately while it's writing to redis. </p></li>
  <li><p><strong>new idea</strong> Use <code>fastcgi_finish_request()</code> in order to flush the request back to nginx, but then insert the results into Redis. </p></li>
  <li><p>Any other suggestions?</p></li>
  </ol>
  
  <h2>Edit to Answer question:</h2>
  
  <p>The reliability of this data is not essential. So long as it is a best guess. I wouldn't want to see a swing of say 30% dropped impressions. But I would allow a tolerance of 10% -/+ acurracy. </p>