title: Distributed Tracing logging and Integrating with Logstash,Kibana and ElasticSearch
body: <p>I have implemented a Distributed Transaction Logging library with Tree like Structure as mention in Google Dapper(<a href="http://research.google.com/pubs/pub36356.html" rel="nofollow">http://research.google.com/pubs/pub36356.html</a>) and eBay CAL Transaction Logging Framework(<a href="http://devopsdotcom.files.wordpress.com/2012/11/screen-shot-2012-11-11-at-10-06-39-am.png" rel="nofollow">http://devopsdotcom.files.wordpress.com/2012/11/screen-shot-2012-11-11-at-10-06-39-am.png</a>).</p>
  
  <p><strong>Log Format</strong></p>
  
  <pre><code>TIMESTAMP   HOSTNAME  DATACENTER  ENVIRONMENT  EVENT_GUID PARENT_GUID TRACE_GUID APPLICATION_ID TREE_LEVEL TRANSACTION_TYPE TRANSACTION_NAME STATUS_CODE DURATION(in ms) PAYLOAD(key1=value2,key2=value2)
  </code></pre>
  
  <p><strong>GUID HEX NUMBER FORMAT</strong></p>
  
  <pre><code>MURMER_HASH(HOSTNAME + DATACENTER + ENVIRONMENT)-JVM_THREAD_ID-(TIME_STAMP+Atomic Counter)
  </code></pre>
  
  <p>What I would like to do is to integrate this format with Kibana UI and when user want to search and click on on TRACE_GUID it will show something similar to Distributed CALL graph which show where the time was spent.  Here is UI <a href="http://twitter.github.io/zipkin/" rel="nofollow">http://twitter.github.io/zipkin/</a>.  This will be great.  I am not UI developer if some can point me how to do this that will be great.  </p>
  
  <p>Also I would like to know how I can index elastic search payload data so user specify some expression like in payload (duration > 1000) then, Elastic Search will bring all the loglines that satisfy condition. Also, I would like to index Payload as Name=Value pair so user can query  (key3=value2 or key4 = <em>exception</em>) some sort of regular expression.  Please let me know if this can be achieved.  Any help pointer would be great.. </p>
  
  <p>Thanks,
  Bhavesh</p>