title: How to differentiate Encryption from Compression in real time traffic
body: <p>I work on network traffic filter and I need to classify input IP-traffic into 3 groups:</p>
  
  <ul>
  <li>Encrypted traffic</li>
  <li>Compressed traffic</li>
  <li>Neither encrypted nor compressed traffic</li>
  </ul>
  
  <p>Compressed and Encrypted data have many similar attributes, such as entropy and character dispersion in content histograms. So it is difficult to distinguish them.</p>
  
  <p>There is a way to classify data when chunk is big (> 1MB), using Chi Square Distribution,   Pi Approximation Error, and Entropy methods. <a href="http://www.devttys0.com/2013/06/differentiate-encryption-from-compression-using-math/" rel="nofollow noreferrer">http://www.devttys0.com/2013/06/differentiate-encryption-from-compression-using-math/</a></p>
  
  <p>This algorithm is not accurate with small amount of data (~1k).</p>
  
  <p><strong>The question</strong>:</p>
  
  <p>How to differentiate Encryption from Compression in real time traffic (~1k data blocks) with more than 90 percents accuracy?</p>
  
  <p>I know that modern DPI(Deep Packet Inspection) systems already have some of algorithms implemented to deal with this problem.</p>
  
  <p>Do you have some ideas?</p>