title: How to understand synergy in information theory?
body: <p>In information theory, <strong>multivariate mutual information (MMI)</strong> could be synergy (negative) or redundancy (positive). To simulate this two cases, assuming three variables X, Y and Z, all of them takes 0 or 1 (binary variable). And we repeat sampling them 12 times.</p>
  
  <p><strong>Case 1:</strong></p>
  
  <pre><code>X = [ 0 0 0 0 0 0 1 1 1 1 1 1 ]
  Y = [ 0 0 0 0 1 1 0 0 1 1 1 1 ]
  Z = [ 0 0 1 1 1 1 0 0 0 0 1 1 ]
  </code></pre>
  
  <p>In this case, we assume a mechanism among XYZ taht when both Y and Z are 0 or 1, X takes 0 or 1 respectively. When Y = 0, Z = 1, then X takes 0, and Y = 1, Z = 0, then X takes 1.</p>
  
  <p>The <strong>mmi(X,Y,Z) = -0.1699</strong> in this case, indicating a synergy effect among three variable.</p>
  
  <p><strong>Case 2:</strong></p>
  
  <pre><code>X = [ 0 0 0 0 0 0 1 1 1 1 1 1 ]
  Y = [ 0 0 0 0 0 1 0 1 1 1 1 1 ]
  Z = [ 0 1 1 1 1 1 0 0 0 0 0 1 ]
  </code></pre>
  
  <p>the machanism in this case is same as above. The difference is there are more samples of XY takes different value and less samples of both XY are 0 or 1.</p>
  
  <p>The <strong>mmi(X,Y,Z) = 0.0333</strong>, indicating a redundancy.</p>
  
  <p>So far, can I say in these two cases, synergy and redundancy show the similar mechanism (or relationship) among three variables? But how do we understand redundancy and particularly synergy in realistic data?</p>