title: Need to make a very large array [2^16+1][2^16+1] - size of array is too large :(
body: <p>Greetings</p>
  
  <p>I need to calculate a first-order entropy (Markov source, like on wiki here <a href="http://en.wikipedia.org/wiki/Entropy_(information_theory" rel="nofollow">http://en.wikipedia.org/wiki/Entropy_(information_theory</a>) of a signal that consists of 16bit words. 
  This means, i must calculate how frequently each combination of a->b (symbol b appears after a) is happening in the data stream.
  When i was doing it for just 4 less significant or 4 more significant bits, i used a two dimensional array, where first dimension was the first symbol and second dimension was the second symbol.</p>
  
  <p>My algorithm looked like this</p>
  
  <ol>
  <li>Read current symbol</li>
  <li>Array[prev_symbol][curr_symbol]++</li>
  <li>prev_symbol=curr_symbol</li>
  <li>Move forward 1 symbol</li>
  </ol>
  
  <p>Then, Array[a][b] would mean how many times did symbol b going after symbol a has occurred in a stream.</p>
  
  <p>Now, i understand that array in C is a pointer that is incremented to get exact value, like to get element [3][4] from array[10][10] i have to increment pointer to array[0][0] by (3*10+4)(size of variable stored in array). I understand that the problem must be that 2^32  elements of type unsigned long must be taking too much. </p>
  
  <p>But still, is there a way to deal with it?</p>
  
  <p>Or maybe there is another way to accomplish this?</p>