title: How to build (or precompute) a histogram from a file too large for memory?
body: <p>Is there a graphing library for python that doesn't require storing all raw data points as a <strong><code>numpy</code></strong> array or list in order to graph a histogram?  </p>
  
  <p>I have a dataset <strong>too large for memory</strong>, and I don't want to use subsampling to reduce the data size.</p>
  
  <p>What I'm looking for is a library that can take the output of a generator (each data point yielded from a file, as a <strong><code>float</code></strong>), and build a histogram on the fly.</p>
  
  <p>This includes computing bin size as the generator yields each data point from the file.</p>
  
  <p><strong>If such a library doesn't exist</strong>, I'd like to know if <code>numpy</code> is able to precompute a counter of <code>{bin_1:count_1, bin_2:count_2...bin_x:count_x}</code> from yielded datapoints.</p>
  
  <p>Datapoints are held as a vertical matrix, in a tab file, arranged by <code>node-node-score</code> like below:</p>
  
  <pre><code>node   node   5.55555
  </code></pre>
  
  <p><strong>More information:</strong></p>
  
  <ul>
  <li>104301133 lines in data (so far)</li>
  <li>I don't know the min or max values</li>
  <li>bin widths should be the the same</li>
  <li>number of bins could be 1000</li>
  </ul>
  
  <p><strong>Attempted Answer:</strong></p>
  
  <pre><code>low = np.inf
  high = -np.inf
  
  # find the overall min/max
  chunksize = 1000
  loop = 0
  for chunk in pd.read_table('gsl_test_1.txt', header=None, chunksize=chunksize, delimiter='\t'):
      low = np.minimum(chunk.iloc[:, 2].min(), low)
      high = np.maximum(chunk.iloc[:, 2].max(), high)
      loop += 1
  lines = loop*chunksize
  
  nbins = math.ceil(math.sqrt(lines))   
  
  bin_edges = np.linspace(low, high, nbins + 1)
  total = np.zeros(nbins, np.int64)  # np.ndarray filled with np.uint32 zeros, CHANGED TO int64
  
  
  # iterate over your dataset in chunks of 1000 lines (increase or decrease this
  # according to how much you can hold in memory)
  for chunk in pd.read_table('gsl_test_1.txt', header=None, chunksize=2, delimiter='\t'):
  
      # compute bin counts over the 3rd column
      subtotal, e = np.histogram(chunk.iloc[:, 2], bins=bin_edges)  # np.ndarray filled with np.int64
  
      # accumulate bin counts over chunks
      total += subtotal
  
  
  plt.hist(bin_edges[:-1], bins=bin_edges, weights=total)
  # plt.bar(np.arange(total.shape[0]), total, width=1)
  plt.savefig('gsl_test_hist.svg')
  </code></pre>
  
  <p><strong>Output:</strong>
  <a href="https://i.stack.imgur.com/IC1DT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IC1DT.png" alt="normal dist, mu=00, sigma=30"></a></p>