title: Counting csv column occurrences on the fly in Python
body: <p>I have a csv file that has 200 million lines in it. </p>
  
  <p>The best way to load in this file is line by line using a csv reader (as I have many of these files and so parallelising the code later requires not loading in massive datasets and overloading RAM). </p>
  
  <p>I'm trying to count the number of occurrences of values in a certain column and record both their values and frequency in a dictionary. For example, count the number of unique IDs in a column and the number of times those IDs appear. </p>
  
  <p>Here is an example of how I would do this:</p>
  
  <pre><code>import csv
  from tqdm import tqdm
  
  field_names = ['A','B','ID','C','D']
  filename = '/PATH/file'
  
  ID_dict = {}
  with open(data_path+filename) as f:
      reader = csv.DictReader(f,field_names,delimiter=',')
      for row in tqdm(reader):
          label = row['ID']
          if label not in ID_dict.keys():
                  ID_dict[label] = 0
          ID_dict[label] += 1
  </code></pre>
  
  <p>So the column labelled 'ID' is what I'm interested in here, but imagine it has ~200 million entries in it.  </p>
  
  <p>Iterating through all those lines and populating a dictionary is slow (it would take around 10 hours on my machine). </p>
  
  <p>Alternatively, appending the values to a new array and then using Counter to find the number of appearances of each unique element also takes too long. (see <a href="https://stackoverflow.com/questions/12282232/how-do-i-count-unique-values-inside-a-list">How do I count unique values inside a list</a>)</p>
  
  <p>Is there a much faster way of doing this that I'm missing? Perhaps there is a Pandas way that is faster?
  Thanks in advance </p>