title: Splitting csv file based on a particular column using Python
body: <p>I'm a Python beginner, and have made a few basic scripts. My latest challenge is to take a very large csv file (10gb+) and split it into a number of smaller files, based on the value of a particular variable in each row.</p>
  
  <p>For example, the file may look like this:</p>
  
  <pre><code>Category,Title,Sales
  "Books","Harry Potter",1441556
  "Books","Lord of the Rings",14251154
  "Series", "Breaking Bad",6246234
  "Books","The Alchemist",12562166
  "Movie","Inception",1573437
  </code></pre>
  
  <p>And I would want to split the file into separate files:
  Books.csv, Series.csv, Movie.csv</p>
  
  <p>In reality there will be hundreds of categories, and they will not be sorted. In this case they are in the first column but in future they may not be.</p>
  
  <p>I've found a few solutions online but nothing in Python. There is a really simple AWK command that can do this in one line, but I cannot get access to AWK in work.</p>
  
  <p>I've written the following code which works, but I think it is probably very inefficient. Can anybody suggest how to speed it up?</p>
  
  <pre><code>import csv
  
  #Creates empty set - this will be used to store the values that have already been used
  filelist = set()
  
  #Opens the large csv file in "read" mode
  with open('//directory/largefile', 'r') as csvfile:
  
      #Read the first row of the large file and store the whole row as a string (headerstring)
      read_rows = csv.reader(csvfile)
      headerrow = next(read_rows)
      headerstring=','.join(headerrow) 
  
      for row in read_rows:
  
          #Store the whole row as a string (rowstring)
          rowstring=','.join(row)
  
          #Defines filename as the first entry in the row - This could be made dynamic so that the user inputs a column name to use
          filename = (row[0])
  
          #This basically makes sure it is not looking at the header row.
          if filename != "Category":
  
              #If the filename is not in the filelist set, add it to the list and create new csv file with header row.
              if filename not in filelist:    
                  filelist.add(filename)
                  with open('//directory/subfiles/' +str(filename)+'.csv','a') as f:
                      f.write(headerstring)
                      f.write("\n")
                      f.close()    
              #If the filename is in the filelist set, append the current row to the existing csv file.     
              else:
                  with open('//directory/subfiles/' +str(filename)+'.csv','a') as f:
                      f.write(rowstring)
                      f.write("\n")
                      f.close()
  </code></pre>
  
  <p>Thanks!</p>