title: Is there a way to pass an open file to pandas read_csv
body: <p>I have a csv file that is too large to read into memory all at once (~46 GB), and I want a subset of that data which can fit into memory.  I am able to read in part of the file, filter out what I do not want by doing an inner join with a previously created filtering index, and then continue on to the next part of the file like below:</p>
  
  <pre><code>    with open('Filtered_By_Month/all_data.csv','r') as f:
          cols=np.array(f.readline().strip('\n').split(','))#column names
          data=[]
          df=None
          for i in f:
              data.append(i.strip('\n').split(',')
              if len(data)==1000000:
                  df_sub=pd.DataFrame(data,columns=cols).set_index('KEY_')
                  df_sub=df_sub.join(filtered_keys,how='inner')
                  data=[]
                  if df is None:
                      df=df_sub
                  else:
                      df=pd.concat([df,df_sub])
  </code></pre>
  
  <p>This seems to do the trick except it is extremely slow.  An alternative would be to use pandas read_csv function along with the nrows and skip_rows arguments, however this becomes really slow as well since each time I call the read_csv method, the file is reopened and the cursor is at the first line, and I have to iterate through the lines until I have gone past skip_rows.  Is there a way to combine both of these methods so that I pass a file to read_csv that is already open and thus I do not need to have the cursor start all over again.</p>