title: How to merge two large Pandas dataframes based on their ID - by reading the biggest file through chunks?
body: <p>I have two CSV files, df1_50GB.csv and df2_1GB.csv. I would like to merge the data frames based on the matched ID value. 
  here is what I did so far</p>
  
  <pre><code>f50GB =  "df1_50GB.csv"
  f1GGB =  "df1_1GGB.csv"
  result_file =  "output.csv"
  
  tmp50GB = pd.read_csv(f50GB, nrows=3)
  tmp1GB = pd.read_csv(f1GGB, nrows=3)
  
  # creating an empty bucket to save result
  df_result = pd.DataFrame(columns=(tmp50GB.columns.append(tmp1GB.columns)).unique())
  df_result.to_csv(result_file, index_label=False)
  tmp1GB = pd.read_csv(f1GGB)
  # Save (append) data which the ID of f1GB  mached with f50GB i.e (x)  #
  def save_merged_csv(x):
      df = pd.merge(x, tmp1GB, on='ID', how='left')
      df.to_csv(result_file, mode="a", header=False, index=False)
  
  
  # create the iterator
  csv_reader = pd.read_csv(
      f50GB,
      iterator=True,
      chunksize=100000)
  
  [save_merged_csv(r) for r in csv_reader]
  </code></pre>
  
  <p>But I keep getting errors 'TypeError: Can only merge Series or DataFrame objects, a  was passed'
  [UPDATE] The error is now fixed. Thanks to @Michael Gardner. But I am not sure if the logic is correct. The file size of the result is too big. It is also very slow. So,
   I will leave it here if there is room to improve. </p>