title: Concatenating multiple csv files into a single csv with the same header - Python
body: <p>I am currently using the below code to import 6,000 csv files (with headers) and export them into a single csv file (with a single header row).</p>
  
  <pre><code>#import csv files from folder
  path =r'data/US/market/merged_data'
  allFiles = glob.glob(path + "/*.csv")
  stockstats_data = pd.DataFrame()
  list_ = []
  
  for file_ in allFiles:
      df = pd.read_csv(file_,index_col=None,)
      list_.append(df)
      stockstats_data = pd.concat(list_)
      print(file_ + " has been imported.")
  </code></pre>
  
  <p>This code works fine, but it is slow. It can take up to 2 days to process. </p>
  
  <p>I was given a single line script for Terminal command line that does the same (but with no headers). This script takes 20 seconds.</p>
  
  <pre><code> for f in *.csv; do cat "`pwd`/$f" | tail -n +2 &gt;&gt; merged.csv; done 
  </code></pre>
  
  <p>Does anyone know how I can speed up the first Python script? To cut the time down, I have thought about not importing it into a DataFrame and just concatenating the CSVs, but I cannot figure it out. </p>
  
  <p>Thanks.</p>