title: How do i merge (i.e 'concat') 100+ .csv files using the pandas module?
body: <p>I'm a newbie to data science with python. So, I wanted to play around with the following data "<a href="https://www.ssa.gov/OACT/babynames/limits.html" rel="nofollow noreferrer">https://www.ssa.gov/OACT/babynames/limits.html</a>." The main problem here is that instead of giving me one file containing the data for all years, it contains a separate file for each year. Furthermore, each separate file also lacks column headings. </p>
  
  <p>FYI, the data contains the names, genders and some identification number of all registered US citizens from 1910 onwards. The data is available to the public (intended to aid demographers tracking trends in popular names). </p>
  
  <p>Thus, one major problem I'm facing is the need to edit more than 100 files directly (manually, open each and edit) so as to ensure that all column headings are the same (which is required for a function like concat to work).</p>
  
  <p>Another big problem is the sheer magnitude of the task. It's very, very inefficient to use concat for 100* files, as well as use up more than 100 lines of code in just scanning/reading your data</p>
  
  <p>Of course, 'concat' was built for this, but I think it's quite inefficient to use it for around 130 files. Regarding the missing column headings, I've manually edited some files, but there are just too many to be edited directly.</p>
  
  <pre><code>names2010 = pd.read_csv("../yob2010.txt")
  names2011 = pd.read_csv("../yob2011.txt")
  names = pd.concat([names2010, names2011])
  </code></pre>
  
  <p>intuitively, this is what I want to avoid>   </p>
  
  <pre><code>#rough notation 
  names = pd.concat([names1910, names1911 ..., names2017, names2018])
  </code></pre>
  
  <p>this is just for two years' worth of data. I need to create a single data frame consisting of all data from the years 1910 to 2018.</p>
  
  <p>update: I've figured out how to combine all different .txt files, but still need to resolve for column headings.</p>
  
  <pre><code>dataframes = pd.read_csv("../yob1910.txt")
  for year in range(1911, 2019):
         temp_frame = pd.read_csv("../yob{}.txt".format(year))
         dataframes = pd.concat([temp_frame, dataframes])
  </code></pre>