title: Reading text files from subfolders and folders and creating a dataframe in pandas for each file text as one observation
body: <p>I have the following architecture of the text files in the folders and subfolders.</p>
  
  <p>I want to read them all and create a df. I am using this code, but it dont work well for me as the text is not what I checked and the files are not equivalent to my counting.</p>
  
  <p><a href="https://i.stack.imgur.com/6kqy1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6kqy1.png" alt="folders- level 1"></a>
  <a href="https://i.stack.imgur.com/bSP1K.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bSP1K.png" alt="files in first folder and so on"></a></p>
  
  <pre><code>l = [pd.read_csv(filename,header=None, encoding='iso-8859-1') for filename in glob.glob("2018_01_01/*.txt")]
  main_df = pd.concat(l, axis=1)
  main_df = main_df.T
  for i in range(2):
      l = [pd.read_csv(filename, header=None, encoding='iso-8859-1',quoting=csv.QUOTE_NONE) for filename in glob.glob(str(foldernames[i+1])+ '/' + '*.txt')]
      df = pd.concat(l, axis=1)
      df = df.T
      main_df = pd.merge(main_df, df)
  </code></pre>
  
  <p><a href="https://drive.google.com/open?id=1qiaggsLDNJfLlz45cURN1q7XMCkadQBQ" rel="nofollow noreferrer">file</a></p>