title: Live updating graph from increasing amount of csv files
body: <p>I need to analyse some spectral data in real-time and plot it as a self-updating graph.
  The program I use outputs a text file every two seconds.</p>
  <p>Usually I do the analysis after gathering the data and the code works just fine. I create a dataframe, where each csv file represents a column. The problem is, with several thousands of csv files, the import becomes very slow and creating a dataframe out of all the csv files takes usually more than half an hour.
  Below the code for creating the dataframe from multiple csv files.</p>
  <pre><code>    ''' import, append and concat files into one dataframe '''
      all_files = glob.glob(os.path.join(path, filter + &quot;*.txt&quot;)) # path to the files by joining path and file name
      all_files.sort(key=os.path.getmtime)
      data_frame = []
      name = []
      for file in (all_files):
          creation_time = os.path.getmtime(file)
          readible_date = datetime.fromtimestamp(creation_time)
          df = pd.read_csv(file, index_col=0, header=None, sep='\t', engine='python', decimal=&quot;,&quot;, skiprows = 15)
          df.rename(columns={1: readible_date}, inplace=True)
          data_frame.append(df)
          full_spectra = pd.concat(data_frame, axis=1)
      for column in full_spectra.columns:
          time_step = column - full_spectra.columns[0]
          minutes = time_step.total_seconds()/60
          name.append(minutes)    
      full_spectra.columns = name
      return full_spectra
  </code></pre>
  <p>The solution I thought of was using the watchdog module and everytime a new textfile is created it gets appended as a new column to the existing dataframe and the updated dataframe is plotted. Because then, I do not need to loop over all csv files all the time.</p>
  <p><a href="https://gist.github.com/mastro35/0c433bb13d0efa347ce0fbf8ec1cb225" rel="nofollow noreferrer">I found a very nice example on how to use watchdog here</a></p>
  <p>My problem is, I could not find a solution how after detecting the new file with watchdog, to read it and append it to the existing dataframe.</p>
  <p>A minimalistic example code should look something like this:</p>
  <pre><code>def latest_filename():
  &quot;&quot;&quot;a function that checks within a directoy for new textfiles&quot;&quot;&quot;
      return(filename)
  
  df = pd.DataFrame() #create a dataframe
  
  newdata  = pd.read_csv(latest_filename) #The new file is found by watchdog
  
  df[&quot;newcolumn&quot;] = newdata[&quot;desiredcolumn&quot;] #append the new data as column
  
  df.plot() #plot the data
  
  </code></pre>
  <p>The plotting part should be easy and my thoughts were to <a href="https://pythonprogramming.net/live-graphs-matplotlib-tutorial/" rel="nofollow noreferrer">adapt the code presented here</a>. I am more concerned with the self-updating dataframe.</p>
  <p>I appreciate any help or other solutions that would solve my issue!</p>