title: How to handle incoming real time data with python pandas
body: <p>Which is the most recommended/pythonic way of handling live incoming data with pandas?</p>
  <p>Every few seconds I'm receiving a data point in the format below:</p>
  <pre><code>{'time' :'2013-01-01 00:00:00', 'stock' : 'BLAH',
   'high' : 4.0, 'low' : 3.0, 'open' : 2.0, 'close' : 1.0}
  </code></pre>
  <p>I would like to append it to an existing DataFrame and then run some analysis on it.</p>
  <p>The problem is, just appending rows with DataFrame.append can lead to performance issues with all that copying.</p>
  <h3>Things I've tried:</h3>
  <p>A few people suggested preallocating a big DataFrame and updating it as data comes in:</p>
  <pre><code>In [1]: index = pd.DatetimeIndex(start='2013-01-01 00:00:00', freq='S', periods=5)
  
  In [2]: columns = ['high', 'low', 'open', 'close']
  
  In [3]: df = pd.DataFrame(index=t, columns=columns)
  
  In [4]: df
  Out[4]: 
                      high  low open close
  2013-01-01 00:00:00  NaN  NaN  NaN   NaN
  2013-01-01 00:00:01  NaN  NaN  NaN   NaN
  2013-01-01 00:00:02  NaN  NaN  NaN   NaN
  2013-01-01 00:00:03  NaN  NaN  NaN   NaN
  2013-01-01 00:00:04  NaN  NaN  NaN   NaN
  
  In [5]: data = {'time' :'2013-01-01 00:00:02', 'stock' : 'BLAH', 'high' : 4.0, 'low' : 3.0, 'open' : 2.0, 'close' : 1.0}
  
  In [6]: data_ = pd.Series(data)
  
  In [7]: df.loc[data['time']] = data_
  
  In [8]: df
  Out[8]: 
                      high  low open close
  2013-01-01 00:00:00  NaN  NaN  NaN   NaN
  2013-01-01 00:00:01  NaN  NaN  NaN   NaN
  2013-01-01 00:00:02    4    3    2     1
  2013-01-01 00:00:03  NaN  NaN  NaN   NaN
  2013-01-01 00:00:04  NaN  NaN  NaN   NaN
  </code></pre>
  <p>The other alternative is building a list of dicts. Simply appending the incoming data to a list and slicing it into smaller DataFrames to do the work.</p>
  <pre><code>In [9]: ls = []
  
  In [10]: for n in range(5):
     .....:     # Naive stuff ahead =)
     .....:     time = '2013-01-01 00:00:0' + str(n)
     .....:     d = {'time' : time, 'stock' : 'BLAH', 'high' : np.random.rand()*10, 'low' : np.random.rand()*10, 'open' : np.random.rand()*10, 'close' : np.random.rand()*10}
     .....:     ls.append(d)
  
  In [11]: df = pd.DataFrame(ls[1:3]).set_index('time')
  
  In [12]: df
  Out[12]: 
                          close      high       low      open stock
  time                                                             
  2013-01-01 00:00:01  3.270078  1.008289  7.486118  2.180683  BLAH
  2013-01-01 00:00:02  3.883586  2.215645  0.051799  2.310823  BLAH
  </code></pre>
  <p>or something like that, maybe processing the input a little bit more.</p>