title: Dataframe from complex data structure
body: <p>Say I need to have data stored as follows:</p>
  
  <pre><code>[[[{}][{}]]]
  </code></pre>
  
  <p>or a list of lists of two lists of dictionaries</p>
  
  <p>where:</p>
  
  <p><code>{}</code>: dictionaries containing data from individual frames observing an event. (There are two observers/stations, hence two dictionaries.)</p>
  
  <p><code>[{}][{}]</code>: two lists of all the individual frames related to a single event, one from each observer/station.</p>
  
  <p><code>[[{}][{}]]</code>: list of all events on a single night of observation.</p>
  
  <p><code>[[[{}][{}]]]</code>: list of all nights.</p>
  
  <p>Hopefully that's clear. What I want to do is create two pandas dataframes where all dictionaries from <code>station_1</code> are stored in one, and all dictionaries from <code>station_2</code> are stored in the other.</p>
  
  <p>My current method is as follows (where <code>data</code> is the above data structure):</p>
  
  <pre><code>for night in range(len(data)):
  
      station_1 = pd.DataFrame(data[night][0])
      station_2 = pd.DataFrame(data[night][1])
  
      all_station_1.append(station_1)
      all_station_2.append(station_2)
  
  all_station_1 = pd.concat(all_station_1)
  all_station_2 = pd.concat(all_station_2)
  </code></pre>
  
  <p>My understanding though is that the for loop must be horribly inefficient since I will be scaling the application of this script way up from my sample dataset this cost could easily become unmanageable.</p>
  
  <p>So, any advice for a smarter way of proceeding would be appreciated! I feel like pandas is so user friendly there's gotta be an efficient way of dealing with any kind of data structure but I haven't been able to find it on my own yet. Thanks!</p>