title: Flattening nested JSON included embedded array in Python using Pandas
body: <p>I have a JSON-array from a <em>mongoexport</em> containing data from the Beddit sleeptracker. Below is an example of one of the truncated documents (removed some unneeded detail). </p>
  
  <pre><code>    {
          "user" : "xxx",
          "provider" : "beddit",
          "date" : ISODate("2016-11-30T23:00:00.000Z"),
          "data" : [ 
              {
                  "end_timestamp" : 1480570804.26226,
                  "properties" : {
                      "sleep_efficiency" : 0.8772404,
                      "resting_heart_rate" : 67.67578,
                      "short_term_resting_heart_rate" : 61.36963,
                      "activity_index" : 50.51958,
                      "average_respiration_rate" : 16.25667,
                      "total_sleep_score" : 64,
                  },
                  "date" : "2016-12-01",
                  "session_range_start" : 1480545636.55059,
                  "start_timestamp" : 1480545636.55059,
                  "session_range_end" : 1480570804.26226,
                  "tags" : [ 
                      "not_enough_sleep", 
                      "long_sleep_latency"
                  ],
                  "updated" : 1480570805.25201
              }
          ],
          "__v" : 0
      }
  </code></pre>
  
  <p>Several related questions like <a href="https://stackoverflow.com/questions/40146043/parsing-nested-json-into-multiple-dataframe-using-pandas-python">this</a> and <a href="https://stackoverflow.com/questions/40702633/extract-nested-json-in-pandas-dataframe?rq=1">this</a> do not seem to work for the data structure above. As recommended in other related questions I am trying to stay away from looping over each row for performance reasons (the full dataset is ~150MB). How would I flatten out the <em>"data"</em>-key with json_normalize so that each <em>key</em> is at the top-level? I would prefer one DataFrame where e.g. <em>total_sleep_score</em> is a column. </p>
  
  <p>Any help is much appreciated! Even though I know how to 'prepare' the data using JavaScript, I would like to be able to understand and do it using Python. </p>
  
  <p>edit (request from comment to show preferred structure): </p>
  
  <pre><code>{
      "user" : "xxx",
      "provider" : "beddit",
      "date" : ISODate("2016-11-30T23:00:00.000Z"),
      "end_timestamp" : 1480570804.26226,
      "properties.sleep_efficiency" : 0.8772404,
      "properties.resting_heart_rate" : 67.67578,
      "properties.short_term_resting_heart_rate" : 61.36963,
      "properties.activity_index" : 50.51958,
      "properties.average_respiration_rate" : 16.25667,
      "properties.total_sleep_score" : 64,
      "date" : "2016-12-01",
      "session_range_start" : 1480545636.55059,
      "start_timestamp" : 1480545636.55059,
      "session_range_end" : 1480570804.26226,
      "updated" : 1480570805.25201,
      "__v" : 0
  }
  </code></pre>
  
  <p>The 'properties' append is not necessary but would be nice.</p>