title: Python Panda Reading Embedded JSON File
body: <p>I have a csv file named people.</p>
  <p>The structure is like below:</p>
  <p>people.csv</p>
  <pre><code>&quot;id&quot;,&quot;name&quot;,&quot;age&quot;,&quot;mate&quot;
  ------------------------
  'a1234','Bob',26,&quot;file1234.json&quot;
  'a2345','Allen',45,&quot;file2345.json&quot;
  'a3456','Kim',18,&quot;file3456.json&quot;
  'a4567','Peter',39,&quot;file4567.json&quot;
  'a5678','Tim',55,&quot;file5678.json&quot;
  #....many more similar records....
  </code></pre>
  <p>JSON file Sample:</p>
  <p><strong>file1234.json</strong></p>
  <pre class="lang-json prettyprint-override"><code>{
      &quot;name&quot;: &quot;Bob&quot;,
      &quot;id&quot;: &quot;a1234&quot;,
      &quot;age&quot;: 26,
      &quot;familymember&quot;: [
          {
              &quot;name&quot;: &quot;Tim&quot;,
              &quot;id&quot;: &quot;a5678&quot;,
              &quot;parentJson&quot;: &quot;file5678.json&quot;,
              &quot;age&quot;: 55
          },
          {
              &quot;name&quot;: &quot;Joe&quot;,
              &quot;id&quot;: &quot;a9876&quot;,
              &quot;parentJson&quot;: &quot;file9876.json&quot;,
              &quot;age&quot;: 19
          }
      ]
  }
  </code></pre>
  <p><strong>file5678.json</strong></p>
  <pre class="lang-json prettyprint-override"><code>{
      &quot;name&quot;: &quot;Tim&quot;,
      &quot;id&quot;: &quot;a5678&quot;,
      &quot;age&quot;: 55,
      &quot;familymember&quot;: [
          {
              &quot;name&quot;: &quot;Bob&quot;,
              &quot;id&quot;: &quot;a1234&quot;,
              &quot;parentJson&quot;: &quot;file1234.json&quot;,
              &quot;age&quot;: 22
          },
          {
              &quot;name&quot;: &quot;Joe&quot;,
              &quot;id&quot;: &quot;a9876&quot;,
              &quot;parentJson&quot;: &quot;file9876.json&quot;,
              &quot;age&quot;: 20
          }
      ]
  }
  </code></pre>
  <p>The goal for me is to find the inconsistent ages among the JSON files. For example: In the original file, Bob is 26, file1234 indicates Bob is 26, however, in the file5678, Bob is 22, which is inconsistent. The second example is Joe, in the file1234, Joe is 22 and in file5678, Joe is 20. (id is the unique key for everyone)</p>
  <p>The idea I think is process all the json files and generate something like:</p>
  <pre><code>d=collections.defaultdict(set)
  d[id].add(age)
  
  res=[]
  for k,v in d.items():
      if len(d[k])&gt;1:
          res.append(k)
  return res
  </code></pre>
  <p>The challenging part is really how to flatten all the JSON files and save them into a dictionary type or Panda data frame given the assumption that you have unlimited CPU and able to parallel processing as much as possible so all JSON files can be processed in parallel.</p>
  <p>What will be an efficient and elegant way to process JSON in JSON and in Parallel.</p>