title: Extract list from dict of lists then append to dataframe
body: <p>I'm trying to extract a field from a json that contains a list then append that list to a dataframe, but I'm running in to a few different errors.</p>
  
  <p>I think I can write it to a csv then read the csv with Pandas, but I'm trying to avoid writing any files. I know that I can also use StringIO to make a csv, but that has issues with null bytes. Replacing those would be (i think) another line-by-line step that will further extend the time the script takes to complete... i'm running this against a query that returns thens of thousands of results so keeping it fast and simple is a priority</p>
  
  <p>First I tried this: </p>
  
  <pre><code>hit_json = json.loads(hit)
  for ln in hit_json.get('hits').get('hits'):
      df = df.append(ln['_source'], ignore_index=True)
  print(df)
  </code></pre>
  
  <p>This gives me a result that looks like this:</p>
  
  <pre><code>1    2           3      4 
  a    b    d,e,f...      x
  </code></pre>
  
  <p>Then I tried this:</p>
  
  <pre><code> df = df.append(ln['_source']['payload'], ignore_index=True)
  </code></pre>
  
  <p>But that gives me this error:</p>
  
  <pre><code>TypeError: cannot concatenate object of type "&lt;class 'str'&gt;"; only pd.Series, 
  pd.DataFrame, and pd.Panel (deprecated) objs are valid
  </code></pre>
  
  <p>What I'm looking for would be something like this:</p>
  
  <pre><code>0  1  2  3  4
  d  e  f  g  h
  </code></pre>
  
  <p>On top of this... I need to figure out a way to handle a specific string in this list that contains a comma... which may be a headache that's best handled in a different question... something like:</p>
  
  <pre><code># Obviously this is incorrect but I think you get the idea :)
  str.replace(',', '^')
      except if ',' followed by ' '
  </code></pre>
  
  <p>Greatly appreciate any help!  </p>
  
  <p><strong><em>EDITING TO ADD JSON AS REQUESTED</em></strong></p>
  
  <pre><code>{
  "_index": "sanitized",
  "_type": "sanitized",
  "_id": "sanitized".,
  "_score": sanitized,
  "_source": {
      "sanitized": sanitized,
      "sanitized": "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,\"34,35\",36,37,38,39,40",
      "sanitized": "sanitized",
      "sanitized": ["sanitized"],
      "sanitized": "sanitized",
      "sanitized": "sanitized",
      "sanitized": "sanitized",
      "sanitized": "sanitized",
  }
  }]
  }
  }
  </code></pre>