title: Quickest way to store dataframes containing list fields
body: <p>I'm working with a pandas dataframe that contains 3 fields:</p>
  <ul>
  <li>Id field (ex. 1234)</li>
  <li>text field (ex. &quot;some text&quot;)</li>
  <li>list field (ex. ['som', 'ome', 'tex', 'ext'])</li>
  </ul>
  <p>When using <code>pd.DataFrame.to_csv()</code>, the list column is stored as a literal and must be re-converted when loading back into memory. Because the table has over 3 million rows, this takes upwards of 30 min when using the loading process <code>pd.read_csv(path, encoding='latin-1', converters={'list_field':eval})</code>.</p>
  <p>I have tried using pandas' <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.explode.html" rel="nofollow noreferrer">explode</a> function to give each list element its own row, save the exploded table, and then aggregate using <code>pd.DataFrame.group_by()</code> upon loading, but this seemed to take significantly longer.</p>
  <p>I have also considered <code>json.dump() / load()</code> or <code>pickle</code> but my understanding is that these packages also store lists as literals and must be converted upon loading.</p>
  <p>Is there an efficient/quick way of doing this <strong>to reduce the load time</strong> of the dataframe?</p>