title: Fast JSON serialization (and comparison with Pickle) for cluster computing in Python?
body: <p>I have a set of data points, each described by a dictionary.  The processing of each data point is independent and I submit each one as a separate job to a cluster.  Each data point has a unique name, and my cluster submission wrapper simply calls a script that takes a data point's name and a file describing all the data points.  That script then accesses the data point from the file and performs the computation.</p>
  
  <p>Since each job has to load the set of all points only to retrieve the point to be run, I wanted to optimize this step by serializing the file describing the set of points into an easily retrievable format. </p>
  
  <p>I tried using JSONpickle, using the following method, to serialize a dictionary describing all the data points to file:</p>
  
  <pre><code>def json_serialize(obj, filename, use_jsonpickle=True):
      f = open(filename, 'w')
      if use_jsonpickle:
      import jsonpickle
      json_obj = jsonpickle.encode(obj)
      f.write(json_obj)
      else:
      simplejson.dump(obj, f, indent=1)   
      f.close()
  </code></pre>
  
  <p>The dictionary contains very simple objects (lists, strings, floats, etc.) and has a total of 54,000 keys.  The json file is ~20 Megabytes in size.</p>
  
  <p>It takes ~20 seconds to load this file into memory, which seems very slow to me.  I switched to using pickle with the same exact object, and found that it generates a file that's about 7.8 megabytes in size, and can be loaded in ~1-2 seconds.  This is a significant improvement, but it still seems like loading of a small object (less than 100,000 entries) should be faster.  Aside from that, pickle is not human readable, which was the big advantage of JSON for me.</p>
  
  <p>Is there a way to use JSON to get similar or better speed ups?  If not, do you have other ideas on structuring this?</p>
  
  <p>(Is the right solution to simply "slice" the file describing each event into a separate file and pass that on to the script that runs a data point in a cluster job? It seems like that could lead to a proliferation of files).</p>
  
  <p>thanks.</p>