title: Unable to load a previously dumped pickle file of large size in Python
body: <p>I used cPickle and protocol version 2 to dump some computation results.  The code looks like this:</p>
  
  <pre><code>&gt; f = open('foo.pck', 'w')
  &gt; cPickle.dump(var, f, protocol=2)
  &gt; f.close()
  </code></pre>
  
  <p>The variable var is a tuple of length two.  The type of var[0] is a list and var[1] is a numpy.ndarray.</p>
  
  <p>The above code segment successfully generated a file with large size (~1.7G).</p>
  
  <p>However, when I tried to load the variable from foo.pck, I got the following error.</p>
  
  <pre><code>ValueError                                Traceback (most recent call last)
  /home/user_account/tmp/&lt;ipython-input-3-fd3ecce18dcd&gt; in &lt;module&gt;()
  ----&gt; 1 v = cPickle.load(f)
  ValueError: buffer size does not match array size
  </code></pre>
  
  <p>The loading codes looks like the following.</p>
  
  <pre><code>&gt; f= open('foo.pck', 'r')
  &gt; v = cPickle.load(f)
  </code></pre>
  
  <p>I also tried to use pickle (instead of cPickle) to load the variable, but got a similar error msg as follows.</p>
  
  <pre><code>ValueError                                Traceback (most recent call last)
  /home/user_account/tmp/&lt;ipython-input-3-aa6586c8e4bf&gt; in &lt;module&gt;()
  ----&gt; 1 v = pickle.load(f)
  
  /usr/lib64/python2.6/pickle.pyc in load(file)
     1368 
     1369 def load(file):
  -&gt; 1370     return Unpickler(file).load()
     1371 
     1372 def loads(str):
  
  /usr/lib64/python2.6/pickle.pyc in load(self)
      856             while 1:
      857                 key = read(1)
  --&gt; 858                 dispatch[key](self)
      859         except _Stop, stopinst:
      860             return stopinst.value
  
  /usr/lib64/python2.6/pickle.pyc in load_build(self)
     1215         setstate = getattr(inst, "__setstate__", None)
     1216         if setstate:
  -&gt; 1217             setstate(state)
     1218             return
     1219         slotstate = None
  
  ValueError: buffer size does not match array size
  </code></pre>
  
  <p>I tried the same code segments to a much smaller size data and it worked fine.  So my best guess is that I reached the loading size limitation of pickle (or cPickle).  However, it is strange to dump successfully (with large size variable) but failed to load.</p>
  
  <p>If this is indeed a loading size limitation problem, how should I bypass it? If not, what can be the possible cause of the problem?  </p>
  
  <p>Any suggestion is appreciated.  Thanks!</p>