title: Pickle File too large to load
body: <p>The problem that I am having is that I have a very large pickle file (2.6 Gb) that I am trying to open but each time I do so I get a memory error. I realize now that I should have used a database to store all the information but its too late now. The pickle file contains dates and text from the U.S. Congressional record that was crawled from the internet (took about 2 weeks to run). </p>
  
  <p>Is there any way I can access the information that I dumped into the pickle file incrementally or a way to convert the pickle file into a SQL database or something else that I can open without having to re-input all the data. I really don't want to have to spend another 2 weeks re-crawling the congressional record and imputing the data into a database. </p>
  
  <p>Thanks a bunch for your help</p>
  
  <p>EDIT*</p>
  
  <p>code for how the object gets pickled:</p>
  
  <pre><code>def save_objects(objects): 
      with open('objects.pkl', 'wb') as output: 
          pickle.dump(objects, output, pickle.HIGHEST_PROTOCOL)
  
  def Main():   
      Links()
      file = open('datafile.txt', 'w')
      objects = []
      with open('links2.txt', 'rb') as infile:
          for link in infile: 
              print(link)
              title, text, date = Get_full_text(link)
              article=Doccument(title, date, text)
              if text != None:
                  write_to_text(date, text)
                  objects.append(article)
                  save_objects(objects)
  </code></pre>
  
  <p>This is the program with the error: </p>
  
  <pre><code>def Main():
      file = open('objects1.pkl', 'rb') 
      object = pickle.load(file)
  </code></pre>