title: Flask Session (filesystem) UnPicklingError on session objects, is there a size limitation or am I getting corruptions?
body: <p>Python2.7 / Centos7.5 / Apache2.4.6 + mod_wsgi</p>
  <p>I'm using flask/jinja to generate some interactive html forms and saving data to the session file between changes (flask_session / type=filesystem). Its randomly breaking with an Unpickling error during the _prune function of werkzeug's contrib/cache.py and I'm at a loss for why. Deleting the session files would fix the issue until it cropped up again. I've since been able to reproduce the error, by forcing the session file to grow in size (making the form longer), so I suspect the size is related, but it doesn't entirely explain why they were happening in the first place.
  <strong>Is there some size limitations of server-side session files / pickles that I'm running into?</strong>
  It's not perfectly predictable, I just know that when I start overloading things it will eventually happen. AFAIK nothing is being sent over to client-side except the jinja-rendered html.</p>
  <p>I load a decent-sized OrderedDict as a session item:</p>
  <pre class="lang-py prettyprint-override"><code>session['saved'] = OrderedDict(items_list)
  </code></pre>
  <p>The depth of the dict is only 3 levels, with a total of ~ 300 keys when it runs the risk of crashing.</p>
  <p>I've tried changing the session items threshold, changing the permanence of the session file, updating werkzeug, and I have a workaround by forcing deletion of the session file on the UnpicklingError (which doesn't seem to affect the client-side experience at all). But that's a manual edit of the werkzeug lib file and is definitely not a long-term solution.</p>
  <pre class="lang-py prettyprint-override"><code>Traceback (most recent call last):
  File &quot;/usr/lib64/python2.7/site-packages/flask/app.py&quot;, line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File &quot;/usr/lib64/python2.7/site-packages/flask/app.py&quot;, line 1816, in full_dispatch_request
    return self.finalize_request(rv)
  File &quot;/usr/lib64/python2.7/site-packages/flask/app.py&quot;, line 1833, in finalize_request
    response = self.process_response(response)
  File &quot;/usr/lib64/python2.7/site-packages/flask/app.py&quot;, line 2114, in process_response
    self.session_interface.save_session(self, ctx.session, response)
  File &quot;/usr/lib64/python2.7/site-packages/flask_session/sessions.py&quot;, line 355, in save_session
    total_seconds(app.permanent_session_lifetime))
  File &quot;/usr/lib/python2.7/site-packages/werkzeug/contrib/cache.py&quot;, line 815, in set
    self._prune()
  File &quot;/usr/lib/python2.7/site-packages/werkzeug/contrib/cache.py&quot;, line 764, in _prune
    expires = pickle.load(f)
  UnpicklingError: invalid load key, '*'.
  </code></pre>
  <h3>Relevant python code (trimmed and modified for readability):</h3>
  <pre class="lang-py prettyprint-override"><code>from flask import Flask, session, request, render_template
  from flask_session import Session
  from collections import OrderedDict
  app = Flask(__name__)
  SESSION_TYPE = app.config['SESSION_TYPE'] = 'filesystem'
  SESSION_FILE_DIR = app.config['SESSION_FILE_DIR'] = os.path.join(local_dir,'flask_session')
  SESSION_FILE_THRESHOLD = app.config['SESSION_FILE_THRESHOLD'] = 100
  Session.init_app(app)
  
  @app.route('/', defaults={'path': ''}, methods=['POST','GET'])
  @app.route('/&lt;path:path&gt;', methods=['POST','GET'])
  def configuration(path):
      #Do Some Stuff
      session['saved'] = OrderedDict(items_list)
      return render_template('configuration.html', session=session)
  </code></pre>
  <h3>Relevant jinja</h3>
  <pre><code>% for element, details in session.saved.items()
      &lt;tr id=&quot;{{element}}&quot; name=&quot;{{details['type']}}&quot;&gt;
              &lt;td&gt;
          &lt;select id=&quot;{{element}}&quot; name=&quot;{{element}}&quot;&gt;
              % for item, desc in details[&quot;items&quot;]
                  &lt;option value=&quot;{{item}}&quot;&gt;{{desc}}&lt;/option&gt;
              % endfor
          &lt;/select&gt;
          &lt;/td&gt;
          &lt;/tr&gt;
  </code></pre>
  <h3>_prune function from werkzeug/contrib/cache.py for convenience</h3>
  <pre class="lang-py prettyprint-override"><code> def _prune(self):
          if self._threshold == 0 or not self._file_count &gt; self._threshold:
              return
  
          entries = self._list_dir()
          now = time()
          for idx, fname in enumerate(entries):
              try:
                  remove = False
                  with open(fname, &quot;rb&quot;) as f:
                      expires = pickle.load(f)
                  remove = (expires != 0 and expires &lt;= now) or idx % 3 == 0
  
                  if remove:
                      os.remove(fname)
              except (IOError, OSError):
                  pass
  # Add Exception to delete file with pickle errors
              except pickle.UnpicklingError:
                  os.remove(fname)
          self._update_count(value=len(self._list_dir()))
  </code></pre>
  <p>I don't see it being relevant, considering it should all be server side, but running the same scripts in a Windows environment with the flask development server doesn't present the pickling error. Instead I get a socket error in self.flush() - An established connection was aborted by the software in your host machine. There's no crash either, it just carries on. I suspect that's just error handling, but thought it was worth mentioning.</p>