title: Key: value store in Python for possibly 100 GB of data, without client/server
body: <p>There are many solutions to serialize a small dictionary: <code>json.loads</code>/<code>json.dumps</code>, <code>pickle</code>, <code>shelve</code>, <code>ujson</code>, or even by using <code>sqlite</code>.</p>
  
  <p>But when dealing with possibly 100 GB of data, it's not possible anymore to use such modules that would possibly rewrite the whole data when closing / serializing.</p>
  
  <p><code>redis</code> is not really an option because it uses a client/server scheme.</p>
  
  <p>Question: <strong>Which key:value store, serverless, able to work with 100+ GB of data, are frequently used in Python?</strong></p>
  
  <p>I'm looking for a solution with a standard "Pythonic" <code>d[key] = value</code> syntax:</p>
  
  <pre><code>import mydb
  d = mydb.mydb('myfile.db')
  d['hello'] = 17          # able to use string or int or float as key
  d[183] = [12, 14, 24]    # able to store lists as values (will probably internally jsonify it?)
  d.flush()                # easy to flush on disk 
  </code></pre>
  
  <p>Note: <a href="https://docs.python.org/2/library/bsddb.html#module-bsddb" rel="noreferrer">BsdDB</a> (BerkeleyDB) seems to be deprecated. There seems to be a <a href="https://github.com/jtolds/leveldb-py" rel="noreferrer">LevelDB for Python</a>, but it doesn't seem well-known - and I <a href="https://github.com/google/leveldb/issues/528#issuecomment-347626773" rel="noreferrer">haven't found</a> a version which is ready to use on Windows. Which ones would be the most common ones?</p>
  
  <hr>
  
  <p>Linked questions: <a href="https://stackoverflow.com/questions/47237807/use-sqlite-as-a-keyvalue-store">Use SQLite as a key:value store</a>, <a href="https://stackoverflow.com/questions/61088235/flat-file-nosql-solution">Flat file NoSQL solution</a></p>