title: maintaining an on-disk concurrent hash table
body: <p>What are some options for maintaining a simple mapping of key/value pairs on disk for use by a large number of concurrent processes that are both reading from and writing to the table?  To give a sense of scale, the table has on the order of 10 million entries (mapping strings to numbers) and is accessed by about 10,000 simultaneous processes distributed across a network that share an underlying file system.  Each process can either read or write, and writes can take the form of modification of existing key/value pairs or insertion of new ones.  A single transaction typically involves 1 to 1000 key/value pairs (but all pairs are independent).  Atomic operations and consistency of transactions are essential.  I've implemented this with a "pile of files" approach and file locking (flock on linux), but this does not have the best scaling characteristics.  Sqlite does not seem to be a viable option given the number of concurrent writers and the separation of processes from data by a network, and a full relational database management system seems like overkill given that the queries are never more complicated than extracting the value of a given key (or the values of a set of keys).  I want to avoid re-inventing the wheel if there is already a solution designed for this scenario.  Reads and writes should be as fast as possible (for the given hardware) with locking done at the smallest scale possible (or practical) to minimize queueing/blocking of other processes.  Occasionally (in a non-concurrent mode), a large portion of the table needs to be rewritten (deletion, insertion, and modification of millions of key/value pairs) and these large-scale updates should also be as fast as possible.</p>