Take the 2-minute tour ×

I frequently run into the problem, that I have arrays of simple (POD) objects and need to save it to the disk, search the contents by one column and retrieve it.

Let's take a simple cache as example:

id: Integer
expires: Date
query: String
result: String

The common operations are saving an object (#1), retrieving an object by id (#2) and deleting everything after a specific date (#3), without using a DSL that has to be parsed at runtime (#4).
Since I know the format my data has I don't need any support for storing arbitrary documents (#5).

This should be common enough to lead to a flood of libraries doing just this, but all I'm seeing is key-value-stores like bdb, tokyocabinet, etc. (which don't work, because of #3), full-fletched SQL-databases including SQLite, MySQL etc. (#4), and completely schemaless databases like CouchDB, MongoDB and so on (#5). Storing it as plain CSV/XML/JSON works reasonably well, but doesn't do #2 and #3 well.

I'm looking for something like Boost's Multi-Index (but using the disk as storage) or Squeryl but using a native backend instead of a glorified DSL-to-SQL-Compiler.
Is there anything like that or am I damned to either parse CSV by hand or write massive amounts of boilerplate just to get to use the benefits of a SQL-database?

share|improve this question
Sorry to ask but why aren't you just using a SQL database for this? You really don't gain much by shaving off the query parsing step, which is the only overhead it'd have over something non-SQL... benchmarks have shown raw InnoDB engine performance comparable to MongoDB/CouchDB/etc. –  Dan Grossman Feb 18 '11 at 15:20
Sure, I could use a SQL database, but a simple table-db should be a common enough requirement. –  tstenner Feb 19 '11 at 10:08
add comment

2 Answers

It sounds like something that is typically done in a normal SQL database. If you create a good factory method to save and load these objects, each of those factories (if you got more than one object type) needs to implement only two queries (save and load) and maybe a third to update. If your objects map directly to table records, you can even create a base object that just maps properties to field names and generates these queries for you. Since almost every language has libraries for almost every database, you can set this up within a day, even if you don't use one of the many DBO features that many languages support. A simple implementation will probably take less than 200 lines of code and is probably already more powerful than what you're doing now with CSVs.

share|improve this answer
add comment

MongoDB (and probably other noSQL databases) allow you to have more than one index on a table. But even with key/value type systems there's nothing to stop you creating a second table using the date as the key and referencing the key from the data table as the payload and using that as an index.

However unless you've got very specific reasons for using a non-relational database, I would recommend sticking with a relational database - sooner or later you're going to have to start debugging your data / analysing patterns, handling more complex data models etc.

share|improve this answer
I considered using 2 KV-stores (one for the ID an one for the other index), but as you said it makes debugging a real pain. –  tstenner Feb 19 '11 at 10:12
add comment

Your Answer


