Take the 2-minute tour ×

I.e. allowing you to efficiently access and update data as you would in a database.

My specific situation is the following:

I have a very large Monte Carlo simulation + optimization and the data is sorted into a few dictionaries with 600k+ tuple keys each, formatted like this:

simple_dictionary[(year, month, day, hour, minute)] = value_or_small_list_or_small_numpy_array

The above, it turns out, is actually slightly slower than nested dictionaries:

simple_dictionary[year][month][day][hour][minute] = value_or_small_list_or_small_numpy_array

This leads me to believe - and please correct me if I'm wrong - that the latter format checks less keys than the former. This assumption is based on my code (which is too long to post). My code does not create new keys after __init__, but has every key/value pair called at least once each iterations. However, not all values are updated.

Assuming this can/should run from RAM, is there an alternative to the above code that is more efficient?

share|improve this question
What do you mean by 'checks less keys' here? –  Lattyware Dec 3 '12 at 19:36
I'm sorry, my knowledge is fairly limited as I am not a computer scientist, but I assumed Python had to do some internal processing before it knew the exact location of the data in the RAM. –  TimY Dec 3 '12 at 19:40

1 Answer 1

up vote 4 down vote accepted

Have you thought about trying an in-memory sqlite3 database? That would be about as "similar to a database" as you can get, and it's all in memory. :)

Using the connection string :memory: gives you an in memory database.

share|improve this answer
This is what I used. There's a python sqllite3 library so just use it. docs.python.org/2/library/sqlite3.html –  Dean Toader Dec 3 '12 at 21:47
@DragosToader, Did you get a chance to compare? If so, did you get much better results? –  TimY Dec 3 '12 at 22:05
The other great thing about using sqllite3 is that if the dataset gets too big, you could just connect without the :memory: connection param. –  Dean Toader Dec 3 '12 at 23:49
I loaded a complete Windows Registry export of keys/values first as a dictionary, second as a sqllite3 db. The sqllite3 db was faster from what I remember. I had to use SQL to access the data though -- which was better in my case. It got even faster after I created a composite index on my (key,value) key. –  Dean Toader Dec 3 '12 at 23:54
Interesting - I'll have a look and see if this is applicable for my application. Thank you! –  TimY Dec 4 '12 at 9:48

Your Answer


