Take the 2-minute tour ×

Possible Duplicate:
Is there a memory efficient and fast way to load big json files in python?

So I have some rather large json encoded files. The smallest is 300MB, but this is by far the smallest. The rest are multiple GB, anywhere from around 2GB to 10GB+.

So I seem to run out of memory when trying to load the file with Python. I'm currently just running some tests to see roughly how long dealing with this stuff is going to take to see where to go from here. Here is the code I'm using to test:

from datetime import datetime
import json

print datetime.now()

f = open('file.json', 'r')

print datetime.now()

Not too surprisingly, Python gives me a MemoryError. It appears that json.load() calls json.loads(f.read()), which is trying to dump the entire file into memory first, which clearly isn't going to work.

Any way I can solve this cleanly?

I know this is old, but I don't think this is a duplicate. While the answer is the same, the question is different. In the "duplicate", the question is how to read large files efficiently, whereas this question deals with files that won't even fit in to memory at all. Efficiency isn't required.

share|improve this question

marked as duplicate by Piotr Dobrogost, WATTO Studios, Barmar, Andro Selva, ThinkingStiff Oct 2 '12 at 6:11


Similar if not the same question: stackoverflow.com/questions/2400643/… –  tskuzzy Apr 30 '12 at 10:39
The issue is that if the JSON file is one giant list (for example), then parsing it into Python wouldn't make much sense without doing it all at once. I guess your best bet is to find a module that handles JSON like SAX and gives you events for starting arrays and stuff, rather than giving you objects. Unfortunately, that doesn't exist in the standard library. –  Lattyware Apr 30 '12 at 10:40
Well, I kind of want to read it in all at once. One of my potential plans is to go through it once and stick everything in a database so I can access it more efficiently. –  Knyght Apr 30 '12 at 10:45
If you can't fit the entire file as text into memory, I sincerely doubt you'll fit the entire file as Python objects into memory. If you want to put it in a database, my answer could be helpful. –  Lattyware Apr 30 '12 at 10:46
For any non-trivial task processing of json files such sizes can easy take weeks or months. –  yazu Apr 30 '12 at 10:50

1 Answer 1

up vote 25 down vote accepted

The best option appears to be using something like ijson - a module that will work with JSON as a stream, rather than as a block file.

Edit: Also worth a look - kashif's comment about json-streamer.

share|improve this answer
Thanks! I'll check that out in hopes that I don't have to resort to using Java. –  Knyght Apr 30 '12 at 10:51
Yeah I think I'll end up using Java but this is the answer to the question I actually asked. Thanks again. –  Knyght Apr 30 '12 at 13:01
I found that ijson requires complete json before it will stream it - I would have preferred something that can work with partial json as it becomes available. Couldn't find anything so wrote my own, its called jsonstreamer and is available at github and at the cheeseshop –  kashif Dec 19 at 11:38
