title: Streaming json parser
body: <p>I am looking to implement a streaming json parser for a very, very large JSON file (~ 1TB) that I'm unable to load into memory. One option is to use something like <a href="https://github.com/stedolan/jq" rel="nofollow noreferrer">https://github.com/stedolan/jq</a> to convert the file into json-newline-delimited, but there are various other things I need to do to each json object, that makes this approach not ideal.</p>
  
  <p>Given a very large json object, how would I be able to parse it object-by-object, similar to this approach in xml: <a href="https://www.ibm.com/developerworks/library/x-hiperfparse/index.html" rel="nofollow noreferrer">https://www.ibm.com/developerworks/library/x-hiperfparse/index.html</a>.</p>
  
  <p>For example, in pseudocode:</p>
  
  <pre><code>with open('file.json','r') as f:
      json_str = ''
      for line in f: # what if there are no newline in the json obj?
          json_str += line
          if is_valid(json_str):
              obj = json.loads(json_str)
              do_something()
              json_str = ''
  </code></pre>
  
  <p>Additionally, I did not find <code>jq -c</code> to be particularly fast (ignoring memory considerations). For example, doing <code>json.loads</code> was just as fast (and a bit faster) than using <code>jq -c</code>. I tried using <code>ujson</code> as well, but kept getting a corruption error which I believe was related to the file size. </p>
  
  <pre><code># file size is 2.2GB
  &gt;&gt;&gt; import json,time
  &gt;&gt;&gt; t0=time.time();_=json.loads(open('20190201_itunes.txt').read());print (time.time()-t0)
  65.6147990227
  
  $ time cat 20190206_itunes.txt|jq -c '.[]' &gt; new.json
  real    1m35.538s
  user    1m25.109s
  sys 0m15.205s
  </code></pre>
  
  <p>Finally, here is an example 100KB json input which can be used for testing: <a href="https://hastebin.com/ecahufonet.json" rel="nofollow noreferrer">https://hastebin.com/ecahufonet.json</a></p>