title: How can I process huge JSON files as streams in Ruby, without consuming all memory?
body: <p>I'm having trouble processing a huge JSON file in Ruby. What I'm looking for is a way to process it entry-by-entry without keeping too much data in memory.</p>
  
  <p>I thought that <a href="https://github.com/brianmario/yajl-ruby" rel="nofollow noreferrer">yajl-ruby</a> gem would do the work but it consumes all my memory. I've also looked at <a href="https://github.com/dgraham/yajl-ffi" rel="nofollow noreferrer">Yajl::FFI</a> and JSON:Stream gems but there it is clearly stated:</p>
  
  <blockquote>
    <p>For larger documents we can use an IO object to stream it into the
    parser. We still need room for the parsed object, but the document
    itself is never fully read into memory.</p>
  </blockquote>
  
  <p>Here's what I've done with Yajl:</p>
  
  <pre><code>file_stream = File.open(file, "r")
  json = Yajl::Parser.parse(file_stream)
  json.each do |entry|
      entry.do_something
  end
  file_stream.close
  </code></pre>
  
  <p>The memory usage keeps getting higher until the process is killed.</p>
  
  <p>I don't see why Yajl keeps processed entries in the memory. Can I somehow free them, or did I just misunderstood the capabilities of Yajl parser?</p>
  
  <p>If it cannot be done using Yajl: is there a way to do this in Ruby via any library?</p>