title: Parsing with R huge json with many different keys
body: <h2>BACKGROUND</h2>
  <p>I have a relatively simple but huge (~GB) json file with the following structure</p>
  <pre><code>[{&quot;a&quot;:92,&quot;b&quot;:{&quot;2019-01-01T00:00:14&quot;:102,&quot;2019-01-01T00:01:15&quot;:123,...}},
   {&quot;a&quot;:20,&quot;b&quot;:{&quot;2019-01-01T01:00:21&quot;:104,&quot;2019-01-01T01:01:25&quot;:321,...}}]
  </code></pre>
  <p>The periods ... in the nested structure indicate that this element has a variable number of entries, up to about 60.</p>
  <p>Ultimately I want to be able to make an R-data.frame out of this, with either of the following two structures (option 1 is prefered)</p>
  <p><strong>Option 1</strong></p>
  <pre><code>a, Time,                    Value
  92, &quot;2019-01-01T00:00:14&quot;,   102
  92, &quot;2019-01-01T00:01:15&quot;,   123
  20, &quot;2019-01-01T01:00:21&quot;,   104
  20, &quot;2019-01-01T01:01:25&quot;,   321
  </code></pre>
  <p><strong>Option 2</strong></p>
  <pre><code>a,  Time_1,                  Value_1, Time_2,                 Value_2, ...
  92, &quot;2019-01-01T00:00:14&quot;,   102,    &quot;2019-01-01T00:01:15&quot;,   123
  20, &quot;2019-01-01T01:00:21&quot;,   104,    &quot;2019-01-01T01:01:25&quot;,   321
  </code></pre>
  <p>It is not so important that the manipulation is done in R. All I care is that the final data is in one of the specified formats.</p>
  <hr />
  <h2>HOW I HAVE THOUGHT ABOUT IT</h2>
  <p>I have been able to write an R-program that (with jsonlite and data.table) converts the json into a data.frame with the first structure.
  However this technique works with files up to 200MB or so. For bigger files it just takes a prohibitive amount of time.</p>
  <p>An approach could be to directly modify the structure of the json object, a bit like one can use xslt to change the structure of xml objects. But I am not aware of such a tool.
  Another would be to somehow specify the structure of the json for a parser to extract the information. But I don't know how to do this either.</p>
  <p>Besides I just found the R-package jqr, which sounds very promising but I do not quite understand how to use it <a href="https://cran.r-project.org/web/packages/jqr/jqr.pdf" rel="nofollow noreferrer">https://cran.r-project.org/web/packages/jqr/jqr.pdf</a></p>
  <hr />
  <h2>PROBLEM</h2>
  <p>My main problem is that</p>
  <ol>
  <li>the file is really big</li>
  <li>the keys of the nested structure are not &quot;unique&quot; because they are timestamps. So a &quot;standard&quot; parser will convert each unique value into a column of a dataframe with the key as columnname and as value the corresponding value, potentially generating a LOT of columns.</li>
  </ol>