title: Rewriting a JSON file into a CSV efficiently in Bash
body: <p>I want to efficiently rewrite a large json, which has always the same field names, into a csv, ignoring its keys. </p>
  
  <p>To give a concrete example, here is a large JSON file (<code>tempST.json</code>):
  <a href="https://gist.githubusercontent.com/pedro-roberto/b81672a89368bc8674dae21af3173e68/raw/e4afc62b9aa3092c8722cdbc4b4b4b6d5bbc1b4b/tempST.json" rel="nofollow noreferrer">https://gist.githubusercontent.com/pedro-roberto/b81672a89368bc8674dae21af3173e68/raw/e4afc62b9aa3092c8722cdbc4b4b4b6d5bbc1b4b/tempST.json</a></p>
  
  <p>If I rewrite just fields <code>time</code>, <code>ancestorcount</code> and <code>descendantcount</code> from this JSON into a CSV I should get:</p>
  
  <pre><code>1535995526,1,1
  1535974524,1,1
  1535974528,1,2
  ...
  1535997274,1,1
  </code></pre>
  
  <p>The following script <code>tempSpeedTest.sh</code> writes the value of the fields <code>time</code>, <code>ancestorcount</code> and <code>descendantcount</code> into each line of the csv:</p>
  
  <pre><code>rm tempOutput.csv
  jq -c '.[]' &lt; tempST.json | while read line; do 
  descendantcount=$(echo $line | jq '.descendantcount')
  ancestorcount=$(echo $line | jq '.ancestorcount')
  time=$(echo $line | jq '.time')
  echo "${time},${ancestorcount},${descendantcount}" &gt;&gt; tempOutput.csv
  done
  </code></pre>
  
  <p>However the script takes around 3 minutes to run, which is unsatisfying:</p>
  
  <pre><code>&gt;time bash tempSpeedTest.sh
  real    2m50.254s
  user    2m43.128s
  sys 0m34.811s
  </code></pre>
  
  <p>What is a faster way to achieve the same result?</p>