title: Transforming high-redundancy CSV data into nested JSON using jq (or awk)?
body: <p>Say I have the following CSV data in <code>input.txt</code>:</p>
  <pre><code>broker,client,contract_id,task_type,doc_names
  alice@company.com,John Doe,33333,prove-employment,important-doc-pdf
  alice@company.com,John Doe,33333,prove-employment,paperwork-pdf
  alice@company.com,John Doe,33333,submit-application,blah-pdf
  alice@company.com,John Doe,00000,prove-employment,test-pdf
  alice@company.com,John Doe,00000,submit-application,test-pdf
  alice@company.com,Jane Smith,11111,prove-employment,important-doc-pdf
  alice@company.com,Jane Smith,11111,submit-application,paperwork-pdf
  alice@company.com,Jane Smith,11111,submit-application,unimportant-pdf
  bob@company.com,John Doe,66666,submit-application,pdf-I-pdf
  bob@company.com,John Doe,77777,submit-application,pdf-J-pdf
  </code></pre>
  <p>And I'd like to transform it into the following JSON:</p>
  <pre class="lang-json prettyprint-override"><code>[
  {&quot;broker&quot;: &quot;alice@company.com&quot;,
      &quot;clients&quot;: [
      {
          &quot;client&quot;: &quot;John Doe&quot;,
          &quot;contracts&quot;: [
          {
              &quot;contract_id&quot;: 33333,
              &quot;documents&quot;: [
              {
                  &quot;task_type&quot;: &quot;prove-employment&quot;,
                  &quot;doc_names&quot;: [&quot;important-doc-pdf&quot;, &quot;paperwork-pdf&quot;]
              },
              {
                  &quot;task_type&quot;: &quot;submit-application&quot;,
                  &quot;doc_names&quot;: [&quot;blah-pdf&quot;]
              }
              ]
          },
          {
              &quot;contract_id&quot;: 00000,
              &quot;documents&quot;: [
              {
                  &quot;task_type&quot;: &quot;prove-employment&quot;,
                  &quot;doc_names&quot;: [&quot;test-pdf&quot;]
              },
              {
                  &quot;task_type&quot;: &quot;submit-application&quot;,
                  &quot;doc_names&quot;: [&quot;test-pdf&quot;]
              }
              ]
          }
          ]
      },
      {
          &quot;client&quot;: &quot;Jane Smith&quot;,
          &quot;contracts&quot;: [
          {
              &quot;contract_id&quot;: 11111,
              &quot;documents&quot;: [
              {
                  &quot;task_type&quot;: &quot;prove-employment&quot;,
                  &quot;doc_names&quot;: [&quot;important-doc-pdf&quot;]
              },
              {
                  &quot;task_type&quot;: &quot;submit-application&quot;,
                  &quot;doc_names&quot;: [&quot;paperwork-pdf&quot;, &quot;unimportant-pdf&quot;]
              }
              ]
          }
          ]
      }
      ]
  },
  
  {&quot;broker&quot;: &quot;bob@company.com&quot;,
      &quot;clients&quot;: [
      {
          &quot;client&quot;: &quot;John Doe&quot;,
          &quot;contracts&quot;: [
          {
              &quot;contract_id&quot;: 66666,
              &quot;documents&quot;: [
              {
                  &quot;task_type&quot;: &quot;submit-application&quot;,
                  &quot;doc_names&quot;: [&quot;pdf-I-pdf&quot;]
              }
              ]
          },
          {
              &quot;contract_id&quot;: 77777,
              &quot;documents&quot;: [
              {
                  &quot;task_type&quot;: &quot;submit-application&quot;,
                  &quot;doc_names&quot;: [&quot;pdf-J-pdf&quot;]
              }
              ]
          }
          ]
      }
      ]
  }
  ]
  
  </code></pre>
  <p>Based on a quick search, it seems like people recommend <code>jq</code> for this type of task. I read some of the manual and played around with it for a bit, and I'm understand that it's meant to be used by composing its filters together to produce the desired output.</p>
  <p>So far, I've been able to transform each line of the CSV into a list of strings for example with <code>jq -Rs '. / &quot;\n&quot; | .[] | . / &quot;,&quot;'</code>.</p>
  <p>But I'm having trouble with something even a bit more complex, like assigning a key to each value on a line (not even the final JSON form I'm looking to get). This is what I tried: <code>jq -Rs '[inputs | . / &quot;\n&quot; | .[] | . / &quot;,&quot; as $line | {&quot;broker&quot;: $line[0], &quot;client&quot;: $line[1], &quot;contract_id&quot;: $line[2], &quot;task_type&quot;: $line[3], &quot;doc_name&quot;: $line[4]}]'</code>, and it gives back <code>[]</code>.</p>
  <p>Maybe <code>jq</code> isn't the best tool for the job here? Perhaps I should be using <code>awk</code>? If all else fails, I'd probably just parse this using Python.</p>
  <p>Any help is appreciated.</p>