title: How to convert data to desired format and write to a file - Python + Apache Beam
body: <p>I have a <code>.ndjson</code> file looks like this:</p>
  <pre><code>{&quot;property_id&quot;: &quot;107&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FB9E-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 80000, &quot;date_of_transfer&quot;: &quot;2021-05-07 00:00&quot;, &quot;postcode&quot;: &quot;BL2 2GY&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;14&quot;, &quot;SAON&quot;: &quot;&quot;, &quot;street&quot;: &quot;RIVER VIEW COURT&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;BOLTON&quot;, &quot;district&quot;: &quot;BOLTON&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  {&quot;property_id&quot;: &quot;108&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FB9F-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 330000, &quot;date_of_transfer&quot;: &quot;2021-02-26 00:00&quot;, &quot;postcode&quot;: &quot;SK6 4AN&quot;, &quot;property_type&quot;: &quot;S&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;F&quot;, &quot;PAON&quot;: &quot;18&quot;, &quot;SAON&quot;: &quot;&quot;, &quot;street&quot;: &quot;GUYWOOD LANE&quot;, &quot;locality&quot;: &quot;ROMILEY&quot;, &quot;town_city&quot;: &quot;STOCKPORT&quot;, &quot;district&quot;: &quot;STOCKPORT&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  {&quot;property_id&quot;: &quot;109&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FBA0-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 215000, &quot;date_of_transfer&quot;: &quot;2021-02-19 00:00&quot;, &quot;postcode&quot;: &quot;M1 2BL&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;40&quot;, &quot;SAON&quot;: &quot;APARTMENT 022&quot;, &quot;street&quot;: &quot;HILTON STREET&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;MANCHESTER&quot;, &quot;district&quot;: &quot;MANCHESTER&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  {&quot;property_id&quot;: &quot;109&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FBD3-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 226500, &quot;date_of_transfer&quot;: &quot;2021-02-08 00:00&quot;, &quot;postcode&quot;: &quot;M1 2BL&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;40&quot;, &quot;SAON&quot;: &quot;APARTMENT 727&quot;, &quot;street&quot;: &quot;HILTON STREET&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;MANCHESTER&quot;, &quot;district&quot;: &quot;MANCHESTER&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  {&quot;property_id&quot;: &quot;109&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FBF8-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 262000, &quot;date_of_transfer&quot;: &quot;2021-05-14 00:00&quot;, &quot;postcode&quot;: &quot;M1 2BL&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;40&quot;, &quot;SAON&quot;: &quot;APARTMENT 025&quot;, &quot;street&quot;: &quot;HILTON STREET&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;MANCHESTER&quot;, &quot;district&quot;: &quot;MANCHESTER&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  ...
  </code></pre>
  <p>I used Apache Beam to read it and grouped the data by <code>property_id</code>, then I wrote the output to a json file, but the data looks like this:</p>
  <pre><code>('107', [PPD(property_id='107', transaction_unique_id='{C3C3F9B5-FB9E-362B-E053-6B04A8C03ACC}', price=80000, date_of_transfer='2021-05-07 00:00', postcode='BL2 2GY', property_type='F', old_new='N', duration='L', PAON='14', SAON='', street='RIVER VIEW COURT', locality='', town_city='BOLTON', district='BOLTON', country='GREATER MANCHESTER', PPD_category_type='A', record_status='A')])
  ('108', [PPD(property_id='108', transaction_unique_id='{C3C3F9B5-FB9F-362B-E053-6B04A8C03ACC}', price=330000, date_of_transfer='2021-02-26 00:00', postcode='SK6 4AN', property_type='S', old_new='N', duration='F', PAON='18', SAON='', street='GUYWOOD LANE', locality='ROMILEY', town_city='STOCKPORT', district='STOCKPORT', country='GREATER MANCHESTER', PPD_category_type='A', record_status='A')])
  ('109', [PPD(property_id='109', transaction_unique_id='{C3C3F9B5-FBA0-362B-E053-6B04A8C03ACC}', price=215000, date_of_transfer='2021-02-19 00:00', postcode='M1 2BL', property_type='F', old_new='N', duration='L', PAON='40', SAON='APARTMENT 022', street='HILTON STREET', locality='', town_city='MANCHESTER', district='MANCHESTER', country='GREATER MANCHESTER', PPD_category_type='A', record_status='A'), PPD(property_id='109', transaction_unique_id='{C3C3F9B5-FBD3-362B-E053-6B04A8C03ACC}', price=226500, date_of_transfer='2021-02-08 00:00', postcode='M1 2BL', property_type='F', old_new='N', duration='L', PAON='40', SAON='APARTMENT 727', street='HILTON STREET', locality='', town_city='MANCHESTER', district='MANCHESTER', country='GREATER MANCHESTER', PPD_category_type='A', record_status='A'), PPD(property_id='109', transaction_unique_id='{C3C3F9B5-FBF8-362B-E053-6B04A8C03ACC}', price=262000, date_of_transfer='2021-05-14 00:00', postcode='M1 2BL', property_type='F', old_new='N', duration='L', PAON='40', SAON='APARTMENT 025', street='HILTON STREET', locality='', town_city='MANCHESTER', district='MANCHESTER', country='GREATER MANCHESTER', PPD_category_type='A', record_status='A')])
  ...
  </code></pre>
  <p>We can see that for <code>property_id = '109'</code>, it grouped three records, but the output format above is really weird...Does anything know why is that and how can I convert it to a newline delimited JSON format and then write to a JSON file?</p>
  <p>Expected format looks something like (not sure if this is valid newline delimited json format, but the idea is to include the transactions for the same <code>property_id</code> (for example <code>109</code>) in an array):</p>
  <pre><code>{&quot;property_id&quot;: &quot;107&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FB9E-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 80000, &quot;date_of_transfer&quot;: &quot;2021-05-07 00:00&quot;, &quot;postcode&quot;: &quot;BL2 2GY&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;14&quot;, &quot;SAON&quot;: &quot;&quot;, &quot;street&quot;: &quot;RIVER VIEW COURT&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;BOLTON&quot;, &quot;district&quot;: &quot;BOLTON&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  {&quot;property_id&quot;: &quot;108&quot;, &quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FB9F-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 330000, &quot;date_of_transfer&quot;: &quot;2021-02-26 00:00&quot;, &quot;postcode&quot;: &quot;SK6 4AN&quot;, &quot;property_type&quot;: &quot;S&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;F&quot;, &quot;PAON&quot;: &quot;18&quot;, &quot;SAON&quot;: &quot;&quot;, &quot;street&quot;: &quot;GUYWOOD LANE&quot;, &quot;locality&quot;: &quot;ROMILEY&quot;, &quot;town_city&quot;: &quot;STOCKPORT&quot;, &quot;district&quot;: &quot;STOCKPORT&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}
  {&quot;property_id&quot;: &quot;109&quot;, &quot;transactions&quot;: [{&quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FBA0-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 215000, &quot;date_of_transfer&quot;: &quot;2021-02-19 00:00&quot;, &quot;postcode&quot;: &quot;M1 2BL&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;40&quot;, &quot;SAON&quot;: &quot;APARTMENT 022&quot;, &quot;street&quot;: &quot;HILTON STREET&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;MANCHESTER&quot;, &quot;district&quot;: &quot;MANCHESTER&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;},{&quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FBD3-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 226500, &quot;date_of_transfer&quot;: &quot;2021-02-08 00:00&quot;, &quot;postcode&quot;: &quot;M1 2BL&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;40&quot;, &quot;SAON&quot;: &quot;APARTMENT 727&quot;, &quot;street&quot;: &quot;HILTON STREET&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;MANCHESTER&quot;, &quot;district&quot;: &quot;MANCHESTER&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;},{&quot;transaction_unique_id&quot;: &quot;{C3C3F9B5-FBF8-362B-E053-6B04A8C03ACC}&quot;, &quot;price&quot;: 262000, &quot;date_of_transfer&quot;: &quot;2021-05-14 00:00&quot;, &quot;postcode&quot;: &quot;M1 2BL&quot;, &quot;property_type&quot;: &quot;F&quot;, &quot;old_new&quot;: &quot;N&quot;, &quot;duration&quot;: &quot;L&quot;, &quot;PAON&quot;: &quot;40&quot;, &quot;SAON&quot;: &quot;APARTMENT 025&quot;, &quot;street&quot;: &quot;HILTON STREET&quot;, &quot;locality&quot;: &quot;&quot;, &quot;town_city&quot;: &quot;MANCHESTER&quot;, &quot;district&quot;: &quot;MANCHESTER&quot;, &quot;country&quot;: &quot;GREATER MANCHESTER&quot;, &quot;PPD_category_type&quot;: &quot;A&quot;, &quot;record_status&quot;: &quot;A&quot;}]}
  ...
  </code></pre>
  <p>Can someone help please, I'm very new to beam, any help would be appreciated. Thanks.</p>