title: How to process 20GB of data file from HDFS
body: <p>I have a requirement to get a unique JSON record depending on certain conditions from 20GB of data present in a file(depending on queryId). Each line in the file is a JSON record.
  EX :</p>
  <p><code>{&quot;sessionID&quot;:&quot;xyz&quot;,&quot;appID&quot;:&quot;abc&quot;,&quot;tenantID&quot;:&quot;123&quot;,&quot;realmId&quot;:&quot;buyer_2021&quot;,&quot;topology&quot;:&quot;top456&quot;,&quot;queryId&quot;:&quot;12346278&quot;,&quot;signalType&quot;:&quot;USER_QUERY_RESULTS&quot;,XYZ content.....}</code></p>