title: Convert 17GB JSON file to a numpy array
body: <p>I have a big 17 GB JSON file placed in hdfs . I need to read that file and convert into nummy array which is then passed into K-Means clustering algorithm. I tried many ways but system slows down and getting a memory error or kernel dies.</p>
  
  <p><a href="https://i.stack.imgur.com/t4cSg.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/t4cSg.jpg" alt="JSON structure"></a></p>
  
  <p>the code i tried is </p>
  
  <pre><code>from hdfs3 import HDFileSystem
  import pandas as pd
  import numpy as nm
  import json
  hdfs = HDFileSystem(host='hostname', port=8020)
  with hdfs.open('/user/iot_all_valid.json/') as f:
   for line in f:
          data = json.loads(line)
  df = pd.DataFrame(data)
  dataset= nm.array(df)
  </code></pre>
  
  <p>I tried using ijson but still not sure which is the right way to do this in faster way.</p>