title: Inserting JSON data to remote Postgres table with Python Psycopg2 taking too long time.Is this the right approach?
body: <p>I have 2GB json data from Mongodb export.I am trying make some analytics with those data and post those analytics data into remote Postgres table.This is my code:</p>
  <pre><code>import json
  
  from psycopg2 import connect, Error
  import dateutil.parser as parser
  from datetime import datetime
  
  record_list = []
  
  for line in open('december.json', 'r'):
      record_list.append(json.loads(line))
  
      #print(record_list[0])
      
  
  
      devices = []
  
      for key, values in record_list[0].items():
  
  
         if key == &quot;deviceSerial&quot;:
            devices.append(values)
      test_list = list(set(devices)) 
      device_data = []
  
      for x in test_list:
          device_1 = [i for i in record_list if (i['deviceSerial'] == x)] 
          datetime_object_first = parser.parse(device_1[0]['createdAt'],fuzzy=True) 
          datetime_object_last = parser.parse(device_1[len(device_1)-1]['createdAt'],fuzzy=True) 
  
          devices = device_1[0]['deviceSerial']
          device_model = device_1[0]['device']
          device_Usage=round((datetime.timestamp(datetime_object_last)-datetime.timestamp(datetime_object_first))/60/1000,3)
  
  
          #calculating device usage
          device_data_elements ={
  
              'deviceModel': device_model,
              'deviceSerial':devices,
              'deviceUsageInMin':device_Usage
       
          }
  
          device_data.append(device_data_elements)
  
          if type(device_data) == list:
  
  
  
              first_record = device_data[0]
  
  
          columns = list(first_record.keys())
  
          #print (&quot;\ncolumn names:&quot;, columns)
  
          #print(device_data)
  
          table_name = &quot;test_data&quot;
  sql_string = 'INSERT INTO {} '.format( table_name )
  sql_string += &quot;(&quot; + ', '.join(columns) + &quot;)\nVALUES &quot;
  
  
  
  # enumerate over the record
  for i, record_dict in enumerate(device_data):
  
      # iterate over the values of each record dict object
      values = []
      for col_names, val in record_dict.items():
  
          if type(val) == str:
            
              val = val.replace(&quot;'&quot;, &quot;''&quot;)
              val = &quot;'&quot; + val + &quot;'&quot;
  
          values += [ str(val) ]
  
          # join the list of values and enclose record in parenthesis
      sql_string += &quot;(&quot; + ', '.join(values) + &quot;),\n&quot;
  
  # remove the last comma and end statement with a semicolon
  sql_string = sql_string[:-2] + &quot;;&quot;
  
  print (&quot;\nSQL statement:&quot;)
  print (sql_string)
  
  try:
      # declare a new PostgreSQL connection object
      conn = connect(
          dbname = &quot;events_data&quot;,
          user = &quot;chao&quot;,
          host = &quot;localhost&quot;,
          
          # attempt to connect for 10 seconds then raise exception
          connect_timeout = 10
      )
  
      cur = conn.cursor()
      print (&quot;\ncreated cursor object:&quot;, cur)
      
  #Post data into postgres table
  except (Exception, Error) as err:
      print (&quot;\npsycopg2 connect error:&quot;, err)
      conn = None
      cur = None
  
  if cur != None:
  
      try:
          
          cur.execute( sql_string )
          conn.commit()
  
          print ('\nfinished INSERT INTO execution')
  
      except (Exception, Error) as error:
          print(&quot;\nexecute_sql() error:&quot;, error)
          conn.rollback()
  
      # close the cursor and connection
      cur.close()
      conn.close()
  </code></pre>
  <p>I am doing some calculations(analytics) with this script.So there is some for loop in the script.When data length is less, script inserts those analysis into Postgres successfully.If data length is big, it takes too long time and I waited even 12h but no success .Right now my script is running locally and reading data also locally.What would be the best approach to read and manipulate large volume data and post the analytics to Postgres table.This is my sample json data.</p>
  <pre><code>
  [
      {
     &quot;createdAt&quot;: &quot;Fri Nov 27 2020 08:07:39 GMT+0000 &quot;,
     &quot;sessionId&quot;: null,
     &quot;text&quot;: null,
     &quot;device&quot;: null,
     &quot;deviceSerial&quot;: null
   },
   {
     &quot;createdAt&quot;: &quot;Tue Sep 01 2020 06:59:18 GMT+0000&quot;,
     &quot;sessionId&quot;: null,
     &quot;text&quot;: &quot;Approve&quot;,
     &quot;device&quot;: &quot;Android&quot;,
     &quot;deviceSerial&quot;: null
   },
   {
     &quot;createdAt&quot;: &quot;Wed Sep 02 2020 08:40:10 GMT+0000&quot;,
     &quot;pageTitle&quot;: &quot;submit option&quot;,
     &quot;sessionId&quot;: null,
     &quot;text&quot;: &quot;launchComponent&quot;,
     &quot;device&quot;: &quot;Android&quot;,
     &quot;deviceSerial&quot;: &quot;636363636890&quot;
   },
   {
     &quot;createdAt&quot;: &quot;Wed Sep 02 2020 08:40:11&quot;,
     &quot;pageTitle&quot;: &quot;quick check&quot;,
     &quot;sessionId&quot;: &quot;88958d89c65f4fcea56e148a5a2838cfhdhdhd&quot;,
     &quot;text&quot;: &quot;&quot;,
     &quot;device&quot;: &quot;Android&quot;,
     &quot;deviceSerial&quot;: &quot;6625839827&quot;
   }
  ]
  </code></pre>