title: How to add fields to existing parquet and generate new parquet (Scala or python)
body: <p>I have existing parquets (say p1) to read into dataframe, and then write it into new parquet file (say p2) after some transformation;</p>
  
  <p>The process is like:</p>
  
  <pre><code>val df1 = spark.read.parquet(s"path_to_p1_1")
  df1.createOrReplaceTempView("table1")
  
  val df2 = spark.read.parquet(s"path_to_p1_2")
  df2.createOrReplaceTempView("table2")
  
  val q = s"""
  select 
         cast(ADDRESS as String) as ADDRESS,
         cast(CITY as String) as CITY,
         cast(STATE as String) as STATE,
  .......80 fields.......
  FROM 
      ( SELECT * FROM table1
          UNION
      SELECT * FROM table2 ) A
    """
  
  val result = spark.sql(q)
  res.repartition(1).write.mode(SaveMode.Overwrite).parquet(s"path_to_p2")
  </code></pre>
  
  <p>It comes to the need to derive geo info (long and lat) from three columns (address, city, state) and add it back to the p2, or create a new parquet file p3.</p>
  
  <p>The geo part is like below:</p>
  
  <pre><code>import requests
  
  http_str = 'https://maps.googleapis.com/maps/api/geocode/json?address='
  addr = '1600+Amphitheatre+Parkway,+Mountain+View,+CA'
  #addr = '181 University Ave, Toronto, ON, CANADA'
  response = requests.get(http_str + addr)
  
  resp_json_payload = response.json()
  latlong = resp_json_payload['results'][0]['geometry']['location']
  lat = latlong.get('lat')
  lng = latlong.get('lng')
  </code></pre>
  
  <p>lat and lng is the two derived value I want to add to the existing p2 parquet (prefered) or a new p3 parquet.</p>
  
  <p>What is the best way to do this?</p>
  
  <p>Thank you very much.</p>