title: Pyspark: How to aggregate data for all the elements in a list in Pyspark?
body: <p>I am storing all the string fields in a list object. Then currently i am passing each field inside the for loop to calculate the aggregated count.</p>
  
  <p>I am looking for a way to get the aggregated count at once for all the string columns. Please help.</p>
  
  <p>Sample data:</p>
  
  <p>Dataframe(Input_Data) is having these records</p>
  
  <pre><code>NoOfSegments,SegmentID,Country
  3,2,Bangalore
  3,2,Bangalore
  3,3,Delhi
  3,2,Delhi
  3,3,Delhi
  3,1,Pune
  3,3,Bangalore
  3,1,Pune
  3,1,Delhi
  3,3,Bangalore
  3,1,Delhi
  3,3,Bangalore
  3,3,Pune
  3,2,Delhi
  3,3,Pune
  3,2,Pune
  3,2,Pune
  3,3,Pune
  3,1,Bangalore
  3,1,Bangalore
  </code></pre>
  
  <p>MY code:</p>
  
  <pre><code>        input_data.createOrReplaceTempView('input_data')
  
          sub="string"
          category_columns = [name for name, data_type in input_data.dtypes
                                  if sub in data_type]
          df_final_schema = StructType([StructField("Country", StringType())
                             , StructField("SegmentID", IntegerType())
                             , StructField("total_cnt", IntegerType())
                          ])
          df_final=spark.createDataFrame([],df_final_schema)
  
          for cat_col in category_columns:
              query="SELECT {d_name} as Country,SegmentID ,(count(*) over(partition by {d_name},SegmentID)/ count(*) over(partition by NoOfSegments))*100 as total_cnt  from input_temp order by {d_name},SegmentID".format(d_name=cat_col)
              new_df=hc.sql(query)
              df_final = df_final.union(new_df)
  </code></pre>
  
  <p>Result:</p>
  
  <p><a href="https://i.stack.imgur.com/nnIJO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nnIJO.png" alt="enter image description here"></a></p>
  
  <p>Is there any way I can pass all the string columns and calculate the above result at once a data frame?</p>