title: How to use a list of aggregate expressions with groupby in pyspark?
body: <p>I have a dataframe with 2 columns like this:</p>
  
  <pre><code>+----+---+
  |ptyp|sID|
  +----+---+
  |  CO|111|
  |  CO|222|
  |  CO|222|
  |  CO|222|
  |  CO|111|
  |  CD|111|
  |  CD|222|
  |  CD|222|
  |  CD|333|
  |  CD|333|
  |  CD|333|
  |  AG|111|
  |  AG|111|
  |  AG|111|
  |  AG|222|
  +----+---+
  </code></pre>
  
  <p>Given an input <code>n</code>, for each <code>pytp</code>, I want to add columns which display the top <code>n</code> <code>sIDs</code> (in terms of number of times they appear for that <code>pytp</code>). I also want to print the number of times each <code>sID</code> occurs in a column <code>sIDval</code> for each group. </p>
  
  <p>For example, if <code>n = 2</code>, I want the output to be like this:</p>
  
  <pre><code>+----+-------+-----------+-------+-----------+
  |ptyp|topSID1|topSID1_val|topSID2|topSID2_val|
  +----+-------+-----------+-------+-----------+
  |  AG|    111|          3|    222|          1|
  |  CO|    222|          3|    111|          2|
  |  CD|    333|          3|    222|          2|
  +----+-------+-----------+-------+-----------+
  
  </code></pre>
  
  <p>I am using <code>UDFs</code> to calculate this:</p>
  
  <pre><code>@F.udf
  def mode(x, top_rank):
      c = Counter(x).most_common(top_rank)
      sz = len(c)
      return c[min(top_rank-1, sz-1)][0]
  
  @F.udf
  def modeval(x, top_rank):
      c = Counter(x).most_common(top_rank)
      sz = len(c)
      return c[min(top_rank-1, sz-1)][1]
  </code></pre>
  
  <p>And I am storing the aggregate expressions required for each new column in a list <code>newcols</code>:</p>
  
  <pre><code>newcols = []
  n = 3
  
  for r in range(1, num_ranks+1):
      newcols.append([mode(F.collect_list('sID'), F.lit(r)).alias('topSID' + str(r))])
      newcols.append([modeval(F.collect_list('sID'), F.lit(r)).alias('topSID' + str(r) +'_val')])
  </code></pre>
  
  <p>Now if I know that <code>n=3</code>, I can do it in this way:</p>
  
  <pre><code>df.groupBy('ptyp').agg(*newcols[0], *newcols[1], *newcols[2], \
                         *newcols[3], *newcols[4], *newcols[5])
  </code></pre>
  
  <p>Is there a way I can use generalize this for any value of <code>n</code>?
  I tried </p>
  
  <pre><code>df.groupBy('ptyp').agg([*e for e in new_cols])
  </code></pre>
  
  <p>and</p>
  
  <pre><code>df.groupBy('ptyp').agg((*e for e in new_cols))
  </code></pre>
  
  <p>and many more variations, but all of them give errors.</p>
  
  <p>Now I have resorted to aggregating one at a time and doing a join but that is very expensive.</p>
  
  <p>Is there a way to do this in the manner I've tried above?</p>