title: Pyspark create a summary table with calculated values
body: <p>I have a data frame that looks like this:</p>
  <pre><code>+--------------------+---------------------+-------------+------------+-----+
  |tpep_pickup_datetime|tpep_dropoff_datetime|trip_distance|total_amount|isDay|
  +--------------------+---------------------+-------------+------------+-----+
  | 2019-01-01 09:01:00|  2019-01-01 08:53:20|          1.5|        2.00| true|
  | 2019-01-01 21:59:59|  2019-01-01 21:18:59|          2.6|        5.00|false|
  | 2019-01-01 10:01:00|  2019-01-01 08:53:20|          1.5|        2.00| true|
  | 2019-01-01 22:59:59|  2019-01-01 21:18:59|          2.6|        5.00|false|
  +--------------------+---------------------+-------------+------------+-----+
  </code></pre>
  <p>and I want to create a summary table which calculates the <code>trip_rate</code> for all the night trips and all the day trips (<code>total_amount</code> column divided by <code>trip_distance</code>). So the end result should look like this:</p>
  <pre><code>+------------+-----------+
  | day_night  | trip_rate |
  +------------+-----------+
  |Day         | 1.33      |
  |Night       | 1.92      |
  +------------+-----------+
  </code></pre>
  <p>Here is what I'm trying to do:</p>
  <pre><code>df2 = spark.createDataFrame(
      [
          ('2019-01-01 09:01:00','2019-01-01 08:53:20','1.5','2.00','true'),#day
          ('2019-01-01 21:59:59','2019-01-01 21:18:59','2.6','5.00','false'),#night
          ('2019-01-01 10:01:00','2019-01-01 08:53:20','1.5','2.00','true'),#day
          ('2019-01-01 22:59:59','2019-01-01 21:18:59','2.6','5.00','false'),#night
      ],
      ['tpep_pickup_datetime','tpep_dropoff_datetime','trip_distance','total_amount','day_night'] # add your columns label here
  )
  
  day_trip_rate = df2.where(df2.day_night == 'Day').withColumn(&quot;trip_rate&quot;,F.sum(&quot;total_amount&quot;)/F.sum(&quot;trip_distance&quot;))
  night_trip_rate = df2.where(df2.day_night == 'Night').withColumn(&quot;trip_rate&quot;,F.sum(&quot;total_amount&quot;)/F.sum(&quot;trip_distance&quot;))
  </code></pre>
  <p>I don't believe I'm even approaching it the right way. And I'm getting this error:(
  <code>raise AnalysisException(s.split(': ', 1)[1], stackTrace) pyspark.sql.utils.AnalysisException: &quot;grouping expressions sequence is empty, and '</code>tpep_pickup_datetime<code>' is not an aggregate function.</code></p>
  <p>Can someone help me know how to approach this to get that summary table?</p>