title: Slow PySpark script with very low volume
body: <p>It takes about 40 minutes on a 8core/32G Windows computer to run following Python script. Why is it so slow?</p>
  
  <pre><code>for he in range(1, 25):
      he_str = str(he)
      ### df_all is a dataframe that contains only 3200 records ###
      ### df_all does contain 146 columns. Maybe this is why? ###
      df_all = df_all.withColumn('PROFIT_INC_HE' + he_str, functions.lit(0))
      df_all = df_all.withColumn('PROFIT_DEC_HE' + he_str, functions.lit(0))
  
      ### TIER_PRICE_FACTORS is list of 4 elements ###
      for tiers in TIER_PRICE_FACTORS:
          tiers_str = str(tiers).replace('.', '')
  
          df_all = df_all.withColumn('PROFIT_INC_HE' + he_str, functions.col('PROFIT_INC_HE' + he_str) \
                                      + functions.col('BID_PROFIT_INC_HE' + he_str + '_' + tiers_str))
  
          df_all = df_all.withColumn('PROFIT_DEC_HE' + he_str, functions.col('PROFIT_DEC_HE' + he_str) \
                                      + functions.col('BID_PROFIT_Dec_HE' + he_str + '_' + tiers_str))
  </code></pre>