title: Spark program takes a really long time to complete execution
body: <p>Spark is supposed to finish data processing at lightening speed. But I guess I'm not using the right functionalities for my program to make Spark work that way. </p>
  
  <p>This is how my program looks like:</p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark import SQLContext
  from pyspark.sql.types import *
  from pyspark.sql import Row
  from pyparsing import re
  import time
  
  start_time = time.time()
  sc = SparkContext("local","test")
  sqlContext = SQLContext(sc)
  
  def func1(lines):
      for line in lines:
          qtype = re.search("qtype=(\S+)",line)
          try:
              url = re.search(" url=(\S+)",line)
              url=url.group(1)
          except:
              url="null"
  
          time = re.search("(^\S+)",line)
          .... #extracting field names
          row = Row(time = time.group(1),ttms = ttms.group(1),chi = chi.group(1),pssc = pssc.group(1),cqhm = cqhm.group(1),rtype = rtype.group(1),rdetails = rdetails.group(1),rurl = rurl.group(1),qtype = qtype.group(1),phn = phn.group(1),fqdn = fqdn,url = url)
  
      return row
  
  file = sc.textFile("C:\\Logs\\filename.log").cache()
  result = file.map(lambda x: x.split("\n"))
  lines = result.map(func1).collect()
  df = sqlContext.createDataFrame(lines)
  df.registerTempTable("df")
  df.show()
  
  for line in df :
      query1 = sqlContext.sql("SELECT qtype,rtype,COUNT(qtype,rtype) from df GROUP BY qtype,rtype").collect()
      query2 = sqlContext.sql("SELECT qtype, COUNT(qtype) from df GROUP BY qtype").collect()
      query3 = sqlContext.sql("SELECT rtype, COUNT(rtype) from df GROUP BY rtype").collect()
      query4 = sqlContext.sql("SELECT rtype, COUNT(rtype) from df WHERE qtype = \"HTTP\" GROUP BY rtype").collect()
  
  df1 = sqlContext.createDataFrame(query1)
  df2 = sqlContext.createDataFrame(query2)
  df3 = sqlContext.createDataFrame(query3)
  df4 = sqlContext.createDataFrame(query4)
  df1.toPandas().to_csv("C:\\Sample logs\\sqlcsv1.csv")
  df2.toPandas().to_csv("C:\\Sample logs\\sqlcsv2.csv")
  df3.toPandas().to_csv("C:\\Sample logs\\sqlcsv3.csv")
  df4.toPandas().to_csv("C:\\Sample logs\\sqlcsv4.csv")
  print(time.time() - start_time)
  </code></pre>
  
  <p>This program takes almost 200 seconds to execute which is a very long time. I can't figure out the reason. (My log file contains around 34k log lines). I tried using spark's <code>filter</code> for regex expressions but I get the error that <code>rdd is not iterable</code>. So I need to know in what way can I optimize my program to make it run faster.
  Also, I get the warning that <code>stage x contains task of very large size</code>. Tried broadcasting <code>lines</code> but gave an error.</p>