title: PySpark execute all test cases in a directory
body: <p>I am trying to develop a script which will run all the spark sql queries kept in a directory.
  I have been able to do that in Python, but pyspark is a different game.
  Below is the python script I use to read and execute all query files in a directory.</p>
  
  <blockquote>
    <p>import sys,csv,sqlite3,codecs,unicodedata, string,glob, os,c
    conn=psycopg2.connect(database="xxx", user="xxxx", password="xxxx",
    host="localhost", port="5432") cur = conn.cursor() print("done")</p>
    
    <p>with open("*.txt", "r") as ins:  for line in ins:<br>
    words=line.split('|')    print(words)    query=words[0]<br>
    pmicode=words[1]    print(query)
         cur = conn.cursor()
         cur.execute(query) conn.commit() conn.close()</p>
  </blockquote>
  
  <p>Is it possible to replicate this in PySpark ?</p>
  
  <p>Thanks,
  Pankaj</p>