title: Reading files from HDFS directory and creating a RDD in Spark using Python
body: <p>I have some text files and I want to create an RDD using these files. 
  The text files are stored in 'Folder_1' and 'Folder_2' and these folders are stored in the folder 'text_data'</p>
  
  <p>When the files are stored in local storage, the following code works : </p>
  
  <pre><code>#Reading the corpus as an RDD
  
  data_folder = '/home/user/text_data'
  
  def read_data(data_folder):
      data = sc.parallelize([])
      for folder in os.listdir(data_folder):
          for txt_file in os.listdir( data_folder + '/' + folder  ):
              temp = open( data_folder + '/' + folder + '/' + txt_file)
              temp_da = temp.read()
              temp_da = unicode(temp_da, errors = 'ignore')
              temp.close()
              a = [ ( folder, temp_da) ]
              data = data.union(sc.parallelize( a ) )
      return data
  </code></pre>
  
  <p>The function read_data returns an RDD consisting of the text files.</p>
  
  <p>How can I perform the above function if I move the 'text_data' folder to an HDFS directory?</p>
  
  <p>The code is to be deployed in a Hadoop-Yarn Cluster running SPARK.</p>