title: Spark: Load multiple files, perform same operation and merge into a single dataFrame
body: <p>I have lots of small, individual .txt files. For each of these files, I have multiple rows divided by a space into 2 columns, start_time and end_time (a float number).</p>
  
  <p>I'd like to:</p>
  
  <ul>
  <li>load all the .txt files</li>
  <li>for each row compute a new column that contains (end_time - start_time)</li>
  <li>for each row add a new column with the name of the file</li>
  <li>In the end, I want to get a single dataFrame with this schema:</li>
  </ul>
  
  <pre><code>+------------+--------------+------------+------------+
  |  file_name |   start_time |   end_time |   duration |
  +------------+--------------+------------+------------+
  </code></pre>
  
  <p>I know that I can simply make a loop for each file and each row and add one single line at a time to a dataframe, but I'd like to know if there is a faster way to do this.<br>
  I am not interested in the order in which things are done, but the speed of the final result. 
  I see that existing functions like <em>textFile()</em> and <em>wholeTextFiles()</em> are provided in SparkContext but I could not figure out how to use them to get to do what I want.</p>
  
  <p>Any direction or recommendations are greatly appreciated!</p>
  
  <p>(Sorry about my poor English)</p>
  
  <p><strong>Update:</strong></p>
  
  <p>Thanks to @Shu for the help, this is the final code that I use to solve my problem</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql.functions import split, reverse, input_file_name
  
  original_schema = [StructField("Start", FloatType(), True),
                      StructField("End", FloatType(), True)]
  
  data_structure = StructType(original_schema)
  
  df = self.spark_session.read.\
      csv(path=PATH_FILES+'\\*.txt', header=False, schema=data_structure, sep='\t').\
      withColumn("Filename", reverse(split(input_file_name(), "/")).getItem(0) ).\
      withColumn("duration", col("End") - col("Start"))
  
  df.show(20, False)
  </code></pre>