title: Using Spark, how can I pickup a filename when loading everything into a dataframe?
body: <p>My code sample looks like this.</p>
  
  <pre><code>from  pyspark.sql.functions import input_file_name
  from pyspark.sql import SQLContext
  from pyspark.sql.types import *
  sqlContext = SQLContext(sc)
  
  customSchema = StructType([ \
  StructField("asset_id", StringType(), True), \
  StructField("price_date", StringType(), True), \
  etc., 
  StructField("close_price", StringType(), True), \
  StructField("filename", StringType(), True)])
  
  fullpath = 'path_to files_and_all_credentials'
  </code></pre>
  
  <p>Now, to get everything in a dataframe, with the file path added as a column, I would expect the code to be something like this...</p>
  
  <pre><code>df = spark.load(fullPath, withColumn("filename", input_file_name(), header='false', schema = customSchema, sep='|')
  </code></pre>
  
  <p>or...</p>
  
  <pre><code>df = spark.read.format("csv"), \
     .option("header", "false"), \
     .option(schema = customSchema), \
     .option(delimiter = "|"), \
     .load(fullPath), \
     .withColumn("filename", input_file_name()
  </code></pre>
  
  <p>When I run this sample code, I get an 'invalid syntax' error message.  This should be a pretty straightforward thing, I think.  How can I make this work?  Thank you!!</p>