title: Apache Spark (PySpark) handling null values when reading in CSV
body: <p>I'm trying to read in flight data from the Department of Transportation. It is stored in a CSV, and keep getting <code>java.lang.NumberFormatException: null</code></p>
  
  <p>I have tried setting the <code>nanValue</code> to the empty string, as it's default value is <code>NaN</code>, but this hasn't worked. </p>
  
  <p>My current code is:</p>
  
  <pre><code>spark = SparkSession.builder \
      .master('local') \
      .appName('Flight Delay') \
      .getOrCreate()
  
  schema = StructType([
      StructField('Year', IntegerType(), nullable=True),
      StructField('Month', IntegerType(), nullable=True),
      StructField('Day', IntegerType(), nullable=True),
      StructField('Dow', IntegerType(), nullable=True),
      StructField('CarrierId', StringType(), nullable=True),
      StructField('Carrier', StringType(), nullable=True),
      StructField('TailNum', StringType(), nullable=True),
      StructField('Origin', StringType(), nullable=True),
      StructField('Dest', StringType(), nullable=True),
      StructField('CRSDepTime', IntegerType(), nullable=True),
      StructField('DepTime', IntegerType(), nullable=True),
      StructField('DepDelay', DoubleType(), nullable=True),
      StructField('TaxiOut', DoubleType(), nullable=True),
      StructField('TaxiIn', DoubleType(), nullable=True),
      StructField('CRSArrTime', IntegerType(), nullable=True),
      StructField('ArrTime', IntegerType(), nullable=True),
      StructField('ArrDelay', DoubleType(), nullable=True),
      StructField('Cancelled', DoubleType(), nullable=True),
      StructField('CancellationCode', StringType(), nullable=True),
      StructField('Diverted', DoubleType(), nullable=True),
      StructField('CRSElapsedTime', DoubleType(), nullable=True),
      StructField('ActualElapsedTime', DoubleType(), nullable=True),
      StructField('AirTime', DoubleType(), nullable=True),
      StructField('Distance', DoubleType(), nullable=True),
      StructField('CarrierDelay', DoubleType(), nullable=True),
      StructField('WeatherDelay', DoubleType(), nullable=True),
      StructField('NASDelay', DoubleType(), nullable=True),
      StructField('SecurityDelay', DoubleType(), nullable=True),
      StructField('LateAircraftDelay', DoubleType(), nullable=True)
  ])
  
  flts = spark.read \
      .format('com.databricks.spark.csv') \
      .csv('/home/william/Projects/flight-delay/data/201601.csv',
           schema=schema, nanValue='', header='true')
  </code></pre>
  
  <p>Here is the CSV I'm working with: <a href="http://pastebin.com/waahrgqB" rel="nofollow noreferrer">http://pastebin.com/waahrgqB</a> </p>
  
  <p>The last row there is where it breaks and raises the <code>java.lang.NumberFormatException: null</code></p>
  
  <p>It seems that some numeric columns are empty strings, while others are just empty. Can someone please help me with this?</p>