title: Set schema in pyspark dataframe read.csv with null elements
body: <p>I have a data set (example) that when imported with </p>
  
  <pre><code>df = spark.read.csv(filename, header=True, inferSchema=True)
  df.show()
  </code></pre>
  
  <p>will assign the column with 'NA' as a stringType(), where I would like it to be IntegerType() (or ByteType()).</p>
  
  <p><a href="https://i.stack.imgur.com/XggfZ.png" rel="noreferrer"><img src="https://i.stack.imgur.com/XggfZ.png" alt="inferSchema"></a></p>
  
  <p>I then tried to set </p>
  
  <pre><code>schema = StructType([
      StructField("col_01", IntegerType()),
      StructField("col_02", DateType()),
      StructField("col_03", IntegerType())
  ])
  df = spark.read.csv(filename, header=True, schema=schema)
  df.show()
  </code></pre>
  
  <p>The output shows the entire row with <em>'col_03' = null</em> to be null.</p>
  
  <p><a href="https://i.stack.imgur.com/59B8w.png" rel="noreferrer"><img src="https://i.stack.imgur.com/59B8w.png" alt="entire_row_null"></a></p>
  
  <p>However <em>col_01</em> and <em>col_02</em> return appropriate data if they are called with</p>
  
  <pre><code>df.select(['col_01','col_02']).show()
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/FTOGn.png" rel="noreferrer"><img src="https://i.stack.imgur.com/FTOGn.png" alt="row_actually_not_null"></a></p>
  
  <p>I can find a way around this by post casting the data type of <em>col_3</em></p>
  
  <pre><code>df = spark.read.csv(filename, header=True, inferSchema=True)
  df = df.withColumn('col_3',df['col_3'].cast(IntegerType()))
  df.show()
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/7xyWd.png" rel="noreferrer"><img src="https://i.stack.imgur.com/7xyWd.png" alt="import_then_cast"></a></p>
  
  <p>, but I think it is not ideal and would be much better if I can assign the data type for each column directly with setting schema.</p>
  
  <p>Would anyone be able to guide me what I do incorrectly? Or casting the data types after importing is the only solution? Any comment regarding performance of the two approaches (if we can make assigning schema to work) is also welcome.</p>
  
  <p>Thank you,</p>