title: Spark - handle blank values in CSV file
body: <p>Let's say I've got a simple pipe delimited file, with missing values:</p>
  
  <pre><code>A|B||D
  </code></pre>
  
  <p>I read that into a dataframe:</p>
  
  <pre><code>val foo =  spark.read.format("csv").option("delimiter","|").load("/path/to/my/file.txt")
  </code></pre>
  
  <p>The missing third column, instead of being a null value, has the string null:</p>
  
  <pre><code>+-----------+----+
  |_c0|_c1| _c2|_c3|
  +-----------+----+
  |a  |b  |null|c  |
  </code></pre>
  
  <p>I would like to be able to control what Spark does with that missing value.  According to the <a href="https://docs.databricks.com/data/data-sources/read-csv.html" rel="nofollow noreferrer">CSV Documentation</a>, there is a <code>nullValue</code> option:</p>
  
  <blockquote>
    <p>nullValue: string that indicates a null value, any fields matching
    this string will be set as nulls in the DataFrame.</p>
  </blockquote>
  
  <p>But since I don't have any value there, I can't seem to control it that way.</p>
  
  <p>Short of testing every single field of every single file I read in, is there way to control what spark does with these blanks?  For example, replace it with 'N/A'.</p>
  
  <p>We're using Spark 2.1, if it's relevant.</p>