title: How to handle null values when writing to parquet from Spark
body: <p>Until recently <code>parquet</code> did not support <code>null</code> values - a questionable premise. In fact a recent version did finally add that support:</p>
  
  <p><a href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md" rel="noreferrer">https://github.com/apache/parquet-format/blob/master/LogicalTypes.md</a> </p>
  
  <p>However it will be a long time before <code>spark</code> supports that new <code>parquet</code> feature - if ever. Here is the associated (<code>closed - will not fix</code>) JIRA:</p>
  
  <p><a href="https://issues.apache.org/jira/browse/SPARK-10943" rel="noreferrer">https://issues.apache.org/jira/browse/SPARK-10943</a></p>
  
  <p>So what are folks doing with regards to null column values <em>today</em> when writing out <code>dataframe</code>'s to <code>parquet</code> ?   I can only think of <em>very</em> ugly horrible hacks like writing empty strings and .. well .. I have <em>no</em> idea what to do with numerical values to indicate <code>null</code> - short of putting some sentinel value in and having my code check for it (which is inconvenient and bug prone).</p>