title: Spark sql how to explode without losing null values
body: <p>I have a Dataframe that I am trying to flatten. As part of the process, I want to explode it, so if I have a column of arrays, each value of the array will be used to create a separate row. For instance, </p>
  
  <pre class="lang-none prettyprint-override"><code>id | name | likes
  _______________________________
  1  | Luke | [baseball, soccer]
  </code></pre>
  
  <p>should become </p>
  
  <pre class="lang-none prettyprint-override"><code>id | name | likes
  _______________________________
  1  | Luke | baseball
  1  | Luke | soccer
  </code></pre>
  
  <p>This is my code</p>
  
  <pre class="lang-java prettyprint-override"><code>private DataFrame explodeDataFrame(DataFrame df) {
      DataFrame resultDf = df;
      for (StructField field : df.schema().fields()) {
          if (field.dataType() instanceof ArrayType) {
              resultDf = resultDf.withColumn(field.name(), org.apache.spark.sql.functions.explode(resultDf.col(field.name())));
              resultDf.show();
          }
      }
      return resultDf;
  }
  </code></pre>
  
  <p>The problem is that in my data, some of the array columns have nulls. In that case, the entire row is deleted. So this dataframe:</p>
  
  <pre class="lang-none prettyprint-override"><code>id | name | likes
  _______________________________
  1  | Luke | [baseball, soccer]
  2  | Lucy | null
  </code></pre>
  
  <p>becomes </p>
  
  <pre class="lang-none prettyprint-override"><code>id | name | likes
  _______________________________
  1  | Luke | baseball
  1  | Luke | soccer
  </code></pre>
  
  <p>instead of </p>
  
  <pre class="lang-none prettyprint-override"><code>id | name | likes
  _______________________________
  1  | Luke | baseball
  1  | Luke | soccer
  2  | Lucy | null
  </code></pre>
  
  <p>How can I explode my arrays so that I don't lose the null rows? </p>
  
  <p>I am using Spark 1.5.2 and Java 8</p>