title: Scala: Delete empty array values from a Spark DataFrame
body: <p>I'm a new learner of Scala. Now given a DataFrame named df as follows:</p>
  
  <pre><code>+-------+-------+-------+-------+
  |Column1|Column2|Column3|Column4|
  +-------+-------+-------+-------+
  | [null]|  [0.0]|  [0.0]| [null]|
  | [IND1]|  [5.0]|  [6.0]|    [A]|
  | [IND2]|  [7.0]|  [8.0]|    [B]|
  |     []|     []|     []|     []|
  +-------+-------+-------+-------+
  </code></pre>
  
  <p>I'd like to delete rows if all columns is an empty array (4th row).</p>
  
  <p>For example I might expect the result is:</p>
  
  <pre><code>+-------+-------+-------+-------+
  |Column1|Column2|Column3|Column4|
  +-------+-------+-------+-------+
  | [null]|  [0.0]|  [0.0]| [null]|
  | [IND1]|  [5.0]|  [6.0]|    [A]|
  | [IND2]|  [7.0]|  [8.0]|    [B]|
  +-------+-------+-------+-------+
  </code></pre>
  
  <p>I'm trying to use isNotNull (like <code>val temp=df.filter(col("Column1").isNotNull &amp;&amp; col("Column2").isNotNull &amp;&amp; col("Column3").isNotNull &amp;&amp; col("Column4").isNotNull).show()</code>
  ) but still show all rows.</p>
  
  <p>I found python solution of using a Hive UDF from <a href="https://stackoverflow.com/questions/32445351/selecting-empty-array-values-from-a-spark-dataframe">link</a>, but I had hard time trying to convert to a valid scala code. I would like use scala command similar to the following code:</p>
  
  <pre><code>val query = "SELECT * FROM targetDf WHERE {0}".format(" AND ".join("SIZE({0}) &gt; 0".format(c) for c in ["Column1", "Column2", "Column3","Column4"]))
  val sqlContext = new org.apache.spark.sql.SQLContext(sc)
  sqlContext.sql(query)
  </code></pre>
  
  <p>Any help would be appreciated. Thank you.</p>