title: Select not null values in dataframes in Spark
body: <p>I'm reading a CSV file in Spark 2.0 and counting not null values in a column using the following:</p>
  
  <pre><code>val df = spark.read.option("header", "true").csv(dir)
  
  df.filter("IncidntNum is not null").count()
  </code></pre>
  
  <p>and it works fine when I test it using spark-shell. When I create a jar file containing the code and submit it to spark-submit, I get an exception at second line above :</p>
  
  <pre><code>Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
  extraneous input '' expecting {'(', 'SELECT', ..
  == SQL ==
  IncidntNum is not null
  ^^^
  
          at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:197)
  </code></pre>
  
  <p>Any idea why this would happen when I'm using the code that works in spark-shell?</p>