title: Spark - Scala - Remove Columns from a dataframe based on condition
body: <p>We have a specific need wherein I will have to <code>drop</code> columns from a <code>dataframe</code> which has only one unique value in that column. The following is what we are doing</p>
  
  <pre><code>val rawdata = spark.read.format("csv").option("header","true").option("inferSchema","true").load(filename)
  </code></pre>
  
  <p>Subsequently to find unique values in all columns we are using the <code>HyperLog++</code> algorithm supported in spark </p>
  
  <pre><code>val cd_cols  = rawdata.select(rawdata.columns.map(column =&gt; approxCountDistinct(col(column)).alias(column)): _*)
  </code></pre>
  
  <p>The output is</p>
  
  <pre><code>scala&gt; cd_cols.show
  +----+----------+---------+---+---------+--------------+---------+----------+----------------+---------+--------------+-------------+
  |  ID|First Name|Last Name|Age|Attrition|BusinessTravel|DailyRate|Department|DistanceFromHome|Education|EducationField|EmployeeCount|
  +----+----------+---------+---+---------+--------------+---------+----------+----------------+---------+--------------+-------------+
  |1491|       172|      154| 43|        2|             3|      913|         3|              30|        1|             6|            1|
  +----+----------+---------+---+---------+--------------+---------+----------+----------------+---------+--------------+-------------+
  </code></pre>
  
  <p>Notice that I have two columns which has a 1 as the unique value. I want to create another <code>dataframe</code> which has all columns except those two columns (<code>Education</code> and <code>EmployeeCount</code>) </p>
  
  <p>I tried using a for loop, but was not very happy and also tried </p>
  
  <pre><code>cd_cols.columns.filter(colName =&gt; cd_cols.filter(colName) &lt;= 1)
  </code></pre>
  
  <p>that is also not working.</p>
  
  <p>Is there a smarter way to do this please.</p>
  
  <p>Thanks</p>
  
  <p>Bala</p>