title: In Spark data frame, add value to new column if value in another column is in broadcast variable array
body: <p>I have an array as a broadcast variable and it contains Integers: </p>
  
  <pre><code>broadcast_array.value
  Array(72159153, 72159163, 72159202, 72159203, 72159238, 72159398, 72159447, 72159448, 72159455, 72159492...
  </code></pre>
  
  <p>I have a column in a dataset (call is <code>col_id</code> which contains <code>IntegerType</code> values that might be in the <code>broadcast_array</code>, but they might not.</p>
  
  <p>I am only trying to create a new column (call it <code>new_col</code>) that checks if <code>col_id</code> value for each row is in <code>broadcast_array</code>. If so, the new column value should be <code>Available</code>, else it can be <code>null</code></p>
  
  <p>So I have something like:</p>
  
  <pre><code>val my_new_df = df.withColumn("new_col", when(broadcast_array.value.contains($"col_id"), "Available"))
  </code></pre>
  
  <p>But I keep getting this error: </p>
  
  <pre><code>Name: Unknown Error
  Message: &lt;console&gt;:45: error: type mismatch;
   found   : Boolean
   required: org.apache.spark.sql.Column
     val my_new_df = df.withColumn("new_col", when(broadcast_array.value.contains($"col_id"), "Available"))
                                                                                             ^
  StackTrace: 
  </code></pre>
  
  <p>What is most confusing to me is that I thought the <code>when</code> statement requires a conditional that outputs some Boolean, but here it's saying it requires a Column. </p>
  
  <p>How should I go about adding a value to a new column based on whether the value in an existing column can be found in a predefined Array or not?</p>