title: PySpark drop columns based on column names / String condition
body: <p>I want to drop columns in a pyspark dataframe that contains any of the words in the <code>banned_columns</code> list and form a new dataframe out of the remaining columns</p>
  
  <pre><code>banned_columns = ["basket","cricket","ball"]
  drop_these = [columns_to_drop for columns_to_drop in df.columns if columns_to_drop in banned_columns]
  
  df_new = df.drop(*drop_these)
  </code></pre>
  
  <p>The idea of <code>banned_columns</code> is to drop any columns that start with <code>basket</code> and <code>cricket</code>, and columns that contain the word <code>ball</code> anywhere in their name.</p>
  
  <p>The above is what I did so far, but it does not work (as in the new dataframe still contains those columns names)</p>
  
  <p>Example of dataframe</p>
  
  <pre><code> sports1basketjump | sports
  </code></pre>
  
  <p>In the above column name example, it will drop the column <code>sports1basketjump</code> because it contains the word basket.</p>
  
  <p>Moreover, is using the <code>filter</code> or/and <code>reduce</code> functions adds optimization than creating list and for loops?</p>