title: concatenate columns and selecting some columns in Pyspark data frame
body: <p>I have a data frame like below in <code>pyspark</code></p>
  
  <pre><code>data = [
  (1, 'a', '', 'b', '', 'c', '123_abc', 'sam', 'NY'),
  (2, 'b', 'abc_123', 'd', '', 'e', '', 'Tim', 'NJ'),
  (3, 'c', '', 'f', '', 'g', '', 'Jim', 'SFO')] 
  
  df = sc.parallelize(data).toDF(["id", "abc_abled", "abc_serial", "bca_abled", "bca_serial", "cca_abled", "cca_serial", "name", "city"])
  
  
  
  df
  DataFrame[id: int, abc_abled: string, abc_serial: string, bca_abled: string, bca_serial: string, cca_abled: string, cca_serial: string, name: string, city: string]
  
  df.show()
  
  +---+---------+----------+---------+----------+---------+----------+----+----+
  | id|abc_abled|abc_serial|bca_abled|bca_serial|cca_abled|cca_serial|name|city|
  +---+---------+----------+---------+----------+---------+----------+----+----+
  |  1|        a|      null|        b|      null|        c|   123_abc| sam|  NY|
  |  2|        b|   abc_123|        d|      null|        e|      null| Tim|  NJ|
  |  3|        c|      null|        f|      null|        g|      null| Jim| SFO|
  +---+---------+----------+---------+----------+---------+----------+----+----+
  </code></pre>
  
  <p>I want to create a new data frame by selecting the following columns and <code>concatenate certain column values</code>.</p>
  
  <pre><code>df1
  DataFrame[id:int, serial_number: string, name:string, city:string]
  
  df1.show()
  
  +---+-------------+----------+
  | id|serial_number|name| city|
  +---+-------------+----------+
  |  1|      123_abc| sam|   NY|
  |  2|      abc_123| Tim|   NJ|
  |  3|             | Jim|  SFO|
  +---+-------------+----+-----+
  </code></pre>
  
  <p>Here <code>serial_number</code> will be <code>all columns that end with _serial</code> concatenated. <code>If the columns have null values ignore the values and concatenate other strings</code></p>
  
  <p>How can I achieve that?</p>