title: How concatenate Two array in pyspark
body: <p>I have a pyspark Dataframe.</p>
  
  <p>Example:</p>
  
  <pre><code>ID   |    phone   |  name &lt;array&gt;  | age &lt;array&gt;
  -------------------------------------------------
  12   | 827556     | ['AB','AA']    |  ['CC']
  -------------------------------------------------
  45   |  87346     |  null          |   ['DD']
  -------------------------------------------------
  56   |  98356     |  ['FF']        |  null
  -------------------------------------------------
  34   |  87345     |   ['AA','BB']  |  ['BB']
  </code></pre>
  
  <hr>
  
  <p>I want to concatenate the 2 arrays name and age.
  I did it like this:</p>
  
  <pre><code>df = df.withColumn("new_column", F.concat(df.name, df.age))
  df = df.select("ID", "phone", "new_column")
  </code></pre>
  
  <p>But I got some missing columns, it seems the <code>concat function</code> works on a String not on an array and remove the duplicates:</p>
  
  <p>Result expected:</p>
  
  <pre><code>ID   |    phone   |  new_column &lt;array&gt;  
  ----------------------------------------
  12   | 827556     | ['AB','AA','CC']    
  ----------------------------------------
  45   |  87346     |  ['DD']             
  ----------------------------------------
  56   |  98356     |  ['FF']        
  ----------------------------------------
  34   |  87345     |   ['AA','BB']    
  ----------------------------------------
  </code></pre>
  
  <p>How can I concatenate 2 arrays in pyspark knowing that I'm using <code>Spark version &lt; 2.4</code></p>
  
  <p>Thank you</p>