title: How to create a new array of substrings from string array column in a spark dataframe
body: <p>I have a spark dataframe. One of the columns is an array type consisting of an array of text strings of varying lengths. I am looking for a way to add a new column that is an array of the unique left 8 characters of those strings.</p>
  <p>df.printSchema()</p>
  <pre><code>root
  (...)
   |-- arr_agent: array (nullable = true)
   |    |-- element: string (containsNull = true)
  </code></pre>
  <p>example data from column &quot;arr_agent&quot;:</p>
  <pre><code>[&quot;NRCANL2AXXX&quot;, &quot;NRCANL2A&quot;]
  [&quot;UTRONL2U&quot;, &quot;BKRBNL2AXXX&quot;, &quot;BKRBNL2A&quot;]
  [&quot;NRCANL2A&quot;]
  [&quot;UTRONL2U&quot;, &quot;REUWNL2A002&quot;, &quot;BKRBNL2A&quot;, &quot;REUWNL2A&quot;, &quot;REUWNL2N&quot;]
  [&quot;UTRONL2U&quot;, &quot;UTRONL2UXXX&quot;, &quot;BKRBNL2A&quot;]
  [&quot;MQBFDEFFYYY&quot;, &quot;MQBFDEFFZZZ&quot;, &quot;MQBFDEFF&quot;  ]
  </code></pre>
  <p>What I need to have in the new column:</p>
  <pre><code>[&quot;NRCANL2A&quot;]
  [&quot;UTRONL2U&quot;, &quot;BKRBNL2A&quot;]
  [&quot;NRCANL2A&quot;]
  [&quot;UTRONL2U&quot;, &quot;BKRBNL2A&quot;, &quot;REUWNL2A&quot;, &quot;REUWNL2N&quot;]
  [&quot;UTRONL2U&quot;, &quot;BKRBNL2A&quot;]
  [&quot;MQBFDEFF&quot;  ]
  </code></pre>
  <p>I already tried to define a udf that does it for me.</p>
  <pre><code>from pyspark.sql import functions as F
  from pyspark.sql import types as T
  
  def make_list_of_unique_prefixes(text_array, prefix_length=8):
      out_arr = set(t[0:prefix_length] for t in text_array)
      return(out_arr)
  
  make_list_of_unique_prefixes_udf = F.udf(lambda x,y=8: make_list_of_unique_prefixes(x,y), T.ArrayType(T.StringType()))
  </code></pre>
  <p>But then calling:</p>
  <pre><code>df.withColumn(&quot;arr_prefix8s&quot;, F.collect_set( make_list_of_unique_prefixes_udf(F.col(&quot;arr_agent&quot;) ))) 
  </code></pre>
  <p>Throws an error
  <code>AnalysisException: grouping expressions sequence is empty,</code></p>
  <p>Any tips would be appreciated.
  thanks</p>