title: Remove special character from a column in dataframe
body: <p>I am trying to remove a special character (<strong>å</strong>) from a column in a dataframe.</p>
  
  <p>My data looks like:</p>
  
  <pre><code>ClientID,PatientID 
  AR0001å,DH_HL704221157198295_91
  AR00022,DH_HL704221157198295_92
  </code></pre>
  
  <p>My original data is approx 8TB in size from which I need to get rid of this special character.</p>
  
  <p>Code to load data:</p>
  
  <pre><code>reader.option("header", true)
    .option("sep", ",")
    .option("inferSchema", false)
    .option("charset", "ISO-8859-1")
    .schema(schema)
    .csv(path)
  </code></pre>
  
  <p>After loading into dataframe when I do <code>df.show()</code> it shows:</p>
  
  <pre><code>+--------+--------------------+
  |ClientID|           PatientID|
  +--------+--------------------+
  |AR0001Ã¥|DH_HL704221157198...|
  |AR00022 |DH_HL704221157198...|
  +--------+--------------------+
  </code></pre>
  
  <p>Code I used to try to replace this character:</p>
  
  <pre><code>df.withColumn("ClientID", functions.regexp_replace(df.col("ClientID"), "\å", ""));
  </code></pre>
  
  <p>But this didn't work. While loading the data in dataframe if I change the charset to "UTF-8" it works.</p>
  
  <p>I am not able to find a solution with the current charset (ISO-8859-1).</p>