title: How do I remove non-ascii characters (e.g Ð±§•¿µ´‡»Ž®ºÏƒ¶¹) from texts in pandas dataframe columns?
body: <p>How do I remove non-ascii characters (e.g Ð±§•¿µ´‡»Ž®ºÏƒ¶¹) from texts in pandas dataframe columns?</p>
  
  <p>I have tried the following but no luck</p>
  
  <pre><code>df = pd.read_csv(path, index_col=0)
  for col in df.columns:
  for j in df.index:
      markup1 = str(df.ix[j, col]).replace("\r", "")
      markup1 = markup1.replace("\n", "")
      markup1 = markup1.decode('unicode_escape').encode('ascii','ignore').strip()
  soup = BeautifulSoup(markup1, 'lxml')
  df.ix[j, col] = soup.get_text()
  print df.ix[j, 'requirements']
  </code></pre>
  
  <p>I tried using regex yet it wouldn't work.</p>
  
  <pre><code>markup1 = str(df.ix[j, 'requirements']).replace("\r", "")
  markup1 = markup1.replace("\n", "")
  markup1 = re.sub(r'[^\x00-\x7F]+', ' ', markup1)
  </code></pre>
  
  <p>I still keep getting the non-ascii characters. Any suggestion would be appreciated.</p>
  
  <p>I have added the first three rows of the df below:</p>
  
  <pre><code>                                              col1               col2  \
  1.0                          H1B SPONSOR FOR L1/L2/OPT  US, NY, New York
  2.0                             Graphic / Web Designer     US, TX, Austin
  3.0  Full Stack Developer (.NET or equivalent + Jav...             GR, ,
  
                  col3  col4  \
  1.0                  NaN   NaN
  2.0  Sales and Marketing   NaN
  3.0                  NaN   NaN
  
                                                col5  \ 
  1.0  i28 Technologies has demonstrated expertise in...
  2.0  outstanding people who believe that more is po...
  3.0                                                NaN
  
                                                col6  \
  1.0  Hello,Wish you are doing good...              ...
  2.0  The Graphic / Web Designer will manage, popula...
  3.0   You?ll have to join the Moosend dojo. But, yo...
  
                                                col7  \
  1.0  JAVA, .NET, SQL, ORACLE, SAP, Informatica, Big...
  2.0  Bachelor?s degree in Graphic Design, Web Desig...
  3.0  ? .NET or equivalent (Java etc.)? MVC? Javascr...
  
                                                col8 col9
  1.0                                                NaN    f
  2.0  CSD offers a competitive benefits package for ...    f
  3.0  You?ll be working with the best team in town.....    f
  </code></pre>