title: Converting domain names to idn in python
body: <p>I have a long list of domain names which I need to generate some reports on. The list contains some IDN domains, and although I know how to convert them in python on the command line:</p>
  
  <pre><code>&gt;&gt;&gt; domain = u"pfarmerü.com"
  &gt;&gt;&gt; domain
  u'pfarmer\xfc.com'
  &gt;&gt;&gt; domain.encode("idna")
  'xn--pfarmer-t2a.com'
  &gt;&gt;&gt; 
  </code></pre>
  
  <p>I'm struggling to get it to work with a small script reading data from the text file.</p>
  
  <pre><code>#!/usr/bin/python
  
  import sys
  
  infile = open(sys.argv[1])
  
  for line in infile:
      print line,
      domain = unicode(line.strip())
      print type(domain)
      print "IDN:", domain.encode("idna")
      print
  </code></pre>
  
  <p>I get the following output:</p>
  
  <pre><code>$ ./idn.py ./test 
  pfarmer.com
  &lt;type 'unicode'&gt;
  IDN: pfarmer.com
  
  pfarmerü.com
  Traceback (most recent call last):
    File "./idn.py", line 9, in &lt;module&gt;
      domain = unicode(line.strip())
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 7: ordinal not in range(128)
  </code></pre>
  
  <p>I have also tried:</p>
  
  <pre><code>#!/usr/bin/python
  
  import sys
  import codecs
  
  infile = codecs.open(sys.argv[1], "r", "utf8")
  
  for line in infile:
      print line,
      domain = line.strip()
      print type(domain)
      print "IDN:", domain.encode("idna")
      print
  </code></pre>
  
  <p>Which gave me:</p>
  
  <pre><code>$ ./idn.py ./test       
  Traceback (most recent call last):
    File "./idn.py", line 8, in &lt;module&gt;
      for line in infile:
    File "/usr/lib/python2.6/codecs.py", line 679, in next
      return self.reader.next()
    File "/usr/lib/python2.6/codecs.py", line 610, in next
      line = self.readline()
    File "/usr/lib/python2.6/codecs.py", line 525, in readline
      data = self.read(readsize, firstline=True)
    File "/usr/lib/python2.6/codecs.py", line 472, in read
      newchars, decodedbytes = self.decode(data, self.errors)
  UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-5: unsupported Unicode code range
  </code></pre>
  
  <p>Here is my test data file:</p>
  
  <pre><code>pfarmer.com
  pfarmerü.com
  </code></pre>
  
  <p>I'm very aware of my need to understand unicode now.</p>
  
  <p>Thanks,</p>
  
  <p>Peter</p>