title: Extracting Words using nltk from German Text
body: <p>I am trying to extract words from a german document, when I use th following method as described in the nltk tutorial, I fail to get the words with language specific special characters.</p>
  
  <pre><code>ptcr = nltk.corpus.PlaintextCorpusReader(Corpus, '.*');
  words = nltk.Text(ptcr.words(DocumentName))
  </code></pre>
  
  <p>What should I do to get the list of words in the document?</p>
  
  <p>An example with <code>nltk.tokenize.WordPunctTokenizer()</code> for the german phrase <code>Veränderungen über einen Walzer</code> looks like:</p>
  
  <pre><code>In [231]: nltk.tokenize.WordPunctTokenizer().tokenize(u"Veränderungen über einen Walzer")
  
  Out[231]: [u'Ver\xc3', u'\xa4', u'nderungen', u'\xc3\xbcber', u'einen', u'Walzer']
  </code></pre>
  
  <p>In this example "ä" is treated as a delimiter,even though "ü" is not. </p>