title: What is proper Tokenization algorithm? & Error: TypeError: coercing to Unicode: need string or buffer, list found
body: <p>I'm doing an Information Retrieval Task. As part of pre-processing I want to doing. </p>
  
  <ol>
  <li>Stopword removal</li>
  <li>Tokenization</li>
  <li>Stemming (Porter Stemmer)</li>
  </ol>
  
  <p>Initially, I skipped tokenization. As a result I got terms like this:</p>
  
  <pre><code>broker
  broker'
  broker,
  broker.
  broker/deal
  broker/dealer'
  broker/dealer,
  broker/dealer.
  broker/dealer;
  broker/dealers),
  broker/dealers,
  broker/dealers.
  brokerag
  brokerage,
  broker-deal
  broker-dealer,
  broker-dealers,
  broker-dealers.
  brokered.
  brokers,
  brokers.
  </code></pre>
  
  <p>So, Now I realized importance of tokenization. Is there any standard algorithm for tokenization for English language? Based on <code>string.whitespace</code> and commonly used puncuation marks. I wrote </p>
  
  <pre><code>def Tokenize(text):
      words = text.split(['.',',', '?', '!', ':', ';', '-','_', '(', ')', '[', ']', '\'', '`', '"', '/',' ','\t','\n','\x0b','\x0c','\r'])    
      return [word.strip() for word in words if word.strip() != '']
  </code></pre>
  
  <ol>
  <li>I'm getting <code>TypeError: coercing to Unicode: need string or buffer, list found</code> error! </li>
  <li>How can this Tokenization routine be improved?</li>
  </ol>