title: Python Regular Expression nltk website extraction
body: <p>Hi I have never had to deal with regex before and I'm trying to preprocess some raw text with Python and NLTK. 
  when I tried to tokenize the document using :</p>
  
  <pre><code>tokens = nltk.regexp_tokenize(corpus, sentence_re)
  sentence_re = r'''(?x)  # set flag to allow verbose regexps
    ([A-Z])(\.[A-Z])+\.?  # abbreviations, e.g. U.S.A.
  | \w+(-\w+)*            # words with optional internal hyphens
  | \$?\d+(\.\d+)?%?      # currency and percentages, e.g. $12.40, 82%
  | \#?\w+|\@?\w+         # hashtags and @ signs
  | \.\.\.                # ellipsis
  | [][.,;"'?()-_`]       # these are separate tokens
  | ?:http://|www.)[^"\' ]+ # websites
  '''
  </code></pre>
  
  <p>its not able to take all of the website as one single token:</p>
  
  <pre><code>print toks[:50]
  ['on', '#Seamonkey', '(', 'SM', ')', '-', 'I', 'had', 'a', 'short', 'chirp',   'exchange', 'with', '@angie1234p', 'at', 'the', '18thDec', ';', 'btw', 'SM', 'is', 'faster', 'has', 'also', 'an', 'agile', '...', '1', '/', '2', "'", '...', 'user', 'community', '-', 'http', ':', '/', '/', 'bit', '.', 'ly', '/', 'XnF5', '+', 'ICR', 'http', ':', '/', '/']
  </code></pre>
  
  <p>any help is greatly appreicated. Thanks so much!</p>
  
  <p>-Florie</p>