title: Why isn't the token_pattern parameter in Tfidfvectorizer working with scikit learn?
body: <p>I have this text:</p>
  
  <pre><code>data = ['Hi, this is XYZ and XYZABC is $$running']
  </code></pre>
  
  <p>I am using the following tfidfvectorizer:</p>
  
  <pre><code>vectorizer = TfidfVectorizer(
              stop_words='english',
              use_idf=False, 
              norm=None,
              min_df=1,
              tokenizer = tokenize,
              ngram_range=(1, 1),
              token_pattern=u'\w{4,}')
  </code></pre>
  
  <p>I am fitting the data as follows:</p>
  
  <pre><code>tdm =vectorizer.fit_transform(data)
  </code></pre>
  
  <p>Now, when I print </p>
  
  <pre><code>vectorizer.get_feature_names()
  </code></pre>
  
  <p>I get this:</p>
  
  <pre><code>[u'hi', u'run', u'thi', u'xyz', u'xyzabc']
  </code></pre>
  
  <p>My question is why am I getting 'hi' and 'xyz' even thought I mentioned that I want it to capture only words that have at least 4 characters? - token_pattern=u'\w{4,}'</p>