title: Is there some fixed overhead running time to sklearn transform?
body: <p>I was hoping to use the PCA module in sklearn to do some real-time transformation on user features in our app (for a spam detection algorithm). The lead of our team has specified that the model would need to make a prediction under 100 ms (preferably 10-50 ms). So far in my timing tests, I'm getting about 150 ms. This is with very small amount of data (i.e. the minimum amount I could conceivably use in a real application). Just wondering if anyone has come upon this issue and if so, what can I do about it? At some point, would I need to take the plunge and roll my own PCA? My first thought was that if sklearn can't do it that quickly, I can't imagine my naive code would be able to do it any faster. Is Python just to limited performance-wise for a real-time application like this?</p>
  
  <p>To build the model, there is a set of transformers combined in a feature union:</p>
  
  <pre><code>    transformer_list=[
          ('age', Pipeline([
              ('selector', ColumnSelector(column='birthdate')),
              ('birthdate', BirthdateTransformer()),
              ('vect', DictVectorizer(sparse=False))
          ])),
          ('domain', Pipeline([
              ('selector', ColumnSelector(column='email')),
              ('counter', EmailDomainName()),
              ('vect', DictVectorizer(sparse=False))])),
          ('sex', Pipeline([
              ('selector', ColumnSelector(column='sex')),
              ('dict', DictTransformer()),
              ('vect', DictVectorizer(sparse=False))
          ]))],
      n_jobs=2,
      transformer_weights=weights)
  </code></pre>
  
  <p>The only custom classes there are BirthdateTransformer and EmailDomainName:</p>
  
  <pre><code>class BirthdateTransformer(TransformerMixin):
      """
      takes list of Unix timestamps (seconds since epoch) and converts to list of ages in years
      """
  
      def fit(self, dates, y=None):
          return self
  
      def transform(self, timestamps, y=None):
          return [{'age group': age_group(ts / 1000)} for ts in timestamps]
  
  class EmailDomainName(TransformerMixin):
      """
      Class for building sklearn Pipeline step. This class takes a list of email addresses and returns a list of dicts
      created by the Counter class from the collections module.
      """
  
      regex = re.compile(r"@[\w.]+")
  
      def pre_filter(self, email_list):
  
          for e in email_list:
              match = self.regex.search(e)
  
              if match is not None:
                  yield match.group()
              else:
                  yield ""
  
      def fit(self, x, y=None):
          return self
  
      def transform(self, email_list):
          """
          Use regular expression to pull out domain name from email address and convert to list of dicts containing
          count of characters.
          :param email_list:
          :return:
          """
  
          return [{k: v for (k, v) in Counter(result).most_common()} for result in
                  self.pre_filter(email_list)]
  </code></pre>
  
  <p>Then I simply pass in a Pandas dataframe with the data and call transform() on the data to be projected into PCA space.</p>