title: How to save classifier in sklearn with Countvectorizer() and TfidfTransformer()
body: <p>Below is some code for a classifier. I used pickle to save and load the classifier instructed in this <a href="https://stackoverflow.com/questions/10592605/save-classifier-to-disk-in-scikit-learn">page</a>. However, when I load it to use it, I cannot use the <code>CountVectorizer()</code> and <code>TfidfTransformer()</code> to convert raw text into vectors that the classifier can use.</p>
  <p>The only I was able to get it to work is analyze the text immediately after training the classifier, as seen below.</p>
  <pre><code>import os
  import sklearn
  from sklearn.datasets import load_files
  
  from sklearn.feature_extraction.text import TfidfTransformer
  from sklearn.model_selection import train_test_split
  from sklearn.naive_bayes import MultinomialNB
  from sklearn.metrics import confusion_matrix
  
  from sklearn.feature_extraction.text import CountVectorizer
  import nltk
  
  import pandas
  import pickle
  
  class Classifier:
  
      def __init__(self):
  
          self.moviedir = os.getcwd() + '/txt_sentoken'
  
      def Training(self):
  
          # loading all files. 
          self.movie = load_files(self.moviedir, shuffle=True)
  
  
          # Split data into training and test sets
          docs_train, docs_test, y_train, y_test = train_test_split(self.movie.data, self.movie.target, 
                                                                    test_size = 0.20, random_state = 12)
  
          # initialize CountVectorizer
          self.movieVzer = CountVectorizer(min_df=2, tokenizer=nltk.word_tokenize, max_features=5000)
  
          # fit and tranform using training text 
          docs_train_counts = self.movieVzer.fit_transform(docs_train)
  
  
          # Convert raw frequency counts into TF-IDF values
          self.movieTfmer = TfidfTransformer()
          docs_train_tfidf = self.movieTfmer.fit_transform(docs_train_counts)
  
          # Using the fitted vectorizer and transformer, tranform the test data
          docs_test_counts = self.movieVzer.transform(docs_test)
          docs_test_tfidf = self.movieTfmer.transform(docs_test_counts)
  
          # Now ready to build a classifier. 
          # We will use Multinominal Naive Bayes as our model
  
  
          # Train a Multimoda Naive Bayes classifier. Again, we call it &quot;fitting&quot;
          self.clf = MultinomialNB()
          self.clf.fit(docs_train_tfidf, y_train)
  
  
          # save the model
          filename = 'finalized_model.pkl'
          pickle.dump(self.clf, open(filename, 'wb'))
  
          # Predict the Test set results, find accuracy
          y_pred = self.clf.predict(docs_test_tfidf)
  
          # Accuracy
          print(sklearn.metrics.accuracy_score(y_test, y_pred))
  
          self.Categorize()
  
      def Categorize(self):
          # very short and fake movie reviews
          reviews_new = ['This movie was excellent', 'Absolute joy ride', 'It is pretty good', 
                        'This was certainly a movie', 'I fell asleep halfway through', 
                        &quot;We can't wait for the sequel!!&quot;, 'I cannot recommend this highly enough', 'What the hell is this shit?']
  
          reviews_new_counts = self.movieVzer.transform(reviews_new)         # turn text into count vector
          reviews_new_tfidf = self.movieTfmer.transform(reviews_new_counts)  # turn into tfidf vector
  
  
          # have classifier make a prediction
          pred = self.clf.predict(reviews_new_tfidf)
  
          # print out results
          for review, category in zip(reviews_new, pred):
              print('%r =&gt; %s' % (review, self.movie.target_names[category]))
  </code></pre>