title: Pandas DataFrame incompatible with sci-kit fit_transform() function
body: <p>So I have created a classifier that distinguishes fraudulent messages from genuine messages. Snippet of the code is as follows:</p>
  
  <pre><code># Import training set as DataFrame from CSV
  dataset = pd.read_csv('data.csv', sep=',')
  class_names = { 1: 'no-flag', 2: 'flag' }
  
  # Separate training data to message, class pairs
  X_train, y_train = dataset.iloc[:,0], dataset.iloc[:, 1]
  
  messages = pd.read_csv('messages.csv', header=None)
  X_predict = messages.iloc[:,0]
  
  print "TRAIN:\n"
  print type(X_train)
  print "PREDICT:\n"
  print type(X_predict)
  
  # Vectorise text data
  vect = TfidfVectorizer(ngram_range=(1, 2), lowercase=True, preprocessor=sanitise_message)
  X_train_tfidf = vect.fit_transform(X_train)
  X_predict_tfidf = vect.transform(X_predict)
  </code></pre>
  
  <p>I used to run this with ten-fold cross validation on the training set, using:</p>
  
  <pre><code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=1)
  </code></pre>
  
  <p>And that used to work fine. Now I want to use the entire training set as training data, and predict unclassified data. However, the call to <code>X_predict_tfidf = vect.transform(X_predict)</code> throws an error, as follows:</p>
  
  <pre><code>Traceback (most recent call last):
  File "post-test.py", line 3, in &lt;module&gt;
  classify()
  File "/Users/user/Documents/MyTutor/mi_datawarehouse/classifier.py", line 90, in classify
  X_predict_tfidf = vect.transform(X_predict)
  File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1409, in transform
  X = super(TfidfVectorizer, self).transform(raw_documents)
  File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 923, in transform
  _, X = self._count_vocab(raw_documents, fixed_vocab=True)
  File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 792, in _count_vocab
  for feature in analyze(doc):
  File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 266, in &lt;lambda&gt;
  tokenize(preprocess(self.decode(doc))), stop_words)
  File "/Users/user/miniconda2/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 119, in decode
  raise ValueError("np.nan is an invalid document, expected byte or "
  ValueError: np.nan is an invalid document, expected byte or unicode string.
  </code></pre>
  
  <p>What is interesting is that the types of both X_train and X_predict are identical:</p>
  
  <pre><code>TRAIN:
  &lt;class 'pandas.core.series.Series'&gt;
  PREDICT:
  &lt;class 'pandas.core.series.Series'&gt;
  </code></pre>
  
  <p>What am I doing wrong? I've been going crazy over this as I've looked everywhere, including the <code>scikit-learn</code> docs.</p>
  
  <p>NOTE: this is NOT a duplicate of <a href="https://stackoverflow.com/questions/39303912/tfidfvectorizer-in-scikit-learn-valueerror-np-nan-is-an-invalid-document">a similar question</a>, I have tried everything in that question and nothing worked. The data structures and problem are slightly different.</p>