title: Python Pipeline returns NaN Scores When Used in Cross Validation
body: <p>I want to create a pipeline with sklearn including some preprocessing steps and a final step with a model to fit to the data. I use this pipeline to get scores by cross-validation. Later on I would like to use the pipeline in <code>GridSearchCV</code> for parameters optimization.</p>
  <p>As to now, the preprocessing steps include:</p>
  <ul>
  <li>One step in which some colums are dropped, using a <code>ColumnsRemoval()</code> class that I created,</li>
  <li>One step which is specific to each feature type (categorical or numerical). To simplify in the example below I have just included a <code>StandardScaler()</code> for numerical features and <code>OneHotEncoder()</code> for categorical features.</li>
  </ul>
  <p>The problem is that the scores I get are all <code>nan</code>. It runs quite fast and it seems as if empty arrays where being passed into the model:</p>
  <pre class="lang-py prettyprint-override"><code>from sklearn.linear_model import LinearRegression
  from sklearn.preprocessing import StandardScaler, OneHotEncoder
  from sklearn.model_selection import cross_validate
  import numpy as np
  
  # Create random dataframe
  num_data = np.random.random_sample((5,4))
  cat_data = ['good','bad','fair','excellent','bad']
  col_list_stack = ['SalePrice','Id','TotalBsmtSF','GrdLivArea']
  data = pd.DataFrame(num_data, columns = col_list_stack)
  
  data['Quality'] = cat_data
  
  X_train = data.drop(labels = ['SalePrice'], axis = 1)
  y_train = data['SalePrice']
  
  #------------------------------------------------------------#
  # create a custom transformer to remove columns
  class ColumnsRemoval(BaseEstimator, TransformerMixin):
      def __init__(self, skip = False, remove_cols = ['Id','TotalBsmtSF']):
          self._remove_cols = remove_cols
          self._skip = skip
          
      def fit(self, X, y = None):
          return self
                  
      def transform(self, X, y = None):
          if not self._skip:
              return X.drop(labels = self._remove_cols,axis = 1)
          else:
              return X
  
  #------------------------------------------------------------#
  # PIPELINE and cross-validation        
  # Preprocessing steps common to numerical and categorical data
  preprocessor_common = Pipeline(steps=[
      ('remove_features', ColumnsRemoval())])
  
  # Separated preprocessing steps
  numeric_transformer = Pipeline(steps=[
      ('scaler', StandardScaler())])
  
  categorical_transformer = Pipeline(steps=[    
      ('onehot', OneHotEncoder(handle_unknown='ignore'))])
  
  preprocessor_by_cat = ColumnTransformer(
      transformers=[
          ('num', numeric_transformer, ['GrdLivArea']),
          ('cat', categorical_transformer, ['Quality'])], remainder = 'passthrough')
  
  # Full pipeline with model
  pipe = Pipeline(steps = [('preprocessor_common', preprocessor_common),
                      ('preprocessor_by_cat', preprocessor_by_cat),
                    ('model', LinearRegression())])
  
  # Use cross validation to obtain scores
  scores = cross_validate(pipe, X_train, y_train,
              scoring = [&quot;neg_mean_squared_error&quot;,&quot;r2&quot;], cv = 4)
  </code></pre>
  <p>I have tried the following:</p>
  <ul>
  <li>Using only one of the preprocessing steps plus the model in the pipeline. When I use <code>preprocessor_by_cat</code> + <code>model</code> steps in the pipeline, I get score values. Using <code>preprocessor_common</code> + <code>model</code> steps gives <code>nan</code> scores as well</li>
  <li>Perform both preprocessing steps in a pipeline (<code>preprocessor_common</code>+<code>preprocessor_by_cat</code>), <code>.fit_transform()</code> the training data, and then send it to cross_validate(), roughly as below:</li>
  </ul>
  <pre class="lang-py prettyprint-override"><code>pipe = Pipeline(steps = [('preprocessor_common', preprocessor_common),
                              ('preprocessor_by_cat', preprocessor_by_cat),
                        ])
  X_processed = pipe.fit_transform(X_train)
  
  # Use cross validation to obtain scores
  scores = cross_validate(LinearRegression(), X_processed, y_train,
              scoring = [&quot;neg_mean_squared_error&quot;,&quot;r2&quot;], cv = 4)
  </code></pre>
  <p>From my understanding, doing the preprocessing in a pipeline or doing the preprocessing + the model to the pipeline are the same, which is why I believe it is a problem to get <code>NaN</code> values.</p>
  <p>I hope the problem is clear, congratulations if you made it this far :)</p>