title: How do I correctly use PCA followed by Logistic Regression?
body: <p>In the program, I am scanning a number of brain samples taken in a time series of 40 x 64 x 64 images every 2.5 seconds. The number of 'voxels' (3D pixels) in each image is thus ~ 168,000 ish (40 * 64 * 64), each of which is a 'feature' for an image sample. </p>
  
  <p>I thought of using Principle Component Analysis (PCA) because of the rediculously high <code>n</code> to perform dimensionality reduction. I am aware PCA does not actually reduce the number of features. Correct me if I'm wrong, but PCA will produce a new set of features  from the original ones. The new features however must follow certain conditions.</p>
  
  <p>I definined a method to get the number of components :</p>
  
  <pre><code>def get_optimal_number_of_components():
      cov = np.dot(X,X.transpose())/float(X.shape[0])
      U,s,v = svd(cov)
  
      S_nn = sum(s)
  
      for num_components in range(0,s.shape[0]):
          temp_s = s[0:num_components]
          S_ii = sum(temp_s)
          if (1 - S_ii/float(S_nn)) &lt;= 0.01:
              return num_components
  
      return s.shape[0]
  </code></pre>
  
  <p>This function will return the number of components such that 99% of the variance from the original data is retained. Now, we can create these components :</p>
  
  <pre><code>#Scaling the values
  X = scale(X)
  
  n_comp = get_optimal_number_of_components()
  print 'optimal number of components = ', n_comp
  
  pca = PCA(n_components = n_comp)
  X_new = pca.fit_transform(X)
  </code></pre>
  
  <p>I get the optimal number of components = 1001 on running the program on this dataset. This number agrees with the plot I obtained on executing :</p>
  
  <pre><code>#Cumulative Variance explains
  var1 = np.cumsum(np.round(pca.explained_variance_ratio_, decimals=4)*100)
  
  plt.plot(var1)
  plt.title('Principle Component Analysis for Feature Selection')
  plt.ylabel('Percentage of variance')
  plt.xlabel('Number of voxels considered')
  
  plt.show()
  </code></pre>
  
  <p>After this PCA stage is complete, I used the newly created 'X_new' in place of X for the next stage : Logistic Regression</p>
  
  <pre><code>#After PCA
  from sklearn.cross_validation import train_test_split
  X_train, X_test, y_train, y_test = train_test_split(X_new, y, test_size=0.10, random_state=42)
  
  classifier = LogisticRegression()
  classifier.fit(X_train,y_train)
  </code></pre>
  
  <p>When I test for the accuracy, I get around <strong>77.57%</strong></p>
  
  <p>But this is less than when I just analysed the middle voxel samples (like 9K samples in the middle of brain image). I was wondering if I pipelined PCA and logistic regression correctly.</p>
  
  <p>I even tried this in another method using <code>sklearn.pipeline</code> :</p>
  
  <pre><code>pipe = Pipeline(steps=[('pca', pca), ('logistic', classifier)])
  
  pipe.set_params(pca__n_components = n_comp).fit(X_train,y_train)
  
  print 'score = ', pipe.score(X_test,y_test)
  </code></pre>
  
  <p>But I got the exact same accuracy of 77.57%. Am I implementing PCA + Logistic Regression Correctly? There must be something wrong, I just can't figure out what it is. </p>