title: scikit learn kernel ridge regression produces very different result using linear kernel than simple linear regression
body: <p>My understanding of the kernel regression is when using linear kernel for ridge regression with no penalty, results should be similar to linear regression. somehow in the toy example linear regression has much better Rsq. part of the problem I think is SK is using sample space for the kernel matrix instead of the smaller of sample and feature space and in this case feature space is much smaller. how do I &quot;fix&quot; the issue or maybe I have done something wrong - a newbie to kernel regression. linear kernel regression has around -3 score and linear regression score is about 50%</p>
  <pre><code>from sklearn.datasets import load_diabetes
  from sklearn.kernel_ridge import KernelRidge
  from sklearn.utils import shuffle
  from sklearn.metrics.pairwise import pairwise_kernels
  import numpy as np
  
  
  diab = load_diabetes()
  X, y = shuffle(diab.data, diab.target)
  X_train, X_test = X[:1000, :], X[100:, :]
  y_train, y_test = y[:1000], y[100:]
  
  # krr = KernelRidge(kernel='polynomial', degree=2, coef0=1)
  # krr = KernelRidge(kernel='rbf', degree=2, coef0=0)
  krr = KernelRidge(kernel='linear', alpha=0.0001)
  # krr = KernelRidge(kernel='laplacian')
  
  kernel_train = np.dot(X_train, X_train.T)  # linear kernel
  
  # krr.fit(kernel_train, y_train)
  krr.fit(X_train, y_train)
  
  #kernel_test = np.dot(X_test, X_train[svc.support_, :].T)
  
  # kernel_test = np.dot(X_test, X_train.T)
  # y_pred = krr.predict(kernel_test)
  y_pred = krr.predict(X_test)
  # krr.score(kernel_test, y_test), krr.score(kernel_train, y_train)
  krr.score(X_test, y_test)
  </code></pre>