title: SVR hyperparameter selection and visualisation
body: <p>I am just a beginner in data analysis. I want to use 'Cross-validation Grid Search method&quot; to determine the parameters gamma and C of the Radial Basis Function (RBF) kernel SVM. I don't know where I should put my data on this code, and what data type I should use (training or target data)?</p>
  <p>For SVR</p>
  <pre><code>import numpy as np
  import pandas as pd
  from math import sqrt
  from sklearn.tree import DecisionTreeRegressor
  import matplotlib.pyplot as plt
  from sklearn.ensemble import AdaBoostRegressor
  from sklearn.metrics import mean_squared_error,explained_variance_score
  from TwoStageTrAdaBoostR2 import TwoStageTrAdaBoostR2 # import the two-stage algorithm
  from sklearn import preprocessing
  from sklearn import svm
  from sklearn.svm import SVR
  from sklearn.model_selection import GridSearchCV
  from sklearn.metrics import classification_report
  from matplotlib.colors import Normalize
  from sklearn.svm import SVC
  
  # Data import (source)
  source= pd.read_csv(sourcedata)
  
  # Data import (target)
  data= pd.read_csv(targetdata)
  
  # Sample Size
  datatrain = data.sample(n=60, random_state=1)
  datatest = data[~dataL.index.isin(data.index)]
  
  # Merge training set data (source and target)
  train = pd.concat([source, datatrain], sort=False)
  train.reset_index(inplace=True, drop=True)
  datatest.reset_index(inplace=True, drop=True)
  
  # Variable input
  X_train, y_train = train[['x1', 'x2']].values, train['y'].values
  X_test, y_test = FL[['x1', 'x2']].values, FL['y'].values
  
  # Parameter setting
  #sample_size = [n_source1+n_source2+n_source3+n_source4+n_source5, n_target_train]
  n_estimators = 100
  steps = 8
  fold = 5
  random_state = np.random.RandomState(1)
  sample_size = [350, 60]
  
  #1  twostage tradaboost.r2
  regr_1 = TwoStageTrAdaBoostR2(SVR(C=50, gamma='auto'),
                        n_estimators = n_estimators, sample_size = sample_size,
                        steps = steps, fold = fold,
                        random_state = random_state)
  regr_1.fit(X_train, y_train)
  y_pred1 = regr_1.predict(X_test)
  print(&quot;MSE of regular two stage trAdaboostR2--model1:&quot;,sqrt(mean_squared_error(y_test, y_pred1)))
  
  
  #Plot the results
  plt.figure()
  plt.scatter(y_test, y_test-y_pred1, c=&quot;black&quot;, label=&quot;TwoStageTrAdaBoostR2_model1&quot;, s=10)
  plt.xlabel(&quot;CAR&quot;)
  plt.ylabel(&quot;Err&quot;)
  plt.title(&quot;Two-stage Transfer Learning Boosted Decision Tree Regression&quot;, loc='left', fontsize=12, fontweight=0, color=&quot;orange&quot;)
  plt.legend()
  plt.show()
  </code></pre>
  <p>for cross-validation grid search methods(best parameters):</p>
  <pre><code># Cross validation grid search (best parameters) 
  parameter_candidates = [
    {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['linear']},
    {'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
  ]
  svr = svm.SVC()
  clf = grid_search.GridSearchCV(svr, parameters, c=5 ,n_jobs=-1)
  clf.fit(X_train, y_train)
  print('Best score for data:', clf.best_score_)
  print('Best C:',clf.best_estimator_.C) 
  print('Best Kernel:',clf.best_estimator_.kernel)
  print('Best Gamma:',clf.best_estimator_.gamma)
  </code></pre>
  <p>For visualization of parameter effects</p>
  <pre><code>c_range = np.logspace(-2, 2, 4)
  gamma_range = np.logspace(-2, 2, 5)
  tuned_parameters = [{'kernel': ['rbf'],'C': c_range,'gamma':gamma_range},
                      {'kernel': ['linear'], 'C': c_range,'gamma':gamma_range}]
  
  svr = svm.SVR()
  clf = GridSearchCV(svr,param_grid=tuned_parameters,verbose=2,n_jobs=-1,
                     scoring='explained_variance')
  clf.fit(X_train, y_train)
  
  print('Best score for data:', clf.best_score_)
  print('Best C:',clf.best_estimator_.C) 
  print('Best Kernel:',clf.best_estimator_.kernel)
  print('Best Gamma:',clf.best_estimator_.gamma)
  
  # scores for rbf kernel
  n = len(gamma_range)*len(c_range)
  scores_rbf = clf.cv_results_['mean_test_score'][:n].reshape(len(gamma_range),
                                                              len(c_range))
  
  # scores for rbf kernel
  scores_linear = clf.cv_results_['mean_test_score'][n:].reshape(len(gamma_range),
                                                                 len(c_range))
  
  class MidpointNormalize(Normalize):
  
      def __init__(self, vmin=None, vmax=None, midpoint=None, clip=False):
          self.midpoint = midpoint
          Normalize.__init__(self, vmin, vmax, clip)
  
      def __call__(self, value, clip=None):
          x, y = [self.vmin, self.midpoint, self.vmax], [0, 0.5, 1]
          return np.ma.masked_array(np.interp(value, x, y))
  
  
  
  plt.figure(figsize=(8, 6))
  plt.subplots_adjust(left=.2, right=0.95, bottom=0.15, top=0.95)
  plt.imshow(scores_rbf, interpolation='nearest', cmap=plt.cm.hot,
             norm=MidpointNormalize(vmin=0.2, midpoint=0.92))
  plt.xlabel('gamma')
  plt.ylabel('C')
  plt.colorbar()
  plt.xticks(np.arange(len(gamma_range)), gamma_range, rotation=45)
  plt.yticks(np.arange(len(c_range)), c_range)
  plt.title('Validation accuracy')
  plt.show()
  </code></pre>
  <p>When I used this code, I found the following output Heatmap plot!
  <a href="https://i.stack.imgur.com/xF57u.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xF57u.png" alt="enter image description here" /></a></p>
  <p>But I am trying to get a Heatmap like this one
  <a href="https://i.stack.imgur.com/ankYU.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ankYU.png" alt="enter image description here" /></a></p>