title: How to choose C and gamma AFTER grid search using libSVM (RBF kernel) for best possible generalisation?
body: <p>I am aware of the abundance of questions asking about choosing the 'best' C and gamma values for SVM (RBF kernel). <strong>The standard answer is a grid search, however, my questions starts after the results of the grid search.</strong> Let me explain:</p>
  
  <p>I have a data set of 10 subjects on which I perform leave-one-subject-out-xfold-validation meaning I perform a grid search on each left-out subject. In order to not optimise on this training data I do not want to choose the best C and gamma parameter by building the mean accuracy over all 10 models and search for the maximum. Considering one model within the xfold, I could perform another xfold only on the training data wihtin this model (not involving the left out validation subject). But you can imagine the computational effort and I do not have enough time atm for this. </p>
  
  <p>Since the <strong>grid search</strong> for each of the 10 models <strong>resulted in a wide range of good C and gamma parameters (difference between accuracy of only 2-4%, see Figure 1)</strong> I thought about a different way.</p>
  
  <p><strong>I defined a region within the grid, which only contains the accuracies that have a difference of 2% to the maximum accuracy of this grid. All other accuracy values with a difference higher than 2% are set to zero (see Figure 2).</strong> I do this for every model and build the intersect between the regions of every model. This results in a much smaller region of C and gamma values that would produce accuracies within 2% of the max. accuracy for each model. However, the range is still rather big. <strong>So I thought about choosing the C-gamma pair with the lowest C as this would mean that I am the furthest away from overfitting and closer to a good generalisation. Can I argue like that?</strong></p>
  
  <hr>
  
  <p><strong>How would I generally choose a C and gamma within this region of C-gamma pairs, which all proofed to be reliable adjustments for my classifier in all 10 models?</strong>
  Should I focus on minimising the C parameter? Or should I focus on minimising the C AND the gamma paramater? </p>
  
  <hr>
  
  <p>I found a related answer here (<a href="https://stackoverflow.com/questions/23391151/are-high-values-for-c-or-gamma-problematic-when-using-an-rbf-kernel-svm">Are high values for c or gamma problematic when using an RBF kernel SVM?</a>) that says a combination of high C AND high gamma would mean overfitting. I understood that the value of gamma changes the width of the gaussian curve around data points, but I still cant get my head around what it practically means within a data set.</p>
  
  <p>The post brought me to another idea. Could I use the number of SVs related to the number of data points as a criterium to choose between all the C-gamma pairs? A low (number of SVs/number of data points) would mean a better generalisation? <strong>I am willing to loose accuracy as it shouldnt effect the outcome I am interested in, if I get in return a better generalisation (at least from a theoretical point of view).</strong> </p>
  
  <p><img src="https://i.stack.imgur.com/gxQg3.png" alt="Balanced accuries after grid search"></p>
  
  <p><img src="https://i.stack.imgur.com/llcN6.png" alt="Balanced accuries that follow my region and intersect criterium"></p>