title: Types of Training vs Test Error for Random Forest Classification Algorithm (Assessing Variance)
body: <p>I have 2 questions that I would like to ascertain if possible (questions are bolded):</p>
  <p>I've recently understood (I hope) the random forest classification algorithm, and have tried to apply it using sklearn on Python on a rather large dataset of pixels derived from satellite images (with the features being the different bands, and the labels being specific features that I outlined by myself, i.e., vegetation, cloud, etc). I then wanted to understand if the model was experiencing a variance problem, and so the first thought that came to my mind was to compare between the training and testing data.</p>
  <p>Now this is where the confusion kicks in for me - I understand that there have been many different posts about:</p>
  <ol>
  <li>How CV error should/should not be used compared to the out of bag (OOB) error</li>
  <li>How by design, the training error of a random forest classifier is almost always ~0 (i.e., fitting my model on the training data and using it to predict on the same set of training data) - seems to be the case regardless of the tree depth</li>
  </ol>
  <p>Regarding point 2, it seems that I can never compare my training and test error as the former will always be low, and so I decided to use the OOB error as my 'representative' training error for the entire model. I then realized that the OOB error might be a pseudo test error as it essentially tests trees on points that they did not specifically learn (in the case of bootstrapped trees), and so I defaulted to CV error being my new 'representative' training error for the entire model.</p>
  <p>Looking back at the usage of CV error, I initially used it for hyperparameter tuning (e.g., max tree depth, number of trees, criterion type, etc), and so I was again doubting myself if I should use it as my official training error to be compared against my test error.</p>
  <p>What makes this worse is its hard for me to validate what I think is true based on posts across the web because each answers only a small part and might contradict each other, and so would anyone kindly help me with my <strong>predicament on what to use as my official training error that will be compared to my test error</strong>?</p>
  <p>My second question revolves around how the OOB error might be a pseudo test error based on datapoints not selected during bootstrapping. If that were true, <strong>would it be fair to say this does not hold if bootstrapping is disabled</strong> (the algorithm is technically still a random forest as features are still randomly subsampled for each tree, its just that the correlation between trees are probably higher)?</p>
  <p>Thank you!!!!</p>