title: Feature importance 'gain' in XGBoost
body: <p>I want to understand how the feature importance in xgboost is calculated by 'gain'. From <a href="https://towardsdatascience.com/be-careful-when-interpreting-your-features-importance-in-xgboost-6e16132588e7" rel="noreferrer">https://towardsdatascience.com/be-careful-when-interpreting-your-features-importance-in-xgboost-6e16132588e7</a>:</p>
  
  <blockquote>
    <p>‘Gain’ is the improvement in accuracy brought by a feature to the branches it is on. The idea is that before adding a new split on a feature X to the branch there was some wrongly classified elements, after adding the split on this feature, there are two new branches, and each of these branch is more accurate (one branch saying if your observation is on this branch then it should be classified as 1, and the other branch saying the exact opposite).</p>
  </blockquote>
  
  <p>In scikit-learn the feature importance is calculated by the gini impurity/information gain reduction of each node after splitting using a variable, i.e. weighted impurity average of node - weighted impurity average of left child node - weighted impurity average of right child node (see also: <a href="https://stats.stackexchange.com/questions/162162/relative-variable-importance-for-boosting">https://stats.stackexchange.com/questions/162162/relative-variable-importance-for-boosting</a>)</p>
  
  <p>I wonder if xgboost also uses this approach using information gain or accuracy as stated in the citation above. I've tried to dig in the code of xgboost and found out this method (already cut off irrelevant parts):</p>
  
  <pre><code>def get_score(self, fmap='', importance_type='gain'):
      trees = self.get_dump(fmap, with_stats=True)
  
      importance_type += '='
      fmap = {}
      gmap = {}
      for tree in trees:
          for line in tree.split('\n'):
              # look for the opening square bracket
              arr = line.split('[')
              # if no opening bracket (leaf node), ignore this line
              if len(arr) == 1:
                  continue
  
              # look for the closing bracket, extract only info within that bracket
              fid = arr[1].split(']')
  
              # extract gain or cover from string after closing bracket
              g = float(fid[1].split(importance_type)[1].split(',')[0])
  
              # extract feature name from string before closing bracket
              fid = fid[0].split('&lt;')[0]
  
              if fid not in fmap:
                  # if the feature hasn't been seen yet
                  fmap[fid] = 1
                  gmap[fid] = g
              else:
                  fmap[fid] += 1
                  gmap[fid] += g
  
      return gmap
  </code></pre>
  
  <p>So 'gain' is extracted from dump file of each booster but how is it actually measured?</p>