title: ID3 and C4.5: How Does "Gain Ratio" Normalize "Gain"?
body: <p>The ID3 algorithm uses "Information Gain" measure.</p>
  
  <p>The C4.5 uses "Gain Ratio" measure which is Information Gain divided by <code>SplitInfo</code>, whereas <code>SplitInfo</code> is high for a split where records split evenly between different outcomes and low otherwise.</p>
  
  <p><strong>My question is:</strong> </p>
  
  <p>How does this help to solve the problem that Information Gain is biased towards splits with many outcomes? I can't see the reason. <code>SplitInfo</code> doesn't even take into account the number of outcomes, just the distribution of records in the split.</p>
  
  <p>It may very well be that there is a low number of outcomes (say 2), and the records are split evenly between those 2 outcomes. In that case, <code>SplitInfo</code> is high, Gain Ratio is low, and a split with few outcomes is less likely to be chosen by C4.5.</p>
  
  <p>On the other hand, it may be that there is a low number of outcomes, but the distribution is far from even. In that case, <code>SplitInfo</code> is low, Gain Ratio is high, and a split with many outcomes is more likely to be chosen.</p>
  
  <p>What am I missing?</p>