<p>Ensembles are used to fight overfitting / improve generalization or to fight specific weaknesses / use strength of different classifiers. They can be applied in any classification task.</p>

<p>I used ensembles in <a href="https://arxiv.org/abs/1707.09725" rel="nofollow noreferrer">my masters thesis</a>. The <a href="https://github.com/MartinThoma/msthesis-experiments" rel="nofollow noreferrer">code is on Github</a>.</p>

<h2>Example 1</h2>

<p>For example, think of a binary problem where you have to tell if a data point is of class A or B. This could be an image and you have to decide if there is a (A) a dog or (B) a cat on it. Now you have two classifiers (1) and (2) (e.g. two neural networks, but trained in different ways; or one SVM and a decision tree, or ...). They make the following errors:</p>

<pre><code>(1):   Predicted
T    | A    B
R  ------------
U  A | 90%  10%
E  B | 50%  50%


(2):   Predicted
T    | A    B
R  ------------
U  A | 60%  40%
E  B | 40%  60%
</code></pre>

<p>You could, for example, combine them to an ensemble by first using (1). If it predicts <code>B</code>, then you can use (2). Otherwise you stick with it.</p>

<p>Now, what would be the expected error, (falsely) assuming both are independent)?</p>

<p>If the true class is <code>A</code>, then we predict with 90% the true result. In 10% of the cases we predict <code>B</code> and use the second classifier. This one gets it right in 60% of the cases. This means if we have <code>A</code>, we predict <code>A</code> in <code>0.9 + 0.1*0.6 = 0.96 = 96%</code> of the cases.</p>

<p>If the true class is <code>B</code>, we predict in <code>50%</code> of the cases <code>B</code>. But we also need to get it right the second time, so only in <code>0.5*0.6 = 0.3 = 30%</code> of the cases we get it right there.</p>

<p>So in this simple example we made the situation better for one class, but worse for the other.</p>

<h2>Example 2</h2>

<p>Now, lets say we have 3 classifiers with </p>

<pre><code>      Predicted
T    | A    B
R  ------------
U  A | 60%  40%
E  B | 40%  60%
</code></pre>

<p>each, but the classifications are independent. What do you get when you make a majority vote?</p>

<p>If you have class A, the probability that at least two say it is class A is</p>

<pre><code>  0.6 * 0.6 * 0.6 + 0.6 * 0.6 * 0.4 + 0.6 * 0.4 * 0.6 + 0.4 * 0.6 * 0.6
= 1*0.6^3 + 3*(0.6^2 * 0.4^1)
= (3 nCr 3) * 0.6 + (3 nCr 2) * (0.6^2 * 0.4^1)
= 0.648
</code></pre>

<p>The same goes for the other class. So we improved the classifier to</p>

<pre><code>      Predicted
T    | A    B
R  ------------
U  A | 65%  35%
E  B | 35%  65%
</code></pre>

<h2>Code</h2>

<p>See <a href="http://scikit-learn.org/stable/modules/ensemble.html" rel="nofollow noreferrer">sklearns page on Ensembles</a> for code.</p>

<p><strong>The most specific example of ensemble learning are random forests.</strong></p>
What are some specific examples of Ensemble Learning?
<p>What are some concrete real life examples which can be solved using Boosting/Bagging algorithms? Code snippets would be greatly appreciated.</p>