Checking for ambiguities in decision tree <p>I'm using sklearn's decision tree to replace a messy and unmantainable implementation of business rules as a long if-elif-else chain. I validate the tree using thousands of test cases for all labels, but sometimes the table with rules I'm using as training data have errors and some tests fail randomly.</p> <p>I need a way to validate the tree, beyond the test cases for the outcomes. Is it correct to assume that if all leaf nodes have a gini = 0.0, there will be no random variation in classification across trees generated with a different random seed? If I need to enforce that on my application, is it reasonable to check for that when updating the training data?</p> <p>Notice that my case is not a typical classification problem, since I already have the decision tree implemented in code, and I want to use an algorithm to generate an equivalent tree from carefully tailored data, rather than a real world data sample, simply because in my case maintaining the data set with business rules is easier than maintaining the code.</p> <p>So, in my data set the features will ideally cover all the possible range of values and give one unambiguous label for that. For instance, while a real world training set might be something like:</p> <pre><code>features = [[1], [1.1], [2], [2.3]] labels = ['sativa', 'sativa', 'indica', 'indica'] </code></pre> <p>And an algorithm can come up randomly with a tree1 like:</p> <pre><code>if feature &lt; 1.75: return 'sativa' else: return 'indica' </code></pre> <p>And a tree2 like:</p> <pre><code>if feature &lt; 1.55: return 'sativa' else: return 'indica' </code></pre> <p>However, my training set wouldn't have the gaps where randomness occurs. It would be like:</p> <pre><code>features = [[1], [1.9], [2], [2.3]] labels = ['sativa', 'sativa', 'indica', 'indica'] </code></pre> <p>So, regardless of the initial random state, the tree would always be (obviously, ignoring differences below 0.1):</p> <pre><code>if feature &lt; 1.95: return 'sativa' else: return 'indica' </code></pre> <p>My problem is precisely that I need to validate if the training set has an error and there's a gap of values where random variation can occur, or if the same set of features is being assigned to different labels. Fixing the random state doesn't solve that problem, it only guarantees the same data will always generate the same tree.</p> <p>So, is there any way to determine if this happens with a tree, other than validating the data for those problems before generating the tree, or running a comprehensive testing a number of times big enough to rule out the random variation?</p>