I am working on a pattern identification/classification problem on an imbalanced dataset, with target to non target proportion in population approx as 1%:99%. There are around 0.5 million records in my dataset.

I am restricted to use SAS E-Miner for this analysis. Currently I am using the following approach:

  1. Give appropriate decision weights (profit matrix)
  2. Undersample on the majority class or "good" records.
  3. Running a decision tree on the sample.

My questions are:

  1. How can I undo the effect of undersampling of majority class?
  2. Does giving appropriate decision weights actually helps in removing the bias introduced because of undersampling? Or are these really two independent things.
  3. Even if these decision weights are applied, how do we determine the optimal decision threshold for basing our decisions.

I have tried a boosting algorithm (without adjusting prior probabilities and without using decision weights) but the number of rules/patterns that get thrown up are around 20+ which seems like a mild concern to me.

Would appreciate any inputs from CV community folks.

  • $\begingroup$ What boosting algorithm are you using? Boosted models with 100s or 1000s of trees are common, so 20+ shouldn't be an issue. $\endgroup$ – Zach Jan 3 '14 at 14:48
  • $\begingroup$ I am really not sure which is the boosting algorithm thats default to the starts group node, but there is another node in SAS EM called Gradient Boosting (which is not giving any encouraging results). The point that I want to make is, I run this boosting technique (not the Gradient one) on my dataset without any adjustment of priors/resampling/profit loss matrix for 5 iterations (no. of loops) and the result gave me around 20+ business rules/patterns. $\endgroup$ – aj_dataminer Jan 6 '14 at 10:25

Removing samples from the majority class may cause the classifier to miss important concepts/features pertaining to the majority class.

One strategy called informed undersampling demonstrated good results. Unsupervised learning algorithm is used to perform independent random sampling from majority class. Multiple classifiers based on the combination of each majority class subset with the minority class data are chosen.

Another example of informed undersampling uses the K-nearest neighbor (KNN) classifier to achieve undersampling. One of the four methods on KNN, looks most straightforward, called NearMiss-3, selects a given number of the closest majority samples for each minority sample to guarantee that every minority sample is surrounded by some majority samples. However, another method, NearMiss-2, in which the majority class samples are selected if their average distance to the three farthest minority class samples are the smallest, is proved the most competitive approach in imbalanced learning.

The profit (cost) matrix can be considered as a numerical representation of the penalty of classifying samples from one class to another. In decision tree,

(1) cost-sensitive adjustments can be applied to the decision threshold;

ROC curve is applied to plot the range of performance values as the decision threshold is moved from the point where the total misclassifications on majority class are maximally costly to the point where total misclassifications on the minority class are maximally costly. The most dominant point on the ROC curve corresponds to the final decision threshold. Read this paper for more details.

(2) cost-sensitive considerations can be given to the split criteria at each node;

This is achieved by fitting an impurity function, and the split with maximum fitting accuracy at each node is selected. This tutorial generalizes the effects of decision tree growth for any choice of spit criteria.

(3) cost-sensitive pruning schemes can be applied to the tree.

Pruning improves generalization by removing leaves with class probability estimates below a specified threshold. Laplace smoothing method on pruning technique is described in the same tutorial here to reduce the probability that pruning removes leaves on the minority class.

| cite | improve this answer | |

First of all, you don't need to down-sample, unless you don't have enough computing power to fit the model to the full dataset. An alternative approach is to assign target observations a weight of 99 and the non-target observations a weight of 1. This will mean the model considers 1 target mis-classification equal to 99 non-target mis-classifications, and will bias the model towards the smaller class without the need for down-sampling.

Basically, when down-sampling, you are throwing out information, which reduces the precision of your classifier. Since the positive class is usually more interesting than the negative class this usually isn't a big problem, but if you can use all the data, you should! Adjusting your weights is another way to tell the model what is important and what is not.

Finally, regardless of your approach, you can look at how your model performs on a test set. Calculate a ROC curve, which will allow you to see what the tradeoff is between true-positives and false positives for your model and determine a decision threshold. You can also use your profit matrix in this step.

| cite | improve this answer | |

Your Answer

