<p>One way that has worked for me in the past to handle highly imbalanced datasets is Synthetic Minority Oversampling Technique (SMOTE). Here is the paper for better understanding:</p>

<p><a href="https://arxiv.org/pdf/1106.1813.pdf" rel="nofollow noreferrer">SMOTE Paper</a></p>

<p>This works by synthetically oversampling the minority class or classes for that matter. To quote the paper:</p>

<blockquote>
  <p>The minority class is over-sampled by taking each minority class
  sample and introducing synthetic examples along the line segments
  joining any/all of the k minority class nearest neighbors. Depending
  upon the amount of over-sampling required, neighbors from the k
  nearest neighbors are randomly chosen.</p>
</blockquote>

<p>This then will move closer towards balancing out your dataset. There is an implementation of SMOTE in the <a href="https://imbalanced-learn.readthedocs.io/en/stable/" rel="nofollow noreferrer">imblearn</a> package in python. </p>

<p>Here is a good read about <a href="https://imbalanced-learn.readthedocs.io/en/stable/auto_examples/over-sampling/plot_comparison_over_sampling.html#more-advanced-over-sampling-using-adasyn-and-smote" rel="nofollow noreferrer">different oversampling algorithms</a>. It includes oversampling using <a href="https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.ADASYN.html" rel="nofollow noreferrer">ADASYN</a> as well as <a href="https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html" rel="nofollow noreferrer">SMOTE</a>.</p>

<p>I hope this helps.</p>
Best way to handle imbalanced dataset for multi-class classification in Auto-Sklearn
<p>I'm using Auto-Sklearn and have a dataset with 42 classes that are heavily imbalanced. What is the best way to handle this imbalance? As far as I know, two approaches to handle imbalanced data within machine learning exist. Either using a resampling mechanism such as over- or under-sampling (or a combination of both) or to solve it on an algorithmic-level by choosing an inductive bias that would require in-depth knowledge about the algorithms used within Auto-Sklearn. I'm not quite sure on how to handle this problem. Is it anyhow possible to solve the imbalance directly within Auto-Sklearn or do I need to use resampling strategies as offered by e.g. imbalanced-learn? Which evaluation metric should be used after the models have been computed? The roc_auc_score for multiple classes is available since sklearn==0.22.1. However, Auto-Sklearn only supports sklearn up to version 0.21.3. Thanks in advance!   </p>