This tutorial is meant to be run in Google Colab. If you are not using this notebook in Google Colab, please install the package on your own computer, download the data from figshare, and then skip ahead to the data loading step.

---
### Install the Popalign repo
---

The user can install PopAlign with the following command:

`pip install git+https://github.com/thomsonlab/popalign`

In general, we recommend installing the PopAlign package and its required dependencies in a dedicated environment (created with Anaconda for example). This package is built to run in Python 3 and will not work with Python 2.

After installing the PopAlign package with pip, the user can import it in their Python code:

`import popalign as PA`

```
!pip install git+https://github.com/thomsonlab/popalign.git
```

    Collecting git+https://github.com/thomsonlab/popalign.git
      Cloning https://github.com/thomsonlab/popalign.git to /tmp/pip-req-build-h4k327xb
      Running command git clone -q https://github.com/thomsonlab/popalign.git /tmp/pip-req-build-h4k327xb
    Requirement already satisfied (use --upgrade to upgrade): popalign==0.1 from git+https://github.com/thomsonlab/popalign.git in /usr/local/lib/python3.7/dist-packages
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (1.20.3)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (1.1.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (1.4.1)
    Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (0.0)
    Requirement already satisfied: plotly in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (4.4.1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (3.2.2)
    Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (0.11.1)
    Requirement already satisfied: adjustText in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (0.7.3)
    Requirement already satisfied: ipywidgets in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (7.6.3)
    Requirement already satisfied: fastcluster in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (1.2.3)
    Requirement already satisfied: umap-learn in /usr/local/lib/python3.7/dist-packages (from popalign==0.1) (0.5.1)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->popalign==0.1) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->popalign==0.1) (2018.9)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sklearn->popalign==0.1) (0.22.2.post1)
    Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.7/dist-packages (from plotly->popalign==0.1) (1.3.3)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from plotly->popalign==0.1) (1.15.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->popalign==0.1) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->popalign==0.1) (1.3.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->popalign==0.1) (2.4.7)
    Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->popalign==0.1) (3.5.1)
    Requirement already satisfied: ipython>=4.0.0; python_version >= "3.3" in /usr/local/lib/python3.7/dist-packages (from ipywidgets->popalign==0.1) (5.5.0)
    Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->popalign==0.1) (5.1.3)
    Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->popalign==0.1) (5.0.5)
    Requirement already satisfied: jupyterlab-widgets>=1.0.0; python_version >= "3.6" in /usr/local/lib/python3.7/dist-packages (from ipywidgets->popalign==0.1) (1.0.0)
    Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->popalign==0.1) (4.10.1)
    Requirement already satisfied: numba>=0.49 in /usr/local/lib/python3.7/dist-packages (from umap-learn->popalign==0.1) (0.51.2)
    Requirement already satisfied: pynndescent>=0.5 in /usr/local/lib/python3.7/dist-packages (from umap-learn->popalign==0.1) (0.5.2)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn->popalign==0.1) (1.0.1)
    Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.7/dist-packages (from widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (5.3.1)
    Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (2.6.1)
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (0.8.1)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (0.7.5)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (4.8.0)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (1.0.18)
    Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (56.1.0)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (4.4.2)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->popalign==0.1) (0.2.0)
    Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->popalign==0.1) (2.6.0)
    Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->popalign==0.1) (4.7.1)
    Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel>=4.5.1->ipywidgets->popalign==0.1) (5.1.1)
    Requirement already satisfied: jupyter-client in /usr/local/lib/python3.7/dist-packages (from ipykernel>=4.5.1->ipywidgets->popalign==0.1) (5.3.5)
    Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->umap-learn->popalign==0.1) (0.34.0)
    Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (0.10.0)
    Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (1.5.0)
    Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (2.11.3)
    Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (5.6.1)
    Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect; sys_platform != "win32"->ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (0.7.0)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0; python_version >= "3.3"->ipywidgets->popalign==0.1) (0.2.5)
    Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets->popalign==0.1) (22.0.3)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (2.0.1)
    Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (0.7.1)
    Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (0.8.4)
    Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (3.3.0)
    Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (0.5.0)
    Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (0.3)
    Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (1.4.3)
    Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (0.5.1)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets->popalign==0.1) (20.9)
    Building wheels for collected packages: popalign
      Building wheel for popalign (setup.py) ... done
      Created wheel for popalign: filename=popalign-0.1-cp37-none-any.whl size=3272511 sha256=d23190ef260691709b244a0db5f260cabf0638570357ee89f187d574f8367de0
      Stored in directory: /tmp/pip-ephem-wheel-cache-ldi5vgwj/wheels/c1/b5/02/5df2b8d9cfac11b9f0ea8d6cff18a063fad5d0f93dc7425af4
    Successfully built popalign

```
!pip install -U numpy
```

    Collecting numpy
      Downloading https://files.pythonhosted.org/packages/a5/42/560d269f604d3e186a57c21a363e77e199358d054884e61b73e405dd217c/numpy-1.20.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.3MB)
         |████████████████████████████████| 15.3MB 268kB/s
    ERROR: tensorflow 2.5.0 has requirement numpy~=1.19.2, but you'll have numpy 1.20.3 which is incompatible.
    ERROR: datascience 0.10.6 has requirement folium==0.2.1, but you'll have folium 0.8.3 which is incompatible.
    ERROR: albumentations 0.1.12 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.
    Installing collected packages: numpy
      Found existing installation: numpy 1.19.5
        Uninstalling numpy-1.19.5:
          Successfully uninstalled numpy-1.19.5
    Successfully installed numpy-1.20.3

```
import popalign as PA
```

---

### Mount Google Drive into colab

---

```
from google.colab import drive
drive.mount("/content/drive")
```

    Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

```
%cd 'drive/My Drive/'
```

    /content/drive/My Drive

```
!mkdir 'PopAlignTutorial'
%cd 'PopAlignTutorial'
```

    /content/drive/My Drive/PopAlignTutorial

```
ls
```

    data_inprogress_v1.pickle  __MACOSX/  output_drugscreen/  PopAlign_Data/

```

```

---
The following codeblock is useful for saving your progress.

When you want to take a break, uncomment the save block and execute.

When you want to load, comment the save block and uncomment the load block.  Make sure that the save block is commented otherwise you will accidentally overwrite the file.

```
import pickle

# '''
# Save data from a multiplexed experiment
# '''
# with open('data_inprogress_v1.pickle', 'wb') as handle:
#     pickle.dump(pop, handle)

# '''
# Load data from a multiplexed experiment
# '''
# with open('data_inprogress_v1.pickle', 'rb') as handle:
#     pop = pickle.load(handle)
```

---

### Grab the Drug Response data from PopAlign figshare link
https://doi.org/10.6084/m9.figshare.11837097

```
!wget  --no-check-certificate https://ndownloader.figshare.com/files/24645260

# If the above link is not correct, go to the doi link,
# right-click on the Download button and select 'Copy Link Address'

```

    --2021-05-26 16:06:30--  https://ndownloader.figshare.com/files/24645260
    Resolving ndownloader.figshare.com (ndownloader.figshare.com)... 54.73.104.177, 34.248.162.188, 52.48.193.78, ...
    Connecting to ndownloader.figshare.com (ndownloader.figshare.com)|54.73.104.177|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/24645260/PopAlign_Data.zip [following]
    --2021-05-26 16:06:31--  https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/24645260/PopAlign_Data.zip
    Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 52.218.20.220
    Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|52.218.20.220|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 242971639 (232M) [application/zip]
    Saving to: ‘24645260’

    24645260            100%[===================>] 231.71M  11.5MB/s    in 24s

    2021-05-26 16:06:56 (9.79 MB/s) - ‘24645260’ saved [242971639/242971639]

```
!unzip 24645260
```

    Archive:  24645260
       creating: PopAlign_Data/
       creating: PopAlign_Data/MM2/
      inflating: PopAlign_Data/MM2/genes.tsv
       creating: __MACOSX/
       creating: __MACOSX/PopAlign_Data/
       creating: __MACOSX/PopAlign_Data/MM2/
      inflating: __MACOSX/PopAlign_Data/MM2/._genes.tsv
      inflating: PopAlign_Data/MM2/barcodes.tsv
      inflating: __MACOSX/PopAlign_Data/MM2/._barcodes.tsv
      inflating: PopAlign_Data/MM2/MM2.mtx
      inflating: __MACOSX/PopAlign_Data/MM2/._MM2.mtx
      inflating: __MACOSX/PopAlign_Data/._MM2
       creating: PopAlign_Data/MM4/
      inflating: PopAlign_Data/MM4/genes.tsv
       creating: __MACOSX/PopAlign_Data/MM4/
      inflating: __MACOSX/PopAlign_Data/MM4/._genes.tsv
      inflating: PopAlign_Data/MM4/barcodes.tsv
      inflating: __MACOSX/PopAlign_Data/MM4/._barcodes.tsv
      inflating: PopAlign_Data/MM4/MM4.mtx
      inflating: __MACOSX/PopAlign_Data/MM4/._MM4.mtx
      inflating: __MACOSX/PopAlign_Data/._MM4
       creating: PopAlign_Data/MM3/
      inflating: PopAlign_Data/MM3/genes.tsv
       creating: __MACOSX/PopAlign_Data/MM3/
      inflating: __MACOSX/PopAlign_Data/MM3/._genes.tsv
      inflating: PopAlign_Data/MM3/barcodes.tsv
      inflating: __MACOSX/PopAlign_Data/MM3/._barcodes.tsv
      inflating: PopAlign_Data/MM3/MM3.mtx
      inflating: __MACOSX/PopAlign_Data/MM3/._MM3.mtx
      inflating: __MACOSX/PopAlign_Data/._MM3
       creating: PopAlign_Data/healthy1/
      inflating: PopAlign_Data/healthy1/barcodes.tsv
       creating: __MACOSX/PopAlign_Data/healthy1/
      inflating: __MACOSX/PopAlign_Data/healthy1/._barcodes.tsv
      inflating: PopAlign_Data/healthy1/gene.tsv
      inflating: __MACOSX/PopAlign_Data/healthy1/._gene.tsv
      inflating: PopAlign_Data/healthy1/healthy1.mtx
      inflating: __MACOSX/PopAlign_Data/healthy1/._healthy1.mtx
      inflating: __MACOSX/PopAlign_Data/._healthy1
       creating: PopAlign_Data/MM1/
      inflating: PopAlign_Data/MM1/genes.tsv
       creating: __MACOSX/PopAlign_Data/MM1/
      inflating: __MACOSX/PopAlign_Data/MM1/._genes.tsv
      inflating: PopAlign_Data/MM1/barcodes.tsv
      inflating: __MACOSX/PopAlign_Data/MM1/._barcodes.tsv
      inflating: PopAlign_Data/MM1/MM1.mtx
      inflating: __MACOSX/PopAlign_Data/MM1/._MM1.mtx
      inflating: __MACOSX/PopAlign_Data/._MM1
       creating: PopAlign_Data/healthy2/
      inflating: PopAlign_Data/healthy2/barcodes.tsv
       creating: __MACOSX/PopAlign_Data/healthy2/
      inflating: __MACOSX/PopAlign_Data/healthy2/._barcodes.tsv
      inflating: PopAlign_Data/healthy2/gene.tsv
      inflating: __MACOSX/PopAlign_Data/healthy2/._gene.tsv
      inflating: PopAlign_Data/healthy2/healthy2.mtx
      inflating: __MACOSX/PopAlign_Data/healthy2/._healthy2.mtx
      inflating: __MACOSX/PopAlign_Data/._healthy2
       creating: PopAlign_Data/drugscreen/
      inflating: PopAlign_Data/drugscreen/features.tsv
       creating: __MACOSX/PopAlign_Data/drugscreen/
      inflating: __MACOSX/PopAlign_Data/drugscreen/._features.tsv
      inflating: PopAlign_Data/drugscreen/drugscreen.mtx
      inflating: __MACOSX/PopAlign_Data/drugscreen/._drugscreen.mtx
      inflating: PopAlign_Data/drugscreen/meta.csv
      inflating: __MACOSX/PopAlign_Data/drugscreen/._meta.csv
      inflating: PopAlign_Data/drugscreen/barcodes.tsv
      inflating: __MACOSX/PopAlign_Data/drugscreen/._barcodes.tsv
      inflating: __MACOSX/PopAlign_Data/._drugscreen
      inflating: __MACOSX/._PopAlign_Data

```
%rm 24645260
```

---

### First we will load in data from raw files.

---

In this tutorial, we will use the drug screen dataset from the PopAlign paper. This dataset was produced using multiplexing (Multi-seq).

For multiplexed experiments, we store all cells into a single matrix file, and store sample information for each cell in a metadata file. This format requires the paths to the .mtx file, the gene list, the barcode list and a metadata file. The meta data file must be a .csv file that includes at least two columns:
cell_barcode and sample_id . The first column contains the cell barcodes and the second column contains the matching sample names for those cells. The cells from the different samples will be selected from the matrix based on their barcode index.

If you do not have multiplexed data, you can also use `PA.load_samples()` to load samples from individual files.

Both loading functions (`PA.load_multiplexed()` and `PA.load_samples()`) produce a dictionary that contains the loaded data, genes and various information. That object (`pop`) is used throughout the entire analysis. The `outputfolder` parameter (default: 'output') defines where the results of the analysis will be saved.

You can load more data into an existing pop object by specifying the `existing_obj` parameter (default: `None`)

The sample gene expression matrices are stored individually within the pop object. To access the gene expression matrix of 'sample1', use the following:

`pop['samples']['sample1']['M']`

The gene labels are stored in the pop object:

`pop['genes']`

Important: Make sure to provide valid gene ids. Some custom references modify the gene ids, and will cause downstream issues when performing GSEA or cell typing, which rely on correct gene names.

```
# Load in data from raw files to rerun entire analysis.
# Please note that some of the results may shift due to slightly different parameters that are learnt.
# Additionally you will have to manually adjust some variables (like component number) in
# the code to account for changes in how the components are numbered.
'''
Load data from a screen experiment and genes from a file

Parameters
----------
matrix : str
  Path to a sparse matrix
barcodes : str
  Path to a .tsv 10X barcodes file
metafile : str
  Path to a metadata file. Must contains `cell_barcodes` and `sample_id` columns
controlstring: string
  String containing common name across all control samples
genes : str
  Path to a .tsv 10X gene file. Optional if existing_obj is provided
outputfolder : str, optional
  Path (or name) of the output folder to create
existing_obj : dict, optional
  Object previously returned by either load_samples() or load_screen(). New samples will be added to that object
only : list, optional
  List of sample names to load (other samples with names not in list will not be loaded)
col : str, optional
  Name of a specific column in the meta data to use
value : str or int, optional
  Value in the specified meta data column `col` to use to filter samples to load
'''

folder = 'PopAlign_Data/drugscreen/'

mymatrix = folder + 'drugscreen.mtx'
mybarcodes = folder + 'barcodes.tsv'
mygenes = folder + 'features.tsv'
mymetadata = folder + 'meta.csv'

pop = PA.load_multiplexed(matrix=mymatrix,
                     barcodes=mybarcodes,
                     genes=mygenes,
                     metafile=mymetadata,
                     controlstring='CTRL', # This can also be directly reset later using: pop['controlstring'] = 'newname'
                     outputfolder='output_drugscreen',
                     only=[], # list of sample names to only load the specified samples
                     col=None, # either None or a column name from the meta data
                     value=None, # if col != None, specify value in column to filter samples
                     existing_obj=None)
```

You can use the PA.print_ncells() command to print the number of cells loaded into each sample:

```
PA.print_ncells(pop)
```

    Meprednisone 	 943
    Dexrazoxane HCl (ICRF-187, ADR-529) 	 720
    Etodolac 	 898
    Phenylbutazone 	 671
    Pomalidomide 	 804
    Budesonide 	 726
    Loteprednol etabonate 	 646
    CTRL6 	 695
    Betamethasone Valerate 	 1004
    Febuxostat 	 848
    Sulfasalazine 	 796
    CTRL2 	 433
    Flurbiprofen 	 834
    Mianserin HCl 	 744
    Naproxen Sodium 	 670
    CTRL4 	 592
    Triamcinolone Acetonide 	 866
    Acetylcysteine 	 903
    Everolimus (RAD001) 	 486
    Isotretinoin 	 927
    Ibuprofen 	 893
    Doramapimod (BIRB 796) 	 908
    Allopurinol Sodium 	 666
    Tranilast 	 1081
    Cyclosporine 	 738
    Adapalene 	 806
    Thalidomide 	 603
    Celecoxib 	 652
    CTRL3 	 668
    Ketoprofen 	 750
    Loratadine 	 545
    Tazarotene 	 859
    Bimatoprost 	 899
    Cetirizine DiHCl 	 426
    Mesalamine 	 589
    Mercaptopurine (6-MP) 	 625
    CTRL1 	 476
    Busulfan 	 703
    CTRL5 	 721
    Zafirlukast 	 863
    Zileuton 	 891
    Latrepirdine 2HCl 	 838
    Rapamycin (Sirolimus) 	 337
    Alprostadil 	 749
    Masitinib (AB1010) 	 291
    Methotrexate 	 699
    Total number of cells loaded: 33482

---

### Normalization

---

Next we will normalize each dataset.

We normalize by using the following equation:

\begin{align}
g'_i = \log(\beta \frac{g_i}{\sum_{i}^{n}{g_i}}  + 1)
\end{align}

where $n$ is the total number of genes, and $\beta$ is a scaling factor, and we add a $1$ pseudo-count to each gene expression value. Dividing by the total number of transcripts per cell allows us to correct for technical differences in transcript capture per cell.  Scaling by $\beta$ enables us to rescale the value so that the quantity inside the $log()$ is roughly equal to the number of raw transcripts collected. We found that by setting $\beta =1000$ (roughly the median number of total transcript counts in a cell), we achieve a smooth transition in the distribution of transformed $g'_i$ when raw $g_i$ values step from $0$ to $1$.

The scaling factor value can be either set by the user or computed automatically (`scaling_factor=None`). When computed automatically, multiple factors are tested and the factor that minimizes the difference between the mean of the original data D prior to normalization and the mean of the normalized, scaled data is used.

The normalization function is:

`PA.normalize(pop, scaling_factor=None, ncells=None)`

Gene expression matrices are normalized in-place, i.e. the matrices are directly updated in the pop object (`pop['samples']['sample1']['M']`). `ncells` sets the number of randomly subsampled cells used to compute the scaling_factor. If `None`, all cells are used.

```
'''
Normalize the samples of object `pop` and applies a normalization factor

Parameters
----------
pop : dict
    Popalign object
scaling_factor : int or None, optional
    Number used to scale the data values. If None, that factor is computed automatically
ncells : int or None
    Number of cells to randomly subsample to try different normalization factors to use less memory. If None, all cells are used.
'''

PA.normalize(pop, scaling_factor=1000)
```

    Performing column normalization

```
import numpy as np
```

---

### Gene Filtering

---

Most genes in the dataset will not be very informative, as they will be expressed at very low levels (near or at 0) or are not expressed differentially between datasets or subpopulations. To filter out only the most informative genes, we find genes whose variance are higher than would be predicted by a Poisson distribution (for which $CV = \frac{1}{\sqrt(mean)}$) . Thus, we plot for all genes: log(CV) ~ log(mean). All genes will lie along a $ y = - 1/2 x $ line, but genes which have CV above that line are supra-Poisson. You can set the `offset` to that y=-½ x line that adjusts the filter threshold. The log of the offset parameter is added to the fitted line's intercept (i.e. `offset = 1` will result in no change to the line).

Plot is displayed in line and saved in: output/qc/gene_filtering.pdf

Once supra-Poisson genes are determined using `plot_gene_filter()`, we use the `filter()` function to generate a filtered and logged version of the data. `filter()` also provides options to eliminate ribosomal genes starting with RPS or RPL (`remove_ribsomal=True`) or mitochondrial genes starting with MT(`remove_mitochondrial=True`).

The filtered and logged matrices are stored as 'M_norm', and can be accessed with:

`pop['samples']['sample1']['M_norm']`

The non-filtered matrices are also logged in-place. The filtered genes can be accessed with:

`pop['filtered_genes']`

```
'''
Plot genes by their log(mean) and log(coefficient of variation)
Can be used multiple times to find best offset

Parameters
----------
offset: float
    Value (its log) will be added to the intercept of the linear fit to filter genes
'''

PA.plot_gene_filter(pop, offset=1.3)
```

```
'''
Filter genes from data in `pop`
Can discard ribosomal genes that start with RPS or RPL
Can discard mitochondrial genes that start with MT-

Parameters
----------
pop :dict
    Popalign object
remove_ribsomal : bool
    Wether to remove or not the ribosomal genes
remove_mitochondrial : bool
    Wether to remove or not the mitochondrial geneset
'''
PA.filter(pop, remove_ribsomal=False, remove_mitochondrial=False)
```

    Filtering genes and logging data

```
len(pop['filtered_genes'])
```

    1081

---
You can also remove red blood cells from the data based on high expression of  hemoglobin genes. We use the following genes:

*  Mouse: 'HBB-BT','HBB-BS' ,'HBA-A1','HBA-A2','HBB-B1','HBB-B2'
*  Human: 'HBB', 'HBA1', 'HBA2'

We sum the expression level for these genes, and use Otsu's method to find a threshold between two peaks.  Cells which exceed the threshold are flagged as red blood cells and removed from the analysis.

We do not do this for the PBMC drugscreen dataset since we do not find red blood cells after extended culture (> 24hr).

```
'''
Remove red blood cells from data

Parameters
----------
pop : dict
    Popalign object
species : str
    name of experiment species. Can be human or mouse
'''
# PA.removeRBC(pop, species='human')
```

---

### Dimensionality reduction

---

We build models in a common low-dimensional space defined by gene expression features or programs, which can be discovered using many linear dimensionality reduction techniques. Using a linear dimensionality reduction method is important for preserving distance after transformation.

In the package, we support both features built from principal components analysis (PCA) or orthogonal non-negative matrix factorizaion (oNMF), although support is more extensive for oNMF features (i.e. we are still developing downstream compatibility for models built with PCA features).

oNMF factors the original data matrix, $D^\text{train}$, into two matrices $D \approx FC$. $F$ is a $n x m$ matrix of features, where each column is a feature vector, n is the number of genes, and m is the number of features. $C$ is a $m x k$ matrix of coefficients multiplying those features, where k is the number of cells. Factorization occurs through minimization of an objective function with positivity and orthogonality constraints:

\begin{align}
&\arg\min_{FC}  || D^\text{train} - F   C||_2 \nonumber \\
& \text{subject to } \ F^T F = I , C_{i j} \geq 0, F_{i j} \geq 0,
\end{align}

The positivity constraints on $F$ and $C$, and orthogonality constraint on $F$ compel a parts-based representation (Lee and Seung, 1999). This means feature vectors tend to capture blocks of genes which co-occur, and are orthogonal to each other.

After running the oNMF algorithm (implemented , each feature vector is normalized by their respective L2-norm, so that each unit of distance along a feature corresponds to the same unit distance in the original gene expression space.

To choose the number of features, we first compute the oNMF featuresets across a range of $m$ and then select the number of features ($m) that optimizes a loss function described in the next section.

**References:**

Lee, D. D., and H. S. Seung. 1999. “Learning the Parts of Objects by Non-Negative Matrix Factorization.” Nature 401 (6755): 788–91.

Y. Li and A. Ngom, "The non-negative matrix factorization toolbox for biological data mining," BMC Source Code for Biology and Medicine, vol 8, pp. 10, April 2013.

```
'''
Compute feature spaces and minimize the reconstruction error to pick a final feature space
Run Gene Set Enrichment Analysis (GSEA)

Parameters
----------
pop : dict
    Popalign object
ncells : int
    Number of cells to use
nfeats : int or list of ints
    Number(s) of features to use
nreps : int
    Number of repetitions to perform for each k in nfeats
niter : int
    Maximum number of iterations to perform for each instance of the algorithm
'''

PA.onmf(pop, ncells=5000, nfeats=list(range(1,20)), nreps=2, niter=500);
```

    Computing W matrices
    Orthogonal NMF performed with 201 iterations

    Orthogonal NMF performed with 201 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Orthogonal NMF performed with 500 iterations

    Computing reconstruction errors

```
len(pop['filtered_genes'])
```

    1081

---

### Selecting the number of features

---

To perform oNMF, we must select $m$, the number of features to be extracted through.  Choosing $m$ involves balancing the tension between the 'expressiveness' in the feature set and its dimensionality. Higher $m$ reduces the error in the representation while also breaking up blocks of genes into smaller modules that represent independent gene expression pathways with finer granularity. However, as $m$ increases, the typical computational and sampling challenges associated with high dimensionality emerge.

Practically, we balance this tension in PopAlign by constructing a loss function with a penalty that increases with $m$:

\begin{align}
\arg \min_{m} \ f(m) = MSE + j \cdot c_{\alpha,0}\cdot m^{\alpha}
\end{align}

First, we compute the mean squared error (MSE):  $ \ \frac{1}{n} ||D^{\text{train}} - F_m C_m||^2$.  This error is, then, incremented by the righthand term, which penalizes higher values of $m$ and hence the dimensionality of the feature set.   To standardize the analysis computationally, we normalize the MSE values by their maximum value. Then, for each $\alpha$, we define the baseline constant as: $c_{\alpha,0} = 0.1 / (m_{max})^{\alpha}$ (PopAlign default $m_{max}=30$). This baseline constant tunes the penalty term to be a similar scale to the MSE curve, while the multiplier j gives the user the flexibility to modulate the exact value of the constant.  We compute the $f(m)$ curves for various values of $\alpha$ and its associated constant $c_{\alpha}$, and allow the user to choose parameter settings in order to achieve a coarser or finer decomposition of cell state.

Once the featureset is chosen, we cast the entire data into the chosen featureset using non-negative least squares. For each cell's gene expression profile $g$, we find the coefficients vector, $c$ :

\begin{align}
&\arg\min_{c} ||g - F \ c||_2 \ , \
& c_i \geq 0
\end{align}

To access the feature space:

`pop['W']`

To access the feature data of sample1 in feature space, use the following:

`pop['samples']['sample1']['C']`

Gene set enrichment analysis (GSEA) is run for each feature from the feature space. The top feature labels can be found here:

`pop['top_feat_labels']`

To acess the top 20 gene set labels assigned to each feature:

`pop['feat_labels']`

```
'''
Choose featureset from stored oNMF calculations.
Either user directly supplies a preferred m value or the hyperparameters for the loss function

NB: Currently only supports oNMF features

Parameters
----------
errors : list
  list of MSE errors from oNMF
alpha : float
  power of polynomial
multiplier : float
  multiplies constant C in f(m)

Plots can be found in output folder: qc/features/

'''

# save files for one featureset
PA.choose_featureset(pop, alpha = 3, multiplier=3)
```

---

### Build probabilistic models

---

The PopAlign framework builds probabilistic models of each sample and uses these models to compare samples at the population, subpopulation, and gene expression level.

All samples are modeled with a Gaussian Mixture Model (GMM). A GMM is a mixture of Gaussian densities that fit a given set of data points in an m-dimensional feature space.

\begin{align}
P(c)= \sum^{k}_{i=1} w_i \ \mathcal{N}(c; \mu_i, \Sigma_i) \nonumber
\end{align}

Each Gaussian $\mathcal{N}_i$ is parameterized with :

*   mean ($\mu_i$}: describes the mean cell-state in terms of gene expression features [m x 1 vector]
*   covariance matrix ($\Sigma_i$): describes spread and shape of the local density [m by m matrix]
*   weight($w_i$): describes the relative proportion of this density in the model [scalar]

Models are fit using sci-kit learn with expectation-maximization. To find the best possible model for a sample, build_gmms() scans over a list of component numbers (`ks`), and build multiple replicates for each k. For a given sample, the sample data is split into two sets: a training set and a validation set. The training set is used to fit the GMMs and the validation set is used to evaluate model performance. Once all models are fit, the validation set is used to rank the models based on either the Bayesian Information Criterion (BIC) or Akaike Information Criterion (AIC). We find the AIC often performs better. The model with the highest IC score is then retained and stored in the pop object.

To access the GMM of sample1 :

`pop['samples']['sample1']['gmm']`

---

#### Cell Typing

After building models, we can also type the components based on gene expression for specific markers using a simple algorithm.

First, all genes are normalized by the maximum for that gene. For each cell vector, we find the gene with the maximum score, and associate the cell with the corresponding cell type for that marker. Each cell is also classified as belonging to the Gaussian component based on max likelihood. Then, we label each Gaussian component with the cell type that is most prevalent within its classified cells.

If no dictionary is supplied, the components will be labeled with their component number (which is arbitrarily determined during the GMM learning process).

The type labels of the GMM for "`sample1`" can be found at:

`pop['samples']['sample1']['gmm_types']`

```
# Define markers for cell types
pbmc_types = {
		'Monocytes' : [
			'CD14',
			'CD33',
			'LYZ',
			'FCER1G',
			'LGALS3',
			'CSF1R',
			'ITGAX',
			'ITGAM',
			'CD86',
			'HLA-DRB1'],
		'B-cells' : [
			'MS4A1',
			'CD19',
			'CD79A'],
    'T cells' : [
			'CD27',
			'CD69',
			'CD2',
			'CD3D',
			'CXCR3',
			'CCL5',
			'IL7R',
			'CXCL8',
			'GZMK'],
	}

```

```
'''
Build a Gaussian Mixture Model on feature projected data for each sample

Parameters
----------
pop : dict
  Popalign object
ks : int or tuple
  Number or range of components to use
niters : int
  number of iterations to build for each k in `ks` during model selection phase
training : int or float
  If training is float, the value will be used a percentage to select cells for the training set. Must follow 0<value<1
  If training is int, that number of cells will be used for the training set.
nreplicates : int
  Number of replicates to generate. These replicate model will be used to provide confidence intervals later in the analysis.
  Each replicate is stored with zero-indexed numbering:  pop['samples'][x]['replicates'][0] is the first replicate
reg_covar : str or float
  If 'auto', the regularization value will be computed from the feature data (For oNMF: reg_covar is set to 1% of the max eigenvalue of all data)
  If float, value will be used as reg_covar parameter to build GMMs
types : dict, str or None
  Dictionary of cell types.
  If None, a default PBMC cell types dictionary is provided
only: list or str, optional
  Sample label or list of sample labels. Will force GMM construction for specified samples only. Defaults to None
featuretype: str
  either 'pca' or 'onmf'
criteria : str
  either 'bic' or 'aic'
'''

PA.build_gmms(pop,
              ks=(3), # For this analysis, we find that setting the k directly works well
              niters=2,
              training=.8,
              nreplicates=0,
              reg_covar='auto',
              types=pbmc_types, # either None, 'defaultpbmc' or a dictionary
              criteria='aic',
              only=None)
```

    Building model for Meprednisone (1 of 46)
    Building model for Dexrazoxane HCl (ICRF-187, ADR-529) (2 of 46)
    Building model for Etodolac (3 of 46)
    Building model for Phenylbutazone (4 of 46)
    Building model for Pomalidomide (5 of 46)
    Building model for Budesonide (6 of 46)
    Building model for Loteprednol etabonate (7 of 46)
    Building model for CTRL6 (8 of 46)
    Building model for Betamethasone Valerate (9 of 46)
    Building model for Febuxostat (10 of 46)
    Building model for Sulfasalazine (11 of 46)
    Building model for CTRL2 (12 of 46)
    Building model for Flurbiprofen (13 of 46)
    Building model for Mianserin HCl (14 of 46)
    Building model for Naproxen Sodium (15 of 46)
    Building model for CTRL4 (16 of 46)
    Building model for Triamcinolone Acetonide (17 of 46)
    Building model for Acetylcysteine (18 of 46)
    Building model for Everolimus (RAD001) (19 of 46)
    Building model for Isotretinoin (20 of 46)
    Building model for Ibuprofen (21 of 46)
    Building model for Doramapimod (BIRB 796) (22 of 46)
    Building model for Allopurinol Sodium (23 of 46)
    Building model for Tranilast (24 of 46)
    Building model for Cyclosporine (25 of 46)
    Building model for Adapalene (26 of 46)
    Building model for Thalidomide (27 of 46)
    Building model for Celecoxib (28 of 46)
    Building model for CTRL3 (29 of 46)
    Building model for Ketoprofen (30 of 46)
    Building model for Loratadine (31 of 46)
    Building model for Tazarotene (32 of 46)
    Building model for Bimatoprost (33 of 46)
    Building model for Cetirizine DiHCl (34 of 46)
    Building model for Mesalamine (35 of 46)
    Building model for Mercaptopurine (6-MP) (36 of 46)
    Building model for CTRL1 (37 of 46)
    Building model for Busulfan (38 of 46)
    Building model for CTRL5 (39 of 46)
    Building model for Zafirlukast (40 of 46)
    Building model for Zileuton (41 of 46)
    Building model for Latrepirdine 2HCl (42 of 46)
    Building model for Rapamycin (Sirolimus) (43 of 46)
    Building model for Alprostadil (44 of 46)
    Building model for Masitinib (AB1010) (45 of 46)
    Building model for Methotrexate (46 of 46)

```

```

#### You can also render the model predicted probabilities as a 2D heatmap in PC dimensions:

```
'''
Render the models as a probabilistic heatmap

Parameters
----------
pop : dict
    Popalign object
figsizegrouped : tuple
    Figure size for the grid rendering plotof all samples together
figsizesingle : tuple
    Figure size of an individual sample rendering plot
mode : str
    One of grouped, individual or unique.
    Grouped will render the models individually and together in a separate grid
    Inidividual will only render the models individually
    Unique will render the data's unique model

Plots can be found in output folder: renderings/
'''
PA.render_models(pop, figsizegrouped=(30,30), samples=pop['order'], figsizesingle=(6,5), mode='grouped');
```

    Meprednisone
    ['Monocytes', 'Monocytes', 'T cells']
    Dexrazoxane HCl (ICRF-187, ADR-529)
    ['Monocytes', 'Monocytes', 'T cells']
    Etodolac
    ['Monocytes', 'T cells', 'Monocytes']
    Phenylbutazone
    ['Monocytes', 'T cells', 'Monocytes']
    Pomalidomide
    ['Monocytes', 'T cells', 'Monocytes']
    Budesonide
    ['Monocytes', 'T cells', 'Monocytes']
    Loteprednol etabonate
    ['Monocytes', 'Monocytes', 'T cells']
    CTRL6
    ['Monocytes', 'Monocytes', 'T cells']
    Betamethasone Valerate
    ['Monocytes', 'T cells', 'Monocytes']
    Febuxostat
    ['Monocytes', 'T cells', 'Monocytes']
    Sulfasalazine
    ['Monocytes', 'Monocytes', 'T cells']
    CTRL2
    ['Monocytes', 'T cells', 'Monocytes']
    Flurbiprofen
    ['Monocytes', 'T cells', 'Monocytes']
    Mianserin HCl
    ['Monocytes', 'T cells', 'Monocytes']
    Naproxen Sodium
    ['Monocytes', 'Monocytes', 'T cells']
    CTRL4
    ['Monocytes', 'Monocytes', 'T cells']
    Triamcinolone Acetonide
    ['Monocytes', 'T cells', 'Monocytes']
    Acetylcysteine
    ['Monocytes', 'Monocytes', 'T cells']
    Everolimus (RAD001)
    ['Monocytes', 'Monocytes', 'T cells']
    Isotretinoin
    ['Monocytes', 'T cells', 'Monocytes']
    Ibuprofen
    ['T cells', 'Monocytes', 'Monocytes']
    Doramapimod (BIRB 796)
    ['Monocytes', 'Monocytes', 'T cells']
    Allopurinol Sodium
    ['Monocytes', 'T cells', 'Monocytes']
    Tranilast
    ['Monocytes', 'Monocytes', 'T cells']
    Cyclosporine
    ['Monocytes', 'T cells', 'Monocytes']
    Adapalene
    ['Monocytes', 'T cells', 'Monocytes']
    Thalidomide
    ['Monocytes', 'T cells', 'Monocytes']
    Celecoxib
    ['Monocytes', 'T cells', 'Monocytes']
    CTRL3
    ['Monocytes', 'T cells', 'Monocytes']
    Ketoprofen
    ['Monocytes', 'T cells', 'Monocytes']
    Loratadine
    ['Monocytes', 'T cells', 'Monocytes']
    Tazarotene
    ['T cells', 'Monocytes', 'Monocytes']
    Bimatoprost
    ['Monocytes', 'T cells', 'Monocytes']
    Cetirizine DiHCl
    ['Monocytes', 'T cells', 'Monocytes']
    Mesalamine
    ['T cells', 'Monocytes', 'Monocytes']
    Mercaptopurine (6-MP)
    ['Monocytes', 'Monocytes', 'T cells']
    CTRL1
    ['Monocytes', 'Monocytes', 'T cells']
    Busulfan
    ['T cells', 'Monocytes', 'Monocytes']
    CTRL5
    ['T cells', 'Monocytes', 'Monocytes']
    Zafirlukast
    ['Monocytes', 'T cells', 'Monocytes']
    Zileuton
    ['Monocytes', 'Monocytes', 'T cells']
    Latrepirdine 2HCl
    ['Monocytes', 'Monocytes', 'T cells']
    Rapamycin (Sirolimus)
    ['Monocytes', 'T cells', 'Monocytes']
    Alprostadil
    ['Monocytes', 'T cells', 'Monocytes']
    Masitinib (AB1010)
    ['Monocytes', 'T cells', 'Monocytes']
    Methotrexate
    ['Monocytes', 'T cells', 'Monocytes']

---

### Align subpopulations

---

'Alignment' in the PopAlign framework is a process of associating subpopulations(represented by mixture components) based on transcriptional similarity. Alignment for each mixture component in a given sample is performed by finding the 'closest' reference mixture component in terms of position and shape in feature space. Mathematically, to define closeness, we use the Jeffreys divergence, a statistical metric of similarity on probability distributions. Specifically, for each mixture component $\phi^\text{test}_{i} \in \{\phi_i(\mathbf{c})\}^\text{test}$, we find a $\phi^\text{ref}_j \in \{\phi_{j}(\mathbf{c})\}^{\text{ref}}$, the closest mixture component in the reference set:
\begin{align}
\arg\min_{j}  D_{\text{JD}} \ (\phi^\text{test}_i(\mathbf{c})\ \| \ \phi^\text{ref}_j(\mathbf{c})),
\end{align}
where the minimization is performed over each $\{\phi^\text{ref}_i(\mathbf{c})\}$ in the set of reference mixture components.

The orientation of alignment can impact the exact associations made. In the PopAlign package, we provide the option to compute alignments in either a one-way or a two-way configuration:

*  'test2ref' (default):  Each component in every test sample is associated with its closest component in the reference sample. The underlying conceptual idea is that the reference sample contains a dictionary of cell-states whose complete representation in other samples is important. For some reference components, there may be no alignments, or in other cases there may be multiple alignments, potentially indicating a branching event. The caveat in this orientation is that sometimes spurious alignments can be generated when cell types are missing from the reference population. In that case, it may be more appropriate to choose a reference with a complete set of cell types or to use the more stringent 'conservative' option.

*  'ref2test': Each component in the reference sample is associated with its closes component in the test sample. Here, governing idea is that the reference population has a collection of cell states for which it is important to find their best-match within the test samples.

*  'conservative': two-way alignment, component pairs are only aligned if they are each others best match

To access the alignments, use:

`pop['samples']['sample1']['alignments']`
`pop['samples']['sample1']['replicates'][repnumber]['alignments']`

In this array, the first column stores the index of the sample component , the second column stores the index of the reference component and the third column stores the Jeffreys divergence between the two densities.

The pairwise Jeffreys divergences between sample1 and the reference sample model can be found at:

`pop['samples']['sample1']['fullalignments']` or
`pop['samples']['sample1']['replicates'][repnumber]['fullalignments']`

In this array, the rows indicate the test sample mixture components and the columns are the reference sample mixture components.

### **All L1 Comparisons for Chari, Weissbourd, Gehring, Ferraioli Below**

```
'''
Align the components of each sample's model to the components of a reference model

Parameters
----------
pop : dict
    Popalign object
ref : str
    Name of reference sample
method : str
    Method to perform the alignment
    If 'conservative', the reference component and the test component have to be each other's best match to align
    If 'test2ref', the closest reference component is found for each test component
    If 'ref2test', the closest test component is found for each test component
figsizedeltas : tuple, optional
    Size of the figure for the delta plot. Default is (10,5)
figsizeentropy : tuple, optional
    Size of the figure for the entropy plot. Default is (10,5)
'''
PA.align(pop, ref='CTRL1',
         method='test2ref', # one of: test2ref, ref2test, conservative
         figsizedeltas=(10,10),
         figsizeentropy=(10,10))
```

```
import pandas as pd
```

```
meta = pd.read_csv(folder + 'meta.csv')
meta.head()
```

      cell_barcode
      sample_id
      sample_number

      0
      AAACCCAAGGGTTAGC1
      Meprednisone
      66

      1
      AAACCCACACTAGGCC1
      Dexrazoxane HCl (ICRF-187, ADR-529)
      73

      2
      AAACCCACACTGAGGA1
      Etodolac
      3

      3
      AAACCCACATGGTACT1
      Phenylbutazone
      6

      4
      AAACCCAGTGGGCTTC1
      Pomalidomide
      52

```
#Set names
names = np.unique(meta.sample_id)
len(names)
```

    46

```
#Set control/reference (same as in paper)
gmm = pop['samples']['CTRL1']['gmm']
controlC = PA.get_coeff(pop, 'CTRL1')
controlPred = gmm.predict(C)
```

```
#Go through all perturbation conditions
labs = []
dists = []
perturb = []

for n in names:

  #Get predictions for types
  gmm = pop['samples'][n]['gmm']
  C = PA.get_coeff(pop, n)

  prediction = gmm.predict(C)

  aligns = pop['samples'][n]['alignments']

  labels = pop['samples'][n]['gmm_types']

  #Using aligments, get L1 distance from centroid of test cells in each of three groups to corresponding centroid in control
  for i in range(3):

    test = int(aligns[i,0])
    con = int(aligns[i,1])

    #Embeddings
    testMat = C[prediction == test,:]
    controlMat = controlC[controlPred == con,:]

    #Get centroids
    diff = testMat.mean(axis=0) - controlMat.mean(axis=0)
    #
    dist = np.linalg.norm(diff,1)

    labs += [labels[test]]
    dists += [dist]
    perturb += [n]

perturbDists = pd.DataFrame()
perturbDists['Type'] = labs
perturbDists['Distance'] = dists
perturbDists['Perturbation'] = perturb

perturbDists.head()

```

      Type
      Distance
      Perturbation

      0
      Monocytes
      3.418679
      Acetylcysteine

      1
      Monocytes
      2.842481
      Acetylcysteine

      2
      T cells
      1.230305
      Acetylcysteine

      3
      Monocytes
      5.714756
      Adapalene

      4
      T cells
      2.455030
      Adapalene

```
pop['samples']['Acetylcysteine']['alignments']
```

    array([[0.        , 0.        , 0.1116422 ],
           [1.        , 0.        , 0.22988867],
           [2.        , 2.        , 0.30961045]])

```
gmm = pop['samples']['CTRL1']['gmm']
otherC = PA.get_coeff(pop, 'CTRL1')

otherPred = gmm.predict(otherC)

otherLab = pop['samples']['CTRL1']['gmm_types']

m_o_idx = np.where(np.array(otherLab) == 'Monocytes')
t_o_idx = np.where(np.array(otherLab) == 'T cells')

#Embeddings

testMat = otherC[np.isin(otherPred, m_o_idx),:]
testMat.shape
#controlMat = otherC[otherPred.isin(t_o_idx),:]
```

    (396, 16)

```
#Control cell distances
controls = ['CTRL1','CTRL2','CTRL3','CTRL4','CTRL5','CTRL6']

#Go through all perturbation conditions
labs = []
dists = []
perturb = []

for n in controls:

  #Get predictions for types
  gmm = pop['samples'][n]['gmm']
  C = PA.get_coeff(pop, n)

  prediction = gmm.predict(C)

  labels = pop['samples'][n]['gmm_types']

  m_idx = np.where(np.array(labels) == 'Monocytes')
  t_idx = np.where(np.array(labels) == 'T cells')
  #Using aligments, get L1 distance from centroid of test cells in each of three groups to corresponding centroid in control
  others = [i for i in controls if i != n]
  for o in others:

    #Get predictions for types
    gmm = pop['samples'][o]['gmm']
    otherC = PA.get_coeff(pop, o)

    otherPred = gmm.predict(otherC)

    otherLab = pop['samples'][o]['gmm_types']

    m_o_idx = np.where(np.array(otherLab) == 'Monocytes')
    t_o_idx = np.where(np.array(otherLab) == 'T cells')

    #Embeddings

    testMat = C[np.isin(prediction,m_idx),:]
    controlMat = otherC[np.isin(otherPred,t_o_idx),:]

    #Get centroids
    diff = testMat.mean(axis=0) - controlMat.mean(axis=0)
    #
    dist = np.linalg.norm(diff,1)

    labs += ['Monocytes']
    dists += [dist]
    perturb += [n]

    #Embeddings
    testMat = C[np.isin(prediction,t_idx),:]
    controlMat = otherC[np.isin(otherPred,m_o_idx),:]

    #Get centroids
    diff = testMat.mean(axis=0) - controlMat.mean(axis=0)
    #
    dist = np.linalg.norm(diff,1)

    labs += ['T cells']
    dists += [dist]
    perturb += [n]

controlDists = pd.DataFrame()
controlDists['Type'] = labs
controlDists['Distance'] = dists
controlDists['Control'] = perturb

controlDists.head()

```

      Type
      Distance
      Control

      0
      Monocytes
      26.541359
      CTRL1

      1
      T cells
      25.707942
      CTRL1

      2
      Monocytes
      25.468264
      CTRL1

      3
      T cells
      25.496175
      CTRL1

      4
      Monocytes
      25.589360
      CTRL1

```
min(controlDists.Distance)
```

    25.208007542183847

```
max(perturbDists.Distance)
```

    12.323728932635067

Plot L1 Distance results

```
import matplotlib.pyplot as plt
import matplotlib

%matplotlib inline
import seaborn as sns
sns.set(style="whitegrid")
from sklearn.metrics import pairwise_distances

matplotlib.rcParams['figure.dpi'] =300
```

```
plt.figure(figsize=(8,4))
plt.grid(False)
plt.hist(controlDists.Distance, color='skyblue',bins=5,alpha=0.6,label ='Inter-type Distances' ,density=True)
plt.hist(perturbDists.Distance,color='purple',bins= 15,alpha=0.4,label = 'Intra-type Distances (State Shifts)',density=True)

plt.yticks(fontsize=10 )
plt.xticks(fontsize=10 )

plt.xlabel('Pairwise L1 Distance',fontsize=10)
plt.ylabel('Frequency',fontsize=10)
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.,fontsize=10)
plt.show()
#sns.histplot(data=l1dists, x="sepal_width", color="red", label="Sepal Width")

```

```
#Top perturbed conditions from original analysis
mono_top = ['Triamcinolone Acetonide','Budesonide','Betamethasone Valerate','Loteprednol etabonate','Alprostadil','Rapamycin (Sirolimus)','Everolimus (RAD001)','Meprednisone']
t_top = ['Dexrazoxane HCl (ICRF-187, ADR-529)']

#['Dexrazoxane HCl (ICRF-187, ADR-529)','Loteprednol etabonate','Acetylcysteine','Budesonide','Alprostadil','Rapamycin (Sirolimus)','Everolimus (RAD001)','Meprednisone']

```

```
def topRet(topBool):
  if topBool:
    return 'Top'
  else:
    return 'Other'
```

```
test = perturbDists[perturbDists.Type == 'Monocytes'].groupby(['Perturbation']).mean()
test.head()
```

      Distance

      Perturbation

      Acetylcysteine
      3.130580

      Adapalene
      4.510336

      Allopurinol Sodium
      3.901668

      Alprostadil
      7.845527

      Betamethasone Valerate
      9.681471

```
test['Top'] = [topRet(i in mono_top) for i in test.index]
test.head()
```

      Distance
      Top

      Perturbation

      Acetylcysteine
      3.130580
      Other

      Adapalene
      4.510336
      Other

      Allopurinol Sodium
      3.901668
      Other

      Alprostadil
      7.845527
      Top

      Betamethasone Valerate
      9.681471
      Top

```
plt.figure(figsize=(4,2))
plt.grid(False)
s = sns.scatterplot(data=test.sort_values(by='Distance'), x='Perturbation', y='Distance',hue='Top',palette=['grey','red'],s=10,alpha=0.6,legend = False) #sort_values(['Dist','Type'])

plt.xticks(
    rotation=90,
    horizontalalignment='right',
    fontweight='light',
    fontsize=4
)
plt.yticks(fontsize=5 )
plt.ylabel('L1 Distance',fontsize=5)
plt.xlabel('Perturbation/Condition',fontsize=5)

plt.show()
```

```
test2 = perturbDists[perturbDists.Type == 'T cells'].groupby(['Perturbation']).mean()
test2.head()
```

      Distance

      Perturbation

      Acetylcysteine
      1.230305

      Adapalene
      2.455030

      Allopurinol Sodium
      1.338489

      Alprostadil
      2.340914

      Betamethasone Valerate
      3.754966

```
test2['Top'] = [topRet(i in t_top) for i in test2.index]
test2.head()
```

      Distance
      Top

      Perturbation

      Acetylcysteine
      1.230305
      Other

      Adapalene
      2.455030
      Other

      Allopurinol Sodium
      1.338489
      Other

      Alprostadil
      2.340914
      Other

      Betamethasone Valerate
      3.754966
      Other

```
plt.figure(figsize=(4,2))
plt.grid(False)
s = sns.scatterplot(data=test2.sort_values(by='Distance'), x='Perturbation', y='Distance',hue='Top',palette=['grey','red'],s=10,alpha=0.6,legend=False) #sort_values(['Dist','Type'])

plt.xticks(
    rotation=90,
    horizontalalignment='right',
    fontweight='light',
    fontsize=4
)
plt.yticks(fontsize=5 )
plt.ylabel('L1 Distance',fontsize=5)
plt.xlabel('Perturbation/Condition',fontsize=5)

plt.show()
```


# DISTIL Usage Example: CIFAR-10 with Rare Classes

Not all datasets have an even spread of classes among the set of labels. Indeed, a dataset might have only a couple elements that have a particular class as a label. Such classes are considered *rare*, and extra work is required to achieve good model performance on these examples. The typical fix is to provide more data with the rare-class label; however, this issue is complicated by the fact that this data is usually part of a massive unlabeled pool of data. Here, we show how to use DISTIL's implementation of [SIMILAR](https://arxiv.org/abs/2107.00717) to mine these rare examples for labeling.

# Preparation

## Installation and Imports

```python
# Get DISTIL
!git clone https://github.com/decile-team/distil.git
!pip install -r distil/requirements/requirements.txt

import copy
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
import torch
import torch.optim as optim
import torch.nn.functional as F

from torch import nn
from torch.autograd import Variable
from torch.utils.data import Dataset, Subset, ConcatDataset, DataLoader
from torchvision import transforms
from torchvision.datasets import cifar

sys.path.append('distil/')
from distil.active_learning_strategies import SMI, GLISTER, BADGE, EntropySampling, RandomSampling  # All active learning strategies showcased in this example
from distil.utils.models.resnet import ResNet18                                                     # The model used in our image classification example
from distil.utils.train_helper import data_train                                                    # A utility training class provided by DISTIL
from distil.utils.utils import LabeledToUnlabeledDataset                                            # A utility wrapper class that removes labels from labeled PyTorch dataset objects
```

    Cloning into 'distil'...
    remote: Enumerating objects: 3324, done.
    remote: Counting objects: 100% (1281/1281), done.
    remote: Compressing objects: 100% (812/812), done.
    remote: Total 3324 (delta 794), reused 841 (delta 461), pack-reused 2043
    Receiving objects: 100% (3324/3324), 23.05 MiB | 21.87 MiB/s, done.
    Resolving deltas: 100% (2067/2067), done.
    Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple/
    Collecting sphinxcontrib-bibtex>=2.3.0
      Downloading sphinxcontrib_bibtex-2.4.1-py3-none-any.whl (38 kB)
    Collecting multipledispatch==0.6.0
      Downloading multipledispatch-0.6.0-py3-none-any.whl (11 kB)
    Collecting scikit-learn==0.23.0
      Downloading scikit_learn-0.23.0-cp37-cp37m-manylinux1_x86_64.whl (7.3 MB)
         |████████████████████████████████| 7.3 MB 12.8 MB/s
    Requirement already satisfied: numpy>=1.14.2 in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 4)) (1.19.5)
    Requirement already satisfied: scipy>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 5)) (1.4.1)
    Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 6)) (1.10.0+cu111)
    Requirement already satisfied: tqdm>=4.24.0 in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 7)) (4.62.3)
    Requirement already satisfied: numba>=0.43.0 in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 8)) (0.51.2)
    Collecting submodlib
      Downloading https://test-files.pythonhosted.org/packages/55/62/88e02a0e170498f38f7b9ce22b3e0a6a3cf9c82a33d3553da693c5c52872/submodlib-1.1.5.tar.gz (83 kB)
         |████████████████████████████████| 83 kB 1.1 MB/s
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
        Preparing wheel metadata ... done
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 12)) (1.1.5)
    Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (from -r distil/requirements/requirements.txt (line 13)) (0.11.1+cu111)
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from multipledispatch==0.6.0->-r distil/requirements/requirements.txt (line 2)) (1.15.0)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.23.0->-r distil/requirements/requirements.txt (line 3)) (1.1.0)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn==0.23.0->-r distil/requirements/requirements.txt (line 3)) (3.0.0)
    Collecting Sphinx>=2.1
      Downloading Sphinx-4.3.1-py3-none-any.whl (3.1 MB)
         |████████████████████████████████| 3.1 MB 55.1 MB/s
    Collecting pybtex-docutils>=1.0.0
      Downloading pybtex_docutils-1.0.1-py3-none-any.whl (4.8 kB)
    Requirement already satisfied: docutils>=0.8 in /usr/local/lib/python3.7/dist-packages (from sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (0.17.1)
    Collecting pybtex>=0.20
      Downloading pybtex-0.24.0-py2.py3-none-any.whl (561 kB)
         |████████████████████████████████| 561 kB 52.6 MB/s
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.4.0->-r distil/requirements/requirements.txt (line 6)) (3.10.0.2)
    Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.43.0->-r distil/requirements/requirements.txt (line 8)) (0.34.0)
    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.43.0->-r distil/requirements/requirements.txt (line 8)) (57.4.0)
    Requirement already satisfied: PyYAML>=3.01 in /usr/local/lib/python3.7/dist-packages (from pybtex>=0.20->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (3.13)
    Collecting latexcodec>=1.0.4
      Downloading latexcodec-2.0.1-py2.py3-none-any.whl (18 kB)
    Collecting sphinxcontrib-htmlhelp>=2.0.0
      Downloading sphinxcontrib_htmlhelp-2.0.0-py2.py3-none-any.whl (100 kB)
         |████████████████████████████████| 100 kB 12.9 MB/s
    Collecting sphinxcontrib-applehelp
      Downloading sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
         |████████████████████████████████| 121 kB 55.1 MB/s
    Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (1.1.5)
    Collecting sphinxcontrib-jsmath
      Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
    Requirement already satisfied: Jinja2>=2.3 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.11.3)
    Collecting sphinxcontrib-devhelp
      Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
         |████████████████████████████████| 84 kB 4.3 MB/s
    Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.2.0)
    Requirement already satisfied: imagesize in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (1.3.0)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (21.3)
    Collecting sphinxcontrib-qthelp
      Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
         |████████████████████████████████| 90 kB 11.3 MB/s
    Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.6.1)
    Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (0.7.12)
    Requirement already satisfied: babel>=1.3 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.9.1)
    Requirement already satisfied: requests>=2.5.0 in /usr/local/lib/python3.7/dist-packages (from Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.23.0)
    Requirement already satisfied: pytz>=2015.7 in /usr/local/lib/python3.7/dist-packages (from babel>=1.3->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2018.9)
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from Jinja2>=2.3->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.0.1)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.5.0->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.5.0->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.5.0->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.5.0->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (2.10)
    Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from submodlib->-r distil/requirements/requirements.txt (line 11)) (0.0)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->-r distil/requirements/requirements.txt (line 12)) (2.8.2)
    Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision->-r distil/requirements/requirements.txt (line 13)) (7.1.2)
    Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->Sphinx>=2.1->sphinxcontrib-bibtex>=2.3.0->-r distil/requirements/requirements.txt (line 1)) (3.0.6)
    Building wheels for collected packages: submodlib
      Building wheel for submodlib (PEP 517) ... done
      Created wheel for submodlib: filename=submodlib-1.1.5-cp37-cp37m-linux_x86_64.whl size=512655 sha256=5f42dc2a440923593a5ac15c93121875e37f2103e924b5f2e7d3836a671a3b77
      Stored in directory: /root/.cache/pip/wheels/17/3c/03/48ce7dd03798c0564b61e61020e733443aed88e115518442b4
    Successfully built submodlib
    Installing collected packages: latexcodec, sphinxcontrib-qthelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, sphinxcontrib-devhelp, sphinxcontrib-applehelp, scikit-learn, pybtex, Sphinx, pybtex-docutils, submodlib, sphinxcontrib-bibtex, multipledispatch
      Attempting uninstall: scikit-learn
        Found existing installation: scikit-learn 1.0.1
        Uninstalling scikit-learn-1.0.1:
          Successfully uninstalled scikit-learn-1.0.1
      Attempting uninstall: Sphinx
        Found existing installation: Sphinx 1.8.6
        Uninstalling Sphinx-1.8.6:
          Successfully uninstalled Sphinx-1.8.6
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    imbalanced-learn 0.8.1 requires scikit-learn>=0.24, but you have scikit-learn 0.23.0 which is incompatible.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
    Successfully installed Sphinx-4.3.1 latexcodec-2.0.1 multipledispatch-0.6.0 pybtex-0.24.0 pybtex-docutils-1.0.1 scikit-learn-0.23.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-bibtex-2.4.1 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 submodlib-1.1.5

## Preparing CIFAR-10 with Rare Classes

The CIFAR10 dataset contains 60,000 32x32 color images in 10 different classes.The 10 different classes represent airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each class. The training set contains 50,000 images, and the test set contains 10,000 images. Here, we do a simple setup of the CIFAR10 dataset that we will use in this example. More importantly, we define a split on CIFAR10's training set into an initial labeled seed set and an unlabeled set. We also impose an artificial imbalance among the classes to simulate a rare-class scenario.

```python
# Define the name of the dataset and the path that PyTorch should use when downloading the data
data_set_name = 'CIFAR10'
download_path = '.'

# Define the number of classes in CIFAR10
nclasses = 10

# Define transforms on the dataset splits of CIFAR10. Here, we use random crops and horizontal flips for training augmentations.
# Both the train and test sets are converted to PyTorch tensors and are normalized around the mean/std of CIFAR-10.
cifar_training_transform = transforms.Compose([transforms.RandomCrop(32, padding=4), transforms.RandomHorizontalFlip(), transforms.ToTensor(), transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])
cifar_test_transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

# Get the dataset objects from PyTorch. Here, CIFAR10 is downloaded, and the transform is applied when points
# are retrieved.
cifar10_full_train = cifar.CIFAR10(download_path, train=True, download=True, transform=cifar_training_transform)
cifar10_test = cifar.CIFAR10(download_path, train=False, download=True, transform=cifar_test_transform)

# Get the dimension of the images. Here, we simply take the very first image of CIFAR10
# and query its dimension.
dim = np.shape(cifar10_full_train[0][0])

# We now define a train-unlabeled split for the sake of the experiment. Here, we specify the ratio of common classes to rare classes
# for both the initial training seed set and the unlabeled set. We use a ratio of 10:1.
num_examples_per_common_class_seed = 400
num_examples_per_rare_class_seed = 40

num_examples_per_common_class_unlabeled = 3000
num_examples_per_rare_class_unlabeled = 300

# We create the imbalance on classes 5,6,7,8,9.
rare_classes = [5,6,7,8,9]

# Create the imbalance by choosing which indices of the full training set to assign to the initial labeled seed set and the unlabeled set
train_idx = []
unlabeled_idx = []

for class_idx in range(nclasses):

    # Retrieve all the indices of the elements in CIFAR-10 whose label matches class_idx
    full_idx_class = torch.where(torch.Tensor(cifar10_full_train.targets) == class_idx)[0].cpu().numpy()

    # Determine how many points to add for this class depending on the rarity of the class
    if class_idx in rare_classes:
        class_num_training_examples_to_add = num_examples_per_rare_class_seed
        class_num_unlabeled_examples_to_add = num_examples_per_rare_class_unlabeled
    else:
        class_num_training_examples_to_add = num_examples_per_common_class_seed
        class_num_unlabeled_examples_to_add = num_examples_per_common_class_unlabeled

    # Choose randomly a subset of these indices. These will be added to the initial training seed set.
    class_train_idx = np.random.choice(full_idx_class, size=class_num_training_examples_to_add, replace=False)

    # Choose randomly a subset of the remaining indices. These will be added to the unlabeled set.
    remaining_class_idx = np.array(list(set(full_idx_class) - set(class_train_idx)))
    class_unlabeled_idx = np.random.choice(remaining_class_idx, size=class_num_unlabeled_examples_to_add, replace=False)

    # Add the chosen indices to the growing subset lists
    train_idx.extend(class_train_idx)
    unlabeled_idx.extend(class_unlabeled_idx)

# Create the train and unlabeled subsets based on the index lists above. While the unlabeled set constructed here technically has labels, they
# are only used when querying for labels. Hence, they only exist here for the sake of experimental design.
cifar10_train = Subset(cifar10_full_train, train_idx)
cifar10_unlabeled = Subset(cifar10_full_train, unlabeled_idx)
```

    Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./cifar-10-python.tar.gz

      0%|          | 0/170498071 [00:00<?, ?it/s]

    Extracting ./cifar-10-python.tar.gz to .
    Files already downloaded and verified

## Preparing the Model

Here, we use DISTIL's provided implementation of the [ResNet-18](https://arxiv.org/abs/1512.03385) architecture. We also create a model directory to store trained models in this example.

```python
net = ResNet18()
```

# Initial Round

## Training

Now that we have prepared the training data and the model, we can begin training an initial model. We use DISTIL's provided [training loop](https://github.com/decile-team/distil/blob/main/distil/utils/train_helper.py) to do training.

```python
# Define the training arguments to use.
args = {'n_epoch':      300,    # Stop training after 300 epochs.
        'lr':           0.01,   # Use a learning rate of 0.01
        'batch_size':   20,     # Update the parameters using training batches of size 20
        'max_accuracy': 0.99,   # Stop training once the training accuracy has exceeded 0.99
        'optimizer':    'sgd',  # Use the stochastic gradient descent optimizer
        'device':       "cuda" if torch.cuda.is_available() else "cpu"  # Use a GPU if one is available
        }

# Create the training loop using our training dataset, provided model, and training arguments.
# Train an initial model.
dt = data_train(cifar10_train, net, args)
trained_model = dt.train()
```

    Training..
    Epoch: 105 Training accuracy: 0.991

## Evaluation

How does our initial model do on the test set? Furthermore, how does our initial model do on the *rare classes* of the test set? Luckily, the training loop provided by DISTIL also provides a way to measure the accuracy of the model on a given dataset. We measure both accuracies here.

```python
# Get the full test accuracy
full_test_accuracy = dt.get_acc_on_set(cifar10_test)
print(F"Full Test Accuracy: {full_test_accuracy}")

# Get the per-class test accuracies
rare_indices = []
for class_idx in range(nclasses):

    # Get the indices of the test set corresponding to this class
    test_rare_class_subset_idx = torch.where(torch.Tensor(cifar10_test.targets) == class_idx)[0].cpu().numpy()

    if class_idx in rare_classes:
        rare_indices.extend(test_rare_class_subset_idx)

    # Get the accuracy on this class subset
    cifar10_test_class_subset = Subset(cifar10_test, test_rare_class_subset_idx)
    rare_class_test_accuracy = dt.get_acc_on_set(cifar10_test_class_subset)
    print(F"Class {class_idx} Test Accuracy: {rare_class_test_accuracy}")

# Get accuracy on all rare points
cifar10_test_rare_subset = Subset(cifar10_test, rare_indices)
rare_test_accuracy = dt.get_acc_on_set(cifar10_test_rare_subset)
print(F"Rare Test Accuracy: {rare_test_accuracy}")
```

    Full Test Accuracy: 0.5509
    Class 0 Test Accuracy: 0.839
    Class 1 Test Accuracy: 0.894
    Class 2 Test Accuracy: 0.689
    Class 3 Test Accuracy: 0.738
    Class 4 Test Accuracy: 0.735
    Class 5 Test Accuracy: 0.142
    Class 6 Test Accuracy: 0.286
    Class 7 Test Accuracy: 0.36
    Class 8 Test Accuracy: 0.35
    Class 9 Test Accuracy: 0.476
    Rare Test Accuracy: 0.3228

As we can see, the test performance on the rare classes is awful. Can we rectify this issue by adding rare-class examples?

# Mining Rare Classes

## Preparing a Query Set

In this example, we know that there are rare classes in the unlabeled dataset that can help us improve the accuracy of the model. How do we select these points if we do not know *where* they are in the unlabeled dataset? Here, we use [SIMILAR](https://arxiv.org/abs/2107.00717) to select rare-class examples from the unlabeled set. SIMILAR requires access to a query set of points that it uses to choose similar unlabeled points. Hence, we must first prepare this query set.

Where do we get the query set? Luckily, we already have a couple points in our training dataset to choose.

```python
# Go over the training dataset, getting the indices of all the rare-class examples
rare_training_example_indices = []
for index, (_, label) in enumerate(cifar10_train):
    if label in rare_classes:
        rare_training_example_indices.append(index)

# Create a query set that contains only the rare-class examples of the training dataset
rare_class_query_set = Subset(cifar10_train, rare_training_example_indices)
```

## Using DISTIL's SIMILAR Implementation: SMI

Now that we have the query set, we are ready to use DISTIL's implementation of SIMILAR. In particular, we use the submodular mutual information [strategy](https://github.com/decile-team/distil/blob/main/distil/active_learning_strategies/smi.py) that is detailed in [SIMILAR](https://arxiv.org/abs/2107.00717). This will allow us to select a set of points to label within a specified budget $k$.

Specifically, the strategy attempts to maximize the [submodular mutual information](https://arxiv.org/abs/2006.15412) between a subset $\mathcal{A}$ of size no greater than $k$ of the unlabeled dataset $\mathcal{U}$ and the query set $\mathcal{Q}$:

\begin{align}
\text{argmax}_{\mathcal{A} \subseteq \mathcal{U}, |\mathcal{A}|\leq k} I_F(\mathcal{A};\mathcal{Q})
\end{align}

where $F$ is a submodular set function.

```python
# Define arguments for SMI
selection_strategy_args = {'device':        args['device'],       # Use the device used in training
                           'batch_size':    args['batch_size'],   # Use the batch size used in training
                           'smi_function':  'fl2mi',              # Use a facility location function, which captures representation information
                           'metric':        'cosine',             # Use cosine similarity when determining the likeness of two data points
                           'optimizer':     'LazyGreedy'          # When doing submodular maximization, use the lazy greedy optimizer
                          }

# Create the SMI selection strategy. Note: We remove the labels from the unlabeled portion of CIFAR-10 that we created earlier.
# In a practical application, one would not have these labels a priori.
selection_strategy = SMI(cifar10_train, LabeledToUnlabeledDataset(cifar10_unlabeled), rare_class_query_set, trained_model, nclasses, selection_strategy_args)

# Disable the augmentations used in the training dataset. Since all augmentations come from the cifar10_full_train object, we set its transform to the
# transform used by the test set.
cifar10_full_train.transform = cifar_test_transform

# Do the selection, which will return the indices of the selected points with respect to the unlabeled dataset.
budget = 750
selected_idx = selection_strategy.select(budget)

# Re-enable augmentations
cifar10_full_train.transform = cifar_training_transform
```

## Labeling the Points

Now that we know which points should be labeled, we can present them to human labelers for annotation. Here, we can do so automatically since we already know their labels for the sake of the example.

```python
# Form a labeled subset of the unlabeled dataset. Again, we already have the labels,
# so we simply take a subset. Note, however, that the selection was done without the
# use of the labels and that we would normally not have these labels. Hence, the
# following statement would usually require human effort to complete.
smi_human_labeled_dataset = Subset(cifar10_unlabeled, selected_idx)
```

## Characterizing the Selection

Now that we have selected and labeled these new points, we can add them to the training dataset and retrain our model. Before that, how many points did we select that actually were rare-class points?

```python
# Go over the newly labeled dataset, tallying the number of points seen in each class.
smi_class_counts = [0 for x in range(nclasses)]
for _, label in smi_human_labeled_dataset:
    smi_class_counts[label] += 1

# Print each class count
for class_idx, class_count in enumerate(smi_class_counts):
    print(F"Class {class_idx} count: {class_count}")

# Print total rare count
total_rare_count = sum([smi_class_counts[i] for i in rare_classes])
print(F"Total Rare Count: {total_rare_count}")
```

    Class 0 count: 47
    Class 1 count: 40
    Class 2 count: 43
    Class 3 count: 118
    Class 4 count: 81
    Class 5 count: 45
    Class 6 count: 65
    Class 7 count: 108
    Class 8 count: 74
    Class 9 count: 129
    Total Rare Count: 421

We were able to get a good number of rare-class points. For comparison sake, how many rare points would we get using some of DISTIL's other strategies?

**BADGE**

```python
# Repeat the previous steps
selection_strategy = BADGE(cifar10_train, LabeledToUnlabeledDataset(cifar10_unlabeled), trained_model, nclasses, selection_strategy_args)

cifar10_full_train.transform = cifar_test_transform
budget = 750
selected_idx = selection_strategy.select(budget)
cifar10_full_train.transform = cifar_training_transform

badge_human_labeled_dataset = Subset(cifar10_unlabeled, selected_idx)

badge_class_counts = [0 for x in range(nclasses)]
for _, label in badge_human_labeled_dataset:
    badge_class_counts[label] += 1

for class_idx, class_count in enumerate(badge_class_counts):
    print(F"Class {class_idx} count: {class_count}")

total_rare_count = sum([badge_class_counts[i] for i in rare_classes])
print(F"Total Rare Count: {total_rare_count}")
```

    Class 0 count: 114
    Class 1 count: 92
    Class 2 count: 139
    Class 3 count: 158
    Class 4 count: 125
    Class 5 count: 16
    Class 6 count: 29
    Class 7 count: 28
    Class 8 count: 19
    Class 9 count: 30
    Total Rare Count: 122

**Random**

```python
# Repeat the previous steps
selection_strategy = RandomSampling(cifar10_train, LabeledToUnlabeledDataset(cifar10_unlabeled), trained_model, nclasses, selection_strategy_args)

cifar10_full_train.transform = cifar_test_transform
budget = 750
selected_idx = selection_strategy.select(budget)
cifar10_full_train.transform = cifar_training_transform

random_human_labeled_dataset = Subset(cifar10_unlabeled, selected_idx)

random_class_counts = [0 for x in range(nclasses)]
for _, label in random_human_labeled_dataset:
    random_class_counts[label] += 1

for class_idx, class_count in enumerate(random_class_counts):
    print(F"Class {class_idx} count: {class_count}")

total_rare_count = sum([random_class_counts[i] for i in rare_classes])
print(F"Total Rare Count: {total_rare_count}")
```

    Class 0 count: 146
    Class 1 count: 128
    Class 2 count: 124
    Class 3 count: 142
    Class 4 count: 143
    Class 5 count: 16
    Class 6 count: 9
    Class 7 count: 14
    Class 8 count: 11
    Class 9 count: 17
    Total Rare Count: 67

**Entropy**

```python
# Repeat the previous steps
selection_strategy = EntropySampling(cifar10_train, LabeledToUnlabeledDataset(cifar10_unlabeled), trained_model, nclasses, selection_strategy_args)

cifar10_full_train.transform = cifar_test_transform
budget = 750
selected_idx = selection_strategy.select(budget)
cifar10_full_train.transform = cifar_training_transform

entropy_human_labeled_dataset = Subset(cifar10_unlabeled, selected_idx)

entropy_class_counts = [0 for x in range(nclasses)]
for _, label in entropy_human_labeled_dataset:
    entropy_class_counts[label] += 1

for class_idx, class_count in enumerate(entropy_class_counts):
    print(F"Class {class_idx} count: {class_count}")

total_rare_count = sum([entropy_class_counts[i] for i in rare_classes])
print(F"Total Rare Count: {total_rare_count}")
```

    Class 0 count: 96
    Class 1 count: 54
    Class 2 count: 126
    Class 3 count: 168
    Class 4 count: 172
    Class 5 count: 22
    Class 6 count: 34
    Class 7 count: 34
    Class 8 count: 22
    Class 9 count: 22
    Total Rare Count: 134

Hence, we can see that SMI does comparatively much better at selecting rare instances.

# Improving Performance

## Re-Training

Let us re-train our model using the newly selected points.

```python
# Create a new training dataset by concatenating what we have with the newly labeled points
new_training_dataset = ConcatDataset([cifar10_train, smi_human_labeled_dataset])
new_dt = data_train(new_training_dataset, copy.deepcopy(net), args)
new_trained_model = new_dt.train()
```

    Training..
    Epoch: 102 Training accuracy: 0.992

## Evaluation

Now, let us see the accuracy improvement.

```python
# Get the full test accuracy
full_test_accuracy_before = dt.get_acc_on_set(cifar10_test)
full_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test)

print(F"Full Test Accuracy Improvement: {full_test_accuracy_before} to {full_test_accuracy_after}")

# Get the per-class test accuracies
rare_indices = []
for class_idx in range(nclasses):

    # Get the indices of the test set corresponding to this class
    test_rare_class_subset_idx = torch.where(torch.Tensor(cifar10_test.targets) == class_idx)[0].cpu().numpy()

    if class_idx in rare_classes:
        rare_indices.extend(test_rare_class_subset_idx)

    # Get the accuracy on this class subset
    cifar10_test_class_subset = Subset(cifar10_test, test_rare_class_subset_idx)

    rare_class_test_accuracy_before = dt.get_acc_on_set(cifar10_test_class_subset)
    rare_class_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_class_subset)
    print(F"Class {class_idx} Test Accuracy: {rare_class_test_accuracy_before} to {rare_class_test_accuracy_after}")

# Get accuracy on all rare points
cifar10_test_rare_subset = Subset(cifar10_test, rare_indices)
rare_test_accuracy_before = dt.get_acc_on_set(cifar10_test_rare_subset)
rare_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_rare_subset)
print(F"Rare Test Accuracy: {rare_test_accuracy_before} to {rare_test_accuracy_after}")
```

    Full Test Accuracy Improvement: 0.5509 to 0.6512
    Class 0 Test Accuracy: 0.839 to 0.843
    Class 1 Test Accuracy: 0.894 to 0.918
    Class 2 Test Accuracy: 0.689 to 0.649
    Class 3 Test Accuracy: 0.738 to 0.769
    Class 4 Test Accuracy: 0.735 to 0.812
    Class 5 Test Accuracy: 0.142 to 0.283
    Class 6 Test Accuracy: 0.286 to 0.441
    Class 7 Test Accuracy: 0.36 to 0.537
    Class 8 Test Accuracy: 0.35 to 0.621
    Class 9 Test Accuracy: 0.476 to 0.639
    Rare Test Accuracy: 0.3228 to 0.5042

## Comparison

What would the accuracy improvement look like if we had used the other methods?

**BADGE**

```python
# Repeat the process
new_training_dataset = ConcatDataset([cifar10_train, badge_human_labeled_dataset])
new_dt = data_train(new_training_dataset, copy.deepcopy(net), args)
new_trained_model = new_dt.train()

full_test_accuracy_before = dt.get_acc_on_set(cifar10_test)
full_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test)

print(F"Full Test Accuracy Improvement: {full_test_accuracy_before} to {full_test_accuracy_after}")

rare_indices = []
for class_idx in range(nclasses):

    test_rare_class_subset_idx = torch.where(torch.Tensor(cifar10_test.targets) == class_idx)[0].cpu().numpy()

    if class_idx in rare_classes:
        rare_indices.extend(test_rare_class_subset_idx)

    cifar10_test_class_subset = Subset(cifar10_test, test_rare_class_subset_idx)

    rare_class_test_accuracy_before = dt.get_acc_on_set(cifar10_test_class_subset)
    rare_class_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_class_subset)
    print(F"Class {class_idx} Test Accuracy: {rare_class_test_accuracy_before} to {rare_class_test_accuracy_after}")

cifar10_test_rare_subset = Subset(cifar10_test, rare_indices)
rare_test_accuracy_before = dt.get_acc_on_set(cifar10_test_rare_subset)
rare_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_rare_subset)
print(F"Rare Test Accuracy: {rare_test_accuracy_before} to {rare_test_accuracy_after}")
```

    Training..
    Epoch: 122 Training accuracy: 0.994
    Full Test Accuracy Improvement: 0.5509 to 0.623
    Class 0 Test Accuracy: 0.839 to 0.883
    Class 1 Test Accuracy: 0.894 to 0.941
    Class 2 Test Accuracy: 0.689 to 0.725
    Class 3 Test Accuracy: 0.738 to 0.748
    Class 4 Test Accuracy: 0.735 to 0.797
    Class 5 Test Accuracy: 0.142 to 0.276
    Class 6 Test Accuracy: 0.286 to 0.312
    Class 7 Test Accuracy: 0.36 to 0.492
    Class 8 Test Accuracy: 0.35 to 0.441
    Class 9 Test Accuracy: 0.476 to 0.615
    Rare Test Accuracy: 0.3228 to 0.4272

**Random**

```python
# Repeat the process
new_training_dataset = ConcatDataset([cifar10_train, random_human_labeled_dataset])
new_dt = data_train(new_training_dataset, copy.deepcopy(net), args)
new_trained_model = new_dt.train()

full_test_accuracy_before = dt.get_acc_on_set(cifar10_test)
full_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test)

print(F"Full Test Accuracy Improvement: {full_test_accuracy_before} to {full_test_accuracy_after}")

rare_indices = []
for class_idx in range(nclasses):

    test_rare_class_subset_idx = torch.where(torch.Tensor(cifar10_test.targets) == class_idx)[0].cpu().numpy()

    if class_idx in rare_classes:
        rare_indices.extend(test_rare_class_subset_idx)

    cifar10_test_class_subset = Subset(cifar10_test, test_rare_class_subset_idx)

    rare_class_test_accuracy_before = dt.get_acc_on_set(cifar10_test_class_subset)
    rare_class_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_class_subset)
    print(F"Class {class_idx} Test Accuracy: {rare_class_test_accuracy_before} to {rare_class_test_accuracy_after}")

cifar10_test_rare_subset = Subset(cifar10_test, rare_indices)
rare_test_accuracy_before = dt.get_acc_on_set(cifar10_test_rare_subset)
rare_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_rare_subset)
print(F"Rare Test Accuracy: {rare_test_accuracy_before} to {rare_test_accuracy_after}")
```

    Training..
    Epoch: 118 Training accuracy: 0.991
    Full Test Accuracy Improvement: 0.5509 to 0.5915
    Class 0 Test Accuracy: 0.839 to 0.771
    Class 1 Test Accuracy: 0.894 to 0.932
    Class 2 Test Accuracy: 0.689 to 0.758
    Class 3 Test Accuracy: 0.738 to 0.612
    Class 4 Test Accuracy: 0.735 to 0.806
    Class 5 Test Accuracy: 0.142 to 0.232
    Class 6 Test Accuracy: 0.286 to 0.425
    Class 7 Test Accuracy: 0.36 to 0.341
    Class 8 Test Accuracy: 0.35 to 0.499
    Class 9 Test Accuracy: 0.476 to 0.539
    Rare Test Accuracy: 0.3228 to 0.4072

**Entropy**

```python
# Repeat the process
new_training_dataset = ConcatDataset([cifar10_train, entropy_human_labeled_dataset])
new_dt = data_train(new_training_dataset, copy.deepcopy(net), args)
new_trained_model = new_dt.train()

full_test_accuracy_before = dt.get_acc_on_set(cifar10_test)
full_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test)

print(F"Full Test Accuracy Improvement: {full_test_accuracy_before} to {full_test_accuracy_after}")

rare_indices = []
for class_idx in range(nclasses):

    test_rare_class_subset_idx = torch.where(torch.Tensor(cifar10_test.targets) == class_idx)[0].cpu().numpy()

    if class_idx in rare_classes:
        rare_indices.extend(test_rare_class_subset_idx)

    cifar10_test_class_subset = Subset(cifar10_test, test_rare_class_subset_idx)

    rare_class_test_accuracy_before = dt.get_acc_on_set(cifar10_test_class_subset)
    rare_class_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_class_subset)
    print(F"Class {class_idx} Test Accuracy: {rare_class_test_accuracy_before} to {rare_class_test_accuracy_after}")

cifar10_test_rare_subset = Subset(cifar10_test, rare_indices)
rare_test_accuracy_before = dt.get_acc_on_set(cifar10_test_rare_subset)
rare_test_accuracy_after = new_dt.get_acc_on_set(cifar10_test_rare_subset)
print(F"Rare Test Accuracy: {rare_test_accuracy_before} to {rare_test_accuracy_after}")
```

    Training..
    Epoch: 116 Training accuracy: 0.993
    Full Test Accuracy Improvement: 0.5509 to 0.6483
    Class 0 Test Accuracy: 0.839 to 0.879
    Class 1 Test Accuracy: 0.894 to 0.906
    Class 2 Test Accuracy: 0.689 to 0.699
    Class 3 Test Accuracy: 0.738 to 0.769
    Class 4 Test Accuracy: 0.735 to 0.829
    Class 5 Test Accuracy: 0.142 to 0.182
    Class 6 Test Accuracy: 0.286 to 0.456
    Class 7 Test Accuracy: 0.36 to 0.457
    Class 8 Test Accuracy: 0.35 to 0.623
    Class 9 Test Accuracy: 0.476 to 0.683
    Rare Test Accuracy: 0.3228 to 0.4802