# Group Metrics

The `fairlearn` package contains algorithms which enable machine learning models to minimise disparity between groups. The `metrics` portion of the package provides the means required to verify that the mitigation algorithms are succeeding.


```python
import numpy as np
import pandas as pd
import sklearn.metrics as skm
```

## Ungrouped Metrics

At their simplest, metrics take a set of 'true' values $Y_{true}$ (from the input data) and predicted values $Y_{pred}$ (by applying the model to the input data), and use these to compute a measure. For example, the _recall_ or _true positive rate_ is given by
\begin{equation}
P( Y_{pred}=1 | Y_{true}=1 )
\end{equation}
That is, a measure of whether the model finds all the positive cases in the input data. The `scikit-learn` package implements this in [sklearn.metrics.recall_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html).

Suppose we have the following data:


```python
Y_true = [0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1]
Y_pred = [0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1]
```

we can see that the prediction is 1 in five of the ten cases where the true value is 1, so we expect the recall to be 0.0.5:


```python
skm.recall_score(Y_true, Y_pred)
```

## Metrics with Grouping

When considering fairness, each row of input data will have an associated group label $g \in G$, and we will want to know how the metric behaves for each $g$. To help with this, `fairlearn` provides wrappers, which take an existing (ungrouped) metric function, and apply it to each group within a set of data.

Suppose in addition to the $Y_{true}$ and $Y_{pred}$ above, we had the following set of labels:


```python
group_membership_data = ['d', 'a', 'c', 'b', 'b', 'c', 'c', 'c', 'b', 'd', 'c', 'a', 'b', 'd', 'c', 'c']

df = pd.DataFrame({ 'Y_true': Y_true, 'Y_pred': Y_pred, 'group_membership_data': group_membership_data})
df
```

We can see that for the groups 'a' and 'd' the recall is 0 (none of the true positives were identified), for 'b' the recall is 0.5 and for 'c' the recall is 0.75.

The `fairlearn.metrics.metric_by_group` routine can calculate all of these for us. This takes as arguments an _ungrouped_ metric (such as `sklearn.metrics.recall_score`), the arrays $Y_{true}$ and $Y_{pred}$ and an array of group labels (and optionally, if the ungrouped metric supports it, an array of sample weights), and produces `GroupMetricResult` object:


```python
import fairlearn.metrics as flm

group_metrics = flm.metric_by_group(skm.recall_score, Y_true, Y_pred, group_membership_data, sample_weight=None)

print("Overall recall = ", group_metrics.overall)
print("recall by groups = ", group_metrics.by_group)
```

Note that the overall recall is the same as that calculated above in the Ungrouped Metric section, while the `by_group` dictionary matches the values we calculated by inspection from the table above.

In addition to these basic scores, `metric_by_group` also records the maximum and minimum values of the metric, the groups for which these occurred, and also the difference and ratio between the maximum and minimum:


```python
print("min recall over groups = ", group_metrics.minimum)
print("occurred for groups: ", group_metrics.argmin_set)
print()
print("max recall over groups = ", group_metrics.maximum)
print("occurred for groups: ", group_metrics.argmax_set)
print()
print("range in recall = ", group_metrics.range)
print("ratio in recall = ", group_metrics.range_ratio)
```

## Supported Ungrouped Metrics

To be used by `metric_by_group` the supplied Python function must take arguments of `y_true` and `y_pred`:
```python
my_metric_func(y_true, y_pred)
```
An additional argument of `sample_weight` is also supported:
```python
my_metric_with_weight(y_true, y_pred, sample_weight)
```
The `sample_weight` argument is always invoked by name, and _only_ if the user supplies a `sample_weight` argument.

## Convenience Wrapper

Rather than require a call to `metric_by_group` each time, we also provide a function which turns an ungrouped metric into a grouped one. This is called `make_group_metric`:


```python
group_recall_score = flm.make_group_metric(skm.recall_score)

results = group_recall_score(Y_true, Y_pred, group_membership_data)

print("Overall recall = ", results.overall)
print("recall by groups = ", results.by_group)
```


```python

```

bd6f7b16360b69d49bab2d69204f260e58f82b84
ipynb
Jupyter Notebook
notebooks/Group Metrics.ipynb
arita37/fairlearn
8f2cb25ca6a0c0aadcff90b927936de6d53ff5fa
MIT
2019-11-30T09:02:42.000Z
2019-12-02T10:24:29.000Z
notebooks/Group Metrics.ipynb
arita37/fairlearn
8f2cb25ca6a0c0aadcff90b927936de6d53ff5fa
MIT
notebooks/Group Metrics.ipynb
arita37/fairlearn
8f2cb25ca6a0c0aadcff90b927936de6d53ff5fa
MIT
Qwen/Qwen-72B
1. YES
2. YES