<p>Update: The Fairlearn documentation now has a FAQ section on this topic <a href="https://fairlearn.org/main/faq.html" rel="nofollow noreferrer">https://fairlearn.org/main/faq.html</a> Search for &quot;Does Fairlearn support multi-class classification?&quot;</p>
<p>Previous answer:
Fairlearn's metrics are designed for binary classification or regression. You could evaluate the various labels individually, of course. If you have a specific idea of what you'd like to see please open a <a href="https://github.com/fairlearn/fairlearn/issues/new/choose" rel="nofollow noreferrer">new feature request</a>.</p>
<p>Fairlearn does support a variety of metrics, not just accuracy. The user guide has a full list: <a href="https://fairlearn.org/v0.6.0/user_guide/assessment.html#scalar-results-from-metricframe" rel="nofollow noreferrer">https://fairlearn.org/v0.6.0/user_guide/assessment.html#scalar-results-from-metricframe</a></p>
<p>One example that comes to mind for a paper doing multi-class classification while thinking about fairness is <a href="https://arxiv.org/abs/2003.00827" rel="nofollow noreferrer">CheXclusion</a> by Seyyed-Kalantari et al. They mostly look into TPR differences when classifying chest x-rays.</p>
<p>The Fairlearn community would definitely be interested in hearing about your use case. Perhaps there's some way we can help. Feel free to reach out via Gitter or by creating your feature request (as mentioned above).</p>
Fairness metrics for multi-class classification
<p>Are there any metrics implemented in Fairlearn or any published papers that I can refer to for use-cases around fairness measurement of multi-class classification where the metrics are AP and not accuracy? Thanks!</p>