Update README.md
Browse files
README.md
CHANGED
@@ -496,7 +496,25 @@ BLEU score on [mExpresso](https://huggingface.co/facebook/seamless-expressive#me
|
|
496 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
497 |
| 1.23.0 | canary-1b | 23.84 | 35.74 | 28.29 |
|
498 |
|
|
|
499 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
500 |
|
501 |
## NVIDIA Riva: Deployment
|
502 |
|
|
|
496 |
|:-----------:|:---------:|:----------:|:----------:|:----------:|
|
497 |
| 1.23.0 | canary-1b | 23.84 | 35.74 | 28.29 |
|
498 |
|
499 |
+
## Model Fairness Evaluation
|
500 |
|
501 |
+
As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset", we assessed the parakeet-tdt-1.1b model for fairness. The model was evaluated on the CausalConversations-v1 dataset, and the results are reported as follows:
|
502 |
+
|
503 |
+
### Gender Bias:
|
504 |
+
|
505 |
+
| Gender | Male | Female | N/A | Other |
|
506 |
+
| :--- | :--- | :--- | :--- | :--- |
|
507 |
+
| Num utterances | 19325 | 24532 | 926 | 33 |
|
508 |
+
| % WER | 14.64 | 12.92 | 17.88 | 126.92 |
|
509 |
+
|
510 |
+
### Age Bias:
|
511 |
+
|
512 |
+
| Age Group | (18-30) | (31-45) | (46-85) | (1-100) |
|
513 |
+
| :--- | :--- | :--- | :--- | :--- |
|
514 |
+
| Num utterances | 15956 | 14585 | 13349 | 43890 |
|
515 |
+
| % WER | 14.64 | 13.07 | 13.47 | 13.76 |
|
516 |
+
|
517 |
+
(Error rates for fairness evaluation are determined by normalizing both the reference and predicted text, similar to the methods used in the evaluations found at https://github.com/huggingface/open_asr_leaderboard.)
|
518 |
|
519 |
## NVIDIA Riva: Deployment
|
520 |
|