---
license: cc
language:
- ve
- ts
- zu
- xh
- nso
- tn
library_name: transformers
tags:
- tshivenda
- low-resouce
- masked-language-model
- south africa
---
# Zabantu - Explorting Multilingual Language Model training for South African Bantu Languages

Zabantu( "Za" for South Africa, "bantu" for Bantu languages) is a collection of masked language models that have been trained from scratch using a compact dataset comprising various subsets of Bantu languages spoken in South Africa. These models are inspired by the work done on AfriBERTa, which demonstrated the effectiveness of training on XLM-R architecture using a smaller dataset. The focus of this work was to use LLMs to advance NLP applications in Tshivenda and also to serve as a benchmark for future works covering Bantu languages.

## Model Overview

This model card provides an overview of the multilingual language models developed for South African languages, with a specific focus on advancing Tshivenda natural language processing (NLP) coverage. Zabantu-XLMR refers to a fleet of models trained on different combinations of South African Bantu languages. These include:

- Zabantu-VEN: A monolingual language model trained on 73k raw sentences in Tshivenda
- Zabantu-NSO: A monolingual language model trained on 179k raw sentences in Sepedi
- Zabantu-NSO+VEN: A bilingual language model trained on 179k raw sentences in Sepedi and 73k sentences in Tshivenda
- Zabantu-SOT+VEN: A multilingual language model trained on 479k raw sentences from Sesotho, Sepedi, Setswana, and Tshivenda
- Zabantu-BANTU: A multilingual language model trained on 1.4M raw sentences from 9 South African Bantu languages

## Model Details

- **Model Name:** Zabantu-XLMR
- **Model Version:** 1.0.0
- **Model Architecture:** [XLM-RoBERTa architecture](https://arxiv.org/abs/1911.02116)
- **Model Size:** 80 - 250 million parameters
- **Language Support:** Tshivenda, Nguni languages (Zulu, Xhosa, Swati), Sotho languages (Northern Sotho, Southern Sotho, Setswana), and Xitsonga.

## Intended Use

The Zabantu models are intended to be used for various NLP tasks involving Tshivenda and related South African languages. In addition, the model can be fine-tuned on a variety of downstream tasks, such as:

- Text classification and sentiment analysis in Tshivenda and related languages.
- Named Entity Recognition (NER) for identifying entities in Tshivenda text.
- Machine Translation between Tshivenda and other South African languages.
- Cross-lingual document retrieval and question answering.

## Performance and Limitations

- **Performance:** The Zabantu models demonstrate promising performance on various NLP tasks, including news topic classification with competitive results compared to similar pre-trained cross-lingual models such as [AfriBERTa](https://huggingface.co/castorini/afriberta_base) and [AfroXLMR](https://huggingface.co/Davlan/afro-xlmr-base).

**Monolingual test F1 scores on News Topic Classification**

| Weighted F1 [%] | Afriberta-large | Afroxlmr | zabantu-nsoven | zabantu-sotven | zabantu-bantu |
|-----------------|-----------------|----------|----------------|----------------|---------------|
| nso             | 71.4            | 71.6     | 74.3           | 69             | 70.6          |
| ven             | 74.3            | 74.1     | 77             | 76             | 75.6          |

**Few-shot(50 shots) test F1 scores on News Topic Classification**

| Weighted F1 [%] | Afriberta | Afroxlmr | zabantu-nsoven | zabantu-sotven | zabantu-bantu |
|-----------------|-----------|----------|----------------|----------------|---------------|
| ven             | 60        | 62       | 66             | 69             | 55            |

- **Limitations:**

Although efforts have been made to include a wide range of South African languages, the model's coverage may still be limited for certain dialects. We note that the training set was largely dominated by Setwana and IsiXhosa.

We also acknowledge the potential to further improve the model by training it on more data, including additional domains and topics.

As with any language model, the generated output should be carefully reviewed and post-processed to ensure accuracy and cultural sensitivity.

## Training Data

The models have been trained on a large corpus of text data collected from various sources, including [SADiLaR](https://repo.sadilar.org/handle/20.500.12185/7), [Leipnets](https://wortschatz.uni-leipzig.de/en/download/Venda#ven_community_2017), [Flores](https://github.com/facebookresearch/flores), [CC-100](https://data.statmt.org/cc-100/), [Opus](https://opus.nlpl.eu/opus-100.php) and various South African government websites. The training data