|
--- |
|
license: mit |
|
language: |
|
- ar |
|
metrics: |
|
- accuracy |
|
pipeline_tag: summarization |
|
library_name: PyTorch |
|
tags: |
|
- PyTorch |
|
- Arabic |
|
- Abstractive-Summarization |
|
- 174M |
|
- Scratch |
|
- Base |
|
--- |
|
# Arab Bart |
|
Implemented the [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension |
|
](https://arxiv.org/abs/1910.13461) paper from scratch using `PyTorch` for an abstractive summarization task in Arabic. |
|
|
|
>[!IMPORTANT] |
|
> The model inferenc is not ready, i mean you can't loading it directly from the `Transformers` library. |
|
> |
|
> As soon as possible i will create an inference API, and integrate the model with the Transformers library. |
|
> |
|
|
|
## Goal |
|
|
|
Reproduce the BART model from scratch to understand its architecture in depth, using the minimum available resources. |
|
|
|
## Size |
|
|
|
The model size: `174M parameters`. |
|
|
|
## Task |
|
Abstractive Summarization in Arabic. |
|
|
|
## Data |
|
The dataset used is the [XL-Sum(Arabic Subset)](https://github.com/csebuetnlp/xl-sum?tab=readme-ov-file#:~:text=Arabic,Download) dataset. I chose this dataset because it's well-suited for our task. Additionally, it's written in pure Arabic, which makes it the best choice. The original source: [BBC Arabic](https://www.bbc.com/arabic). |
|
|
|
- Features (columns): |
|
- text: the full text (source sequences). |
|
- summary: the summary of the text (target sequences). |
|
|
|
- Size: |
|
- train: `32,473 rows`. |
|
- validation: `4689 rows`. |
|
- test: `4689 rows`. |
|
|
|
## Results |
|
|
|
| Epoch | Loss(train) | Loss(validation) | Epoch Time (hours) | Training Time (hours) | Device | |
|
|:-----:|:-----------:|:----------------:|:------------------:|:---------------------:|:--------:| |
|
| 1 | 10.03 | 9.72 | 0.23 | 1.1 | 1 x L4OS | |
|
| 2 | 9.61 | 9.44 | 0.22 | 1.1 | 1 x L4OS | |
|
| 3 | 9.36 | 9.22 | 0.22 | 1.1 | 1 x L4OS | |
|
| 4 | 9.16 | 9.05 | 0.22 | 1.1 | 1 x L4OS | |
|
| 5 | 9.01 | 8.92 | 0.22 | 1.1 | 1 x L4OS | |
|
|
|
|
|
## License |
|
|
|
This model is licensed under the `MIT` License. |
