anakin87 commited on
Commit
b94035b
·
1 Parent(s): 9339ac9

add comparison details

Browse files
Files changed (1) hide show
  1. README.md +20 -3
README.md CHANGED
@@ -119,7 +119,7 @@ It can be used for [Extractive Q&A](https://huggingface.co/tasks/question-answer
119
 
120
  [Evaluation notebook](https://github.com/anakin87/electra-italian-xxl-cased-squad-it/blob/main/evaluation.ipynb)
121
 
122
- ## Usage in Transformers
123
  Model checkpoints are available for usage in Pytorch. They can be used directly with pipelines as:
124
 
125
  ```python
@@ -129,11 +129,28 @@ qa = pipeline('question-answering', model='anakin87/electra-italian-xxl-cased-sq
129
  qa(question="Qual è il soprannome di Vasco Rossi?", context="Vasco Rossi, noto anche semplicemente come Vasco e in passato con l'appellativo Blasco (Zocca, 7 febbraio 1952), è un cantautore italiano")
130
  >>> {'score': 0.93, 'start': 80, 'end': 86, 'answer': 'Blasco'}
131
  ```
132
- ## Usage in Haystack
133
 
134
  TO DO: link notebook
135
 
136
- ## Training details
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
  [Training notebook](https://github.com/anakin87/electra-italian-xxl-cased-squad-it/blob/main/training.ipynb)
138
 
139
  **Hyperparameters**
 
119
 
120
  [Evaluation notebook](https://github.com/anakin87/electra-italian-xxl-cased-squad-it/blob/main/evaluation.ipynb)
121
 
122
+ ## Usage in Transformers 🚀
123
  Model checkpoints are available for usage in Pytorch. They can be used directly with pipelines as:
124
 
125
  ```python
 
129
  qa(question="Qual è il soprannome di Vasco Rossi?", context="Vasco Rossi, noto anche semplicemente come Vasco e in passato con l'appellativo Blasco (Zocca, 7 febbraio 1952), è un cantautore italiano")
130
  >>> {'score': 0.93, 'start': 80, 'end': 86, 'answer': 'Blasco'}
131
  ```
132
+ ## Usage in Haystack 🚀🚀🚀
133
 
134
  TO DO: link notebook
135
 
136
+ ## Comparison ⚖️
137
+
138
+ | Model | EM | F1 | Model size (pytorch) |
139
+ |------------------------------------------------------------|-------|-------|----------------------|
140
+ | it5/it5-large-question-answering | 69.10 | 78.00 | 3.13 GB |
141
+ | **anakin87/electra-italian-xxl-cased-squad-it (this one)** | 66.03 | 77.47 | 437 MB |
142
+ | it5/it5-base-question-answering | 66.30 | 76.10 | 990 MB |
143
+ | it5/mt5-base-question-answering | 66.30 | 75.70 | 2.33 GB |
144
+ | antoniocappiello/bert-base-italian-uncased-squad-it | 63.80 | 75.30 | 440 MB |
145
+ | luigisaetta/squad_it_xxl_cased_hub1 | 63.95 | 75.27 | 440 MB |
146
+ | it5/it5-efficient-small-el32-question-answering | 64.50 | 74.70 | 569 MB |
147
+ | mrm8488/bert-italian-finedtuned-squadv1-it-alfa | 62.51 | 74.16 | 440 MB |
148
+ | mrm8488/umberto-wikipedia-uncased-v1-finetuned-squadv1-it | 60.50 | 72.41 | 443 MB |
149
+ | it5/it5-small-question-answering | 61.90 | 71.60 | 308 MB |
150
+ | it5/mt5-small-question-answering | 56.00 | 66.00 | 1.2 GB |
151
+ | DrQA-it trained on SQuAD-it | 56.10 | 65.90 | ? |
152
+
153
+ ## Training details 🏋️‍
154
  [Training notebook](https://github.com/anakin87/electra-italian-xxl-cased-squad-it/blob/main/training.ipynb)
155
 
156
  **Hyperparameters**