model update
Browse files
README.md
CHANGED
@@ -29,61 +29,42 @@ model-index:
|
|
29 |
metrics:
|
30 |
- name: BLEU4
|
31 |
type: bleu4
|
32 |
-
value:
|
33 |
- name: ROUGE-L
|
34 |
type: rouge-l
|
35 |
-
value:
|
36 |
- name: METEOR
|
37 |
type: meteor
|
38 |
-
value:
|
39 |
- name: BERTScore
|
40 |
type: bertscore
|
41 |
-
value:
|
42 |
- name: MoverScore
|
43 |
type: moverscore
|
44 |
-
value:
|
45 |
- name: QAAlignedF1Score (BERTScore)
|
46 |
type: qa_aligned_f1_score_bertscore
|
47 |
-
value:
|
48 |
- name: QAAlignedRecall (BERTScore)
|
49 |
type: qa_aligned_recall_bertscore
|
50 |
-
value:
|
51 |
- name: QAAlignedPrecision (BERTScore)
|
52 |
type: qa_aligned_precision_bertscore
|
53 |
-
value:
|
54 |
- name: QAAlignedF1Score (MoverScore)
|
55 |
type: qa_aligned_f1_score_moverscore
|
56 |
-
value:
|
57 |
- name: QAAlignedRecall (MoverScore)
|
58 |
type: qa_aligned_recall_moverscore
|
59 |
-
value:
|
60 |
- name: QAAlignedPrecision (MoverScore)
|
61 |
type: qa_aligned_precision_moverscore
|
62 |
-
value:
|
63 |
---
|
64 |
|
65 |
# Model Card of `lmqg/bart-large-tweetqa-qag`
|
66 |
-
This model is fine-tuned version of [facebook/bart-large](https://huggingface.co/facebook/bart-large) for question generation task on the
|
67 |
-
[lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
|
68 |
-
This model is fine-tuned on the end-to-end question and answer generation.
|
69 |
|
70 |
-
Please cite our paper if you use the model ([https://arxiv.org/abs/2210.03992](https://arxiv.org/abs/2210.03992)).
|
71 |
-
|
72 |
-
```
|
73 |
-
|
74 |
-
@inproceedings{ushio-etal-2022-generative,
|
75 |
-
title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
|
76 |
-
author = "Ushio, Asahi and
|
77 |
-
Alva-Manchego, Fernando and
|
78 |
-
Camacho-Collados, Jose",
|
79 |
-
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
|
80 |
-
month = dec,
|
81 |
-
year = "2022",
|
82 |
-
address = "Abu Dhabi, U.A.E.",
|
83 |
-
publisher = "Association for Computational Linguistics",
|
84 |
-
}
|
85 |
-
|
86 |
-
```
|
87 |
|
88 |
### Overview
|
89 |
- **Language model:** [facebook/bart-large](https://huggingface.co/facebook/bart-large)
|
@@ -96,42 +77,46 @@ Please cite our paper if you use the model ([https://arxiv.org/abs/2210.03992](h
|
|
96 |
### Usage
|
97 |
- With [`lmqg`](https://github.com/asahi417/lm-question-generation#lmqg-language-model-for-question-generation-)
|
98 |
```python
|
99 |
-
|
100 |
from lmqg import TransformersQG
|
|
|
101 |
# initialize model
|
102 |
-
model = TransformersQG(language=
|
|
|
103 |
# model prediction
|
104 |
-
|
105 |
-
|
106 |
```
|
107 |
|
108 |
- With `transformers`
|
109 |
```python
|
110 |
-
|
111 |
from transformers import pipeline
|
112 |
-
# initialize model
|
113 |
-
pipe = pipeline("text2text-generation", 'lmqg/bart-large-tweetqa-qag')
|
114 |
-
# question generation
|
115 |
-
question = pipe('Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.')
|
116 |
-
|
117 |
-
```
|
118 |
-
|
119 |
-
## Evaluation Metrics
|
120 |
|
|
|
|
|
121 |
|
122 |
-
|
123 |
-
|
124 |
-
| Dataset | Type | BLEU4 | ROUGE-L | METEOR | BERTScore | MoverScore | Link |
|
125 |
-
|:--------|:-----|------:|--------:|-------:|----------:|-----------:|-----:|
|
126 |
-
| [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) | default | 0.152 | 0.35 | 0.279 | 0.913 | 0.623 | [link](https://huggingface.co/lmqg/bart-large-tweetqa-qag/raw/main/eval/metric.first.answer.paragraph.questions_answers.lmqg_qag_tweetqa.default.json) |
|
127 |
|
|
|
128 |
|
129 |
-
### Metrics (QAG)
|
130 |
|
131 |
-
|
132 |
-
|:--------|:-----|--------------------------------:|---------------------------------:|-----:|
|
133 |
-
| [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) | default | 0.925 | 0.647 | [link](https://huggingface.co/lmqg/bart-large-tweetqa-qag/raw/main/eval/metric.first.answer.paragraph.questions_answers.lmqg_qag_tweetqa.default.json) |
|
134 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
135 |
|
136 |
|
137 |
|
@@ -158,7 +143,6 @@ The full configuration can be found at [fine-tuning config file](https://hugging
|
|
158 |
|
159 |
## Citation
|
160 |
```
|
161 |
-
|
162 |
@inproceedings{ushio-etal-2022-generative,
|
163 |
title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
|
164 |
author = "Ushio, Asahi and
|
|
|
29 |
metrics:
|
30 |
- name: BLEU4
|
31 |
type: bleu4
|
32 |
+
value: 15.18
|
33 |
- name: ROUGE-L
|
34 |
type: rouge-l
|
35 |
+
value: 34.99
|
36 |
- name: METEOR
|
37 |
type: meteor
|
38 |
+
value: 27.91
|
39 |
- name: BERTScore
|
40 |
type: bertscore
|
41 |
+
value: 91.27
|
42 |
- name: MoverScore
|
43 |
type: moverscore
|
44 |
+
value: 62.25
|
45 |
- name: QAAlignedF1Score (BERTScore)
|
46 |
type: qa_aligned_f1_score_bertscore
|
47 |
+
value: 92.47
|
48 |
- name: QAAlignedRecall (BERTScore)
|
49 |
type: qa_aligned_recall_bertscore
|
50 |
+
value: 92.21
|
51 |
- name: QAAlignedPrecision (BERTScore)
|
52 |
type: qa_aligned_precision_bertscore
|
53 |
+
value: 92.74
|
54 |
- name: QAAlignedF1Score (MoverScore)
|
55 |
type: qa_aligned_f1_score_moverscore
|
56 |
+
value: 64.66
|
57 |
- name: QAAlignedRecall (MoverScore)
|
58 |
type: qa_aligned_recall_moverscore
|
59 |
+
value: 64.03
|
60 |
- name: QAAlignedPrecision (MoverScore)
|
61 |
type: qa_aligned_precision_moverscore
|
62 |
+
value: 65.39
|
63 |
---
|
64 |
|
65 |
# Model Card of `lmqg/bart-large-tweetqa-qag`
|
66 |
+
This model is fine-tuned version of [facebook/bart-large](https://huggingface.co/facebook/bart-large) for question & answer pair generation task on the [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) (dataset_name: default) via [`lmqg`](https://github.com/asahi417/lm-question-generation).
|
|
|
|
|
67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
68 |
|
69 |
### Overview
|
70 |
- **Language model:** [facebook/bart-large](https://huggingface.co/facebook/bart-large)
|
|
|
77 |
### Usage
|
78 |
- With [`lmqg`](https://github.com/asahi417/lm-question-generation#lmqg-language-model-for-question-generation-)
|
79 |
```python
|
|
|
80 |
from lmqg import TransformersQG
|
81 |
+
|
82 |
# initialize model
|
83 |
+
model = TransformersQG(language="en", model="lmqg/bart-large-tweetqa-qag")
|
84 |
+
|
85 |
# model prediction
|
86 |
+
question_answer_pairs = model.generate_qa("William Turner was an English painter who specialised in watercolour landscapes")
|
87 |
+
|
88 |
```
|
89 |
|
90 |
- With `transformers`
|
91 |
```python
|
|
|
92 |
from transformers import pipeline
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
93 |
|
94 |
+
pipe = pipeline("text2text-generation", "lmqg/bart-large-tweetqa-qag")
|
95 |
+
output = pipe("Beyonce further expanded her acting career, starring as blues singer Etta James in the 2008 musical biopic, Cadillac Records.")
|
96 |
|
97 |
+
```
|
|
|
|
|
|
|
|
|
98 |
|
99 |
+
## Evaluation
|
100 |
|
|
|
101 |
|
102 |
+
- ***Metric (Question & Answer Generation)***: [raw metric file](https://huggingface.co/lmqg/bart-large-tweetqa-qag/raw/main/eval/metric.first.answer.paragraph.questions_answers.lmqg_qag_tweetqa.default.json)
|
|
|
|
|
103 |
|
104 |
+
| | Score | Type | Dataset |
|
105 |
+
|:--------------------------------|--------:|:--------|:---------------------------------------------------------------------|
|
106 |
+
| BERTScore | 91.27 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
107 |
+
| Bleu_1 | 44.55 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
108 |
+
| Bleu_2 | 31.15 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
109 |
+
| Bleu_3 | 21.58 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
110 |
+
| Bleu_4 | 15.18 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
111 |
+
| METEOR | 27.91 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
112 |
+
| MoverScore | 62.25 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
113 |
+
| QAAlignedF1Score (BERTScore) | 92.47 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
114 |
+
| QAAlignedF1Score (MoverScore) | 64.66 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
115 |
+
| QAAlignedPrecision (BERTScore) | 92.74 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
116 |
+
| QAAlignedPrecision (MoverScore) | 65.39 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
117 |
+
| QAAlignedRecall (BERTScore) | 92.21 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
118 |
+
| QAAlignedRecall (MoverScore) | 64.03 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
119 |
+
| ROUGE_L | 34.99 | default | [lmqg/qag_tweetqa](https://huggingface.co/datasets/lmqg/qag_tweetqa) |
|
120 |
|
121 |
|
122 |
|
|
|
143 |
|
144 |
## Citation
|
145 |
```
|
|
|
146 |
@inproceedings{ushio-etal-2022-generative,
|
147 |
title = "{G}enerative {L}anguage {M}odels for {P}aragraph-{L}evel {Q}uestion {G}eneration",
|
148 |
author = "Ushio, Asahi and
|