lbourdois commited on
Commit
a2ef4bf
·
verified ·
1 Parent(s): 8b7d71c

Improve language tag

Browse files

Hi! As the model is multilingual, this is a PR to add other languages than English to the language tag to improve the referencing. Note that 29 languages are announced in the README, but only 13 are explicitly listed. I was therefore only able to add these 13 languages.

Files changed (1) hide show
  1. README.md +48 -37
README.md CHANGED
@@ -1,37 +1,48 @@
1
- ---
2
- license: mit
3
- language:
4
- - zh
5
- - en
6
- base_model:
7
- - Qwen/Qwen2.5-7B-Instruct
8
- pipeline_tag: text-generation
9
- ---
10
-
11
- # Dataset Information
12
-
13
- We introduce an omnidirectional and automatic RAG benchmark, **OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain**, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework, including:
14
-
15
- 1. a matrix-based RAG scenario evaluation system that categorizes queries into five task classes and 16 financial topics, leading to a structured assessment of diverse query scenarios;
16
- 2. a multi-dimensional evaluation data generation approach, which combines GPT-4-based automatic generation and human annotation, achieving an 87.47% acceptance ratio in human evaluations on generated instances;
17
- 3. a multi-stage evaluation system that evaluates both retrieval and generation performance, result in a comprehensive evaluation on the RAG pipeline;
18
- 4. robust evaluation metrics derived from rule-based and LLM-based ones, enhancing the reliability of assessments through manual annotations and supervised fine-tuning of an LLM evaluator.
19
-
20
- Useful Links: 📝 [Paper](https://arxiv.org/abs/2412.13018) • 🤗 [Hugging Face](https://huggingface.co/collections/RUC-NLPIR/omnieval-67629ccbadd3a715a080fd25) • 🧩 [Github](https://github.com/RUC-NLPIR/OmniEval)
21
-
22
- We have trained two models from Qwen2.5-7B by the lora strategy and human-annotation labels to implement model-based evaluation.Note that the evaluator of hallucination is different from other four.
23
-
24
- We provide the evaluator for the hallucination metric in this repo.
25
-
26
- # 🌟 Citation
27
- ```bibtex
28
- @misc{wang2024omnievalomnidirectionalautomaticrag,
29
- title={OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain},
30
- author={Shuting Wang and Jiejun Tan and Zhicheng Dou and Ji-Rong Wen},
31
- year={2024},
32
- eprint={2412.13018},
33
- archivePrefix={arXiv},
34
- primaryClass={cs.CL},
35
- url={https://arxiv.org/abs/2412.13018},
36
- }
37
- ```
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - zho
5
+ - eng
6
+ - fra
7
+ - spa
8
+ - por
9
+ - deu
10
+ - ita
11
+ - rus
12
+ - jpn
13
+ - kor
14
+ - vie
15
+ - tha
16
+ - ara
17
+ base_model:
18
+ - Qwen/Qwen2.5-7B-Instruct
19
+ pipeline_tag: text-generation
20
+ ---
21
+
22
+ # Dataset Information
23
+
24
+ We introduce an omnidirectional and automatic RAG benchmark, **OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain**, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework, including:
25
+
26
+ 1. a matrix-based RAG scenario evaluation system that categorizes queries into five task classes and 16 financial topics, leading to a structured assessment of diverse query scenarios;
27
+ 2. a multi-dimensional evaluation data generation approach, which combines GPT-4-based automatic generation and human annotation, achieving an 87.47% acceptance ratio in human evaluations on generated instances;
28
+ 3. a multi-stage evaluation system that evaluates both retrieval and generation performance, result in a comprehensive evaluation on the RAG pipeline;
29
+ 4. robust evaluation metrics derived from rule-based and LLM-based ones, enhancing the reliability of assessments through manual annotations and supervised fine-tuning of an LLM evaluator.
30
+
31
+ Useful Links: 📝 [Paper](https://arxiv.org/abs/2412.13018) • 🤗 [Hugging Face](https://huggingface.co/collections/RUC-NLPIR/omnieval-67629ccbadd3a715a080fd25) • 🧩 [Github](https://github.com/RUC-NLPIR/OmniEval)
32
+
33
+ We have trained two models from Qwen2.5-7B by the lora strategy and human-annotation labels to implement model-based evaluation.Note that the evaluator of hallucination is different from other four.
34
+
35
+ We provide the evaluator for the hallucination metric in this repo.
36
+
37
+ # 🌟 Citation
38
+ ```bibtex
39
+ @misc{wang2024omnievalomnidirectionalautomaticrag,
40
+ title={OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain},
41
+ author={Shuting Wang and Jiejun Tan and Zhicheng Dou and Ji-Rong Wen},
42
+ year={2024},
43
+ eprint={2412.13018},
44
+ archivePrefix={arXiv},
45
+ primaryClass={cs.CL},
46
+ url={https://arxiv.org/abs/2412.13018},
47
+ }
48
+ ```