metadata

language:
  - zh
license: apache-2.0
widget:
  - text: 生活的真谛是[MASK]。

Zhouwenwang-110M model (Chinese)，one model of Fengshenbang-LM.

Zhouwenwang-110M apply a new unified structure, and jointly developed by the IDEA-CCNL and Zhuiyi Technology. In the pre-training, the model considers LM (Language Model) and MLM (Mask Language Model) tasks uniformly, and adds rotational position coding, so that the model has the ability to generate and understand. Zhouwenwang-110M is the largest model for LM and MLM tasks in the Chinese field. It will continue to be optimized in the direction of model scale, knowledge integration, and supervision task assistance.

Usage

There is no structure of Zhouwenwang-110M in Transformers, you can run follow code to get structure of Zhouwenwang-110M from Fengshenbang-LM

git clone https://github.com/IDEA-CCNL/Fengshenbang-LM.git

Load Model

from model.roformer.modeling_roformer import RoFormerModel    
from model.roformer.configuration_roformer import RoFormerConfig
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("IDEA-CCNL/Zhouwenwang-110M")
config = RoFormerConfig.from_pretrained("IDEA-CCNL/Zhouwenwang-110M")
model = RoFormerModel.from_pretrained("IDEA-CCNL/Zhouwenwang-110M")

Scores on downstream chinese tasks (without any data augmentation)

Model	afqmc	tnews	iflytek	ocnli	cmnli	wsc	csl
roberta-wwm-ext-large	0.7514	0.5872	0.6152	0.777	0.814	0.8914	0.86
Zhouwenwang-1.3B	0.7463	0.6036	0.6288	0.7654	0.7741	0.8849	0. 8777

Citation

If you find the resource is useful, please cite the following website in your paper.

https://github.com/IDEA-CCNL/Fengshenbang-LM