Continuous Training and Fine-tuning for Domain-Specific Language Models in Medical Question Answering

Large language models exhibit promising general capabilities but often lack specialized knowledge for domain-specific tasks. Developing domain experts from a base model enables a range of applications without prohibitive training costs. This work demonstrates a method using <PRE_TAG>continuous training</POST_TAG> and instruction <PRE_TAG>fine-tuning</POST_TAG> to rapidly adapt Llama 2 base models to the Chinese medical domain. We first conduct <PRE_TAG>continuous training</POST_TAG> on 1B tokens from Chinese medical references to teach relevant vocabulary and knowledge. The models are then fine-tuned on 54K examples sourced from the Chinese National Medical Licensing Examination. Experiments on Chinese medical data confirm the effectiveness of this approach, producing a model comparable to GPT-3.5-turbo while using way less computational resource. The resulting domain-specific model could be useful for various Chinese medical applications. More broadly, this provides a template for domain-specific training of large language models in areas where pre-trained models lack the required expertise, such as law, science, and engineering.