# LLMs as for Doctors: Leveraging Medical LLMs to  Assist Doctors,
Not Replace Them

Wenya Xie, Qingying Xiao, Yu Zheng, Xidong Wang, Junying Chen, Ke Ji,
Anningzhe Gao, Xiang Wan, Feng Jiang∗, Benyou Wang
Shenzhen Research Institute of Big Data
The Chinese University of Hong Kong, Shenzhen
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Feng Jiang and Benyou Wang are the corresponding authors.

###### Abstract

The recent success of Large Language Models (LLMs) has had a significant
impact on the healthcare field, providing patients with medical advice,
diagnostic information, and more. However, due to a lack of professional
medical knowledge, patients are easily misled by generated erroneous
information from LLMs, which may result in serious medical problems. To
address this issue, we focus on tuning the LLMs to be medical assistants who
collaborate with more experienced doctors. We first conduct a two-stage survey
by inspiration-feedback to gain a broad understanding of the real needs of
doctors for medical assistants. Based on this, we construct a Chinese medical
dataset called DoctorFLAN to support the entire workflow of doctors, which
includes 92K Q&A samples from 22 tasks and 27 specialists. Moreover, we
evaluate LLMs in doctor-oriented scenarios by constructing the DoctorFLAN-test
containing 550 single-turn Q&A and DotaBench containing 74 multi-turn
conversations. The evaluation results indicate that being a medical assistant
still poses challenges for existing open-source models, but DoctorFLAN can
help them significantly. It demonstrates that the doctor-oriented dataset and
benchmarks we construct can complement existing patient-oriented work and
better promote medical LLMs research.

LLMs as for Doctors: Leveraging Medical LLMs to  Assist Doctors,
Not Replace Them

Wenya Xie, Qingying Xiao, Yu Zheng, Xidong Wang, Junying Chen, Ke Ji,
Anningzhe Gao, Xiang Wan, Feng Jiang∗, Benyou Wang††thanks: Feng Jiang and
Benyou Wang are the corresponding authors. Shenzhen Research Institute of Big
Data The Chinese University of Hong Kong, Shenzhen<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Current large language models for healthcare applicationsYunxiang et al.
(2023); Singhal et al. (2023b); Wang et al. (2023a); Chen et al. (2023a),
primarily focus on providing online consultations to patientsZhang et al.
(2023); Wang et al. (2023b); Chen et al. (2023b). These models have
demonstrated substantial potential and value, not only in reducing healthcare
costs but also in enhancing the convenience of medical consultations.

However, several challenges hinder their deployment in real-world
scenariosUmapathi et al. (2023); Wang et al. (2023c). Firstly, patients often
lack the professional knowledge necessary to verify the accuracy of the
model’s outputs. This can pose significant risks when these models provide
incorrect or misleading advice. Furthermore, issues such as model
hallucination currently impede their practical application Fan et al. (2023).

Recognizing these challenges, we advocate for repositioning LLMs in healthcare
as supportive collaborators alongside experienced doctors who can identify and
correct inaccuracies in the model’s outputs, ensuring safer implementation.
However, effectively integrating LLMs into medical practice necessitates the
development of new tasks that go beyond the online consultation LLMs for
patients. For example, doctors may need assistance with differential
diagnosis, which is not required for patients in online consultation
scenarios.

To achieve this goal, we conduct a two-stage survey using an inspiration-
feedback method to gain a comprehensive understanding of the real needs of
doctors for medical assistants. From the survey, we identify four tasks that
doctors consider most effective under LLM’s assistance, positioning them as
ideal starting points in practice.

Building upon these insights, we develop DoctorFLAN, a comprehensive Chinese
medical dataset that captures the full spectrum of the diagnostic process.
Additionally, to assess the real-world clinical capabilities of these
assistant LLM models, we designate DoctorFLAN-test for single-turn evaluations
and introduce DotaBench, a three-turn evaluation that can mimic actual medical
interactions.

After that, we benchmark the existing medical LLMs and our model DotaGPT
trained on DoctorFLAN automatically and manually. Our evaluation results
indicate that existing online consultations medical models, struggle with the
diverse and complex tasks of real-world assistant roles. Conversely, models
trained on DoctorFLAN exhibit robust performance across tasks in DoctorFLAN-
test and DotaBench. Moreover, based on our findings, some of the tasks
prioritized by doctors (e.g., Preoperative Education) in the previous survey
are promising candidates for initial deployment due to the models’ impressive
performance in these areas.

The contributions of this work are manifold. 1) We first propose LLMs for
Doctors, a paradigm shift in the application of LLMs in healthcare, and
conduct a two-stage survey to identify the real needs of clinical doctors in
their daily practice. 2) We build the DoctorFLAN, containing about 92K samples
covering 22 tasks in the entire workflow of doctors, to fill the gap for
training LLMs’ assistance to doctors with professional responses. 3) We
develop the first expert-involved benchmark to evaluate LLMs in doctor-
oriented scenarios with the single-turn DoctorFLAN-test and the multi-turn
DotaBench, complementing the existing evaluation designed for patient-oriented
scenarios.

##  2 LLMs as Doctors or for Doctors?

LLMs in healthcare applications fulfill two crucial roles: LLMs as Doctors
that autonomously provide medical consultants for patients and LLMs for
Doctors, which assist doctors in their daily work (Knoll et al., 2022; Cai et
al., 2022; Krishna et al., 2020). In this section, we first discuss the
challenges with LLMs as Doctors and advocate for a strategic shift toward
safer and more effective applications.

###  2.1 Challenges with LLMs as Doctors

Popular Medical LLMs such as ChatDoctor Yunxiang et al. (2023) and HuatuoGPT
Zhang et al. (2023) are designed to autonomously provide medical
consultations. These models are trained on extensive patient-doctor dialogues
to simulate medical advice. However, current implementations of such medical
models exhibit relatively low accuracy rates (Fan et al., 2023). Exposing them
to patients who don’t have professional medical knowledge directly may pose
significant risks as they could receive misleading responses. For example, a
patient consulting an LLM about severe stomachache and fever might receive
incorrect advice to simply rest, potentially overlooking critical conditions
like appendicitis that require immediate medical intervention.

###  2.2 LLMs for Doctors

Recognizing the discriminative ability of doctors with expertise against the
errors caused by models, we propose LLMs for Doctors, which have more
practical significance and safer implementation by positioning the LLM as an
assistant. Doctors can quickly identify and correct errors in diagnosis made
by LLMs for Doctors, thereby safeguarding patient health through critical
intervention.

However, to develop LLMs that truly support doctors, we need to address
significant gaps in existing datasets for model training, which we categorize
as LLMs as Doctors datasets. As shown in Table 1, current LLMs as Doctors
datasets often fail to capture the full scope of medical practice111See
details in Appendix A. (e.g., OMCD), do not align well with real-world
clinical scenarios (e.g., KG), or lack comprehensive responses (e.g., MCQA).

[b]  Paradigm Dataset Data Entire Clinical Comprehensive Type Workflow
Scenarios Responses Huatuo-26M OMCD ✗ ✓ ✓ HealthCareMagic100k OMCD ✗ ✓ ✓
ChatDoctor10k OMCD ✗ ✓ ✓ MedDialog OMCD ✗ ✓ ✓ LLMs as webMedQA OMCD ✗ ✓ ✓
Doctors KUAKE-QIC OMCD ✗ ✓ ✓ CMeKG KG ✓ ✗ ✗ cMedQA2 MCQA ✓ ✓ ✗ MedMCQA MCQA ✓
✓ ✗ LLMs for DoctorFLAN DAQA ✓ ✓ ✓ Doctors &DotaBench

Table 1: Comparison of existing medical training datasets. OMCD represents
Online Medical Consultant Dialogue; KG represents Knowledge Graph; MCQA
represents multiple-choice Question Answer; DAQA represents Doctor-Assistant
Question Answer. Detailed URLs are collected in Appendix B.

Given the fundamental shortcomings in these datasets, the models trained on
them inherently struggle to effectively handle the diversity of real-world
medical environments and deliver comprehensive responses222Refer to the
performance of HuatuoGPT in Table 3. HuatuoGPT is a typical LLMs as Doctors
model.. To address these issues, it is crucial to develop targeted datasets
and establish comprehensive benchmark evaluations for LLMs for Doctors,
specifically designed to reflect the everyday clinical scenarios of doctors.

Figure 1: Overview of task categories in the LLMs for Doctors dataset across
four phases: Admission, Diagnosis, Treatment, and Discharge, illustrating
typical and alternative patient care pathways in solid and dashed lines,
respectively.

###  2.3 Identifying Doctors’ Needs for LLMs

To accurately capture the medical scenarios where LLMs can be most beneficial
serving as LLMs for Doctors, we design a two-stage heuristic-feedback task
collection method to ensure the tasks involved in the medical workflow are
professional and practical. Additionally, we conduct further analysis to
determine which tasks are considered most effective for doctors when assisted
by LLMs, aiming to prioritize these tasks for early model deployment.

####  2.3.1 Task Collection

Our initial step is to collaborate with experienced doctors to identify
essential tasks in their daily workflows using a heuristic top-down method.
Subsequently, we validate the universality of these tasks through an extensive
user survey across multiple medical institutions with feedback.

Heuristic Task Definition. We organize a symposium with 16 healthcare
professionals333Details of the healthcare professionals are provided in
Appendix C. from four distinct specialties to establish task definitions based
on heuristics, ensuring the tasks we identify are closely aligned with the
practical needs of doctors. During the discussion, experts systematically
categorize the medical workflow into 4 stages (pre-diagnosis, diagnosis,
treatment, and post-treatment) with 22 tasks. It comprehensively covers all
scenarios for the doctor’s workflow, from the triage to the creation of
follow-up plans, as shown in Figure 1. Unlike typical online consultations,
which primarily address pre-treatment issues, such as initial diagnoses, and
disease prevention, our approach extensively covers a wide range of tasks that
doctors might encounter in a hospital setting, encompassing both inpatient and
outpatient tasks.

Task Confirmation with Feedback. Then, we survey doctors from over a dozen
top-tier hospitals with their feedback to validate the universality of these
tasks. In the survey444For detailed methodology and the questionnaire used in
the survey, see Appendix D., doctors are asked to rate the helpfulness of each
task with LLMs’ assistance, where a rating scale from 1 to 5 is given, with
scores of 4 and 5 indicating high and critical applicability, respectively. In
over 86% (71/82) of the valid questionnaires collected, the higher scores
achieved in all tasks demonstrate their effectiveness and importance as
recognized by the surveyed doctors, as shown in Figure 2.

Figure 2: Comparative assessment of task efficiency scores for each task
according to our survey. The Task Efficiency Score quantifies the potential of
various tasks to enhance operational efficiency in medical practice,
reflecting improvements in time management, resource use, and overall workflow
efficacy.

####  2.3.2 Analysis of Tasks Prioritized for Doctors

To identify which tasks should be prioritized for initial deployment of LLMs
for Doctors, we analyze the efficiency scores that doctors assign to each task
in the survey. It is noticed that tasks such as Triage, Case Summary,
Medication Inquiry, and Preoperative Education have particularly high scores,
which doctors consider highly beneficial due to the tasks’ repetitive nature
(e.g., case summarization, preoperative education), relatively low medical
risk (e.g., triage), and extensive information requirements (e.g., medication
inquiry).

##  3 Dataset Construction

To address these needs, we develop the DoctorFLAN and DotaBench for single-
turn Q&A and multi-turn conversation scenarios separately.

###  3.1 DoctorFLAN

To develop a dataset that comprehensively covers the entire medical workflow,
we construct DoctorFLAN based on the tasks previously defined. We collect
source data from a wide range of medical data and complete the construction of
the dataset through two steps: instruction normalization and response
polishing. Additionally, we also ensure the high quality of the data by
manually verifying some of the sampled data.

Data Collection. Our dataset primarily comprises multiple-choice questions
sourced from Medtiku555https://www.medtiku.com/, a repository of expert-level
medical examination questions, and various online resources. These questions
are selected due to their extensive coverage of professional medical knowledge
and their relevance to real-world clinical scenarios. In addition, we
supplement our dataset with encyclopedia entries666Hosted by 120 Ask,
https://m.120ask.com/ for constructing inquiry tasks such as Medicine Inquiry.
We also incorporat a subset of the PromptCBLUE dataset (Zhu et al., 2023),
selected for its diverse tasks and high quality. After collecting all the
original data, we perform deduplication and categorization using crafted
regular expressions as part of the preprocessing stage.

Refinement of Original Data. Given the lack of practicality of the original
data, we adopt a refinement process consisting of two key components:
refinement of instructions and enhancement of responses. First, We ask doctors
to manually draft the instructions for the 22 task categories to ensure that
these instructions can reflect actual use scenarios. Second, in response to
the newly refined instructions, we employ GPT-4 777GPT-4 is chosen for its
superior performance among various models we tested. to generate updated
answers. Specifically, we provide GPT-4 with reference materials, such as the
original multiple-choice question answers, to serve as a basis for generating
responses. It enables GPT-4 to produce responses that are not only accurate
but also tailored to the required context and depth, aligning with the
practical demands and professional standards expected in the medical field.

Data Verification. To ensure that the responses generated by the GPT-4 model
are factually accurate and realistic, we use a structured review process in
which a sample of 1050 responses (50 data points per item across 22 items) are
reviewed by three medical professionals, each reviewing 350 items.

The verification criteria include Correctness, where a response is considered
correct if it contains no factual errors, and Practicality, where a response
is deemed practical if it is more effective than the original answer. Our
results demonstrate a high level of correctness (100%) and practicality
(99.9%), underscoring the robustness of the DoctorFLAN. Further details (e.g.,
the graphical user interface used for verification and the detailed analysis
of failure cases) are available in Appendix F.

###  3.2 DotaBench

Extending the single-turn dataset DoctorFLAN, we introduce DotaBench to
evaluate multi-turn dialogues among medical assistants. This enhancement
reflects real-world needs where medical assistants must handle multiple rounds
of Q&A to thoroughly assess a condition, requiring continuous and consistent
responses across interactions.

Data Collection. To ensure its high professionalism, we select CMB-Clin (Wang
et al., 2023c), a multi-round question-answering dataset derived from real,
complex medical records, as the data source for DotaBench.

Refinement of Original Data. However, the samples from CMB-Clin, sourced from
medical exams, exhibit a formal written style and often lack continuity
between rounds of questioning.

To better simulate the multi-turn interactions typical between medical
assistants and doctors, we engage medical professionals to rephrase these
questions in a manner consistent with their routine inquiry processes. In
handling multiple rounds of questioning, they are instructed to maintain
logical sequence and coherence across three question items, thereby creating a
more realistic simulation of conversational turns.

We employ the original Q&A sets as a reference for evaluation. Unlike
DoctorFLAN, which directly involves LLMs in data generation, DotaBench is
crafted without LLM intervention, thereby eliminating the need for subsequent
data verification and ensuring more controlled evaluation conditions.

| DoctorFLAN | DotaBench
---|---|---
Type | Single-turn | 3-turns
Split | train | test | test
Specialist | 27 | 27 | -
Task | 22 | 22 | -
#Q/task | -* | 25 | -
#Q in total | 91,330 | 550 | 74

Table 2: The Statistics of DoctorFLAN and DotaBench Dataset. * Each task has a
different number of questions, as detailed in Appendix E.

###  3.3 Data Statistic

The statistical analysis of the DoctorFLAN and DotaBench datasets is presented
in Tabel 2. The DoctorFLAN dataset comprises 91,880 instances across 22
distinct tasks, involving 27 medical specialties in total as detailed in
Appendix E, demonstrating the comprehensive coverage of DoctorFLAN in real
clinical scenarios. In addition, we have extracted a subset of 25 instances
from each task, referred to as DoctorFLAN-test for evaluation. The DotaBench
dataset includes 74 instances of 3-turn conversations.

##  4 Experiments

###  4.1 Experimental Setup

####  4.1.1 Evaluation Models

Existing Baselines. To ensure a comprehensive understanding of how different
models perform in real-world medical assistance, We evaluate the following
Chinese medical LLMs on DoctorFLAN-test and DotaBench: BianQue-2 (Chen et al.,
2023b), DISC-MedLLM (Bao et al., 2023), HuatuoGPT (Zhang et al., 2023),
HuatuoGPT-II (Chen et al., 2023a). In addition to these domain-specific
models, we evaluate some models that perform strongly on generic tasks:
Qwen-1.8B-Chat Bai et al. (2023), Baichuan-13B-Chat, Baichuan2-7B-Chat and
Baichuan2-13B-Chat (Yang et al., 2023), Yi-6B-Chat and Yi-34B-Chat, Claude-3,
GPT-3.5, GPT-4. More details about the selected models can be found in
Appendix G.

Training Model. We also train models named DotaGPT on the DoctorFLAN training
set as a baseline. We use DoctorFLAN training set to train DotaGPT series on
two backbones: Yi-6B and Baichuan2-7B-Base, respectively, as detailed in
Appendix H.

####  4.1.2 Evaluation Method

Considering both accuracy, reliability, and cost, our evaluation methodology
incorporates both automatic and human evaluations.888For model evaluation, the
decoding hyperparameters used across all models are max_new_tokens = 1024,
top_p = 0.7, temperature = 0.5, and repetition_penalty = 1.1.

Automatic Evaluation. We use GPT-4999gpt-4-0125-preview for evaluation, which
has proven effective in the preliminary stages of development (Liu et al.,
2023b), rather than traditional metrics like F1 and BLEU, as these metrics
often misjudge similar sentences that were semantically similar. During the
automatic evaluation phase, we adopt the evaluation methodology utilized by
MT-Bench (Zheng et al., 2024), as the criteria and prompts designed for this
scoring are detailed in Appendix I. To ensure the accuracy of the GPT-4
scoring process, we supply the model with relevant contextual references and
direct it to base its scoring on these references. Furthermore, GPT-4 is
instructed to utilize a set of well-defined criteria for evaluation, thereby
enhancing the meaningfulness of the scores assigned.

Human Evaluation. To validate the accuracy of the automatic evaluation, we
conduct a human evaluation involving six medical experts assessing six models
using DoctorFLAN-test, and three medical experts evaluating three models on
DotaBench. Details of the evaluation procedures and guidelines are provided in
Appendix J.

Model | Size | DoctorFLAN-test | DotaBench
---|---|---|---
Pre-Diagnosis | During Diagnosis | Treatment | Post-Treatment | Average | Average
General Models
Qwen-1.8B-Chat | 1.8B | 5.28 | 4.56 | 3.96 | 5.44 | 4.48 | 5.77
Baichuan-13B-Chat | 13B | 6.20 | 6.51 | 6.31 | 7.55 | 6.57 | 8.17
Baichuan2-7B-Chat | 7B | 6.32 | 6.36 | 6.34 | 7.70 | 6.59 | 8.33
Baichuan2-13B-Chat | 13B | 6.76 | 6.85 | 6.94 | 7.81 | 7.04 | 8.33
Yi-6B-Chat | 6B | 7.00 | 6.83 | 6.83 | 7.66 | 6.98 | 8.27
Yi-34B-Chat | 34B | 7.36 | 7.38 | 7.95 | 8.78 | 7.80 | 8.81
Medical Models
BianQue-2 | 6B | 5.56 | 3.27 | 3.65 | 4.78 | 3.72 | 4.92
DISC-MedLLM | 13B | 5.56 | 4.23 | 3.54 | 5.14 | 4.24 | 5.81
HuatuoGPT | 7B | 5.32 | 4.24 | 3.72 | 4.92 | 4.29 | 6.78
HuatuoGPT-II | 7B | 7.60 | 7.02 | 6.69 | 7.42 | 7.03 | 8.55
DotaGPT (Yi-6B) | 6B | 8.32 | 7.62 | 7.68 | 8.44 | 7.81 $\uparrow$ 11.9% | 9.05  $\uparrow$ 9.4%
DotaGPT (Baichuan2-7B-Base) | 7B | 8.48 | 8.01 | 8.23 | 8.80 | 8.25 $\uparrow$ 25.2% | 9.00  $\uparrow$ 8.0%
GPT-3.5 | N/A | 6.40 | 6.85 | 6.26 | 6.74 | 6.64 | 8.64
Claude-3 | N/A | 7.80 | 8.38 | 8.28 | 8.76 | 8.38 | 9.46
GPT-4 | N/A | 8.00 | 8.41 | 8.28 | 9.04 | 8.42 | 9.41
Average Score | - | 6.90 | 6.59 | 6.50 | 7.43 | - | -

Table 3: Automatic Evaluation Results on DoctorFLAN-test and DotaBench. For
detailed task-by-task results on DoctorFLAN-test, see Appendix K.1; for
results by turn on DotaBench, see Appendix K.2.

###  4.2 Automatic Evaluation Results

Models | Average Score
---|---
BianQue-2 | 4.58
HuatuoGPT | 4.97
DISC-MedLLM | 5.36
Baichuan2-7B-Chat | 6.69
DotaGPT (Baichuan2-7B-Base) | 7.83
GPT-4 | 8.06

Table 4: Human Evaluation Results on DoctorFLAN-test. For detailed task-by-
task results, see Appendix J.

####  4.2.1 On the DoctorFLAN-test

The performance of general models and medical models are shown in Table 3. It
is also noteworthy that BianQue-2 and HuatuoGPT, typical models representing
LLMs as Doctors, exhibit poor performance. Analysis by phase reveals that
models generally score lower during the diagnosis and treatment phases tasks
compared to an average score of 6.9 in the pre-diagnosis phase tasks and 7.43
in the post-treatment phase tasks for all models. Interestingly, medical
models do not always exhibit stronger performance than general LLMs. For
instance, DISC-MedLLM(4.24) underperforms compared to its general counterpart
Baichuan-13B-Chat(6.57), underscoring the risks of excessive specialization,
indicating that a balance between specialization and general adaptability is
essential for broader applicability.

In contrast, our DotaGPT variants, fine-tuned on the DoctorFLan dataset,
demonstrate significant performance improvements over their respective chat
model counterparts. Specifically, the variant fine-tuned with
Baichuan-2-7B-base shows a substantial improvement of 25.2%, reaching a
performance score of 8.25. Similarly, the DotaGPT variant fine-tuned with
Yi-6B outperforms the Yi-6B-Chat by 11.9%.

The improvement on both backbones highlights the effectiveness of DoctorFLAN
and brings our models’ performance close to those of leading proprietary
models such as Claude-3 and GPT-4.

Model | Average Score
---|---
HuatuoGPT-II | 7.98
Baichuan2-7B-Chat | 8.25
DotaGPT (Baichuan2-7B-Base) | 8.54
Table 5: Human Evaluation Results on the DotaBench.

####  4.2.2 On DotaBench

As shown in Table 3, our DotaGPT variants significantly outperform models of
comparable size on DotaBench. Surprisingly, it even outperforms the
Yi-34B-Chat, which is significantly larger in model size. This strong
performance of the DotaGPT variants on DotaBench, which includes out-of-domain
(OOD) challenges, underscores their robust training on the DoctorFLAN-training
dataset. This training not only boosts their capabilities within specific
domain tasks but also effectively generalizes to out-of-domain contexts.

Figure 3: Pearson correlation between human and automatic evaluations on
DoctorFLAN-test, illustrating task-level consistency.

Model | Triage | Case | Medication | Preoperative | Average
---|---|---|---|---|---
Summary | Inquiry | Education
General Models
Qwen-1.8B-Chat | 5.28 | 5.64 | 3.56 | 5.00 | 4.87
Baichuan-13B-Chat | 6.20 | 8.52 | 6.67 | 7.67 | 7.27
Baichuan2-7B-Chat | 6.32 | 8.84 | 5.60 | 7.44 | 7.05
Baichuan2-13B-Chat | 6.76 | 8.93 | 6.56 | 8.49 | 7.69
Yi-6B-Chat | 7.00 | 8.28 | 6.96 | 8.28 | 7.63
Yi-34B-Chat | 7.36 | 8.56 | 5.88 | 9.28 | 7.77
Medical Models
BianQue-2 | 5.56 | 1.28 | 3.08 | 1.56 | 2.87
DISC-MedLLM | 5.56 | 3.20 | 3.96 | 4.24 | 4.24
HuatuoGPT | 5.32 | 4.48 | 4.60 | 4.56 | 4.74
HuatuoGPT-II | 7.60 | 7.04 | 6.88 | 7.56 | 7.27
DotaGPT (Yi-6B) | 8.32 | 6.48 | 7.48 | 8.84 | 7.78
DotaGPT (Baichuan2-7B-base) | 8.48 | 6.96 | 8.00 | 9.04 | 8.12
GPT-3.5 | 6.40 | 9.08 | 6.32 | 7.60 | 7.35
Claude-3 | 7.80 | 9.32 | 7.36 | 9.00 | 8.37
GPT-4 | 8.00 | 9.24 | 6.60 | 8.88 | 8.18

Table 6: Performance of General and Medical Models on Tasks Prioritized for
Doctors within the DoctorFLAN-test.

###  4.3 Human Evaluation Results

Overall Performance. As indicated in Table 4, DotaGPT (Baichuan2-7B-Base)
demonstrates exceptional performance in manual evaluations, substantially
surpassing both HuatuoGPT and Baichuan2-7B-Chat and closing to GPT-4 on the
DoctorFLAN-test. Furthermore, on the DotaBench, DotaGPT (Baichuan2-7B-Base)
not only outperforms its general counterpart, Baichuan2-7B-Chat but also
exceeds another SOTA model, HuatuoGPT-II, achieving the highest scores among
the three models evaluated.

Correlation with Automatic Evaluations. To verify the reliability of our
evaluation methods, we conducted a task-level correlation analysis between
human and automatic evaluations, aligned with the structured review process by
healthcare professionals who evaluated specific tasks across models. Our
analysis, covering 132 data points, reveals a Pearson correlation coefficient
of 0.84, indicating strong consistency between evaluation modes, as shown in
Figure 3.

##  5 Analysis

To expedite the deployment of LLM assistants in clinical settings, we first
analyze four key tasks identified by doctors in the previous survey (see
Section 1). We then conduct a detailed case study to further illustrate the
advantages of models trained using our DoctorFLAN dataset. Additionally, we
compare our evaluation results with other benchmarks to demonstrate the
necessity of our specialized assessments.

###  5.1 Performance on Tasks Prioritized for Doctors

We analyze model performance on tasks specifically relevant to doctors, as
detailed in Table 6. In the Case Summary task, models such as Yi-34B-Chat and
DotaGPT (Baichuan2-7B-Base) excel due to advanced training datasets and
modeling techniques, showcasing enhanced text generation capabilities. They
demonstrate high accuracy and are suitable for immediate use in clinical
settings to improve workflow efficiency. Conversely, the Medication Inquiry
task presents significant challenges to several models, requiring precise
medical knowledge. While specialized medical models like HuatuoGPT-II and
DotaGPT (Baichuan2-7B-Base) outperform general counterparts, the need for
domain-specific accuracy suggests that these models should only be deployed
once they have matured further, or when used, their outputs should be verified
with external tools to ensure reliability. This cautious approach is necessary
to prevent errors that could impact patient safety.

Model | CMB-Exam | DoctorFLAN-test | Rank Change
---|---|---|---
Yi-34B-Chat | 69.17 | 7.80 (3) | -2
Yi-6B-Chat | 65.87 | 6.98 (5) | -3
GPT-4 | 59.46 | 8.42 (1) | +2
HuatuoGPT-II | 59.00 | 7.03 (4) | +0
DotaGPT (Baichuan2-7B-Base) | 52.42 | 8.25 (2) | +3
Baichuan2-7B-Chat | 43.33 | 6.59 (6) | +0
GPT-3.5 | 38.31 | 6.64 (7) | +0
DISC-MedLLM | 32.47 | 4.24 (10) | +2
HuatuoGPT | 30.73 | 4.29 (9) | +0
Qwen-1.8B-Chat | 27.43 | 4.48 (8) | -2
BianQue-2 | 3.92 | 3.72 (11) | +0

Table 7: Comparative performance of medical LLMs on the CMB-Exam and
DoctorFLAN-test(rankings). CMB-Exam scores are cited from (Wang et al.,
2023c).

Query:
---
Based on the patient’s history, symptoms, and examination results, list
several possible diagnoses first,
then explain why these diagnoses may correspond to the patient’s condition and
how to confirm the final diagnosis through further examination or observation.
<Case description>:
The patient was a male, 11 months old, whose parents complained of a 2-week-
old white reflection in the centre of the right eye, with no redness of the
eye and no
strabismus. There was no family history of this condition, and the patient
denied any history of trauma. He had a full-term delivery with no history of
oxygen
intake and no history of birth injury.
Golden:
Based on the history and symptoms of the child, the following are a few
possible initial diagnoses:
1\. Congenital cataract - The child is young and has no other medical history
such as prematurity…
2\. Retinoblastoma (Retinoblastoma) - Although there is no such case in the
family…
…
Why these diagnoses may be consistent with the patient’s condition:
\- Congenital cataract: fundus reflections may be produced by a light source
hitting the cloudy lens..
\- Retinoblastoma: white fundus reflections (also known as cat’s eye reflex)
in children are typical of retinoblastoma.
\- Consider developmental anomalies such as PHPV: usually occurring
unilaterally…
DotaGPT (Baichuan2-7B-Base):
Based on the information you have provided in your medical history, here are
some possible preliminary diagnoses and their plausibility:
#### Possible Diagnoses:
1\. Congenital Cataract✓:
\- Rationale: A cataract usually presents as a white reflection in the pupil
area or at the location of the lens… ✓
\- FURTHER EXAMINATION: Ultrasound may be performed to assess whether the
structure and position of the lens is normal…
2\. Retinoblastoma (Retinoblastoma)✓:
\- Rationale: In infants and children, retinoblastoma is the most common
malignant tumour of the eye. Although usually associated with symptoms such as
redness
of the eye and strabismus, early lesions may also present only as white
reflections.✓
…
HuatuoGPT:
Based on the history you have provided, here are some possible diagnoses:
1\. Conjunctivitis: Conjunctivitis is a common eye condition usually caused by
a viral or bacterial infection…
2\. Cataracts: Cataracts are a common eye disease in the elderly, usually
resulting in clouding of the lens and affecting vision.
Cataracts are also a possible diagnosis if your child has no other symptoms
such as fever or cough, such as headaches or vomiting.✗
3\. Keratitis: Keratitis is an inflammatory disease … Keratitis is also a
possible diagnosis if your child has no other symptoms such as fever or
cough…✗
4\. Glaucoma: Glaucoma is a common eye disease … Glaucoma is also a possible
diagnosis if your child has no other symptoms such as fever or cough… ✗
We can perform fundus examination, visual field examination and corneal
staining to help determine the cause and confirm the diagnosis.

Table 8: The Case Study of Comparative Evaluation of Model Responses to a
Medical Differential Diagnosis Task. The green font indicates correct
information, the red font indicates incorrect information.

###  5.2 Case Study

As detailed in Table 8, we examine a Differential Diagnosis scenario to
compare HuatuoGPT, representing LLMs as Doctors, with DotaGPT, exemplifying
LLMs for Doctors. Despite HuatuoGPT’s fluent responses, it struggled with
critical patient details such as the  white reflection in the centre of the
right eye, showing a significant lack of contextual understanding necessary
for accurately completing tasks required by doctors. Conversely, DotaGPT not
only accurately diagnosed the condition but also provided detailed,
professional reasoning, demonstrating its superior ability to integrate
specific patient information and clinical context effectively.

###  5.3 Comparison with Other Benchmarks

We compare the performance of models on our DoctorFLAN-test with their results
on the CMB-Exam Wang et al. (2023c), a medical multiple-choice benchmark, as
detailed in Table 7. The comparison reveals inconsistent rankings between the
two benchmarks, highlighting that success in multiple-choice scenarios does
not necessarily predict similar outcomes in tasks that demand detailed
narrative responses and complex reasoning typical of practical medical
settings.

##  6 Related Work

Medical LLMs for Application Current medical LLMs typically fine-tune general
models using doctor-patient dialogues to enhance medical capabilities. Efforts
like Bentsao (Wang et al., 2023b), PMC-LLaMA (Wu et al., 2023), Med-PaLM2
(Singhal et al., 2023a), DISC-MedLLM (Bao et al., 2023), and HuatuoGPT (Zhang
et al., 2023) aim to deepen medical knowledge integration. However, while
these models are primarily trained to simulate AI doctors, the development of
models as doctor’s assistants remains underexplored.

Clinical Benchmarking In medical LLMs, benchmarks like PubMedQA (Jin et al.,
2019), MedQA (Jin et al., 2021), MultiMedQA Singhal et al. (2023a), MedMCQA
(Pal et al., 2022), cMedQA2 (Zhang et al., 2018), and CMB (Wang et al., 2023c)
have focused on assessing knowledge accuracy through multiple-choice
questions. Additionally, evaluations like PromptCBLUE (Zhu et al., 2023) focus
on assessing the abilities of LLMs in traditional NLP tasks, such as Named
Entity Recognition. Recent benchmarks such as HealthSearchQA (Singhal et al.,
2023a) and WebMedQA (He et al., 2019) have shifted to open-ended formats,
testing LLMs’ ability to generate coherent responses for patient inquiries.
While the above benchmarks evaluate factual accuracy and response quality,
there’s a gap in assessing LLMs for doctor-specific tasks.

Medical Instruction and Conversation Data Several datasets such as Huatuo-26M
(Li et al., 2023), HealthCareMagic100k (Yunxiang et al., 2023), ChatDoctor10k
(Yunxiang et al., 2023), and MedDialog (Zeng et al., 2020) focus on online
patient-doctor dialogues, primarily addressing outpatient inquiries without
extending to inpatient scenarios. Additionally, structured resources like KG
and MCQAs, including cMedQA2 (Zhang et al., 2018) and MedMCQA (Pal et al.,
2022), offer structured medical knowledge but fall short in providing the
depth needed for comprehensive clinical responses.

##  7 Conclusion

In this paper, we emphasize a paradigm shift from developing LLMs as Doctors
to LLMs for Doctors. We conduct a two-stage survey to accurately determine the
real-world needs of doctors and create the DoctorFLAN dataset to address the
training deficiencies of previous models. Additionally, we introduce DotaBench
as a complementary evaluation framework to evaluate the ability of popular
medical LLM as the doctor assistant. The benchmarking results show that most
existing LLM models still face difficulties in being competent as medical
assistants, but the superiority of DotaGPT demonstrates that the dataset we
construct can help improve this ability. It proves that the dataset and
benchmark we construct are effective and a beneficial supplement to current
medical LLM research.

## Limitations

The DoctorFLAN is currently only available in Chinese and may require
supplementation in other languages. Consequently, it cannot be guaranteed that
DotaGPT trained on DoctorFLAN will perform well in languages other than the
one it has been tested. However, the methodology employed to create DoctorFLAN
can be applied universally across different languages. Additionally, although
DotaGPT has demonstrated impressive performance on the benchmarks, it is
important to exercise caution when using its outputs, particularly in real-
world doctor-assistant interactions.

## Ethics Statement

Ensuring the privacy and security of data is paramount in the development of
medical applications. The datasets used in this study, primarily derived from
Medtiku, an open-source repository of medical examination questions, and
PromptCBLUEZhu et al. (2023), are both freely available for use. Additionally,
we incorporate data from an internet medical encyclopedia hosted by 120
Ask101010https://m.120ask.com/, also open to the public.

Given the potential issues with the credibility of content generated by
DotaGPT, we are committed to strictly regulating the model’s use to prevent
misuse. Our datasets, DoctorFLAN and DotaBench, will be released under terms
that uphold the highest ethical standards. This commitment ensures that while
advancing the capabilities of large language models in healthcare, we also
safeguard sensitive medical data.

## References

  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Bao et al. (2023) Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, and Zhongyu Wei. 2023.  Disc-medllm: Bridging general large language models and real-world medical consultation.  _arXiv preprint arXiv:2308.14346_.
  * Cai et al. (2022) Pengshan Cai, Fei Liu, Adarsha Bajracharya, Joe Sills, Alok Kapoor, Weisong Liu, Dan Berlowitz, David Levy, Richeek Pradhan, and Hong Yu. 2022.  Generation of patient after-visit summaries to support physicians.  In _Proceedings of the 29th International Conference on Computational Linguistics (COLING)_.
  * Chen et al. (2023a) Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, et al. 2023a.  Huatuogpt-ii, one-stage training for medical adaption of llms.  _arXiv preprint arXiv:2311.09774_.
  * Chen et al. (2023b) Yirong Chen, Zhenyu Wang, Xiaofen Xing, Zhipei Xu, Kai Fang, Junhong Wang, Sihang Li, Jieling Wu, Qi Liu, Xiangmin Xu, et al. 2023b.  Bianque: Balancing the questioning and suggestion ability of health llms with multi-turn health conversations polished by chatgpt.  _arXiv preprint arXiv:2310.15896_.
  * Fan et al. (2023) Yaxin Fan, Feng Jiang, Benyou Wang, Peifeng Li, and Haizhou Li. 2023.  Quantifying self-diagnostic atomic knowledge in chinese medical foundation model: A computational analysis.  _arXiv e-prints_ , pages arXiv–2310.
  * He et al. (2019) Junqing He, Mingming Fu, and Manshu Tu. 2019.  Applying deep matching networks to chinese medical question answering: A study and a dataset.  _BMC Medical Informatics and Decision Making_ , 19(2):52.
  * Jin et al. (2021) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  _Applied Sciences_ , 11(14):6421.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019.  Pubmedqa: A dataset for biomedical research question answering.  _arXiv preprint arXiv:1909.06146_.
  * Knoll et al. (2022) Tom Knoll, Francesco Moramarco, Alex Papadopoulos Korfiatis, Rachel Young, Claudia Ruffini, Mark Perera, Christian Perstl, Ehud Reiter, Anya Belz, and Aleksandar Savkov. 2022.  User-driven research of medical note generation software.  _arXiv preprint arXiv:2205.02549_.
  * Krishna et al. (2020) Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, and Zachary C Lipton. 2020.  Generating soap notes from doctor-patient conversations using modular summarization techniques.  _arXiv preprint arXiv:2005.01795_.
  * Li et al. (2023) Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023.  Huatuo-26m, a large-scale chinese medical qa dataset.  _arXiv preprint arXiv:2305.01526_.
  * Liu et al. (2023a) Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2023a.  Benchmarking large language models on cmexam–a comprehensive chinese medical exam dataset.  _arXiv preprint arXiv:2306.03030_.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b.  Gpteval: Nlg evaluation using gpt-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_.
  * Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022.  Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.  In _Conference on Health, Inference, and Learning_ , pages 248–260. PMLR.
  * Singhal et al. (2023a) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023a.  Large language models encode clinical knowledge.  _Nature_ , 620(7972):172–180.
  * Singhal et al. (2023b) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023b.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_.
  * Umapathi et al. (2023) Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023.  Med-halt: Medical domain hallucination test for large language models.  _arXiv preprint arXiv:2307.15343_.
  * Wang et al. (2023a) Guangyu Wang, Guoxing Yang, Zongxin Du, Longjun Fan, and Xiaohu Li. 2023a.  Clinicalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation.  _arXiv preprint arXiv:2306.09968_.
  * Wang et al. (2023b) Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023b.  Huatuo: Tuning llama model with chinese medical knowledge.  _arXiv preprint arXiv:2304.06975_.
  * Wang et al. (2023c) Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, et al. 2023c.  Cmb: A comprehensive medical benchmark in chinese.  _arXiv preprint arXiv:2308.08833_.
  * Wu et al. (2023) Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023.  Pmc-llama: Further finetuning llama on medical papers.  _arXiv preprint arXiv:2304.14454_.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_.
  * Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al. 2023.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_.
  * Yunxiang et al. (2023) Li Yunxiang, Li Zihan, Zhang Kai, Dan Ruilong, and Zhang You. 2023.  Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge.  _arXiv preprint arXiv:2303.14070_.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zeng et al. (2020) Guangtao Zeng, Wenmian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoyu Zhang, et al. 2020.  Meddialog: Large-scale medical dialogue datasets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9241–9250.
  * Zhang et al. (2023) Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. 2023.  Huatuogpt, towards taming language model to be a doctor.  _arXiv preprint arXiv:2305.15075_.
  * Zhang et al. (2021) Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, et al. 2021.  Cblue: A chinese biomedical language understanding evaluation benchmark.  _arXiv preprint arXiv:2106.08087_.
  * Zhang et al. (2018) Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018.  Multi-scale attentive interaction networks for chinese medical question answer selection.  _IEEE Access_ , 6:74061–74071.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhu et al. (2023) Wei Zhu, Xiaoling Wang, Huanran Zheng, Mosha Chen, and Buzhou Tang. 2023.  Promptcblue: A chinese prompt tuning benchmark for the medical domain.  _arXiv preprint arXiv:2310.14151_.

##  Appendix A Dataset Comparison

We compare tasks within two types of datasets, using KUAKE-QICZhang et al.
(2021)111111An evaluation dataset used by HuatuoGPTZhang et al. (2023). as a
representative example of LLMs as Doctors datasets and DoctorFLAN for LLMs for
Doctors datasets. We find while there is some overlap in tasks between these
datasets, DoctorFLAN includes 17 additional tasks not present in KUA KE-QIC,
as detailed in Figure 4. These additional tasks reflect DoctorFLAN’s broader
coverage of in-hospital medical scenarios, such as surgery planning, which are
generally absent in online consultation contexts.

Figure 4: Visual comparison of task overlap between LLMs as Doctors and LLMs
for Doctors datasets, illustrating unique and shared tasks in DoctorFLAN.

##  Appendix B Dataset and Corresponding URLs

Below are the datasets used in our research along with their corresponding
URLs:

  * •

HealthCareMagic100k: https://github.com/Kent0n-Li/ChatDoctor

  * •

ChatDoctor10k: https://github.com/Kent0n-Li/ChatDoctor

  * •

MedDialog: https://github.com/UCSD-AI4H/Medical-Dialogue-System

  * •

CMeKG: https://github.com/king-yyf/CMeKG

  * •

cMedQ&A2: https://github.com/zhangsheng93/cMedQ&A2

  * •

MedMCQ&A: https://medmcqa.github.io

##  Appendix C Details of Healthcare Professionals Involved in the Symposium

The symposium convenes healthcare professionals from four distinct medical
specialties: internal medicine, gynecology, surgery, and pediatrics. Each
specialty is represented by four experts, comprising two mid-level and two
senior specialists.

##  Appendix D Details of the Survey Process

In this section, we present the distribution process of the questionnaire, our
method for selecting valid responses, and the full text of the questionnaire
itself.

###  D.1 Design of the Survey

The questionnaire is divided into two main sections. The first section aims to
gauge doctors’ evaluations of the usefulness of 22 specific tasks to their
work efficiency, rated on a scale of 1 to 5. A score of 4 signifies that the
LLM assistance is helpful but not essential, while a score of 5 indicates its
crucial role in significantly enhancing efficiency. The second section seeks
to understand doctors’ preferences regarding the LLMs.

###  D.2 Survey Target Respondents

Doctors are sampled from over 12 Grade A tertiary care hospitals to ensure the
universality of the tasks we developed. Feedback and suggestions are gathered
from these doctors through the questionnaires.

###  D.3 Survey Collection

We collect a total of 82 responses, which are then refined to 71 valid
responses through a detailed review. The validity of the responses is
determined based on two stringent criteria: 1) the completion time is less
than one-third of the average duration observed across all submissions,
indicating a potential lack of due consideration; and 2) a marked uniformity
in responses (e.g., repetitive selection of the same response option),
suggesting insufficient engagement with the content of the questionnaire. This
dual-condition criterion ensures that respondents have engaged with the
material thoughtfully, recognizing and reflecting on the varied tasks the
survey examined. On average, respondents spend 191.82 seconds on the
questionnaires deemed valid. Importantly, the survey is designed to protect
participant anonymity by not collecting any personal information.

###  D.4 Detailed Questionnaire on Medical AI Assistant Requirements

Below is the English translation of the original questionnaire, which is
provided to the doctors in Chinese.
Dear Doctor,

In your daily medical practice, if there is a medical AI assistant ("AI
Assistant") available to provide assistance, which problems would you most
like it to solve? To better meet your needs, we have listed tasks that may
improve medical efficiency and envisioned how the AI assistant could assist
with these tasks. You are invited to assess the importance of these tasks
according to the scoring guide.

Scoring Guide:

  * •

Very Important (5 points): Indicates that the assistance of the AI assistant
will significantly improve your work efficiency (High Efficiency Score).

  * •

Important (4 points): Indicates that the AI assistant can improve your work
efficiency, but it is not crucial (Moderate Efficiency Score).

  * •

Neutral (3 points): Indicates that the AI assistant provides some assistance
to your work, but it is not essential (Neutral Efficiency Score).

  * •

Not Very Important (2 points): Indicates that the AI assistant will not be of
much help to your work (Low Efficiency Score).

  * •

Not Important At All (1 point): Indicates that the assistance of the AI
assistant in the task is completely unhelpful to your work (No Efficiency
Score).

### Pre-diagnosis

  1. 1.

Guide to Diagnosis: Based on the patient’s condition, the medical assistant
directs the patient to the correct department. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

### During Diagnosis

  1. 2.

Inquiry prompt: During the doctor-patient conversation, the medical assistant
provides prompts and suggestions for inquiries. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  2. 3.

Symptom query: You query information related to symptoms (such as causes of
the symptom) based on symptom description, and the medical assistant returns
the query results. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  3. 4.

Disease query: You query disease-related information (such as specific disease
diagnosis process and methods, complications, and prognosis) based on the
disease name, and the medical assistant returns the query results. [Single
choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  4. 5.

Initial diagnosis: The medical assistant assists the doctor in preliminary
diagnosis decisions and analysis of the condition. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  5. 6.

Summarizing the medical record from the initial consultation dialogue: The
medical assistant automatically summarizes the medical record information
based on the content of the doctor-patient conversation. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  6. 7.

Differential diagnosis: The medical assistant provides differential diagnosis
assistance based on the patient’s diagnostic information. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  7. 8.

Next steps for examination: After providing the patient’s information to the
medical assistant, it can suggest further examination and testing
recommendations for the patient. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  8. 9.

Interpretation of examination results: The medical assistant assists in
interpreting the results of examinations and tests. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  9. 10.

Confirmation of diagnosis: Based on all the patient’s diagnostic information,
the medical assistant can assist the doctor in making the final diagnosis.
[Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  10. 11.

Disease classification and grading: The medical assistant classifies and
grades the disease based on the severity of the patient’s condition. [Single
choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

### Treatment Phase

  1. 12.

Emergency care: In case of an emergency situation with the patient, the
medical assistant provides guidance and suggestions for emergency treatment
measures. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  2. 13.

Treatment plan: Based on the patient’s medical record, the medical assistant
provides possible treatment plan suggestions. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  3. 14.

Drug query: You query information related to a drug (such as
contraindications, etc.) based on the drug name, and the medical assistant
returns the query results. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  4. 15.

Medication advice: Based on the patient’s medical record, the medical
assistant provides medication advice and plans. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  5. 16.

Complications: Based on the patient’s medical record, the medical assistant
helps identify and prevent possible complications. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  6. 17.

Adjustment of the treatment plan: The medical assistant will suggest possible
adjustments to the treatment plan based on changes in the patient’s condition
recorded in the medical record. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  7. 18.

Necessity of surgery: Based on the patient’s medical record, the medical
assistant assesses whether the patient indeed needs to undergo surgery.
[Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  8. 19.

Surgery plan: For patients needing surgery, the medical assistant provides
surgery plans and related suggestions. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  9. 20.

Pre-surgery education: The medical assistant provides pre-surgery education
and preparation information for patients about to undergo surgery. [Single
choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

### Post-diagnosis

  1. 21.

Health guidance: Based on the patient’s medical record, the medical assistant
provides guidance on a healthy lifestyle. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

  2. 22.

Follow-up plan: Based on the patient’s medical record, the medical assistant
assists in arranging the patient’s subsequent follow-up plan. [Single choice]*

     * •

Not Important At All

     * •

Not Very Important

     * •

General

     * •

Important

     * •

Very Important

### Other Questions

  1. 23.

Beyond the tasks listed in the four stages above, other tasks you think
require the assistance of a medical assistant (optional). [Open-ended
question]

  2. 24.

As a doctor, which type of disease-related information are you most concerned
about? [Single choice]*

     * •

A Detailed physiological and pathological mechanisms of the disease

     * •

B Epidemiological data of the disease

     * •

C Latest treatment advancements and clinical guidelines related to the disease

     * •

D Diagnostic methods of the disease

     * •

E Other _________

  3. 25.

As a doctor, which type of drug-related information are you most concerned
about? [Single choice]*

     * •

A Mechanism of action of the drug

     * •

B Indications and usage of the drug

     * •

C Interactions and side effects of the drug

     * •

D Safety of the drug and precautions to be taken

     * •

E Other _________

  4. 26.

As a doctor, when you query information related to treatment plans, which
aspects do you usually pay most attention to? [Single choice]*

     * •

A Detailed treatment methods and means

     * •

B Cost and effectiveness of the treatment plan

     * •

C Adjustments to lifestyle and precautions during the treatment period

     * •

D Potential complications and side effects of the treatment plan

     * •

E Other _________

  5. 27.

As a doctor, you prefer the answers from the medical assistant to be [Single
choice]*

     * •

A Concise and easy to understand

     * •

B Professional and clinically accurate

     * •

C Detailed explanation

     * •

D Friendly and gentle in tone

  6. 28.

Have you ever used large language models (e.g., GPT-3.5/GPT-4/Wenxin
Yiyuan/Huatuo) in your work? [Single choice]*

     * •

Yes

     * •

No (Please skip to the end of the questionnaire and submit your answers)

  7. 29.

The most frequently used large language model is [Single choice]*

     * •

GPT-3.5

     * •

GPT-4

     * •

Wenxin Yiyan

     * •

HuatuoGPT

     * •

Other _________

  8. 30.

Please rate your satisfaction with using this model. [Single choice]*

     * •

Very dissatisfied

     * •

Dissatisfied

     * •

Neutral

     * •

Satisfied

     * •

Very satisfied

  9. 31.

Reasons for dissatisfaction [Multiple choice]*

     * $\square$

A Often unable to follow my commands or understand my questions

     * $\square$

B The large language model often cannot correctly answer my questions

     * $\square$

C The large language model cannot provide correct justifications, and I cannot
trust its conclusions

     * $\square$

D Other _________

##  Appendix E Data Statistics

Table 9 provides a detailed breakdown of the data volume corresponding to each
specific task within the DoctorFLAN. Table 10 showcases the distribution of
cases as a percentage across different specialties.

Phase | Key Task | Task Description | n | Original Data Source
---|---|---|---|---
Pre-Diagnosis | Triage | Provide department suggestions based on patient’s condition | 2535 | MCQA
During Diagnosis | Inquiry Prompts | Guide further questioning based on patient’s medical history | 1000 | MCQA
Symptom Inquiry | Retrieve encyclopedic knowledge related to the symptom | 6000 | Encyclopedia
Disease Inquiry | Retrieve encyclopedic knowledge related to the disease | 12000 | Encyclopedia
Initial Diagnosis | Identifying potential diseases based on patient history and simple tests | 13358 | MCQA
Case Summary | Compile key points from doctor-patient dialogue into a patient case | 2807 | PromptCBLUE
Differential Diagnosis | Distinguish between diseases with similar symptoms | 1000 | MCQA
Next Examinations | Decide on necessary laboratory or imaging tests for diagnosis confirmation | 6500 | MCQA
Test Results Interpretation | Explain the meaning of patient’s test results | 1000 | MCQA
Definitive Diagnosis | Confirm the final diagnosis based on all available information | 15062 | MCQA
Disease Grading | Determine the category and stage of the disease based on established medical criteria | 1000 | MCQA
Treatment | Emergency Advice | Provide guidance on handling emergencies and surgical situations | 1043 | MCQA
Treatment Plan | Develop a suitable treatment plan based on the diagnosis | 5923 | MCQA
Medication Inquiry | Retrieve encyclopedic knowledge related to the medicine | 10000 | Encyclopedia
Medication Advice | Provide specific medication recommendations for the patient | 2621 | MCQA
Complications Analysis | Identify potential and preventable complications | 1000 | MCQA
Treatment Adjustment | Modify the treatment plan based on the patient’s response to initial treatment | 1000 | MCQA
Surgery Necessity | Evaluate if surgical treatment is necessary and beneficial | 4708 | MCQA
Surgical Plan | Develop a detailed surgical plan based on the patient’s condition | 1000 | MCQA
Preoperative Education | Educate the patient about the surgical procedure and postoperative care | 769 | MCQA
Post-Treatment | Health Guidance | Provide advice on maintaining health and preventing disease recurrence | 1000 | MCQA
Follow-up Plan | Develop a plan for regular check-ups and ongoing care | 1000 | MCQA
Total |  |  | 91,800 |

Table 9: Data Volume Distribution for Various Tasks in the DoctorFLAN. MCQA represents multiple-choice Question Answer. Specialist | Percentage (%)
---|---
Gastroenterology | 7.81
Pediatrics | 7.25
Obstetrics & Gynecology | 6.74
Respiratory Medicine | 6.83
Cardiology | 6.65
Neurology | 6.34
General Surgery | 5.73
Stomatology | 5.01
Nephrology | 4.72
Hepatology | 4.06
Orthopedics | 3.70
Urology | 3.78
Hematology | 3.82
Endocrinology | 3.76
Oncology | 3.64
Emergency Medicine | 3.14
Infectious Disease | 3.13
Traditional Chinese Medicine | 2.76
Ophthalmology | 1.86
Rheumatology & Immunology | 1.75
Neurosurgery | 1.78
Dermatology | 1.73
Otorhinolaryngology(ENT) | 1.50
Vascular Surgery | 0.74
Multidisciplinary | 0.62
Spine Surgery | 0.55
Cardiothoracic Surgery | 0.31
Other | 0.30
Table 10: Percentage of cases per specialist

##  Appendix F Verification Details of DoctorFLAN Data

###  F.1 Data Verification Setting

Due to the extensive size of the dataset and budget constraints, full
verification is not feasible; instead, we sample 50 data points per task,
resulting in a total of 1050 samples for doctors to review. A senior
professional with a high-level title conducted the verification, dedicating 10
hours to the task. This approach balances thoroughness with practical
limitations, ensuring credible verification within the available resources.

To ensure the reliability of our dataset, we establish specific criteria for
data verification focusing on the following aspects:

Correctness: This criterion evaluates the accuracy of the "GPT-4 Answer". An
answer is deemed correct if it is factually accurate within the medical
context. If the "GPT-4 Answer" is entirely correct and free of medical factual
errors, please select "Yes"; otherwise, select "No".

Practicality: This criterion assesses the usefulness of the "GPT-4 Answer"
compared to the "Original Answer". If the "GPT-4 Answer" is deemed more
practical, select "Yes"; if not, select "No".

Illustrations of the graphical user interface used for this verification are
shown in Figures 5 and 6.

Figure 5: Interface for Data Verification Figure 6: Interface for Data
Verification (Translated)

###  F.2 Case Study: A Failure in Data Verification

In a detailed examination of the data verification stage, we identify an
instance where a doctor noted the lack of practicality, commenting on the
"lack of specific details.", as shown in Figure 7. Such feedback suggests that
the responses refined by GPT-4 can sometimes fall short in complex practical
medical contexts, highlighting an area for future improvement.

An Example Lacking Practicality Compared to the Original Answer Instruction:
脸部疼痛:提供症状的详细概述、可能的原因以及与之相关的疾病。包括症状的定义、典型表现、可能引起症状的常见病因，以及与该症状相关的其他健康问题。
Translation: Facial pain: Provide a detailed overview of the symptoms,
possible causes, and related diseases. Include the definition of the symptoms,
typical manifestations, common causes that may cause the symptoms, and other
health issues related to these symptoms. Reference:
associated_diseases:颞下间隙感染、慢性上颌窦炎、颞动脉炎、下颌下间隙感染、颌下间隙感染、颌骨骨髓炎、骨囊肿、颞骨岩部炎、急性颌骨骨髓炎、急性颌下腺炎
overview:脸部疼痛是指在脸部的一边或者双边疼痛，或者额头处疼痛。这种疼痛可以是钝痛，也可以是波动痛，还可能是非常强烈的刺痛。面部疼痛或痉挛，一般没有生命攸关的威胁，但痛者却十分痛苦，即使能忍住疼痛，表情麻木不自然。
cause:面部疼痛通常由局部组织感染或炎症引起，有的是隐隐的跳痛，有的是剧烈的刺痛，表现出的病症不太一样。肌肉发炎是最为人所熟悉的脸痛原因，也叫颞颌关节异常。其次三叉神经炎，带状疱疹。
也可以见于口腔炎症刺激导致的面部疼痛，也可以见于局部外伤刺激导致的疼痛情况的。 Translation: associated_diseases:
Infection in the infratemporal space, chronic maxillary sinusitis, temporal
arteritis, infection under the lower jaw, infection in the submandibular
space, osteomyelitis of the jaw, bone cyst, petrositis of the temporal bone,
acute osteomyelitis of the jaw, acute submandibular sialadenitis overview:
Facial pain refers to pain on one or both sides of the face, or at the
forehead. This pain can be dull or throbbing and can be extremely sharp.
Facial pain or spasms, generally not life-threatening, but the sufferer is
very miserable; even if the pain is bearable, the facial expression appears
unnatural. cause: Facial pain is usually caused by local tissue infection or
inflammation; some are throbbing pain, some are severe stabbing pain, and the
presented symptoms vary. Muscle inflammation is the most familiar cause of
facial pain, also known as temporomandibular joint disorder. Followed by
trigeminal neuralgia, herpes zoster. It can also be seen in facial pain caused
by oral inflammation, and pain caused by local trauma. Answer:
面部疼痛是临床上比较常见的症状，其特点和表现形式多种多样。患者可能会描述面部疼痛为钝痛、波动痛、烧灼感或者极端的刺痛。在考虑面部疼痛的病因时，需要综合考虑痛觉的分布、性质、诱发和缓解因素以及伴随症状。
根据<书本知识>，与面部疼痛相关联的疾病包括但不限于颞下间隙感染、慢性上颌窦炎、颞动脉炎、下颌下间隙感染、颌下间隙感染、颌骨 Translation:
Facial pain is a relatively common clinical symptom, characterized by various
forms and manifestations. Patients may describe facial pain as dull pain,
throbbing pain, burning sensation, or extreme stabbing pain. When considering
the causes of facial pain, it is necessary to comprehensively consider the
distribution, nature, triggering and alleviating factors, and accompanying
symptoms. According to <textbook knowledge>, diseases associated with facial
pain include, but are not limited to, infection in the infratemporal space,
chronic maxillary sinusitis, temporal arteritis, infection under the lower
jaw, infection in the submandibular space, and osteomyelitis of the jaw.
Doctor Comment: 没有原参考答案具体详细 Translation: The answer lacks specific detail
compared to the original reference answer. Figure 7: Example of a response
judged by doctors to be less practical compared to the original, illustrating
a potential challenge in ensuring the utility of LLM-generated content in
medical data.

##  Appendix G Model Details

BianQue-2 BianQue-2 is a medical language model fine-tuned on ChatGLM-6B (Zeng
et al., 2022), specifically oriented towards patient-centric conversations.

DISC-MedLLM. DISC-MedLLM is a domain-specific model crafted for conversational
healthcare, leveraging the high-quality DISC-Med-SFT dataset. It is built upon
the Baichuan-13B-Base architecture and is tailored for deep interactions
within the medical domain.

HuatuoGPT. An advanced language model specifically fine-tuned from the
Baichuan-7B architecture for Chinese medical consultations. HuatuoGPT is
trained on a comprehensive Chinese medical corpus with the goal to provide
professional-grade support in medical consultation scenarios, enhancing the
decision-making process in healthcare.

HuatuoGPT-II. Recognized as the state-of-the-art among Chinese medical
language models, HuatuoGPT-II utilizes the Baichuan2-7B-Base architecture. The
7B version, chosen for our evaluations, integrates extensive medical
knowledge.

Qwen-1.8B-Chat Part of the Qwen series, this general-purpose language model is
trained on trillions of tokens. It employs advanced techniques such as
Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback
(RLHF) to enhance its performance across diverse conversational contexts.

Baichuan-13B-Chat 121212https://huggingface.co/baichuan-inc/Baichuan-13B-Chat.
An extension of the Baichuan-13B-Base, this model integrates enhanced dialogue
capabilities in both Chinese and English.

Baichuan2-7B-Chat. Derived from the Baichuan2-7B-Base, this model shares a
common architecture with several other specialized models but is fine-tuned
for more general applications, balancing robustness and versatility.

Baichuan2-13B-Chat. An extension of the Baichuan2-13B-Base, this model is
enhanced to handle more complex and diverse tasks in conversational AI. It
offers increased computational power and improved response quality, suitable
for advanced conversational systems.

Yi-6B-Chat Yi-6B-Chat excels in both English and Chinese dialogue, aligning
closely with user preferences. While it performs well, it exhibits some
limitations when compared to more advanced models like GPT-4.

Yi-34B-Chat The larger counterpart within the Yi series, Yi-34B-Chat,
showcases enhanced performance in bilingual settings and is favored in
platforms such as AlpacaEval and Chatbot Arena. It outperforms the Yi-6B-Chat
in handling complex dialogues and user interactions but still trails behind
the capabilities of GPT-4.

Claude-3 Developed by Anthropic, Claude-3 131313We use Claude 3 Opus. is
designed to engage in natural language conversations across a broad spectrum
of tasks, emphasizing human-like text generation and comprehension.

GPT-3.5 A product of OpenAI, GPT-3.5 141414We use gpt-3.5-turbo-0125 is
designed to deliver conversational interactions, drawing on a vast corpus of
diverse textual data to generate responsive and contextually appropriate
dialogues.

GPT-4 As the successor to GPT-3.5, GPT-4 151515We use gpt-4-turbo-2024-04-09.
by OpenAI advances further in language comprehension and generation,
significantly enhancing interaction quality and precision in diverse
conversational contexts.

##  Appendix H Details of Model Training

To prevent catastrophic forgetting of general knowledge, we use general
datasets like Evol-instruct (Xu et al., 2023), ShareGPT
161616https://sharegpt.com/ in addition to DoctorFLAN-training and
supplemented it. Furthermore, to augment the model’s proficiency in medical
knowledge, we have included data from the training set of CMExam (Liu et al.,
2023a) in our training regimen. We train our model with a maximum sequence
length of 4096 to accommodate conversational data. The training parameters
included a learning rate of 5e-5 and a batch size of 4 per GPU, conducted 3
epochs on four A100 GPUs.

Task | Qwen-1.8B-Chat | Baichuan-13B-Chat | Baichuan2-7B-Chat | Baichuan2-13B-Chat | Yi-6B-Chat | Yi-34B-Chat | BianQue-2 | DISC-MedLLM | HuatuoGPT | HuatuoGPT-II | DotaGPT | DotaGPT | GPT-3.5 | Claude-3 | GPT-4
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(Baichuan2-7B-Base) | (Yi-6B)
Triage | 5.28 | 6.20 | 6.32 | 6.76 | 7.00 | 7.36 | 5.56 | 5.56 | 5.32 | 7.60 | 8.48 | 8.32 | 6.40 | 7.80 | 8.00
Inquiry Prompts | 7.16 | 8.39 | 8.08 | 8.49 | 8.92 | 8.20 | 2.00 | 6.56 | 4.48 | 8.20 | 9.04 | 9.04 | 8.12 | 8.96 | 9.12
Symptom Inquiry | 5.80 | 6.68 | 6.56 | 7.13 | 7.24 | 7.56 | 2.80 | 3.76 | 4.72 | 7.44 | 8.20 | 7.24 | 6.76 | 8.32 | 8.52
Disease Inquiry | 3.40 | 6.92 | 6.44 | 7.51 | 6.44 | 7.44 | 2.76 | 4.64 | 4.56 | 7.28 | 7.52 | 6.96 | 6.64 | 8.4 | 8.04
Initial Diagnosis | 4.44 | 5.47 | 5.60 | 6.00 | 6.44 | 6.84 | 3.96 | 4.80 | 4.72 | 7.36 | 7.80 | 8.44 | 6.08 | 8.36 | 8.16
Case Summary | 5.64 | 8.52 | 8.84 | 8.93 | 8.28 | 8.56 | 1.28 | 3.20 | 4.48 | 7.04 | 6.96 | 6.48 | 9.08 | 9.32 | 9.24
Differential Diagnosis | 4.04 | 6.41 | 6.64 | 7.09 | 6.72 | 6.64 | 4.28 | 4.52 | 4.28 | 7.00 | 8.68 | 7.56 | 6.64 | 8.44 | 8.88
Next Examinations | 5.52 | 7.45 | 7.24 | 7.88 | 8.00 | 7.88 | 5.68 | 4.60 | 4.48 | 6.84 | 8.80 | 8.32 | 7.12 | 8.92 | 8.88
Test Results Interpretation | 3.80 | 6.44 | 6.64 | 6.63 | 7.12 | 8.36 | 4.24 | 3.92 | 4.28 | 7.28 | 8.72 | 8.20 | 6.64 | 8.72 | 8.84
Definitive Diagnosis | 2.96 | 4.15 | 3.84 | 5.03 | 4.52 | 6.40 | 2.92 | 3.52 | 3.36 | 6.68 | 7.56 | 6.96 | 6.20 | 6.72 | 6.56
Disease Grading | 2.88 | 4.68 | 3.76 | 3.85 | 4.64 | 5.96 | 2.80 | 2.80 | 3.04 | 5.12 | 6.84 | 7.04 | 5.20 | 7.64 | 7.84
Emergency Advice | 3.76 | 5.81 | 7.60 | 7.52 | 7.08 | 8.68 | 3.28 | 3.28 | 3.24 | 7.40 | 8.88 | 8.56 | 6.32 | 8.84 | 8.88
Treatment Plan | 4.48 | 7.04 | 6.56 | 7.45 | 7.20 | 8.56 | 3.92 | 4.12 | 3.88 | 6.72 | 8.92 | 8.44 | 7.28 | 8.54 | 8.88
Medication Inquiry | 3.56 | 6.67 | 5.60 | 6.56 | 6.96 | 5.88 | 3.08 | 3.96 | 4.60 | 6.88 | 8.00 | 7.48 | 6.32 | 7.36 | 6.60
Medication Advice | 2.48 | 4.07 | 4.04 | 4.87 | 4.88 | 7.00 | 2.96 | 2.84 | 3.64 | 5.32 | 6.68 | 5.96 | 4.80 | 7.12 | 7.08
Complications Analysis | 4.04 | 6.61 | 6.28 | 7.11 | 7.84 | 8.28 | 4.28 | 3.20 | 3.48 | 6.80 | 8.84 | 7.68 | 6.60 | 8.72 | 8.72
Treatment Adjustment | 4.32 | 6.43 | 6.12 | 6.87 | 6.96 | 8.68 | 3.56 | 3.76 | 3.16 | 6.64 | 8.36 | 8.16 | 6.36 | 8.77 | 8.84
Surgery Necessity | 5.44 | 7.28 | 7.72 | 8.09 | 7.60 | 8.44 | 4.36 | 3.96 | 4.40 | 7.32 | 8.44 | 7.52 | 7.20 | 8.64 | 8.92
Surgical Plan | 3.60 | 6.61 | 6.80 | 7.07 | 6.08 | 8.04 | 3.76 | 3.16 | 3.36 | 6.40 | 7.72 | 7.64 | 5.20 | 8.28 | 8.28
Preoperative Education | 5.00 | 7.67 | 7.44 | 8.49 | 8.28 | 9.28 | 1.56 | 4.24 | 4.56 | 7.56 | 9.04 | 8.84 | 7.60 | 9.00 | 8.88
Health Guidance | 6.68 | 8.31 | 8.36 | 8.41 | 7.96 | 9.00 | 5.36 | 5.72 | 5.64 | 7.92 | 9.00 | 8.60 | 7.68 | 8.92 | 9.08
Follow-Up Plan | 4.20 | 6.80 | 7.04 | 7.20 | 7.36 | 8.56 | 4.20 | 4.56 | 4.20 | 6.92 | 8.60 | 8.28 | 5.80 | 8.60 | 9.00
Average | 4.48 | 6.57 | 6.59 | 7.04 | 6.98 | 7.80 | 3.57 | 4.24 | 4.29 | 7.03 | 8.25 | 7.81 | 6.64 | 8.38 | 8.42

Table 11: Detailed Auto Evaluation Results for the DoctorFLAN-test

Task | BianQue-2 | HuatuoGPT | DISC-MedLLM | Baichuan2-7B-Chat | DotaGPT (Baichuan2-7B-Base) | GPT-4
---|---|---|---|---|---|---
Triage | 4.24 | 5.04 | 5.00 | 6.28 | 7.00 | 7.76
Inquiry Prompts | 3.72 | 4.28 | 6.00 | 5.88 | 7.88 | 7.68
Symptom Inquiry | 3.60 | 7.08 | 6.00 | 8.60 | 8.68 | 8.84
Disease Inquiry | 4.84 | 6.24 | 6.16 | 7.56 | 8.24 | 8.56
Initial Diagnosis | 4.88 | 4.88 | 5.72 | 6.36 | 8.60 | 8.80
Case Summary | 1.68 | 2.80 | 3.56 | 4.32 | 3.84 | 5.76
Differential Diagnosis | 5.04 | 4.16 | 4.44 | 5.68 | 7.00 | 7.36
Next Steps for Examination | 5.44 | 5.28 | 4.84 | 7.40 | 8.60 | 8.28
Test Results Interpretation | 4.20 | 4.44 | 5.40 | 7.96 | 8.72 | 8.44
Definitive Diagnosis | 4.96 | 4.32 | 4.84 | 4.96 | 8.36 | 7.84
Disease Grading | 4.00 | 4.28 | 5.96 | 4.08 | 6.80 | 7.44
Emergency Advice | 4.04 | 4.84 | 4.64 | 8.88 | 9.36 | 8.76
Treatment Plan | 5.64 | 4.12 | 5.28 | 6.32 | 7.36 | 8.16
Medication Inquiry | 5.72 | 5.24 | 5.76 | 6.48 | 8.68 | 7.60
Complications Analysis | 5.04 | 4.24 | 5.00 | 6.20 | 7.32 | 8.20
Treatment Adjustment | 5.36 | 4.96 | 6.04 | 7.32 | 7.72 | 8.56
Surgery Necessity | 5.00 | 5.80 | 6.44 | 8.32 | 8.72 | 8.68
Surgical Plan | 4.72 | 5.08 | 4.84 | 7.80 | 8.80 | 8.40
Preoperative Education | 2.64 | 5.24 | 4.76 | 6.96 | 7.36 | 8.12
Health Guidance | 6.52 | 6.96 | 6.52 | 8.32 | 9.00 | 8.96
Follow-Up Plan | 4.88 | 4.28 | 5.36 | 6.20 | 7.16 | 7.68
Average | 4.58 | 4.97 | 5.36 | 6.69 | 7.83 | 8.06

Table 12: Detailed Human Evaluation Results for the DoctorFLAN-test

##  Appendix I Automatic Evaluation Prompt

Figure 8 and Figure 9 show the evaluation prompt we designed for the
DoctorFLAN-test and DotaBench.

Evaluation Prompt for DoctorFLAN-test System Prompt: Please act as an
impartial judge and evaluate the quality of the response provided by an AI
assistant to the user question displayed below. Requirements: Your assessment
should focus primarily on the consistency between the assistant’s answer and
the reference answer. Begin your evaluation by providing a short explanation.
Be as objective as possible. After providing your explanation, you must rate
the response on a scale of 1 to 10 by strictly following this format:
"[[Rating ]]", for example: "Rating: [[5]]". Prompt: [Question ] {question}
[The Start of Reference Answer] {reference} [The End of Reference Answer] [The
Start of Assistant’s Answer] {answer} [The End of Assistant’s Answer ] Figure
8:  Evaluation Prompt for DoctorFLAN-test. Evaluation Prompt for DotaBench
System Prompt: Please act as an impartial judge and evaluate the quality of
the response provided by an AI assistant to the user question displayed below.
Requirements: Your assessment should focus on the overall quality of the
responses based on the following criteria: Accuracy: Evaluate the correctness
and reliability of the information provided. Coherence: Assess the clarity and
logical flow of the responses. Relevance: Determine how closely each response
addresses the question asked. Thoroughness: Judge the depth and completeness
of the response in covering the topic. You will be given the assistant’s
answer and some references. The reference consists of Q&A pairs related to the
patient, which are completely accurate and can be used as a reliable source of
truth. You evaluation should focus on the assistant’s answer to the first
question. Begin your evaluation by providing a short explanation. Be as
objective as possible. After providing your explanation, you must rate the
response on a scale of 1 to 10 by strictly following this format: "[[Rating
]]", for example: "Rating: [[5]]". Prompt: <|The Start of Reference|>
{reference} <|The End of Reference|> <|The Start of Assistant A’s Conversation
with User|> ### User: {question_1} ### Assistant A: {answer_1} <|The End of
Assistant A’s Conversation with User|> Figure 9:  Evaluation Prompt for
DotaBench

##  Appendix J Details of the Human Evaluation Process

###  J.1 DoctorFLAN-test

For the DoctorFLAN-test, we engaged six healthcare professionals with varying
levels of experience to ensure a comprehensive evaluation. The team of
evaluators consisted of three mid-level professionals with 5-6 years of
experience, two associate seniors each with 12 years of experience, and one
senior professional with 26 years of experience. To maintain consistency in
assessments, each evaluator was assigned specific questions across all six
models. A total of 550 questions were distributed among the evaluators, with
four handling 92 questions each, and two assessing 91 questions each,
culminating in 3,300 evaluated responses. Each evaluator spent approximately
six hours on this task.

###  J.2 DotaBench

For the DotaBench, which consisted of 74 data points, three doctors
participated in the evaluation process. Each doctor spent three hours
reviewing the data.

###  J.3 Compensation

Compensation for the evaluators was based on their professional seniority.
Senior professionals were paid an hourly rate of 250 RMB, while mid-level
professionals received 165 RMB per hour.

###  J.4 User Interface for Human Evaluation

Figures 10, 11, 12, 13 and 14 display screenshots of the doctor evaluation
website. These figures illustrate the scoring criteria, content, and options
available for evaluation.

Figure 10: Scoring Standards: This screenshot displays the evaluation
criteria, outlining the standards and guidelines for scoring responses in the
evaluation process. Figure 11: Explanation of Website Components: This image
provides a detailed explanation of the content and structure of the evaluation
website, illustrating how various elements are organized for user interaction.
Figure 12: Scoring Process: Showcases the interface for scoring, detailing how
evaluators can assign scores based on the outlined criteria and submit their
evaluations. Figure 13: Scoring Standards(Translated): This screenshot
displays the evaluation criteria, outlining the standards and guidelines for
scoring responses in the evaluation process. Figure 14: Explanation of Website
Components(Translated): This image provides a detailed explanation of the
content and structure of the evaluation website, illustrating how various
elements are organized for user interaction.

##  Appendix K Details of Results

###  K.1 Detailed Automatic Task-by-Task Results onDoctorFLAN-test

Table 11 presents the scores of all models for each task.

###  K.2 Detailed Automatic Turn-by-Turn Results for DotaBench

Table 13 presents the results for each turn across all models evaluated in the
DotaBench benchmark.

Model | Size | DotaBench
---|---|---
Turn 1 | Turn 2 | Turn 3
General Models
Qwen-1.8B-Chat | 1.8B | 5.74 $\pm$0.15 | 5.81 $\pm$0.42 | 5.72 $\pm$0.69
Baichuan-13B-Chat | 13B | 8.07 $\pm$0.92 | 8.54 $\pm$0.13 | 7.90 $\pm$0.02
Baichuan2-7B-Chat | 7B | 8.15 $\pm$0.05 | 8.55 $\pm$0.15 | 8.29 $\pm$0.08
Baichuan2-13B-Chat | 13B | 8.06 $\pm$0.06 | 8.72 $\pm$0.03 | 8.20 $\pm$0.15
Yi-6B-Chat | 6B | 7.84 $\pm$0.05 | 8.36 $\pm$0.04 | 8.61 $\pm$0.03
Yi-34B-Chat | 34B | 9.03 $\pm$0.06 | 8.72 $\pm$0.09 | 8.68 $\pm$0.03
Medical Models
BianQue-2 | 6B | 5.60 $\pm$0.07 | 4.50 $\pm$0.17 | 4.67 $\pm$0.04
DISC-MedLLM | 13B | 5.38 $\pm$0.11 | 5.77 $\pm$0.23 | 6.29 $\pm$0.17
HuatuoGPT | 7B | 6.32$\pm$0.24 | 6.78$\pm$0.07 | 7.25$\pm$0.03
HuatuoGPT-II | 7B | 8.43 $\pm$0.06 | 8.79 $\pm$0.20 | 8.44 $\pm$0.13
DotaGPT (Yi-6B) | 6B | 9.20 $\pm$0.10 | 9.03 $\pm$0.08 | 8.91 $\pm$0.10
DotaGPT (Baichuan2-7B-Base) | 7B | 8.96 $\pm$0.07 | 9.23 $\pm$0.06 | 8.82 $\pm$0.16
GPT-3.5 | N/A | 8.27 $\pm$0.07 | 8.70 $\pm$0.01 | 8.94 $\pm$0.11
Claude-3 | N/A | 9.32 $\pm$0.08 | 9.56 $\pm$0.05 | 9.50 $\pm$0.06
GPT-4 | N/A | 9.27 $\pm$0.03 | 9.50 $\pm$0.04 | 9.47 $\pm$0.04

Table 13: Detailed DotaBench Scores by Turn

###  K.3 Detailed Human Evaluation Results for the DoctorFLAN-test

Table 12 presents the human evaluation results for DoctorFLAN-test.