capbtabboxtable[][]

# SciGLM: Training Scientific Language Models with
Self-Reflective Instruction Annotation and Tuning

Dan Zhang1,2,†, Ziniu Hu3, Sining Zhoubian1,2,†, Zhengxiao Du1,2,†,
Kaiyu Yang3, Zihan Wang1,2,†, Yisong Yue3, Yuxiao Dong1, Jie Tang1
1The Knowledge Engineering Group (KEG), Tsinghua University; 2Zhipu AI;
3California Institute of Technology

###### Abstract

Large Language Models (LLMs) have shown promise in assisting scientific
discovery. However, such applications are currently limited by LLMs’
deficiencies in understanding intricate scientific concepts, deriving symbolic
equations, and solving advanced numerical calculations. To bridge these gaps,
we introduce SciGLM, a suite of scientific language models able to conduct
college-level scientific reasoning. Central to our approach is a novel self-
reflective instruction annotation framework to address the data scarcity
challenge in the science domain. This framework leverages existing LLMs to
generate step-by-step reasoning for unlabelled scientific questions, followed
by a process of self-reflective critic-and-revise. Applying this framework, we
curated SciInstruct, a diverse and high-quality dataset encompassing
mathematics, physics, chemistry, and formal proofs. We fine-tuned the ChatGLM
family of language models with SciInstruct, enhancing their capabilities in
scientific and mathematical reasoning. Remarkably, SciGLM consistently
improves both the base model (ChatGLM3-6B-Base) and larger-scale models (12B
and 32B), without sacrificing the language understanding capabilities of the
base model. This makes SciGLM a suitable foundational model to facilitate
diverse scientific discovery tasks. For the benefit of the wider research
community, we release SciInstruct, SciGLM, alongside a self-reflective
framework and fine-tuning code at https://github.com/THUDM/SciGLM.

††footnotetext: † work done while these authors interned at Zhipu AI.

##  1 Introduction

Figure 1: Average accuracy on CEval-Sci and SciEval benchmarks of different
LLMs. Our SciGLM Models fine-tuned on ChatGLM3 are shown in bold.

Large language models (LLMs) have shown potential to assist and accelerate
scientific discovery Taylor et al. (2022); AI4Science and Quantum (2023),
including protein prediction Chen et al. (2023a), weather forecasting Bi et
al. (2023) and geoscience understanding Deng et al. (2023). Despite these
promising proof-of-concept trials, recent studies Sun et al. (2023); Wang et
al. (2023c); Rein et al. (2023) indicate a significant performance gap in LLMs
when answering basic scientific problems. According to SciBench, even advanced
models like ChatGPT and GPT-4 struggle with college-level scientific problems,
achieving only 28.52% accuracy even with the Chain of Thought (CoT) prompting
strategy. These scientific questions, such as using the Planck distribution
for energy output calculations Wang et al. (2023c), require an in-depth
understanding of complex scientific concepts, rigorous numerical computing,
and sophisticated reasoning skills. Without equipping these capabilities to
solve basic scientific questions, all ambitious visions of LLMs assisting
scientific discovery could be unreliable. This creates a substantial incentive
for developing a foundational scientific language model.

However, training LLMs to understand science is much more challenging than
many general reasoning tasks: On one hand, the knowledge and complexity
required for solving college-level scientific tasks for different subjects
(e.g., math, physics, chemistry) is larger than many other language tasks. As
we state above, solving a physics question requires finding the correct
combination of physical concepts and axioms, choice and deduction of formal
equations, numerical calculations, etc. Improving existing LLM for such a
complex question requires improving ability in all these perspectives.
Therefore, the core of building scientific LLMs is to build large-scale and
high-quality instruction datasets, which shall cover different capabilities
for solving scientific problems. We thus aggregate data from different domains
aiming to cover these abilities: 1) for understanding scientific concepts, we
gather a substantial amount of physics and chemistry questions that require
basic science knowledge; 2) for numerical calculation, we crawl and utilize
additional and more advanced mathematical calculation data; 3) for rigorous
deduction of symbolic equations, we incorporate formal theorem proofs written
in Lean. Such a mixture of datasets enables the trained model not to overfit
to a single subject but to acquire some general and fundamental skills to
solve different scientific tasks.

On the other hand, the scale of available instruction data from the internet
for scientific problems is way smaller than other tasks. As scientific content
often requires a higher level of expertise to create, and most high-quality
information is often protected by intellectual property. Most data we can
legally access only contain question-answer (QA) pairs without detailed chain-
of-thought reasoning steps. However, merely training LLMs on QA pairs will
lead to very bad results and even harm their general language capability. To
get high-quality reasoning steps as instruction (I) and create (QIA) pairs, we
propose a self-reflective instruction annotation framework that asks LLM to
autonomously annotate, critique and revise reasoning steps, with minimal human
intervention. Specifically, LLM first tries to generate both reasoning steps
and answer given question (Q) only; then, for those outputs with incorrect
answer prediction, we ask LLM itself to identify the error type, based on
which to address the error and revise the output. At last, we provide LLM with
the correct answer to finalize their annotated instruction. Such a self-
reflective annotation framework solely utilizes AI rather than humans to
collect instructions, while guaranteeing the quality and addressing the
potential mistakes of existing LLM with careful answer checking and LLM self-
reflection.

Moreover, real-world data extracted by Optical Character Recognition (OCR) and
instructions generated by the self-reflective framework can still contain
mistakes. Therefore, it is crucial to process unrefined data and obtain high-
quality data to enhance model performance. To create high-quality instructions
and filter out noisy data, we choose specific standard questions and answers
as positive samples and generate their corresponding negative samples using
LLMs. These samples can then be utilized to train a data classifier. After
consolidating the questions and answers produced by self-reflective annotation
and filtering out, we construct SciInstruct, a comprehensive dataset
containing scientific information for fine-tuning. In this work, we choose the
bilingual ChatGLM Du et al. (2022); Zeng et al. (2022) series model as the
backbones. The ChatGLM series model is an open-source dialogue pre-training
model with smooth dialogue and low deployment barriers. By fine-tuning the
ChatGLM series model on SciInstruct, we obtain the SciGLM model.

We evaluate SciGLM through three types of evaluation tasks, including
scientific test sets — CEval-Hard Huang et al. (2023), CEval-Sci Huang et al.
(2023), MMLU-Sci Hendrycks et al. (2020), and SciEval Sun et al. (2023) —
mathematical test sets — MATH Hendrycks et al. (2021), Mathematics Davies et
al. (2021), SAT-Math Zhong et al. (2023), MMLU-Math Hendrycks et al. (2020),
CEval-Math Huang et al. (2023) — and general tasks — MMLU Hendrycks et al.
(2020), CEval Huang et al. (2023), MBPP Austin et al. (2021), and GSM8K Cobbe
et al. (2021). As shown in Figure 1, compared to the base model
(ChatGLM3-6B-Base), the generalized MAmmoTH (CoT setting), and other existing
models, our model achieves performance improvement in scientific reasoning and
mathematical reasoning benchmarks, as well as general tasks. Note that SciGLM
(6B scale) obtains 64.09% accuracy on the SAT-Math dataset, which outperforms
the open-source state-of-the-art MAmmoTH-7B.

We highlight our contributions are listed below from four perspectives:

  * •

From the data perspective, we construct and present SciInstruct, a
comprehensive scientific instruction tuning dataset that includes math,
physics, chemistry problems, and formal proofs.

  * •

From the method perspective, we propose two critical sub-modules for
constructing instructions: self-reflective annotation and filtering out noisy
instructions.

  * •

From the model perspective, we utilize the ChatGLM series model as the
backbone, training SciGLM on SciInstruct, and show performance improvement on
various scientific and mathematical benchmarks, without sacrificing general
language understanding tasks.

Figure 2: The pipeline of constructing SciInstruct. On the far left is a mix
of training datasets. The purpose of the annotation is to supplement chain-of-
thought processes with reflective generation. The goal of the filter is to
train a data classifier and generate high-quality instructions.

##  2 SciInstruct Dataset

Many research (Zeng et al., 2023; Yue et al., 2023) have shown that fine-
tuning pre-trained LLMs on high-quality language instructions can gain
performance improvement by enabling the model to better utilize the knowledge
memorized through pre-training, follow more accurate and human-readable
reasoning styles and language formats. However, the main challenges of
constructing scientific instructions include the knowledge and complexity
required and the smaller scale of available data, which are introduced in
Section 1.

Therefore, in this section, we seek to tackle these obstacles by creating
SciInstruct to enhance the scientific problem-solving capabilities of LLMs.
Figure 2 illustrates our meticulous design of essential sub-modules aimed at
gathering large-scale, high-quality instructions to enhance LLMs’ scientific
task performance. These critical sub-modules encompass self-reflective
instruction annotation and noisy instruction filtering. SciInstruct comprises
a total of 172,216 meticulously verified instructions.

###  2.1 Diverse Instruction Collection

Our objective is to build a comprehensive and diverse dataset that encompasses
scientific representation and ensures versatility, referred to as SciInstruct.
To achieve this, we will focus on scientific fields and curate several top-
tier datasets that are extensively utilized and cover a wide range of
scientific disciplines and complex tasks, such as mathematics, physics,
chemistry, and formal proofs. To initiate the process, we will gather
questions from a variety of sources, including textbooks, pedagogical
materials, and problem sets. Subsequently, we will extract pertinent content
using OCR technology. Finally, we will enhance the chain of thoughts of
question and answer with the extracted content in LaTeX format.

Instruction Subject. As show on the left side of Figure 2, we initially
present the specifics of each instruction subject.

$\bullet$ Physics. This subject aims to address the challenge of processing
complex physical problems with step-by-step solutions and assessing the
ability of LLMs to comprehend and analyze physics problems. It is observed
that public training datasets lack college-level physics knowledge, such as
Fundamentals of Physics and Physical Chemistry. To address this gap, a large
set of physical questions from various subjects (e.g., dynamics, quantum
physics, electrodynamics, etc.) from typical physical textbooks, along with a
comprehensive dictionary of physics problems and solutions, has been
collected. Similar to Math, only a subset of physics questions includes
detailed solutions. Note that problems requiring interpretation of visual
content, such as diagrams, have been excluded.

$\bullet$ Chemistry. Just like Physics, obtaining professional-level Chemistry
instructions can be challenging. To address this, we utilized a similar
approach as the one used for Physics to construct the Chemistry dataset. We
gathered questions from various chemistry subjects, including college
chemistry, analysis chemistry, general chemistry, biology chemistry, inorganic
chemistry, and more. Unlike Physics and Math, Chemistry emphasizes knowledge
concepts and features fewer reasoning questions. In total, we amassed 59,243
problems for Physics and Chemistry combined.

$\bullet$ Math. College-level mathematical tasks are characterized by their
intricate logic and high complexity. This subject has undergone extensive
evaluation using LLMs. We have gathered a wide array of math problems
encompassing various mathematical subjects such as Calculus, Algebra, and
Advanced Mathematics. These problems come in different formats, including
multi-choice questions, calculations, and complex problem-solving scenarios.
Our collection process involves sourcing problems from public Q&A websites,
problem sets, and textbooks. Among the 90,436 math problems we have amassed,
some are partial, consisting solely of questions without solutions, while
others include both questions and corresponding solutions and answers in text
format.

$\bullet$ Formal Proofs (Lean). SciInstruct also includes data from formal
mathematics. In formal math, theorems and proofs are expressed in formal logic
instead of natural language. They look like computer programs. For example,
“theorem gcd_self (n : Nat) : gcd n n = n” is a theorem that says: For any
natural number n, the greatest common divisor of n and itself is n. And “cases
n <;> simp [gcd, mod_self]” is a proof of this theorem.

There are many languages for writing formal theorems and proofs; examples
include Coq Barras et al. (1997), Isabelle Nipkow et al. (2002), and Lean de
Moura et al. (2015). We chose to include data from Lean, as it is currently
the most popular in the math community. Therefore, it has abundant theorems
and proofs written by mathematicians, covering diverse topics in graduate-
level mathematics, such as analysis, geometry, and algebra. Specifically, we
process the data from LeanDojo Yang et al. (2023) and obtain 40,248
instructions for theorem proof. For each theorem and its proof, we form an
instruction like below:  This is an example of an instruction format for Lean.
Problem:  theorem gcd_self (n : Nat) : gcd n n = n
Answer:  cases n <;> simp [gcd, mod_self]

Instruction Format for Math, Physics, and Chemistry. To make our dataset
consistent, we define a unified template for problem-solving process that
matches the logical flow of human thinking for both Math, Physics, and
Chemistry. For each question, we first list the basic analysis of which
scientific concepts (or knowledge points) this question ask, and then list
step-by-step reasoning steps of detailed solution, and eventually summarize
the answer. We show the template as follow:  This is an example of an
instruction format.
Problem: $\ast\ast\ast$.
Answer:
Analysis: This question examines $\ast\ast\ast$ knowledge points.
Step 1: $\ast\ast\ast$.
Step i: $\ast\ast\ast$.
Step n: $\ast\ast\ast$.
To sum up, the answer to this question is $\ast\ast\ast$.

Summary. In total, we gathered 189,927 raw questions, the majority of which
lacked step-by-step instructions. We aim to supplement these through a self-
reflective annotation process.

Figure 3: The workflow and quantity of data for self-reflective instruction
annotation framework.

###  2.2 Self-Reflective Instruction Annotation

This stage aims to enhance instructions by incorporating the self-reflective
instruction annotation into the chain-of-thought process.

Annotate Step-by-step Reasoning Steps as Instruction. As shown in Section 2,
many questions provide only the final answer without the accompanying chain of
thought. Given the limited availability of high-quality datasets, there is
significant value in augmenting these solutions. To address questions lacking
a chain of thought, we utilize GPT-4-0613 to generate and compare chains of
thought, retaining the correct thought.

Reflective generation. GPT-4-0613 occasionally fails to consistently produce
accurate CoT processes. Drawing on prior research demonstrating the capacity
of language models for self-correction Shinn et al. (2023); Li et al. (2023a),
we refine the CoT process using a self-reflective framework. The final
reflective generation process entails three stages. Initially, we employ a
simple CoT prompt (Prompt 1) to obtain step-by-step solutions for each
question. By applying the GPT-4 labeling method, we filter out incorrect
solutions, resulting in 19,824 correct solutions. Subsequently, the incorrect
solutions from stage one, along with their respective questions, enter stage
two. Here, a reflective prompt (Prompt 2) assists GPT-4 in generating correct
answers by analyzing its errors from stage one. The newly generated solutions
are then filtered again by GPT-4, and the undesirable ones proceed to stage
three. In stage three, based on Prompt 2, we incorporate the real answer as a
direct hint in the prompt (Prompt 3) to further aid in question resolution.
The final segment of correct solutions is obtained after the generation and
filtering process. We illustrate the reflective generation in Figure 3 and
quantify the amount of data generated by each process.

###  2.3 Filtering Out Noisy Instruction

To effectively train SciInstruct on LLMs, it is essential to convert questions
(solutions) from raw text format to LaTeX format. However, when using OCR to
extract questions, the extracted content may be noisy and pose challenges. For
instance, results extracted from OCR may be incomplete and unable to be
successfully compiled. Thus, the objective of this sub-module is to filter out
noisy data and produce high-quality instructions, as depicted in the middle of
Figure 2. To accomplish this, we create positive and negative samples to train
a data classifier.

Data Generation. We randomly selected a subset of questions from our labeled
dataset of 11,553 questions as positive samples. To generate negative samples,
we prompted ChatGLM2-6B, GPT-3.5-turbo-0613, and GPT-4-0613 to provide step-
by-step answers to selected questions. We filtered inaccurate answers from
ChatGLM2-6B and labeled the solutions from GPT-3.5-turbo-0613 and GPT-4-0613
using a formatted prompt method demonstrated in Figure 7. These solutions were
merged with annotated solutions from the original dataset to train our
classifier. The composition of the merged dataset is detailed in Table 1.

Table 1: Composing data based on a limited set of labeled examples to train a
data classifier and filter out noisy instructions.

Source | # Positive Samples | # Negative Samples | Total
---|---|---|---
Original Clean Data | 11,553 | 0 | 11,553
ChatGLM2-6B | 0 | 23,106 | 23,106
GPT-3.5-Turbo-0613 | 745 | 2075 | 2,820
GPT-4-0613 | 332 | 3,898 | 4,230
Total | 12,630 | 29,079 | 41,709

Data Classifier. We improved dataset quality by training a linear data
classifier based on Table 1 using ChatGLM3-6B-Base features. Positive samples
included accurate answers from GPT-3.5-turbo-0613 and GPT-4-0613 for selected
questions, while negative samples were erroneous answers from the same models.
The classifier outputs a logit ranging from -1 to 1, with higher scores
indicating more reliable answers. This logit is used to rank and select high-
quality data from the noisy dataset. Table 2 demonstrates the effectiveness of
supervised fine-tuning on both filtered and unfiltered datasets at a 6B scale,
showing that models trained on a filtered dataset perform better on average,
emphasizing the importance of data quality over quantity for scientific tasks.

Table 2: Ablation study of filter step. We arranged the samples by score and
excluded the lowest 10%. The resulting numbers represent the average weighted
accuracy on evaluation tasks.

Type | Ave. Math | Ave. Sci | Ave. Bench
---|---|---|---
Unfiltered | 39.03 | 52.54 | 43.53
Filtered | 40.85 | 52.28 | 44.66

Error Analysis. As our classifier filter is trained on labeled datasets and
generated solutions, errors in negatively labeled solutions from ChatGLM2-6B,
GPT-3.5-turbo-0613, and GPT-4-0613 can significantly impact the classifier’s
performance. Therefore, we conduct an error analysis and categorize them into
Calculation mistakes, False reasoning, and Comprehensive mistakes. This
analysis is detailed in Figure 8, demonstrating the classifier’s capacity to
recognize these errors in the dataset.

Summary. Based on the aforementioned key sub-modules, we have constructed the
SciInstruct dataset, which comprises 172,216 instructions, as illustrated in
Table 3.

Table 3: Statistics of SciInstruct across subjects.

Subject | Math | Physics & Chemistry | Formal Proofs (Lean) | Total
---|---|---|---|---
# Number | 89,934 | 42,034 | 40,248 | 172,216

###  2.4 Instruction-Tuning with SciInstruct

Next, we present our approach to tuning instructions, referred to as our
training setup. We have selected the ChatGLM series model as our foundational
model. More specifically, we have opted for the open bilingual chat LLM,
ChatGLM3-6B, as well as its base version, ChatGLM3-6B-Base. After establishing
the base model, we have standardized all data into a chatbot-style format.
Subsequently, we have fine-tuned the foundational model using the SciInstruct,
enabling us to validate our constructed SciInstruct. Throughout the fine-
tuning process, we have conducted experiments using the Huggingface
transformers library. For the 6B, 12B, and 32B models, we have utilized a
learning rate of 3e-6, employed a linear scheduler, and trained for two
epochs. To efficiently train the model, we have leveraged DeepSpeed
Rajbhandari et al. (2020) training.

##  3 Experiments

###  3.1 Evaluation Setup

Table 4: Overview of Scientific Evaluation Tasks. (${\dagger}$: Pre-Algebra,
${\ddagger}$: Inter-Algebra, $\S$: Algebra, $\triangle$: Probability,
$\diamondsuit$: NumTheory, $\vartriangle$: Statistics, $\blacktriangle$:
Calculus, $\blacktriangledown$: Geometry, $\blacklozenge$: Physics,
$\bigstar$: Chemistry, $\clubsuit$: Biology)

Eval Dataset | # Samples | Answer Form | Subject
---|---|---|---
CEval-Hard | 152 | Multi-choice |  $\triangle,\blacktriangle,\diamondsuit,\vartriangle$, $\blacklozenge$, $\bigstar$
CEval-Sci | 210 | Multi-choice |  $\triangle,\diamondsuit,\vartriangle,\blacktriangle$, $\blacklozenge$, $\bigstar$, $\clubsuit$
MMLU-Sci | 1,855 | Multi-choice |  $\S,\triangle,\diamondsuit,\vartriangle,\blacktriangle$, $\blacklozenge$, $\bigstar$, $\clubsuit$
SciEval | 15,901 | Multi-choice |  $\blacklozenge$, $\bigstar$, $\clubsuit$
MATH | 5,000 | Open-formed | ${\dagger},{\ddagger},\S,\triangle,\diamondsuit,\blacktriangle,\blacktriangledown$
Mathematics | 1,000 | Open-formed | ${\dagger},{\ddagger},\diamondsuit,\blacktriangle$
SAT-Math | 220 | Multi-choice | ${\ddagger},\triangle,\blacktriangledown$
MMLU-Math (MI) | 974 | Multi-choice | $\S,\triangle,\diamondsuit,\blacktriangle$
CEval-Math | 90 | Multi-choice | $\vartriangle,\triangle,\diamondsuit,\blacktriangle$
MMLU-Math | 1,064 | Multi-choice | $\S,\triangle,\diamondsuit,\vartriangle,\blacktriangle$

  * 1

.

Scientific and Mathematical Tasks. In Table 4, the evaluation tasks are
summarized. Ten tasks have been chosen as out-of-domain benchmark datasets,
encompassing CEval-Hard, CEval-Sci, MMLU-Sci, SciEval Sun et al. (2023), MATH,
Mathematics, SAT-Math, MMLU-Math from MathInstruction, CEval-MATH, and MMLU-
MATH. These evaluation datasets cover a broad range of subjects, including
math, physics, and chemistry.

  * •

CEval-Hard Huang et al. (2023) comprises the eight subjects at the hard level
from CEval.

  * •

CEval-Sci Huang et al. (2023) is selected from CEval including mathematical,
physical, and chemical subjects.

  * •

MMLU-Sci Hendrycks et al. (2020) is similar to CEval-Sci and includes
mathematical, physical, and chemical subjects selected from MMLU.

  * •

SciEval Sun et al. (2023) serves as a comprehensive, multi-disciplinary
benchmark for assessing subjective question-answering capabilities.

  * •

MATH Hendrycks et al. (2021) is a standard benchmark for evaluating the
mathematical ability of LLMs.

  * •

Mathematics Davies et al. (2021) mainly includes Algebra, NumTheory, and
Calculus.

  * •

SAT-Math Zhong et al. (2023) is a math task including Intermediate Algebra,
Probability, and Calculus.

  * •

MMLU-Math (MI) Hendrycks et al. (2020) evaluation is the same as MAmmoTH Yue
et al. (2023), including Algebra, Probability, NumTheory, and Calculus.

  * •

CEval-MATH Huang et al. (2023) only includes mathematical subjects extracted
from CEval.

  * •

MMLU-Math Hendrycks et al. (2020) differs from MMLU-Math (MI) in that this
evaluation dataset replaces the Formatic subject with an Elementary subject.

General Tasks. Furthermore, we assess the generalization abilities of tasks
across various scales when fine-tuning models. These tasks include assessing
knowledge abilities (MMLU and CEval), code generation (MBPP), and elementary-
level mathematical reasoning (GSM8K).

Evaluation Metrics. The default setting for the fine-tuned SciGLM model
inherently provides the CoT solution. Hence, we conduct all experiments using
CoT settings. To thoroughly and accurately evaluate the capabilities of
different models, we employ the accuracy metric for all tasks except for code
generation, for which we use the pass@1 metric.

Table 5: Results on Scientific Reasonins tasks. Experiments indicate that
SciGLM fine-tuning consistently outperforms the base ChatGLM2 and ChatGLM3
across various parameter scales. Avg. Sci represents the weighted average
score of all scientific tasks within the same evaluation category, while Avg.
Bench signifies the weighted average score on both scientific and mathematical
tasks. Bold spotlights the best performance among both closed-source models
and open-source models. Underline denotes the second-best result among open-
source models. Results marked as † are benchmarked by ourselves.

Model |  | CEval-Hard | CEval-Sci | MMLU-Sci | SciEval | Avg. Sci | Avg.{Sci+Math}
---|---|---|---|---|---|---|---
(API, parameter details unknown)
GPT-4 |  | 54.96 | 60.55 | - | 73.93 | - | -
GPT-3.5-turbo |  | 41.37 | 46.83 | - | 66.97 | - | -
Claude-v1.3 |  | 39.14 | 44.64 | - | 63.45 | - | -
(# parameter = 6B$\sim$7B)
LLaMA-2-7B |  | 28.29† | 30.00† | 30.41 | 28.37 | 28.29 | 24.12
Galactica-6.7B |  | 11.84† | 11.44† | 30.68 | 50.87 | 26.21 | 20.00
ChatGLM2-6B |  | 29.61† | 45.71† | 37.09† | 53.02† | 45.27 | 34.39
ChatGLM2-6B-Base |  | 32.90† | 40.95† | 38.06† | 50.38† | 43.13 | 33.46
ChatGLM3-6B |  | 36.84† | 38.57† |  41.78† | 56.56† | 45.64 | 33.63
ChatGLM3-6B-Base |  |  45.40† |  54.29† | 40.16† | 61.69† | 50.38 | 41.89
SciGLM (ChatGLM3-6B-Base) |  | 50.66 | 59.05 | 42.32 | 61.49 | 53.38 | 46.79
(# parameter = 12B$\sim$13B)
LLaMA-2-13B |  | 19.74† | 19.05† | 35.85 | 36.96 | 27.90 | 27.51
Vicuna-13B |  | - | - | 32.13 | 53.93 | - | -
ChatGLM2-12B |  | 21.05† | 49.52† | 41.19† | 55.86† | 41.91 | 37.21
ChatGLM2-12B-Base |  | 35.54† | 47.14† | 46.79† | 59.61† | 47.27 | 39.63
SciGLM (ChatGLM2-12B-Base) |  | 47.37 | 55.71 | 46.95 | 56.35 | 51.60 | 43.57
(# parameter = 30B$\sim$32B)
Galactica-30B |  | - | - | 35.53 | 54.96 | - | -
ChatGLM3-32B-Base |  | 53.95† | 64.29† | 50.30† | 67.38† | 58.98 | 51.88
SciGLM (ChatGLM3-32B-Base) |  | 57.89 | 67.14 | 53.53 | 68.56 | 61.78 | 54.53

Table 6: Results on Mathmatical Reasoning. SciGLM outperforms all base models
across various parameter scales. The Avg. Math represents the weighted average
score of all mathematical tasks within the evaluation category. The use of
Bold highlights the best result among closed-source and open-source models,
while Underline denotes the second-best result among open-source models.
Results marked as † are benchmarked by ourselves.

Model |  | MATH | Mathematics | SAT-Math | MMLU-Math (MI) | CEval-Math | MMLU-Math | Avg. Math
---|---|---|---|---|---|---|---|---
(API, parameter details unknown) |
GPT-4 |  | 42.50 | - | 95.00 | - | 53.91 | - | -
GPT-3.5-turbo |  | 34.10 | - | 70.90 | - | 40.81 | - | -
Claude-v1.3 |  | - | - | - | - | 37.66 | - | -
(# parameter = 6B$\sim$7B) |
LLaMA-2-7B |  | 2.50 | 6.00 | 26.80 | 29.80 | 30.00† | 29.04 | 20.69
Galactica-6.7B |  | 2.20 | 4.60 | 17.50 | 28.00 | 14.48† | 28.36 | 15.86
WizardMath-7B |  | 10.70 | 9.30 | 25.40 | 31.10 | - | - | -
MAmmoTH (CoT)-7B |  | 10.40 | 9.20 | 32.70 | 39.90 | - | - | -
MetaMath-7B |  | 19.80 | - | - | - | - | - | -
MAmmoTH & MetaMath-7B |  | 24.10 | 18.30 | 41.40 | 44.40 | - | - | -
ChatGLM2-6B |  | 6.90† | 14.30† | 39.55† | 38.91† | 36.67† | 37.41 | 28.95
ChatGLM2-6B-Base |  | 7.84† | 17.10† | 34.55† | 40.45† | 32.22† | 39.57† | 28.62
ChatGLM3-6B |  | 9.92† | 11.60† | 39.09† | 41.07† | 21.11† | 42.95† | 27.62
ChatGLM3-6B-Base |  |  25.38† |  29.30† |  55.91† | 31.83† |  40.00† | 34.96† | 36.23
SciGLM (ChatGLM3-6B-Base) |  | 24.42 | 29.70 | 64.09 | 49.90 | 47.78 | 38.53 | 42.40
(# parameter = 12B$\sim$13B) |
LLaMA-2-13B |  | 3.90 | 11.50 | 32.70 | 34.40 | 18.89† | 33.83 | 22.54
Vicuna-13B |  | 5.80 | 10.00 | 34.00 | 34.10 | - | 29.79 | -
WizardMath-13B |  | 14.00 | 14.10 | 24.50 | 32.10 | - | - | -
MAmmoTH (CoT)-13B |  | 10.20 | 9.60 | 40.90 | 36.60 | - | - | -
MAmmoTH & MetaMath-13B |  | 26.18 | 20.60 | 48.18 | 48.25 | - | - | -
ChatGLM2-12B |  | 9.82† | 18.60† | 52.27† | 42.71† | 37.78† | 43.33† | 34.08
ChatGLM2-12B-Base |  | 11.92† | 21.50† | 43.64† | 46.00† | 37.78† | 46.43† | 34.54
SciGLM (ChatGLM2-12B-Base) |  | 12.92 | 20.60 | 59.09 | 48.77 | 38.89 | 49.06 | 38.22
(# parameter = 30B$\sim$32B) |
Galactica-30B |  | 12.70 | 11.80 | 37.70 | 37.90 | - | - | -
ChatGLM3-32B-Base |  | 31.60† | 38.60† | 67.73† | 50.10† | 51.11† | 43.70† | 47.14
SciGLM (ChatGLM3-32B-Base) |  | 32.30 | 33.60 | 69.55 | 59.75 | 55.56 | 47.46 | 49.70

Table 7: Results on general language understanding tasks. Fine-Tuning does not
sacrifice most language tasks, and only drop a little bit on code generation
task.

Model | MMLU | CEval | MBPP | GSM8K | Avg.
---|---|---|---|---|---
GPT-4 | 86.40 | 68.70 | 83.00 | 87.10 | 81.30
ChatGLM3-6B-Base | 61.32 | 67.09 | 55.80 | 72.93 | 64.28
SciGLM (6B-Base) | 61.38 | 67.16 | 45.00 | 72.93 | 61.62
ChatGLM3-32B-Base | 69.05 | 79.94 | 58.20 | 81.80 | 72.24
SciGLM (32B-Base) | 70.08 | 79.64 | 56.60 | 82.64 | 71.80

Figure 4: SciGLM consistently improving the ChatGLM3-Base through instructing-
tuning on SciInstruct for different model size and different tasks.

Baselines. We consider the following baselines:

  * •

GPT-4 OpenAI (2023): We consider the closed-source model GPT-4, which uses CoT
prompting.

  * •

GLM Du et al. (2022); Zeng et al. (2022) and LLaMA Touvron et al. (2023) Base:
We select ChatGLM2-6B (Base), ChatGLM3-6B (Base), ChatGLM2-12B (Base), and
ChatGLM3-32B-Base as the base models. In addition, we consider LLaMA-2-7B and
LLaMA-2-13B as baselines.

  * •

Continue Pre-training: We include Galactica Taylor et al. (2022) although we
note that its continued pre-training does not necessarily include college-
level scientific reasoning.

  * •

Dataset-specific Tuning: For the dataset-specific tuning, we consider
WizardMath Luo et al. (2023), MAmmoTH-CoT Yue et al. (2023), and MetaMath Yu
et al. (2023), which adapt to the MATH dataset.

We employ a standardized evaluation framework to compare GLM and LLaMA Base
baselines fairly. To gauge performance in the MATH task, we utilize zero-shot
and 8-shot configurations to determine the highest accuracy. Additionally, for
Mathematics, SAT-Math, MMLU, and CEval, we employ a chat module for
assessment. When dealing with multiple-choice questions, we formulate the
prompt as "Therefore, among A through D, the answer is".

###  3.2 Main Results

Table 5, Table 6, and Table 7 present experimental findings for scientific and
mathematical benchmarks, as well as general tasks. The results show that
training on the provided SciInstruct benefits reasoning tasks, improving
performance in scientific reasoning (i.e., CEval-Sci, SciEval, MMLU-Sci) and
transferring to mathematical tasks. Such performance improvement is consistent
with different scale of based model parameters, across 6B, 12B and 32B. In
addition, SciGLM’s performance improvement on scientific reasoning does not
sacrifice its general language understanding capabilities. As shown in Figure
4, on standard task like MMLU and elementary math GSM8K, SciGLM even achieves
slight performance improvement. On code generation like MBPP, the performance
is slightly lower but is overall consistent.

###  3.3 Ablation Studies

####  3.3.1 Impact of Subject

Figure 5: Influence of different domain in fine-tuning data towards target
tasks. Weight is calculated by (Acc. (SciGLM) - Acc. (Exclude subjects) / Acc.
(SciGLM)) under the leave-one-out subject setting.

We conduct experiments to study the effect of subjects in SciGLM. We finetuned
the ChatGLM3-6B-Base model on multiple settings without anyone subject. Since
SciGLM is trained on SciInstruct composed of multiple subjects, we are
interested in investigating the impact of each subject. As shown in Figure 5,
the results from our subject ablation on ten evaluation benchmarks indicate
that multiple subjects can obtain the best performance and are very efficient
in the subject collection. However, we noted a slight decrease in performance
when leaving out a subject, implying that potential gap.

Figure 6: Performance Improment over different scale of instruction data. The
x-axis represents the proportion of instruction per domain, and y-axis
represents the average accuracy on corresponding datasets.

####  3.3.2 Impact of Number of Questions

One of the main factors that impacts the performance of the model is the
number and quality of the training dataset, This raises the question of how
the dataset size impacts the model’s performance. To investigate this, we
evaluate the effect of the number of input instructions. As shown in Figure 6,
we show the ablation results. Overall, the accuracy improves as the number of
instructions increases, which is consistent with the scaling law found in Yuan
et al. (2023).

##  4 Related Work

###  4.1 Scientific Reasoning Datasets

Mathematical Datasets. In the past, the emphasis in mathematical reasoning
tasks was primarily on fundamental arithmetic reasoning datasets Koncel-
Kedziorski et al. (2015); Roy and Roth (2016); Patel et al. (2021), addressing
basic mathematical problems such as AddSub Hosseini et al. (2014). Later on,
to address realistic math word problems, some more difficult datasets are
proposed Ling et al. (2017); Amini et al. (2019); Cobbe et al. (2021);
Azerbayev et al. (2023). To construct a more difficult and diversified
dataset, NumGLUE (Mishra et al., 2022) and LiLA (Mishra et al., 2022) have
been introduced to enhance current research. However, their focus remains
primarily on elementary, middle school, and high school math problems. Given
the rapid advancement of LLMs, to assess their capacity and constraints in
addressing more intricate math problems, MMLU Hendrycks et al. (2020) with
college-level math problems has been proposed. To address the more challenging
college-level math problems, PoT Chen et al. (2023b); Wang et al. (2023c) is
proposed. Our proposed SciInstruct covers more complex and more diversified
math problems at the college level.

Scientific Datasets. To address the scientific ability of LLMs, SciBench Wang
et al. (2023c) proposes various solutions like calling existing tools (python
and Wolfram). SciEval Sun et al. (2023) presents a multi-level evaluation
benchmark for scientific research. GPQA Rein et al. (2023) builds a graduate-
level Q&A benchmark to evaluate more difficult physics and chemistry
questions.

###  4.2 General Reasoning with LLMs

Assisted by Chain-of-Thought prompting Wei et al. (2022); Kojima et al.
(2022); Wang et al. (2022c) and Tree-or-Though Yao et al. (2023), LLMs have
brought decent reasoning performance improvements. Especially, on the
challenging BIG-Bench tasks, the CoT technique has already outperformed human
ability Suzgun et al. (2022). Later on, to address reasoning tasks, a few
studies Nye et al. (2021); Wang et al. (2022a, b); Li et al. (2023b); Wang et
al. (2023b); Yu et al. (2023); Drozdov et al. (2022); Zhou et al. (2022)
propose various methods to leverage LLMs and retain step-by-step processes.
With the rapid development of LLM Agents, several works Yao et al. (2022);
Zeng et al. (2023) propose to utilize external tools to enhance the reasoning
capabilities of LLMs. For example, ReAct Yao et al. (2022) attempts to call
existing tools like web search engines to improve the reasoning skills of
LLMs. Indeed, leveraging the programs as thought processes is a natural way.
Several recent studies have utilized programs as thought processes, such as
PoT Chen et al. (2022), to improve the reasoning abilities of LLMs. To enhance
LLMs’ reasoning performance in solving mathematical or scientific problems
with PoT, several methods have been suggested, including Self-critic Gou et
al. (2023), Self-eval Xie et al. (2021), and Plan-and-solve Wang et al.
(2023a). For example, self-critic Gou et al. (2023) and self-eval Xie et al.
(2021) propose to adopt self-evaluation to improve the generated program’s
robustness. Different from these two methods, plan-and-solve Wang et al.
(2023a) adopts more detailed planning instructions to generate a high-level
reasoning plan for LLMs. Indeed, these methods have demonstrated that they can
obtain great capabilities over PoT.

###  4.3 Scientific Reasoning with LLMs

Recently, there have been advances to bridge the gaps in reasoning difficulty
and evaluation subjects from three perspectives. (1) High-level evaluation
like SciBench Wang et al. (2023c) and GPQA Rein et al. (2023), which evaluates
the scientific reasoning capabilities of LLMs at the college level and even
graduate level. In addition, SciEval Sun et al. (2023) provides a multi-level
LLMs evaluation benchmark to address data leakage problems and subjective
question/answer evaluation ability issues. (2) Continued pre-training like
Galactica Taylor et al. (2022) and MINERVA Lewkowycz et al. (2022), which
continue to train their respective base LLMs on multiple web texts including
science-related or math-related corpus. In particular, Galactica trains 106
billion tokens. This continued pre-training approach explores the potential of
LLMs for science and contributes to the open-source models for the scientific
community, but it is computationally expensive. Mistral Jiang et al. (2023)
leverages grouped-query attention and sliding window attention to improve
performance and fine-tune public instruction datasets. (3) Dataset-specific
fine-tuning like RFT Yuan et al. (2023), WizardMath Luo et al. (2023), MAmmoTH
Yue et al. (2023), and MetaMath Yu et al. (2023), which constructs certain
datasets including GSM8K and MATH, conducts supervised fine-tuning of LLMs and
evaluates the popular benchmarks GSM8K and MATH. Compared to base LLMs, RFT,
WizardMath, and MetaMath implement the accuracy increment on GSM8K by 11.5%
and improve the accuracy on MATH by 8.7% at a 6B $\sim$ 7B scale. However,
these approaches degrade the performance of out-of-domain tasks like MMLU-Math
Hendrycks et al. (2020). To improve the generalization to wide math reasoning
tasks, MAmmoTH not only improves the in-domain (IND) performance like GSM8K
but also generalizes to broader mathematical tasks by building a spectrum of
math instruction tuning datasets including in-domain and out-of-domain
datasets. However, this series of methods mainly focuses on mathematical
reasoning tasks.

###  4.4 Instruction Tuning in LLMs

To align language models with human preferences and effective objectives, some
works Zeng et al. (2023); Xu et al. (2023) design instruction tuning. The
instruction tuning aims to mine the potential capabilities by aligning with
and responding to human preference. Early on, instruction tuning focuses on
improving the instruction-following abilities of LLMs for general purposes.
Represented by FLAN Wei et al. (2021) and T0 Sanh et al. (2021), they aim to
understand the generalization abilities of LLMs for instruction tuning. Later,
to comprehend the efficacy and performance of enlarging the instructional
tuning datasets on models, FLAN-v2 Chung et al. (2022); Longpre et al. (2023)
has been proposed to validate this goal. However, these methods build training
instruction tuning datasets in a human-annotated manner. Recently, various
studies Wang et al. (2022d); Xu et al. (2023); Peng et al. (2023); Zeng et al.
(2023); Zhou et al. (2023); Wang et al. (2023d) have started to construct
synthetic instruction following datasets distilled from some LLMs like
GPT-3.5-turbo and GPT-4. Like these works, Platypus Lee et al. (2023)
constructs a small-scale instruction for the following dataset by utilizing a
domain-specialized dataset, aiming to enhance the reasoning capabilities of
LLMs.

##  5 Conclusion

In this work, we aim to improve the scientific reasoning capabilities of LLMs
not limited to mathematical tasks. By collecting diverse instructions, self-
reflective annotation instructions, and filtering out instructions on
mathematics, physics, and chemistry subjects, we present a high-level and
high-quality dataset, referred to as SciInstruct, to enhance and enrich the
scientific knowledge of LLMs. Using SciInstruct, we train SciGLM by fine-
tuning ChatGLM series models, significantly outperformed on scientific and
mathematical benchmarks over the base models and even some state-of-the-art
LLMs that has order of magnitude more parameters. Our research underscores the
significance of diverse training data as well as LLM self-annotation and
correctness for enhancing general reasoning capability, even for hard domain
like science. In the future, we aim to enhance the model’s capabilities by
synthesizing data purely from unstructured data (like academic papers and
knowledge base), and explore more advanced self-improving techniques to
bootstrap LLM reasoning.

## Limitation

In this section, we discuss more limitations during the research of
SciInstruct and SciGLM.

Scale of dataset and model. Even though our training dataset has expanded to
approximately 170k, improving model performance still necessitates access to
an even larger dataset. Presently, our model’s experimental outcomes are
carried out at 6B, 12B, and 32B parameters, leading to a relatively better
performance. However, it’s important to note that these performances are
constrained by the model’s scale. Moving forward, it’s worth exploring the
potential benefits of leveraging larger-scale datasets and models to further
improve performance.

Using data classifier to enhance the generation of models. In line with what
was discussed in Section 2.3, we employ a data classifier to boost the
instruction quality, yielding improved performance as shown in Table 2.
However, we anticipate that the data classifier, also referred to as the
reward model, could provide even greater benefits. One particular avenue of
improvement could be bootstrapping data to improve the ability of the base
model.

## Acknowledgements

We would like to thank the data annotators for their help and support. All GPU
compute and API costs are provided by Zhipu AI.

## References

  * AI4Science and Quantum (2023) Microsoft Research AI4Science and Microsoft Azure Quantum. 2023.  The impact of large language models on scientific discovery: a preliminary study using gpt-4.  _arXiv preprint arXiv:2311.07361_.
  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019.  Mathqa: Towards interpretable math word problem solving with operation-based formalisms.  _arXiv preprint arXiv:1905.13319_.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_.
  * Azerbayev et al. (2023) Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023.  Llemma: An open language model for mathematics.
  * Barras et al. (1997) Bruno Barras, Samuel Boutin, Cristina Cornes, Judicaël Courant, Jean-Christophe Filliatre, Eduardo Gimenez, Hugo Herbelin, Gerard Huet, Cesar Munoz, Chetan Murthy, et al. 1997.  _The Coq proof assistant reference manual: Version 6.1_.  Ph.D. thesis, Inria.
  * Bi et al. (2023) Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. 2023\.  Accurate medium-range global weather forecasting with 3d neural networks.  _Nature_ , 619:1–6.
  * Chen et al. (2023a) Bo Chen, Xingyi Cheng, Yangli ao Geng, Shen Li, Xin Zeng, Boyan Wang, Jing Gong, Chiming Liu, Aohan Zeng, Yuxiao Dong, Jie Tang, and Le Song. 2023a.  xtrimopglm: Unified 100b-scale pre-trained transformer for deciphering the language of protein.  _bioRxiv_.
  * Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_.
  * Chen et al. (2023b) Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023b.  Theoremqa: A theorem-driven question answering dataset.  _arXiv preprint arXiv:2305.12524_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Davies et al. (2021) Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. 2021.  Advancing mathematics by guiding human intuition with ai.  _Nature_ , 600(7887):70–74.
  * de Moura et al. (2015) Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris Van Doorn, and Jakob von Raumer. 2015.  The Lean theorem prover (system description).
  * Deng et al. (2023) Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Le Zhou, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. 2023.  Learning a foundation language model for geoscience knowledge understanding and utilization.  _arXiv preprint arXiv:2306.05064_.
  * Drozdov et al. (2022) Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022.  Compositional semantic parsing with large language models.  _arXiv preprint arXiv:2209.15003_.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335.
  * Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023.  Critic: Large language models can self-correct with tool-interactive critiquing.  _arXiv preprint arXiv:2305.11738_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_.
  * Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014\.  Learning to solve arithmetic word problems with verb categorization.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 523–533.
  * Huang et al. (2023) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, et al. 2023.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _arXiv preprint arXiv:2305.08322_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015.  Parsing algebraic word problems into equations.  _Transactions of the Association for Computational Linguistics_ , 3:585–597.
  * Lee et al. (2023) Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. 2023.  Platypus: Quick, cheap, and powerful refinement of llms.  _arXiv preprint arXiv:2308.07317_.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models.  _Advances in Neural Information Processing Systems_ , 35:3843–3857.
  * Li et al. (2023a) Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Heng Huang, Jiuxiang Gu, and Tianyi Zhou. 2023a.  Reflection-tuning: Data recycling improves llm instruction-tuning.  _arXiv preprint arXiv:2310.11716_.
  * Li et al. (2023b) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023b.  Making language models better reasoners with step-aware verifier.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5315–5333.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_.
  * Luo et al. (2023) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _arXiv preprint arXiv:2308.09583_.
  * Nipkow et al. (2002) Tobias Nipkow, Markus Wenzel, and Lawrence C Paulson. 2002.  _Isabelle/HOL: a proof assistant for higher-order logic_.
  * Nye et al. (2021) Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021.  Show your work: Scratchpads for intermediate computation with language models.  _arXiv preprint arXiv:2112.00114_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.  Are nlp models really able to solve simple math word problems?  _arXiv preprint arXiv:2103.07191_.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.  Instruction tuning with gpt-4.  _arXiv preprint arXiv:2304.03277_.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE.
  * Rein et al. (2023) David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2023.  Gpqa: A graduate-level google-proof q&a benchmark.  _arXiv preprint arXiv:2311.12022_.
  * Roy and Roth (2016) Subhro Roy and Dan Roth. 2016.  Solving general arithmetic word problems.  _arXiv preprint arXiv:1608.01413_.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_.
  * Sun et al. (2023) Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2023.  Scieval: A multi-level large language model evaluation benchmark for scientific research.  _arXiv preprint arXiv:2308.13149_.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022\.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022\.  Galactica: A large language model for science.  _arXiv preprint arXiv:2211.09085_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2022a) Boshi Wang, Xiang Deng, and Huan Sun. 2022a.  Iteratively prompt pre-trained language models for chain of thought.  _arXiv preprint arXiv:2203.08383_.
  * Wang et al. (2022b) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022b.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  _arXiv preprint arXiv:2212.10001_.
  * Wang et al. (2023a) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  _arXiv preprint arXiv:2305.04091_.
  * Wang et al. (2023b) Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhifang Sui. 2023b.  Making large language models better reasoners with alignment.  _arXiv preprint arXiv:2309.02144_.
  * Wang et al. (2023c) Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2023c.  Scibench: Evaluating college-level scientific problem-solving abilities of large language models.  _arXiv preprint arXiv:2307.10635_.
  * Wang et al. (2022c) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022c.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wang et al. (2023d) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023d.  How far can camels go? exploring the state of instruction tuning on open resources.  _arXiv preprint arXiv:2306.04751_.
  * Wang et al. (2022d) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022d.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021.  An explanation of in-context learning as implicit bayesian inference.  _arXiv preprint arXiv:2111.02080_.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_.
  * Yang et al. (2023) Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. 2023.  LeanDojo: Theorem proving with retrieval-augmented language models.  In _Neural Information Processing Systems (NeurIPS)_.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint arXiv:2309.12284_.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023.  Scaling relationship on learning mathematical reasoning with large language models.  _arXiv preprint arXiv:2308.01825_.
  * Yue et al. (2023) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_.
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023.  Agenttuning: Enabling generalized agent abilities for llms.  _arXiv preprint arXiv:2310.12823_.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zhong et al. (2023) Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023.  Agieval: A human-centric benchmark for evaluating foundation models.  _arXiv preprint arXiv:2304.06364_.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_.

##  6 Appendix

###  6.1 GPT Labeling

As shown in Figure 7, we present the process of GPT-4 labeling.

Figure 7: We present the process of GPT-4 labeling. Given a prompt, question,
solution, and real answer, GPT-4 can give the final label output.

###  6.2 Detailed Error Analysis

As shown in Figure 8, we give detailed error examples, which are summarized in
Section 2.3.

Figure 8: Error analysis of three types of self-reflection for noisy
instruction filtering.

###  6.3 Detailed Experimental Results

Domain-wise Results in Figure 4. We present the detailed subject results in
Table 8 and Table 9 for Figure 4.

Table 8: Detailed results of physical subjects for Figure 1. Bold denotes the
best result in the closed-source models and open-source models.

model | Ave. | CEval | MMLU | SciEval
---|---|---|---|---
Avg. |  | College
---
Physics
| High
---
School
Physics
| Middle
---
School
Physics
Avg. |  | College
---
Physics
| Conceptual
---
Physics
| Middle
---
School
Physics
Physics.
ChatGLM3-6B-Base | 54.32 | 63.16 | 42.11 | 63.16 | 84.21 | 45.59 | 39.22 | 55.32 | 34.44 | 46.95
SciGLM (ChatGLM3-6B-Base) | 56.59 | 66.67 | 47.37 | 63.16 | 89.47 | 46.52 | 34.31 | 55.32 | 41.06 | 47.56
ChatGLM3-32B-Base | 67.85 | 78.95 | 57.89 | 78.95 | 100.00 | 56.76 | 42.16 | 62.55 | 57.62 | 51.22
SciGLM (ChatGLM3-32B-Base) | 69.65 | 80.70 | 63.16 | 78.95 | 100.00 | 58.61 | 41.18 | 66.81 | 57.62 | 56.71

Table 9: Detailed results of chemical subjects for Figure 1. Bold denotes the
best result in the closed-source models and open-source models.

model | Ave. | CEval | MMLU | SciEval
---|---|---|---|---
Avg. |  | College
---
Chemistry
| High
---
School
Chemistry
| Middle
---
School
Chemistry
Avg. |  | College
---
Chemistry
| High
---
School
Chemistry
Chemistry.
ChatGLM3-6B-Base | 58.25 | 66.67 | 45.83 | 63.16 | 95.00 | 49.83 | 45.00 | 52.22 | 55.76
SciGLM (ChatGLM3-6B-Base) | 58.55 | 68.25 | 41.67 | 73.68 | 95.00 | 48.84 | 47.00 | 49.75 | 57.27
ChatGLM3-32B-Base | 66.44 | 69.84 | 41.67 | 78.95 | 95.00 | 63.04 | 51.00 | 68.97 | 60.15
SciGLM (ChatGLM3-32B-Base) | 69.05 | 71.42 | 45.83 | 78.95 | 95.00 | 66.67 | 57.00 | 71.43 | 61.40

###  6.4 SciInstruct Examples

We present SciInstrcut examples for Math(Figure 10), Physics (Figure 11),
Chemistry (Figure 12), and Formal Proofs (Figure 13).

###  6.5 SciInstruct Examples

As shown in Figure 9, we present a concrete example and model output from
SciEval, including the ChaGLM3-6B-Base and SciGLM.

Problem How does electric field affect capacitance? A. Electric field
decreases capacitance. B. Electric field doesn’t affect capacitance. C.
Electric field changes capacitance based on geometry. D. Electric field
increases capacitance. Correct Answer
B Answer produced by ChatGLM3-6B-Base
D Answer produced by SciGLM (ChatGLM3-6B-Base)
$\checkmark$ B

Figure 9: An example problem from SciEval is accurately solved by SciGLM
(ChaGLM3-6B-Base). For this problem, the category is physics; the topic is
Electric Forces and Fields; the ability is Scientific Calculation; the type is
multiple-choice; the task name is SocraticQA.

Problem
Using the method of undetermined coefficients to answer the below question.
$y^{\prime\prime}-2y^{\prime}+5y=\sin x,y(0)=1,y^{\prime}(0)=1$ Answer
Analysis: The key points of this problem are the undetermined coefficients and
the differential equation of parameter variations. This test uses the Method
of undetermined coefficients to solve differential equation problems. Step1:
the auxiliary equation is $r^{2}-2r+5=0$ with roots $r=1\pm 2i$, so the
complementary solution is: $\iota_{e}(x)=e^{x}(c_{1}\cos 2x+c_{2}\sin 2x).$
Try the particular solution: $y_{p}(x)=A\cos x+B\sin x$,
$y_{p}^{\prime}=-A\sin x+B\cos x$ Step2: substituting, we have: $(-A\cos
x-B\sin x)-2(-A\sin x+B\cos x)+5(A\cos x+B(\sin x)=\sin x$
$\quad\Rightarrow(4A-2B)\cos x+(2A+4B)\sin x=\sin x.$ Then:
$4A-2B=0,2A+4B=1\quad\Rightarrow\quad A=\frac{1}{10}$, $B=\frac{1}{3}$ and the
general solution is: $y(x)=y_{1}(x)+y_{p}(x)=e^{\prime}(c_{1}\cos
2x+c_{2}2x)+\frac{1}{10}\cos x+\frac{1}{3}\sin x$ But,
$1=y(0)=c_{1}+\frac{1}{10}\quad\Rightarrow\quad c_{1}=\frac{3}{10}$ and
$1=y^{\prime}(0)=2c_{2}+c_{1}+\frac{1}{5}\quad\Rightarrow\quad
c_{2}=-\frac{1}{26}$ Step3: Thus the solution to the initial-value problem is:
$y(x)=e^{x}\left(\frac{9}{10}\cos 2x-\frac{1}{20}\sin
2x\right)+\frac{1}{10}\cos x+\frac{1}{5}\sin x.$ To sum up, the answer is:
$y(x)=e^{x}\left(\frac{9}{10}\cos 2x-\frac{1}{20}\sin
2x\right)+\frac{1}{10}\cos x+\frac{1}{5}\sin x.$

Figure 10: An instruction example on the math subject.

Problem
Two lightbulbs, one rated 30 W at 120 V and another rated 40 W at 120 V, are
arranged in two different circuits. a. The two bulbs are first connected in
parallel to a 120 V source. i. Determine the resistance of the bulb rated 30 W
and the current in it when it is connected in this circuit. ii. Determine the
resistance of the bulb rated 40 W and the current in it when it is connected
to this circuit. b. The bulbs are now connected in series with each other and
have a 120 V source. i. Determine the resistance of the bulb rated 30 W and
the current in it when it is connected in this circuit. ii. Determine the
resistance of the bulb rated 40 W and the current in it when it is connected
to this circuit. c. In the spaces below, number the bulbs in each situation
described, in order of their brightness. (1= brightest, 4 = dimmest) ____30 W
bulb in the parallel circuit ____40 W bulb in the parallel circuit ____30 W
bulb in the series circuit ____40 W bulb in the series circuit d. Calculate
the total power dissipated by the two bulbs in each of the following cases. i.
The parallel circuit ii. The series circuit Answer Aanalysis: Use formulas to
calculate resistance, current, and total power. (a) i: Calculate the
resistance and current of the light bulb. $\mathrm{P=V^{2}/R}$ given R=480
$\Omega$ and V=IR given I=0.25 A ii: Calculate the resistance and current of
the light bulb. $\mathrm{P=V^{2}}/R$ given R=360 $\Omega$ and V=IR given
I=0.33 A (b) i./ii. Calculate the resistance and current of the light bulb.
The resistances are unchanged $480~{}\Omega$ and $360~{}\Omega$. The total
resistance in series is $480~{}\Omega$ \+ 360 $\Omega$ = 840 $\Omega$ making
the total current $\mathrm{I=V/R=0.14~{}A}$ which is the same value for both
resistors in series. (c) Compare the brightness of the light bulb. The bulbs
are brightest in parallel, where they provide their labeled values of 40 W and
30 W. In series, it is the larger resistor (the 30 W bulb) that glows brighter
with a larger potential difference across it in series. This gives the order
from top to bottom as 2 1 3 4. (d) i: Calculate the total power consumed by
two light bulbs. In parallel, they each operate at their rated voltage so they
each provide their rated power
and$\mathrm{P}_{\mathrm{T}}=30\mathrm{W}+40\mathrm{W}=70\mathrm{W}$ ii:
Calculate the total power consumed by two light bulbs In series
$\mathbf{P_{T}}=\mathbf{V_{T}}^{2}/\mathbf{R_{T}}=17\mathbf{W}$ In summary,
(a) i: $\mathrm{P=480}$ $\Omega$ and $\mathrm{V=0.25A}$ ii:
$\mathrm{P=360}~{}\Omega$ and $\mathrm{V=0.33A}$ (b) i/ii:
$\mathrm{P=840~{}\Omega}$ and $\mathrm{V=0.14A}$ (c) This gives the order from
top to bottom as 2 1 3 4. (d) i: $\mathrm{P}_{\mathrm{T}}=70\mathrm{W}$ ii:
$\mathbf{P_{T}}=17\mathbf{W}$

Figure 11: An instruction example on the physics subject.

Problem
Consider a mixture of the two solids,
$\mathrm{BaCl_{2}}+2\mathrm{H_{2}}\mathrm{O}$ (FM 244.26) and $\mathrm{KCl}$
(FM 74.551), in an unknown ratio. (The notation $\mathrm{BaCl_{2}}\cdot
2\mathrm{H_{2}}\mathrm{O}$ means that a crystal is formed with two water
molecules for each $\mathrm{BaCl_{2}}$.) When the unknown is heated to
$160^{\circ}\mathrm{C}$ for $1\mathrm{~{}h}$, the water of crystallization is
driven off: $\mathrm{BaCl_{2}}\cdot
2\mathrm{H_{2}}\mathrm{O}(s)\stackrel{{\scriptstyle
160^{\circ}\mathrm{C}}}{{\longrightarrow}}\mathrm{BaCl_{2}}(s)+2\mathrm{H_{2}}\mathrm{O}(\mathrm{g})\\\
$ A sample originally weighing $1.7839\mathrm{~{}g}$ weighed
$1.5623\mathrm{~{}g}$ after heating. Calculate the weight percent of
$\mathrm{Ba},\mathrm{K}$, and $\mathrm{Cl}$ in the original sample. Answer
Analysis: The content of this question is to calculate the weight percentage.
Step1: Formula and atomic masses:
$\mathrm{Ba}(137.327),\mathrm{Cl}(35.453),\mathrm{K}(39.098)$,
$\mathrm{H_{2}}\mathrm{O}(18.015),\mathrm{KCl}(74.551),\mathrm{BaCl_{2}}\cdot
2\mathrm{H_{2}}\mathrm{O}(244.26),\mathrm{H_{2}}\mathrm{O}$ lost $=1.7839-$
$1.5623=0.2216\mathrm{~{}g}=1.2301\times 10^{-2}\mathrm{~{}mol}$ of
$\mathrm{H_{2}}\mathrm{O}$. For $2\mathrm{~{}mol}\mathrm{H_{2}}\mathrm{O}$
lost, $1\mathrm{~{}mol}$ $\mathrm{BaCl_{2}}\cdot 2\mathrm{H_{2}}\mathrm{O}$
must have been present. $\frac{1}{2}\left(1.2301\times
10^{-2}\mathrm{~{}mol}\mathrm{H_{2}}\mathrm{O}\right.$ lost $)=$ $6.1504\times
10^{-3}\mathrm{~{}mol}\mathrm{BaCl_{2}}\cdot
2\mathrm{H_{2}}\mathrm{O}=1.5024\mathrm{~{}g}$. The $\mathrm{Ba}$ and
$\mathrm{Cl}$ contents of the $\mathrm{BaCl_{2}}\cdot
2\mathrm{H_{2}}\mathrm{O}$ are
$\mathrm{Ba}=\left(\frac{137.33}{244.26}\right)(1.5024\mathrm{~{}g})=0.84469\mathrm{~{}g}$
$\mathrm{Cl}=\left(\frac{2(35.453)}{244.26}\right)(1.5024\mathrm{~{}g})=0.43613\mathrm{~{}g}$
Step2: Because the total sample weighs $1.783\mathrm{~{}g}$ and contains
$1.5024\mathrm{~{}g}$ of $\mathrm{BaCl_{2}}\cdot 2\mathrm{H_{2}}\mathrm{O}$,
the sample must contain $1.7839-1.5024=0.2815\mathrm{~{}g}$ of $\mathrm{KCl}$,
which contains
$\mathrm{K}=\left(\frac{39.098}{74.551}\right)\left(0.281_{5}\right)=0.14763\mathrm{~{}g}$
$\mathrm{Cl}=\left(\frac{35.453}{74.551}\right)\left(0.281_{5}\right)=0.13387\mathrm{~{}g}$
Weight percent of each element: $\mathrm{Ba}=\frac{0.84469}{1.7839}=47.35\%$
$\mathrm{K}=\frac{0.14763}{1.7839}=8.28\%$
$\mathrm{Cl}=\frac{0.43613+0.13387}{1.7839}=31.95\%$ In summary, the weight
percent of $\mathrm{Ba}$ is $47.35\%$,the weight percent of $\mathrm{K}$ is
$8.28\%$, the weight percent of $\mathrm{Cl}$ is $31.95\%$.

Figure 12: An instruction example on the chemistry subject.

Problem
lemma unfold_sub $\alpha$ [subtraction_monoid $\alpha$] (a b c : $\alpha$) (h
: a + -b = c) : a - b = c :=. Answer
by rw [sub_eq_add_neg, h] Problem lemma subtype.exists_pi_extension ι : Sort*
$\alpha$ : ι → Sort* [ne : Π i, nonempty (α i)] p : ι → Prop (f : Π i :
subtype p, $\alpha$ i) : ∃ g : Π i : ι, $\alpha$ i, (λ i : subtype p, g i) = f
:= Answer begin tactic.classical, refine ⟨λ i, if hi : p i then f ⟨i, hi⟩ else
classical.choice (ne i), funext _⟩, rintro ⟨i, hi⟩, exact dif_pos hi end

Figure 13: Examples on formal proofs.