#  Mamo: a  Mathematical  Modeling Benchmark with Solvers

Xuhan Huang
The Chinese University of Hong Kong, Shenzhen
<EMAIL_ADDRESS>
&Qingning Shen
The Chinese University of Hong Kong, Shenzhen
<EMAIL_ADDRESS>
&Yan Hu
The Chinese University of Hong Kong, Shenzhen
Shenzhen Research Institute of Big Data
<EMAIL_ADDRESS>
&Anningzhe Gao
Shenzhen Research Institute of Big Data
<EMAIL_ADDRESS>
&Benyou Wang 11footnotemark: 1
The Chinese University of Hong Kong, Shenzhen
Shenzhen Research Institute of Big Data
<EMAIL_ADDRESS>Corresponding authors<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Mathematical modeling involves representing real-world phenomena, systems, or
problems using mathematical expressions and equations to analyze, understand,
and predict their behavior. Given that this process typically requires
experienced experts, there is an interest in exploring whether Large Language
Models (LLMs) can undertake mathematical modeling to potentially decrease
human labor. To evaluate of LLMs in mathematical modeling, we introduce a new
benchmark, Mamo, that transcends traditional result-oriented assessments.
Unlike conventional methods that primarily assess LLMs based on the accuracy
of solutions to mathematical problems, our approach offers deeper insight into
the modeling process itself. By focusing on the processes LLMs undertake
rather than the correctness of their final solutions, Mamo pioneers a novel
evaluation paradigm. This shift underscores the importance of understanding
the inherent modeling capabilities of LLMs, paving the way for a more nuanced
and comprehensive analysis of their problem-solving strategies. Our work marks
a significant advancement in the field, suggesting a new direction for future
research by emphasizing the evaluation of LLMs’ modeling processes over the
mere correctness of answers. This benchmark not only facilitates a better
understanding of LLMs’ mathematical modeling capabilities but also sets a new
standard for evaluating their performance in complex problem-solving
scenarios.

##  1 Introduction

Recent advancements in Large Language Models (LLMs) have garnered widespread
interest, demonstrating remarkable capabilities across a broad spectrum of
natural language processing tasks Ouyang et al. (2022); OpenAI (2023); Nijkamp
et al. (2022); Tang et al. (2024). However, the domain of mathematics remains
a critical aspect of LLM evaluation Wei et al. (2022). This focus not only
gauges LLMs’ proficiency in comprehending and addressing mathematical
challenges but also serves as a profound indicator of their underlying
abilities in abstract conceptualization and logical reasoning, among other
intellectual faculties.

Natural language captures human complexity and nuance, while mathematical
language excels in precision and modeling the natural world (Giordano et al.,
2013). Bridging their gap requires high-level cognitive skills akin to
Artificial General Intelligence (AGI), a task at which LLMs show promise.
However, the challenge lies in the evaluation of modeling, as perfectly
representing reality may be inherently elusive.

In this endeavor, solvers111In optimization, solvers are algorithms used to
find the best solution to a problem. They search through possible options to
maximize or minimize an objective under given constraints if any. By using
solves, The challenge becomes problem formulation under a given solver or a
solver set.  becomes critically important as it standardizes the modeling
output and could provide a way to falsifiably validate the produced answers.
Recent studies have already combined LLMs with solvers Yang et al. (2023);
Feng et al. (2023); Pan et al. (2023). In the field of optimization,
AhmadiTeshnizi et al. (2023b) introduced OptiMUS, an LLM-based agent that uses
a solver to address optimization problems. The key task for LLM in OptiMUS
involves first generating a mathematical formulation of a real-world problem,
followed by writing Python code to invoke a solver. This implies the potential
for testing LLM’s modeling capabilities.

We thus develope Mamo, a new benchmark that leverages solvers to assess the
mathematical modeling prowess of large language models (LLMs). Our approach
shifts from conventional outcome-focused evaluation to process-focused
evaluation, offering in-depth insights into the LLMs’ problem-solving
strategies. We concentrate on the models’ mathematical modeling skills,
delegating the task of solving abstract problems to solvers to circumvent
errors arising from the models’ computational capabilities.

The contributions of this paper are as below:

  1. 1.

Clarification of a Mathematical Modeling Framework for LLMs in Natural
Language Domains: We have meticulously clarified the concept of mathematical
modeling within Large Language Models (LLM), focusing on the realm of natural
language challenges. This effort has culminated in the establishment of a
comprehensive framework designed to guide the mathematical modeling process in
LLMs, addressing both theoretical underpinnings and practical applications.

  2. 2.

Innovation in Benchmarking Strategy through Solver Integration: Our research
introduces a pioneering benchmarking strategy for the assessment of
mathematical modeling process. By incorporating solvers into the benchmarking
methodology, this strategy facilitates a rigorous evaluation process. This
novel approach sets a new standard for the assessment in NLP research.

  3. 3.

Creation of the Mamo Benchmark for Extensive Mathematical Modeling Assessment:
We have developed a new benchmark, named Mamo, specifically tailored for the
evaluation of mathematical modeling capabilities. Mamo encompasses a wide
array of modeling questions, focusing on areas such as ordinary differential
equations and optimization problems within linear programming and mixed-
integer linear programming frameworks. With a total of 1059 meticulously
curated questions, Mamo stands as a comprehensive tool for assessing the
breadth and depth of mathematical modeling proficiency.

##  2 Related Work

Recent studies have made significant strides in applying LLMs to mathematical
problem-solving, introducing innovative datasets for comprehensive evaluation.
Frieder et al. (2024) and Yuan et al. (2023) have focused on datasets that
test LLMs across various math problems and arithmetic expressions,
respectively. Lewkowycz et al. (2022) explored training models using natural
language paired with LaTeX-formatted mathematical content from arXiv,
enhancing syntactic understanding. Zhang et al. (2024) developed the CARP
dataset for computation-intensive challenges, while He et al. (2024)
introduced OlympiadBench, a bilingual and multimodal benchmark for
competition-level mathematical reasoning and proofs. Liu et al. (2024)
constructed MathBench, which is designed with a carefully structured
difficulty hierarchy and focuses on assessing the understanding of theoretical
knowledge in LLMs. AhmadiTeshnizi et al. (2023a) constructed NLP4LP, a
benchmark comprising 52 linear programming (LP) and mixed-integer linear
programming (MILP) problems. These efforts collectively push the boundaries of
LLMs in mathematical cognition, showcasing the growing complexity and
specificity of tasks LLMs are being trained to tackle.

The pursuit of enhancing and devising novel solvers has seen a variety of
research efforts, with a notable focus on the integration of LLMs. Yang et al.
(2023) investigated the potential of employing LLMs as direct solvers,
pioneering the exploration of LLMs’ autonomous problem-solving capabilities.
He-Yueya et al. (2023) introduced a hybrid approach that marries an LLM with
an external symbolic solver, specifically for equation solving, marking a
significant advancement in combining neural and symbolic computation.
Furthermore, Pan et al. (2023) developed LOGIC-LM, a cutting-edge framework
that fuses LLMs with symbolic solvers, aimed at enhancing the solving of
logical problems. In a closely related study, AhmadiTeshnizi et al. (2023a)
introduced OptiMUS, an LLM-based agent explicitly crafted for formulating and
solving optimization problems, thus demonstrating the broad applicability and
potential of LLMs in tackling intricate computational challenges. These
endeavors underscore the innovative directions being explored in the realm of
solver development, leveraging the robust capabilities of LLMs.

##  3 Background

###  3.1 Mathematical Modeling

Giordano et al. (2013) emphasize the role of mathematical models as
intermediaries that transform real-world problems into structured mathematical
forms, facilitating the discovery of solutions. Despite LLMs’ proven
competencies across various tasks, their capacity for developing mathematical
models remains uncertain. This involves two key phases: model formulation,
which demands a thorough comprehension of the problem, and model resolution,
typically addressed by computational solvers. To accurately gauge LLMs’
potential in this domain, our strategy focuses on their ability to construct
mathematical models, entrusting calculations to specialized solvers. This
approach highlights LLMs’ creativity and generalization skills, simultaneously
capitalizing on solvers’ computational efficiency.

##### Motivations to benchmark mathematical modeling for LLMs

We evaluate mathematical models for two reasons. The first reason is out of
curiosity: We aim to investigate whether LLMs can do what expert humans can.
Mathematical modeling is considered a highly advanced capability that we
believe experts possess. This is a beneficial test for exploring the
boundaries of LLMs’ capabilities and to what extent they can approach human
intelligence. On the other hand, in terms of evaluating the models themselves,
most current benchmarks, especially those related to mathematics like GSM8K
Cobbe et al. (2021), etc., show that the performance has become saturated.
Even a 7B model Yu et al. (2023b); Liu et al. (2023) (sometimes with a
verifier-like model Yu et al. (2023a)) can achieve a accuracy exceeding 80%,
closely approaching that of GPT 4. Therefore, we believe there is a need to
construct more challenging mathematical datasets. Traditional mathematical
modeling is a labor-intensive, custom process reliant on human expertise,
raising the question: Can large language models automate or assist in this
process?

###  3.2 Using Solvers in Mathematical Modeling

The model generated by the LLM is then subjected to verification against real-
world data to assess its validity, ensuring it addresses the problem logically
and sensibly. The solver, an algorithmic tool designed to find solutions to
mathematical models, is also involved in this process and it is also
beneficial for verification.

Different types of solvers are specialized for various classes of problems,
such as optimization solvers. However, they share the common property: They
are all algorithmic tools designed to find solutions to mathematical models.

Rationale of Using solver in Mathematical Modeling for LLMs These solvers
operationalize the abstract constructs created by LLMs, translating them into
tangible outputs that can be analyzed and validated against real-world data
and phenomena. The solver is a crucial instrument that resolves the model
presented by the LLM, and by comparing the solution it provides with the
correct answer, we assess the accuracy of the mathematical model generated by
the LLM.

##  4  Philosophy of Mamo Benchmark

###  4.1 Methodology: exact answer verification with solvers

We utilize solver to resolve this mathematical model. The solver operates at
the final state of the question-solving process, meaning it directly computes
the final answer from the model without intermediate steps. By executing the
solver and juxtaposing its output ($\widehat{A}$) with the correct answer (A)
to the original problem, we can verify the accuracy of the Language Model’s
mathematical
model(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\widehat{M}}$).
This final-state approach ensures that our assessment is squarely focused on
the Language Model’s capacity to model the problem, abstracting from any
additional computational steps that might otherwise obscure the evaluation of
the model’s validity.

Figure 1: The pipeline to use exact answer verification via an additional
solver.

###  4.2 Scope of Mathematical Modeling

We construct the category of Mathematical Modeling by referring to Giordano et
al. (2013). In the context of Natural Language we propose a framework in
mathematical modelling in LLM, specified in solving natural language problem,
see Tab. 1.

Category | In Mamo
---|---
Changes and Differences | ✗
Proportionality and Geometric Similarity | ✗
Optimization Problem Modeling | ✓
Differential Equations | ✓
Probabilistic Modeling | ✗
Table 1: The taxonomy of mathematical modeling. In Mamo benchmark, we only
include the categories where sophisticated Solvers is available.

Given the broad spectrum of mathematical modeling and the availability of
specific solvers, our benchmark is meticulously designed to encompass domains
where these tools can be effectively utilized. This section outlines the
rationale for our focus on certain areas of mathematical modeling, based on
solver availability and the clarity of the modeling process.

Availability of Sophisticated Solvers: Advanced optimization solvers, such as
COPT Ge et al. (2023), along with Python libraries for solving ordinary
differential equations (ODEs), offer a strong foundation for testing the
capabilities of Large Language Models (LLMs). These tools enable a detailed
and automated evaluation of the LLMs’ abilities to abstract, formulate, and
accurately solve mathematical problems within these specific domains.

Challenges in Modeling Changes and Differences, Proportionality and Geometric
Similarity: Although simpler tools like calculators could address problems
related to modeling changes and differences, proportionality and geometric
similarity, the lack of a defined, structured modeling process complicates the
implementation of our final-state-approach benchmark. Our objective is to
assess the LLMs’ skill in developing comprehensive mathematical models that
adhere to a solvable framework. The absence of a distinct modeling procedure
in these areas impedes a standardized and objective assessment of an LLM’s
modeling efficiency.

Following the considerations mentioned above, we have developed our benchmark
in the fields of optimization, and ODE, due to the lack of advanced and
universal Partial Differential Equations (PDE) solvers. In the optimization
segment, our focus is on Linear Programming (LP) and Mixed-Integer Linear
Programming (MILP) problems. Compared to NLP4LP, introduced by AhmadiTeshnizi
et al. (2023b) and focusing exclusively on optimization, our benchmark
includes a larger number of problems. This benchmark set is highly scalable,
with potential future expansions into nonlinear optimization and probabilistic
modeling. Additionally, as new and advanced solvers emerge, it could be
expanded into areas like Partial Differential Equations (PDE) and beyond.

##  5 Data Collection and Synthesis

###  5.1 Data Selection and Source Credibility

The data set for our benchmark consists of a blend of manually selected and
GPT-generated questions. All items within the data set have been subjected to
a meticulous review process conducted by qualified collectors and domain
experts to ensure their validity and relevance.

##### Annotator Qualifications

The benchmarking process is conducted by a team of qualified individuals with
extensive expertise in various mathematical disciplines. 222See the details of
the annotator qualifications in Appendix. B:

####  5.1.1 Manually Selected Questions

  * •

Source and Reliability: When doing manually selecting, we refer to the
textbooks, exercises and the solutions in ODE and optimization area(Thomas et
al., 2016; Stewart et al., 2014; Lial et al., 2017; Braun, 1993; Boyce &
DiPrima, 2012; Giordano et al., 2014; Zill, 2013; Giordano et al., 2014;
Bertsimas & Tsitsiklis, 1997; Hurlbert, 2010). The selecting process focus on
deriving the mathematical model in the questions and assign a reasonable
scenario. The source ensure the correctness of the answer of each question.

  * •

Rewriting Process: Each question is meticulously rewritten to fit the
benchmark’s criteria(the properties in Section 5.3), which may involve
assigning new scenarios or adapting the question and its answer to align with
our testing methodology. This process also circumvents potential licensing
issues, as the original content is significantly transformed.

##### Data Synthesis

The process of annotating data starts with the creation of typical
mathematical constructs, into which random parameters are introduced to give
them specificity. This is followed by the computation of answers to establish
a ground truth. Subsequently, GPT-4 is employed to craft real-life scenarios
based on these predefined models, effectively translating abstract
mathematical concepts into tangible, context-rich problems. This approach not
only tests the model’s ability to handle mathematical reasoning but also its
capacity to contextualize mathematical principles within real-world
situations. See the example in Appendix. E.

##### Cross Validation

GPT-4 is employed to confirm that the newly generated questions are amenable
to being solved by solvers using the final-state approach, ensuring the
benchmark’s focus remains on the LLM’s modeling capabilities. Moreover, we
employ a comprehensive review process, re-examining each question for
consistency and correctness, thereby ensuring the quality of our dataset. See
the review details in Appendix. A.

###  5.2 Data Statistics and Visualization

In the section on Ordinary Differential Equations (ODE), we present a total of
346 problems: 196 are based on first-order equations, 110 on second-order
equations, and 40 on systems of equations, offering a comprehensive
exploration across different levels.

The optimization section is divided into two segments: Easy LP, featuring 652
high school-level Mixed Integer Linear Programming (MILP) problems, and
Complex LP, comprising 211 undergraduate-level problems that integrate both LP
and MILP.

See the word cloud of the combined data in Figure 2

Figure 2: The word cloud of the data after filtering the requirement words
such as ’significant figures’, ’rounded’

###  5.3 Guidance of Data Synthesis

The synthesis of problems for our benchmark are governed by specific criteria
designed to evaluate the mathematical modeling capacity of Large Language
Models (LLMs). The guidelines ensure that problems are appropriate for our
final-state approach and yield numerical answers suitable for automated
verification. The following are the key considerations in our benchmark
guidance:

######  Property 1.

Final-State Approach Solvability: Problems should be structured so that they
can be solved by a solver in a final-state approach, meaning the solution
process should not require intermediate steps once the mathematical model has
been formulated by the LLM.

######  Property 2.

Numerical Answers: The answers to all questions must be numerical, allowing
for clear-cut evaluation and comparison with the solver’s output.

######  Property 3.

Significant Figures and Precision: Each question will explicitly state the
required significant figures or the level of numerical precision for the
answer. This clarity ensures that the LLM’s output can be properly evaluated
against the solver’s results, maintaining consistency and rigor in the
assessment of the model’s accuracy.

######  Property 4.

Modeling Capacity Test: To specifically test the LLM’s modeling capabilities:

  * •

For Ordinary Differential Equations (ODEs), questions should focus on
determining the value of the original function at a specific time $t_{0}$.

  * •

For optimization problems, questions should aim to find the optimal value.

These constraints facilitate the testing accuracy and the convenience of
verifying potentially multiple valid solutions to a single problem.

######  Property 5.

Real-World Problem Context: Questions should be framed as real-world problems
without explicitly stating the underlying mathematical model, challenging the
LLM to demonstrate its capacity for abstraction and application in practical
scenarios.

##  6 Evaluation

###  6.1 Evaluation Protocol

####  6.1.1 Evaluation Details

Our evaluation protocol is meticulously designed to test the LLMs’
mathematical modeling process. For ODE problems, the LLM is prompted to
generate Python code from a natural language description. The code is
executed, and its output is compared with the correct answer to assess
accuracy. In optimization problems, the LLM is asked to express the model in
.lp format, after which the COPT solver is used to find the optimal value for
comparison with the correct answer.

##### Model output

In ODE parts, we ask LLMs to output Python code calling solvers ( solve_ivp
odeint in SciPy, and dsolve in NumPy). In the optimization sections, we prompt
LLMs to generate the standard .lp file. Compared to the testing methodology in
NLP4LP (AhmadiTeshnizi et al., 2023b), which involves prompting the model to
output Python code for invoking the solver, the .lp format more closely aligns
with the standard optimization form, making it more readable. Furthermore,
optimization solvers such as COPT (Ge et al., 2023) can directly read and
solve the .lp file, minimizing potential issues related to the LLM’s coding
capabilities. To better adhere to the format, we conducted tests using 3-shot
learning. Detailed information about the number of few-shot samples used in
our experiments and the result can be found in the Appendix. (see Appendix C
and Appendix. G).

##### Mitigation of Formatting Errors

It is essential to distinguish errors in modeling from those in coding or file
formatting. Therefore, if the LLM’s output contains syntax errors or incorrect
.lp formatting, we need to rectify these issues without altering the
underlying logic of the code or model. This step ensures that our evaluation
focuses solely on the modeling process. To achieve this, we use a language
model to fix any syntax errors, referring to it as the code modifier. In our
experiments, we use the original language model and GPT-4-0613 as code
modifiers. For the sake of reproducibility, we choose the orignal LLM; To
ensure the comparison focuses exclusively on modeling capacity, we choose
GPT-4-0613 as the code modifier due to its exceptional coding proficiency and
operational reliability, as well as its stability than other provisional
release Language Models. See the comparison of the result from different code
modifier in Section 6.3 and the prompts in Appendix. G. Besides, at the
beginning of the testing, we do the first clean of the codes, clearing up the
string like " “‘python “‘ " that will affect our testing.

####  6.1.2 Evaluation Metrics

Given the potential for minor discrepancies in numerical solutions, our
protocol includes a comparison rule that accommodates for slight inaccuracies.
Let $O$ denote the output from the LLM, and $A$ denote the standard answer.
The comparison is defined by the following criteria 333 See the evaluation
script in Appendix. F

Precision Adjustment for Floating-Point Numbers If the standard answer, $A$,
is a floating-point number, the precision adjustment involves calculating the
number of significant digits, $n$, after the decimal point. Both $O$ and $A$
are then scaled by $10^{n}$ for comparison.

General Comparison Criterion The overall correctness is determined based on
the following conditions, applicable to both adjusted and original values:

$\text{Correctness}=\begin{cases}1&\text{if }\left|\frac{O-A}{A}\right|\leq
1\times 10^{-4}\text{ or }|O-A|\leq 1,\\\ 0&\text{otherwise.}\end{cases}$ (1)

This structured evaluation ensures that the LLM’s output is accurately
assessed against the standard answer with consideration for the specific
requirements of numerical precision and the inherent variability in
computational answers.

###  6.2 Benchmarking Results

Models | ODE | Optimization
---|---|---
First order | Second order | System | All | Easy LP | Complex LP
GPT-4o | 69.39% | 73.64% | 42.50% | 67.63% | 87.27% | 23.22%
GPT-4o † |  69.90% |  74.55% | 42.50% |  68.21% | 87.42% |  32.23%
GPT-4-turbo | 66.84% | 69.09% | 42.50% | 65.12% |  87.88% | 23.70%
GPT-4-turbo † | 67.86% | 70.00% | 42.50% | 65.99% |  87.88% | 30.33%
GPT-4 | 60.71% | 65.45% | 42.50% | 60.12% | 86.50% | 22.27%
GPT-4 † | 61.73% | 67.27% | 45.00% | 61.56% | 86.96% | 24.64%
GPT-3.5-turbo | 17.35% | 16.36% | 7.50% | 15.90% | 81.29% | 9.95%
GPT-3.5-turbo † | 25.51% | 31.82% | 15.00% | 26.30% | 84.82% | 9.95%
Claude-3-Sonnet | 53.57% | 61.82% | 30.00% | 53.47% | 83.59% | 25.12%
Claude-3-Sonnet † | 55.61% | 63.64% | 35.00% | 55.78% | 84.20% | 25.59%
Claude-3-Haiku | 42.35% | 60.91% | 20.00% | 45.66% | 78.53% | 19.91%
Claude-3-Haiku † | 52.04% | 63.64% | 22.50% | 52.31% | 85.89% | 21.33%
Gemini-1-pro | 26.53% | 35.45% | 2.50% | 26.59% | 78.99% | 15.31%
Gemini-1-pro † | 35.71% | 39.09% | 7.50% | 33.53% | 80.06% | 15.31%
Gemini-1.5-pro | 66.33% | 66.36% |  47.50% | 64.16% | 85.28% | 30.81%
Gemini-1.5-pro † | 67.35% | 66.36% | 52.50% | 65.32% | 85.28% | -
DeepSeek-v2 | 40.82% | 45.45% | 32.50% | 41.33 % | 86.50% | 6.16%
DeepSeek-v2 † | 42.35% | 64.55% | 32.50% | 48.27% | 86.66% | 6.16%
DeepSeek-coder | 32.14% | 49.09% | 15.00% | 35.55% | 75.77% | 18.96%
DeepSeek-coder † | 34.18% | 49.09% | 15.00% | 36.71% | 76.99% | 18.96%
Table 2: Evaluation result on Mamo Benchmark. The remark $\dagger$ indicates
it additionally uses the original LLM to rectify syntax error of the code and
.lp file generated by the corresponding LLM. Here GPT-4-turbo denotes
GPT-4-turbo-2024-04-09 and GPT-4o denotes GPT-4o-2024-05-13, while GPT-4
refers to GPT-4-0613

Our evaluation includes both closed-source LLMs and open-source LLMs. The
closed-source commercial LLMs include the GPT-4 series OpenAI (2023), the
Claude series 444https://www.anthropic.com/news/claude-3-family, and the
Gemini series 555https://deepmind.google/technologies/gemini/. The open-source
LLMs include DeepSeek-v2 DeepSeek-AI (2024) and DeepSeek-coder Guo et al.
(2024). Table 2 presents the evaluation results of different language models
on the Mamo Benchmark. The performance of these models is assessed both in
their original form and when enhanced with the language model itself to
rectify syntax errors (indicated by the dagger symbol †). Table 3 displays the
performance results when GPT-4-0613 is selected as the code modifier.

Models | ODE | Optimization
---|---|---
First order | Second order | System | All | Easy LP | Complex LP
GPT-4o †† |  69.90% |  74.55% | 42.50% |  68.21% | 87.42% | 29.86%
GPT-4-turbo †† | 67.85% | 70.00% | 42.50% | 65.99% |  88.19% | 28.44%
GPT-4 †† | 61.73% | 67.27% | 45.00% | 61.56% | 86.96% | 24.64%
GPT-3.5-turbo †† | 47.96% | 60.91% | 20.00% | 48.84% | 85.43% | 10.43%
Claude-3-Sonnet †† | 58.16% | 66.36% | 35.00% | 59.09% | 85.12% | 28.44%
Claude-3-Haiku †† | 53.06% | 65.45% | 25.00% | 53.76% | 86.81% | 21.33%
Gemini-1-pro †† | 43.37% | 54.55% | 15.00% | 43.64% | 80.37% | 16.27%
Gemini-1.5-pro †† | 67.35 % | 67.27% |  52.50% | 65.61% | 85.28% |  34.12%
DeepSeek-v2 †† | 56.12% | 68.18% | 32.50% | 57.23% | 86.81% | 6.16%
DeepSeek-coder †† | 33.67% | 49.09% | 17.50% | 36.71% | 78.83% | 18.96%
Table 3: Evaluation result on Mamo Benchmark. The remark $\dagger\dagger$
indicates it additionally uses the GPT-4-0613 to rectify syntax error of the
code and .lp file generated by the corresponding LLM.

The evaluation results on the Mamo Benchmark reveal that GPT-4o stands out
among the models in both ODE and LP tasks. Specifically, GPT-4o achieves the
highest performance in ODE tasks with 69.90% in first-order and 74.55% in
second-order equations, and also excels in LP tasks with 87.42% in easy LP and
32.23% in complex LP scenarios when using the LLM to rectify syntax errors.
Additionally, DeepSeek-v2, as an open-source LLM, demonstrates significant
capability with 48.27% overall in ODE tasks and 86.66% in easy LP tasks when
enhanced with syntax correction, highlighting its potential despite being
open-source.

The inclusion of the LLM itself as code modifier to rectify syntax errors
results in significant improvements across various models. For instance,
GPT-3.5 Turbo’s performance in ODE tasks increased from 15.90% to 26.30%
overall and Claude-3-Haiku shows a remarkable improvement in ODE tasks from
45.66% to 52.31%. Similarly, DeepSeek-v2, as an open-source LLM, demonstrates
significant capability with its overall ODE performance increasing from 41.33%
to 48.27% when enhanced with syntax correction.

Selecting GPT-4 as a code modifier generally results in better performance
than using the LLM itself for self-improvement. For instance, DeepSeek-v2’s
accuracy in ODE tasks improves from 48.27% with self-improvement to 57.23%
when GPT-4 is used as the code modifier. This pattern indicates that GPT-4 is
a more effective code modifier, enhancing the accuracy and performance of
various models more reliably than self-improvement.

###  6.3 On the format errors and code modifiers

Models | ODE | Optimization
---|---|---
format | raw | self-improved | GPT-4-improved | raw | self-improved | GPT-4-improved
GPT-4o | 96.82% | 99.42% | 99.42% | 92.93% | 97.91% | 97.91%
GPT-4-turbo | 96.82% | 99.13% | 98.84% | 93.86% | 97.80% | 97.56%
GPT-4 | 94.51% | 97.69% | 97.69% | 90.96% | 95.13% | 95.13%
GPT-3.5-turbo | 32.37% | 60.40% | 94.22% | 87.95% | 89.80% | 96.76%
Claude-3-Sonnet | 84.68% | 93.06% | 97.40% | 91.89% | 93.97% | 96.76%
Claude-3-Haiku | 78.03% | 92.77% | 98.27% | 82.61% | 92.58% | 96.29%
Gemini-1.0-pro | 56.36% | 69.94% | 88.73% | 88.06% | 89.11% | 94.21%
Gemini-1.5-pro | 93.64% | 98.27% | 98.84% | 95.48% | - | 96.87%
deepseek-v2 | 70.81% | 83.81% | 96.24% | 90.27% | 92.00% | 94.90%
deepseek-coder | 88.44% | 93.64% | 95.95% | 92.58% | 96.29% | 98.84%
Table 4: Correct format rate on raw, self-improved, and GPT4-improved format.

The data presented in Table 4 provides a comparative analysis of the
performance (rate of correct format) of various models using different
improvement methods. The table considers three scenarios: no improvement,
self-improvement, and improvement provided by GPT-4.

The GPT-3.5-turbo model shows the most dramatic improvement when enhanced by
GPT-4, jumping from a 32.37% success rate to 94.22%. Interestingly, Claude-3
models (Sonnet, and Haiku) and DeepSeek models (DeepSeek-v2, DeepSeek-coder)
also exhibit high improvement rates when enhanced by GPT-4, highlighting the
compatibility of GPT-4’s improvement methods with other models. In the realm
of Optimization tasks, the GPT-3.5-turbo model again shows a significant
improvement when enhanced by GPT-4, jumping from a 87.95% success rate to
96.76%.

The improvement from GPT-4 is generally superior to both self-improvement and
no improvement. This indicates that GPT-4 can serve as a more effective and
fair code modifier. By using GPT-4 for syntax error correction, we can better
examine the modeling ability of different LLMs without the confounding factor
of formatting errors. Notably, in ODE tasks, the correct format rate for all
models, except one, is generally greater than 94%, with the lowest being
88.73%. In Optimization tasks, the correct format rate is even higher, with
the lowest rate being 94.21%.

##  7 Conclusion and Future Directions

In this article, we introduce Mamo, a novel benchmark that utilizes solvers to
evaluate the mathematical modeling capabilities of large language models
(LLMs). We propose a shift in the evaluation paradigm from traditional
outcome-oriented evaluation to process-oriented evaluation. This shift allows
us to scrutinize the modeling process undertaken by LLMs, providing a more
detailed and comprehensive assessment of their mathematical problem-solving
abilities. Our evaluation focuses more on the mathematical modeling
capabilities of the models, leaving the process of solving abstract
mathematical problems to the solvers, thereby avoiding the impact of errors
such as computational capabilities of the models on their actual modeling
abilities.

Our benchmark represents an important step in understanding the fundamental
modeling capabilities of LLMs. By focusing on the modeling process rather than
the correctness of the final solution, we are able to gain valuable insights
into the strengths and limitations of current LLMs. This approach provides new
directions for future research, emphasizing the importance of the modeling
process in solving mathematical problems.

The introduction of Mamo paves new avenues for future research. The benchmark
can be expanded to include a wider variety of mathematical problems and
solvers, broadening its applicability and providing a more comprehensive
evaluation of the mathematical modeling capabilities of LLMs. Insights gained
from the benchmark can be used to guide the development of future LLMs. By
understanding the strengths and weaknesses of current models in solving
mathematical problems, researchers can design new models that better handle
these tasks. We will extend our model list to other models (Llama2 series,
Mistral series etc) in our further study.

##### Limitations

While the current benchmarking methodology necessitates LLMs to generate
Python code for ODEs and .lp files for optimization problems, it opens up
exciting avenues for future research. The process, while effective, may
inadvertently influence the LLMs’ focus towards formalization over the
conceptual modeling, potentially affecting the depth of mathematical
reasoning. This presents an opportunity to explore innovative benchmark
designs that can better distinguish between an LLM’s modeling prowess and its
formalization abilities. Future research could aim to develop methodologies
that allow LLMs to demonstrate their mathematical modeling capabilities
without the intermediary step of formalization, leading to a more nuanced and
comprehensive assessment of their abilities

## Ethics Statement:

The benchmark is designed with a stringent ethical framework to ensure that
all problems are socially responsible and do not perpetuate or encourage
harmful biases or stereotypes. Questions are meticulously reviewed to avoid
any content that could lead to the dissemination of misinformation, support
unethical practices, or cause harm to individuals or groups. This includes but
is not limited to issues of privacy, security, and the potential for misuse of
the model. Furthermore, the benchmark abstains from any problem that could
indirectly endorse unethical behaviors or decisions in real-world scenarios,
upholding the highest standards of academic integrity and social
responsibility.

## References

  * AhmadiTeshnizi et al. (2023a) Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell.  Optimus: Optimization modeling using mip solvers and large language models.  _arXiv preprint arXiv:2310.06116_ , 2023a.
  * AhmadiTeshnizi et al. (2023b) Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell.  OptiMUS: Optimization Modeling Using MIP Solvers and large language models, October 2023b.  URL http://arxiv.org/abs/2310.06116.  Issue: arXiv:2310.06116 arXiv:2310.06116 [cs].
  * Bertsimas & Tsitsiklis (1997) Dimitris Bertsimas and John N. Tsitsiklis.  _Introduction to linear optimization_.  Athena Scientific, 1997.
  * Boyce & DiPrima (2012) William E. Boyce and Richard C. DiPrima.  _Elementary differential equations and boundary value problems_.  Wiley, 10 edition, 2012.
  * Braun (1993) Martin Braun.  _Differential equations and their applications an introduction to applied mathematics_.  Springer, 4 edition, 1993.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * DeepSeek-AI (2024) DeepSeek-AI.  Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
  * Feng et al. (2023) Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen.  Language models can be logical solvers.  _arXiv preprint arXiv:2311.06158_ , 2023.
  * Frieder et al. (2024) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner.  Mathematical capabilities of chatgpt.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ge et al. (2023) Dongdong Ge, Qi Huangfu, Zizhuo Wang, Jian Wu, and Yinyu Ye.  Cardinal Optimizer (COPT) user guide.  https://guide.coap.online/copt/en-doc, 2023.
  * Giordano et al. (2013) Frank R. Giordano, Maurice D. Weir, and William P. Fox.  _A First Course in Mathematical Modeling_.  Brooks/Cole, Cengage Learning, 5th edition, 2013.
  * Giordano et al. (2014) Frank R. Giordano, Steven B. Horton, and William P. Fox.  _A first course in mathematical modeling_.  Brooks/Cole, 5 edition, 2014.
  * Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang.  Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024.
  * He et al. (2024) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.  Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.  _arXiv preprint arXiv:2402.14008_ , 2024.
  * He-Yueya et al. (2023) Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D Goodman.  Solving math word problems by combining language models with symbolic solvers.  _arXiv preprint arXiv:2304.09102_ , 2023.
  * Hurlbert (2010) Glenn Hurlbert.  _Linear Optimization: The simplex workbook_.  Springer, 2010.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.  Solving quantitative reasoning problems with language models.  _Advances in Neural Information Processing Systems_ , 35:3843–3857, 2022.
  * Lial et al. (2017) Margaret L. Lial, Raymond N. Greenwell, and Nathan P. Ritchey.  _Calculus with applications_.  Pearson, 11 edition, 2017.
  * Liu et al. (2023) Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang.  Tinygsm: achieving> 80% on gsm8k with small language models.  _arXiv preprint arXiv:2312.09241_ , 2023.
  * Liu et al. (2024) Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen.  Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark, 2024.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_ , 2022.
  * OpenAI (2023) R OpenAI.  Gpt-4 technical report. arxiv 2303.08774.  _View in Article_ , 2(5), 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Pan et al. (2023) Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang.  Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning.  _arXiv preprint arXiv:2305.12295_ , 2023.
  * Stewart et al. (2014) James Stewart, Dan Clegg, and Saleem Watson.  _Calculus: Early transcendentals_.  Cengage, 8 edition, 2014.
  * Tang et al. (2024) Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei.  Mathscale: Scaling instruction tuning for mathematical reasoning.  _arXiv preprint arXiv:2403.02884_ , 2024.
  * Thomas et al. (2016) George B. Thomas, Maurice D. Weir, Joel Hass, and Frank R. Giordano.  _Thomas’ calculus_.  Pearson, 13 edition, 2016.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.  Large language models as optimizers.  _arXiv preprint arXiv:2309.03409_ , 2023.
  * Yu et al. (2023a) Fei Yu, Anningzhe Gao, and Benyou Wang.  Outcome-supervised verifiers for planning in mathematical reasoning.  _arXiv preprint arXiv:2311.09724_ , 2023a.
  * Yu et al. (2023b) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint arXiv:2309.12284_ , 2023b.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang.  How well do large language models perform in arithmetic tasks?  _arXiv preprint arXiv:2304.02015_ , 2023.
  * Zhang et al. (2024) Beichen Zhang, Kun Zhou, Xilin Wei, Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen.  Evaluating and improving tool-augmented computation-intensive math reasoning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Zill (2013) Dennis G. Zill.  _A first course in differential equations with modeling applications_.  Cengage, 10 edition, 2013.

##  Appendix A Cross Review

In ODE part, we have the following review process:

  1. 1.

We employed GPT-4 to confirm that the newly generated questions are amenable
to being solved by solvers using the final-state approach. There were only 10
out of 383 questions are invalid. All were filtered, 373 questions were left.

  2. 2.

Then we conducted a thorough evaluation of the questions. Our dataset
initially comprised $373$ questions. We identified $29$ questions that lacked
sufficient information, $21$ that had incorrect answers, and $18$ that
featured unclear statements. Following our assessment, $27$ questions were
deleted, and $41$ were corrected to ensure clarity and correctness.

Finally, we have 346 problems in ODE part.

We have also conducted the cross-review process. See the details in Appendix.
A.1 and the results on Appendix. A.1.1.

In the optimization section, particularly with easy LP part, out of 688
entries, 12 were found to be completely invalid (not meeting the criteria), 26
had incorrect answers, and 6 had misleading descriptions. All invalid entries
were filtered out, and the remaining issues were corrected manually. Among
them, 8 of them were corrected and others (36 questions) were filtered out. In
complex LP part, for the original 211 questions, 40 were found synthetized by
wrong mathematical models. All of them were corrected.

###  A.1 Cross-Review Process

Furthermore, adhering to the scaling law evident in the dataset’s
distribution, we selected 50 questions from the ODE parts in our dataset for a
deeper analysis. Four independent reviewers (see their qualification in
Section B) were tasked with assessing 50 questions each. The responses could
either be numeric or labeled as “error” if a question was deemed problematic.
This section outlines the metrics used to evaluate the reviewers’ responses
against pre-defined correct answers, which also include the “error” label for
any flawed questions. We use the same metric as described in Subsection 6.1.2.
Here are our results:

####  A.1.1 Results

The effectiveness of the review process was quantified using the metrics
defined in Subsection 6.1.2, focusing on the accuracy and inter-rater
reliability among reviewers. The statistical outcomes are summarized as
follows:

  * •

Average Cohen’s Kappa: The average Cohen’s Kappa across all reviewer pairs was
0.60, indicating a moderate to substantial agreement. This suggests that
reviewers generally agreed on the classification of answers, though there were
variations in some cases.

  * •

Minimum and Maximum Cohen’s Kappa: The minimum Kappa value recorded was 0.50,
and the maximum was 0.71. The spread in these values highlights areas where
alignment and training could potentially enhance consistency.

  * •

Accuracy Rates: The individual accuracy rates were as follows:

    * –

Reviewer 1: 90.0%

    * –

Reviewer 2: 84.0%

    * –

Reviewer 3: 88.0%

    * –

Reviewer 4: 74.0%

These rates reflect the precision with which each reviewer matched the
standard answers, including their recognition of problematic questions
correctly labeled as “error.”

These results provide insights into the overall effectiveness and areas of
improvement for the cross-review process, particularly in terms of aligning
understanding and interpretation of the evaluation criteria among reviewers.

##  Appendix B Annotator Qualifications

The benchmarking process benefits from the expertise of reviewers with robust
backgrounds in mathematics, underscored by their academic accomplishments. The
team is composed of individuals whose education encompasses critical
mathematical disciplines, including calculus (I and II), linear algebra (both
introductory and advanced levels), optimization, probability, and ordinary
differential equations. The average Grade Point Average (GPA) across these
foundational courses is approximately 3.89 out of a 4.0 scale. This high level
of academic achievement demonstrates the reviewers’ strong grasp of essential
mathematical concepts and their application. Furthermore, the team is enriched
by members who hold Ph.D. degrees in mathematics, further solidifying the
depth of expertise and analytical skills available for the benchmarking
process.

###  B.1 Qualification of Reviewers

We select the reviewers mentioned in Appendix A.1 based on the following
criteria:

  1. 1.

Undergraduate students who have taken fundamental mathematical courses
(Calculus and Linear Algebra) and have an average GPA of at least 3.5/4.0.

  2. 2.

Have completed or are currently taking courses in Ordinary Differential
Equations (ODE).

  3. 3.

Meet one of the following English proficiency requirements: TOEFL > 80, IELTS
> 6.5, a minimum grade of B+ in all English classes, or a Gaokao English score
> 120.

These criteria ensure their ability to read questions in English and solve ODE
problems.

##  Appendix C Few-shot Experiment

To test the sensitivity of different models to the number of prompts, we
conducted a sensitivity analysis using the scaling law to examine the effect
of the number of shots on model performance. In our analysis, we tested the
performances of 11 models on ODE problems using prompts with 0 shot, 1 shot, 3
shots, 5 shots, and 10 shots. The results, shown in Figure 3, illustrate the
ODE problem-solving accuracy of different models under varying numbers of
shots. Similarly, we performed sensitivity tests on optimization problems,
assessing the accuracy of 10 models with prompts of 0 shot, 1 shot, 3 shots,
and 5 shots. The results are depicted in Figure 4.

Figure 3: Results of models accuracy and shots on ODE problems Figure 4:
Results of models accuracy and shots on optimization problems

Additionally, we specifically tested the 0-shot performance of GPT-4-0613 on
all questions in Easy LP and Complex LP categories. The accuracy of GPT-4-0613
was 66.56% in Easy LP and 14.69% in Complex LP, resulting in an overall LP
accuracy of 53.65%, consistent with the results obtained from sampling.

##  Appendix D Testing Process

We use 3-shot learnging in testing, see examples in G

Figure 5: Example of testing process in optimization Figure 6: Example of
testing process in ODE

##  Appendix E Data Synthesis

Figure 7 shows the example of synthesis process in ODE.

Figure 7: Example of synthesis process in ODE

##  Appendix F Evaluation script

    def comp(output, standard_answer):
        dif = abs(float(output) - float(standard_answer))
        if float(standard_answer) == 0:
            rate = dif * 1e-4
        else:
            rate = dif / float(standard_answer)
        if abs(rate) <= 1e-4:
            return 1
        else:
            return 0

    def compare_output_with_standard(output, standard_answer):
        try:
            float_output = float(output)
        except ValueError:
            return False

        if ’.’ in standard_answer:
            digit = len(standard_answer.split(’.’)[1])
            s_ans = float(standard_answer) * 10 ** digit
            ans = float_output * 10 ** digit
            return (abs(ans - s_ans) <= 1 or comp(output, standard_answer))
        else:
            s_ans = float(standard_answer)
            ans = float_output
            return (abs(ans - s_ans) <= 1 or comp(output, standard_answer))

##  Appendix G Prompts

Assume you are a virtual assistant with expertise in optimization,
specifically in creating .lp files for linear programming problems. Your task
is to translate given natural language problems into optimization models,
formatted as .lp files.When you receive a question, it might include
mathematical expressions in LaTeX format. Your job is to interpret these
expressions accurately and model the problem in .lp format. Your response must
adhere to the following guidelines: \- The optimization model must be written
in .lp format, adhering to the conventions and syntax appropriate for linear
programming problems. \- The model should be designed so that solving it
yields the optimal value, which directly answers the question posed. \- Your
response should be an entire .lp file, ready to be processed by a linear
programming solver. Ensure that the file contains no comments or extraneous
content beyond the model itself. \- Handle LaTeX expressions with care to
ensure that the mathematical aspects of the problem are accurately represented
in the .lp model. \- If the solution needs to be rounded to an integer, make
use of the ’General’ integer constraint in the .lp file to specify integer
variables, please do not use ’General’ if there are not requirement for the
integer variables. Here comes the examples: Example_1: (the input) A
manufacturing company produces two types of products: $X$ and $Y$. The
production cost for each unit of product $X$ is $50$, while the cost for each
unit of product $Y$ is $10$. There are constraints in the production process,
such that twice the number of units produced from product $X$, plus the number
of units from product $Y$, cannot exceed 200 due to resource limitations. In
addition, to meet market demand, four times the number of units produced from
product $X$, plus the number of units from product $Y$, must be at least 50.
Considering these constraints and given that both products can only be
produced in whole numbers due to their physical nature, what is the minimum
total cost needed for producing these items while satisfying all conditions?
Provide your answer rounded to the nearest dollar. Your response: Minimize
obj: 50 x + 10 y Subject To c1: 2 x + y <= 200 c2: 4 x + y >= 50 Bounds x >= 0
y >= 0 Generals x y End […] Please craft the .lp file according to these
instructions, focusing on delivering a model that is directly solvable to
obtain the answer. And Please follow the syntax like examples to write the .lp
file. Here comes the question: [question] Generate the contents of an .lp file
for this problem, starting with the objective function and followed by the
constraints, without any additional sentences. The constraints should be
formatted as ’variable + variable >= number’ for inequalities, all the
variables shoule on the left hand side of the inequality . Ensure there is a
space between variables and their coefficients. Your response: Figure 8: The
prompt for testing in optimization part. [question] refers to a question in
the benchmark. […] refers to the examples. We use 3-shot in testing

Assume you are a virtual assistant with expertise in ordinary differential
equations (ODEs), particularly in formulating ODE models from natural language
descriptions and solving them using Python’s $`solve\\_ivp`$, $`odeint`$ from
SciPy, and $`dsolve`$ from SymPy. Your primary task is to convert the given
natural language problems into ODE models and then apply ODE solvers to find
the solutions. When you receive a question, it will often contain mathematical
expressions in LaTeX format, which you need to interpret accurately. Your
response must be in the form of Python code that directly outputs the final
solution to the problem upon execution. This Python code should adhere to the
following criteria: 1\. The output should only display the final answer of the
problem. 2\. Utilize a ’final-state-approach’ where the code immediately
prints the solution after solving the ODE, without showing any intermediate
steps. 3\. Ensure the answer is rounded to the specified number of significant
figures or decimal places. Use Python’s ‘round‘ function for decimal rounding.
For significant figures, include and use the following function in your code:
$```python$ def round_to_significant_figures(num, sig_figs): if num != 0:
return round(num, -int(math.floor(math.log10(abs(num))) + (1 - sig_figs)))
else: return 0 # Handles the case of num being 0 $```$ 4\. Your response
should be entirely in Python code, formatted to run directly without
modifications. 5\. Handle LaTeX expressions in the problem statement carefully
to ensure accurate modeling and solution. Here comes the examples: […] Please
process the problem according to these instructions, focusing solely on
delivering the Python code that meets these requirements. And Please directly
generate the code without any explaintion(except the comments in the code).
the following lines are forbidden: ′′′ Here’s the Python code that solves the
given problem and meets the specified requirements: $```$python $```$ This
code uses the $`solve_{i}vp`$ function from SciPy to solve the initial value
problem for the given differential equation. The
$`round_{t}o_{s}ignificant_{f}igures`$ function is included to round the final
answer to the specified number of significant figures. Upon execution, the
code will directly output the amount of pollutant left in the tank after 5
minutes, rounded to five significant figures. ′′′ Please avoid the above
sentences. Take a deep breathe before answering the question. This is a piece
of cake to you. Here comes the question: [question] Your response: Figure 9:
The prompt for testing in ODE part. [question] refers to a question in the
benchmark. […] refers to the examples.  You are experienced python engineer,
the following codes may have some errors (in syntax), with the error
information [error_info], please fix the errors to make sure the code can run
successfully. If there is no error, please return the original code. And
please do not change the original logic of the code. Your response should be
entirely in Python code, formatted to run directly without modifications. Take
a deep breathe before answering the question. This is a piece of cake to you.
PLEASE do not response anything except the code, No other comments outside the
code are allowed. The followings are forbidden: ”’ The code is almost correct,
but there is a syntax error in the function ‘round_to_significant_figures‘.
The round function is missing a closing parenthesis. Here is the corrected
version: “‘python “‘ ”’ Please avoid resposing above sentences. Please output
only the corrected code, with no additional text or explanations. Here comes
the code: [code] The correct code is: Figure 10: The prompt for fixing syntax
error in ODE. [error] refers to the message when executing the Python code.
While [code] refer to the code which contains syntax error You are experienced
engineer in optimization, please fix the errors to make sure the code can run
successfully. If there is no error, please return the original code. And
please do not change the original logic of the lp file. Your response should
be entirely in .lp format, formatted to run directly without modifications.
Take a deep breathe before answering the question. This is a piece of cake to
you. PLEASE do not response anything except the code, No comments are allowed.
The followings are forbidden: ”’ This is the correct .lp file: “‘lp “‘ ”’
Please avoid resposing above sentences. Please output only the corrected code,
with no additional text or explanations. Here comes the lp: [lp_code] Generate
the correct .lp format, starting with the objective function and followed by
the constraints, without any additional sentences. The constraints should be
formatted as ’variable + variable >= number’ for inequalities, all the
variables shoule on the left hand side of the inequality. For example, make a
+ b + c <= d into a + b + c - d <= 0. Ensure there is a space between
variables and their coefficients (coefficients should be numerical). The
correct lp code is: Figure 11: The prompt for fixing syntax error in LP.
[lp_code] refer to the lp code which contains syntax error