# Variable Sampling MPC via Differentiable Time-Warping Function

Zehui Lu, Shaoshuai Mou The authors are with the School of Aeronautics and
Astronautics, Purdue University, IN 47907, USA<EMAIL_ADDRESS>work was supported in part by grants from the NASA University Leadership
Initiative (ULI), Northrop Grumman Corporation, and Rolls-Royce Corporation.

###### Abstract

Designing control inputs for a system that involves dynamical responses in
multiple timescales is nontrivial. This paper proposes a parameterized time-
warping function to enable a non-uniformly sampling along a prediction horizon
given some parameters. The horizon should capture the responses under faster
dynamics in the near future and preview the impact from slower dynamics in the
distant future. Then a variable sampling MPC (VS-MPC) strategy is proposed to
jointly determine optimal control and sampling parameters at each time
instant. VS-MPC adapts how it samples along the horizon and determines optimal
control accordingly at each time instant without any manual tuning or trial
and error. A numerical example of a wind farm battery energy storage system is
also provided to demonstrate that VS-MPC outperforms the uniform sampling MPC.

##  I Introduction

In many applications such as energy management systems, transportation,
aerospace systems, and process control systems, a primary task is to make
real-time decisions or scheduling while optimizing a specific objective and
not violating some constraints [1]. If a dynamical model of such a system is
available prior, one commonly used method is model predictive control (MPC)
[1]. To formulate an MPC problem, a discrete-time dynamical model of the
system is required to construct a prediction of system behaviors over a
certain prediction horizon. To achieve an optimum of a given objective, MPC
methods usually explore all possible control inputs while guaranteeing that
these control inputs can forward propagate the given dynamics correctly and
not violate any given constraints.

When MPC is applied to multi-timescale systems, such as power grids[2, 3],
chemical processes [4, 5, 6], aerospace systems[7, 8, 9], and electrified
vehicles[10, 11], the slower dynamics often require a longer prediction
horizon. This usually leads to a higher dimension MPC problem potentially with
a larger computational burden. To address this challenge, the singular
perturbation theory[12] has been explored broadly, which decomposes a multi-
timescale system into two subsystems with faster and slower timescales,
respectively. Then a MPC controller can be developed for each of these two
subsystems. But this method is only applicable to those systems whose dynamics
can be explicitly decomposed into two subsystems with faster and slower
timescales, respectively.

Another method of controlling multi-timescale systems is hierarchical MPC
(H-MPC) [8, 2, 9, 10, 13, 14]. The H-MPC method first computes an optimal
reference by an MPC given slower dynamics over a relatively long prediction
horizon. Then, a reference tracking problem is solved by another controller
given faster dynamics over a shorted prediction horizon and thus some optimal
control inputs can be obtained. Besides time delays arising from communication
among controller, choosing a proper quantity as the reference value requires
prior knowledge of the specific system.

Briefly, to make MPC able to deal with multi-timescale systems well, the
prediction horizon should be longer to capture more look-ahead information in
the distant future while the sampling rate of MPC should be small enough to
provide more accurate prediction in the near future.  A multi-horizon MPC (MH-
MPC) [11, 15] has been studied recently, which combines a short receding
horizon and a long shrinking horizon altogether in one MPC formulation. The
short receding horizon indicates relatively accurate prediction with a higher
sampling rate, whereas the long shrinking horizon extends to the end of the
trip with a lower sampling rate. Even though MH-MPC exploits preview
information over a longer horizon, it introduces an extra computational
burden, especially at the beginning of the trip, because the dimension of the
MH-MPC problem varies and depends on the current progress over the entire
trip. In addition, MH-MPC requires the entire trip to be finite. Another
direction to develop MPC for multi-timescale systems is by the non-uniform
sampling MPC (NS-MPC) [16, 17, 18, 19], in which the prediction horizon is
partitioned into multiple parts and each part has a different sampling rate.
The dimension of the decision variables, i.e., the number of prediction steps,
is assumed to be fixed in order to avoid an extra computational footprint.
Both NS-MPC and MH-MPC requires manual tuning some parameters to to obtain
better performance. And determining the optimal settings involves trial and
error, and requires expert knowledge of the specific system.

Instead of manually tuning some parameters of a prediction horizon, this paper
seeks a differentiable time mapping from sampling time to actual time such
that it can describe any non-uniform sampling under some parameterizations.
Time-warping functions represent this kind of mapping, which was originally
proposed to deal with the time misalignment between two temporal
sequences[20], or between human demonstrations and system observations [21,
22, 23]. To describe the faster dynamics in the near future precisely and
preview the impact from the slower dynamics in the distant future, there are
some constraints on a time warping function, which require the function’s
differentiability. Then the function’s parameters can be a part of decision
variables to be optimized while designing control inputs at run-time.

To control a multi-timescale system by one MPC controller and avoid the manual
tuning of sampling, this paper proposes a variable sampling MPC (VS-MPC)
strategy to accurately capture the responses under the faster dynamics in the
near future and preview the impact from the slower dynamics in the distant
future.  In detail, a differentiable time warping function describes the
timeline of a prediction horizon.  The function is parameterized by some
decision variables and an optimal control problem jointly determines the
control inputs and function parameters at each time instant without any manual
tuning on the horizon. As the situation changes at run-time, VS-MPC adapts how
it samples along the horizon and then determines optimal control accordingly.
Section IV also studies how the proposed VS-MPC strategy performs in a
specific application where a control strategy needs to be designed to control
a battery energy storage system (BESS) for a wind farm. Section IV also
includes a performance comparison for several methods.

_Notation_ The real number set is $\mathbb{R}$. The non-negative real number
set is $\mathbb{R}_{\geq 0}$. Let
$\text{col}\\{\boldsymbol{v}_{1},\cdots,\boldsymbol{v}_{a}\\}$ denote a column
stack of elements $\boldsymbol{v}_{1},\cdots,\boldsymbol{v}_{a}$, which may be
scalars, vectors or matrices, i.e.
$\text{col}\\{\boldsymbol{v}_{1},\cdots,\boldsymbol{v}_{a}\\}\triangleq{\begin{bmatrix}{\boldsymbol{v}_{1}}^{\prime}&\cdots&{\boldsymbol{v}_{a}}^{\prime}\end{bmatrix}}^{\prime}$.
For a scalar $x\in\mathbb{R}$, $[x]^{+}\triangleq x$ when $x\geq 0$ and 0
otherwise. $\mathcal{N}(\mu,\sigma^{2})$ indicates a normal distribution with
a mean $\mu$ and a standard deviation $\sigma$.

##  II Problem Formulation

Suppose the continuous-time open-loop system dynamics for a plant are
described by

$\dot{\boldsymbol{x}}(t)=\boldsymbol{f}_{c}(\boldsymbol{x}(t),\boldsymbol{u}(t)),$
(1)

where $t\in\mathbb{R}_{\geq 0}$ denotes time,
$\boldsymbol{x}(t)\in\mathbb{R}^{n}$ denotes state at time $t$,
$\boldsymbol{u}(t)\in\mathbb{R}^{m}$ denotes input at time $t$, and
$\boldsymbol{f}_{c}:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{r}\mapsto\mathbb{R}^{n}$
denotes the nonlinear dynamics.

The open-loop control $\boldsymbol{u}(t)$ is determined by discrete-time model
predictive control with sampling in the following way.  Let $N$ denote the
number of steps in a prediction horizon and $\boldsymbol{x}_{k}$ denote the
value of $\boldsymbol{x}(t)$ at the sampling time $t_{k}$, $k=0,1,2,\cdots$,
i.e. $\boldsymbol{x}_{k}=\boldsymbol{x}(t_{k})$. By Euler integration with
non-uniform sampling time $\Delta_{k}>0$ at time $t_{k}$, one reaches the
following discretization of the continuous system in (1):

$\boldsymbol{x}_{k+1}=\boldsymbol{x}_{k}+\Delta_{k}\boldsymbol{f}_{c}(\boldsymbol{x}_{k},\boldsymbol{u}_{k}).$
(2)

Let

$J\triangleq
J(\boldsymbol{x}_{0:N|k},\boldsymbol{u}_{0:N-1|k},\Delta_{0:N-1|k}),$

where
$\boldsymbol{x}_{0:N|k}\triangleq\text{col}\\{\boldsymbol{x}_{k},\boldsymbol{x}_{k+1|k}\cdots,\boldsymbol{x}_{k+N|k}\\}\in\mathbb{R}^{n(N+1)}$
denotes the state at current time $t_{k}$ and the states from the future time
$t_{k+1}$ to $t_{k+N}$ that are predicted at time $t_{k}$; similarly
$\boldsymbol{u}_{0:N-1|k}\triangleq\text{col}\\{\boldsymbol{u}_{k|k},\cdots,\boldsymbol{u}_{k+N-1|k}\\}\in\mathbb{R}^{mN}$;
$\Delta_{0:N-1|k}\triangleq\text{col}\\{\Delta_{k|k},\cdots,\Delta_{k+N-1|k}\\}\in\mathbb{R}^{N}$
denotes the non-uniform sampling time intervals from time $t_{k}$ to
$t_{k+N-1}$ that are determined at current time $t_{k}$. Note that for a
uniform sampling, $\Delta_{k}$ is a constant for any time $t_{k}$.

Then at time $t_{k}$, given the current state $\boldsymbol{x}_{k}$, the
optimal control can be determined by —s— u_0:N-1—k Δ_0:N-1—k
J(x_0:N—k,u_0:N-1—k, Δ_0:N-1—k) x_k+j+1—k = x_k+j—k + Δ_k+j—k f_c(x_k+j—k,
u_k+j—k) ∀j=0,⋯,N-1 with given  x_k g(x_0:N—k, u_0:N-1—k, Δ_0:N-1—k) ≤0
h(x_0:N—k, u_0:N-1—k, Δ_0:N-1—k) = 0, where
$\boldsymbol{g}(\boldsymbol{x}_{0:N},\boldsymbol{u}_{0:N-1},\Delta_{0:N-1|k})$
denotes a column stack of inequality constraints;
$\boldsymbol{h}(\boldsymbol{x}_{0:N},\boldsymbol{u}_{0:N-1},\Delta_{0:N-1|k})$
denotes a column stack of equality constraints; $\leq$ and $=$ in these
constraints indicate element-wise inequality and equality. The discrete-time
optimal control that are determined at time $k$ will be denoted by
$\boldsymbol{u}^{*}_{0:N-1|k}$. Then in a receding horizon fashion, the system
will perform the optimal control $\boldsymbol{u}^{*}_{k|k}$ at time $t_{k}$,
update its states at time $t_{k+1}$, and then rerun the optimal control
problem (II) with current state $\boldsymbol{x}_{k+1}$. This procedure will be
performed repeatedly under a certain frequency.

The problem of interest is to find the discrete-time optimal control
$\boldsymbol{u}^{*}_{0:N-1|k}$ and sampling steps $\Delta_{0:N-1|k}$ jointly
at each time $t_{k}$, without any manual tuning of $\Delta_{0:N-1|k}$.

##  III Approach

This section introduces a variable sampling MPC (VS-MPC) strategy, which
partitions a prediction horizon with non-uniform sampling by a time warping
function. The time warping function is parameterized by some decision
variables and describes the mapping from sampling time to actual time. Then at
each time instant, VS-MPC solves an optimal control problem in which its
decision variables consist of the control inputs in (II) and the parameters of
time warping function. With the situation changing at each time instant, VS-
MPC finds optimal control inputs and sampling settings jointly without any
manual tuning.

###  III-A Time-warping Function

This paper proposes a differentiable time-warping function
$w:\mathbb{R}\mapsto\mathbb{R}$ and denote $t=w(\tau)$, where $\tau\geq 0$
denotes the sampling time and $t\geq 0$ denotes the actual time. The states
are sampled at $\tau=0,1,\cdots$ and the actual time associated with $\tau$
are $t=w(0),w(1),\cdots$. Some general constraints are considered on this
time-warping function:

$w(0)=0,\quad\frac{\partial
w(\tau)}{\partial\tau}\Big{|}_{\tau=\hat{\tau}}>0,\ \forall\hat{\tau}\geq 0$
(3)

Given the time-warping function $w(\cdot)$ and the formulation of (II), the
time interval between two adjacent time instants is

$\Delta_{j}=w(j+1)-w(j).$

Fig. 1 shows two most common time warping functions for MPC. The left one
indicates a uniform sampling when formulating an MPC problem, and the right
one indicates a non-uniform sampling which partitions the entire horizon into
multiple parts. Note that the right one is not differentiable at certain
points.

Figure 1: Two most common time warping functions for MPC.

To consider the system’s future behavior when developing control, a single
optimal control problem is often required to cover a total time $T$ given $N$
steps in the prediction horizon, i.e. $w(N)=T$. To make the prediction horizon
adjustable, one can rewrite $w(N)=T$ as

$\underline{\alpha}T\leq w(N)\leq\overline{\alpha}T,$ (4)

where $\underline{\alpha}>0$ and $\overline{\alpha}>\underline{\alpha}$.
Intuitively, one usually introduces small $\Delta_{k}$ in the near future and
larger $\Delta_{k}$ in the distant future, which leads to one additional
constraint:

$\frac{\partial w(\tau)}{\partial\tau}\Big{|}_{\tau=\tau_{2}}-\frac{\partial
w(\tau)}{\partial\tau}\Big{|}_{\tau=\tau_{1}}\geq 0,\
\forall\tau_{2}\geq\tau_{1}\geq 0.$ (5)

Without losing of generality, this paper chooses a polynomial with degree
$s=2$ to represent a time-warping function:

$t=\hat{w}(\tau,\boldsymbol{\beta})=\beta_{1}\tau+\beta_{2}\tau^{2},$

where
$\boldsymbol{\beta}={\begin{bmatrix}\beta_{1}&\beta_{s}\end{bmatrix}}^{\prime}\in\mathbb{R}^{s}$
is the coefficient vector that parameterizes the time warping function
$\hat{w}$. The constraints (3) and (5) can be rewritten as

$\begin{split}&\beta_{1}+2\beta_{2}\tau>0,\ \forall\tau\geq 0\\\
&2\beta_{2}\geq 0,\ \forall\tau\geq 0.\end{split}$ (6)

Thus the parameter $\boldsymbol{\beta}$ should satisfy the following
conditions:

$\begin{split}&\underline{\alpha}T\leq\hat{w}(N,\boldsymbol{\beta})\leq\overline{\alpha}T,\\\
&\beta_{1}>0,\ \beta_{2}\geq 0.\end{split}$ (7)

Note that at time $t_{k}$, the current time warping function
$\hat{w}_{k}(\tau,\boldsymbol{\beta})=\hat{w}(\tau,\boldsymbol{\beta})+t_{k}$
and
$\frac{\partial\hat{w}_{k}(\tau,\boldsymbol{\beta})}{\partial\tau}=\frac{\partial\hat{w}(\tau,\boldsymbol{\beta})}{\partial\tau}$.

A larger $s$ means that the parameterized time-warping function can represent
a more complicated time mapping. However, it is impossible to construct
constraints on $\boldsymbol{\beta}$ such that the conditions (3) and (5) are
satisfied when $s\geq 6$ because there is no algebraic solution to general
polynomial equations of degree five or higher with arbitrary coefficients, as
per Abel–Ruffini Theorem [24]. Hence, one cannot obtain a constraint for
$\boldsymbol{\beta}\in\mathbb{R}^{s}$ to satisfy (6) when $s\geq 6$.

Although a polynomial function is used to parameterize the time-warping
function, a time-warping function can be any functions, as long as the
constraints (3), (4), and (5) is satisfied. This paper uses polynomial
parameterization due to its simplicity.

###  III-B Variable Sampling MPC

If a non-uniform sampling is determined and well tuned given current
situation, this sampling is not necessarily suitable for next time instant,
which can be caused by external disturbance, etc. But variable sampling MPC
(VS-MPC) adapts how it samples along the horizon and determines optimal
control accordingly at each time instant $t_{k}$, without any manual tuning or
trial and error.  The VS-MPC strategy with variable sampling includes an
optimal control problem at an arbitrary time $t_{k}$, which is formulated as
follows: —s— u_0:N-1—k β_k ∈R^2  J(x_0:N—k,u_0:N-1—k, Δ_0:N-1—k) x_k+j+1—k =
x_k+j—k + Δ_k+j—k f_c(x_k+j—k, u_k+j—k) ∀j=0,⋯,N-1 with given  x_k g(x_0:N—k,
u_0:N-1—k, Δ_0:N-1—k) ≤0 h(x_0:N—k, u_0:N-1—k, Δ_0:N-1—k) = 0 Δ_k+j—k =
^w(j+1, β_k) - ^w(j, β_k) ∀j=0, ⋯, N-1 αT ≤^w(N,β_k) ≤¯αT -β_1 ¡0, -β_2 ≤0.
At time $t_{k}$, VS-MPC determines the optimal control and how it samples
given (III-B). Then it performs the optimal control and repeats this process
at next time instant.  The detailed explanation of the algorithm for VS-MPC is
shown below, where $\frac{1}{dt_{mpc}}$ indicates the MPC frequency.

Input: $k=0$, $t_{0}$, $x(t_{0})$, $N$, $dt_{mpc}>0$

1 while  _true_ do

2 $\boldsymbol{u}^{*}_{0:N-1|k},\ \boldsymbol{\beta}^{*}_{k}\leftarrow$ solve
$\eqref{prog:mpc_with_time_warping}$

3

4
$\hat{w}_{k}(\cdot,\boldsymbol{\beta})=\hat{w}(\cdot,\boldsymbol{\beta})+t_{k}$

5

6 $\boldsymbol{u}_{k}\leftarrow$ interpolate $\boldsymbol{u}^{*}_{0:N-1|k}$
for time $[t_{k},t_{k}+dt_{mpc}]$ given
$\hat{w}_{k}(\cdot,\boldsymbol{\beta})$ and zero-order hold

7 perform $\boldsymbol{u}_{k}$ until $t_{k}+dt_{mpc}$

8 $t_{k}\leftarrow t_{k}+dt_{mpc}$

9 $k\leftarrow k+1$

10

Algorithm 1 Variable Sampling MPC

##  IV Simulations

This section shows how the VS-MPC strategy is applied to a battery energy
storage system (BESS) for a wind farm, and compares the revenue regarding two
MPC strategies with uniform sampling and variable sampling.

###  IV-A Battery Energy Storage System for Wind Farm

This subsection discusses how to design controls of a BESS by MPC to provide
reserves to mitigate wind power intermittency and the following problem
formulation originates from Ref. [25]. In particular, an MPC strategy will be
used to control the charge and discharge of the battery in BESS to reduce the
negative impact caused by the wind intermittency. As shown in Fig. 2, the MPC
strategy is required to decide, at each time instant, how much wind power goes
to the power grid and how much goes to the BESS or how much power the BESS
discharge and then goes to the grid.

Figure 2: A battery energy storage system for a wind farm.

Assume that the nameplate capacity [MWh] of a wind farm is denoted by
$Q_{n}>0$. Assuming that the efficiencies of both charge and discharge are
perfect, and the responses of both are instantaneously fast, the dynamics of
the battery state of charge (SOC) is governed as follows:

$x(t)=-\frac{P_{batt}(t)}{Q_{c}},$ (8)

where $x(t)\in[0,1]$ is the battery SOC at time $t$;
$P_{batt}(t)\in\mathbb{R}$ is the battery discharge power [MW] and
$P_{batt}(t)<0$ indicates the battery is charging at time $t$; $Q_{c}>0$ is
the battery capacity [MWh].

Suppose that the control input is the scheduling wind power
$u(t)\in\mathbb{R}_{\geq 0}$ at time $t$ that goes to the grid and denote the
actual wind power at time $t$ as $w_{a}(t)\in\mathbb{R}_{\geq 0}$, then
$P_{batt}(t)=u(t)-w_{a}(t)$. The system dynamics of the BESS are written as
follows:

$\dot{x}(t)=f_{c}(x(t),u(t))=\frac{w_{a}(t)-u(t)}{Q_{c}}.$ (9)

Assume that $w_{a}(t)$ is unknown prior to determining the power scheduling
$u(t)$ at time $t$ but there is a wind power forecasting
$w_{f}(t)\in\mathbb{R}_{\geq 0}$ available at time $t$. Then, one needs to
determine the power scheduling based on wind power forecasting.

The cost function includes the revenue of selling wind power to the grid, the
expense of scheduling conventional reserves based on wind forecasting, the
expense of dispatching conventional reserves due to the mismatch between
actual and forecasted wind power, and the expense of ramping services[25].
Denote the power reserve requirement at time $t_{k}$ as
$r(u,w_{f})=[u(t_{k})-w_{f}(t_{k})]^{+}$ and the wind power shortage due to
the imperfect forecasting at time $t_{k}$ as
$d(u,w_{a})=[u(t_{k})-w_{a}(t_{k})]^{+}$. Denote
$\overline{P}(x_{k})\in\mathbb{R}_{\geq 0}$ as the battery discharging power
limit given the SOC $x_{k}$ at time $t_{k}$. Similarly,
$\underline{P}(x_{k})\in\mathbb{R}_{\leq 0}$ denotes the battery charging
power limit. Then the cost $c_{k}$ at time $t_{k}$ is defined by

$\begin{split}c_{k}=&-\alpha_{1}u_{k}+\alpha_{2}\Big{[}r\big{(}u_{k},w_{f}(t_{k})\big{)}-\overline{P}(x_{k})\Big{]}^{+}\\\
&+\alpha_{3}\Big{[}d\big{(}u_{k},w_{a}(t_{k})\big{)}-\overline{P}(x_{k})\Big{]}^{+}\\\
&+\alpha_{4}|u_{k}-u_{k-1}|.\end{split}$ (10)

Note that one cannot evaluate $c_{k}$ at current time $t_{k}$ because
$w_{a}(t_{k})$ is unknown. Instead, the estimated cost $\hat{c}_{k+j|k}$ of
time $t_{k+j}$ that is predicted at time $t_{k}$ is defined as

$\begin{split}\hat{c}_{k+j|k}=&-\alpha_{1}u_{k+j|k}+(\alpha_{2}+\alpha_{3})\Big{[}r\big{(}u_{k+j|k},w_{f}(t_{j+k})\big{)}\\\
&-\overline{P}(x_{k+j|k})\Big{]}^{+}+\alpha_{4}\ell(u_{k+j|k}-u_{k+j-1|k}),\end{split}$
(11)

where $\ell:\mathbb{R}\mapsto\mathbb{R}_{>0}$ denotes a smooth approximation
of absolute value function $|\cdot|$ and $\ell(x)=\sqrt{x^{2}+0.01}$. The
coefficients $\alpha_{1},\alpha_{2},\alpha_{3}$, and $\alpha_{4}$ are the unit
price of electricity generation, reserve scheduling, reserve dispatch, and
ramping services in the wholesale market, respectively. And these coefficients
are determined based on statistics in [26], i.e.

$\alpha_{1}=1,\alpha_{2}=1.03,\alpha_{3}=1,\alpha_{4}=0.5455.$ (12)

Thus, the total estimated cost over time horizon $[t_{k},t_{k+N}]$ is defined
by

$\begin{split}\hat{J}_{k}=\sum_{j=0}^{N-1}\hat{c}_{k+j|k}\Delta_{k+j|k},\end{split}$
(13)

where $u_{k-1|k}$ indicates the previous control input at time $t_{k-1}$.
Hence, the optimal control problem with variable sampling (III-B) at each time
$t_{k}$ can be rewritten as follows: —s— u_0:N-1—k β_k ∈R^2  ^Jkw(N, β)
x_k+j+1—k = x_k+j—k + Δ_k+j—k f_c(x_k+j—k, u_k+j—k) Δ_k+j—k = w(j+1, β_k) -
w(j, β_k) αT ≤w(N,β_k) ≤¯αT -β_1 ¡0, -β_2 ≤0. SOC_min ≤x_k+j—k ≤SOC_max 0
≤u_k+j—k ≤Q_n P(x_k+j—k) ≤u_k+j—k - w_f(t_k+j) ≤¯P(x_k+j—k) ∀j=0,⋯,N-1 with
given  x_k, where the objective $\frac{\hat{J}_{k}}{w(N,\boldsymbol{\beta})}$
measures the average cost over the entire prediction horizon $[0,\
w(N,\boldsymbol{\beta})]$.

###  IV-B Result

The parameters are: $N=10$, $T=1$ hour, $Q_{n}=400$ MWh,
$\underline{\alpha}=1$, $\overline{\alpha}=4$, $SOC_{min}=0.3$,
$SOC_{max}=0.9$, $x(0)=0.4$. When $Q_{c}\leq Q_{n}$, the battery charge and
discharge limits are defined as follows [25]:

$\overline{P}(x)=Q_{c}x,\ \underline{P}(x)=Q_{c}(x-1),\ x\in[0,1].$ (14)

When $Q_{c}>Q_{n}$, the limits are defined by

$\overline{P}(x)=\begin{cases}Q_{c}x,&\text{$x\in[0,\
\frac{Q_{n}}{Q_{c}}]$}\\\ Q_{n},&\text{$x\in[\frac{Q_{n}}{Q_{c}},\
1]$}\end{cases},$ (15)

and

$\underline{P}(x)=\begin{cases}-Q_{n},&\text{$x\in[0,\
1-\frac{Q_{n}}{Q_{c}}]$}\\\ Q_{c}(x-1),&\text{$x\in[1-\frac{Q_{n}}{Q_{c}},\
1]$}\end{cases}.$ (16)

Figure 3: The actual and forecasting wind power trajectories.
$w_{f}(t)=120\text{sin}(\frac{\pi
t}{3})+100\text{sin}(2\pi\frac{t+2}{3}+0.4)+150$. The actual wind $w_{a}(t)$
equals to the forecasting $w_{t}(t)$ adding a random noise whose distribution
is $\mathcal{N}(0,40^{2})$. The actual wind is clipped to zero when negative.

The simulation’s time step is 0.1 hour and the total simulation horizon is 24
hour. The wind trajectories are shown in Fig. 3. Ref. [25] also proposes a
heuristic control algorithm, which is written as follows:

$u(t)=w_{f}(t)\cdot 2x(t).$ (17)

This section adopts both the heuristic control algorithm and an MPC strategy
with uniform sampling for revenue comparisons. The prediction horizon for the
MPC with uniform sampling is 1 hour and includes 10 steps.

Fig. 4 compares the average revenue (the converse of the total cost) given
different battery capacities and control strategies when wind forecasting is
perfect. The battery capacity varies from 200 MWh (50% of the nameplate) to
1200 MWh (300%). And all the average revenues are normalized as the revenue of
200 MWh given uniform sampling MPC is set to 1. Fig. 4 shows that the proposed
VS-MPC strategy outperforms the other two method for all battery capacities
listed in the figure. As the battery capacity is increasing, the revenue for
two MPC strategies increases because a larger battery has more capability to
compensate for wind intermittency. The revenue difference between two MPC
strategies is roughly constant as the battery capacity growing. Since the
maximum length of prediction horizon for VS-MPC is constant over these cases,
the exclusive look-ahead information that VS-MPC obtains is nearly the same.
And the revenue difference between the heuristic control algorithm and MPC
strategies grows when the battery capacity rises because the heuristic
algorithm does not exploit any look-ahead information.

Figure 4: Average revenue (normalized) when wind forecasting is perfect. The
number over red bar is the relative difference of revenue between two MPC
strategies.

Fig. 5 compares the average revenue given different battery sizes and control
strategies when wind forecasting is imperfect (wind trajectories shown in Fig.
3). Fig. 5 also shows that the VS-MPC strategy outperforms the other two
method for all battery capacities. And all the other observations are
consistent with those mentioned in Fig. 4.

Figure 5: Average revenue (normalized) when wind forecasting is imperfect.

Fig. 6 shows the trajectories of the control input (power scheduling) and the
state (SOC) for 3 methods when $Q_{c}$ = 400 MWh. Since the heuristic
algorithm does not use any prediction, its SOC oscillates around 0.5 due to
the forecasting error. Since it does not use much battery capacity for
compensating the wind intermittency, its average revenue underperforms the MPC
strategies. As for the VS-MPC, since it is possible to exploit much more look-
ahead information by a longer horizon, its behavior around 5, 10, 17, 22 hours
is more smooth than the uniform sampling MPC, which reduces the ramping cost.

(a) The trajectories of input for 3 methods.

(b) The trajectories of SOC for 3 methods.

Figure 6: Details about the BESS system when the wind forecasting is
imperfect. $Q_{c}$ = 400 MWh.

##  V Conclusion

This paper proposes a variable sampling model predictive control (VS-MPC)
strategy, which can deal with multi-timescale systems with only one
controller. Unlike the existing non-uniform sampling MPC (NS-MPC) or multi-
horizon MPC (MH-MPC) strategies, VS-MPC does not require manual tuning on some
parameters for the prediction horizon. Instead, VS-MPC constructs a
differentiable and parameterized time warping function to describe the
sampling nature of a non-uniform horizon. Then an optimization program jointly
determines the optimal control inputs and the parameters for the time warping
function at each time instant. Lastly, this paper uses an example of BESS for
a wind farm to demonstrate the performance of VS-MPC. In addition, some
revenue comparisons for several methods are performed and they show that VS-
MPC outperforms the other two methods, including the uniform sampling MPC
strategy.

Potential future work includes the application of VS-MPC into more realistic
and complicated systems, such as battery thermal management systems,
integrated power and thermal management systems for hybrid electric vehicles,
HVAC (heating, ventilation, and air conditioning) systems, etc.

## References

  * [1] J. H. Lee, “Model predictive control: Review of the three decades of development,” International Journal of Control, Automation and Systems, vol. 9, no. 3, pp. 415–424, 2011.
  * [2] W. C. Clarke, C. Manzie, and M. J. Brear, “Hierarchical economic mpc for systems with storage states,” Automatica, vol. 94, pp. 138–150, 2018.
  * [3] R. Zafar, J. Ravishankar, J. E. Fletcher, and H. R. Pota, “Multi-timescale model predictive control of battery energy storage system using conic relaxation in smart distribution grids,” IEEE Transactions on Power Systems, vol. 33, no. 6, pp. 7152–7161, 2018.
  * [4] M. Baldea and P. Daoutidis, “Control of integrated process networks—a multi-time scale perspective,” Computers & chemical engineering, vol. 31, no. 5-6, pp. 426–444, 2007.
  * [5] Y. Q. Gao, L. Yang, Y. Fan, and Q. Shao, “Thermodynamics and kinetics simulations of multi-time-scale processes for complex systems,” International Reviews in Physical Chemistry, vol. 27, no. 2, pp. 201–227, 2008\.
  * [6] Z. Wei, J. Zhao, D. Ji, and K. J. Tseng, “A multi-timescale estimator for battery state of charge and capacity dual estimation based on an online identified model,” Applied energy, vol. 204, pp. 1264–1274, 2017.
  * [7] Y. Shtessel, C. Hall, and M. Jackson, “Reusable launch vehicle control in multiple-time-scale sliding modes,” Journal of Guidance, Control, and Dynamics, vol. 23, no. 6, pp. 1013–1020, 2000.
  * [8] J. P. Koeln, H. C. Pangborn, M. A. Williams, M. L. Kawamura, and A. G. Alleyne, “Hierarchical control of aircraft electro-thermal systems,” IEEE transactions on control systems technology, vol. 28, no. 4, pp. 1218–1232, 2019\.
  * [9] W. Wang and J. P. Koeln, “Hierarchical multi-timescale energy management for hybrid-electric aircraft,” in Dynamic Systems and Control Conference, vol. 84270, p. V001T11A002, American Society of Mechanical Engineers, 2020.
  * [10] M. R. Amini, I. Kolmanovsky, and J. Sun, “Hierarchical mpc for robust eco-cooling of connected and automated vehicles and its application to electric vehicle battery thermal management,” IEEE Transactions on Control Systems Technology, vol. 29, no. 1, pp. 316–328, 2020.
  * [11] Q. Hu, M. R. Amini, I. Kolmanovsky, J. Sun, A. Wiese, and J. B. Seeds, “Multihorizon model predictive control: An application to integrated power and thermal management of connected hybrid electric vehicles,” IEEE Transactions on Control Systems Technology, vol. 30, no. 3, pp. 1052–1064, 2021\.
  * [12] P. Kokotović, H. K. Khalil, and J. O’reilly, Singular perturbation methods in control: analysis and design.  SIAM, 1999.
  * [13] M. Farina, X. Zhang, and R. Scattolini, “A hierarchical multi-rate mpc scheme for interconnected systems,” Automatica, vol. 90, pp. 38–46, 2018.
  * [14] H. C. Pangborn, C. E. Laird, and A. G. Alleyne, “Hierarchical hybrid mpc for management of distributed phase change thermal energy storage,” in 2020 American Control Conference (ACC), pp. 4147–4153, IEEE, 2020.
  * [15] Q. Hu, M. R. Amini, A. Wiese, M. Tascillo, J. B. Seeds, I. Kolmanovsky, and J. Sun, “A spatial data-driven vehicle speed prediction framework for energy management of hevs using multi-horizon mpc with non-uniform sampling,” in 2022 American Control Conference (ACC), IEEE, 2022.
  * [16] C. K. Tan, M. J. Tippett, and J. Bao, “Model predictive control with non-uniformly spaced optimization horizon for multi-timescale processes,” Computers & Chemical Engineering, vol. 84, pp. 162–170, 2016.
  * [17] O. Gomozov, J. P. F. Trovao, X. Kestelyn, and M. R. Dubois, “Adaptive energy management system based on a real-time model predictive control with nonuniform sampling time for multiple energy storage electric vehicle,” IEEE Transactions on Vehicular Technology, vol. 66, no. 7, pp. 5520–5530, 2016\.
  * [18] D. Liao-McPherson, S. Kim, K. Butts, and I. Kolmanovsky, “A cascaded economic model predictive control strategy for a diesel engine using a non-uniform prediction horizon discretization,” in 2017 IEEE Conference on Control Technology and Applications (CCTA), pp. 979–986, IEEE, 2017.
  * [19] T. Brüdigam, D. Prader, D. Wollherr, and M. Leibold, “Model predictive control with models of different granularity and a non-uniformly spaced prediction horizon,” in 2021 American Control Conference (ACC), pp. 3876–3881, IEEE, 2021.
  * [20] H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization for spoken word recognition,” IEEE transactions on acoustics, speech, and signal processing, vol. 26, no. 1, pp. 43–49, 1978.
  * [21] P. Kingston and M. Egerstedt, “Time and output warping of control systems: Comparing and imitating motions,” Automatica, vol. 47, no. 8, pp. 1580–1588, 2011.
  * [22] A. Vakanski, I. Mantegh, A. Irish, and F. Janabi-Sharifi, “Trajectory learning for robot programming by demonstration using hidden markov model and dynamic time warping,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 42, no. 4, pp. 1039–1052, 2012.
  * [23] W. Jin, T. D. Murphey, D. Kulić, N. Ezer, and S. Mou, “Learning from sparse demonstrations,” IEEE Transactions on Robotics, 2022.
  * [24] M. I. Rosen, “Niels hendrik abel and equations of the fifth degree,” The American Mathematical Monthly, vol. 102, no. 6, pp. 495–505, 1995.
  * [25] C.-T. Li, H. Peng, and J. Sun, “Mpc for reducing energy storage requirement of wind power systems,” in 2013 American Control Conference, pp. 6607–6612, IEEE, 2013.
  * [26] I. M. Monitor, “2010 state of the market report for the miso electricity markets,” 2011.


# SciEx: Benchmarking Large Language Models on Scientific Exams with Human
Expert Grading and Automatic Grading

Tu Anh Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon
Reiß,
Jueun Lee, Nathan Lerzer, Fabian Ternava, Jianfeng Gao, Alexander Waibel,
Tamim Asfour,
Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan
Niehues
Karlsruhe Institute of Technology, Karlsruhe, Germany
<EMAIL_ADDRESS>
###### Abstract

With the rapid development of Large Language Models (LLMs), it is crucial to
have benchmarks which can evaluate the ability of LLMs on different domains.
One common use of LLMs is performing tasks on scientific topics, such as
writing algorithms, querying databases or giving mathematical proofs. Inspired
by the way university students are evaluated on such tasks, in this paper, we
propose SciEx - a benchmark consisting of university computer science exam
questions, to evaluate LLMs ability on solving scientific tasks. SciEx is (1)
multilingual, containing both English and German exams, and (2) multi-modal,
containing questions that involve images, and (3) contains various types of
freeform questions with different difficulty levels, due to the nature of
university exams. We evaluate the performance of various state-of-the-art LLMs
on our new benchmark. Since SciEx questions are freeform, it is not
straightforward to evaluate LLM performance. Therefore, we provide human
expert grading of the LLM outputs on SciEx. We show that the free-form exams
in SciEx remain challenging for the current LLMs, where the best LLM only
achieves 59.4% exam grade on average. We also provide detailed comparisons
between LLM performance and student performance on SciEx. To enable future
evaluation of new LLMs, we propose using LLM-as-a-judge to grade the LLM
answers on SciEx. Our experiments show that, although they do not perform
perfectly on solving the exams, LLMs are decent as graders, achieving 0.948
Pearson correlation with expert grading.

SciEx: Benchmarking Large Language Models on Scientific Exams with Human
Expert Grading and Automatic Grading

Tu Anh Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon
Reiß, Jueun Lee, Nathan Lerzer, Fabian Ternava, Jianfeng Gao, Alexander
Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher,
Klemens Böhm, Jan Niehues Karlsruhe Institute of Technology, Karlsruhe,
Germany<EMAIL_ADDRESS>

##  1 Introduction

In recent years, Large Language Models (LLMs) have proven their usefulness
across a wide range of tasks, from conversational agents to code generation
Rajkumar et al. (2022); Abbasian et al. (2023); Liao et al. (2023). Given the
fast pace of development in the field, with an increasing number of LLMs being
trained and released, it is important to have indicators of LLM performance on
different domains. This can be achieved by establishing evaluation benchmarks
that assess the capabilities of LLMs across diverse use cases.

One use case of LLMs is to handle scientific tasks. Some previous works have
introduced benchmarks containing questions on science topics Welbl et al.
(2017); Lu et al. (2022); Gilson et al. (2022); Schubert et al. (2023); Zhang
et al. (2024). However, these benchmarks are limited to multiple-choice
questions. This restricts the variability of questions, such as instruction-
follow ones like "write a mathematical proof for this statement …".
Additionally, it is difficult to ask certain types of questions in a multiple-
choice way without including the answer in the question itself. Multiple-
choice benchmarks therefore create a gap between testing and actual usage,
since they only evaluate whether the LLMs choose the correct answer, whereas
in real life, the users are more likely to ask open-ended questions to the
LLMs. In contrast, some other works have introduced freeform question
benchmarks. These works either convert multiple-choice questions to freeform
questions Bhakthavatsalam et al. (2021), or focus on a specific type of
problem such as answering questions related to a paper Dasigi et al. (2021),
thus still limiting the variability of the questions.

In this paper, we introduce a new benchmark, termed SciEx (Scientific Exams),
designed to evaluate this capability. Inspired by the way students are
evaluated in university, we created the benchmark by evaluating the
performance of LLMs on university computer science exams. SciEx’s questions
are in various formats, from multiple choice to open-ended, thus suitable to
evaluate LLM’s capabilities of generating free-text answers that fit the
requirements of the questions. It is multilingual, containing exams in both
German and English. It is multimodal, as exam questions can also contain
figures. The set of questions is a good mix of different difficulty levels
since they are designed for university exams. This enables us to evaluate LLMs
on different levels, and we found that stronger LLMs tend to perform better on
more difficult questions.

Unlike the previous multiple-choice benchmarks, the questions in SciEx are
freeform, making it non-trivial how to evaluate the LLM output. Therefore, we
make use of expert grading, i.e., having the lecturers grade the LLM output
the same way they would grade student answers. We also ask the experts to
perform qualitative analysis of the LLM output. With expert grading, we
provide a highly reliable way of evaluating LLMs, which is more reliable than
previous work that uses crowd-sourced evaluation. Expert grading by lecturers
also provides an opportunity to compare LLMs’ performance to university
student performance in a similar setting. We find that the stronger LLMs,
i.e., Claude and GPT-4V, are able to outperform the student average. However,
they are still far from perfect, achieving only 59% across SciEx exams.

Since new LLMs are constantly being released, we cannot fully rely on expert
grading for evaluation. Therefore, we provide an automatic grading scheme by
using LLM as a judge so that future LLMs can also be evaluated on SciEx.
Interestingly, we find that, although LLMs do not perform too well as
examinees, they perform well as graders, achieving over 0.948 Pearson
correlation to expert grading in the best setting.

In summary, our contributions are as follows:

  * •

SciEx111We release SciEx under CC BY-NCSA 4.0 license. Code:
https://github.com/TuAnh23/SciEx. Data:
https://huggingface.co/datasets/tuanh23/SciEx. \- a freeform, multimodal,
multilingual benchmark consisting of university computer science exams,
outputs of various LLMs on the exams, and expert grading of the LLM output.

  * •

Detailed quantitative and qualitative analysis comparing LLM to student
performance.

  * •

Automatic grading with 0.948 Pearson correlation to expert grading

##  2 Related Work

##### General-Purpose LLM Benchmarks

In order to rank different LLMs, there are several commonly used public
benchmarks. For example, Zheng et al. (2024) introduced MT-bench and Chatbot
Arena. MT-bench is a multi-turn question set; and Chatbot Arena is a
crowdsourced battle platform for LLMs where the users can ask their questions
and vote for the better LLM answer. Another benchmark is MMLU Hendrycks et al.
(2020), which is a multitask dataset covering multiple domains such as
mathematics, US history and law.

##### Scientific LLM Benchmarks

To specifically focus on the scientific domain, previous studies have
established benchmarks, such as SciQ Welbl et al. (2017) and ScienceQA Lu et
al. (2022), which feature questions spanning various scientific subjects. More
recent works have focused on benchmarking LLMs on solving exam questions on
some narrow science domains such as medical Gilson et al. (2022) or neurology
Schubert et al. (2023). M3Exam Zhang et al. (2024), in contrast, provides exam
questions to benchmark LLMs which span over multiple topics and multiple
educational levels (primary, middle, and high school). However, all benchmarks
mentioned above are limited to multiple-choice questions. While this
simplifies the evaluation process, it does not allow us to assess the LLMs’
capability to generate natural text.

Other studies have instead provided scientific benchmarks with open-ended
questions. Some examples are Qasper Dasigi et al. (2021) and ARC-DA
Bhakthavatsalam et al. (2021). However, Qasper only focuses on questions about
NLP papers rather than on general computer science topics. ARC-DA is closer to
our work, since it contains open-ended questions taken from science exam and
quiz sources. However, these are created by converting questions that were
originally multiple-choice, thus not covering certain types of typical
freeform questions (e.g. those that require mathematical proofs, or long
explanations).

Different from these works, SciEx is created from university computer science
exams, thus naturally providing diversity in the types of questions as well as
having freeform format.

##### Freeform Answer Evaluation

Compared to benchmarks with multiple-choice questions, evaluating LLMs’
performance on freeform questions is not straightforward. Similar to
evaluation conditions in tasks such as machine translation or summarization,
there are multiple correct answers, or multiple ways to express a correct
answer for a single input. Therefore, it is insufficient to evaluate a model’s
output by comparing it to a gold standard answer. Ideally, in these cases, we
can evaluate by human judgment. For example, the ARC-DA benchmark
Bhakthavatsalam et al. (2021) uses a crowdscoring pipeline for evaluation.
Chatbot Arena Zheng et al. (2024) also uses crowdsourcing, where the users
vote between pairs of LLM output. However, human evaluation is inherently non-
scalable. Therefore, previous works have used automated metrics. Some
traditional metrics such as BLEU Papineni et al. (2002) or ROUGE Lin (2004)
compare the model’s output to some gold-standard answer on the surface level,
i.e., word matching. More advanced metrics, such as s BERTScore Zhang* et al.
(2020), BLEURT Sellam et al. (2020), and BARTScore Yuan et al. (2021), are
model-based, thus being able to evaluate answers on the semantic level.

One recent approach is to use LLMs for evaluation, termed “LLM-as-a-judge”.
Liu et al. (2023); Chiang and Lee (2023); Zheng et al. (2024) find that,
although still prone to biases, LLM-as-a-judge for textual modality has high
agreement with human scoring when a strong judge LLM is used. However, when
including images, Chen et al. (2024) find that the performance of LLM-as-a-
judge is no longer as well correlated to human judgment. Nevertheless, LLM-as-
a-judge is a promising way to perform scalable evaluation.

In our work, we make use of LLM-as-a-judge for automatic grading of LLM
answers on SciEx exams, and find that they have good correlation to human
expert grading on both text-only and image-related questions.

##  3 The SciEx Benchmark

The components of SciEx are as follows.

##### Univeristy Exams

SciEx contains university computer science exams in a unified JSON format. The
exams are taken from the following computer science courses at the Karlsruhe
Institute of Technology from the 2022/2023/2024 semesters:

  * •

Natural Language Processing (NLP)

  * •

Advanced Artificial Intelligence (AI2)

  * •

Deep Learning and Neural Networks (DLNN)

  * •

Deep Learning for Computer Vision (DL4CV2)

  * •

Human–Computer Interaction (HCI)

  * •

Databases (DBS) for the years 2022 and 2023

  * •

Computer Graphics (CG)

  * •

Theoretical Foundations of Computer Science (TGI)

  * •

Algorithms (ALGO)

Descriptions of the exams are in Appendix A. In total, SciEx contains 10
exams, among which 5 exams are in English and 7 exams are in German (some
exams are provided in both languages).

There are in total 154 unique questions. Each question is annotated with (1)
the maximum points that can be achieved and (2) a difficulty level among Easy,
Medium, Hard. Most questions are provided with gold reference answer and
average student performance. The detailed per-question statistics are shown in
Table 1.

| Question count
---|---
Total | 154
English / German* | 95 / 97
Text-only / Image-related | 121 / 33
Easy / Medium / Hard | 51 / 71 / 32
With / Without reference | 120 / 34
With / Without student average | 117 / 37
*: Some questions are provided bilingually.
Table 1: Question-level statistics for SciEx.

##### LLM-Generated Answers

SciEx contains answers produced by 7 LLMs on the exam questions. The details
of the LLMs are shown in Table 2. In Table 2, only Llama3 was not used to
solve the exam, since it was released at a later point of conducting this
paper. In total, we obtained 1120 question-answer pairs.

| Full name | # Params | Quant. | Handle Image
---|---|---|---|---
Proprietary |  |  |  |
Claude | Claude-3-opus-20240229 | - | - | yes
GPT-4v | gpt-4-vision-preview | - | - | yes
GPT-3.5 | gpt-3.5-turbo-0125 | - | - | no
Open source |  |  |  |
Llama3 | MaziyarPanahi/Meta-Llama-3-70B-Instruct-GGUF | 70B | 4 bit | no
Mixtral | Mistralai/Mixtral-8x7B-Instruct-v0.1 | 8x7B | 5 bit | no
Qwen | Qwen/Qwen-72B | 72B | 2 bit | no
Mistral | Mistralai/Mistral-7B-Instruct-v0.2 | 7B | - | no
Llava | Llava-hf/Llava-v1.6-Mistral-7b-hf | 7B | - | yes
Table 2: Details of the LLMs in consideration.

##### Expert Grading and Automatic Grading

Each question-answer pair is assigned a score by an expert. In order to guide
future work to evaluate new LLMs on SciEx without relying on human expert
grading, we also provide automatic grading generated by Mixtral, Llama3 and
GPT4V.

###  3.1 Data Creation

The data creation process is described as follows.

##### Exam Collection

We collect university exams from different courses. We additionally ask the
lecturers to provide us with the reference answers, the difficulty level of
each question, and the average student grades on each question.

##### Exam Formatting

We convert every exam into a unified JSON format. Each exam includes a list of
questions, where each question includes an index, its content, and potentially
path to any related images. An example is shown in Appendix B.

##### LLM-Generated Answers

We pass the exams to the LLMs listed in Table 2 (except Llama3 due to later
release), one question at a time. Questions that contain images are handled
differently depending on the LLM. For the text-only LLMs, we exclude the
images and only pass the question text to the models. For Llava, since it is
trained to handle only 1 image at a time, we concatenate the images into one,
with blank padding around the images as separators before feeding it to the
model. Claude and GPT-4V can take multiple images, however, there is no pre-
defined way of referencing the image within the text. In our work, we
reference the image by mentioning the image caption within the question text,
and add the text caption to the image.

Since the considered LLMs can only output text, for questions asking to draw
on images, we ask the LLMs to describe in text what should be drawn.

##### Expert Grading

We then give the LLM answers back to the lecturers, who proceed with grading
the LLM output the same way they would grade student answers. We anonymized
the LLMs’ names in order to avoid bias during exam grading. We also build a
user interface for collecting the grades (see Appendix C for more details).

With expert grading, the evaluation of the LLM output is highly reliable. Most
importantly, the expert graders are generally the ones who designed the exam
questions. We additionally ask the expert graders to provide their comments on
the LLM output to further understand LLMs’ behaviors when solving the exams.

###  3.2 Automatic Grading

In addition to expert grading, we also provide automatic grading using LLM-as-
a-judge, so that we can evaluate future LLMs on SciEx. We use the stronger
models, i.e., Mixtral, Llama3 and GPT-4V, to conduct the grading. Given a
tuple containing question, answer, and maximum score, we ask the LLMs to
output a single score between 0 and the maximum. We include reference answers
to the grading prompt. We ask the LLMs to provide a chain-of-thought reasoning
Wei et al. (2022) before giving the grade. We also include examples for
grading in the prompt, so-called few-shot judge Zheng et al. (2024). Each
example is a tuple consisting of a question, an answer, and the grade. We try
out different settings to select the examples:

  * •

Same question: select examples from the same question but with an answer-grade
pair for a different LLM examinee. This mimics the real-life scenario where we
use the expert resource to grade some answers of the same exam, then use it to
guide the LLM graders.

  * •

Same exam: select examples from a different question but the same exam. Here
the examples are in the same domain as the question-answer pair in
consideration. This mimics the real-life scenario where, e.g., we have expert
grading on exam of the same course from previous years to guide LLM graders.

  * •

Different exam: select examples from a different question and a different
exam. This mimics the real-life scenario where, e.g., we have expert grading
for an exam of another course to guide the LLM graders.

Intuitively, the example-selection settings above have decreasing levels of
relevance to the actual grading query, but increasing easiness to collect.

##  4 Experiments

In this section, we describe our experiments and results. For prompting the
proprietary LLMs, we use their APIs, namely
OpenAI222https://platform.openai.com/ and
Anthropic333https://console.anthropic.com/. For the open-source models, we
obtain model checkpoints from the Huggingface444https://huggingface.co/ model
hub. We perform inference with the LLMs using
llama.cpp555https://github.com/ggerganov/llama.cpp with the default sampling
strategy. The experiments with open-sourced models are conducted on an NVIDIA
RTX A6000 GPU with 48GB VRAM.

For our analysis, we consider the exam-level and question-level grades. An
exam-level grade is the sum of the grades of all questions in the exam.

###  4.1 Quantiative Analysis

We analyze the performance of the LLMs on SciEx with expert grading. For both
exam level and question level, we normalize the grade to be between $0$ and
$100\%$, since they have different scales. We also report on the German grade
scale, where the grades range from $1.0$ to $5.0$, where $1.0$ is the highest
grade and $4.0$ is the passing threshold. We compare the performance of the
LLMs to students from different aspects: language, difficulty level, and
modality, i.e., questions with or without images.

####  4.1.1 General Observations

##### SciEx is Challenging

The performance of the LLMs on SciEx provided by expert grading is shown in
Table 3. The bigger-sized LLMs (Claude, GPT-4V, GPT-3.5, Mixtral and Qwen) can
achieve exam passing grades (i.e., grades that are better than $4.0$ in the
German scale). However, the best-performing model (Claude) only achieves
$59.4\%$ of the maximum points, which is far from perfect.

Compared to the student average, most LLMs have worse performance. Only the
strongest proprietary LLMs, i.e., Claude and GPT-4V, can achieve grades that
are better than the students’.

| Grade (%) | German Scale
---|---|---
Proprietary |  |
Claude | 59.4 | 2.4
GPT-4V | 58.2 | 2.5
GPT-3.5 | 32.8 | 3.9
Open source |  |
Mixtral | 41.1 | 3.5
Qwen | 35.4 | 3.7
Mistral | 25.9 | 4.2
Llava | 21.5 | 4.3
Student avg. | 45.3 | 3.1
Table 3: Average performance of LLMs, exam level.

##### SciEx Versus Other Benchmarks

The ranking of the LLMs on SciEx in Table 3 generally agrees with other public
benchmarks. However, SciEx seems to be more challenging. For example, the best
LLM accuracy achieved on MMLU’s various tasks is $88.8\%$. The best accuracy
achieved on M3Exam multiple choice questions is $72.92\%$. Although these
scores are not directly comparable, it indicates that SciEx provides a more
challenging test set for future LLMs.

####  4.1.2 Influential Factors

##### Difficulty Levels

Figure 1(a) shows the influence of the difficulty level on the examinee
grades. As can be seen, the student performance aligns with the difficulty
level of the questions: they perform better on easier questions. Some weaker
LLMs, e.g., Mixtral, Qwen, GPT-3.5, Llava, align with the students. However,
the stronger LLMs, i.e., Claude and GPT-4V, perform better on harder
questions. This is an indication that, difficulty levels from human
perspective do not always align with LLMs’ perspective. This is also confirmed
by looking at the correlations between the LLM grades and the student grades.
These correlations are between 0.4 to 0.6, indicating that LLM grades and the
student grades are not highly correlated.

In Figure 1(b), we plot the difference between LLM scores and student scores.
The stronger LLMs, i.e., Claude and GPT-4V, outperform the students the most
on hard questions. Weaker LLMs, on the other hand, generally fall behind
students the most on hard questions. A potential reason is that math-type easy
questions are hard for the LLMs, while long-text hard questions are easy for
them.

Looking at each difficulty level independently, we observe that the ranking of
the LLMs changes across different levels. This aligns with the findings made
by Li et al. (2024), where they show that the LLM rankings change on a subset
of evaluation prompts that are artificially labeled as hard.

(a) LLMs’ and students’ scores.

(b) Difference between LLM scores and student scores.

Figure 1: Question-level scores grouped by difficulty.

##### Text-only versus Image-related Questions

Figure 2 shows the influence of images on the difference between LLM and
student scores. Trivially, the LLMs that cannot handle images perform poorly
on the image-related questions. The strong, multi-modal LLMs, i.e., Claude and
GPT4, outperform the students on both image-related questions and text-only
questions, but the performance gap is still larger for text-only questions.
Llava, although can handle images, still falls behind student performance by a
large margin on image-related questions. This shows that LLMs’ image-handling
capability is still not as advanced as for text.

Figure 2: Difference between LLM scores and student scores, question-level,
grouped by with/without images. Only Claude, GPT-4V and Llava can handle
images.

##### Language

Figure 3 shows the influence of languages on the difference between LLM scores
and student scores. When the questions are in English, all LLMs, except for
GPT-3.5, outperform the student average. However, for German, either the LLMs
outperform students by a smaller gap, or fall behind student performance. It
can be concluded that LLMs are still superior in English than other languages
like German, although German can be considered a high-resource language.

Figure 3: Difference between LLM scores and student scores, question-level,
grouped by languages.

Since some models are not made to deal with images, or with languages other
than English, we additionally analyze LLMs’ performance on text-only and
English-only questions. On this subset of questions, the grades obtained by
the models are generally better, and more models would outperform the student
average. More details can be found in Appendix D.

###  4.2 Qualitative Analysis

In this section, we summarize the observations made by the graders while
grading the LLMs.

##### General Behaviours

The graders observed some common behaviors made by the LLMs. Some solutions of
the LLMs were good language-wise but low-quality content-wise. For students,
good language usually correlates strongly with good content. The LLMs tend to
output lengthy answers, since, unlike the students, they do not have a time
constraint. Some LLMs even ignore when the question specifies that they should
”answer briefly“. There are also some failure cases: (1) Claude refuses to
answer the question with "I apologize, but I do not feel comfortable providing
answers related to …" or (2) some LLMs get stuck in decoding loops. Sometimes,
instead of answering the question, LLMs give some text that is (or seems)
related to the task; rephrase the task; or describe how a task of this nature
may be approached in general.

##### Knowledge-type Questions

On some exams such as AI2, DL4CV2, DLNN, CG, questions which students can
answer by learning the lecture content by heart are quite easy for the LLM.
For the DL4CV2 exam, very specific questions about neural network
architectures which are covered in our lecture seem to be quite common
knowledge in the LLMs, which might be due to those papers being included in
the training data. However, for other exams such as HCI, the models lacked
specific course context, which was important for answering many theoretical
and open-ended questions.

##### Math-related Ability

The LLMs tend to fail on the math-related questions, even the basic ones. For
example, they miscount the number of words in a piece of text, or have trouble
comparing numbers. For questions that require writing mathematical proof in
the TGI exam, all LLMs except for GPT-4V and Claude failed. For GPT-4V and
Claude, they are able to pass the TGI exam. Their mistakes are more in line
with those that students would make. That is, they are often not successful
when making actual proof, and the points where the proof breaks sometimes are
the same as the students. Even the better models handle simple geometry
questions poorly and/or struggle to follow instructions of a simple algorithm.

|  | without ref | with ref
---|---|---|---
|  | same question | same exam | diff exam | same question | same exam | diff exam
Text-only questions
Mixtral | 0 shot | 0.232 |  |  | 0.311 |  |
| 1 shot | 0.352 | 0.377 | 0.364 | 0.395 | 0.333 | 0.275
| 2 shot | 0.395 | 0.299 | 0.316 | 0.398 | 0.271 | 0.255
Llama3 | 0 shot | 0.452 |  |  | 0.603 |  |
| 1 shot | 0.573 | 0.547 | 0.500 | 0.672 | 0.581 | 0.645
| 2 shot | 0.598 | 0.522 | 0.546 | 0.644 | 0.596 | 0.575
GPT-4V | 0 shot | 0.607 |  |  | 0.696 |  |
| 1 shot | 0.605 | 0.679 | 0.616 | 0.653 | 0.693 | 0.701
| 2 shot | 0.672 | 0.648 | 0.674 | 0.717 | 0.727 | 0.678
Image-related questions
GPT-4V | 0 shot | 0.677 |  |  | 0.539 |  |
| 1 shot | 0.640 | 0.661 | 0.611 | 0.642 | 0.539 | 0.749
| 2 shot | 0.613 | 0.632 | 0.673 | 0.712 | 0.465 | 0.696
Table 4: LLM grading’s Pearson correlation to expert grading on the question
level. Note that there are only a single scores for zero-shot, since they do
not have different shot settings.

##### Reasoning Ability

The LLMs do not perform well on questions that require deep thinking and
reasoning. For questions of the type “is this statement true or false; reason
for your solution”, the LLMs often said “true” and then just repeated the
statement or reasoned for the opposite of their claim. This is a similar
behavior often seen in students. Sometimes they make self-contradicting
arguments: making a statement and then providing arguments for the other side.

##### Image Handling

GPT-4V, Claude and Llava can handle images. However, only GPT-4V and Claude
have reasonable performance. When the question is about drawing on top of the
figures, sometimes the LLMs successfully describe in words what needs to be
drawn, but occasionally they just hallucinate a non-existing figure file path.

###  4.3 Automatic Grading

In this section, we evaluate the performance of LLM-as-a-judge approach to
automatic grading. We use the expert grades as the gold standard to evaluate
automatic graders. We use Pearson correlation on the normalized scores as our
metric. Since the LLMs are asked to provide the scores on the same scale as
the expert scores, we also provide the Root Mean Squared Error (RMSE) on the
originally-scaled scores as a secondary metric. Note that RMSE would correctly
put more weight on the questions that have more points, however, it is not as
easily interpretable as the Pearson correlation. Therefore, we only report
RMSE in Appendix E.2. The main results are discussed as follows.

####  4.3.1 General

##### LLMs Perform Well as Graders

On the exam level, LLM-as-a-judge performs well for automatic grading. The
best Pearson correlation to expert grading on the exam level, at 0.948, is
achieved by GPT-4V. The open-source Llama3 achieves 0.883 Pearson correlation
to expert grading. These high correlations indicate that, although being far
from perfect in solving SciEx exams (discussed in Section 4.1.1), the stronger
LLMs are quite reliable for grading the exams. This is useful since we would
have to rely less on expert grading to evaluate newly-developed LLMs’
performance on SciEx. The details of graders’ performance under different
settings on the exam level is in Appendix E.1.

On the sentence level, the performance of LLM-as-a-judge is shown in Table 4.
The highest Pearson correlation to expert grading achieved by the LLMs is now
around 0.7, which is lower than on the exam level, but still quite high.
Surprisingly, the performance of GPT-4V on grading image-related questions is
quite comparable to grading text-only questions. This contradicts to the
finding made by Chen et al. (2024). This could potentially be due to the small
number of image-related questions in SciEx, thus the results might not be
generalizable.

##### Few-shot and References Help

The performance of the graders on the question level is shown in Table 4. We
observe that adding more examples (shots) and adding the reference answers in
the prompt generally increases the performance of the LLM graders. GPT-4V is
the strongest grader, followed by Llama3 and Mixtral. This shows that
proprietary LLMs are still stronger as judge, aligning with previous studies
Zheng et al. (2024).

####  4.3.2 Grader-specific Behaviours

##### Mixtral Grader Tends to Give Full Points

As can be seen from Table 4, Mixtral has the worst performance on grading the
exams. We observe that Mixtral tends to give full points to the answers.
Without reference and without examples (0-shot), the portion of answers where
Mixtral outputs full points is 67.6%, significantly higher than Llama3 and
GPT-4V, at 19.1% and 15.1%, respectively. As a result, Mixtral’s precision on
giving full points, at 0.181, is much lower than Llama3 and GPT-4V, at 0.380
and 0.527 respectively. As we add more examples and/or add the reference
answer to the prompt, the problem is lessened. More details can be found in
Appendix E.3.

##### Mixtral and GPT-4V Copy Grade of Example

For Mixtral and GPT-4V graders, when having one example (shot) from the same
question in the prompt without reference, the performance is worse than having
the example from the same exam or from a different exam. We hypothesize that
this is due to these graders tend to copy the grades of the examples when
having a chunk of duplicated text (i.e., the question description) in the
example. This is verified when looking at the statistics: Mixtral and GPT-4V
copy the grade of the example 25% of the time, whereas Llama3 does it 13% of
the time. As a result, Llama3 can best make use of examples from the same
question. The problem is reduced when having more than 1 shot or when the
reference answer is included.

####  4.3.3 Influential Factors

##### Different Examinees

As can be seen in Table 5, GPT-4V grader has better performance than others,
but is more inconsistent: it does worse on grading some LLMs, especially
Claude. This is potentially due to Claude being a better examinee than GPT-4V
itself, as shown in Section 4.1.1. When using the scores from GPT-4V grader to
rank the LLMs, we find that, without reference answer, GPT-4V always ranks
itself higher than Claude. This emphasizes the importance of reference answers
for grading, especially when the grader is weaker than the examinee.

| Graders
---|---
| Mixtral | Llama3 | GPT-4V
Claude | 0.304 | 0.460 | 0.482
GPT-4V | 0.353 | 0.528 | 0.612
Mixtral | 0.251 | 0.472 | 0.564
Qwen | 0.351 | 0.556 | 0.736
GPT-3.5 | 0.333 | 0.522 | 0.697
Mistral | 0.291 | 0.467 | 0.601
Llava | 0.387 | 0.716 | 0.812
Table 5: Graders performance on different examinees.

##### Difficulty Levels

Looking at Table 6, the weaker graders, i.e., Mixtral and Llama3, perform
better on grading easier questions. In contrast, GPT-4V performs better in
grading harder questions.

| Graders
---|---
| Mixtral | Llama3 | GPT-4V
Easy | 0.374 | 0.602 | 0.628
Medium | 0.293 | 0.524 | 0.690
Hard | 0.224 | 0.496 | 0.732
Table 6: Graders performance on difficulty levels.

##  5 Conclusion

In this paper, we proposed SciEx - a benchmark consisting of scientific
university exams, along with expert grading and automatic grading, to evaluate
the abilities of LLMs on science topics. SciEx is multilingual, multi-modal,
and contains a variety of free-form questions. Our experiments show that SciEx
is still quite challenging for current LLMs, where the best LLM can only
achieve $59.4\%$ of the exam score on average. Despite that, the LLMs perform
well as graders, achieving 0.948 Pearson correlation to the expert grades.
This is a promising observation, since we can use strong LLMs for automatic
grading of new LLM examinees on SciEx, rather than relying on expert grading.
We encourage the research community as well as LLM developers and users to
make use of SciEx for evaluating LLMs’ scientific capabilities.

## Limitations

There are certain biases that can occur for SciEx. Firstly, the LLMs do not
have time pressure. Therefore, they can output longer answers, which helps
them get better grades, as there is a higher likelihood that something will be
correct. Secondly, the grading process can not be fully anonymized. It is not
easy to mix the LLM answers with student answers for the lecturer to grade,
since student answers are usually handwritten. Therefore, the lecturers know
when they are grading an LLM, thus can bias the score they give. Thirdly, the
comparison between the LLMs and the students might be unfair, since the
students studied the centralized course material specifically for the exams,
while this is not the case for the LLMs. Lastly, due to the reliance on expert
resources, the size of SciEx is quite small compared to other scientific
benchmarks.

## Ethics

Our work makes use of student statistics to compare against LLMs’ performance.
However, we only use the average of the student grades, without disclosing any
individual student’s information. The student answers are never directly used,
as we only ask for the average graders from the lecturers.

For automatic grading, regardless of the high correlation to expert grading,
they can still be imperfect. We are not suggesting to use LLMs to evaluate
students, but to evaluate new models coming out when it is not possible to do
human evaluation.

## Acknowledgments

## References

  * Abbasian et al. (2023) Mahyar Abbasian, Iman Azimi, Amir M Rahmani, and Ramesh Jain. 2023.  Conversational health agents: A personalized llm-powered agent framework.  _arXiv preprint arXiv:2310.02374_.
  * Bhakthavatsalam et al. (2021) Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. 2021.  Think you have solved direct-answer question answering? try arc-da, the direct-answer ai2 reasoning challenge.  _arXiv preprint arXiv:2102.03315_.
  * Chen et al. (2024) Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Pan Zhou, Yao Wan, and Lichao Sun. 2024.  Mllm-as-a-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark.  _arXiv preprint arXiv:2402.04788_.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021.  A dataset of information-seeking questions and answers anchored in research papers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4599–4610, Online. Association for Computational Linguistics.
  * Gilson et al. (2022) Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, R Andrew Taylor, and David Chartash. 2022.  How does chatgpt perform on the medical licensing exams? the implications of large language models for medical education and knowledge assessment.  _MedRxiv_ , pages 2022–12.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Li et al. (2024) Tianle Li, Wei-Lin Chiang, and Lisa Dunlap. 2024.  Introducing hard prompts category in chatbot arena.  Published on LMSYS.
  * Liao et al. (2023) Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023.  Proactive conversational agents in the post-chatgpt world.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’23, page 3452–3455, New York, NY, USA. Association for Computing Machinery.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: NLG evaluation using gpt-4 with better human alignment.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2511–2522, Singapore. Association for Computational Linguistics.
  * Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Rajkumar et al. (2022) Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022.  Evaluating the text-to-sql capabilities of large language models.  _arXiv preprint arXiv:2204.00498_.
  * Schubert et al. (2023) Marc Cicero Schubert, Wolfgang Wick, and Varun Venkataramani. 2023.  Performance of large language models on a neurology board–style examination.  _JAMA network open_ , 6(12):e2346721–e2346721.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017.  Crowdsourcing multiple choice science questions.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27263–27277. Curran Associates, Inc.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2024) Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2024.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36.

##  Appendix A Exam Description

The overall description of each exam in SciEx is as follows:

  1. 1.

Natural Language Processing (NLP): exam contains questions about word and
sequence representation, language modeling, and pretrained models.

  2. 2.

Advanced Artificial Intelligence (AI2): exam contains questions about natural
language processing, signal processing, automatic speech recognition and
cognitive robotics.

  3. 3.

Deep Learning and Neural Networks (DLNN): exam contains questions about neural
network fundamentals, in-depth questions about multihead self attention and
calculation questions on backpropagation.

  4. 4.

Deep Learning for Computer Vision (DL4CV2): exam contains questions about
semi-supervised learning, weakly supervised learning, multi-modal text-image
models, continual learning, representation learning, interactive segmentation,
transfer learning and generative models from recent literature.

  5. 5.

Human–Computer Interaction (HCI): exam encompasses fundamental HCI subjects
like observational studies, human perception and information processing, user
studies, and system design and design analysis. It requires students to
utilize theoretical knowledge and perform brief analyses based on the given
context.

  6. 6/7.

Databases (DBS): 2 exams from 2022 and 2023, containing questions about ER
(Entity Relationship) modeling, SQL writing and comprehension, relational
algebra, and transaction management.

  7. 8.

Computer Graphics (CG): exam contains questions about color and perception,
raytracing, shading, data structures, transformations, textures, OpenGl,
blending, shaders, procedural modeling, and bezier curves.

  8. 9.

Theoretical Foundations of Computer Science (TGI): exam contains questions
about finite automata, regular languages, push down automata, grammars
(Chomsky hierarchy), Turing machine, formal languages, NP-completeness,
approximation algorithms, decidability. Most questions require writing
mathematical proofs.

  9. 10.

Algorithms (ALGO): exam contains questions on writing proofs (correctness of
algorithms, asymptotic run time analysis), knowing basic algorithms and data
structures, designing simple new algorithms, selecting the right data
structure or algorithm for a given task at hand.

##  Appendix B Exam Formatting

Originally, exams were in different formats, depending on their creator. We
convert the exams into the JSON format, with file paths to images if any. An
example is shown in Figure 4.

(a)

(b)

Figure 4: Exam question before and after being converted to JSON format.

##  Appendix C User Interface for Expert Grading

Figure 5: Open page of the grading UI. Figure 6: Grading UI showing the
question, the answer and textbox to input the grade. Figure 7: Grading UI
where grader can fill in additional information about the question.

We instructed the expert grader to use our user interface (UI) for grading.
Figure 5 shows the open page of the UI, where the grader can choose their exam
and enter their password. Figure 6 shows the page for the grading, where the
expert is shown with the question, the LLM answer to the question, and a text
box to enter the grade. The expert can choose the examinee to grade from the
dropdown on the left-hand side. Figure 7 shows the page to enter additional
information about the exam questions, including the maximal achievable score,
average student performance, gold answer, and difficulty level.

Once the data is collected, we also ask the experts and have their consent to
make the data public.

##  Appendix D Performance on Text-only, English-only Questions

The performance of the LLMs on text-only, English-only questions is shown in
Table 7. Qn this subset of questions, besides GPT-4V and Claude, we can see
that Mixtral and Qwen also have better performance than the student average.

| Grade (%) | German Scale
---|---|---
Proprietary |  |
GPT-4V | 70.8 | 1.4
Claude | 69.2 | 1.6
GPT-3.5 | 47.8 | 2.9
Open source |  |
Mixtral | 61.2 | 2.0
Qwen | 56.8 | 2.4
Mistral | 48.0 | 3.2
Llava | 42.4 | 3.5
Student avg. | 56.5 | 2.4
Table 7: Average performance of the LLMs on the exam level, provided by expert
grading, text-only and English-only questions.

##  Appendix E Grader Performance

###  E.1 Pearson Correlation on Exam Level

The performance of LLM-as-a-judge for automatic grading on the exam level is
shown in Table 8. Note that Mixtral and Llava3 graders have disadvantage since
they cannot take image input for image-related questions.

|  | without ref | with ref
---|---|---|---
|  | same question | same exam | diff exam | same question | same exam | diff exam
Mixtral | 0 shot | 0.404 |  |  | 0.445 |  |
| 1 shot | 0.542 | 0.549 | 0.619 | 0.565 | 0.390 | 0.344
| 2 shot | 0.564 | 0.505 | 0.620 | 0.500 | 0.466 | 0.463
Llama3 | 0 shot | 0.649 |  |  | 0.677 |  |
| 1 shot | 0.812 | 0.731 | 0.706 | 0.883 | 0.770 | 0.772
| 2 shot | 0.771 | 0.729 | 0.768 | 0.788 | 0.738 | 0.785
GPT-4V | 0 shot | 0.911 |  |  | 0.938 |  |
| 1 shot | 0.886 | 0.906 | 0.893 | 0.902 | 0.904 | 0.948
| 2 shot | 0.921 | 0.910 | 0.896 | 0.917 | 0.888 | 0.934
Table 8: LLM graders’ Pearson correlation to expert graders on exam level,
scores normalized. Note that there are only a single scores for zero-shot,
since they do not have different shot settings.

###  E.2 RMSE on Question-level

Since the LLM graders are asked to output the scores in the original scale,
RMSE would be the most informative metric, since it also reflects the
importance of the questions that have higher maximum scores. The LLM graders’
performance in RMSE is shown in Table 9.

|  | without ref | with ref
---|---|---|---
|  | same question | same exam | diff exam | same question | same exam | diff exam
Text-only questions
Mixtral | 0 shot | 3.25 |  |  | 2.96 |  |
| 1 shot | 2.68 | 2.90 | 2.83 | 2.54 | 2.65 | 2.79
| 2 shot | 2.69 | 2.83 | 2.86 | 2.51 | 2.46 | 2.70
Llama3 | 0 shot | 2.66 |  |  | 2.09 |  |
| 1 shot | 1.89 | 2.30 | 2.50 | 1.45 | 1.92 | 1.85
| 2 shot | 1.88 | 2.36 | 2.31 | 1.92 | 1.77 | 1.92
GPT-4V | 0 shot | 1.56 |  |  | 1.20 |  |
| 1 shot | 1.34 | 1.32 | 1.53 | 1.49 | 1.34 | 1.28
| 2 shot | 1.25 | 1.29 | 1.54 | 1.36 | 1.31 | 1.31
Image-related questions
GPT-4V | 0 shot | 0.30 |  |  | 0.28 |  |
| 1 shot | 0.22 | 0.24 | 0.29 | 0.19 | 0.26 | 0.17
| 2 shot | 0.23 | 0.28 | 0.30 | 0.18 | 0.24 | 0.19
Table 9: LLM grading’s RMSE compared to expert grading on the question level.
Note that there are only a single scores for zero-shot, since they do not have
different shot settings.

###  E.3 Performance on Giving Full Points

The performance of the LLM graders on assigning full points to the answers is
shown in Table 10.

|  |  | Max score
---
precision
| Max score
---
predicted (%)
|  | No ref | Ref | No ref | Ref
Mixtral | 0 | 0.181 | 0.196 | 67.0 | 37.2
| 1 | 0.251 | 0.325 | 46.3 | 26.6
| 2 | 0.258 | 0.256 | 45.3 | 37.1
Llama3 | 0 | 0.380 | 0.636 | 19.1 | 5.9
| 1 | 0.438 | 0.579 | 19.9 | 8.5
| 2 | 0.447 | 0.483 | 17.8 | 10.5
GPT-4V | 0 | 0.527 | 0.405 | 15.0 | 11.3
| 1 | 0.422 | 0.526 | 16.8 | 10.2
| 2 | 0.458 | 0.560 | 15.9 | 9.8
Table 10: Performance on giving full points