# Tracking Point Vortices and Circulations via Advected Passive Particles: an
Estimation Approach

Gil Marques12, Marco Martins Afonso1, Sílvio Gama1

###### Abstract

We present a novel method for estimating the circulations and positions of
point vortices using trajectory data of passive particles in the presence of
Gaussian noise. The method comprises two algorithms: the first one calculates
the vortex circulations, while the second one reconstructs the vortex
trajectories. This reconstruction is done thanks to a hierarchy of
optimization problems, involving the integration of systems of differential
equations, over time sub-intervals all with the same amplitude defined by the
autocorrelation function for the advected passive particles’ trajectories. Our
findings indicate that accurately tracking the position of vortices and
determining their circulations is achievable, even when passive particle
trajectories are affected by noise.

11footnotetext: Gil Marques, Marco Martins Afonso and Sílvio Gama are with the
Centro de Matemática da Universidade do Porto, Departamento de Matemática,
Faculdade de Ciências, Universidade do Porto, Rua do Campo Alegre s/n,
4169-007, Porto, Portugal 22footnotetext<EMAIL_ADDRESS>
#### Index Terms

– Fluid flow systems, Optimization algorithms, Numerical algorithms

##  I Introduction

The objective of this work is to showcase a way to find the strength (or
circulation) and trajectory of a system of point vortices by using the data of
trajectories of passive particles. This is within the scope of the problems
proposed by Protas [1], more specifically his Problem $4$, in which he
considers the scenario of reconstructing some state of a vortex dynamical
system by using incomplete or noisy data.

The inverse problem of this, i.e. tracking passive particles advected by point
vortices with known circulations is well documented in the literature. For
example, the work [2] uses Poincaré sections and canonical transformations to
investigate the Hamiltonian dynamics of passive markers in the flow created by
two point vortices. In [3], the authors use data from coherent Lagrangian
structures to determine when and where vortex formation occurs in flows around
a 2D panel, and in [4] the authors employ vortices to approximate the dynamics
of an incompressible flow.

The study of point vortices and passive particles is interesting for
understanding the global evolution of atmospheric and oceanic vortices [5, 6,
7].

Section II serves as an introduction to the formalism of point vortex
dynamics. In Section III we detail the problem we are tackling, presenting a
method to solve it in Section IV. We present the results obtained from testing
the method on simulated corrupted data in Section V, and finally in Section VI
we sum up some conclusions and ideas for future work.

##  II Point Vortices

In the vorticity formulation, the two-dimensional (2D) incompressible
Navier–Stokes equations can be written as

$\frac{\partial\omega}{\partial
t}+\left(\mathbf{u}\cdot\nabla\right)\omega=\nu\nabla^{2}\omega,$ (1)

where $\omega$ is the vorticity, $\mathbf{u}$ is the velocity field and $\nu$
is the kinematic viscosity. In an inviscid flow ($\nu=0$), equation (1), also
known as Euler’s equation, possesses the singular solutions

$\omega\left(x,y,t\right)=\sum_{v}\Gamma_{v}\,\delta\left(\left(x,y\right)-\left(x_{v}\left(t\right),y_{v}\left(t\right)\right)\right).$
(2)

The scalar $\Gamma_{v}$ denotes the circulation of the point vortex located in
$\left(x_{v}\left(t\right),y_{v}\left(t\right)\right)$ and can be thought of
as a measure of the rotation strength of the flow around the point vortex.

These singular solutions of the 2D incompressible Navier–Stokes equations [8,
9, 10, 11] were first studied by Helmholtz [12], and further explored by
Kelvin[13] and Kirchhoff[14]. They correspond to a scenario where the
vorticity of a flow is concentrated in some well-defined points in space and
enable a simpler description of the dynamics of such a system. Knowing the
position and strength of each vortex in the system suffices to obtain the full
velocity field at that instant, and thus knowing the dynamics of the vortices
themselves in some time interval is enough to characterize the velocity field
during that same time interval.

While a single point vortex, if left alone, will stay in the position
$\left(x_{0},y_{0}\right)$ forever and its circulation $\Gamma$ will induce a
time-independent velocity field in the plane, in a system of multiple point
vortices the dynamics become more complex.

Consider a 2D inviscid flow in the complex plane that is described by $N_{v}$
point vortices located in the positions
$z_{v}=x_{v}+iy_{v}\,(v=1,\dots,N_{v}\,;i^{2}=-1)$. The velocity field
generated by each of these vortices will affect the remaining $N_{v}-1$ point
vortices, making them move in the 2D plane, which causes the velocity field to
change in time. Assuming there is no other object or force interfering in the
flow, the motion of point vortices follows the differential equations

$\dot{z}_{v}^{*}=\frac{1}{2\pi i}\sum_{\begin{subarray}{c}s=1\\\ s\neq
v\end{subarray}}^{N_{v}}\frac{\Gamma_{s}}{z_{v}-z_{s}},\quad
v=1,\dots,N_{v}\,,$ (3)

where $\Gamma_{v}$ is the circulation of vortex $v$ and ∗ denotes the complex
conjugate.

However, when looking at real phenomena, it can be difficult to state
precisely where the vortices are located and what is their circulation. A test
particle, or a passive particle, is a particle that does not affect the flow
and thus does not act on the vortices. Thus, they can be considered as point
vortices with zero circulation. These passive particles, whose number we
assume to be $N_{p}\,,$ can be easier to trace and the trajectory
$z_{p}\left(\cdot\right)$ of any of them follows the equations of motion

$\dot{z}_{p}^{*}=\frac{1}{2\pi
i}\sum_{v=1}^{N_{v}}\frac{\Gamma_{v}}{z_{p}-z_{v}},\quad p=1,\dots,N_{p}\,.$
(4)

##  III Problem

Assuming we have a way to track individual passive particles in a flow (for
example, using drones, meteorological sensors, particle image/tracking
velocimetry techniques), consider the problem of finding the circulations and
positions of a system of $N_{v}$ point vortices, while only knowing the number
of vortices in the system and the trajectory of $N_{p}$ passive particles
advected by that system of vortices.

In fact, since the trajectories of passive particles are sampled, what we have
access to are discrete measurements of the particle trajectories. For
simplicity, let us assume that these measurements are taken on $N_{t}$ equally
spaced time instants $t_{k}=t_{0}+hk$ in the time interval
$\left[t_{0},t_{f}\right]$, where $h$ is the time increment.

Thus, one way to approach the problem of reconstituting the vortices
trajectories is to use the information on the particle trajectories to trace
back the vortices that originated them, by trying to reconstitute the measured
particle trajectories using the knowledge of the equations of motion for both
the particles and the vortices. This can be summed up in the following
problem, where $\widetilde{z}_{p}\left(t\right)$ is the position of particle
$p$ at time $t$ as obtained from solving (4):

$\left\\{\begin{aligned}
\text{Minimize}&\quad\sum_{k=0}^{N_{t}-1}\sum_{p=1}^{N_{p}}||\widetilde{z}_{p}\left(t_{k}\right)-z_{p}\left(t_{k}\right)||^{2}\\\
\text{subject to}&\\\ &\dot{\widetilde{z}}_{p}^{*}=\frac{1}{2\pi
i}\sum_{v=1}^{N_{v}}\frac{\Gamma_{v}}{\widetilde{z}_{p}-z_{v}},\quad
p=1,\dots,N_{p}\\\ &\dot{z}_{v}^{*}=\frac{1}{2\pi
i}\sum_{\begin{subarray}{c}s=1\\\ s\neq
v\end{subarray}}^{N_{v}}\frac{\Gamma_{s}}{z_{v}-z_{s}},\quad
v=1,\dots,N_{v}\\\
&\widetilde{z}_{p}\left(t_{0}\right)=z_{p}\left(t_{0}\right)\\\
&z_{v}\left(t_{0}\right)\quad\text{free}\\\
&\Gamma_{v}\qquad\\!\quad\text{free}\end{aligned}\right.,$ (5)

where $||\cdot||$ is the usual Euclidean norm.

As a consequence to solving this problem, we will obtain the information about
the vortices circulations and trajectories.

##  IV Method

Notice that first and foremost, since the particle trajectory data is prone to
unknown measurement errors, we actually do not measure the real
$z_{p}\left(t_{k}\right)$, but a quantity $z_{p}\left(t_{k}\right)+\Delta
z_{p}\left(t_{k}\right)$, where $\Delta z_{p}\left(t\right)$ is the error
associated with that measurement. By using a smoothing algorithm to diminish
the impact of measurement errors in the data, we can then obtain estimations
for the particle trajectories. From here onward, $z_{p}$ will be used to
denote the filtered data in order to not overload the notation.

From the filtered trajectories one can use a finite-differences method to
estimate the velocity of the passive particles $\dot{z}_{p}\left(t\right)$.
However, if velocity data are available, one can smooth such data directly
instead of estimating them from the filtered trajectory data, which is
something that would introduce further errors in the velocity estimates.

We can take advantage of the fact that $\Gamma_{v}$ is constant in time to
divide the problem into two sub-problems. First, we will find the circulations
of the vortices, and afterwards use them to find better estimates for the
vortices trajectories.

###  IV-A Estimating the circulations

For any given time $t$ for which we have data for, the unknowns in (4) are
$\Gamma_{v}$ and $z_{v}\left(t\right)$. These amount, in reality, to $3N_{v}$
unknowns and thus we need to know the trajectories of at least $3N_{v}/2$
particles (each brings two pieces of information) in order to build a system
of equations that is not underdetermined. After building such a system, one
can use a nonlinear solver method to find a solution of the system. This will
require an initial guess for both the circulations and positions of the
vortices that should be provided based on analysis of the data: either the
movement of the particles in the flow or actual visual imagery of the
environment should hint towards what range of values one can expect for these
quantities.

Doing so for time $t_{k}$ will return estimates for the circulations
$\Gamma_{v}$ and for $z_{v}\left(t_{k}\right)$ — the positions of the vortices
at time $t_{k}$. We will identify these estimates as
$\widetilde{\Gamma}_{v_{k}}$ and $\widetilde{z}_{v_{k}}$. The index $k$ on the
circulations is used just to identify that such a value was estimated using
the data at time $t_{k}\,,$ as the circulation is in fact independent of time.

One can then use $\widetilde{z}_{v_{k}}$ as an initial condition for (3) in
order to find an initial guess for $z_{v}\left(t_{k+1}\right)$ and build the
same nonlinear system of equations as before for time $t_{k+1}$, solve it and
obtain estimates for $z_{v}\left(t_{k+1}\right)$ and a new estimate for
$\Gamma_{v}$: $\widetilde{\Gamma}_{v_{k+1}}$. In order to explore the
parameter space further, one can update the initial guess for the circulations
as $\widetilde{\Gamma}_{v_{k}}\left(1+\delta_{v}\right)$, where $\delta_{v}$
is a uniform random variable that ranges from $-\epsilon$ to $\epsilon$
($\epsilon$ is some small positive number).

Repeating this procedure from $t_{0}$ to $t_{f}$ will result in a set of
values $\left\\{\widetilde{\Gamma}_{v_{k}}:k=1,\dots,N_{t}\right\\}$, all of
which are estimates for the vortex circulations. By doing an outlier analysis
one can remove any abnormal value and find estimates for the vortex
circulations using the average or median of the leftover values in order to
obtain $\widetilde{\Gamma}_{v}$, a more robust estimate for these quantities.
One can even repeat this whole procedure using these more robust estimates as
the initial guess in order to refine the results. Algorithm 1 sums up this
procedure.

Algorithm 1 Finding the circulations

1:Set a value for $\epsilon$

2:Estimate the velocities of passive particles

3:Provide an initial guess for the circulations and positions of the vortices
at $t=t_{0}$

4:$k\leftarrow 1$

5:while $k\leq N_{t}$ do

6: Use a nonlinear solver to find $\widetilde{\Gamma}_{v_{k}}$ and
$\widetilde{z}_{v_{k}}$, estimates for $\Gamma_{v}$ and
$z_{v}\left(t_{k}\right)$ with the appropriate initial conditions

7: Use $\widetilde{z}_{v_{k}}$ and (3) to find an estimate for
$z_{v}\left(t_{k+1}\right)$

8: Generate random numbers $\delta_{v}$ uniformly in the interval
$\left[-\epsilon,\epsilon\right]$

9: Use $\widetilde{\Gamma}_{v_{k}}\left(1+\delta_{v}\right)$ as an updated
initial guess for the circulations

10: $k\leftarrow k+1$

11:end while

12:Run an outlier analysis over the values $\widetilde{\Gamma}_{v_{k}}$

13:Obtain a more robust estimate of the circulations using the average or
median of the leftover $\widetilde{\Gamma}_{v_{k}}$ values

###  IV-B Estimating the vortex trajectories

Having obtained the circulations, we have now simplified the problem (5) as
$\Gamma_{v}$ is now fixed at $\widetilde{\Gamma}_{v}$.

It is known that, in general, vortex systems exhibit chaotic dynamics for
$N_{v}\geq 4$ [15, 8]. A practical consequence of this fact is that, when
solving equations such as (5) or (4), even a small error in the initial
condition can be highly amplified given enough time and the computed
trajectories can be completely different from the real ones. Our approach to
deal with this problem is to partition the trajectory and work on the
reconstitution of each part separately and sequentially. However, one needs to
first find a criteria for building such a partition which should take into
account what is known about the chaoticity of the system.

One way to measure the degree of chaos in such a system are Lyapunov
exponents[16, 17]. In particular, the largest Lyapunov exponent measures the
(exponential) rate at which two nearby trajectories separate from each other,
and thus its inverse gives us the time needed for two nearby trajectories to
separate for a factor of $e$. This also tells us that the system has some
degree of memory but, after a certain time, the evolution is mostly
uncorrelated with the states past a certain point in time. Lyapunov exponents
for the system of vortices and for passive particles with a chaotic movement
are different. Nevertheless, in a chaotic system, they seem to be within the
same order of magnitude [18]. Thus, since we have no information about the
vortices themselves, we need to use the passive particles to get an estimate
of such quantities. However, Lyapunov exponents computation is slow and heavy
and, in a scenario where one wants to follow vortices in real time, it is not
a quantity suitable to use. Autocorrelation functions, nonetheless, are fast
to compute and can be used to find the time after which the state of the
system has little to no memory of its initial state.

By computing autocorrelation functions for the passive particles’
trajectories, we can find the time $\tau$ it takes for the system to lose most
of its memory from the initial state. We do this by setting an arbitrary
cutoff $0<\alpha<1$ and computing $\tau:\forall
p\in\left\\{1,\dots,N_{p}\right\\},\,|\,\rho_{p}\left(t>\tau\right)|\leq\alpha\,,$
where $\rho_{p}$ denotes the autocorrelation function for the trajectory of
the particle $p$.

Thus, in order to reconstitute the full vortex trajectories we must
reconstitute $n=\Bigr{\lceil}\frac{t_{f}-t_{0}}{\tau}\Bigr{\rceil}$ partitions
of the form
$P_{j}=\left[t_{0}+j\tau,t_{0}+\left(j+1\right)\tau\right],\,j=0,1,\dots,n-2\,,$
and $P_{n-1}=\left[t_{0}+\left(n-1\right)\tau,t_{f}\right]$. Each of these
constitutes an optimization problem akin to (5) that can be written as

$\left\\{\begin{aligned}
&\text{Minimize}\quad\sum_{k=j\tau/h}^{\left(j+1\right)\tau/h}\sum_{p=1}^{N_{p}}||\widetilde{z}_{p}\left(t_{k}\right)-z_{p}\left(t_{k}\right)||^{2}\\\
&\text{subject to}\\\ &\dot{\widetilde{z}}_{p}^{*}=\frac{1}{2\pi
i}\sum_{v=1}^{N_{v}}\frac{\widetilde{\Gamma}_{v}}{\widetilde{z}_{p}-z_{v}},\quad
p=1,\dots,N_{p}\\\ &\dot{z}_{v}^{*}=\frac{1}{2\pi
i}\sum_{\begin{subarray}{c}s=1\\\ s\neq
v\end{subarray}}^{N_{v}}\frac{\widetilde{\Gamma}_{s}}{z_{v}-z_{s}},\quad
v=1,\dots,N_{v}\\\
&\widetilde{z}_{p}\left(t_{0}+j\tau\right)=z_{p}\left(t_{0}+j\tau\right)\\\
&z_{v}\left(t_{0}+j\tau\right)\quad\text{free}\\\ &t\in
P_{j}\end{aligned}\right.$ (6)

for $j=0,1,\dots,n-2\,,$ and

$\left\\{\begin{aligned}
&\text{Minimize}\,\sum_{k=\left(n-1\right)\tau/h}^{N_{t}-1}\sum_{p=1}^{N_{p}}||\widetilde{z}_{p}\left(t_{k}\right)-z_{p}\left(t_{k}\right)||^{2}\\\
&\text{subject to}\\\ &\dot{\widetilde{z}}_{p}^{*}=\frac{1}{2\pi
i}\sum_{v=1}^{N_{v}}\frac{\widetilde{\Gamma}_{v}}{\widetilde{z}_{p}-z_{v}},\quad
p=1,\dots,N_{p}\\\ &\dot{z}_{v}^{*}=\frac{1}{2\pi
i}\sum_{\begin{subarray}{c}s=1\\\ s\neq
v\end{subarray}}^{N_{v}}\frac{\widetilde{\Gamma}_{s}}{z_{v}-z_{s}},\quad
v=1,\dots,N_{v}\\\
&\widetilde{z}_{p}\left(t_{0}+\left(n-1\right)\tau\right)=z_{p}\left(t_{0}+\left(n-1\right)\tau\right)\\\
&z_{v}\left(t_{0}+\left(n-1\right)\tau\right)\quad\text{free}\\\ &t\in
P_{n-1}\end{aligned}\right.$ (7)

These $n$ problems can thus be solved sequentially by a nonlinear solver,
using $z_{v}\left(t_{0}+\left(j+1\right)\tau\right)$ obtained in the problem
associated to the interval $P_{j}$ as an initial guess for the problem
associated with the interval $P_{j+1}$. Algorithm 2 summarizes this full
process.

Algorithm 2 Finding the vortex trajectories

1:Set $\alpha$

2:$p\leftarrow 1$

3:while $p\leq N_{p}$ do

4: Compute $\rho_{p}\left(t\right)$, the autocorrelation function of
$z_{p}\left(t\right)$

5: Find $\tau_{p}:\,|\,\rho_{p}\left(t>\tau_{p}\right)|\leq\alpha$

6: $p\leftarrow p+1$

7:end while

8:$\tau\leftarrow\min_{1\leq p\leq N_{p}}\left\\{\tau_{p}\right\\}$

9:$n\leftarrow\Bigr{\lceil}\frac{t_{f}-t_{0}}{\tau}\Bigr{\rceil}$

10:$j\leftarrow 1$

11:while $j\leq n$ do

12: Use a nonlinear solver to solve problem (6) on the time interval $P_{j}$

13: $j\leftarrow j+1$

14:end while

15:Use a nonlinear solver to solve problem (7)

##  V Results

We simulated a system comprised of $N_{v}=4$ vortices with circulations
$\Gamma_{v}=v,\,v=1,2,3,4$ and $N_{p}=20$ passive particles. The vortices were
initially placed in the position $z_{1}\left(t=0\right)=2$,
$z_{2}\left(t=0\right)=-1-i$, $z_{3}\left(t=0\right)=\frac{1}{2}+\frac{i}{2}$,
$z_{4}\left(t=0\right)=-2+3i$ and the passive particles were placed in random
positions in the background between the vortices, and we let the system evolve
for a time period of $10^{5}$ units of time using a time step of $10^{-2}$.
The vortex circulations were all chosen to have the same sign in order to
confine the movements of the particles and vortices to a certain area, which
facilitates the simulation and analysis. If there are vortices with opposite
signs in the system, pairs of vortices can be formed and travel long distances
away from the remaining vortices. The simulation ran for a long time in order
to be possible for us to compute a good estimate of the maximum Lyapunov
exponent of the vortex system, for which we found $\sim 0.028$. However, since
the problem of recovering the vortex trajectories is computationally heavy, we
restrained to using a time interval of $10^{3}$ units of time for this
problem.

We then introduced Gaussian white noise
$W\sim\mathcal{N}\left(\mu=0,\sigma^{2}=0.01^{2}\right)$ in the particles’
trajectory data, in order to simulate what actual real data could look like.
After doing so, the data were too corrupted to be directly used, so we first
smoothed them using a Gaussian filter. We then used the smoothed trajectories
to compute particle velocities using a fourth-order finite-differences scheme
and ran Algorithm 1 with $\epsilon=0.1$. Integration was made using the
Runge-–Kutta-–Fehlberg method with a step size of $10^{-2}$, and the nonlinear
equations were solved by MATLABTM routine lsqcurvefit.

We were able to recover the vortices’ circulations with a relative error of
$\sim 0.1\%\,$ (if we use the uncorrupted data, the relative error decreases
to $\sim 10^{-7}$). The obtained values are summarized in Table I, as well as
the relative errors for each individual circulation.

Fig. 1 shows the boxplots for the values of $\widetilde{\Gamma}_{v_{k}}$
obtained by the algorithm. The values in the $y$ axis were shifted down by
$v-1$ for easier visualization purposes.

TABLE I: True values of the circulations and the values recovered by Algorithm 1. The circulations were recovered with a relative error of $\sim 0.1\%$. Vortex $v$ | $1$ | $2$ | $3$ | $4$
---|---|---|---|---
$z_{v}\left(0\right)$ | $2$ | $-1-i$ | $\frac{1}{2}+\frac{i}{2}$ | $-2+3i$
$\Gamma_{v}$ | $1$ | $2$ | $3$ | $4$
$\widetilde{\Gamma}_{v}$ | $0.99625$ | $2.00132$ | $2.99746$ | $4.00258$
$\frac{|\widetilde{\Gamma}_{v}-\Gamma_{v}|}{|\Gamma_{v}|}$ | $3.7\times 10^{-3}$ | $6.6\times 10^{-4}$ | $8.5\times 10^{-4}$ | $6.5\times 10^{-4}$
Figure 1: Boxplot for $\widetilde{\Gamma}_{v}-(v-1)$. The values are shifted
down by $v-1$ for easier visualization purposes. We can see that the
distribution of obtained values is centered around the true value of each
circulation.

Using the obtained values of $\widetilde{\Gamma}_{v}$, we then proceeded to
recover the vortex trajectories using Algorithm 2. We set $\alpha=0.2$,
obtaining $\tau=39.89\,.$ This is in accordance with the value of the Lyapunov
exponent we obtained in the initial simulation ($\sim 0.028$), since
$\tau^{-1}\approx 0.025\,,$ which is of the same order of magnitude as the
Lyapunov exponent. Once again, integration was made using the
Runge–-Kutta–-Fehlberg method with a step size of $10^{-2}$ and the nonlinear
problems were solved by MATLABTM using the routine fmincon with the sqp
algorithm. We were able to recover the trajectories of the vortex system with
a total relative error of $\sim 0.44\%$. (if we use the uncorrupted data, the
relative error decreases to $\sim 8\times 10^{-4}$ ).

Fig. 2 shows a portion of the recovered trajectories of the $4$ vortices with
different colored markers and the (smoothed) trajectory of a passive particle
in a full black line. The color-filled dots mark the initial point of each
trajectory.

Fig. 3 illustrates how the partitioning of the problem helps with controlling
the error of the recovered trajectories. Due to the chaoticity of the system,
errors can grow large and we can see that, in general, by integrating
partitions of the trajectory with a time length similar to the inverse of the
Lyapunov exponent of the vortex system will help to prevent exponential growth
of the errors.

Figure 2: A portion of the recovered trajectories of the $4$ vortices (blue
dots, pink triangles, red crosses and green asterisks identify vortices $1$,
$2$, $3$ and $4$, respectively) and an example of a smoothed trajectory for a
passive particle in a full black line. The color-filled dots mark the initial
point of each trajectory. Figure 3: Relative error for the position of the
vortex system at each time step. The circles identify the discontinuities that
arise from the partition of the problem into various sub-problems. In general
the relative error decreases every time we change from one partition to the
following one.

##  VI Conclusion

We presented the basic ideas on how to try to recover vortex dynamics from
data on the trajectories of advected passive particles. This is a problem
that, as far as we are aware, has not been addressed before and can have
relevant interest since it is easier to track the advected particles than
tracking and measuring the strength of vortices themselves.

The method presented is a first study and we believe it can be improved
largely. For example, using optimization techniques and algorithms
specifically tailored for the problem and its equations, instead of using
generic MATLABTM routines, will likely increase accuracy and speed up the
computations.

Nevertheless, from the results of this study we see that it is possible to
effectively recover the full vortex dynamics with a good degree of accuracy,
even in the presence of noisy data. This opens up the possibility of studying
more complex problems, for instance, predicting positions of vortices and/or
particles at future times, which enables the study of finding controls to move
passive particles to a desired position in a given time.

## Acknowledgements

This work was supported by (i) CMUP, member of LASI, which is financed by
national funds through FCT – Fundação para a Ciência e a Tecnologia, I.P.,
under the project with reference UIDB/00144/2020, and (ii) project SNAP
NORTE-01-0145-FEDER-000085, co-financed by the European Regional Development
Fund (ERDF) through the North Portugal Regional Operational Programme
(NORTE2020) under Portugal 2020 Partnership Agreement. GM thanks grant ref.
PD/BD/150537/2019 through FCT.

## References

  * [1] B. Protas, “Vortex dynamics models in flow control problems,” Nonlinearity, vol. 21, no. 9, pp. R203–R250, 2008.
  * [2] G. Boffetta, A. Celani, and P. Franzese, “Trapping of passive tracers in a point vortex system,” Journal of Physics A: Mathematical and General, vol. 29, no. 14, p. 3749, 1996.
  * [3] Y. Huang and M. A. Green, “Detection and tracking of vortex phenomena using Lagrangian coherent structures,” Experiments in Fluids, vol. 56, no. 7, p. 147, 2015.
  * [4] Z. Qian, Y. Qiu, and Y. Zhang, “Tracking the vortex motion by using Brownian fluid particles,” Physics of Fluids, vol. 33, no. 10, p. 105113, 2021.
  * [5] V. V. Meleshko, M. Y. Konstantinov, A. A. Gurzhi, and T. P. Konovaljuk, “Advection of a vortex pair atmosphere in a velocity field of point vortices,” Physics of Fluids A: Fluid Dynamics, vol. 4, no. 12, pp. 2779–2797, 1992.
  * [6] H. Aref and M. A. Stremler, “Point vortex models and the dynamics of strong vortices in the atmosphere and oceans,” in Fluid Mechanics and the Environment: Dynamical Approaches: A Collection of Research Papers Written in Commemoration of the 60th Birthday of Sidney Leibovich, pp. 1–17, Springer, 2001\.
  * [7] I. I. Mokhov, S. G. Chefranov, and A. G. Chefranov, “Point vortices dynamics on a rotating sphere and modeling of global atmospheric vortices interaction,” Physics of Fluids, vol. 32, no. 10, p. 106605, 2020.
  * [8] P. Newton, The N-Vortex Problem: Analytical Techniques.  Appl. Math. Sci., Springer New York, 2001.
  * [9] A. Chorin, Vorticity and Turbulence.  Appl. Math. Sci., Springer, 1994.
  * [10] G. K. Batchelor, An Introduction to Fluid Dynamics.  Cambridge Mathematical Library, Cambridge University Press, 2000.
  * [11] P. G. Saffman, Vortex dynamics.  Cambridge University Press, 1995.
  * [12] H. Helmholtz, “Über Integrale der hydrodynamischen Gleichungen, welche den Wirbelbewegungen entsprechen.,” vol. 1858, no. 55, pp. 25–55, 1858.
  * [13] W. Thomson (Lord Kelvin), “On vortex motion.,” Trans. R. Soc. Edin, vol. 25, p. 217–260, 1869.
  * [14] G. R. Kirchhoff, “Vorlesungenbër mathematische Physik.,” Mechanik, 1876\.
  * [15] A. Babiano, G. Boffetta, A. Provenzale, and A. Vulpiani, “Chaotic advection in point vortex models and two‐dimensional turbulence,” Phys. of Fluids, vol. 6, no. 7, pp. 2465–2474, 1994.
  * [16] D. Ruelle and S. Isola, Chaotic evolution and strange attractors, vol. 1.  Cambridge University Press, 1989.
  * [17] K. T. Alligood, T. D. Sauer, and J. A. Yorke, Chaos An Introduction to Dynamical Systems.  Springer-Verlag, New York, 1997.
  * [18] G. Marques, M. J. Rodrigues, and S. Gama, “Passive particle dynamics in viscous vortex flow,” in CONTROLO 2020 (J. A. Gonçalves, M. Braz-César, and J. P. Coelho, eds.), (Cham), pp. 352–362, Springer International Publishing, 2021.


tcb@breakable

#  1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge
Aggregators?

Yue Huang1111Equal contribution., Chenrui Fan2111Equal contribution., Yuan
Li3, Siyuan Wu4,
Tianyi Zhou2, Xiangliang Zhang1222Corresponding anthor., Lichao Sun5

1University of Notre Dame 2University of Maryland, College Park 3University of
Cambridge
4Huazhong University of Science and Technology 5Lehigh University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have garnered significant attention due to their
remarkable ability to process information across various languages. Despite
their capabilities, they exhibit inconsistencies in handling identical queries
in different languages, presenting challenges for further advancement. This
paper introduces a method to enhance the multilingual performance of LLMs by
aggregating knowledge from diverse languages. This approach incorporates a
low-resource knowledge detector specific to a language, a language selection
process, and mechanisms for answer replacement and integration. Our
experiments demonstrate notable performance improvements, particularly in
reducing language performance disparity. An ablation study confirms that each
component of our method significantly contributes to these enhancements. This
research highlights the inherent potential of LLMs to harmonize multilingual
capabilities and offers valuable insights for further exploration.

##  1 Introduction

Large Language Models (LLMs) are increasingly recognized for their impressive
capabilities in natural language processing (NLP). Employed across a variety
of domains such as the medical sector (Liu et al., 2023c; Zhang et al.,
2023a), software engineering (Qian et al., 2023), scientific research (Guo et
al., 2023; Li et al., 2024c), and LLM-based agents (Liu et al., 2023b; Guo et
al., 2024; Huang et al., 2024a; Chen et al., 2024a), LLMs have demonstrated
significant utility. Additionally, recent advancements in LLMs have expanded
research (Qin et al., 2024; Li et al., 2024a; Xu et al., 2024b; Chen et al.,
2024b), which focuses on enhancing their ability to process multiple languages
and thereby increasing their accessibility and relevance across diverse
linguistic demographics.

Despite these advancements, LLMs demonstrate inconsistencies when processing
queries in different languages with the same meaning Li et al. (2024d), as
evidenced by the results in Figure 1. This inconsistency not only diminishes
the efficacy and fairness of LLMs but also signals underlying knowledge
conflicts (Xu et al., 2024a) that prevent these models from achieving true
intelligence (Liu et al., 2023b; Huang et al., 2024a). Furthermore, such
inconsistency can erode trust in LLM applications, particularly when users
from varied linguistic backgrounds cannot equally benefit from the technology
(Li et al., 2023b).

Figure 1: The top is an example of distinct answers to the same questions in
different languages. The bottom is the GPT-4’s performance on 300 queries in
HalluEval Li et al. (2023a) of nine different languages.

To address the inconsistency problems in LLMs, we propose a novel method by
leveraging the intrinsic capabilities of LLMs through integrating knowledge
across different languages. Our approach begins with the development of a low-
resource knowledge detector. This detector assesses whether a user’s query
involves knowledge that is underrepresented in the specific language. When the
query does not feature low-resource knowledge, it is directly addressed by the
LLMs. In contrast, if low-resource knowledge is detected, the LLMs will be
required to select the most relevant target language to handle this specific
knowledge. Once the target language is selected, the query is translated into
this language, and the LLMs generate a response based on the translated query.
This response either replaces the original answer or is integrated with it.
Finally, the response is translated back to the original language of the query
and delivered to the user.

We conducted comprehensive experiments using six popular LLMs and five
bilingual datasets (specifically, English and Chinese) to evaluate the
effectiveness of our proposed method. The experimental results demonstrate
that our approach successfully integrates knowledge from different languages,
leading to the improvement in overall performance. Importantly, it
significantly reduces the performance disparities across languages, thereby
addressing the inconsistency issues inherent in LLMs and promoting fairness
for downstream applications. Additionally, our ablation study confirms that
both the low-resource knowledge detector and the language selection process
are crucial to the improvements observed. Overall, our contributions are as
follows:

  * •

We posed an important challenge on the inconsistency of LLMs in downstream
tasks, and the low-resource knowledge in a specific language can be brought
from another language.

  * •

Based on the observation, we propose a method that utilizes the LLMs’ internal
capability to enhance its performance on datasets in different datasets
through a low-resources knowledge detector, language selection process, and
answer replacement & integration.

  * •

We conduct extensive experiments on six popular LLMs and five bilingual
datasets. The results show that our proposed method effectively enhances the
performance of LLMs by integrating knowledge from different languages and
reduce the performance gap in different languages.

##  2 Related Work

###  2.1 Multilingual LLMs

There has been a surge in research and work on Multilingual Large Language
Models (MLLMs) Qin et al. (2024); Li et al. (2024a); Xu et al. (2024b); Chen
et al. (2024b); Etxaniz et al. (2023). For instance, the InternLM, proposed by
Team (2023), is a multilingual language model that has demonstrated excellent
performance on multiple Chinese benchmarks. Similarly, PolyLM Wei et al.
(2023b) is another LLM trained using curriculum learning, surpassing other
open-source models in multilingual tasks. Besides the above multilingual LLMs,
the popular LLMs also include the ChatGLM series developed by Du et al. (2022)
and Zeng et al. (2022), and Baichuan series Yang et al. (2023). To improve
model performance on multilingual tasks, Muennighoff et al. (2023) and Zhang
et al. (2023b) focus on utilizing multilingual training data to fine-tune the
parameters.

In terms of evaluation, Lai et al. (2023) assessed ChatGPT’s performance
across 37 different languages.CulturaX Nguyen et al. (2023) is a multilingual
dataset containing 6.3 trillion tokens across 167 languages, aimed at
promoting the development of multilingual LLMs. Additionally, M3Exam Zhang et
al. (2023c) introduces a dataset derived from real and official human exam
questions, designed for evaluating LLMs in a multilingual, multimodal, and
multilevel context. BUFFET consolidates 15 varied tasks across 54 languages
into a sequence-to-sequence format, offering a standardized set of few-shot
examples and instructions Asai et al. (2023).

###  2.2 Factuality in LLMs

One way to improve the factuality of LLMs is the utilization of knowledge
graphs (KGs)Sun et al. (2024b). For instance, Abu-Rasheed et al. (2024) uses
knowledge graphs to learn explainable recommendations. Yang et al. (2024b)
suggests improving LLMs through the development of knowledge graph-enhanced
LLMs, which offers a method to boost the factual reasoning capabilities of
LLMs. Sun et al. (2024a) utilizes the LLM as an agent to interact with and
navigate through the KGs, identifying relevant entities and relationships, and
conducting reasoning with the knowledge it gathers.

Another method to enhance the factual knowledge of LLMs is the utilization of
prompt engineering. Previous studies propose various prompt methods such as
Chain-of-Thoughts (CoT) Wei et al. (2023a) and Tree-of-Thoughts (ToT) Yao et
al. (2023). Moreover, some studies use knowledge injection to enhance the
domain capability of LLMs Huang and Sun (2024).

###  2.3 Hallucination Mitigation

A significant challenge associated with LLMs is their tendency to generate
seemingly plausible yet fabricated responses, a phenomenon known as
hallucination Sun et al. (2024c). To address this issue and prevent
misinformation Huang et al. (2024b), recent research has introduced various
hallucination mitigation strategies Tonmoy et al. (2024). For example, Feng et
al. (2024) leverage multi-LLM collaboration to decrease hallucinations in LLM
outputs. Additionally, Guan et al. (2024) have developed a novel framework
called Knowledge Graph-based Retrofitting (KGR), which integrates LLMs with
KGs to minimize factual hallucinations during reasoning. Similarly, Manakul et
al. (2023) propose SelfCheckGPT, a sampling method that verifies the accuracy
of responses from black-box models without the need for an external database.

##  3 Methodology

Figure 2: The knowledge domain of a multilingual LLM can be separated into
multiple sections (the figure shows two). The language-specific knowledge
(pure blue or pure orange) in one language can be utilized for improving the
performance in other languages. Figure 3: The average performance of six LLMs
in five datasets. We show the accuracy of Chinese and English domain knowledge
with the query/answer in Chinese and English.

###  3.1 Motivation

Our proposed method draws inspiration from the distinct knowledge domains
inherent to different languages. As illustrated in Figure 2, language-specific
knowledge can serve as supplementary information for another language. Figure
3 demonstrates that when queries related to English domain knowledge are posed
in Chinese, the performance (i.e., accuracy) of LLMs declines compared to
those posed in the English language. Furthermore, Figure 8 reveals that LLMs
often provide correct answers in only one of two languages for a given query,
suggesting the potential to use the correct response to rectify inaccuracies
in the other language. These observations underscore the potential to leverage
the strengths of each language to enhance LLM performance across different
languages. As shown in Figure 4, the proposed method includes three main
modules: low-resource knowledge detection, target language selection, and
answer replacement & integration.

###  3.2 Construction of Low-Resource Dataset

We first construct a low-resource dataset to measure current LLMs’
multilingual knowledge transfer capacity. We also use this dataset to train
our Low-Resource Knowledge Detector in section 3.3. We initial the dataset
with the combination of various existing question-answering datasets including
TriviaQA Joshi et al. (2017), CMMLU Li et al. (2024b), HalluEval Li et al.
(2023a), TruthfulQA Lin et al. (2022), MKQA Longpre et al. (2021), XQuAD
Artetxe et al. (2019), LC-QuAD Trivedi et al. (2017), KgCLUE Xu et al. (2020).
Moreover, we also construct a dataset that uses LLM-powered synthesized data
to cover more knowledge and topics in the training corpus (We call it
MultiGen). The details of the constructed dataset are shown in Appendix C.

To label these data items, we first use an LLM-Human collaboration to label
the samples as Chinese-specific, English-specific, or common sense.
Specifically, to confirm the correctness of the labeling, we infer the GPT-4
twice to label the samples with a temperature of 1.0 to enlarge the potential
uncertainty of its output. We then conduct human inspections of the dataset
where the labels are inconsistent in two labeling processes, to confirm the
labeling and filter out the samples that are too hard or ambiguous for current
LLMs. The statistics of the dataset can be found in Table 1.

Figure 4: The proposed method begins with the query detection of low-resource
knowledge powered by a detector. If low-resource knowledge is detected within
the queries, LLMs then select the language most likely to yield the best
answer. Answer replacement and integration are employed to formulate the final
response.

###  3.3 Low-Resource Knowledge Detector

The multilingual misalignment stems from the unbalanced training data as the
knowledge with low data resources is less likely to be captured by the
language model during the pretraining process. For example, queries about the
details of Chinese history are not well answered by the model if asked in
English as they appear less frequently in the English pretraining corpus. This
phenomenon could be improved by fully utilizing the model’s inherent capacity.
To implement this process, we first adopt a low-resource knowledge detector to
identify these low-resource queries and later borrow knowledge from other
languages for help.

We train a classifier for each source language to identify the low-resource
query for that language. This classifier separates the query about common
sense and language-specified knowledge(e.g. Spanish query about Spanish
culture) from the low-resources query(e.g. Spanish query about Turkish
geography). Queries of the former class are fed into the normal pipeline of
language generation while the latter queries are to be enhanced by the
knowledge of other languages through our design of other modules. Given a
query $x$ in the original language $L_{o}$, the low-resource knowledge
detector $F_{L_{o}}$ works as follow:

$F_{L_{o}}(x)=\left\\{\begin{array}[]{rcl}1&,&\text{$x$ is low-resource query
of $L_{o}$}\\\ 0&,&\text{else}\\\ \end{array}\right.$ (1)

We demonstrate in the experiment that a classifier is effective enough to
distinguish low-resource queries from others. The construction of the training
dataset of $F_{L}$ can be found in subsection 4.1.

The method is cost-effective as it does not require the translation of all
queries to multiple languages considering that low-resource query is only a
small part of user queries. The majority of user queries are related to common
sense and knowledge specified in that language and do not need to go through
the following process.

###  3.4 Target Language Selection

After selecting the low-resource query from the user’s input, we later adopt a
target language selection module to find the most suitable language for that
question(e.g. translating a question in English about Chinese history into
Chinese). Answering the query with its most resourceful language would improve
output quality in terms of correctness and may offer more useful details to
the user. We implement this process by prompting the LLM itself as the
selection is model-dependent. Different LLMs may select different target
languages due to their pretraining corpus. Given the prompt $P_{\text{sel}}$
to help select the target language, the low-resource query $x$, the procedure
of Target Language Selection is defined as follows:

$x^{\prime}\leftarrow\mathrm{Trans}\big{(}x,\text{LLM}(P_{\text{sel}}(x))\big{)},$
(2)

where translator $\mathrm{Trans}(Q,L_{t})$ translates the input $Q$ into
target language $L_{t}$, and LLM is the large language model that selects the
most suitable language for $x$ with prompt $P_{\text{sel}}$.

###  3.5 Answer Replacement & Integration

After translating the original query $x$ to the query in target language
$x^{\prime}$, we use it to prompt the model for the answer in target language
$a_{t}$. We simply translate the answer back to the original language to get
the final answer $a_{\text{final}}$ for the user’s understanding.

$a_{\text{final}}\leftarrow\mathrm{Trans}(a_{t},L_{o}),$ (3)

where $L_{o}$ is the original language of the user’s query.

We also explore the integration of answers in the scenario of open-ended
question answering (the prompt template is shown in Appendix F). We let the
LLM combine and integrate the answer in the target language $a_{t}$ and the
answer in the original language $a_{o}=\text{LLM}(x)$:

$a_{\text{final}}=\text{LLM}(P_{\text{int}}(a_{t},a_{o})),$ (4)

where $P_{\text{int}}$ is the prompt to help LLM integrate between $a_{t}$ and
$a_{o}$, and $a_{\text{final}}$ is the final answer.

Algorithm 1 Proposed Method

0: Query $x$ in original language $L_{o}$

0: Final answer $a_{\text{final}}$

1: Low-Resource Knowledge Detection:

2: Train classifier $F_{L_{o}}$ for language $L_{o}$

3: $isLowResource\leftarrow F_{L_{o}}(x)$

4: if $isLowResource==1$ then

5: Target Language Selection:

6: Define prompt $P_{\text{sel}}$ for selecting target language

7: $L_{t}\leftarrow\text{LLM}(P_{\text{sel}}(x))$

8: $x^{\prime}\leftarrow\mathrm{Trans}(x,L_{t})$

9: Answer Generation:

10: $a_{t}\leftarrow\text{LLM}(x^{\prime})$

11: $a_{o}\leftarrow\mathrm{Trans}(a_{t},L_{o})$

12: Answer Integration:

13: Define prompt $P_{\text{int}}$ for integrating answers

14: $a_{\text{final}}\leftarrow\text{LLM}(P_{\text{int}}(a_{t},a_{o}))$

15: else

16: $a_{\text{final}}\leftarrow\text{LLM}(x)$

17: end if

18: return $a_{\text{final}}$

##  4 Experiments

We chose English and Chinese for our experiments primarily due to their broad
applicability and the availability of resources. Firstly, most LLMs,
particularly open-source ones like the ChatGLM series, perform significantly
in English and Chinese. This trend highlights the advanced development and
optimization of LLMs for these languages, making them ideal for rigorous
testing. Secondly, major LLM benchmarks and datasets predominantly focus on
these two languages. For instance, besides English benchmarks or datasets,
benchmarks such as HalluQA and AlignBench are primarily designed around
English and Chinese, providing a robust framework for evaluating our methods.
Lastly, the linguistic features and data availability in English and Chinese
ensure comprehensive evaluation and validation of our approaches and suggest
that our findings could be extrapolated to other languages. This potential for
cross-linguistic application supports the broader relevance and utility of our
study, choosing English and Chinese as both strategic and impactful.

###  4.1 Experiment Setup

Training Datasets for Detectors. As we need to train the low-resource detector
for each language, for the dataset in English (_e.g._ , TriviaQA) or the
dataset in Chinese (_e.g._ , CMMLU, KgCLUE), we translate them to another
language (_i.e._ , Chinese or English) through translation API
***https://fanyi.youdao.com/.

Detailed Setting. To ensure the reproducibility of results, the temperature
parameter for all LLMs is set to 0. For ChatGPT, GPT-4, and Qwen-turbo, we use
the official API. For Yi-34b, we use the API from
Replicate†††https://replicate.com/. For ChatGLM3 and Llama3-Chinese, we deploy
them locally for inference with a V100 (40G).

Dataset | Chinese | Common | English | Total | Lang.
---|---|---|---|---|---
TriviaQA | 21 | 754 | 1040 | 1815 | En.
CMMLU | 1200 | 2162 | 2751 | 6113 | Ch.
HalluEval | 28 | 923 | 1033 | 1984 | En.
TruthfulQA | 9 | 322 | 212 | 543 | En.
MKQA | 71 | 315 | 1114 | 1500 | En.
XQuAD | 72 | 610 | 503 | 1185 | En.
LC-QuAD | 2 | 640 | 345 | 987 | En.
KgCLUE | 1218 | 610 | 172 | 2000 | Ch.
MultiGen | 1095 | 1121 | 1083 | 3299 | En.
Total | 3716 | 7457 | 8253 | 19426 | /

Table 1: Dataset statistics of the low-resource knowledge detector. "Lang." is
the original language for the dataset. Figure 5: Statistics of the dataset in
our experiments.

Test Datasets. We selected five datasets for our study, comprising four pre-
existing datasets and one that we developed in-house. The following criteria
guided our selection:

  * •

The datasets should not predominantly consist of common-sense questions (i.e.,
questions that are independent of linguistic background), as this minimizes
the potential for LLMs to demonstrate improvement through linguistic
knowledge.

  * •

The datasets should maintain a balance in difficulty; they should not be
overly simplistic or excessively challenging. Datasets that are too easy can
lead to inflated performance metrics for LLMs, thereby reducing the potential
for meaningful improvement. Conversely, datasets that are too challenging can
degrade performance across all linguistic contexts, thereby constraining the
opportunity to enhance performance in the target language by leveraging
knowledge of additional languages.

|  | ChatGLM3 | ChatGPT | GPT-4 | Yi-34b | Qwen-turbo | Llama3-Ch.
---|---|---|---|---|---|---|---
Dataset | Lang. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro.
| (en) | 18.03% | 18.03% | 57.98% | 57.84% | 67.13% | 67.13% | 42.86% | 42.72% | 29.31% | 29.31% | 40.67% | 40.67%
HalluEval | (ch) | 11.23% | 17.34% | 32.07% | 51.40% | 47.99% | 64.36% | 25.10% | 39.67% | 19.35% | 26.09% | 25.35% | 37.19%
| (en) | 20.00% | 25.95% | 34.27% | 30.90% | 51.89% | 54.05% | 38.38% | 47.03% | 25.97% | 37.57% | 22.83% | 19.57%
HalluQA | (ch) | 22.16% | 22.16% | 21.91% | 24.16% | 49.73% | 51.35% | 45.95% | 44.86% | 43.65% | 43.09% | 15.22% | 16.30%
Chinese | (en) | 9.52% | 20.78% | 41.85% | 42.73% | 56.71% | 58.44% | 33.33% | 55.84% | 27.19% | 46.05% | 30.73% | 24.24%
Domain | (ch) | 32.47% | 32.47% | 41.85% | 41.85% | 59.31% | 59.74% | 63.64% | 63.20% | 62.28% | 61.84% | 18.61% | 18.61%
| (en) | 36.32% | 36.32% | 90.53% | 90.37% | 94.09% | 94.09% | 79.33% | 79.17% | 59.59% | 59.47% | 77.27% | 77.16%
triviaQA | (ch) | 21.33% | 31.95% | 54.60% | 82.67% | 82.77% | 91.90% | 59.43% | 75.56% | 41.53% | 52.99% | 43.92% | 65.17%
| (en) | 28.51% | 38.15% | 59.08% | 58.88% | 67.70% | 69.41% | 57.07% | 68.71% | 49.48% | 61.08% | 50.00% | 48.09%
WebQA | (ch) | 48.69% | 48.49% | 57.35% | 57.86% | 72.52% | 72.42% | 76.93% | 76.13% | 71.12% | 71.33% | 37.02% | 38.43%

Table 2: Six LLMs’ performance on our proposed method.

For all datasets in our study, we select QA-pair samples from them and do not
use other extra data to facilitate our evaluation. Totally, we select five
datasets for evaluating our method. These include four existing dataset:
TriviaQA Joshi et al. (2017), HaluEval Li et al. (2023a), HalluQA Cheng et al.
(2023), and WebQA Li et al. (2016). We show the statistics of the datasets we
selected in Figure 5 and the details are shown in Appendix A. In addition to
the four datasets mentioned above, we have constructed a bilingual Chinese-
English dataset tailored to the Chinese domain. Details of the construction
process are provided in Appendix D.

Models. We carefully select six popular LLMs including proprietary and open-
source LLMs that master both English and Chinese: ChatGPT (OpenAI, 2023a),
GPT-4 (OpenAI, 2023b), ChatGLM3 (Zeng et al., 2022; Du et al., 2022), Yi-34b
(AI et al., 2024), Qwen-turbo Bai et al. (2023), and LLama3-Chinese (lla,
2024).

###  4.2 Main Results

We evaluate the effectiveness of our proposed method on five benchmark
datasets and six popular LLMs mentioned above. Each dataset is translated into
a Chinese and an English version for later assessment. We first infer the
models with the queries in the dataset to get the generated answers. We then
leverage GPT-4 as the judge model to compare each generated answer with the
reference answer in the dataset to see if the model produces a correct output.
We calculate the generation accuracy and present the result in Table 2. We
mark the result in green where there is a significant improvement of more than
$1\%$ and mark the result in red if the accuracy decrease by more than $1\%$.

As can be seen from the table, our method can effectively improve the
performance of the model in many scenarios. To be specific, the performance of
the GPT-4 model on the HalluEval dataset in Chinese improves significantly
from $47.99\%$ to $64.36\%$. This means there still exists a large cross-
lingual knowledge gap in advanced models such as GPT-4 and our method
successfully leverages the knowledge across languages to enhance the model’s
performance. It is important to notice that the improvements do not rely on
other models or online resources, they exist due to our leverage of the
model’s inherent capacity.

It can also be observed from Table 2 that most improvements happen in the
language that is different from that of the original dataset, which is also
the part where the models suffer from a weaker performance. The comparison of
the cross-lingual performance gap before and after applying our method is
shown in Figure 6. The figure showcases that our method could significantly
reduce the knowledge gap between languages in all LLMs we evaluate, thus
improving the fairness of the application for users of different languages.

Figure 6: The average performance gap on datasets before and after applying
our method.

###  4.3 Ablation Study

As our generation pipeline consists of several parts, we conduct an ablation
study to validate their effectiveness and expenses.

The Impact of the Low-resource Detector. The low-resource detector serves as a
filter to sift the language-specific queries from the majority of the queries
that involve only commonsense, thus improving efficiency and reducing the
expense of the pipeline. As can be observed in Figure 7, a low-resource query
detector would significantly reduce the average inference time per sample from
more than 9 seconds to less than 6.5 seconds if the ratio of the low-resource
queries is 0.05 in the dataset. When the ratio of the low-resource query in
the dataset increases, the detector passes more samples into the translation
pipeline and increases the average inference time.

Another intriguing finding is that the low-resource detector would increase
the model performance. As shown in Table 3, the performance of the pipeline is
unstable when we remove the low-resource detector. The overall performance
would also drop as we observed in Figure 7. This indicates that the detector
and LLM itself can be complementary. The full result of the models’
performance without the low-resource detector can be found in Table 7.

|  | Yi-34b | Qwen-turbo | Llama3-Ch.
---|---|---|---|---
Dataset | Lang. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro.
| (en) | 42.86% | 41.75% | 29.31% | 29.59% | 40.67% | 40.67%
HalluEval | (ch) | 25.10% | 39.81% | 19.35% | 26.51% | 25.35% | 37.33%
| (en) | 38.38% | 47.03% | 25.97% | 37.57% | 22.83% | 18.48%
HalluQA | (ch) | 45.95% | 45.95% | 43.65% | 39.78% | 15.22% | 20.65%
Chinese | (en) | 33.33% | 57.58% | 27.19% | 48.25% | 30.74% | 24.24%
Domain | (ch) | 63.64% | 57.14% | 62.28% | 62.28% | 18.61% | 22.51%

Table 3: Selected LLMs’ performance on the setting without a low-resource
detector. Figure 7: The relationship of time efficiency and error rate. The
error rate is the percentage sum of all decreasing in five datasets (value in
red on Table 2, Table 3 (w/o Detection), Table 4 (w/o Selection)).

|  | Yi-34b | Qwen-turbo | Llama3-Ch.
---|---|---|---|---
Dataset | Lang. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro.
| (en) | 42.86% | 42.72% | 29.31% | 29.17% | 40.67% | 40.25%
HalluEval | (ch) | 25.10% | 42.58% | 19.35% | 28.61% | 25.34% | 39.97%
| (en) | 38.38% | 46.49% | 25.97% | 38.67% | 22.83% | 21.74%
HalluQA | (ch) | 45.95% | 44.86% | 43.65% | 41.44% | 15.21% | 15.76%
Chinese | (en) | 33.33% | 61.04% | 27.19% | 58.33% | 30.74% | 19.48%
Domain | (ch) | 63.64% | 60.17% | 62.28% | 58.33% | 18.61% | 18.61%

Table 4: Selected LLMs’ performance on the setting without language selection. |  | ChatGPT | GPT-4 | ChatGLM3 | Yi-34b | Qwen-Turbo | Llama3-Ch.
---|---|---|---|---|---|---|---
Type | Lang. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro.
| (ch) | 4.98 | 5.26 | 6.90 | 6.85 | 4.15 | 4.09 | 5.82 | 5.91 | 5.88 | 5.87 | 3.71 | 3.58
Integrate | (en) | 5.92 | 6.02 | 7.32 | 7.54 | 4.02 | 4.07 | 5.86 | 6.13 | 5.59 | 5.78 | 4.60 | 4.60
| (ch) | 4.98 | 5.47 | 6.90 | 6.98 | 4.15 | 4.16 | 5.82 | 6.23 | 5.88 | 6.00 | 3.71 | 3.93
Replace | (en) | 5.92 | 5.97 | 7.32 | 7.12 | 4.02 | 4.26 | 5.86 | 6.25 | 5.59 | 5.88 | 4.60 | 4.54
Table 5: Model performance on AlignBench Liu et al. (2023a) in the setting of
answer replacement and integration.

The Impact of the Language Selection Module. The language selection module can
choose the proper language to answer the question with model-specific choice.
It is also flexible in the multi-lingual setting as the resulting target
language can be more than two as we test. However, we still validate its
effectiveness in the bi-lingual setting, comparing it with the strategy of
using the opposite language when the query is detected as low-resources, and
show our result in Table 4. The trade-off between its cost and error can also
be found in Figure 7.

As we can see from Figure 7, the language selection module only adds a small
inference cost while significantly improving the model performance. This is
due to the existence of the query that is low-resource for both languages, in
which switching to the opposite language may make the situation worse. In
these situations, the language selection module may pick a third language to
better answer the question. The full result of the performance without the
language selection module can be found in Table 8.

The Comparison between Answer Replacement and Integration. We further
investigated the effectiveness of answer replacement and integration
strategies. Given that QA setups with a golden answer may not always
accommodate answer integration effectively (for example, when the answers in
two different languages factually conflict), we opted for a subset in
AlignBench Liu et al. (2023a) as our evaluation dataset. AlignBench provides a
comprehensive, multi-dimensional benchmark designed to assess the alignment of
LLMs in Chinese, featuring a variety of open-ended questions. To create a
bilingual dataset, we translated the Chinese questions into English. For each
response evaluation, we employed an LLM-as-a-judge approach, utilizing the
prompt template from AlignBench. The LLM judge then assigned an overall score
ranging from 1 to 10 to each LLM response. As indicated in Table 5, both
replacement and integration methods significantly enhanced the LLMs’
performance across most datasets. Direct replacement led to more substantial
improvements but also introduced a higher rate of errors, as evidenced by the
performance dips in GPT-4 and Llama3-Ch. Interestingly, the integration method
showed a more pronounced performance improvement in English responses,
suggesting that LLMs may possess stronger capabilities for answer optimization
in English than in Chinese Yang et al. (2024a).

The Impact of Different Detection Models. As we build a different low-resource
detector for each language, the selection of the tokenizer and classification
model would impact the training of the detector thereby influencing the
overall performance. We adopt language-specific Bert and multi-lingual Bert
models to train our low-resource query detector and report the result in Table
6. As shown in the model, using the language-specific model and tokenizer
would slightly improve the result of using a multi-lingual model.

Model | Acc. | Recall | Precision | F1.
---|---|---|---|---
bert-base-chinese (ch) | 86.64 | 86.64 | 86.68 | 86.66
bert-uncased (en) | 94.98 | 94.98 | 94.88 | 94.91
Multilingual Bert (ch) | 86.47 | 86.47 | 86.58 | 86.51
Multilingual Bert (en) | 94.73 | 94.73 | 94.64 | 94.67
Table 6: The impact of model selection on detector training.

##  5 Discussion on Other Approach

As the confidence of the generated content is related to its entropy during
the generation process, a natural idea is to calculate the entropy in
different languages and compare them to decide which is the best language to
answer the question. This approach is widely used for measuring LLMs’
uncertainty and detecting hallucinations Manakul et al. (2023). However, our
trial demonstrates that this approach is infeasible and achieves merely
random-guess-level performance when selecting the right language for the given
queries.

To explore how to leverage entropy-related statistics to select the target
language, we train a model $f$ that takes the statistics as input and outputs
the selection of the language $Y$. The statistics we use for a language $l$
include the entropy of the query $E_{Q_{l}}$, the entropy of the response
$E_{R_{l}}$, the perplexity of the query $P_{Q_{l}}$, and the perplexity of
the response $P_{R_{l}}$. We adopt an MLP as the classification model
$f:(E_{Q_{l}},E_{R_{l}},P_{Q_{l}},P_{R_{l}})\rightarrow Y$ and train the model
on the low-resource query dataset we construct. We trained based on SVM and
random forests in Llama2-7b’s output. The accuracy is no more than 60%. This
is a merely random-guess-level performance when taking the entropy-related
statistics as input. We attribute this to the hallucination issue of LLMs,
that the model may become over-confident even with the wrong answer Groot and
Valdenegro-Toro (2024), which indicates LLMs are not calibrated well now Zhang
et al. (2024).

##  6 Conclusion

This paper presents a method to improve the multilingual capabilities of LLMs
by leveraging knowledge from various languages, which includes a low-resource
knowledge detector, a process for selecting languages, and answer replacement
& integration. Our experiments show significant enhancements in performance,
especially in reducing disparities across languages. Moreover, each module in
our method contributes to the improvement. Overall, this study underscores the
potential of LLMs to unify multilingual functions and provide insights for
future research.

## Limitations

Our method requires training a separate low-resource query detector for each
language. This is not convenient as the developer of a certain language should
construct a low-resource training set himself, which involves collecting
language-specific data. Also, the dataset should be updated with time with the
rise of the new language-specific data.

## Ethics Statement

This study adheres to ethical standards in AI research and development. We
acknowledge that while our methods aim to enhance the multilingual
capabilities of LLMs, they must be implemented with careful consideration of
potential biases. Efforts were made to ensure that the knowledge aggregation
does not disproportionately favor any particular language or cultural
perspective. We also emphasize transparency in our methodologies and findings
to enable scrutiny and replication by the research community. The research was
conducted without utilizing any personally identifiable information, thereby
safeguarding privacy and upholding data protection standards. We commit to
ongoing evaluation of our methods in diverse linguistic settings to address
and mitigate any emergent biases or disparities. This research seeks not only
to advance technology but also to promote inclusivity and fairness in AI
applications across different linguistic and cultural groups. In this paper,
we utilized AI tools to aid in writing and coding, ensuring that they did not
directly contribute to the writing process and that their use adheres to
academic standards. Additionally, we ensured that all datasets and benchmarks
used in the study comply with their intended purposes and standards.

## References

  * lla (2024) 2024\.  Llama3-chinese.  https://github.com/LlamaFamily/Llama-Chinese.
  * Abu-Rasheed et al. (2024) Hasan Abu-Rasheed, Christian Weber, and Madjid Fathi. 2024.  Knowledge graphs as context sources for llm-based explanations of learning recommendations.
  * AI et al. (2024) 01\. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024.  Yi: Open foundation models by 01.ai.
  * Artetxe et al. (2019) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019.  On the cross-lingual transferability of monolingual representations.  _CoRR_ , abs/1910.11856.
  * Asai et al. (2023) Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023.  Buffet: Benchmarking large language models for few-shot cross-lingual transfer.  _arXiv preprint arXiv:2305.14857_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Chen et al. (2024a) Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. 2024a.  Gui-world: A dataset for gui-oriented multimodal llm-based agents.
  * Chen et al. (2024b) Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu, Haihui Pan, Leichao Xu, Dacheng Zhang, Zhipeng Zhang, and Kun Han. 2024b.  Orion-14b: Open-source multilingual large language models.
  * Cheng et al. (2023) Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu. 2023.  Evaluating hallucinations in chinese large language models.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.
  * Etxaniz et al. (2023) Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023.  Do multilingual language models think better in english?
  * Feng et al. (2024) Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024.  Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration.
  * Gao et al. (2024) Chujie Gao, Qihui Zhang, Dongping Chen, Yue Huang, Siyuan Wu, Zhengyan Fu, Yao Wan, Xiangliang Zhang, and Lichao Sun. 2024.  The best of both worlds: Toward an honest and helpful large language model.
  * Groot and Valdenegro-Toro (2024) Tobias Groot and Matias Valdenegro-Toro. 2024.  Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models.
  * Guan et al. (2024) Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. 2024.  Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 18126–18134.
  * Guo et al. (2024) Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024.  Large language model based multi-agents: A survey of progress and challenges.
  * Guo et al. (2023) Taicheng Guo, kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, and Xiangliang Zhang. 2023.  What can large language models do in chemistry? a comprehensive benchmark on eight tasks.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 59662–59688. Curran Associates, Inc.
  * Huang et al. (2024a) Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024a.  Metatool benchmark for large language models: Deciding whether to use tools and which to use.
  * Huang et al. (2024b) Yue Huang, Kai Shu, Philip S. Yu, and Lichao Sun. 2024b.  From creation to clarification: Chatgpt’s journey through the fake news quagmire.  In _Companion Proceedings of the ACM on Web Conference 2024_ , WWW ’24, page 513–516, New York, NY, USA. Association for Computing Machinery.
  * Huang and Sun (2024) Yue Huang and Lichao Sun. 2024.  Fakegpt: Fake news generation, explanation and detection of large language models.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
  * Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.
  * Li et al. (2024a) Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. 2024a.  X-instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions.
  * Li et al. (2024b) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024b.  Cmmlu: Measuring massive multitask language understanding in chinese.
  * Li et al. (2023a) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a.  Halueval: A large-scale hallucination evaluation benchmark for large language models.
  * Li et al. (2016) Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016.  Dataset and neural recurrent sequence labeling model for open-domain factoid question answering.
  * Li et al. (2023b) Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023b.  A survey on fairness in large language models.  _arXiv preprint arXiv:2308.10149_.
  * Li et al. (2024c) Yuan Li, Yue Huang, Yuli Lin, Siyuan Wu, Yao Wan, and Lichao Sun. 2024c.  I think, therefore i am: Benchmarking awareness of large language models using awarebench.
  * Li et al. (2024d) Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. 2024d.  Quantifying multilingual performance of large language models across languages.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.
  * Liu et al. (2023a) Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023a.  Alignbench: Benchmarking chinese alignment of large language models.
  * Liu et al. (2023b) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023b.  Agentbench: Evaluating llms as agents.
  * Liu et al. (2023c) Zhengliang Liu, Yue Huang, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Yiwei Li, Peng Shu, et al. 2023c.  Deid-gpt: Zero-shot medical text de-identification by gpt-4.  _arXiv preprint arXiv:2303.11032_.
  * Longpre et al. (2021) Shayne Longpre, Yi Lu, and Joachim Daiber. 2021.  Mkqa: A linguistically diverse benchmark for multilingual open domain question answering.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.
  * Nguyen et al. (2023) Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023.  Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.
  * OpenAI (2023a) OpenAI. 2023a.  Chatgpt.  https://openai.com/product/chatgpt.
  * OpenAI (2023b) OpenAI. 2023b.  Gpt-4.  https://openai.com/gpt-4.
  * Qian et al. (2023) Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_.
  * Qin et al. (2024) Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2024.  Multilingual large language model: A survey of resources, taxonomy and frontiers.
  * Sun et al. (2024a) Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024a.  Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph.
  * Sun et al. (2024b) Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2024b.  Head-to-tail: How knowledgeable are large language models (llms)? a.k.a. will llms replace knowledge graphs?
  * Sun et al. (2024c) Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. 2024c.  Trustllm: Trustworthiness in large language models.
  * Team (2023) InternLM Team. 2023.  Internlm: A multilingual language model with progressively enhanced capabilities.
  * Tonmoy et al. (2024) SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024.  A comprehensive survey of hallucination mitigation techniques in large language models.  _arXiv preprint arXiv:2401.01313_.
  * Trivedi et al. (2017) Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017.  Lc-quad: A corpus for complex question answering over knowledge graphs.  In _International Semantic Web Conference_ , pages 210–218. Springer.
  * Wei et al. (2023a) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023a.  Chain-of-thought prompting elicits reasoning in large language models.
  * Wei et al. (2023b) Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023b.  Polylm: An open source polyglot large language model.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020.  CLUE: A Chinese language understanding evaluation benchmark.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Xu et al. (2024a) Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024a.  Knowledge conflicts for llms: A survey.
  * Xu et al. (2024b) Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Yuqi Ye, and Hanwen Gu. 2024b.  A survey on multilingual large language models: Corpora, alignment, and bias.
  * Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_.
  * Yang et al. (2024a) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024a.  Large language models as optimizers.
  * Yang et al. (2024b) Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2024b.  Give us the facts: Enhancing large language models with knowledge graphs for fact-aware language modeling.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.
  * Yu et al. (2023) Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023.  Large language model as attributed training data generator: A tale of diversity and bias.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zhang et al. (2023a) Kai Zhang, Jun Yu, Zhiling Yan, Yixin Liu, Eashan Adhikarla, Sunyang Fu, Xun Chen, Chen Chen, Yuyin Zhou, Xiang Li, et al. 2023a.  Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks.  _arXiv preprint arXiv:2305.17100_.
  * Zhang et al. (2024) Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu. 2024.  Calibrating the confidence of large language models by eliciting fidelity.
  * Zhang et al. (2023b) Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023b.  Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models.
  * Zhang et al. (2023c) Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023c.  M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models.

##  Appendix A Dataset Details

  * •

TriviaQA Joshi et al. (2017) is a reading comprehension dataset that features
more than 650,000 question-answer-evidence triples. It consists of lots of
question-answer pairs created by trivia aficionados, along with independently
collected evidence documents—averaging six per question—that offer robust
distant supervision for responding to the queries.

  * •

HaluEval Li et al. (2023a) is a benchmark designed to assess how well LLMs
hallucinations—unverifiable or incorrect content in their outputs. It includes
a collection of generated texts and human-annotated samples that help evaluate
the models’ performance in detecting such errors.

  * •

HalluQA Cheng et al. (2023) is a dataset consisting of 450 carefully crafted
adversarial questions that cover various domains, incorporating elements of
Chinese historical culture, customs, and social phenomena. It aims to evaluate
LLMs on their propensity to produce two types of errors: imitative falsehoods
and factual inaccuracies.

  * •

WebQA Li et al. (2016) is a large-scale, human-annotated real-world QA
dataset, developed to address the scarcity of extensive real-world QA datasets
for neural QA systems.

##  Appendix B Experiment Results

We show the full experiment results in Table 7, Table 8, and Figure 8.

|  | ChatGLM3 | ChatGPT | GPT-4 | Yi-34b | Qwen-turbo | Llama3-Ch.
---|---|---|---|---|---|---|---
Dataset | Lang. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro.
| (en) | 18.03% | 18.03% | 57.98% | 57.84% | 67.13% | 66.99% | 42.86% | 41.75% | 29.31% | 29.59% | 40.67% | 40.67%
HalluEval | (ch) | 11.23% | 17.34% | 32.07% | 52.38% | 47.99% | 65.05% | 25.10% | 39.81% | 19.35% | 26.51% | 25.35% | 37.33%
| (en) | 20.00% | 25.41% | 34.27% | 30.90% | 51.89% | 53.51% | 38.38% | 47.03% | 25.97% | 37.57% | 22.83% | 18.48%
HalluQA | (ch) | 22.16% | 22.70% | 21.91% | 25.28% | 49.73% | 51.89% | 45.95% | 45.95% | 43.65% | 39.78% | 15.22% | 20.65%
Chinese | (en) | 9.52% | 21.21% | 41.85% | 42.73% | 56.71% | 57.58% | 33.33% | 57.58% | 27.19% | 48.25% | 30.74% | 24.24%
Domain | (ch) | 32.47% | 25.54% | 41.85% | 42.29% | 59.31% | 58.44% | 63.64% | 57.14% | 62.28% | 62.28% | 18.61% | 22.51%
| (en) | 36.32% | 36.38% | 90.53% | 90.31% | 94.09% | 93.93% | 79.33% | 78.90% | 59.59% | 59.47% | 77.27% | 77.05%
triviaQA | (ch) | 21.33% | 32.22% | 54.60% | 83.33% | 82.77% | 92.29% | 59.43% | 76.27% | 41.53% | 53.55% | 43.92% | 66.32%
| (en) | 28.51% | 38.96% | 59.08% | 58.98% | 67.70% | 69.61% | 57.07% | 69.71% | 49.48% | 62.11% | 50.00% | 47.08%
WebQA | (ch) | 48.69% | 42.07% | 57.35% | 59.29% | 72.52% | 72.32% | 76.93% | 74.12% | 71.12% | 70.70% | 37.02% | 40.54%

Table 7: Six LLMs’ performance on the setting without a low-resource detector.

|  | ChatGLM3 | ChatGPT | GPT-4 | Yi-34b | Qwen-turbo | Llama3-Ch.
---|---|---|---|---|---|---|---
Dataset | Lang. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro. | Origi. | Impro.
| (en) | 18.03% | 18.31% | 57.98% | 57.70% | 67.13% | 66.57% | 42.86% | 42.72% | 29.31% | 29.17% | 40.67% | 40.25%
HalluEval | (ch) | 11.23% | 18.03% | 32.07% | 56.02% | 47.99% | 66.16% | 25.10% | 42.58% | 19.35% | 28.61% | 25.34% | 39.97%
| (en) | 20.00% | 25.95% | 34.27% | 32.02% | 51.89% | 53.51% | 38.38% | 46.49% | 25.97% | 38.67% | 22.83% | 21.74%
HalluQA | (ch) | 22.16% | 23.78% | 21.91% | 23.60% | 49.73% | 51.35% | 45.95% | 44.86% | 43.65% | 41.44% | 15.21% | 15.76%
Chinese | (en) | 9.52% | 32.03% | 41.85% | 41.85% | 56.71% | 59.31% | 33.33% | 61.04% | 27.19% | 58.33% | 30.74% | 19.48%
Domain | (ch) | 32.47% | 30.74% | 41.85% | 41.41% | 59.31% | 58.44% | 63.64% | 60.17% | 62.28% | 58.33% | 18.61% | 18.61%
| (en) | 36.32% | 35.78% | 90.53% | 89.09% | 94.09% | 93.54% | 79.33% | 78.73% | 59.59% | 58.80% | 77.27% | 76.12%
triviaQA | (ch) | 21.33% | 35.94% | 54.60% | 89.15% | 82.77% | 93.22% | 59.43% | 78.18% | 41.53% | 58.41% | 43.92% | 74.92%
| (en) | 28.51% | 44.38% | 59.08% | 59.90% | 67.70% | 70.81% | 57.07% | 73.72% | 49.48% | 67.70% | 50.00% | 46.48%
WebQA | (ch) | 48.69% | 46.99% | 57.35% | 58.88% | 72.52% | 71.61% | 76.93% | 74.22% | 71.12% | 69.25% | 37.02% | 41.15%

Table 8: Six LLMs’ performance on the setting without language selection.
Figure 8: Performance percentage of LLMs across different datasets. ‘All
correct’ indicates that the LLMs answered correctly in both the Chinese and
English datasets. ‘All wrong’ signifies that the LLMs answered incorrectly in
both datasets. ‘Half’ denotes that the LLMs answered correctly in only one of
the datasets.

##  Appendix C Details of Constructed Dataset

For the generated dataset, inspired by previous studies Huang et al. (2024a);
Yu et al. (2023), we employed attribute-guided prompting to instruct LLMs to
generate relevant questions on specific topics, as illustrated in Table 9. We
chose GPT-4 as our generation model because of its exceptional ability to
follow instructions. The prompt template is shown in Figure 9. For the
generated items, we manually evaluate the correctness of its label to ensure
the data quality.

Table 9: Topics used for data generation. Topic Word
---
History | Literature | Science | Art
Social Sciences | Technology | Philosophy | Geography
Culture | Health | Artificial Intelligence | Machine Learning
Big Data | Blockchain | Internet of Things | Environmental Protection
Sustainable Development | Energy | Finance | Education
Human Genetics | Artificial Life | Space Exploration | Food Science
Sports | Psychology | Political Science | Economics
Sociology | Law |  |
Prompt Template Next, I will provide you with a topic, and you will assist me
in generating data based on this topic. I need you to generate three
categories: questions with an English background, questions with a Chinese
background, and questions with no specific language. I will provide you with
some examples: Question: Piaget believes that communication has two functions,
one is the egocentric function, and the other is? Category: English knowledge
Question: With one byte, how many different codes can be generated at most?
Category: Knowledge with no specific language Question: What are some famous
dishes from Guangdong? Category: Chinese knowledge For each type of question,
you need to generate ten, a total of thirty. You need to return the data in
JSON format, as follows:

    {
    "Question": "Category",
    "Question": "Category",
    "Question": "Category",
    "Question": "Category",
    ...
    }

Please generate the corresponding data in Chinese. The topic I provide is:
[TOPIC] Figure 9: Prompt template for the generated dataset.

##  Appendix D Collection of Chinese Domain Dataset

Our Chinese domain dataset consists of 227 items. This dataset encompasses
knowledge and information specific to Chinese content, including aspects of
geography, history, culture, and more. We sourced the content from a broad
range of Chinese social media platforms and search engines. After initial
curation, we conducted filtering to remove contents that cannot be accurately
translated into English or may result in discrepancies in meaning upon
translation, such as phrases from ancient Chinese.

##  Appendix E Answer Evaluation

We adopt the LLM-as-a-Judge for answer evaluation in all experiments to reduce
the bias that comes from keyword matching. We use GPT-4 for our evaluation due
to its exceptional capability and wide application in previous studies Liu et
al. (2023a); Gao et al. (2024). For the five QA datasets, we use the prompt
template shown in Figure 10.

Prompt Template As a helpful assistant, your task is now to help me assess the
correctness of the provided answers. I will present a question along with its
correct answer. Subsequently, I will also provide you with the answer you need
to evaluate. If the answer to be evaluated correctly expresses the same
meaning as the correct answer or contains the correct answer, then it is
right. Ignore case errors. Although there are some errors in certain
explanations within the answer, as long as the core answer is correct, the
response is considered correct. Return me only one word: ’correct’ or ’wrong’.
Here is the question and its correct answer: Question: [QUESTION] Answer:
[ANSWER] Here is the answer you should evaluate: [RES] Figure 10: Prompt
template for LLM-as-a-Judge. Prompt Template You are a very helpful assistant.
I will provide you with a question and the answers in both Chinese and
English. You need to integrate the Chinese and English answers to provide the
final answer. During the integration process, you need to follow these rules:
1\. You should primarily refer to the Chinese answer, appropriately
integrating parts of the English answer. 2\. If there is a factual conflict
between the English and Chinese answers, you must refer to the Chinese answer.
3\. The integrated answer should be of higher quality than the individual
answers and better address the corresponding question. 4\. The integrated
answer must be all in English Question: [[Q]] Chinese answer: [[CH_RES]]
English answer: [[EN_RES]] Figure 11: Prompt template for integration (For the
situation when the selected language is English). Prompt Template
你是一个非常有帮助的助手。我将给你提供一个问题，以及该问题的中英文的答案。你需要融合中英文答案，给出最终的答案。在融合答案的过程中，你需要遵循下面的规则：
1\. 你需要着重参考英文的答案，适当融合部分中文的答案。 2\. 如果英文的答案与中文的答案发生事实性冲突，你必须参考英文的答案。 3\.
融合后的答案应该比融合前的答案具有更高的质量，更好地回答对应的问题。 4\. 融合后的答案必须全都是中文。 问题：[[Q]] 中文答案：[[CH_RES]]
英文答案：[[EN_RES]] Figure 12: Prompt template for integration (For the situation
when the selected language is Chinese).

##  Appendix F Prompt Template

We show the prompt template used in our study in Figure 9, Figure 10, Figure
12, Figure 11, Figure 14, and Figure 13.

##  Appendix G Screenshots of Human Evaluation

We show the screenshots of human evaluation in Figure 15 and Figure 16.

Prompt Template As a helpful assistant, you need to categorize an English
question, considering that the background of this question is not common in an
English environment. Therefore, you need to choose the most suitable language
for this question. You need to analyze the required language context for the
question first, and then tell me at the end which language you think is most
suitable to answer the question. The question is as follows: Figure 13: Prompt
template for language selection (For the query in English). Prompt Template
作为乐于助人的助理，您需要将一个中文问题进行分类，考虑到该问题背景在中文环境中并不常见，因此您需要返回最适合该问题的语言。你需要首先对问题所需要的语言环境进行分析，然后在最后告诉我你返回的最适合回答该问题的语言。问题如下：
Figure 14: Prompt template for language selection (For the query in Chinese).
Figure 15: Screenshot of human annotation (1). Figure 16: Screenshot of human
annotation (2).