ON GEOMETRY OF THE RÖSSLER
SYSTEM OF EQUATIONS

Valery Dryuma

Institute of Mathematics and Informatics AS Moldova, Kishinev

E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
Abstract

Theory of Riemann extensions of spaces of constant affine connection is
proposed to study the Rössler system of equations

$\frac{dx}{ds}=-(y+z),\quad\frac{dy}{ds}=x+\alpha
y,\quad\frac{dz}{ds}=\beta+xz-\nu z.$

After its presentation in a homogeneous form

${\frac{d}{ds}}\xi(s)+1/5\,{\xi}^{2}-1/5\,\xi\,\rho\,\nu+1/5\,\xi\,\rho\,\alpha+\eta\,\rho+\theta\,\rho=0$
${\frac{d}{ds}}\eta(s)+1/5\,\xi\,\eta-\xi\,\rho-1/5\,\eta\,\rho\,\nu-4/5\,\eta\,\rho\,\alpha=0$
${\frac{d}{ds}}\theta(s)-4/5\,\xi\,\theta+4/5\,\theta\,\rho\,\nu+1/5\,\theta\,\rho\,\alpha-\beta\,{\rho}^{2}=0$
${\frac{d}{ds}}\rho(s)+1/5\,\xi\,\rho-1/5\,{\rho}^{2}\nu+1/5\,{\rho}^{2}\alpha=0,$

where

$x=\frac{\xi}{\rho},\quad y=\frac{\eta}{\rho},\quad z=\frac{\theta}{\rho},$

it can be considered as geodesic equations

$\frac{d^{2}X^{i}}{ds^{2}}+\Pi^{i}_{jk}\frac{dX^{j}}{ds}\frac{dX^{k}}{ds}=0.$

of four dimensional space $M^{4}$ of constant affine connection with the
components $\Pi^{i}_{jk}=\Pi^{i}_{jk}(\alpha,\beta,\nu)$ depending on
parameters.

##  1  From the first order system of equations to the second order systems of
ODE

The systems of the first order differential equations

$\frac{dx^{i}}{ds}=c^{i}+a^{i}_{j}x^{j}+b^{i}_{jk}x^{j}x^{k}$ (1)

depending on the parameters $a,b,c$ are not suitable object of consideration
from usually point of the Riemann geometry.

The systems of the second order differential equations in form

$\frac{d^{2}x^{i}}{ds^{2}}+\Pi^{i}_{kj}(x)\frac{dx^{k}}{ds}\frac{dx^{j}}{ds}=0$
(2)

are best suited to do that.

They can be considered as geodesic of affinely connected space $M^{k}$ in
local coordinates $x^{k}$. The values $\Pi^{i}_{jk}=\Pi^{i}_{kj}$ are the
coefficients of affine connections on $M^{k}$.

With the help of such coefficients can be constructed curvature tensor and
others geometrical objects defined on variety $M^{k}$.

##  2 From affinely connected space to the Riemann space

We shall construct the Riemann space starting from a given affinely connected
space defined by the systems of the second order ODE’s.

With this aim we use the notion of the Riemann extension of nonriemannian
space which was used earlier in the articles of author.

Remind the basic properties of this construction.

With help of the coefficients of affine connection of a given n-dimensional
space can be introduced 2n-dimensional Riemann space $D^{2n}$ in local
coordinates $(x^{i},\Psi_{i})$ having the metric of form

${{}^{2n}}ds^{2}=-2\Pi^{k}_{ij}(x^{l})\Psi_{k}dx^{i}dx^{j}+2d\Psi_{k}dx^{k}$
(3)

where $\Psi_{k}$ are an additional coordinates.

The important property of such type metric is that the geodesic equations of
metric (3) decomposes into two parts

$\ddot{x}^{k}+\Pi^{k}_{ij}\dot{x}^{i}\dot{x}^{j}=0,$ (4)

and

$\frac{\delta^{2}\Psi_{k}}{ds^{2}}+R^{l}_{kji}\dot{x}^{j}\dot{x}^{i}\Psi_{l}=0,$
(5)

where

$\frac{\delta\Psi_{k}}{ds}=\frac{d\Psi_{k}}{ds}-\Pi^{l}_{jk}\Psi_{l}\frac{dx^{j}}{ds}$

and $R^{l}_{kji}$ are the curvature tensor of n-dimensional space with a given
affine connection.

The first part (4) of the full system is the system of equations for geodesic
of basic space with local coordinates $x^{i}$ and it do not contains the
supplementary coordinates $\Psi_{k}$.

The second part (5) of the system has the form of linear $N\times N$ matrix
system of second order ODE’s for supplementary coordinates $\Psi_{k}$

$\frac{d^{2}\vec{\Psi}}{ds^{2}}+A(s)\frac{d\vec{\Psi}}{ds}+B(s)\vec{\Psi}=0.$
(6)

Remark that the full system of geodesics has the first integral

$-2\Pi^{k}_{ij}(x^{l})\Psi_{k}\frac{dx^{i}}{ds}\frac{dx^{j}}{ds}+2\frac{d\Psi_{k}}{ds}\frac{dx^{k}}{ds}=\nu$
(7)

which is equivalent to the relation

$2\Psi_{k}\frac{dx^{k}}{ds}=\nu s+\mu$ (8)

where $\mu,\nu$ are parameters.

The geometry of extended space connects with geometry of basic space. For
example the property of the space to be Ricci-flat $R_{ij}=0$ or symmetrical
$R_{ijkl;m}=0$ keeps also for an extended space.

It is important to note that for extended space having the metric (3) all
scalar curvature invariants are vanished.

As consequence the properties of linear system of equation (5-6) depending
from the the invariants of $N\times N$ matrix-function

$E=B-\frac{1}{2}\frac{dA}{ds}-\frac{1}{4}A^{2}$

under change of the coordinates $\Psi_{k}$ can be of used for that.

First applications the notion of extended spaces for the studying of nonlinear
second order ODE’s connected with nonlinear dynamical systems have been
considered by author (V.Dryuma 2000-2008).

##  3  Eight-dimensional Riemann space for the Rössler system of equations

To investigation the properties of the Rössler system equations

$\frac{dx}{ds}=-(y+z),\quad\frac{dy}{ds}=x+\alpha
y,\quad\frac{dz}{ds}=\beta+xz-\nu z$ (9)

we use its presentation in homogeneous form

${\frac{d}{ds}}\xi(s)+1/5\,{\xi}^{2}-1/5\,\xi\,\rho\,\nu+1/5\,\xi\,\rho\,\alpha+\eta\,\rho+\theta\,\rho=0$
${\frac{d}{ds}}\eta(s)+1/5\,\xi\,\eta-\xi\,\rho-1/5\,\eta\,\rho\,\nu-4/5\,\eta\,\rho\,\alpha=0$
${\frac{d}{ds}}\theta(s)-4/5\,\xi\,\theta+4/5\,\theta\,\rho\,\nu+1/5\,\theta\,\rho\,\alpha-\beta\,{\rho}^{2}=0$
${\frac{d}{ds}}\rho(s)+1/5\,\xi\,\rho-1/5\,{\rho}^{2}\nu+1/5\,{\rho}^{2}\alpha=0,$

where

$x=\frac{\xi}{\rho},\quad y=\frac{\eta}{\rho},\quad z=\frac{\theta}{\rho}.$

The relation between both systems is defined by the conditions

$x(s)=\frac{\xi}{\rho},\quad y(s)=\frac{\eta}{\rho},\quad
z(s)=\frac{\theta}{\rho}.$

Remark that for a given system

$\dot{\xi}=P,\quad\dot{\eta}=Q,\quad\dot{\theta}=R,\quad\dot{\rho}=T$

the condition

$P_{\xi}+Q_{\eta}+R_{\theta}+T_{\rho}=0$

is fulfield.

Such type of the system can be rewriten in the form

$\frac{d^{2}X^{i}}{ds^{2}}+\Pi^{i}_{jk}\frac{dX^{j}}{ds}\frac{dX^{k}}{ds}=0,$

which allow us to consider it as geodesic equations of the space with constant
affine connection.

In our case nonzero components of connection are

$\Pi^{1}_{11}=\frac{1}{5},\quad\Pi^{1}_{14}=\frac{\alpha-\nu}{10},\quad\Pi^{1}_{24}=\frac{1}{2},\quad\Pi^{1}_{34}=\frac{1}{2},\quad\Pi^{2}_{12}=\frac{1}{10},$
$\Pi^{2}_{14}=-\frac{1}{2},\quad\Pi^{2}_{24}=-\frac{4\alpha+\nu}{10},\quad\Pi^{3}_{34}=\frac{4\nu+\alpha}{10},$
$\Pi^{3}_{12}=-\frac{4}{10},\quad\Pi^{3}_{44}=-\beta,\quad\Pi^{4}_{14}=\frac{1}{10},\quad\Pi^{4}_{44}=\frac{\alpha-\nu}{5}.$

The metric of corresponding Riemann space is

$^{8}ds^{2}=-2\,\Pi^{1}_{11}\,Pdx^{2}+2\,\left(-2\,\Pi^{2}_{12}\,Q-2\,\Pi^{3}_{12}\,U\right)dxdy+2\,\left(-2\,\Pi^{1}_{14}\,P-2\,\Pi^{2}_{14}\,Q-2\,\Pi^{4}_{14}\,V\right)dxdu+$
$+2\,\left(-2\,\Pi^{1}_{24}\,P-2\,\Pi^{2}_{24}\,Q\right)dydu+2\,\left(-2\,\Pi^{1}_{34}\,P-2\,\Pi^{3}_{34}\,U\right)dzdu+$
$+\left(-2\,\Pi^{3}_{44}\,U-2\,\Pi^{4}_{44}\,V\right)du^{2}+2\,dxdP+2\,dydQ+2\,dzdU+2\,dudV,$
(10)

where $(P,Q,U,V)$ are an additional coordinates.

Geodesic of the metric (3) for coordinates $(x,y,z,u)$ are

${\frac{d^{2}}{d{s}^{2}}}x(s)+1/5\,\left({\frac{d}{ds}}x(s)\right)^{2}+1/5\,\left({\frac{d}{ds}}u(s)\right)\left({\frac{d}{ds}}x(s)\right)\alpha-1/5\,\left({\frac{d}{ds}}u(s)\right)\left({\frac{d}{ds}}x(s)\right)\nu+\left({\frac{d}{ds}}u(s)\right){\frac{d}{ds}}y(s)+$
$+\left({\frac{d}{ds}}u(s)\right){\frac{d}{ds}}z(s)=0,$
${\frac{d^{2}}{d{s}^{2}}}y(s)+1/5\,\left({\frac{d}{ds}}y(s)\right){\frac{d}{ds}}x(s)-\left({\frac{d}{ds}}u(s)\right){\frac{d}{ds}}x(s)-4/5\,\left({\frac{d}{ds}}u(s)\right)\left({\frac{d}{ds}}y(s)\right)\alpha-1/5\,\left({\frac{d}{ds}}u(s)\right)\left({\frac{d}{ds}}y(s)\right)\nu=0,$
${\frac{d^{2}}{d{s}^{2}}}z(s)-4/5\,\left({\frac{d}{ds}}y(s)\right){\frac{d}{ds}}x(s)+4/5\,\left({\frac{d}{ds}}u(s)\right)\left({\frac{d}{ds}}z(s)\right)\nu+1/5\,\left({\frac{d}{ds}}u(s)\right)\left({\frac{d}{ds}}z(s)\right)\alpha-\beta\,\left({\frac{d}{ds}}u(s)\right)^{2}=0,$
${\frac{d^{2}}{d{s}^{2}}}u(s)+1/5\,\left({\frac{d}{ds}}u(s)\right){\frac{d}{ds}}x(s)+1/5\,\left({\frac{d}{ds}}u(s)\right)^{2}\alpha-1/5\,\left({\frac{d}{ds}}u(s)\right)^{2}\nu=0$

and they have a form of homogeneous Rössler system in the variables

$\xi=\frac{dx}{ds},\quad\eta=\frac{dy}{ds},\theta=\frac{dz}{ds},\quad\rho=\frac{du}{ds}.$

The system of second order differential equations for additional coordinates
can be reduced to the linear system of the first order equations with variable
coefficients

${\frac{d}{dt}}P(t)+A_{1}P(t)+B_{1}Q(t)+C_{1}U(t)+E_{1}V(t)=0,$
${\frac{d}{dt}}Q(t)+A_{2}P(t)+B_{2}Q(t)+C_{2}U(t)+E_{2}V(t)=0,$
${\frac{d}{dt}}U(t)+A_{3}P(t)+B_{3}Q(t)+C_{3}U(t)+E_{3}V(t)=0,$
${\frac{d}{dt}}V(t)+A_{4}P(t)+B_{4}Q(t)+C_{4}U(t)+E_{4}V(t)=0,$

where $A_{i},~{}B_{i},~{}C_{i},~{}E_{i}$ are the functions of the variables
$x,~{}y,~{}z,~{}u)$.

Properties of such type of the systems can be investigated with help of the
Wilczynski invariants.

##  4 Laplace operator

In theory of Riemann spaces the equation

$L\psi=g^{ij}(\frac{\partial^{2}}{\partial x^{i}\partial
x^{j}}-\Gamma^{k}_{ij}\frac{\partial}{\partial x^{k}})\psi(x)=0$ (11)

can be used to the study of the properties of spaces.

For the eight-dimensional space with the metric (3) corresponded the Rössler
system we get the equation on the function
$\psi(x,y,z,u,P,Q,U,V)=\theta(P,Q,U,V)$

$4/5\,{\frac{\partial}{\partial
P}}\theta(P,Q,U,V)+2/5\,P{\frac{\partial^{2}}{\partial{P}^{2}}}\theta(P,Q,U,V)-8/5\,\left({\frac{\partial^{2}}{\partial
P\partial Q}}\theta(P,Q,U,V)\right)U+$
$+2/5\,\left({\frac{\partial^{2}}{\partial P\partial
Q}}\theta(P,Q,U,V)\right)Q-2\,\left({\frac{\partial^{2}}{\partial P\partial
V}}\theta(P,Q,U,V)\right)Q+2/5\,\left({\frac{\partial^{2}}{\partial P\partial
V}}\theta(P,Q,U,V)\right)V+$ $+2\,\left({\frac{\partial^{2}}{\partial
Q\partial V}}\theta(P,Q,U,V)\right)P+2\,\left({\frac{\partial^{2}}{\partial
U\partial V}}\theta(P,Q,U,V)\right)P+8/5\,\left({\frac{\partial}{\partial
V}}\theta(P,Q,U,V)\right)\alpha+$ $+2/5\,\left({\frac{\partial}{\partial
V}}\theta(P,Q,U,V)\right)\nu-2/5\,\left({\frac{\partial^{2}}{\partial{V}^{2}}}\theta(P,Q,U,V)\right)V\nu+2/5\,\left({\frac{\partial^{2}}{\partial{V}^{2}}}\theta(P,Q,U,V)\right)V\alpha+$
$+2/5\,\left({\frac{\partial^{2}}{\partial U\partial
V}}\theta(P,Q,U,V)\right)U\alpha+2/5\,\left({\frac{\partial^{2}}{\partial
Q\partial
V}}\theta(P,Q,U,V)\right)Q\nu+8/5\,\left({\frac{\partial^{2}}{\partial
Q\partial V}}\theta(P,Q,U,V)\right)Q\alpha-$
$-2/5\,\left({\frac{\partial^{2}}{\partial P\partial
V}}\theta(P,Q,U,V)\right)P\nu-2\,\left({\frac{\partial^{2}}{\partial{V}^{2}}}\theta(P,Q,U,V)\right)\beta\,U+8/5\,\left({\frac{\partial^{2}}{\partial
U\partial V}}\theta(P,Q,U,V)\right)U\nu+$
$+2/5\,\left({\frac{\partial^{2}}{\partial P\partial
V}}\theta(P,Q,U,V)\right)P\alpha=0.$

This equation has varies type of particular solutions.

A simplest one is

$\psi(x,y,z,u,P,Q,U,V)={e^{\left(\nu-\alpha\right)P-5\,Q+5\,U+V}}$

at the conditions on parameters of the Rössler system

$\beta={\frac{25}{2}}\,\nu,\quad\alpha=-3/2\,\nu.$

As examples obtained by direct substitutions we get the quadratic solution

$\theta(P,Q,U,V)=$
$=1/18\,{\frac{9\,{Q}^{2}+30\,QP\alpha-18\,QV+25\,{P}^{2}{\alpha}^{2}-30\,P\alpha\,V+9\,{V}^{2}+15\,{P}^{2}+60\,U\alpha\,P+45\,\beta\,UP}{{\it\\_c}_{{2}}}}$

with conditions

$\nu=8/3\,\alpha,\quad\beta=arbitrary.$

Cubic solution

$\theta(P,Q,U,V)=-{\frac{1}{240}}\,{\frac{\left(-96\,\alpha\,{\it
l3}-2880\,{\alpha}^{2}{\it
l1}\right){V}^{3}}{{\beta}^{2}}}-{\frac{1}{240}}\,{\frac{\left(-48\,\beta\,{\it
l3}-1440\,\beta\,{\it l1}\,\alpha\right)U{V}^{2}}{{\beta}^{2}}}+$
$+\left(-{\frac{1}{240}}\,{\frac{\left(60\,\beta\,{\it l3}+1800\,\beta\,{\it
l1}\,\alpha\right){P}^{2}}{{\beta}^{2}}}+{\it l1}\,{U}^{2}\right)V+{\it
k1}\,{U}^{3}+\left({\it l3}\,P+{\it l2}\,Q\right){U}^{2}-$
$-{\frac{1}{240}}\,{\frac{\left(2250\,\beta\,{\it l1}\,\alpha+75\,\beta\,{\it
l3}\right){P}^{2}Q}{{\beta}^{2}}}+\left({\it
m2}\,{Q}^{2}-{\frac{1}{240}}\,{\frac{\left(400\,{\beta}^{2}{\it
l1}+3000\,\beta\,{\it l1}\,\alpha+100\,\beta\,{\it
l3}\right){P}^{2}}{{\beta}^{2}}}\right)U+n{Q}^{3}$

at the condition

$\nu=6\alpha$

and arbitrary coefficients ${\it l},{\it m},{\it k},{\it n}$.

A polynomial solution of degree four

$\theta(P,Q,U,V)=r{Q}^{4}+{\it k1}\,{U}^{4}-2/5\,{\it m3}\,{U}^{3}V+{\it
l2}\,U{Q}^{3}+{\it m2}\,{U}^{2}{Q}^{2}+{\it n2}\,{U}^{3}Q+{\it
m3}\,{U}^{2}{P}^{2}$

at the condition

$\nu=-{\frac{7}{13}}\,\alpha.$

Remark that the properties of of such type of solutions depend on parameters
and may be highly diversified.

More complicated solutions of the Laplace equation can be obtained by
application of the method of $(u,v)$\- transformation developed in the works
of author.

##  5 Eikonal equation

Solutions of eikonal equation

$g{{{}^{i}}{{}^{j}}}\frac{\partial F}{\partial x^{i}}\frac{\partial
F}{\partial x^{j}}=0$ (12)

also gives useful information about the properties of Riemann space.

In particular the condition

$F(x^{1},x^{2},...,x^{i})=0$

where function $F(x^{i})$ satisfies the equation $(\ref{dryuma:eq22})$,
determines $(N-1)$-dimensional hypersurface with normals forming an isotropic
vector field.

For the space with the metric (3) the eikonal equation on the function
$\psi(x,y,z,u,P,Q,U,V)=\eta(P,Q,U,V)$ takes the form

$2/5\,P\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)^{2}-8/5\,\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
Q}}\eta(P,Q,U,V)\right)U+$ $+2/5\,\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
Q}}\eta(P,Q,U,V)\right)Q-2\,\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)Q+$ $+2/5\,\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)P\alpha-2/5\,\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)P\nu+$ $+2/5\,\left({\frac{\partial}{\partial
P}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)V+2\,\left({\frac{\partial}{\partial
Q}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)P-$ $-8/5\,\left({\frac{\partial}{\partial
Q}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)Q\alpha-2/5\,\left({\frac{\partial}{\partial
Q}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)Q\nu+$ $+2\,\left({\frac{\partial}{\partial
U}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)P+8/5\,\left({\frac{\partial}{\partial
U}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)U\nu+$ $+2/5\,\left({\frac{\partial}{\partial
U}}\eta(P,Q,U,V)\right)\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)U\alpha+2/5\,\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)^{2}V\alpha-$ $-2/5\,\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)^{2}V\nu-2\,\left({\frac{\partial}{\partial
V}}\eta(P,Q,U,V)\right)^{2}\beta\,U=0.$ (13)

A simplest solution of this equation is

$\eta(P,Q,U,V)=-{\frac{Q}{\alpha}}+{\frac{V}{\nu-\alpha}}+{\frac{U}{\alpha}}+P$

with condition on the coefficients of the Rössler system

$-5\,\beta\,\alpha-11\,\alpha\,\nu+8\,{\nu}^{2}+3\,{\alpha}^{2}=0.$ (14)

From here we find

$\nu={\frac{11}{16}}\,\alpha+1/16\,\sqrt{25\,{\alpha}^{2}+160\,\beta\,\alpha}$

To provide a more complicated solutions of the equation $(\ref{dryuma:eq22})$
we use the method of $(u,v)$-transformation.

For the sake of convenience we rewrite the equation $(\ref{dryuma:eq22})$ in
the form

$2/5\,x\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)^{2}-8/5\,\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
y}}\eta(x,y,z,p)\right)z+$ $+2/5\,\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
y}}\eta(x,y,z,p)\right)y-2\,\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)y+$ $+2/5\,\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)x\alpha-2/5\,\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)x\nu+$ $+2/5\,\left({\frac{\partial}{\partial
x}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)p+2\,\left({\frac{\partial}{\partial
y}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)x-$ $-8/5\,\left({\frac{\partial}{\partial
y}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)y\alpha-2/5\,\left({\frac{\partial}{\partial
y}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)y\nu+$ $+2\,\left({\frac{\partial}{\partial
z}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)x+8/5\,\left({\frac{\partial}{\partial
z}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)z\nu+$ $+2/5\,\left({\frac{\partial}{\partial
z}}\eta(x,y,z,p)\right)\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)z\alpha+2/5\,\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)^{2}p\alpha-2/5\,\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)^{2}p\nu-$ $-2\,\left({\frac{\partial}{\partial
p}}\eta(x,y,z,p)\right)^{2}\beta\,z=0$ (15)

Now after change of the function and derivatives in accordance with the rules

$\eta(x,y,z,p)\rightarrow u(x,t,z,p),\quad y\rightarrow v(x,t,z,p,$
$\frac{\partial\eta(x,y,z,p)}{\partial x}\rightarrow\frac{\partial
u(x,t,z,p)}{\partial x}-\frac{\frac{\partial u(x,t,z,p)}{\partial
t}}{\frac{\partial v(x,t,z,p))}{\partial t}}\frac{\partial
v(x,t,z,p)}{\partial x},$ $\frac{\partial\eta(x,y,z,p)}{\partial
z}\rightarrow\frac{\partial u(x,t,z,p)}{\partial z}-\frac{\frac{\partial
u(x,t,z,p)}{\partial t}}{\frac{\partial v(x,t,z,p))}{\partial
t}}\frac{\partial v(x,t,z,p)}{\partial z},$
$\frac{\partial\eta(x,y,z,p)}{\partial p}\rightarrow\frac{\partial
u(x,t,z,p)}{\partial p}-\frac{\frac{\partial u(x,t,z,p)}{\partial
t}}{\frac{\partial v(x,t,z,p))}{\partial t}}\frac{\partial
v(x,t,z,p)}{\partial p},$ $\frac{\partial\eta(x,y,z,p)}{\partial
y}\rightarrow\frac{\frac{\partial u(x,t,z,p)}{\partial t}}{\frac{\partial
v(x,t,z,p))}{\partial t}},$

where

$u(x,t,z,p)=t{\frac{\partial}{\partial
t}}\omega(x,t,z,p)-\omega(x,t,z,p),\quad v(x,t,z,p)={\frac{\partial}{\partial
t}}\omega(x,t,z,p)$

we find the equation on the function $\omega(x,t,z,p)$

$\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right)^{2}x-5\,tx{\frac{\partial}{\partial
p}}\omega(x,t,z,p)-p\nu\,\left({\frac{\partial}{\partial
p}}\omega(x,t,z,p)\right)^{2}-5\,\beta\,z\left({\frac{\partial}{\partial
p}}\omega(x,t,z,p)\right)^{2}+$ $+p\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right){\frac{\partial}{\partial
p}}\omega(x,t,z,p)+4\,z\nu\,\left({\frac{\partial}{\partial
z}}\omega(x,t,z,p)\right){\frac{\partial}{\partial
p}}\omega(x,t,z,p)+4\,\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right)zt-$ $-\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right)\left({\frac{\partial}{\partial
t}}\omega(x,t,z,p)\right)t-5\,\left({\frac{\partial}{\partial
t}}\omega(x,t,z,p)\right)\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right){\frac{\partial}{\partial p}}\omega(x,t,z,p)+$
$+p\alpha\,\left({\frac{\partial}{\partial
p}}\omega(x,t,z,p)\right)^{2}+5\,x\left({\frac{\partial}{\partial
z}}\omega(x,t,z,p)\right){\frac{\partial}{\partial
p}}\omega(x,t,z,p)+z\alpha\,\left({\frac{\partial}{\partial
z}}\omega(x,t,z,p)\right){\frac{\partial}{\partial p}}\omega(x,t,z,p)+$
$+4\,t\left({\frac{\partial}{\partial
t}}\omega(x,t,z,p)\right)\alpha\,{\frac{\partial}{\partial
p}}\omega(x,t,z,p)+t\left({\frac{\partial}{\partial
t}}\omega(x,t,z,p)\right)\nu\,{\frac{\partial}{\partial p}}\omega(x,t,z,p)-$
$-x\nu\,\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right){\frac{\partial}{\partial
p}}\omega(x,t,z,p)+x\alpha\,\left({\frac{\partial}{\partial
x}}\omega(x,t,z,p)\right){\frac{\partial}{\partial p}}\omega(x,t,z,p)=0.$ (16)

In spite of the fact that this equation looks not a simple than equation (5)
its particular solutions can be find without trouble.

As example the substitution of the form

$\omega(x,t,z,p)=B(x,t,z)+kpt,$

into the equation (5) lead to expression on the function $B(x,t,z)$

$B(x,t,z)=\alpha\,tx+tz+A(t),$

with arbitrary function $A(t)$ and the conditions on coefficients of the
Rössler system

$\beta=1/5\,{\frac{\alpha\,\left(8+5\,k\right)}{{k}^{2}}},\quad\nu={\frac{\alpha\,\left(1+k\right)}{k}}$

depending from arbitrary parameter $k$.

Using the function $\omega(x,t,z,p)$ we can obtain the solution of the
equation (5) by elimination of the parameter $t$ from the relations

$\eta(x,y,z,p)-t{\frac{\partial}{\partial
t}}\omega(x,t,z,p)+\omega(x,t,z,p)=0,\quad y-{\frac{\partial}{\partial
t}}\omega(x,t,z,p)=0.$

As example at the choice $(A(t)=1+t^{2}$ we get

$\eta(x,y,z,p)=1/4\,{k}^{2}{p}^{2}+\left(-1/2\,yk+1/2\,zk+1/2\,\alpha\,xk\right)p+1/4\,{z}^{2}+\left(1/2\,\alpha\,x-1/2\,y\right)z-1+1/4\,{y}^{2}-1/2\,y\alpha\,x+1/4\,{\alpha}^{2}{x}^{2}.$

At the condition $A(t)=\ln(t)$ we get

$\eta(x,y,z,p)=1-\ln(-\left(-y+\alpha\,x+z+kp\right)^{-1}).$

Remark that parameters $\alpha,\beta,\nu$ in these cases the relation (14) is
satisfied.

To cite another example.

Substitution the expression

$\omega(x,t,z,p)=A(t)x+tz+C(t)p$

into the equation (5) lead to the system of equations on the functions
$A(t),~{}C(t)$

$\left(t\nu\,C(t)+4\,t\alpha\,C(t)-5\,A(t)C(t)-A(t)t\right){\frac{d}{dt}}A(t)+\left(A(t)\right)^{2}-\nu\,A(t)C(t)+\alpha\,A(t)C(t)=0,$
$\left(t\nu\,C(t)+4\,t\alpha\,C(t)-5\,A(t)C(t)-A(t)t\right){\frac{d}{dt}}C(t)+A(t)C(t)-\nu\,\left(C(t)\right)^{2}+\alpha\,\left(C(t)\right)^{2}=0,$
$-5\,\beta\,z\left(C(t)\right)^{2}+\left(5\,z\alpha\,t-5\,A(t)z+5\,z\nu\,t\right)C(t)+3\,A(t)zt=0.$

From this system of equations we find the condition on parameters

$\nu=3/5\,\beta-\alpha$ (17)

and expression on the function $A(t)$

$A(t)=-\beta\,C(t).$

Function $C(t)$ in this case satisfies the equation

$\left(25\,\beta\,C(t)+8\,\beta\,t+15\,t\alpha\right){\frac{d}{dt}}C(t)-8\,\beta\,C(t)+10\,\alpha\,C(t)=0.$

Its solution is defined by the relation

$-t\alpha+\left(C(t)\right)^{-1/2\,{\frac{8\,\beta+15\,\alpha}{-4\,\beta+5\,\alpha}}}{\it\\_C1}\,\alpha-\beta\,C(t)=0.$

Using the expression on the function $C(t)$ we can find the function
$\omega(x,t,z,p)$ and after elimination of the parameter $t$ from
corresponding relations it is possible to get the solution of the eikonal
equation at the condition (17) on parameters of the Rössler system of
equations.

## References

  * [1] Dryuma V., The Riemann Extension in theory of differential equations and their applications,Matematicheskaya fizika, analiz, geometriya, 2003,v.10, No.3, 1–19.
  * [2] Dryuma V., The Invariants, the Riemann and Einstein-Weyl geometries in theory of ODE’s, their applications and all that, New Trends in Integrability and Partial Solvability, p. 115-156, (ed. A.B.Shabat et al.), Kluwer Academic Publishers, (ArXiv: nlin: SI/0303023, 11 March, 2003, 1–37).
  * [3] Dryuma V., Applications of Riemannian and Einstein-Weyl Geometry in the theory of second order ordinary differential equations,Theoretical and Mathematical Physics, 2001, V.128, N 1, 845–855.
  * [4] Valery Dryuma, On theory of surfaces defined by the first order sustems of equations Buletinul Academiei de Stiinte a Republicii Moldova, (matematica), 1(56), 2008, p.161–175.
  * [5] Valery Dryuma, Riemann extensions in theory of the first order systems of differential equations ArXiv:math. DG/0510526 v1 25 Oct 2005, 1–21.
  * [6] Valery Dryuma, Riemann geometry in theory of the first order systems of differential equations ArXiv: 0807.01178 v1 [nlin.SI] 1 Jul 2008, 1–17.


# KB-Plugin: A Plug-and-play Framework for Large Language Models to
Induce Programs over Low-resourced Knowledge Bases

Jiajie Zhang1 , Shulin Cao1, Linmei Hu2, Ling Feng1, Lei Hou1, Juanzi Li1
1Tsinghua University 2Beijing Institute of Technology
<EMAIL_ADDRESS>
Corresponding author.

###### Abstract

Program induction (PI) has become a promising paradigm for using knowledge
bases (KBs) to help large language models (LLMs) answer complex knowledge-
intensive questions. Nonetheless, PI typically relies on a large number of
parallel question-program pairs to make the LLM aware of the schema of the
given KB, and is thus challenging for many low-resourced KBs that lack
annotated data. To this end, we propose KB-Plugin, a plug-and-play framework
that enables LLMs to induce programs over any low-resourced KB. Firstly, KB-
Plugin adopts self-supervised learning to encode the detailed schema
information of a given KB into a pluggable module, namely schema plugin.
Secondly, KB-Plugin utilizes abundant annotated data from a rich-resourced KB
to train another pluggable module, namely PI plugin, which can help the LLM
extract question-relevant schema information from the schema plugin of any KB
and utilize this information to induce programs over this KB. Experiments on
five heterogeneous KBQA datasets show that KB-Plugin achieves better or
comparable performance with 25$\times$ smaller backbone LLM compared to SoTA
PI methods for low-resourced KBs, and even approaches the performance of
supervised methods. Our code and data are available at https://github.com/THU-
KEG/KB-Plugin.

KB-Plugin: A Plug-and-play Framework for Large Language Models to
Induce Programs over Low-resourced Knowledge Bases

Jiajie Zhang1 , Shulin Cao1, Linmei Hu2, Ling Feng1, Lei Hou1††thanks:
Corresponding author., Juanzi Li1 1Tsinghua University 2Beijing Institute of
Technology<EMAIL_ADDRESS>

Figure 1: Illustration of KB-Plugin. By simply plugging the schema plugin of a
KB and the PI plugin, the LLM is injected with the schema information of this
KB and the ability to induce programs over it.

##  1 Introduction

Recently, the usage of knowledge bases (KBs) as external resources to assist
large language models (LLMs) Brown et al. (2020); Zhao et al. (2023) in
answering complex knowledge-intensive questions has gained increasing study
Pan et al. (2023); Li et al. (2023b); Jiang et al. (2023). Among various
methods, program induction (PI) has emerged as a promising paradigm due to its
good interpretability and capacity to support complex reasoning operations Cao
et al. (2022a); Gu et al. (2023); Li et al. (2023b). Given a KB, PI methods
employ LLMs to convert a question into a multi-step program (e.g., KoPL Cao et
al. (2022a) and S-expression Su et al. (2016)), whose execution against the KB
produces the answer. Despite strong capacity, most PI methods rely on
individual training for each KB using a large number of manually annotated
question-program pairs Xie et al. (2022); Li et al. (2023b); Luo et al.
(2023). As for many low-resourced KBs that lack program annotations, how to
enable LLMs to utilize their knowledge via PI remains a challenging problem.

Recent studies Cao et al. (2022b); Li et al. (2023a) have indicated that the
mapping from questions to program sketches (i.e., composed functions without
arguments, such as Find$\rightarrow$ Relate$\rightarrow$ FilterConcept)
primarily correlates with language compositional structures and is thus
transferable across KBs. Hence the main challenge for PI over low-resourced
KBs is to determine the argument for each function Gu and Su (2022), which
requires LLMs to link natural language in a question to corresponding schema
items (i.e., pre-defined relations and concepts) in the KB (e.g., in Fig 1,
the relation “part of network” and the concept “rail network” are arguments of
function Relate and FilterConcept, respectively), so it is important to
provide LLMs adequate information of each schema item. A straightforward
approach is to directly feed all the schema information to the LLM via a
prompt. However, the broad schema of KBs and limited context windows of LLMs
make this infeasible Li et al. (2023a).

Regarding the above challenges, we are inspired by recent studies that claim
that the parameters of LLMs can encode task-specific knowledge Saxena et al.
(2022); Moiseev et al. (2022); Wang et al. (2022). Our basic idea is to encode
schema information of a KB into the parameters of a pluggable module (e.g.,
LoRA Hu et al. (2022)), namely schema plugin, and use another pluggable
module, namely PI plugin, to help the LLM capture question-relevant schema
information from the schema plugin and utilize this information to induce
programs. As illustrated in Fig. 1, by simply plugging the schema plugin of a
KB and the PI plugin, the LLM is injected with the schema information of this
KB and the ability to induce programs over it. We name this framework KB-
Plugin. To implement KB-Plugin, there remain two key problems: (1) By what
task can sufficient information about each schema item in a KB be encoded into
its schema plugin? (2) Without annotated data from the low-resource KBs, how
can the PI plugin learn to extract and utilize question-relevant schema
information from their schema plugins to induce programs over these KBs?

To address the above problems, we propose a novel plugin learning and transfer
framework. First, inspired by prior studies Bordes et al. (2013); Lin et al.
(2015) which show that schema items in a KB can be well represented by fact
triples involving them, we propose to learn schema plugins via a self-
supervised triple completion task. Specifically, given a KB, we plug a schema
plugin into the LLM and tune the plugin to enable the LLM to complete relevant
triples for each schema item in the KB. In this way, the detailed schema
information can be encoded into this schema plugin. As for PI plugin learning,
inspired by Cao et al. (2022b), we utilize abundant program annotations from a
rich-resourced KB. Specifically, we use this KB to generate multiple KBs with
different schemas via alias replacement and train a schema plugin for each of
them. Given a training question, we plug these schema plugins alone with the
PI plugin into the LLM in turn and train the PI plugin to make the LLM
generate the correct program whose arguments conform to the currently plugged
schema plugin. In this way, the PI plugin is forced to learn the skills of
extracting and utilizing question-relevant schema information from the plugged
schema plugin for PI over the corresponding KB. Besides, since the PI plugin
is trained to be compatible with different schema plugins, it can be directly
transferred to other low-resourced KBs and generalize well with their schema
plugins, even if most schema items in these KBs are unseen during its
training.

In experiments, we take Wikidata-based KQA Pro as the rich-resourced KB to
train the PI plugin and evaluate our framework on three Freebase-based
datasets (WebQSP, GraphQ, and GrailQA) and two domain-specific datasets
(MetaQA for movie domain and SoAyBench for academic domain). The results show
that KB-Plugin achieves better or comparable performance with 25$\times$
smaller backbone LLM compared to SoTA PI methods for low-resource KBs. On
GraphQ, GrailQA, and MetaQA, KB-Plugin even surpasses the performance of
several supervised methods.

Our contributions include: (1) proposing KB-Plugin, a novel plug-and-play
framework that enables LLMs to induce programs over any low-resourced KB; (2)
empirical validation of the efficacy of KB-Plugin through comprehensive
experiments on five heterogeneous KBQA datasets.

##  2 Related Work

Low-resourced Program Induction.  Recently, there have emerged three types of
PI methods for low-resourced KBs that lack program annotations, but each of
them has limitations: (1) Few-shot program generation methods Gu et al.
(2023); Li et al. (2023a) utilize in-context learning ability of LLMs to
induce programs with a handful of demonstrations. However, they can only
determine function arguments based on the schema item names due to limited
context windows, so they face challenges in distinguishing similar schema
items. They also suffer from long inference time due to excessive LLM calls or
executing a vast number of potential programs; (2) Few-shot data generation
methods Li et al. (2023c) also employ in-context learning with LLMs to convert
automatically sampled programs into questions, and train a smaller PI model
using the generated question-program pairs. Nonetheless, the generated
questions may not align with programs and often lack diversity due to the
limited number of program templates; (3) Similar to us, program transfer
methods Cao et al. (2022b) also leverage program annotations from a rich-
resourced KB to aid PI for low-resourced KBs. However, they mainly focus on
program sketch transfer and perform poorly without fine-tuning using annotated
question-answer pairs from low-resourced KBs to adapt to their schemas. While
KB-plugin obviates the reliance on any annotated data from low-resourced KBs,
thereby enabling LLMs to easily utilize their knowledge.

Plug-and-Play Modules for LLMs.  In recent years, various parameter-efficient
modules have been proposed to adapt LLMs to different downstream tasks Lester
et al. (2021); Hu et al. (2022); Li and Liang (2021); Pfeiffer et al. (2021) .
These modules show plug-and-play characteristics and can inject task-specific
knowledge and skills into LLMs Xiao et al. (2023); Zhang et al. (2023). Some
researchers also found that pluggable modules for similar tasks encode
knowledge and skills into the parametric space in similar ways Qin et al.
(2021); Su et al. (2022), providing basic rationality for the transferability
of our PI plugin.

##  3 Problem Formulation

In this section, we first provide some necessary definitions and then
formulate our task.

Knowledge Base. A knowledge base (KB) can be formalized as
$\mathcal{KB}=\\{\mathcal{C},\mathcal{E},\mathcal{R},\mathcal{T}\\}$, where
$\mathcal{C}$, $\mathcal{E}$, $\mathcal{R}$ and $\mathcal{T}$ represent the
sets of concepts, entities, relations and fact triples, respectively.
Specifically, $\mathcal{R}=\\{r_{e},r_{c}\\}\cup\mathcal{R}_{l}$, where
$r_{e}$ is “instance of”, $r_{c}$ is “subclass of”, and $\mathcal{R}_{l}$ is
the set of other general relations. Correspondingly, $\mathcal{T}$ can be
divided into there disjoint subsets: (1) “instance of” triples
$\mathcal{T}_{e}=\\{(e,r_{e},c)|e\in\mathcal{E},c\in\mathcal{C}\\}$; (2)
“subclass of” triples
$\mathcal{T}_{c}=\\{(c_{i},r_{c},c_{j})|c_{i},c_{j}\in\mathcal{C}\\}$; (3)
relational triples
$\mathcal{T}_{l}=\\{(e_{i},r,e_{j})|e_{i},e_{j}\in\mathcal{E},r\in\mathcal{R}_{l}\\}$.
Elements in $\mathcal{C}$ and $\mathcal{R}$ are also called the schema items
of $\mathcal{KB}$.

Program Induction. Given a KB $\mathcal{KB}$ and a natural language question
$x=\left\langle w_{1},w_{2},\cdots,w_{|x|}\right\rangle$, program induction
(PI) aims to convert $x$ into a program $y$, which would return the correct
answer when executed against $\mathcal{KB}$. Formally, $y$ is composed of
functions that take a specific type of arguments, and can be serialized as
$y=\left\langle
f_{1}(arg_{1}),\cdots,f_{t}(arg_{t}),\cdots,f_{|y|}(arg_{|y|})\right\rangle,f_{t}\in\mathcal{F},arg_{t}\in\mathcal{E}\cup\mathcal{C}\cup\mathcal{R}\cup\\{\emptyset\\}$.
Here, $\mathcal{F}$ is a set of pre-defined functions that cover basic
reasoning operations on KBs. In this work, we use KoPL Cao et al. (2022a) as
our programming language.

Task Formulation. Suppose we have access to (1) source KB $\mathcal{KB}^{S}$
and source domain data
$\mathcal{D}^{S}=\\{(x_{i}^{S},y_{i}^{S})\\}_{i=1}^{n^{S}}$, which are
question-program pairs for $\mathcal{KB}^{S}$; (2) target KB
$\mathcal{KB}^{T}$, which is low-resourced and has no annotated data. The goal
is to learn a PI model $M_{PI}^{T}$ that can translate a question $x^{T}$ for
$\mathcal{KB}^{T}$ into program $y^{T}$, whose execution on $\mathcal{KB}^{T}$
produces the correct answer.

##  4 Methodology

Figure 2: Overview of our plugin learning and transfer framework: (a) Generate
multiple source KBs with different schemas and augmented source domain data
via alias replacement; (b) Learn an individual schema plugin for each source
KB and the target KB via self-supervised schema-relevant triple completion
task; (c) Train the PI plugin by inducing program for each source KB when
plugging it into the LLM along with the corresponding schema plugin. (d)
Transfer the PI plugin by plugging it into the LLM with the schema plugin of
the target KB and inducing programs over the target KB with constrained
decoding.

As mentioned in the introduction, to enable a LLM $M$ to induce programs over
low-resourced $\mathcal{KB}^{T}$, KB-Plugin learns two types of pluggable
modules for $M$: (1) KB-specific schema plugin $m_{sc}$, which stores
information of schema items of a given KB within its parameters; (2) KB-
transferable PI plugin $m_{PI}$, which encodes the skill of inducing programs
over any KB by extracting and utilizing question-relevant schema information
from the schema plugin of this KB. It is trained with $\mathcal{KB}^{S}$ and
$\mathcal{D^{S}}$ but can be directly transferred to $\mathcal{KB}^{T}$. The
final PI model for $\mathcal{KB}^{T}$ can be formulated as

$M_{PI}^{T}=\mathrm{plug}(M,\\{m_{sc}^{T},m_{PI}\\}),$ (1)

where $m_{sc}^{T}$ is the schema plugin of $\mathcal{KB}^{T}$ and
$\mathrm{plug}(M,\\{\cdot\\})$ means plugging the plugins in $\\{\cdot\\}$
into $M$.

In the following, we will first introduce the architecture of two types of
plugins, then present our plugin learning and transfer framework.

###  4.1 Plugin Architecture

A host of studies have demonstrated that knowledge and skills can be
encapsulated within the parameters of LLMs Saxena et al. (2022); Moiseev et
al. (2022); Wang et al. (2022). Inspired by this, we implement both schema
plugin and PI plugin with LoRA Hu et al. (2022), a popular type of pluggable
module for LLMs with a few trainable parameters.

Specifically, let $L_{M}$ be the set of weight matrices in the self-attention
modules and MLP modules of a LLM $M$. For each $W_{i}\in\mathbb{R}^{d\times
k}$ in $L_{M}$, LoRA modifies its forward pass from $h=W_{i}x$ to
$h=(W_{i}+A_{i}B_{i})x$, where $A_{i}\in\mathbb{R}^{d\times r}$ and
$B_{i}\in\mathbb{R}^{r\times k}$ are two matrices with rank $r\ll\min(d,k)$. A
LoRA plugin $m_{j}$ is thus defined as

$m_{j}=\\{(A_{i}^{m_{j}},B_{i}^{m_{j}})|W_{i}\in L_{M}\\},$ (2)

and $\mathrm{plug}(M,\\{m_{1},\dots,m_{N}\\})$ means replacing all $W_{i}\in
L_{M}$ with $W_{i}+\sum_{j=1}^{N}A_{i}^{m_{j}}B_{i}^{m_{j}}$. If we train
$M^{\prime}=\mathrm{plug}(\mathrm{fz}(M),\\{\mathrm{fz}(m_{1}),\dots,\mathrm{fz}(m_{N-1}),m_{N}\\})$
on a certain task, where $\mathrm{fz}(\cdot)$ represents parameter freezing,
knowledge and skills related to this task will be encoded within $m_{N}$.
Although other parameter-efficient pluggable modules such as prefix-tuning Li
and Liang (2021) can also serve as our plugin modules, the advantages of LoRA
are that it does not increase input length or inference latency.

###  4.2 Plugin Learning and Transfer Framework

There are two primary challenges for learning schema plugins and the PI
plugin: (1) How to encode sufficient information about each schema item of a
KB into a schema plugin? (2) How to ensure that the PI plugin can extract and
utilize useful schema information for program induction from schema plugins of
different KBs, instead of ignoring the schema plugin entirely, directly
learning to induce program over source KB during training, and consequently
losing transferability?

To handle these challenges, we propose a novel plugin learning and transfer
framework, which is illustrated in Fig. 2 and contains four steps: (1)
Generate multiple source KBs $\mathcal{KB}^{S_{1}},\dots,\mathcal{KB}^{S_{N}}$
with different schemas and augmented data
$\mathcal{D}^{S}_{a}=\\{(x_{j}^{S},y_{j}^{S_{1}},\dots,y_{j}^{S_{N}})\\}_{j=1}^{n^{S}}$
based on $\mathcal{KB}^{S}$ and $D^{S}$ via alias replacement, where
$y_{j}^{S_{i}}$ is the golden program for question $x_{j}^{S}$ on
$\mathcal{KB}^{S_{i}}$; (2) Learn individual schema plugin $m_{sc}^{S_{i}}$
for each $\mathcal{KB}^{S_{i}}$ via self-supervised schema-relevant triple-
completion task; (3) Train PI plugin $m_{PI}$ by requiring
$M_{PI}^{S_{1}},\dots,M_{PI}^{S_{N}}$ to generate
$y_{j}^{S_{1}},\dots,y_{j}^{S_{N}}$ given $x_{j}^{S}$, respectively, where
$M_{PI}^{S_{i}}=\mathrm{Plug}(\mathrm{fz}({M}),\\{\mathrm{fz}(m_{sc}^{S_{i}}),m_{PI}\\})$,
so that $m_{PI}$ is forced to extract and utilize schema information from each
$m_{sc}^{S_{i}}$; (4) Learn schema plugin $m_{sc}^{T}$ for $\mathcal{KB}^{T}$
using the same method in (2) and take
$M_{PI}^{T}=\mathrm{plug}(M,\\{m_{sc}^{T},m_{PI}\\})$ as the final PI model
for $\mathcal{KB}^{T}$. We will introduce each step in detail in the
following.

####  4.2.1 KB Generation and Data Augmentation

We utilize the aliases of each schema item to generate multiple KBs with
different schemas based on
$\mathcal{KB}^{S}=\\{\mathcal{C}^{S},\mathcal{E}^{S},\mathcal{R}^{S},\mathcal{T}^{S}\\}$.
As shown in Fig. 2(a), for each schema item
$v\in\mathcal{C}^{S}\cup\mathcal{R}^{S}$, we replace $v$ with $v_{i}$, a
randomly chosen alias of $v$, and record $a_{i}(v)=v_{i}$. For example, the
concept “basketball team” can be replaced with “basket club” and the relation
“member of sports team” can be replaced with “plays for”. Relevant triples in
$\mathcal{T}^{S}$ are also modified with the same alias. In this way,
$\mathcal{KB}^{S_{i}}$ that has a different schema than $\mathcal{KB}^{S}$ is
created. In practice, we let $\mathcal{KB}^{S_{1}}=\mathcal{KB}^{S}$ and
repeat above process $N-1$ times to generate
$\mathcal{KB}^{S_{2}},\dots,\mathcal{KB}^{S_{N}}$.

Similarly, for each question-program pair
$(x^{S}_{j},y^{S}_{j})\in\mathcal{D}^{S}$, suppose $y^{S}_{j}=\left\langle
f_{1}(arg_{1}),\cdots,f_{t}(arg_{t}),\cdots,f_{|y_{j}^{S}|}(arg_{|y_{j}^{S}|})\right\rangle$,
we replace every $arg_{t}\in\mathcal{C}^{S}\cup\mathcal{R}^{S}$ with
$a_{i}(arg_{t})$ to obtain $y^{S_{i}}_{j}$, which is the correct program for
$x_{j}^{S}$ executable on $\mathcal{KB}^{S_{i}}$. We repeat the process for
$\mathcal{KB}^{S_{1}},\dots,\mathcal{KB}^{S_{N}}$ to obtain augmented data
$\mathcal{D}^{S}_{a}=\\{(x_{j}^{S},y_{j}^{S_{1}},\dots,y_{j}^{S_{N}})\\}_{j=1}^{n^{S}}$.

####  4.2.2 Learning of Schema Plugin

Many studies about knowledge graph embedding show that the information of
schema items in a KB can be represented by not only their names but also
triples containing them Bordes et al. (2013); Lv et al. (2018). Inspired by
this, we propose to encode schema information into schema plugins via a self-
supervised triple completion task. As illustrated in Fig. 2(b), to learn the
schema plugin $m_{sc}$ for a given KB
$\mathcal{KB}=\\{\mathcal{C},\mathcal{E},\mathcal{R},\mathcal{T}\\}$, where
$\mathcal{T}=\mathcal{T}_{e}\cup\mathcal{T}_{c}\cup\mathcal{T}_{l}$, we train
$M_{sc}=\mathrm{Plug}(\mathrm{fz}(M),m_{sc})$ to complete relevant triples for
each concept and relation in $\mathcal{KB}$ in sequence-to-sequence form as
follows.

First, for each concept $c\in\mathcal{C}$, we require $M_{sc}$ to complete
relevant “instance of” triples to aggregate the semantic features of entities
belonging to $c$. Specifically, we sample $K$ triples $(e_{k},\textit{instance
of},c)$ from $\mathcal{T}_{e}$ (see Appendix B for detailed sampling
strategy), and use each sampled triple to construct two pairs of verbalized
queries and answer as the inputs and expected outputs for $M_{sc}$:

  * •

“$\langle e_{k}\rangle$ || instance of” $\rightarrow$ “$\langle c\rangle$”;

  * •

“$\langle c\rangle$ || contains instance” $\rightarrow$ “$\langle
e_{k}\rangle$”.

Here, $\langle e_{k}\rangle$ and $\langle c\rangle$ means filling in the names
of $e_{k}$ and $c$, respectively.

Besides, the information of a concept is also related to its sub- and super-
concepts. Therefore, for each triple $(c_{i},\textit{subclass
of},c_{j})\in\mathcal{T}_{c}$, we also construct two queries with answers for
$M_{sc}$:

  * •

“$\langle c_{i}\rangle$ || subclass of” $\rightarrow$ “$\langle
c_{j}\rangle$”;

  * •

“$\langle c_{j}\rangle$ || contains subclass” $\rightarrow$ “$\langle
c_{i}\rangle$”.

Finally, the information of a relation can be learned from its name and the
elements connected by it. Therefore, for each $r\in R_{l}$, we sample $K$
triples $(e_{i},r,e_{j})$ from $\mathcal{T}_{l}$, choose $c_{i}$, $c_{j}$ such
that
$(e_{i},\mathrm{instance\\_of},c_{i}),(e_{j},\mathrm{instance\\_of},c_{j})\in
T_{e}$, and use each $(e_{i},c_{i},r,e_{j},c_{j})$ to construct three queries
with answers:

  * •

“$\langle e_{i}\rangle$ | $\langle c_{i}\rangle$ || $\langle r\rangle$ | forward” $\rightarrow$ “$\langle c_{j}\rangle$ | $\langle e_{j}\rangle$”;

  * •

“$\langle e_{j}\rangle$ | $\langle c_{j}\rangle$ || $\langle r\rangle$ | backward” $\rightarrow$ “$\langle c_{i}\rangle$ | $\langle e_{i}\rangle$”;

  * •

“$\langle e_{i}\rangle$ | $\langle c_{i}\rangle$ || what relation || $\langle c_{j}\rangle$ | $\langle e_{j}\rangle$” $\rightarrow$ “$\langle r\rangle$”.

We empirically find that including $c_{i}$, $c_{j}$ benefits the information
encoding for both concepts and relations.

Let the set of all generated queries and answers be
$D_{sc}=\\{(q_{i},a_{i})\\}_{i=1}^{l}$, then $m_{sc}$ is trained to minimize

$\mathcal{L}_{sc}=-\sum\limits_{(q_{i},a_{i})\in D_{sc}}\log P(a_{i}|q_{i}),$
(3)

where $P(a_{i}|q_{i})$ is the likelihood of $M_{sc}$ generating $a_{i}$ given
$q_{i}$, defined by token-level cross entropy. Note that the learning of
$m_{sc}$ does not rely on any additional data except the KB itself, so we can
train a schema plugin for any KB.

####  4.2.3 Learning of PI Plugin

As illustrated in Fig. 2(c), to learn the PI plugin $m_{PI}$, we first train
individual schema plugin $m_{sc}^{S_{i}}$ for each $\mathcal{KB}^{S_{i}}$.
After that, given $(x_{j}^{S},y_{j}^{S_{1}},\dots,y_{j}^{S_{N}})\in
D_{a}^{S}$, where $x_{i}^{S}$ is a question and $y_{j}^{S_{i}}$ is the golden
program for $x_{j}^{S}$ on $\mathcal{KB}^{S_{i}}$, we train $m_{PI}$ by
feeding $x_{i}^{S}$ to $M_{PI}^{S_{1}},\dots,M_{PI}^{S_{N}}$ and requiring
them to generate $y_{j}^{S_{1}},\dots,y_{j}^{S_{N}}$, respectively. Here,
$M_{PI}^{S_{i}}=\mathrm{Plug}(\mathrm{fz}({M}),\\{\mathrm{fz}(m_{sc}^{S_{i}}),m_{PI}\\})$.
The overall objective can be formulated as:

$\mathcal{L}_{PI}=-\sum_{(x_{j}^{S},y_{j}^{S_{1}},\dots,y_{j}^{S_{N}})\in
D_{a}^{S}}\sum_{i=1}^{N}\log P_{i}(y_{j}^{S_{i}}|x_{j}^{S}),$ (4)

where $P_{i}(y_{j}^{S_{i}}|x_{j}^{S})$ is the likelihood of $M_{PI}^{S_{i}}$
generating $y_{j}^{S_{i}}$ given $x_{j}^{S}$, defined by token-level cross
entropy. To generate programs conforming to different schemas given the same
question, $m_{PI}$ must learn to (1) choose correct functions according to the
compositional structure of the question; (2) extract and utilize question-
relevant schema information for argument determination from the corresponding
schema plugin, because it is the only difference among
$M_{PI}^{S_{1}},\dots,M_{PI}^{S_{N}}$.

####  4.2.4 Plugin Transfer

Once the PI plugin $m_{PI}$ is trained, we directly transfer it to
$\mathcal{KB}^{T}$ as in Fig 2 (d), and let
$M_{PI}^{T}=\mathrm{plug}(M,\\{m_{sc}^{T},m_{PI}\\})$ be the PI model for
$\mathcal{KB}^{T}$. Here, $m_{sc}^{T}$ is the trained schema plugin for
$\mathcal{KB}^{T}$ using the method in Sec. 4.2.2. Since $m_{sc}^{T}$ and
$m_{sc}^{S_{i}}$ are trained with the same tasks, we expect that they encode
schema information into their parameters in similar ways Qin et al. (2021); Su
et al. (2022), so $m_{PI}$ can also extract schema information from
$m_{sc}^{T}$ to help PI over $\mathcal{KB}^{T}$. Besides, to guarantee
$M_{PI}^{T}$ generating valid programs which do not cause execution error or
return an empty answer, we adopt constrained decoding, i.e., after
$M_{PI}^{T}$ generates $f_{1}(arg_{1}),\dots,f_{t}(arg_{t})$, we enumerate all
the valid $f_{t+1}(arg_{t+1})$ following the method of Gu et al. (2023) and
restrict $M_{PI}^{T}$ to only generate one of them. More details are in
Appendix C. We also use beam search to retain top-k programs during decoding
to provide $M_{PI}^{T}$ with a more global view.

##  5 Experiments

###  5.1 Datasets

Source Domain. We use KQA Pro Cao et al. (2022a) as the source domain
datasets. It provides 117,970 questions with diverse compositional structures
and corresponding programs based on a subset of Wikidata Vrandecic and
Krötzsch (2014).

Target Domain.  We use WebQSP Yih et al. (2016), GraphQ Su et al. (2016),
GrailQA Gu et al. (2021), MetaQA Zhang et al. (2018) and SoAyBench Anonymous
(2024) as the target domain datasets. Among them, WebQSP, GraphQ, and GrailQA
are based on Freebase Bollacker et al. (2008). Their KBs contain a large
number of schema items and can evaluate the effectiveness of KB-Plugin for
large-scale KBs. MetaQA and SoAyBench are two datasets in movie and academic
domains, respectively, and can evaluate the effectiveness for specific
domains. For MetaQA, since most of the relations in its KB have been covered
by KQA Pro, we remove these relations and relevant question-program pairs from
KQA Pro to avoid data leakage. For SoAyBench which is originally a tool-using
dataset based on Aminer Tang et al. (2008) APIs, we construct its KB by
collecting relevant data from these APIs. Table 1 shows the statistics of
these datasets and their overlap with source KBs generated from KQA Pro. Most
schema items in the target KBs are unseen in source KBs and most test cases
also involve unseen schema items.

Dataset | $|\mathcal{R}|$ | $|\mathcal{R}_{u}|$ | $|\mathcal{C}|$ | $|\mathcal{C}_{u}|$ | $|\mathcal{D}^{\text{test}}|$ | $|\mathcal{D}_{u}^{\text{test}}|$
---|---|---|---|---|---|---
KQA Pro | 1209 | - | 794 | - | - | -
WebQSP | 412 | 296 | 446 | 363 | 1639 | 1083
GraphQ | 9569 | 8931 | 7298 | 7004 | 2395 | 2340
GrailQA(dev) | 3938 | 3524 | 2018 | 1868 | 6763 | 6578
GrailQA(test) | 3938 | 3524 | 2018 | 1868 | 13231 | -
MetaQA | 9 | 9 | 9 | 3 | 39093 | 39093
SoAyBench | 17 | 11 | 5 | 3 | 792 | 756

Table 1: Statistics for source and target domain datasets and their overlaps
with 16 source KBs generated from KQA Pro. $|\mathcal{R}|$ / $|\mathcal{C}|$
denotes the number of relations / concepts in their KBs. $|\mathcal{R}_{u}|$ /
$|\mathcal{C}_{u}|$ denotes the number of relations / concepts unseen in the
source KBs. $|\mathcal{D}^{\text{test}}|$ and
$|\mathcal{D}_{u}^{\text{test}}|$ denotes the numbers of test cases and test
cases that involve unseen schema items, respectively.

###  5.2 Baselines

For WebQSP, GraphQ, GrailQA, and MetaQA, we mainly compare KB-Plugin with low-
resourced PI methods including (1) few-shot program generation methods Pangu
Gu et al. (2023) and KB-BINDER Li et al. (2023a); (2) few-shot data generation
method APS Li et al. (2023c); (3) program transfer method ProgramTrans Cao et
al. (2022b), where we adopt its results without fine-tuning on target KBs for
fair comparison. In addition, we also provide the results of several
representative supervised models for comparison.

For SoAyBench, we choose tool-using methods that were evaluated on it as
baselines, including DFSDT Qin et al. (2023) and SoAy Anonymous (2024). These
methods solve questions by prompting LLMs to call Aminer APIs in specific
orders via in-context learning. Their processes of determining the composition
of APIs and filling in arguments for each API can also be viewed as program
induction.

We provide detailed descriptions of all the baselines and our evaluation
metrics in Appendix D.1.

###  5.3 Implementation Details

In experiments, we use Llama2-7B Touvron et al. (2023) as the backbone LLM of
KB-Plugin and set the rank $r$ of LoRA to 16. The number of parameters of each
plugin is consequently 40M, which is extremely lightweight. The number of
generated source KBs is set to 16 to balance performance and training
efficiency. The sampling number $K$ in schema plugin learning is set to be
500, 500, 50, 100, 3000, and 1000 for KQA Pro, WebQSP, GraphQ, GrailQA,
MetaQA, and SoAyBench, respectively, to limit the size of the constructed data
for schema plugin learning. We use beam size 5 for all experiments. More
details can be found in Appendix D.2.

Method | WebQSP | GraphQ | GrailQA
---|---|---|---
Test | Dev
Supervised
QGG | 74.0 | - | 36.7 | -
BERT+Ranking | - | 25.0 | 58.0 | -
ArcaneQA | 75.6 | 31.8 | 73.7 | 76.8
RnG-KBQA | 75.6 | - | 74.4 | 76.9
Low-resourced
ProgramTrans | 53.8∗ | - | - | -
APS | 51.1 | - | 57.7 | 62.1
KB-BINDER | 53.2 | 39.5 | 56.0 | -
Pangu | 54.5 | 43.3 | 62.7 | -
KB-Plugin | 57.2 / 61.1∗ | 49.5 | 62.7 | 65.0
w/o schema plugin | 41.0 | 42.8 | - | 57.5
w/ $m_{sc}^{S_{0}}$ | 48.0 | 37.9 | - | 51.0
Table 2: F1 results on WebQSP, GraphQ, and GrailQA. ∗ means using oracle topic entities.  Method | 1-hop | 2-hop | 3-hop
---|---|---|---
Supervised
KV-Mem | 96.2 | 82.7 | 48.9
PullNet | 97.0 | 99.9 | 91.4
EmbedKGQA | 97.5 | 98.8 | 94.8
TransferNet | 97.5 | 100.0 | 100.0
Low-resourced
KB-BINDER | 93.5 | 99.6 | 96.4
KB-Plugin | 97.1 | 100.0 | 99.3
w/o schema plugin | 92.6 | 99.0 | 98.9
w/ $m_{sc}^{S_{0}}$ | 90.4 | 93.6 | 88.6
Table 3: Hit@1 results on MetaQA. Method | Acc
---|---
DFSDT (gpt-3.5-turbo) | 45.7
DFSDT (gpt-4) | 59.7
SoAy (gpt-3.5-turbo) | 67.7
SoAy (gpt-4) | 88.7
KB-Plugin | 90.8
w/o schema plugin | 70.8
w/ $m_{sc}^{S_{0}}$ | 64.0
Table 4: Accuracy results on SoAyBench. Dataset | Method | $\mathcal{D}_{\text{seen}}^{\text{test}}$ | $\mathcal{D}_{\text{unseen}}^{\text{test}}$
---|---|---|---
WebQSP | KB-Plugin | 64.9 | 53.3
w/o schema plugin | 47.6 | 37.6
Gain | +17.4 | +15.7
GraphQ | KB-Plugin | 40.0∗ | 49.7
w/o schema plugin | 70.9∗ | 42.2
Gain | -30.9∗ | +7.5
GrailQA-dev | KB-Plugin | 69.0 | 64.8
w/o schema plugin | 64.9 | 57.3
Gain | +4.1 | +7.5
Table 5: F1 Results of KB-Plugin with and without schema plugin.
$\mathcal{D}_{\text{unseen}}^{\text{test}}$ and
$\mathcal{D}_{\text{seen}}^{\text{test}}$ denote the sets of test cases that
involve and do not involve schema items unseen in the source KBs,
respectively. $*$ means the results may not be indicative since there are only
55 cases in $\mathcal{D}_{\text{seen}}^{\text{test}}$ of GraphQ.

Question I | Which airport to fly into Rome?
---|---
Pangu | Find(Rome) Relate(tourist attractions) (✗)
KB-Plugin w/o schema plugin | Find(Rome) Relate(country) FilterConcept(sovereign state) (✗)
KB-Plugin | Find(Rome) Relate(transport terminus) FilterConcept(airport) (✓)
Relevant Triples | (London, transport terminus, Luton airport), (London, instance of, citytown),
(Luton airport, instance of, airport)
Question II | What role did Paul Mccartney play in the Beatles?
Pangu |  Find(Paul Mccartney) Relate(instruments played) (✗)
KB-Plugin |  Find(Beatles) Relate(member) Find(Paul Mccartney) ReverseRelate(member) And() Relate(role) (✓)
Source Domain Data Pair | What is Jane Lynch’s role in Glee?
Find(Glee) Relate(starring) Find(Jane Lynch) ReverseRelate(starring) And()
Relate(character role)

Table 6: Two typical questions from the test set of WebQSP that KB-Plugin
succeeds while Pangu fails. The incorrect functions and arguments are marked
as red, while the correct ones are marked as green.

###  5.4 Main Results

The results are presented in Table 2, 3 and 4. Compared with Pangu, the SoTA
PI method for low-resourced KBs, KB-Plugin improves the F1 score by 2.7% and
6.2% on WebQSP and GraphQ, respectively, and achieves comparable performance
on GrailQA, despite Pangu using 25$\times$ larger model (175B Codex) and 100
annotated examples from each dataset. Moreover, Pangu needs to call Codex
hundreds of times for a question to score each candidate program, while our
model selects the optimal program via beam search, which is significantly
faster and less costly. Besides, since ProgramTrans, KB-BINDER, and Pangu all
link questions to schema items according to their names only, the superiority
of KB-Plugin also demonstrates the benefits of aggregating additional schema
information from relevant triples via schema plugin learning. KB-Plugin even
surpasses several supervised models on GraphQ and GrailQA, which demand
training using thousands of annotated samples from target KBs, showing the
effectiveness of transferring prior knowledge from rich-resourced KBs.

On MetaQA and SoAyBench, KB-Plugin outperforms all the low-resourced baselines
even though they use more powerful LLMs (i.e., Codex, gpt-3.5-turbo, and
gpt-4), indicating that our framework also performs well for domain-specific
KBs. In particular, KB-Plugin achieves strong performance on par with
supervised SoTAs on MetaQA even if it does not see any target relations from
the source domain.

###  5.5 Ablation Study

To demonstrate the effect of schema plugins, we remove them from our
framework, i.e., we directly train a PI plugin using the source domain data
and transfer it to the target KBs without training any schema plugins.
According to Table 2, 3, 4, and 5, the performance of KB-Plugin without schema
plugins is severely degraded, especially on the test cases that involve schema
items unseen in the source KBs. The experimental results illustrate that (1)
direct PI transfer is difficult due to the substantial difference between the
schemas of source and target KBs; (2) schema plugins of target KBs effectively
encode adequate schema information via the triple completion task, and the PI
plugin can extract and utilize question-relevant schema information from these
schema plugins even though it is never trained with them. In addition, if we
adopt the schema plugin of a source KB, e.g., $m_{sc}^{S_{0}}$, for the target
KBs, the performance of KB-Plugin also drops heavily, showing the necessity of
using matched schema plugin.

Figure 3: KB-Plugin performance with different numbers of generated source
KBs.

To show the rationality of our PI plugin learning method, we evaluate the
performance of PI plugins trained with different numbers of generated source
KBs on WebQSP, GraphQ, and GrailQA, and present the results in Fig. 3. The PI
plugin trained with only one source KB performs poorly, implying that it
ignores the schema plugin entirely and directly learns PI over this source KB.
Once there emerges a new source KB with a different schema, the performance of
the trained PI plugin increases substantially, and there is an apparent trend
that the performance will increase with more generated source KBs. These
results prove that training the PI plugin over multiple source KBs succeeds in
forcing the PI plugin to learn to extract and utilize schema information from
different schema plugins, and the learned skill can be transferred to target
KBs.

###  5.6 Case Study

To better showcase the advantages of KB-Plugin over in-context learning PI
methods, we present a case comparison between KB-Plugin and Pangu in Table 6.
Question I shows the effect of schema plugin learning and utilization. Both
Pangu and KB-Plugin without schema plugin struggle to predict the correct
relation “transport terminus” because it is unseen in the demo examples or
source KBs. The complete KB-Plugin, however, effectively encodes the
information that “transport terminus” is a possible relation between
“citytown” and “airport” into the schema plugin via completing relevant
triples, and succeeds in predicting this relation by utilizing above
information. Question II demonstrates the benefits of harnessing abundant
program annotations from the source domain, where Pangu produces a program
with incorrect function composition because none of its demo examples has a
similar compositional structure, while KB-Plugin induces the correct program
by utilizing prior knowledge learned from the source domain. Further analysis
can be found in Appendix E and F.

##  6 Conclusion

We propose KB-Plugin, a plug-and-play framework that enables LLMs to induce
programs over any low-resourced KB by learning two types of pluggable modules:
KB-specific schema plugin and KB-transferable PI plugin. KB-Plugin achieves
better or comparable performance on five heterogeneous KBQA datasets with much
smaller backbone LLMs compared to SoTA PI methods for low-resourced KBs,
demonstrating its effectiveness for both large-scale and domain-specific KBs.
Ablation study and case study also prove the rationality and further showcase
the advantage of KB-plugin.

##  7 Limitations

We discuss several limitations of KB-Plugin in this section: (1) In the
experiments, we only adopt Llama2-7B as our backbone model due to limited
computing resources. Actually, KB-Plugin is model-agnostic and can also be
applied to more language models with various sizes and architectures. (2) KB-
Plugin requires that the source domain dataset covers questions with diverse
various compositional structures, and performs relatively poorly for questions
whose compositional structures are unseen in the source domain dataset though
they are rare (see Appendix E for details). Future research can focus on
improving the transferability of KB-Plugin across compositional structures. In
practice, we can also continue to train the PI plugin using some self-training
methods such as EGST Li et al. (2023c) to adapt to these questions. (3) In
this work, since both training and evaluation of KB-Plugin require annotated
KBQA datasets, we can only take a single dataset KQA Pro as the source dataset
and take other datasets as the target datasets, which may limit the upper
bounds of KB-Plugin. In the realistic scenario where we need to apply KB-
Plugin for a new KB, we can take all these KBQA datasets as the source domain
datasets so that the trained source schema plugins would be more diverse and
the trained PI plugin would also have stronger transferability and
generalizability.

##  8 Ethical Considerations

Though our framework (as well as other PI methods) can effectively reduce the
probability of LLMs generating inaccurate answers when faced with questions
involving uncommon knowledge, it may still make mistakes if the induced
programs are incorrect. In addition, there is a risk of being hacked through
targeted means such as injecting harmful or nonfactual knowledge into the KBs.
Hence additional care and protective measures should be taken if our framework
is deployed in user-facing applications.

All the datasets and encyclopedias used in this work are publicly published
with permissible licenses.

## References

  * Anonymous (2024) Anonymous. 2024.  Soay: A service-oriented apis applying framework of large language models.
  * Bollacker et al. (2008) Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008_ , pages 1247–1250. ACM.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , pages 2787–2795.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Cao et al. (2022a) Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022a.  KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 6101–6119. Association for Computational Linguistics.
  * Cao et al. (2022b) Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. 2022b.  Program transfer for answering complex questions over knowledge bases.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 8128–8140. Association for Computational Linguistics.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _CoRR_ , abs/2107.03374.
  * Gu et al. (2023) Yu Gu, Xiang Deng, and Yu Su. 2023.  Don’t generate, discriminate: A proposal for grounding language models to real-world environments.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 4928–4949. Association for Computational Linguistics.
  * Gu et al. (2021) Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021.  Beyond I.I.D.: three levels of generalization for question answering on knowledge bases.  In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , pages 3477–3488. ACM / IW3C2.
  * Gu and Su (2022) Yu Gu and Yu Su. 2022.  Arcaneqa: Dynamic program induction and contextualized encoding for knowledge base question answering.  In _Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022_ , pages 1718–1731. International Committee on Computational Linguistics.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spacy: Industrial-strength natural language processing in python.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Jiang et al. (2023) Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023.  Structgpt: A general framework for large language model to reason over structured data.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , pages 9237–9251. Association for Computational Linguistics.
  * Lan and Jiang (2020) Yunshi Lan and Jing Jiang. 2020.  Query graph generation for answering multi-hop complex questions from knowledge bases.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 969–974. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 3045–3059. Association for Computational Linguistics.
  * Li et al. (2023a) Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023a.  Few-shot in-context learning for knowledge base question answering.  _CoRR_ , abs/2305.01750.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 4582–4597. Association for Computational Linguistics.
  * Li et al. (2023b) Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq R. Joty, and Soujanya Poria. 2023b.  Chain of knowledge: A framework for grounding large language models with structured knowledge bases.  _CoRR_ , abs/2305.13269.
  * Li et al. (2023c) Zhenyu Li, Sunqi Fan, Yu Gu, Xiuxing Li, Zhichao Duan, Bowen Dong, Ning Liu, and Jianyong Wang. 2023c.  Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering.  _CoRR_ , abs/2308.12060.
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.  Learning entity and relation embeddings for knowledge graph completion.  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA_ , pages 2181–2187. AAAI Press.
  * Luo et al. (2023) Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, and Wei Lin. 2023.  Chatkbqa: A generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models.  _CoRR_ , abs/2310.08975.
  * Lv et al. (2018) Xin Lv, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018.  Differentiating concepts and instances for knowledge graph embedding.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 1971–1979. Association for Computational Linguistics.
  * Miller et al. (2016) Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016.  Key-value memory networks for directly reading documents.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 1400–1409. The Association for Computational Linguistics.
  * Moiseev et al. (2022) Fedor Moiseev, Zhe Dong, Enrique Alfonseca, and Martin Jaggi. 2022.  SKILL: structured knowledge infusion for large language models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 1581–1588. Association for Computational Linguistics.
  * Nie et al. (2022) Lunyiu Nie, Shulin Cao, Jiaxin Shi, Jiuding Sun, Qi Tian, Lei Hou, Juanzi Li, and Jidong Zhai. 2022.  Graphq IR: unifying the semantic parsing of graph query languages with one intermediate representation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 5848–5865. Association for Computational Linguistics.
  * Pan et al. (2023) Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2023.  Unifying large language models and knowledge graphs: A roadmap.  _CoRR_ , abs/2306.08302.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021.  Adapterfusion: Non-destructive task composition for transfer learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 487–503. Association for Computational Linguistics.
  * Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  _CoRR_ , abs/2307.16789.
  * Qin et al. (2021) Yujia Qin, Xiaozhi Wang, YuSheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. 2021.  Exploring low-dimensional intrinsic task subspace via prompt tuning.  _CoRR_ , abs/2110.07867.
  * Saxena et al. (2022) Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022.  Sequence-to-sequence knowledge graph completion and question answering.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2814–2828. Association for Computational Linguistics.
  * Saxena et al. (2020) Apoorv Saxena, Aditay Tripathi, and Partha P. Talukdar. 2020.  Improving multi-hop question answering over knowledge graphs using knowledge base embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4498–4507. Association for Computational Linguistics.
  * Shi et al. (2021) Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, and Hanwang Zhang. 2021.  Transfernet: An effective and transparent framework for multi-hop question answering over relation graph.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 4149–4158. Association for Computational Linguistics.
  * Su et al. (2016) Yu Su, Huan Sun, Brian M. Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. 2016.  On generating characteristic-rich question sets for QA evaluation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 562–572. The Association for Computational Linguistics.
  * Su et al. (2022) Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022.  On transferability of prompt tuning for natural language processing.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 3949–3969. Association for Computational Linguistics.
  * Sun et al. (2019) Haitian Sun, Tania Bedrax-Weiss, and William W. Cohen. 2019.  Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2380–2390. Association for Computational Linguistics.
  * Tang et al. (2008) Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008.  Arnetminer: extraction and mining of academic social networks.  In _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008_ , pages 990–998. ACM.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Wang et al. (2022) Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022.  Finding skill neurons in pre-trained transformer-based language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 11132–11152. Association for Computational Linguistics.
  * Xiao et al. (2023) Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Xiangyang Li, Zhonghua Li, Zhao Cao, and Maosong Sun. 2023.  Plug-and-play document modules for pre-trained models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 15713–15729. Association for Computational Linguistics.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 602–631. Association for Computational Linguistics.
  * Yao (2015) Xuchen Yao. 2015.  Lean question answering over freebase from scratch.  In _NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015_ , pages 66–70. The Association for Computational Linguistics.
  * Ye et al. (2022) Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022.  RNG-KBQA: generation augmented iterative ranking for knowledge base question answering.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 6032–6043. Association for Computational Linguistics.
  * Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016.  The value of semantic parse labeling for knowledge base question answering.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers_. The Association for Computer Linguistics.
  * Zhang et al. (2018) Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018.  Variational reasoning for question answering with knowledge graph.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 6069–6076. AAAI Press.
  * Zhang et al. (2023) Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2023.  Plug-and-play knowledge injection for pre-trained language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 10641–10658. Association for Computational Linguistics.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A survey of large language models.  _CoRR_ , abs/2303.18223.

##  Appendix A Details of KoPL Functions

Function | Input$\times$Args $\rightarrow$ Output | Description
---|---|---
Find | $E\times\emptyset\rightarrow E$ | find an entity from the KB
FindAll | $\emptyset\times\emptyset\rightarrow E^{\prime}$ | return all entities in the KB
Relate | $(E\cup E^{\prime})\times R\rightarrow E^{\prime}$ | a single hop along a relation
ReverseRelate | $(E\cup E^{\prime})\times R\rightarrow E^{\prime}$ | a reverse hop along a relation
FilterConcept | $E^{\prime}\times C\rightarrow E^{\prime}$ | return entities in a concept
And/Or | $(E^{\prime},E^{\prime})\times\emptyset\rightarrow E^{\prime}$ | intersection/union of two sets
Argmax/Argmin | $E^{\prime}\times R\rightarrow E^{\prime}$ | superlative aggregations
LT/LE/GT/GE | $E\times R\rightarrow E^{\prime}$ |  $<$ / $\leq$ / $>$ / $\geq$
Count | $E^{\prime}\times\emptyset\rightarrow N$ | set cardinality

Table 7: KoPL functions used in this work. $E$: entity; $E^{\prime}$: a set of
entities; $R$: relation; $C$: concept; $N$: integer.

We list KoPL functions used in this work in Table 7. We make some
modifications to the original Cao et al. (2022a) for conciseness. Except Find
taking topic entities as the argument, other functions either have no
arguments or take schema items (i.e., concepts or relations) as their
arguments.

##  Appendix B Triple Sampling Strategy

Let the given KB be
$\mathcal{KB}=\\{\mathcal{C},\mathcal{E},\mathcal{R},\mathcal{T}\\}$, where
$\mathcal{T}=\mathcal{T}_{e}\cup\mathcal{T}_{c}\cup\mathcal{T}_{l}$. For each
$e\in\mathcal{E}$, let $\operatorname{cnt}(e)$ be its popularity (i.e., the
number of its occurrences in $\mathcal{KB}$).

When sampling “instance of” triples for a concept $c\in\mathcal{C}$, we hope
the sampled triples contain representative entities belonging to $c$, so we
sort all $(e_{k},\textit{instance of},c)\in T_{e}$ in descending order of
$\operatorname{cnt}(e_{k})$ and select the first $K$ triples.

When sampling relational triples for a relation $r\in\mathcal{R}_{l}$, we take
both representativeness and diversity into account. Therefore, we sort all
$(e_{i},r,e_{j})\in T_{l}$ in descending order of
$\min(\operatorname{cnt}(e_{i}),\operatorname{cnt}(e_{j}))$ and select the
first $K$ triples.

##  Appendix C Details of Constrained Decoding

In constrained decoding, after $M_{PI}^{T}$ generates $t$ function chunks
$f_{1}(arg_{1}),\dots,f_{t}(arg_{t})$, we enumerate all admissible
$f_{t+1}(arg_{t+1})$ as the candidate set $P_{t+1}$ following the definition
of KoPL functions in Table 7, and constrain $M_{PI}^{T}$ to continue
generating one of these candidate or generating the
$\langle\mathtt{EOS}\rangle$ token to end the decoding process.

Specifically, let $E_{\text{topic}}$ be the set of topic entities in the
question obtained using off-the-shelf entity linkers 111Entity linking is not
a major challenge for PI, and exhaustive fuzzy string matching Yao (2015)
suffices to achieve a reasonable performance.. At $t=0$, we enumerate
Find($e$) for each $e\in E_{\text{topic}}$ as a candidate in $P_{1}$.
Specially, around 5% of questions in GraphQ and GrailQA do not have a topic
entity (e.g., “Who is the heaviest film director?" from GrailQA, whose target
program is FindAll() FilterConcept(director)SelectAmong(weight kg). For these
questions, we follow Pangu Gu et al. (2023) to start constrained decoding from
FindAll()FilterConcept($c$), where $c$ is a topic concept provided by Gu and
Su (2022).

When $t>0$, we execute the current program $p_{t}=\langle
f_{1}(arg_{1}),\dots,f_{t}(arg_{t})\rangle$ to get its denotation (i.e., a set
of entities) and also the concepts, forward relations, and backward relations
that are reachable from the denotation. For each concept $c$, we enumerate
FilterConcept($c$) as a candidate in $P_{t+1}$. For each forward relation $r$,
we enumerate Relate($r$) as a candidate. For each backward relation $r$, we
enumerate ReverseRelate($r$) as a candidate, and also include LT($r$),
LE($r$), GT($r$), and GE($r$) in $P_{t+1}$ if the denotation of $p_{t}$ is a
numerical value such as a quantity or a date. In addition, candidates with
superlatives can be enumerated as Argmax($r$) and Argmin($r$). Also, Count()
can always be included to $P_{t+1}$. If there are multiple topic entities, we
enumerate Find($e^{\prime}$) as a candidate to add a new branch, where
$e^{\prime}\in E_{\text{topic}}$ is a topic entity not in $p_{t}$. When
$p_{t}$ contains multiple branches, we enumerate Or() and And() as candidates
to merge the last two branches.

##  Appendix D Experimental Setup

###  D.1 Details of Baselines and Evaluation Metrics

The details of our baselines are as follows:

Pangu Gu et al. (2023) utilizes potent LLM Codex Chen et al. (2021) to produce
programs in a step-wise fashion via in-context learning. At each step, it
first extends existing programs into new valid candidates by enumerating all
possible next functions with arguments, then scores each candidate using Codex
with several demonstrations and retains the top-k candidates.

KB-BINDER Li et al. (2023a) first lets Codex generate several "draft" programs
for a given question by imitating a few examples, then grounds the arguments
in the drafts to the target KB using similarity search to produce hundreds of
refined programs. The final answer is decided by the majority vote after
executing all these refined programs.

Automatic Program Sampling (APS) Li et al. (2023c) utilizes
gpt-3.5-turbo222https://platform.openai.com/docs/models/gpt-3-5 to translate
automatically sampled programs based on a handful of templates into
corresponding questions via in-context learning, and subsequently fine-tune a
RnG-KBQA Ye et al. (2022) PI model using the generated question-program pairs.

ProgramTrans Cao et al. (2022b) is a program transfer method that first uses a
seq2seq sketch parser to translate the question into a program sketch, then
uses an argument parser to search suitable argument from the KB for each
function. We adopt its results without fine-tuning on the target KBs for fair
comparison.

DFSDT Qin et al. (2023) is the SoTA method for general tool using. To solve a
question, it employs a LLM to call suitable tool APIs in depth-first order. At
each step, the LLM can either (1) call the next API to proceed along a
promising path or (2) undo the current call and call another API to expand a
new path.

SoAy Anonymous (2024) is the SoTA method on SoAyBench. Given a question, it
employs a LLM to first select the most suitable plan (i.e., API combination)
from a candidate pool, then write a Python program with branching and looping
structure following the plan to call APIs to get the answer.

Supervised Methods. For WebQSP, GraphQ, GrailQA, and MetaQA, we also provide
the fully supervised results of several representative models for comparison,
including QGG Lan and Jiang (2020), BERT+Ranking Gu et al. (2021), ArcnaeQA Gu
and Su (2022), RnG-KBQA Ye et al. (2022), KV-MemMiller et al. (2016), PullNet
Sun et al. (2019), EmbedKGQA Saxena et al. (2020) and TransferNet Shi et al.
(2021).

Evalution Metrics. Following these baselines, we use F1 for WebQSP, GraphQ,
and GrailQA, use Hit@1 for MetaQA, and use Accuracy for SoAyBench.

###  D.2 Implementation Details

We train the schema plugins of the source and target KBs for 3 epochs and 1
epoch, respectively. The batch size and learning rate are set to be 128 and
1e-5, respectively. Besides, we train the PI plugin for 1 epoch with batch
size 16 and learning rate 1e-5. For WebQSP, GraphQ, and GrailQA, we use the
same off-the-shelf entity-linker as Pangu to find topic entities; For MetaQA,
we follow our baselines to use oracle topic entities; For SoAyBench, we find
topic entities using spaCy Honnibal et al. (2020).

##  Appendix E Analysis about Question Compositional Structures

Dataset | Seen | Unseen
---|---|---
Num | EM | F1 | Num | EM | F1
GraphQ | 2148 | 71.0 | 52.8 | 247 | 15.4 | 20.4
GrailQA | 6433 | 79.9 | 67.4 | 330 | 10.0 | 16.4
Table 8: Performance of KB-Plugin on test cases whose compositional structures
are seen and unseen in the source dataset KQA Pro. EM means the exact match of
program sketch.

For GraphQ and GrailQA, we translate their SPARQL programs to KoPL programs
using GraphQ Trans Nie et al. (2022) and analyze the performance of KB-Plugin
on the test cases whose question compositional structures (identified by
program sketches) are seen and unseen in the source domain dataset KQA Pro,
respectively. From the results in Table 8 we can see that (1) KQA Pro covers
most of question compositional structures in the target dataset; (2) KB-Plugin
correctly predicts the program sketches for over 70% questions whose
compositional structures are seen in the source domain dataset, implying that
the mapping from questions to program sketches is largely independent of KB
schemas and transferable across KBs, which is consistent with the findings of
Cao et al. (2022b) and Li et al. (2023a); (3) KB-Plugin performs poorly on the
questions with unseen compositional structures though they are relatively
rare, indicating that more advanced transfer techniques across compositional
structures remains to be explored.

##  Appendix F Error Analysis

We analyze 100 incorrect predictions (i.e., F1<1) randomly sampled from the
dev set of GrailQA. The major errors are predicting wrong schema items (36%).
Specially, when facing several schema items with only subtle differences,
e.g., “publisher”(reverse) v.s. “game version published”, KB-plugin tends to
prefer to choose the shorter one due to the inherent defects of beam search.
Besides, 21% errors are due to a wrong termination check where the model
misses the last relation or predicts an additional function. There are also 5%
wrong function predictions. Apart from the above errors caused by our model,
27% errors are caused by unidentified or wrongly identified topic entities
during entity linking, 9% errors are due to ambiguous or wrong annotations,
and the remaining 2% errors are due to the incompletion of KBs.