PREPRINT

The Location of Optimal Object Colors with More Than Two Transitions

Scott A. Burns, University of Illinois at Urbana-Champaign, Urbana, IL 61801.

Email<EMAIL_ADDRESS>
May 14, 2021

Preprint Status Notice: This preprint contains substantially less material
than the version that was eventually accepted for publication in Color
Research and Application. In particular, most of the material contained in the
online supplementary documentation (Reference 27) as been incorporated into
the final version at the request of the reviewers. In addition, an analysis
was performed on the more modern “physiologically-relevant” color-matching
functions that are transformed from cone fundamentals. This additional
analysis is included at the end of the final peer-reviewed version.

Abstract: The chromaticity diagram associated with the 1931 $2^{\circ}$ CIE
color matching functions is shown to be slightly non-convex. While having no
impact on practical colorimetric computations, the non-convexity does have a
significant impact on the shape of some optimal object color reflectance
distributions associated with the outer surface of the object color solid.
Instead of the usual two-transition Schrödinger form, many optimal colors
exhibit higher transition counts. A linear programming formulation is
developed and is used to locate where these higher-transition optimal object
colors reside on the object color solid surface. The regions of higher
transition count appear to have a point-symmetric complementary structure.

1\. Introduction

The color-matching functions (CMFs), when plotted as a set of three-
dimensional vectors in tristimulus space, form an origin-based cone enveloping
all possible colors. Nested within this cone is the object color solid (OCS).
It contains all “object colors” that can be produced by a non-fluorescent
reflecting surface (i.e., having a spectral reflectance distribution between
zero and one), under the action of some specified illuminant. The outer
surface of the OCS contains the “optimal object colors.” Historically, the
optimal colors are believed to be associated with rectangular spectral
reflectance distributions having only values of zero and one and at most two
sudden transitions, known as Schrödinger colors.1 This belief has been echoed
in color science publications for many decades.2-8

West and Brill showed that one condition for all optimal colors to have the
Schrödinger form is that the chromaticity diagram (the so-called horseshoe
diagram) be convex.9 Most historic proofs of optimal colors having the
Schrödinger form assume this convexity, either explicitly or implicitly.

Recently, Davis demonstrated that the CIE 1931 CMFs produce a non-convex
chromaticity diagram.10 His presentation is highly mathematical and intended
for a very specialized audience. The first part of this paper will demonstrate
this non-convexity in a simpler way, using only a single MATLAB function.

The non-convexity is very slight and has no impact on practical colorimetric
computations. It does, however, have a significant impact on the shape of some
optimal object color reflectance distributions. Instead of the usual two-
transition Schrödinger form, many optimal colors exhibit higher transition
counts. A linear programming approach will be developed and be used to compute
the reflectance distribution of optimal object colors, located anywhere on the
OCS surface. It will be used to map regions where these higher-transition
optimal colors reside on the OCS surface.

2\. Convexity of the Chromaticity Diagram

Here we are considering the 1931 $2^{\circ}$ CIE Standard Colorimetric
Observer Data (CMFs) that span 360 nm to 830 nm in 1 nm intervals.11,12 We
designate this 471$\times$3 matrix with the symbol $A$. The chromaticity
coordinates associated with $A$ are found by dividing each row of $A$ by the
row sum. The first two columns are then identified as $x$ and $y$ chromaticity
coordinates. In MATLAB this is accomplished with the statement xy =
A(:,1:2)./sum(A,2);, where $xy$ is a 471$\times$2 matrix of chromaticity
coordinates, producing the well-known “horseshoe” chromaticity diagram.

We now use the notion of a convex hull to assess the convexity of the
chromaticity diagram. When applied to a discrete set of points in two
dimensions, the convex hull is the smallest convex polygon that contains all
of the points. If it turns out that any of the chromaticity coordinates of the
horseshoe reside in the interior of the convex hull, then the horseshoe is not
convex.

The convex hull is generated in MATLAB with the statement k = convhull(xy).
The vector $k$ contains the indices of rows of $xy$ that form the convex hull
boundary. It turns out that only 161 of the 471 rows of $xy$ contribute to
this boundary. Figure 1 depicts graphically where they are. The black dots are
those that form the convex hull boundary and the red dots are those that fall
inside of it.

Figure 1: Chromaticity diagram (black solid line) and convex hull enveloping
it (blue dashed line). Red dots are points on chromaticity diagram that fall
slightly inside the convex hull, indicating a slight non-convexity of the
horseshoe for portions above 574 nm and also a small portion running 435 nm to
453 nm.

Close examination of one of the red dots (figure inset) shows that the
horseshoe polygon (solid black line) falls slightly inside the convex hull
polygon (blue dashed line) in some areas. There are two main regions of non-
convexity: several large pockets of wavelengths above 574 nm, and another
segment running 435 nm to 453 nm. There are also many pockets of non-convexity
at both extreme ends of the visible range.

The non-convexity persists with lower-resolution versions of the 1 nm CMFs.
Figure 2 shows the non-convexity of 5 nm and 10 nm versions of CMFs that are
commonly used in colorimetric computations. It is evident that the non-
convexity is present here as well.

Figure 2: Convex hull analysis of lower-resolution versions of the 1 nm CMFs,
having 5 nm wavelength intervals (left) and 10 nm intervals (right). Red dots
show chromaticity coordinates falling slightly inside the convex hull
boundary.

This non-convexity is extremely small. In chromaticity space units, the
horseshoe and convex hull polygons are only about 0.00005 units apart in the
Figure 1 inset. This very small difference will have no practical impact on
typical colorimetric calculations. It turns out, however, that it does have a
significant impact on the shape of optimal object colors, as discussed in the
next sections.

3\. Computing Optimal Object Color Reflectance Curves

The standard approach to characterizing the object color solid is to start
with two-transition Schrödinger reflectance curves and map them into
tristimulus space to form the boundary of the solid.2-5,13 An alternate
approach is to make no assumption of the shape of the optimal reflectance
curve, but seek points on the surface of the OCS by optimization methods. In
this paper, we follow the latter approach using linear programming (LP).
Historically, LP has been used in colorimetric calculations to compute metamer
mismatch volumes,14-17 to design illuminants to achieve desired rendering
effects,18 and to compute subtractive mixture recipes for colorants based on
the Kubelka-Munk theory.19-21

The idea of using LP to identify the OCS surface is not new. In his 1969 paper
on limits of metamerism (i.e., metamer mismatch volumes), Allen mentions in
passing (without any details) that LP could be used to compute MacAdam limits
(optimal object colors).14 Ohta and Wyszecki repeat Allen’s claim in their
1975 paper on metamer mismatch volumes, but only go so far as to state that it
would be achieved with a simplified version of some of the equations they
present.15 In 2010, Li et al explicitly use LP to determine the shape of
cross-sections of the OCS by first identifying key locations on each cross
section by LP, and then by using LP repeatedly to fill in the curves between
these key locations.22 Although their LP formulation directly computes the
shape of the reflectance curves as a by-product of the computation, the focus
of their paper is instead on identifying the shape of the OCS in tristimulus
space. They apparently did not encounter (or perhaps notice) any cases where
the two-transition assumption was violated (other than inevitable artifacts of
discretization at the two transition locations, which they did mention).

The LP presented in this paper uses only one basic colorimetric equation:

${XYZ}_{W}={A_{W}}^{\prime}\;\rho.$ (1)

In this equation, ${XYZ}_{W}$ is a $3{\times}1$ vector of illuminant-W-
referenced tristimulus values, and $\rho$ is an $n{\times}1$ vector of
reflectance values (0-1), where $n$ is the number of wavelength intervals used
to discretize continuous spectral distributions over the visible range. Matrix
$A_{W}$ is an $n{\times}3$ matrix of illuminant-W-referenced color matching
functions, computed as $\overline{W}A$, where $\overline{W}$ is an
$n{\times}n$ matrix with illuminant $W$ on the main diagonal and zeros
elsewhere. Prime denotes matrix transpose. As mentioned in Section 2, matrix
$A$ is an $n{\times}3$ array of color matching functions (CMFs) by columns,
$A=[\overline{x},\overline{y},\overline{z}]$. It is assumed that the
illuminant has been normalized so that the scalar product
${\overline{y}}\,^{\prime}\cdot W$ equals 100. For the standard 1931
$2^{\circ}$ CIE standard CMFs with 1 nm resolution, $n$ is 471.

Note that the reflective/transmissive version of tristimulus values are being
used here (as opposed to the emissive version), so all tristimulus values are
referenced to an illuminant, hence the $W$ subscript. Even though different
subscripts may be used to identify special tristimulus values in this paper,
it is understood that all of them are $W$-referenced. For example, the white
point, ${XYZ}_{wp}$ is the tristimulus triplet formed as the three row sums of
${A_{W}}^{\prime}$ (three column sums of $A_{W}$).

Figure 3 presents a cut-away view of the interior of the OCS. Four key
locations are identified, (1) the white point, ${XYZ}_{wp}$, (2) the 50% gray
halfway point between the origin and the white point,
${XYZ}_{\mathit{50\%}}={XYZ}_{wp}/2$, (3) an arbitrary point within the OCS,
${XYZ}_{targ}$, which defines a direction with respect to
${XYZ}_{\mathit{50\%}}$, and (4) the point of intersection of a ray in that
direction with the OCS outer surface, ${XYZ}_{opt}$.

Figure 3: An object color solid with key locations identified.

A spherical coordinate system is established, based at
${XYZ}_{\mathit{50\%}}$. Every ${XYZ}_{opt}$ on the surface is associated with
a unique $(\theta,\varphi)$ pair. The spherical coordinate system will come in
handy in a subsequent section, when it comes time to plot a dense array of
optimal color data as viewed from the interior of the OCS.

Consider the vector ${XYZ}_{targ}-{XYZ}_{\mathit{50\%}}$. When this vector is
based at ${XYZ}_{\mathit{50\%}}$, it points in the direction of the ray shown
in Figure 3. Next, consider a scalar, $c$, which serves to magnify the vector
and locate an arbitrary point, ${XYZ}$, along the ray:

${XYZ}={XYZ}_{\mathit{50\%}}+c({XYZ}_{targ}-{XYZ}_{\mathit{50\%}}).$ (2)

Substituting for ${XYZ}$ using Equation 1, we now have an equation in terms of
our unknowns, $\rho$ and $c$. We seek to maximize $c$ to move as far along the
ray as possible, while constraining $\rho$ to be between 0 and 1:

$\begin{split}\underset{\rho,c}{\mathsf{max}}\;\;&c\\\
\mathsf{s.t.}\;\;&{A_{W}}^{\prime}\;\rho={XYZ}_{\mathit{50\%}}+c({XYZ}_{targ}-{XYZ}_{\mathit{50\%}})\\\
&0\leq\rho\leq 1.\end{split}$ (3)

This is a linear program. To solve this LP with MATLAB’s “linprog” function,
it needs to be recast in standard form:

$\begin{split}\underset{x}{\mathsf{min}}\;\;&f^{\prime}x\\\
\mathsf{s.t.}\;\;&A_{eq}x=b_{eq}\\\ &x_{lb}\leq x\leq x_{ub},\end{split}$ (4)

where $f,x,b_{eq},x_{lb}$, and $x_{ub}$ are all vectors and $A_{eq}$ is a
matrix. This recasting of Equation 3 is accomplished by concatenating $\rho$
and $c$ into a single 472$\times$1 vector and by minimizing $-c$.

$\begin{split}\underset{\rho,c}{\mathsf{min}}\;\;&-c\\\
\mathsf{s.t.}\;\;&\left[\begin{array}[]{c|c}{A_{W}}\,^{\prime}&{XYZ}_{\mathit{50\%}}-{XYZ}_{targ}\end{array}\right]\begin{Bmatrix}\rho\\\
c\end{Bmatrix}={XYZ}_{\mathit{50\%}}\\\ &\;0\leq\rho\leq 1.\end{split}$ (5)

The following MATLAB statements produce the optimal color reflectance
distribution, $\rho$, corresponding to a point on the surface of the OCS that
is the intersection of a ray emanating from ${XYZ}_{\mathit{50\%}}$, passing
through ${XYZ}_{targ}$:

>
> >> options = optimoptions(’linprog’, ’Algorithm’, ’dual-simplex’,...
>  ’OptimalityTolerance’, 1e-9, ’ConstraintTolerance’, 3e-9);
> >> x = linprog([zeros(471,1);-1], [ ] ,[ ], [Aw’,XYZ50-XYZtarg], XYZ50,...
>  [zeros(471,1);-Inf], [ones(471,1);Inf], options);
> >> rho = x(1:471);

The two tolerances used internally by the LP solver are tightened up
considerably. These values were determined experimentally to avoid premature
termination of the LP. Note that the MATLAB Optimization Toolbox is required
for function linprog.

The vast majority of reflectance curves generated by this LP will have the
expected two-transition structure. For example, using a target of
XYZtarg=[10;40;30], the LP gives the reflectance shown on the left side of
Figure 4. In some cases, however, more than two transitions are obtained. For
example, if the target is changed to XYZtarg=[49.1;40.3;25.0], a four-
transition reflectance curve is produced, as shown on the right side of Figure
4.

Figure 4: Reflectance curves produced by the LP for two different targets.

It is natural to question the higher-transition result. Is it just an artifact
of the LP solution process? Is it really on the outer surface of the OCS? In
other words, is there a two-transition solution along the same ray that is
even farther from ${XYZ}_{\mathit{50\%}}$, which would make the high-
transition solution an interior point of the OCS, and not optimal? The LP
solution process involves pivots into and out of an active basis, using
tolerances to decide if optimality conditions have been numerically satisfied.
Is it possible that the higher-transition solutions are just barely sub-
optimal points that happen to satisfy the optimality conditions numerically
within the specified tolerances? These issues will be addressed in subsequent
sections, but first, a survey of how often and where the high-transition LP
solutions occur will be presented.

4\. A Map of High-Transition LP Solutions

Figure 3 presented a spherical coordinate system based at
${XYZ}_{\mathit{50\%}}$. A second rectangular coordinate system was also shown
in that figure, defined as
$(X^{\prime},Y^{\prime},Z^{\prime})=(X,Y,Z)-{XYZ}_{\mathit{50\%}}$. Suppose we
wish to visualize the OCS surface by placing our viewpoint at
${XYZ}_{\mathit{50\%}}$ and directing our view either up or down the
$Z^{\prime}$ axis. Figure 5 shows two polar plots of how we can arrange a
mapping of the spherical coordinate system, in both cases, placing the
$X^{\prime}$ axis running to the right. The left plot in Figure 5 visualizes
the upper half of the OCS and the right plot visualizes the lower half.

Figure 5: Polar mapping for upper and lower halves of OCS. Figure 6: A view of
the the top and bottom halves of the OCS as viewed from the 50% gray point,
looking in the $+Z^{\prime}/{-}Z^{\prime}$ directions.

We are now able to create a high-resolution bitmapped graphic by superimposing
a rectangular lattice over each polar plot. The lattice points correspond to
the pixels of a bitmapped image, and each pixel has a corresponding spherical
coordinate pair, $(\theta,\varphi)$. We can formulate a LP for each pixel by
defining
${XYZ}_{targ}={XYZ}_{\mathit{50\%}}+[\mathrm{sin}(\varphi)\;\mathrm{cos}(\theta);\;\;\mathrm{sin}(\varphi)\;\mathrm{sin}(\theta);\;\;\mathrm{cos}(\varphi)]$.
Figure 6 presents a color-coded summary of transition count of the LP solution
associated with each pixel’s ${XYZ}_{targ}$.

Several notable features stand out. First, it is evident that the vast
majority of optimal colors have the two-transition Schrödinger form (the black
and gray regions). Second it appears that there are two regions of higher-
transition optimal colors that are mirror images of one another. The region in
the upper half are all type II-like, having reflectance of 1 everywhere except
for several pockets of 0 values. Conversely, the region in the lower half are
all type I-like, having 0 reflectance everywhere except for several pockets of
1 values. Another way to interpret this is that the complement of a higher-
transition color is a color of the same transition count, but of the opposite
type. A line connecting the two passes through ${XYZ}_{\mathit{50\%}}$ (point
symmetry).

Certainly there are two-transition object colors in the same directions as the
higher-transition colors in Figure 6. How do they compare? The next section
examines that question.

5\. Are Some Schrödinger Colors Sub-Optimal?

The short answer appears to be yes, but only because of the slight non-
convexity of the 1931 $2^{\circ}$ CIE Standard Colorimetric Observer Data. To
demonstrate this, another computational experiment that parallels that shown
in Figure 6 was performed, this time using each ${XYZ}_{targ}$ to identify the
two-transition color along the associated ray. This was accomplished using the
MATLAB code that was presented in a paper by Masaoka and Berns.23 Its original
purpose was to compute “optimal metamers” for Logvinenko’s “object color
space.”24 Here, it is adapted to compute only the two-transition part.***There
is a very minor bug in the Masaoka & Berns MATLAB code. The statement in their
function “optm” XYZopt = sum(T(L(1):L(2),:))... should be instead XYZopt =
sum(T(L(1):L(2),:),1).... The bug occurs in the very rare instance when L(1) =
L(2); in that case, the original statement incorrectly returns a scalar.

By computing the distance from ${XYZ}_{\mathit{50\%}}$ to the tristimulus
values of both the two-transition color and the high-transition color, both
measured along the same ray, a comparison can be made. In all cases, the high-
transition color was found to be slightly farther away from
${XYZ}_{\mathit{50\%}}$ than the two-transition color was. Figure 7 summarizes
this difference graphically. Only the top half is shown, as the bottom half is
again a point symmetric copy.

Figure 7: Difference in distances to the LP solutions and to the two-
transition solutions, measured from ${XYZ}_{\mathit{50\%}}$, both along the
same ray.

The gray region is where the LP solution gives a two-transition color, which
matches the 2-transition color returned by the Masaoka & Berns code. In those
cases, the distances from ${XYZ}_{\mathit{50\%}}$ of the two match to machine
floating point tolerance. In regions of higher transition count, however, the
difference in distances is ten orders of magnitude greater than machine
tolerance. We can conclude that the two-transition colors in the regions where
the LP gives higher-transition optimal colors are sub-optimal and fall
slightly inside the OCS surface.

6\. Effect of Changing CMFs Resolution

In Section 2, it was shown that the 1931 $2^{\circ}$ CIE CMFs retain non-
convexity at other wavelength interval sizes. Here we examine one specific
point on the OCS surface and how the LP high-transition optimal color and two-
transition color (from the Masaoka & Berns code) are affected by resolution of
the CMFs. Figure 8 shows the two solutions for wavelength resolutions of 0.1
nm, 1 nm, 5 nm, and 10 nm. The black line is the two-transition reflectance
curve and the blue region is the LP optimal reflectance curve. (Note that both
are bar plots, one is filled in blue and the other is only outlined.) The
small image inset into each of the four plots shows the transition count
distribution as computed from the CMFs with the various wavelength interval
sizes. It uses the same color coding as shown in Figure 6. Even at 10 nm
intervals, the high-transition behavior persists. The small white dot in each
plot shows where the two reflectance plots are located on the OCS surface.

Figure 8: The effect of different CMFs wavelength interval size.

The reflectance distributions all show magnitudes at the transition points
that are between 0 and 1. This is an artifact of the discretization of the
CMFs. Since both the LP and the Masaoka & Berns code have constraints the
require the reflectance curve to have an associated tristimulus value triplet
that exactly matches a specified value, the only way to satisfy this match is
to allow the transition points to have intermediate values. The way this
should be interpreted is that the “true” transition points fall somewhere
within the finite width of the wavelength interval. Li et al noticed this same
artifact in their 2010 paper, and came to the same conclusion.22

As mentioned in Section 5, the difference in tristimulus values between the LP
solution and the two-transition solution is very small. For example, for the 1
nm case shown in Figure 8, we have the following numerical values:

$\begin{split}&\theta=1.478858\;\mathrm{rad},\;\;\varphi=0.371322\;\mathrm{rad},\;\;{XYZ}_{targ}=\begin{Bmatrix}50.03731\\\
50.36132\\\ 50.94838\end{Bmatrix}\\\
&{XYZ}_{LPsoln}=\begin{Bmatrix}51.79069\\\ 69.37875\\\
99.99523\end{Bmatrix},\;\;\;{XYZ}_{two-trans}=\begin{Bmatrix}51.79066\\\
69.37828\\\ 99.99402\end{Bmatrix}\\\
&\mathrm{Difference\;in\;distance\;from}\;{XYZ}_{\mathit{50\%}}=1.29\times
10^{-3}\end{split}$ (6)

7\. Other Illuminants

So far, all results presented are for an equal-energy illuminant. Here we
examine how other illuminants affect the results. Five high-chroma Munsell
colors were selected, 5R 5/14, 5Y 8/16, 5G 7/10, 5B 6/10, and 5P 4/12, and an
illuminant was created with the same chromaticity, using a “smoothest”
reflectance reconstruction technique.25-26 They are all shown at the top of
Figure 9, scaled to have a maximum value of 1.

Figure 9: Five illuminants with chromaticities matching Munsell 5R 5/14, 5Y
8/16, 5G 7/10, 5B 6/10, and 5P 4/12 and high-transition optimal color regions
associated with each illuminant, plotted on the chromaticity diagram.

The lower portion of Figure 9 shows the regions of high-transition optimal
colors associated with each of the Munsell-based illuminants, plotted on the
chromaticity plane. The small circles are the chromaticities of the
illuminants. The green and orange lines are the main and opposite meridians.
Only the type II-like regions appear since the type I-like regions have very
high chromatic purity, and are compressed along the boundary of the
chromaticity diagram. It is evident that the general distribution of
transition count locations persists across a wide range of illuminants. The
choppiness in Figure 9 is simply an artifact of a difference in graphics
resolution between Figure 6 and 9.

8\. Conclusions

The main conclusion that can be drawn from this presentation is that very
slight non-convexities of the spectral locus can have a significant impact on
the shape of reflectance curves associated with optimal object colors. It is
possible that the non-convexity was introduced by the manipulation of the
Wright-Guild data leading to the 1931 standard.27 The slight non-convexity has
no negative impact on everyday colorimetric computations.

Another interesting observation is that the high-transition solutions appear
to come in point-symmetric pairs, one type I-like and the other type II-like,
but otherwise matching in transition locations. The size of the regions of
high-transition optimal colors, when plotted on the chromaticity diagram,
depends on the illuminant. The largest regions are associated with illuminants
nearest to the boundary non-convexities.

The use of linear programming to find optimal reflectance spectra is not new,
but the way of searching along an arbitrary ray through a target is unique.
This brings up other potential applications. For example, to create a slice of
the three-dimensional OCS for some fixed value of $X$, $Y$, or $Z$, all we
need to do is relocate our coordinate system to the point on the gray line
(connecting the origin and the white point) that has the desired value of $X$,
$Y$, or $Z$, and then sweep the search ray around a circle in the plane of the
other two tristimulus coordinates.

Additional supporting material and additional insights can be found at the
online supplementary documentation for this paper.27

References

[1] Kuehni RG. Erwin Schrödinger, Theorie der Pigmente von grösster
Leuchtkraft (Theory of pigments of greatest lightness). http://www.iscc-
archive.org/pdf/SchroePigments2.pdf Also:
https://web.archive.org/web/20190120185844/http://www.iscc-
archive.org/pdf/SchroePigments2.pdf Short URL: https://bit.ly/39fUdkv Accessed
March 26, 2021.

[2] MacAdam DL. The theory of the maximum visual efficiency of colored
materials. J Opt Soc Am. 1935;25(8):249–252.

[3] MacAdam DL. Maximum visual efficiency of of colored materials. J Opt Soc
Am. 1935;25(11):361–367.

[4] Wyszecki G, Stiles WS. Color Science, Concepts and Methods, Quantitative
Data and Formulae. 2nd ed. New York, NY: John Wiley & Sons; 1982:179-184.

[5] MacAdam DL. Color Measurement, Theme and Variations. 2nd ed. Berlin
Heidelberg New York: Springer-Verlag; 1985:122.

[6] Koenderink JJ., van Doorn AJ. Perspectives on colour space. In: Mausfeld
R, Heyer D eds. Colour Vision: From Light to Object. Oxford, UK: Oxford
University Press; 2003:1–56.

[7] Kuehni RG. Color, An Introduction to Practice and Principles. 3rd ed.
Hoboken, NJ: John Wiley & Sons; 2013:126.

[8] Oleari C. Standard Colorimetry, Definitions, Algorithms and Software.
Chichester, West Sussex, UK: John Wiley & Sons; 2016:352.

[9] West G, Brill M. Conditions under which Schrödinger object colors are
optimal. J Opt Soc Am. 1983;73(9):1223-1225.

[10] Davis G. Convexity and Transitions, a strict examination of the 1931 CIE
inverted-U, April 1, 2020.
https://cran.r-project.org/web/packages/colorSpec/vignettes/convexity.pdf.
Accessed March 26, 2021.

[11] University College London, Color & Vision Research Laboratory. Color
Matching Functions. http://cvrl.ioo.ucl.ac.uk/cmfs.htm. Accessed March 3,
2021.

[12] Rochester Institute of Technology, Munsell Color Science Laboratory.
Useful color data.
https://www.rit.edu/cos/colorscience/rc_useful_data.php Short URL:
https://bit.ly/39hBp4e Excel data at http://www.rit-
mcsl.org/UsefulData/all_1nm_data.xls Short URL: https://bit.ly/3srkdAL
Accessed March 26, 2021.

[13] Martínez-Verdú F, Perales E, Chorro E, de Fez D, Viqueira V, Gilabert E.
Computation and visualization of the MacAdam limits for any lightness, hue
angle, and light source. J Opt Soc Am A. 2007;24(6):1501-1515.

[14] Allen E, Some new advances in the study of metamerism: Theoretical limits
of metamerism; An index of metamerism for observer differences. Color Eng.
1969;7(1):35-40.

[15] Ohta N, Wyszecki G. Theoretical chromaticity-mismatch limits of metamers
viewed under different illuminants. J Opt Soc Am. 1975;65(3):327-333.

[16] Centore P. An open-source algorithm for metamer mismatch bodies, February
4, 2017.
https://www.munsellcolourscienceforpainters.com/ColourSciencePapers/AnOpenSourceAlgorithm
ForMetamerMismatchBodies.pdf Short URL: https://bit.ly/3fhUBCK Accessed March
26, 2021

[17] Mackiewicz M, Rivertz HJ, Finlayson G. Spherical sampling methods for the
calculation of metamer mismatch volumes. J Opt Soc Am A. 2019;36(1):96-104.

[18] Ohta N, Wyszecki G. Designing illuminants that render given objects in
prescribed colors. J Opt Soc Am. 1976;66(3):269-275.

[19] Allen E. Basic equations used in computer color matching. J Opt Soc Am.
1966;56(9):1256-1259.

[20] Belanger PR. Linear-programming approach to color recipe formulations. J
Opt Soc Am.
1974;64(11):1541-1544.

[21] Cogno JA, Jungman D, Conno JC. Linear and quadratic optimization
algorithms for computer color matching. Color Res Appl. 1988;13(1):40-42.

[22] Li C, Luo MR, Cho MS, Kim JS. Linear programming method for computing the
gamut of object color solid. J Opt Soc Am A. 2010;27(5):985-991.

[23] Masaoka K, Berns RS. Computation of optimal metamers. Optics Letters.
2013;38(5):754-756.

[24] Logvinenko AD. An object-color space. J Vision. 2009;9(11):1-23.

[25] Burns SA. Numerical methods for smoothest reflectance reconstruction.
Color Res Appl. 2020;45(1):8-21.

[26] Burns SA. Chromatic adaptation transform by spectral reconstruction.
Color Res Appl. 2019;44(5):682-693.

[27] Burns SA. Supplementary Documentation: The Location of Optimal Object
Colors with More Than Two Transitions. http://scottburns.us/supplementary-
documentation-the-location-of-optimal-object-colors-with-more-than-two-
transitions/ Short URL: https://bit.ly/3rxEZh7 Accessed March 26, 2021.


# Large Knowledge Model: Perspectives and Challenges

Huajun Chen
College of Computer Science, Zhejiang University, Hangzhou, 310027

<EMAIL_ADDRESS>

###### Abstract

Humankind’s understanding of the world is fundamentally linked to our
perception and cognition, with _human languages_ serving as one of the major
carriers of _world knowledge_. In this vein, _Large Language Models_ (LLMs)
like ChatGPT epitomize the pre-training of extensive, sequence-based world
knowledge into neural networks, facilitating the processing and manipulation
of this knowledge in a parametric space. This article explores large models
through the lens of “knowledge”. We initially investigate the role of symbolic
knowledge such as Knowledge Graphs (KGs) in enhancing LLMs, covering aspects
like knowledge-augmented language model, structure-inducing pre-training,
knowledgeable prompts, structured CoT, knowledge editing, semantic tools for
LLM and knowledgeable AI agents. Subsequently, we examine how LLMs can amplify
traditional symbolic knowledge bases, encompassing aspects like using LLM as
KG builder and controller, structured knowledge pretraining, LLM-enhanced
symbolic reasoning, and the amalgamation of perception with cognition.
Considering the intricate nature of human knowledge, we advocate for the
creation of _Large Knowledge Models_ (LKM), specifically engineered to manage
diversified spectrum of knowledge structures. This ambitious undertaking could
entail several key challenges, such as disentangling knowledge representation
from language models, restructuring pre-training with structured knowledge,
and building large commonsense models, among others. We finally propose a
five-“A” principle to distinguish the concept of LKM.

##  1 Language vs Knowledge

###  1.1 Human Language and World Knowledge

Humankind accumulates knowledge about the world in the processes of perceiving
the world, with natural languages as the primary carrier of _world
knowledge_Yule . Historically, the vast majority of world knowledge has been
described, documented, and passed on through natural languages. In addition to
natural languages that record common-sense knowledge, human has also invented
various scientific languages for describing specialized scientific knowledge,
for examples, the mathematical languages that describe mathematical models
VerifyStepByStep ; MATHDataset , chemical languages such as SMILE for
describing molecular structuresSMILE ; Fingerprint , and genetic languages to
model compositions of living organisms GeneOntology ; GOResource ; OntoProtein
.

However, natural languages merely encode world knowledge through sequences of
words, while human cognitive processes extend far beyond simple word
sequences. Therefore, since early inceptions of AI, it has been a fundamental
objective to explore machine-friendly formats for _Knowledge Representations_
(KR) WhatIsKR . Typical examples of these endeavors encompass Description
Logics IntroDL  for representing ontological knowledge, Prolog for rule-based
logic prolog2022 , Semantic Networks for depicting conceptual relationships
woods1975whatsInALink , and various others.

In traditional symbolic AI research, the logical structure and expressiveness
of a KR is pivotal to the reasoning ability of the inference machines WhatIsKR
; IntroDL . Natural language sequences composed of simple words or concepts
are generally considered to be unfavorable for machines to make inference,
while hierarchical taxonomies, complex ontologies, and rule-based logic are
more conducive to machine reasoning. As highlighted later, even in the era of
large models, there persists a positive correlation between the complexity
level of data representations and the reasoning proficiency of large models.

###  1.2 Language Models vs Knowledge Graphs

_Knowledge Graphs_ (KGs) model world entities, mental concepts and their
relationships in form of graph structures, which are typically extracted or
derived from natural language descriptions. With many inspirations from
traditional symbolic AI, KGs integrate natural language descriptions and
structure knowledge. Typical structural forms include: hierarchical
structures(e.g., concept graphs), relational structures(e.g., relational
entity graphs), temporal structures(e.g., event logic graphs), etc. Similar to
natural languages, KGs represent knowledge in an explicitly symbolic way,
thereby offering higher reliability and interpretability to humans.

_Large Language Models_ (LLMs) SurveyOfLLM ; HarnessLLM , such as ChatGPT,
mark a paradigm shift in knowledge representation and processing. By
autonomously compressing or projecting vast textual knowledge in form of human
languages into neural networks, LLMs achieve a parameterized representation
and handling of world knowledge. LLMs internalize deep word patterns and
interactions from extensive text corpora, establishing a wider coverage of
knowledge than conventional symbolic knowledge base. Advanced techniques like
prompt or instruction tuning further imbue LLMs with prior human
knowledge,expanding their generalization capacity and adaptability to new
tasks. Unlike natural language and KGs, LLMs operate on a fully parameterized,
machine-friendly basis, albeit less interpretable to humans.

Figure 1: Language, Knowledge, Language Models and Knowledge Graphs.

In this article, we aim to investigate large models through the lens of
“knowledge”. Our first focus is on how KGs can enhance LLMs. Topics include
knowledge-augmented language modeling, structure-inducing pre-training,
knowledgeable prompts, structured Chains of Thought (CoT), semantic tools for
LLMs, editing knowledge in large models, and the development of knowledgeable
AI agents. We further examine the role of LLMs in enriching traditional
symbolic knowledge bases, encompassing the utilizing of LLMs as KG builders
and controllers, structured knowledge pre-training, LLM-driven symbolic
reasoning, and the integration of perception and cognition. Acknowledging the
intricate nature of human knowledge, we then conceptualize a _Large Knowledge
Model_ (LKM) in purpose to handle the diversity of structures of human
knowledge. This may involve approaches of decoupling knowledge from language
models, restructure pretraining with structure knowledge, building large
commonsense model, among others. To distinguish the concept of LKM, we propose
a five-“A” principle from the aspects of Augmented pretraing, Authentic
knowledge, Accountable reasoning, Abundant knowledge coverage, and Aligned
with knowledge.

##  2 Knowledge-Augmented Language Models

The first step towards _Large Knowledge Models_ is to augment LLM with
knowledge. In this context, “knowledge” specifically refers to more
standardized semantic representations such as domain teminology or ontologies,
more structured knowledge representations such as a KG, or richer logic
descriptions such as text with CoT. We will discuss how knowledge can be
effectively applied in various aspects in the life cycle of a LLM, including
pre-training, fine-tuning with instructions, prompt engineering, Chain-of-
Thought, AI agents, among others.

###  2.1 Knowledge Augmentation

Growing numbers of researches have shown that improving the semantic
standardization of or adding more structured knowledge into the training
corpus would be beneficial to enhance model reasoning capability, either in
pre-training or instruction-tuning stages ToG ; MindMap ; InstructProtein ;
CoK ; UnifyKGLLM ; ClinGen . For examples, code languages, characterized by
their structured nature and abundant computational logic, would be more
conducive to activating the reasoning capabilities of large models compared to
natural languages PoT ; PoT-CIRS . Similarly, Chain-of-Thought (CoT) prompts
CoT ; CoT-SC  contain more logical descriptions than ordinary text, thus being
more effective in stimulating model reasoning ability. Further research
reveals that CoT prompts are more effective when the model scale to a certain
extent, which can be explained as that a larger model contains more knowledge
for reasoning, and the activation of these knowledge requires more logical
prompts.

To delve deeper into the correlations between code structure and model
reasoning, we introduce a study called “Program of Thoughts” PoT-CIRS . This
research begins by establishing metrics for quantifying the complexity of code
structure and code logic. For instance, codes featuring loop operations are
considered as having richer computational logic and assigned with higher
scores compared to those with mere variable assignment operations.
Subsequently, we compile a series of code prompts with different levels of
structure or logic scores, and assess their impact on model reasoning
performance. Experimental analyses on various benchmarks reveal that the code
prompts with higher structure or logic scores are more conducive to improving
model reasoning performance.

Code language is not suitable to represent common-sense knowledge, while
Knowledge Graphs (KGs) are more adept in modeling such type of knowledge. To
further explore the influence of knowledge structure on common-sense
reasoning, we transform structured knowledge in a KG into corresponding code
constructs in a study called KG2Code CodeKGC . For example, we made classes
and properties in a KG transformed to corresponding _Class_ and member
variable declarations found in conventional object-oriented programming
languages. We then use the code-formed KGs as prompts or directly inject them
into base model pre-trainings, and evaluate their impact on downstream tasks.
Experimental results have indicated that models augmented with code-formed KGs
exhibit notable performance improvements in various dimensions, particularly
in extraction tasks.

External knowledge can also enhance the generalization ability of large
models. As discussed in RetroPrompt RetroPrompt ,large models usually rely on
“mechanical memorization” to handle long-tailed or isolated samples rather
than actually learning underlying patterns. This is one of the essential
reasons why many large models have poor generalization ability in few-shot
scenarios. It’s like a student who only knows how to memorize things
mechanically is relatively weak to extrapolate from one instance to others.
RetroPrompt addresses this issue by decoupling knowledge from memory, and
train a external knowledge base (not necessarily a KG here) independently.
Prompts are then enriched by retrieving pertinent knowledge the pretrained
knowledge base. Experiments indicate that RetroPrompt manages to effectively
improve the generalization ability of models particularly in few-shot
scenarios.

###  2.2 Structure Inducing Pre-training

Structures do matter in pretraining. In a study called “Structure-Inducing
Pretraining” StructIndPretrain , the role of imposing relational structure in
pretrained language models is investigated. The basic ideas involve crafting
connections, either at inter-sample or intra-sample level, to enforce
additional structural constraints on the geometric relationships within or
between training samples. These constraints range from shallow, implemented
via auxiliary pretraining objectives, to more intricate, implicitly integrated
through strategies like data augmentation or sample-level contrastive
learning.

A typical example of _inter-sample_ structure constraints manifests in the
injection of a protein-protein interaction network into the pretraining of a
protein language model. This injection infuses the model with sample-level
structural insights, i.e. the relational structures of protein interactions. A
typical example of _intra-sample_ structure manifests in the linking of
entities within a sentence to an external knowledge graph, thereby introducing
correlation information in a KG into the training sentences. Both theoretical
analysis and experimental verifications are performed and indicate a
consistent improvement across a variety of benchmarks.

###  2.3 Knowledgeable Prompts

####  2.3.1 Prompt Engineering as Knowledge Engineering

Prompts refer to natural language text describing the instructions that an AI
should perform, the context of instructions that an AI can inquire of, or the
guidance to think that an AI can follow InstructGPT ; CoT ; FLAN . It has been
proven by numerous practices Auto-CoT ; APE ; AutoPrompt ; KNNPrompt  that the
quality of prompts has a huge impact on model outputs. By providing clear
prompts or instructions with specific details, the model can more accurately
comprehend human intentions, leading to outputs that are more aligned with
human instructions. In contrast to the basic and shallow semantic
understanding fostered during base model training, prompts in the fine-tuning
stage embed more sophisticated and task-specific knowledge.

Prompt engineering is essentially comparable to knowledge engineering.
Increasing number of research indicates that the complexity of prompts is
closely correlated to model reasoning ability. The evolution of prompts ranges
from mere textual prompt, CoT prompt with more logic descriptions to more
structured prompts such as _Program of Thoughts_ PoT , _Tree of Thoughts_ToT ,
and _Graph of Thoughts_ GoT , and even directly using logical rules as prompts
PTR . As the complexity and expressiveness of prompt representations
increases, an improvement of model reasoning ability can be observed. However,
there persist a trade-off between richness of prompt representations and
scalability of prompts acquisition: higher expressiveness in prompts typically
entail increased difficulty for acquisition, which is essentially similar to
traditional knowledge engineering. High-quality prompt engineering is also
time-consuming and laborious as like knowledge engineering.

####  2.3.2 Structured Chain of Thoughts

Chain of Thought (CoT) is a special type of prompts describing the thinking
procedures that can guide large models to imitate human logical thinking. As
previously discussed, employing more intricately structured CoT
representations, such as Tree of Thoughts ToT  or Graph of Thoughts GoT ,
would bolsters the model’s capacity for reasoning.

Moreover, KG is obviously conducive to enhancing the structure of CoT prompts.
On the one hand, The structured knowledge in a KG proves useful in the
formulation and generation of structured CoT prompts CoK ; ToG ; MindMap , to
form a CoT knowledge graph (KGoT). On the other hand, through the entity
linking technologies, structure correlations among concepts and entities in
the KG can be used to expand and enhance CoT in natural language forms, making
the textual CoT more in line with human thinking patterns based on
associations,thereby enhancing their effectiveness in guiding AI reasoning.

####  2.3.3 KG2Instructions

Instruction tuning InstructGPT ; FLAN  refers to further training base models
with task-specific instructions, typically converted from task-specific
training data. Since data from various tasks are converted to an unified input
form as instructions or prompts, intruction-tuning with massive downstream
task would significantly improve model’s generalization ability, that is, the
ability to solve new problems or to complete new tasks.

Knowledge Graphs serve a dual roles of enhancing instruction-tuning. On the
first hand, KGs can be directly injected into instruction prompts. For
instances, KnowPrompt KnowPrompt  utilizes the relational data in a KG to
bolster the contextual depth of instruction prompts. Similarly, KANO KANO
integrates structured knowledge from a chemical KG, sourced from Wikipedia, to
refine functional prompts, thereby improving the learning process for
molecular pretraining.

On the other hand, KGs can be directly converted to instructional training
datasets. As exemplified by KG2Instructions proposed in DeepKE-LLM knowlm , a
KG refined from WikiData is employed to automatically generate vast amount of
training instructions for knowledge extraction. Essentially, these
instructions converted from the KG represent a novel form of distant
supervision, which leverages the structured information in KGs to generate
instructions that guide models in understanding and extracting relevant
information across diverse contexts and tasks. Experiments demonstrate notable
efficacy in enhancing the generalizability of extraction capabilities of large
models.

Compared to free text, a KG is data corpus with richer logical structures and
higher knowledge density. Many fields of knowledge graph data come from long-
term artificial accumulation (such as Gene Ontology) or natural structured
data transformation. Fully mining and using the knowledge structure contained
in the knowledge graph (such as hierarchical concepts, entity relationship,
temporal causality, rule logic, etc.) to enhance and enrich the prompt
instruction or assist in the construction of more logical instruction data
set, can help the model to complete more complex tasks.

###  2.4 Knowledge Editing

A well-known problem of large models is hallucination, TrustworthyLLMs ;
KGHallucination , where models generate responses based not on explicit
textual evidence rather than on the amalgamation of implicit parameters within
the neural network. Once the model goes wrong, it is neither easy to pinpoint
the source of errors, nor trivial to correct the errors in a parameterized
space. Even more seriously, due to the interconnected nature of knowledge,
even minor adjustments to the parameterized knowledge can lead to widespread
and unpredictable effects of “a slight move in one part may affect the
situation as a whole”. This phenomenon is more pronounced in LLM than in
traditional symbolic knowledge bases such as KGs, where changes tend to be
more localizable and manageable.

Knowledge editing EasyEdit  is geared towards the precise modification of
parameterized knowledge of large models. This process aims to rectify
incorrect knowledge, eliminate harmful content, or update outdated
information. Yet, executing edits in the vast parameter space of large-scale
neural models is a nontrivial task. It necessitates firstly locating the
incorrect knowledge, then secondly determining the scopes and boundaries of
necessary modifications. Critically, because the parameters of the neural
network are highly correlated, changing any one of the neurons will have a
“pull-start” effect on the other neurons associated with it. The relationship
between facts explicitly defined in a symbolic knowledge graph can be used to
help determine the correlation relationship between parametric knowledge and
define the boundaries of parameter updates. This understanding assists in
delineating relationships for parameter updates, ensuring that modifications
are both effective and do not inadvertently disrupt related knowledge
structures EditLLMs .

###  2.5 Knowledge Alignment

Broadly speaking, since the majority of information within most of KGs has
undergone manual review and verification, thereby ensuring a high degree of
standardization and accuracy, it can serve as a target for aligning large
models zhang2023knowledgeable . For instance, KGs can be instrumental in
generating feedback signals for the reward model . This is achieved by
contrasting the outputs of the LLM with the factual data, structured
relations, or logic constraints present in the KG. Such a comparison not only
aids in enhancing the accuracy and reliability of the outputs generated by the
LLM but also provides valuable metrics to assess the model’s performance.

###  2.6 Semantic Tools for LLM

Tool-Augmented Language Models ALMSurvey ; Toolformer  are proposed to extend
the functionalities of large models by integrating the ability to invoke
external tools for problems solving beyond their native solution scope. These
external tools can range from mathematical calculators, database interfaces to
specific functional APIs. A complex task may require a orchestration of
calling multiple tools, forming a complex tool calling logic. For LLMs to
adeptly master such complex tool-calling logic remains a significant
challenge. Knowledge Graphs can play a valuable role in this context. They can
enhance the semantic descriptions or model the calling relationships of these
tools, thereby facilitating the modeling of complex API combination logic.
This approach has already seen considerable application in the realm of
traditional semantic web services mcilraith2002adapting , suggesting a
promising avenue for enriching the functional versatility of tool-augmented
large models.

###  2.7 Knowledgeable AI Agents

LLMs such as ChatGPT enables the building of autonomous agents AgentsSurvey ;
Agents ; HuggingGPT ; CAMEL  that can solve complicated problems by
communicating with each other. The deep relations between KGs and Agents
rooted historically. Early KG technologies, such as RDF/OWL RDF2OWL , were
initially developed to enhance interactions among web-based agents and to act
as a Knowledge Interchange Format KIF . For instance, DAML DAML , a forerunner
to the OWL language, stands for a Agent Markup Language, underlining this
connection.

The advent of new LLM technologies would revolutionize the paradigm of
knowledge exchange and interaction among agents. In scenarios where an agent’s
own knowledge is insufficient to solve a problem, it can autonomously seek
assistance from other agents, bypassing the need for manually and accurately
crafted knowledge exchange formats. This capability of LLM, together with
accountability of KGs, foster more autonomous knowledge collaboration and
paves the way for a more robust and effective community of agent
collaborations.

##  3 LLM-enhanced Knowledge Bases

A second step towards _Large Knowledge Model_ is to enhance conventional
symbolic knowledge base with LLM. In this context, we take Knowledge Graphs as
an exemplar type of such a symbolic KB. We will explore the multifaceted roleS
of LLMs in enhancing various stages in the life-cycle of a KG including
aspects such as KG building and management, structured knowledge pretraining,
symbolic reasoning, and others.

###  3.1 LLM as KG Builder

In critical scenarios requiring high reliability, the construction of a
symbol-based knowledge base would ensure accuracy, offer traceable sources for
its content, and provide enhanced interpretability. LLMs with their
proficiency in language understanding, AI-Generated content, and
generalization capabilities, marks a new era in developing symbolic knowledge
bases and their utility in serious or critical applications.

An in-depth evaluation of ChatGPT’s capability in constructing a KG has been
conducted LLMsForKGC . According to the report, ChatGPT does have a notable
ability to build a KG. For instance, ChatGPT can effectively generate a KG by
giving only a simple instruction such as “create a KG for Zhejiang
University”, and easily format it into requested formats such as RDF/OWL.
However, as GPT relies heavily on the AIGC model to generate the knowledge
graph, the correctness rate is roughly 70-80% on average for common domains
and even less than 20% for long-tail domains, due to its hallucination problem
LLMsForKGC . When it comes to extracting knowledge from provided texts,
ChatGPT’s accuracy can exceed 90%. Despite this, its performance still falls
short of the state-of-the-art (SOTA) achievements of existing small models in
this domain A_Multitask__EvaluationofChatGPT ; EventExtraction_for_ChatGPT ;
IE_of_ChatGPT . This discrepancy highlights the need for further refinement in
LLM-based approach to KG construction and information extraction, particularly
in balancing accuracy with the breadth of its generative capabilities.

In order to verify that the correct extraction from chatGPT does not come from
knowledge seen before, that is, there is relevant knowledge in the training
corpus, a fabricated KG LLMsForKGC  was designed and generated. Subsequently,
we generate text based on these fictitious entities and relationships and
presented them to ChatGPT for entity and relation extraction. The results were
noteworthy: ChatGPT demonstrated a high degree of accuracy in identifying
these fabricated entities and relations. This proficiency in generalized
knowledge extraction is likely attributable to its instruction-driven tuning
and the reward-based feedback learning mechanism.

Extraction is, in fact, a rather fundamental AI task. It is closely related to
the concept of understanding, in that effective knowledge extraction from text
is indicative of the model’s semantic comprehension abilities. Therefore, it
is anticipated that models fine-tuned with a focus on extraction instructions
will not only excel in extraction tasks but also show enhanced performance
across a broader spectrum of tasks InstructIE ; TRICE .

We summarize the unique advantages of LLM as universal KG builder as follows.

  * •

_Model-Centric Knowledge Extraction_ : In contrast to conventional methods of
KG construction, ChatGPT leverages its extensive “model knowledge” and
“generative inference” as additional components to extraction. This means that
GPT can extract knowledge embedded within its own parameters or generate
additional knowledge to enhance the extraction outcomes. This approach allows
for the inclusion of additional knowledge in the results that may not be
explicitly present in the source text. Such a strategy signifies a shift from
a text-centric extraction to model-centric approach.

  * •

_Enhanced Generalization in Knowledge Extraction_ : As highlighted earlier,
LLMs demonstrate remarkable precision in identifying previously unseen entity
types and relations. This high level of accuracy is largely attributed to the
models’ advanced generalization capabilities. This adaptability enables LLMs
to effectively process and recognize novel data inputs, transcending the
limitations of traditional extraction methods heavily relying on laborious
labeling efforts.

  * •

_AI-Generated Content for Knowledge Graph Construction_ : LLMs are equipped
with powerful generative capabilities. In cases where knowledge is absent or
insufficient in the model or source text, large models can still synthesize
new knowledge for the KG. However, it is notable that while AIGC extends the
depth and breadth of KGs, it also carries the risk of introducing incorrect
knowledge.

###  3.2 LLM as KG Controller

Large models can function as potent controllers for managing and interacting
with structured knowledge, primarily through their enhanced language
understanding capabilities. These can be manifested in three aspects as
depicted below:

  * •

_Query Templates Enhancement_ : Traditional query answering methods for
symbolic knowledge typically depend on manually crafted query templates. This
process necessitates extensive maintenance of a vast amount of templates.
Large models present an effective solution by automating the generation of
high-quality templates, thereby ensuring greater consistency and lowering the
costs associated with template maintenance.

  * •

_Logical Query Parsing_ : Leveraging the language comprehension abilities of
large models to convert natural language query into structured query formats
(like SPARQL, Cypher, Gremlin, etc.) offers a more direct solution CoK ;
NLQxform . This method frees programmers from the tedious and laborious task
of manually devising query compositions, streamlining the process
significantly.

  * •

_Embedding-based Retrieval Augmentation_ EmbedKGQA : Lastly, large models can
enhance the retrieval process from structured knowledge bases by utilizing
embedding-based techniques. This involves representing queries and the
knowledge base in a high-dimensional space to improve the accuracy and
relevance of the retrieved information.

Overall, these methods leverage the advanced language understanding
capabilities of large models to make the interaction with structured knowledge
more accessible, efficient, and user-friendly.

###  3.3 Structured Knowledge Pretraining and Large Graph Model

Research shows that current large models often do not outperform traditional
small models in tasks inherently reliant on structured knowledge, such as link
prediction, association graph mining, subgraph analysis, or time-series
prediction LLMsForKGC ; Evaluation_on_Relations . The _Large Graph Models_
GraphFundationModel ; GraphGPT  are then proposed for deriving insights from
complex, graph-structured data, which is critical in a variety of advanced
applications, including social network analysis, biological network analysis,
and network security, among others.

In this vein, we can develop special pre-training mechanisms tailored for
encoding structured knowledge, transforming structured knowledge into
instruction formats to boost model generalizability, or devising specific
reward models for graph computation tasks. This approach is particularly
valuable for big data analysis applications dependent on graph-structured
data.

KGTransformer KGTransformer  exemplifies this approach. It employs several
basic tasks for a KG, such as triple classification or head/tail prediction,
to pretrain a structured knowledge graph. For downstream applications such as
image classification, KGTransformer integrates a prompt fine-tuning
mechanisms, which effectively harnesses the knowledge from the pretrained KG,
thus enhancing downstream tasks’ performance.

###  3.4 LLM-enhanced Symbolic Reasoning

“Reasoning” refers to the cognitive process in which people derive inferences
or conclusions based on observed phenomena or facts, combined with known
facts, experiences, or rules ReasoningWithLLMs ; ENeSy . Reasoning is one the
most fundamental challenge in AI and has been extensively studied in many
fields such as philosophy, computer science, and psychology. Due to the broad
concept of reasoning, this article only discusses reasoning within the scope
of language models and knowledge graphs.

KGs and LLMs are both capable of performing reasoning. LLMs support a variety
of reasoning tasks, such as mathematical reasoning, commonsense reasoning,
symbolic manipulation, logical reasoning, and multi-modal reasoning
ReasoningWithLLMs . In the realm of KGs, typical reasoning forms encompass
concept reasoning, knowledge completion, multi-hop query reasoning, rule
learning and rule-based reasoning, inconsistency reasoning, and analogical
reasoning KGReasonWithLogicEmb ; AnalogyReason ; Ruleformer ; ENeSy .

The reasoning supported by KGs relies mainly on explicitly acquired symbolic
knowledge, hence it is more interpretable, and the reasoning process is
relatively reliable. With the integration of methods such as graph neural
networks or KG embedding, reasoning can also be carried out in a vector space.
However, since the knowledge coverage of a KG is not as extensive as that of
LLMs, the robustness and generalization ability are relatively poor.

The reasoning supported by LLM ReasoningWithLLMs  is mainly performed in a
parameterized space, hence it is more of an implicit form of reasoning.
Different from traditional reasoning methods such as graph neural network or
KG embedding, LLM also rely on the richness of knowledge embedded in the large
models. Furthermore, many studies show that the reasoning capabilities of
large models are strongly related to the model size EmergentOfLLMs ; GPT3 .
Certain reasoning abilities can only be activated when the model size reaches
a certain scale. Since large models possess very good generalization
abilities, they are particularly adept at extrapolating to new reasoning
tasks. However, since the reasoning relies on a generative process, it is
prone to hallucination issues, leading to unreliability in the reasoning
results.

Therefore, LLMs and KGs can complement with each other to achieve a more
robust reasoning solution. On the one hand, a large language model can use a
symbolic knowledge graph as an externally accessible tool. When it is
necessary to give a definitive, reliable answer, the reasoning with a symbolic
knowledge graph can be employed. On the other hand, when symbolic knowledge
graph reasoning cannot derive an answer due to lack of knowledge, it is
possible to use the extensive knowledge and high generalization ability of the
large language model to give coarse-grained reasoning. Additionally, the
interplay between LLMs (parametric, yet not entirely reliable) and KGs
(symbolic, more interpretable) allows for their mutual enhancement and
complementation.

###  3.5 Integration of Perception and Cognition

Knowledge Graphs fundamentally models abstract concepts or ontologies about
the world. These ontologies in the human minds are formed through a cognitive
process of abstract thinking, by which a myriad of world elements are firstly
recognized and hierarchical concepts or ontological categories are further
abstracted from understanding these elements. Large models learn rich
knowledge about words and concepts from massive text corpora, excel in concept
recognition and abstraction. This enables more advanced methods for tasks such
as automated construction of conceptual hierarchies, ontological category
expansion, attribute completion, ontology alignment, and concept
normalization.

Demis Hassabis, co-founder of DeepMind, posed a thought-provoking question:
“Can we build from our own perceptions, use deep learning systems, and learn
from basic principles? Can we build up to higher-level thinking and symbolic
thought?”. Present large models still learn about the world from human-
generated text corpora. Whether future large models can rely more on the model
itself to learn world knowledge from the perceptual interaction with physical
world, autonomously abstract concepts and ontologies about the world, and
directly utilize this understanding about the physical world for decision-
making, thereby achieving a profound integration of perception and cognition.

##  4 From LLM to LKM

Human knowledge, characterized by its complexity, is not optimally represented
by mere sequences of free text. Our descriptions of the world often entail a
variety of structured forms. The ultimate goal should to create more advanced
_Large Knowledge Model_ (LKM) proficient in handling and deciphering the
myriad structures of knowledge representation. This section aims to illuminate
a roadmap and identify the key challenges in developing such a Large Knowledge
Model.

###  4.1 Decoupling World Knowledge from Language Models

The essential reason of the “largeness” in LLMs stem from the need for massive
parameters to store vast amount of world knowledge. As highlighted by Bengio
in a recent blog post ModelBasedML , the parameters in these large models
encompass two components: the _World Model_ and _the Inference Machine_. World
models are used to store world knowledge, with actually a large portion of the
neural network parameters serving as this purpose. The inference machine,
responsible for reasoning computation, operate through the capabilities of
language models.

Perhaps because human reasoning ability also heavily relies on language
understanding ability, “Language” and “Knowledge” are actually inseparable in
typical large language model. This blend of language and knowledge in large
models is a departure from traditional symbolic AI systems such as expert
systems, where components like the knowledge base and reasoning engine are
implemented as independent modules. However, Bengio also pointed out that
future large models should probably decouple the world model from the
inference machine, so that part of the stored knowledge can be independently
verified and maintained.

This implies the need to train a separate, likely very _large_ knowledge model
to handle the extensiveness of world knowledge, while conversely, to train a
different language model, not necessarily have to be as large as nowadays’
LLM, to perform reasoning tasks. This division allows for efficient allocation
of model sizes according to their primary functions. Such a separation would
allow the knowledge storage component to be independently verified and
maintained, offering a potential approach to manage model size and enhance the
verifiability and reliability of the model’s knowledge.

###  4.2 Restructured Pretraining with Structure Knowledge

Numerous cognitive scientists posit that humans inherently favor structured
thinking, tending to recall, reason, and plan via associative thinking. Recent
research suggests that the way large models store knowledge lacks discernible
patterns as like a KG StructGPT . In other words, those knowledge that should
be structured correlated does not show the expected associations in the
parameter space, making it difficult for the model to make feasible
correlations among knowledge pieces, and generate the desired answers in a
manner close to human cognition. There might be a sort of “cognitive gap”
between the form of knowledge storage in large models and the organization of
knowledge in the human mind. This cognitive gap becomes problematic in many
critical scenarios when we expect the model to have more controllability, or
to deal with deeper and more complex associative reasoning.

One way to alleviate this problem is to enhance the structure of the training
corpus during the pretraining phase, known as restructure pretraining GraphGPT
. That is, we can use automated or semi-automated methods to enhance the
structured features of the text. For example, we can organize sentences more
logically (essentially, CoT is a method to increase the structure and logic of
texts). We can structure paragraphs more coherently, add links between words,
create connections between training samples, or directly incorporating an
external structured KG to structurally reinforce the entire training corpus.

By making the representation and organization of training data closer to the
organization of knowledge in the human mind, we can reduce the cognitive gap
between large models and human mind, thereby making the knowledge in large
models more transparent, associative, and verifiable. Ultimately, this would
paves a way to transform a LLM into a LKM, for which the structure of
knowledge in a LKM aligns more naturally with human cognitive processes.

###  4.3 Large Commonsense Model

Commonsense encompasses practical know-how and sound judgment regarding
everyday matters, shared by nearly all humans. According to a definition from
Marvin Minsky MinskyTheEmotionMachine , “commonsense knowledge includes facts
about events occurring in time, about the effects of actions by the knower and
others, about physical objects and how they are perceived, and about their
properties and their relations to one another”.

The development of commonsense knowledge bases has been a pivotal challenge
throughout AI’s history. This journey began with the early Cyc project Cyc ,
extended through ConceptNet ConceptNet  as advocated by Marvin Minsky, and
progressed to contemporary knowledge graph initiatives like Yago Yago ,
DBPedia DBPedia , Wikidata Wikidata , BabelNet BabelNet , ATOMIC Atomic , and
ASTER ZHANG2022103740 . The consistent goal has been to equip AI with a
comprehensive commonsense knowledge base. However, these common sense
knowledge bases generally face challenges such as insufficient knowledge
coverage and weak generalization capabilities of reasoning machines.

Large language models, such as ChatGPT, have demonstrated their potential in
harboring vast commonsense knowledge and performing commonsense reasoning.
Their principal strength lies in THE extensive knowledge coverage. However,
they also grapple with challenges such as hallucination and the unreliability
of reasoning results, and they tend to support only basic commonsense
reasoning. A promising research direction for the future could involve
integrating traditional commonsense knowledge bases with large language models
to create an expansive large commonsense knowledge model. Such a model could
potentially harness the strengths of both approaches to overcome their
individual limitations.

###  4.4 Five-“A” Principle of LKM

In this final section, we strive to provide a general description on the
concept of a Large Knowledge Model (LKM) and outline its pivotal
characteristics. We outline the essential aspects of an LKM using a five-“A”
framework as depicted in Figure. 2.

  * •

_Augmented Pretraining_ : Firstly, large knowledge models should be trained
with a diverse range of knowledge structures rather than relying solely on
sequential words. This can be achieved by incorporating additional logical
elements into prompt instructions or by enhancing structural coherence,
whether at the inter-sample or intra-sample level, as higlihgted in structure-
inducing pretraining StructIndPretrain . The manner in which knowledge is
stored within a large model should resemble the organization of knowledge in
human mind.

  * •

_Authentic Knoweldge_ : Secondly, large knowledge models should avoid
generating hallucinations and offer authentic knowledge that can be
independently verified. This can potentially be accomplished by decoupling
knowledge representation from the language model, allowing for the independent
maintenance, verification, and upgrading of the knowledge base.

  * •

_Accountable Reasoning_ : Thirdly, the entire system should facilitate
accountable reasoning processes that enhance both the reliability of answers
and their interpretability to humans. This can be achieved by integrating
symbolic reasoning over an external knowledge base with prompted reasoning by
large language models.

  * •

_Abundant Coverage_ : Fourthly, the entire system should offer abundant
coverage of both general and domain-specific knowledge. This objective can be
possibly accomplished by fostering the development of knowledgeable AI agents
communities through a comprehensive integration of private knowledge bases,
open commonsense knowledge, and LLM-based knowledge exchange among agent
communities.

  * •

_Aligend with Knowledge_ : Finally, the entire system should be in close
alignment with human values and ethics, which have evolved as a form of
advanced knowledge accumulated throughout the history of humanity. This could
be probably achieved through the creation of sophisticated ethics knowledge
bases and knowledge alignment technologies zhang2023knowledgeable , which
treat the ethic knowledge bases as aligning objectives.

Figure 2: Five-“A” principle of LKM.

##  5 Summary

Representing and processing world knowledge has been central to its objectives
Since the advent of AI. Both large language models and knowledge graphs
exhibit distinct advantages and limitations in handling world knowledge. Large
language models excel in language comprehension, whereas knowledge graphs
offer diversified methods for knowledge representations. A deeper integration
of these two technologies promises a more holistic, reliable, and controllable
approach to knowledge processing in the field of artificial intelligence.

Firstly, KGs can significantly contribute to various stages of LLM
development. This include enhancing the model’s capabilities and reducing
training costs by refining the structure and logic of the training corpus or
prompt instructions, and addressing the issue of hallucination generation. KGs
are also poised to play a critical role in the evolution and future
development of AI agent communities.

Secondly, LLMs can also contribute in multiple ways to the traditional
knowledge graph technology stack. For instance, instruction-driven
methodologies can facilitate knowledge extraction with enhanced generalization
capabilities. The advanced language comprehension abilities of LLMs can be
harnessed to significantly improve operations such as querying, answering, and
updating structured knowledge. Moreover, the commonsense reasoning abilities
of large models, grounded in natural language, can complement and strengthen
the symbolic reasoning capabilities of knowledge graphs.

Lastly, the complexity of human knowledge necessitates the use of various
structured descriptions to accurately depict the world. Sequential words alone
fall short as the optimal medium for representing knowledge about world. The
ultimate goal, therefore, is to develop a more advanced large knowledge model
adept at interpreting the myriad structures of world knowledge.

##  6 Acknowledgment

I would like to express my gratitude to all the members of the Knowledge
Engine Lab at Zhejiang University for their invaluable support and
contributions. Additionally, I would extend my sincere thanks to our
collaborators for their partnership. Special appreciation goes to Dr. Mingyang
Chen for his diligent content review.

## References

  * [1] G. Yule.  The study of language (6th ed.).  Cambridge University Press., 2019.
  * [2] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.  Let’s verify step by step.  CoRR, abs/2305.20050, 2023.
  * [3] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the MATH dataset.  In NeurIPS Datasets and Benchmarks, 2021.
  * [4] Stanislaw Jastrzebski, Damian Lesniak, and Wojciech Marian Czarnecki.  Learning to SMILE(S).  CoRR, abs/1602.06289, 2016.
  * [5] Zheng Xu, Sheng Wang, Feiyun Zhu, and Junzhou Huang.  Seq2seq fingerprint: An unsupervised deep molecular embedding for drug discovery.  In BCB, pages 285–294. ACM, 2017.
  * [6] Michael Ashburner, Catherine A. Ball, Judith A. Blake, David Botstein, Heather Butler, J. Michael Cherry, Allan P. Davis, Kara Dolinski, Selina S. Dwight, Janan T. Eppig, Midori A. Harris, David P. Hill, Laurie Issel-Tarver, Andrew Kasarskis, Suzanna Lewis, John C. Matese, Joel E. Richardson, Martin Ringwald, Gerald M. Rubin, and Gavin Sherlock.  Gene ontology: tool for the unification of biology.  Nature Genetics, 25(1):25–29, 2000.
  * [7] The Gene Ontology Consortium.  The gene ontology resource: 20 years and still going strong.  Nucleic Acids Res., 47(Database-Issue):D330–D338, 2019.
  * [8] Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Qiang Zhang, Jiazhang Lian, and Huajun Chen.  Ontoprotein: Protein pretraining with gene ontology embedding.  In ICLR. OpenReview.net, 2022.
  * [9] Randall Davis, Howard E. Shrobe, and Peter Szolovits.  What is a knowledge representation?  AI Mag., 14(1):17–33, 1993.
  * [10] Franz Baader, Ian Horrocks, Carsten Lutz, and Ulrike Sattler.  An Introduction to Description Logic.  Cambridge University Press, 2017.
  * [11] Philipp Körner, Michael Leuschel, João Barbosa, Vítor Santos Costa, Verónica Dahl, Manuel V. Hermenegildo, Jose F. Morales, Jan Wielemaker, Daniel Diaz, Salvador Abreu, and Giovanni Ciatto.  Fifty years of prolog and beyond.  2022\.
  * [12] William A. Woods.  What’s in a link: Foundations for semantic networks.  pages 35–84, 1975.
  * [13] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models.  CoRR, abs/2303.18223, 2023.
  * [14] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu.  Harnessing the power of llms in practice: A survey on chatgpt and beyond.  CoRR, abs/2304.13712, 2023.
  * [15] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo.  Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph.  CoRR, abs/2307.07697, 2023.
  * [16] Yilin Wen, Zifeng Wang, and Jimeng Sun.  Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models.  CoRR, abs/2308.09729, 2023.
  * [17] Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, and Huajun Chen.  Instructprotein: Aligning human and protein language via knowledge instruction.  CoRR, abs/2310.03269, 2023.
  * [18] Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq R. Joty, and Soujanya Poria.  Chain of knowledge: A framework for grounding large language models with structured knowledge bases.  CoRR, abs/2305.13269, 2023.
  * [19] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu.  Unifying large language models and knowledge graphs: A roadmap.  CoRR, abs/2306.08302, 2023.
  * [20] Ran Xu, Hejie Cui, Yue Yu, Xuan Kan, Wenqi Shi, Yuchen Zhuang, Wei Jin, Joyce C. Ho, and Carl Yang.  Knowledge-infused prompting: Assessing and advancing clinical text data generation with large language models.  CoRR, abs/2311.00287, 2023.
  * [21] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  CoRR, abs/2211.12588, 2022.
  * [22] Zhen Bi, Ningyu Zhang, Yinuo Jiang, Shumin Deng, Guozhou Zheng, and Huajun Chen.  When do program-of-thoughts work for reasoning?  CoRR, abs/2308.15452, 2023.
  * [23] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  In NeurIPS, 2022.
  * [24] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  In ICLR. OpenReview.net, 2023.
  * [25] Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang.  Codekgc: Code language model for generative knowledge graph construction.  CoRR, abs/2304.09048, 2023.
  * [26] Xiang Chen, Lei Li, Ningyu Zhang, Xiaozhuan Liang, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen.  Decoupling knowledge from memorization: Retrieval-augmented prompt learning.  In NeurIPS, 2022.
  * [27] Matthew B. A. McDermott, Brendan Yap, Peter Szolovits, and Marinka Zitnik.  Knowledge graph-enhanced molecular contrastive learning with functional prompt.  Nature Machine Intelligence, 5:612–621, 2023.
  * [28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In NeurIPS, 2022.
  * [29] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  In ICLR. OpenReview.net, 2022.
  * [30] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola.  Automatic chain of thought prompting in large language models.  In ICLR. OpenReview.net, 2023.
  * [31] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Large language models are human-level prompt engineers.  In ICLR. OpenReview.net, 2023.
  * [32] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In EMNLP (1), pages 4222–4235. Association for Computational Linguistics, 2020.
  * [33] Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang.  $k$nn prompting: Beyond-context learning with calibration-free nearest neighbor inference.  In ICLR. OpenReview.net, 2023.
  * [34] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  CoRR, abs/2305.10601, 2023.
  * [35] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.  Graph of thoughts: Solving elaborate problems with large language models.  CoRR, abs/2308.09687, 2023.
  * [36] Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun.  PTR: prompt tuning with rules for text classification.  AI Open, 3:182–192, 2022.
  * [37] Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen.  Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction.  In WWW, pages 2778–2788. ACM, 2022.
  * [38] Yin Fang, Qiang Zhang, Ningyu Zhang, Zhuo Chen, Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen.  Knowledge graph-enhanced molecular contrastive learning with functional prompt.  Nature Machine Intelligence, 5:542–553, 2023.
  * [39] Ningyu Zhang, Jintian Zhang, Xiaohan Wang, Honghao Gui, Kangwei Liu, Yinuo Jiang, Xiang Chen, Shengyu Mao, Shuofei Qiao, Yuqi Zhu, Zhen Bi, Jing Chen, Xiaozhuan Liang, Yixin Ou, Runnan Fang, Zekun Xi, Xin Xu, Lei Li, Peng Wang, Mengru Wang, Yunzhi Yao, Bozhong Tian, Yin Fang, Guozhou Zheng, and Huajun Chen.  Knowlm technical report, 2023.
  * [40] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li.  Trustworthy llms: a survey and guideline for evaluating large language models’ alignment.  CoRR, abs/2308.05374, 2023.
  * [41] Garima Agrawal, Tharindu Kumarage, Zeyad Alghami, and Huan Liu.  Can knowledge graphs reduce hallucinations in llms? : A survey.  CoRR, abs/2311.07914, 2023.
  * [42] Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, and Huajun Chen.  Easyedit: An easy-to-use knowledge editing framework for large language models.  CoRR, abs/2308.07269, 2023.
  * [43] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang.  Editing large language models: Problems, methods, and opportunities.  CoRR, abs/2305.13172, 2023.
  * [44] Yichi Zhang, Zhuo Chen, Yin Fang, Lei Cheng, Yanxi Lu, Fangming Li, Wen Zhang, and Huajun Chen.  Knowledgeable preference alignment for llms in domain-specific question answering.  2023\.
  * [45] Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom.  Augmented language models: a survey.  CoRR, abs/2302.07842, 2023.
  * [46] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  CoRR, abs/2302.04761, 2023.
  * [47] Sheila McIlraith and Tran Cao Son.  Adapting golog for composition of semantic web services.  2002\.
  * [48] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huan, and Tao Gui.  The rise and potential of large language model based agents: A survey.  CoRR, abs/2309.07864, 2023.
  * [49] Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan.  Agents: An open-source framework for autonomous language agents.  CoRR, abs/2309.07870, 2023.
  * [50] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving AI tasks with chatgpt and its friends in huggingface.  CoRR, abs/2303.17580, 2023.
  * [51] Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.  CAMEL: communicative agents for "mind" exploration of large scale language model society.  CoRR, abs/2303.17760, 2023.
  * [52] Ian Horrocks, Peter F. Patel-Schneider, and Frank van Harmelen.  From SHIQ and RDF to OWL: the making of a web ontology language.  J. Web Semant., 1(1):7–26, 2003.
  * [53] Michael R. Genesereth.  Knowledge interchange format.  In KR, pages 599–600. Morgan Kaufmann, 1991.
  * [54] Grit Denker, Jerry R. Hobbs, David L. Martin, Srini Narayanan, and Richard J. Waldinger.  Accessing information and services on the daml-enabled web.  In SemWeb, volume 40 of CEUR Workshop Proceedings. CEUR-WS.org, 2001.
  * [55] Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang.  Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities.  CoRR, abs/2305.13168, 2023.
  * [56] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  CoRR, abs/2302.04023, 2023.
  * [57] Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu.  Exploring the feasibility of chatgpt for event extraction.  CoRR, abs/2303.03836, 2023.
  * [58] Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han.  Zero-shot information extraction via chatting with chatgpt.  CoRR, abs/2302.10205, 2023.
  * [59] Honghao Gui, Jintian Zhang, Hongbin Ye, and Ningyu Zhang.  Instructie: A chinese instruction-based information extraction dataset.  CoRR, abs/2305.11527, 2023.
  * [60] Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang.  Making language models better tool learners with execution feedback.  CoRR, abs/2305.13068, 2023.
  * [61] Ruijie Wang, Zhiruo Zhang, Luca Rossetto, Florian Ruosch, and Abraham Bernstein.  Nlqxform: A language model-based question to SPARQL transformer.  CoRR, abs/2311.07588, 2023.
  * [62] Apoorv Saxena, Aditay Tripathi, and Partha P. Talukdar.  Improving multi-hop question answering over knowledge graphs using knowledge base embeddings.  In ACL, pages 4498–4507. Association for Computational Linguistics, 2020.
  * [63] Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song.  Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations.  CoRR, abs/2304.14827, 2023.
  * [64] Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S. Yu, and Chuan Shi.  Towards graph foundation models: A survey and beyond.  CoRR, abs/2310.11829, 2023.
  * [65] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang.  Graphgpt: Graph instruction tuning for large language models.  CoRR, abs/2310.13023, 2023.
  * [66] Wen Zhang, Yushan Zhu, Mingyang Chen, Yuxia Geng, Yufeng Huang, Yajing Xu, Wenting Song, and Huajun Chen.  Structure pretraining and prompt tuning for knowledge graph transfer.  In WWW, pages 2581–2590. ACM, 2023.
  * [67] Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen.  Reasoning with language model prompting: A survey.  In ACL (1), pages 5368–5393. Association for Computational Linguistics, 2023.
  * [68] Zezhong Xu, Wen Zhang, Peng Ye, Hui Chen, and Huajun Chen.  Neural-symbolic entangled framework for complex query answering.  In NeurIPS, 2022.
  * [69] Wen Zhang, Jiaoyan Chen, Juan Li, Zezhong Xu, Jeff Z. Pan, and Huajun Chen.  Knowledge graph reasoning with logics and embeddings: Survey and perspective.  CoRR, abs/2202.07412, 2022.
  * [70] Jiarui Jin, Yangkun Wang, Kounianhua Du, Weinan Zhang, Zheng Zhang, David Wipf, Yong Yu, and Quan Gan.  Inductive relation prediction using analogy subgraph embeddings.  In ICLR. OpenReview.net, 2022.
  * [71] Zezhong Xu, Peng Ye, Hui Chen, Meng Zhao, Huajun Chen, and Wen Zhang.  Ruleformer: Context-aware rule mining over knowledge graph.  In COLING, pages 2551–2560. International Committee on Computational Linguistics, 2022.
  * [72] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent abilities of large language models.  Trans. Mach. Learn. Res., 2022, 2022.
  * [73] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  CoRR, abs/2005.14165, 2020.
  * [74] Yoshua Bengio and Edward J. Hu.  Scaling in the service of reasoning & model-based ml, 2023.
  * [75] Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen.  Structgpt: A general framework for large language model to reason over structured data.  CoRR, abs/2305.09645, 2023.
  * [76] Marvin Minsky.  The emotion machine: Commonsense thinking, artificial intelligence, and the future of the human mind.  Simon and Schuster, 2007.
  * [77] Douglas B. Lenat, Ramanathan V. Guha, Karen Pittman, Dexter Pratt, and Mary Shepherd.  CYC: toward programs with common sense.  Commun. ACM, 33(8):30–49, 1990.
  * [78] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In AAAI, pages 4444–4451. AAAI Press, 2017.
  * [79] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum.  Yago: a core of semantic knowledge.  In WWW, pages 697–706. ACM, 2007.
  * [80] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer.  Dbpedia - A large-scale, multilingual knowledge base extracted from wikipedia.  Semantic Web, 6(2):167–195, 2015.
  * [81] Denny Vrandecic and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  Commun. ACM, 57(10):78–85, 2014.
  * [82] Roberto Navigli and Simone Paolo Ponzetto.  Babelnet: Building a very large multilingual semantic network.  In ACL, pages 216–225. The Association for Computer Linguistics, 2010.
  * [83] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi.  ATOMIC: an atlas of machine commonsense for if-then reasoning.  In AAAI, pages 3027–3035. AAAI Press, 2019.
  * [84] Hongming Zhang, Xin Liu, Haojie Pan, Haowen Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song.  Aser: Towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities.  Artificial Intelligence, 309:103740, 2022.