# Ray Theory of Waves

K. F. Ren1,2∗, M. Yang3∗, Q. Duan2, C. Rozé1, C. Zhang1,4 and X. Han2∗
1CORIA/UMR 6614 CNRS - University of Rouen Normandy BP 12, 76801 Saint-Etienne
du Rouvray, France
2School of Physics, Xidian University, Xi’an 710071, China
3Institute of Radio Frequency Technology and Software, School of Integrated
Circuits and Electronics, Beijing Institute of Technology (BIT), Beijing
100081, China
4School of Information Science and Engineering, Southeast University, Nanjing
210096, China Corresponding authors:
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

In order to deal with the interaction of an electromagnetic wave with large
homogeneous objects of arbitrary shape with smooth surface we develop the _ray
theory of waves_ (RTW) which is composed of the _vectorial complex ray model_
(VCRM) and VCRM based singularity theory. By introducing the wavefront
curvature as an intrinsic property of rays, VCRM permits to predict the
amplitude and the phase of field at any point rigorously in the sense of ray
model. Its combination with the singularity theory remedies the discontinuity
in the ray model. In this letter, the wavefront equation, key physical law of
VCRM describing the relation between the wavefront curvatures of the incident
wave and the refracted/reflected wave, is derived for the most general case of
three dimension scattering. The strategy of the calculation scheme in RTW is
described. Typical applications to the prediction of the rainbow patterns of a
spheroidal drop are presented. The comparison to a rigorous numerical method,
multilevel fast multipole algorithm, shows that RTW can predict very fast and
precisely the scattered field even in the vicinity of caustics.

Ray theory of waves

††preprint: APS/123-QED

Introduction – The interaction of waves (light, electromagnetic or acoustic
waves) with large macroscopic objects concern great number of research
domains. Though the fundamental laws (Maxwell equations, etc.) have been well
established, their applications to practical problems are always challenging.
For example, the scientific community has not efficient methods to predict
with precision the scattered field of a wave by objects of complex forms and
size much larger than the wavelength. This fact has important impact on the
scientific research and the engineering design. In fact, researchers have made
great efforts to improve this situation. Many sophisticated numerical
techniques have been developed in the field of electromagnetic computation
[20, 25], such as the multilevel fast multipole algorithm (MLFMA)[26]. These
methods can be applied to objects of complex shape and composition but they
are resource and time consuming. The calculable size is also limited (less
than 200 wavelengths for dielectric objects[24]). Some rigorous solutions
exist for the particles of simple shapes (spheroid, ellipsoid), but the
calculable size is still at the same order as the pure numerical methods.
Different approximate methods have also been investigated to deal with
specific problems or in specific conditions[18, 7, 17, 25, 2]. For instance,
the catastrophe theory provides a powerful tool to study the morphology of the
scattering patterns near caustics and its classification by using polynomial
functions[4, 2]. So, we are still lack of efficient methods to simulate
directly with precision the scattering patterns of real large objects.

The _Ray Theory of Waves_ (RTW) presented in this letter intends to reply this
demand. It consists of the Vectorial Complex Ray Model (VCRM)[19, 12] and its
combination with singularity theory[6] by using the uniform approximation[5].
Its four _principles_ are: (i). All waves are described by the vectorial
complex rays and each ray possesses three properties: wave vector $k$, complex
electric vector $E$ and wavefront curvature $\mathbb{Q}$ (Fig. 1). (ii). All
properties of a ray after its interaction with a dioptric surface are
calculated by the wavefront equation, the vectorial Snell law and the Fresnel
formulas in wave vector components. (iii). In the region without singularity,
the total field is the summation of the complex amplitudes of all the rays at
that point, (iv). In the vicinity of singular points (caustics, critical
angles, particle surface, …), the singularity theory along with the uniform
approximation will be applied. The physical optics (PO)[9, 27], the
diffraction integration[21], etc. are examples of the special cases. The first
three principles form VCRM, key progress on the interaction of a wave with a
large non-spherical object, and the fourth one is based on the results of
VCRM.

Figure 1: Ray tracing in RTW.

Vectorial Complex Ray Model – _The wavefront equation_ describes the relation
between the wavefront curvatures of the incident wave and the refracted or
reflected wave as function of the curvature of the diopter. It is at the heart
of VCRM and we will derivate it for three dimension (3D) scattering.

Consider a ray incident on a curved surface $\Gamma$ (blue in Fig. 2) at a
point $O$. When an adjacent ray passing by the point $A$ on the incident
wavefront $\Sigma$ (black in Fig. 2) reaches the diopter at $H$, the ray
refracted at $O$ arrives at $E$ on the refracted wavefront $\Sigma^{\prime}$
(green in Fig. 2). We note $B$ as the projection point of $A$ on the tangent
plane of $\Sigma$, $C$ and $G$ the projection points of $H$ respectively on
$\Gamma$ and $\Sigma$’. The curvature of the surface $\Sigma$ is described by
its curvature matrix $\mathbb{Q}_{i}$ in the base
$(\hat{\mbox{\boldmath$u$}}_{1},\hat{\mbox{\boldmath$u$}}_{2})$. Similarly,
the curvatures of $\Sigma^{\prime}$ and $\Gamma$ are described respectively by
their curvature matrices $\mathbb{Q}_{r}$ and $\mathbb{C}$ in the
corresponding bases
$(\hat{\mbox{\boldmath$s$}}_{1},\hat{\mbox{\boldmath$s$}}_{2})$ and
$(\hat{\mbox{\boldmath$v$}}_{1},\hat{\mbox{\boldmath$v$}}_{2})$ (not presented
in the figure for clarity). According to the differential geometry, in the
vicinity of the tangent point, the distance between the point on the curved
surface and the projected point on the tangent surface can be expressed as
function of the curvature matrix. So, the infinitesimal distances
$\delta_{i}=\overline{AB}$, $\delta_{r}=\overline{HG}$ and
$\delta_{c}=\overline{HC}$ are given by

$\delta_{i}=\frac{1}{2}\mbox{\boldmath$t$}_{i}\mathbb{Q}_{i}\mbox{\boldmath$t$}_{i}^{T},\quad\delta_{r}=\frac{1}{2}\mbox{\boldmath$t$}_{r}\mathbb{Q}_{r}\mbox{\boldmath$t$}_{r}^{T},~{}~{}\delta_{c}=\frac{1}{2}\mbox{\boldmath$t$}_{c}\mathbb{C}\mbox{\boldmath$t$}_{c}^{T}$

where $\mbox{\boldmath$t$}_{i}=\overrightarrow{OB}$,
$\mbox{\boldmath$t$}_{r}=\overrightarrow{EG}$ and
$\mbox{\boldmath$t$}_{c}=\overrightarrow{OC}$. The prime T stands for the
transpose of a matrix. Note that the quantities $\delta$ and $t$ are all
infinitely small, but the differential symbol $d$ of $d\delta$ and
$d\mbox{\boldmath$t$}$ is omitted here for clarity.

Figure 2: Schema of the wavefronts and the diopter: an arbitrary wave incident
on a curved surface.

The phase difference between the two wavefronts $\Sigma$ and $\Sigma^{\prime}$
being constant, so $k_{i}\overline{AH}=k_{r}\overline{OE}$. On the other hand,
from the geometry we have
$\overline{AH}=\delta_{i}+\mbox{\boldmath$t$}_{c}\cdot\hat{\mbox{\boldmath$k$}}_{i}-\delta_{c}\hat{\mbox{\boldmath$n$}}\cdot\hat{\mbox{\boldmath$k$}}_{i}$
and
$\overline{OE}=\delta_{r}+\mbox{\boldmath$t$}_{c}\cdot\hat{\mbox{\boldmath$k$}}_{r}-\delta_{c}\hat{\mbox{\boldmath$n$}}\cdot\hat{\mbox{\boldmath$k$}}_{r}$.
These lead to

$k_{i}(\delta_{i}+\mbox{\boldmath$t$}_{c}\cdot\hat{\mbox{\boldmath$k$}}_{i}-\delta_{c}\hat{\mbox{\boldmath$n$}}\cdot\hat{\mbox{\boldmath$k$}}_{i})=k_{r}(\delta_{r}+\mbox{\boldmath$t$}_{c}\cdot\hat{\mbox{\boldmath$k$}}_{r}-\delta_{c}\hat{\mbox{\boldmath$n$}}\cdot\hat{\mbox{\boldmath$k$}}_{r})$

The terms $k_{i}\mbox{\boldmath$t$}_{c}\cdot\hat{\mbox{\boldmath$k$}}_{i}$ and
$k_{r}\mbox{\boldmath$t$}_{c}\cdot\hat{\mbox{\boldmath$k$}}_{r}$ are
independent of any curvatures, so we have $k_{i,\tau}=k_{r,\tau}$. This is
also true for the reflected ray $k_{i,\tau}=k_{l,\tau}$. We establish thus the
Snell law in vector form: the tangent components of the incident, reflected
and refracted wave vectors are equal.

$k_{i,\tau}=k_{l,\tau}=k_{r,\tau}$ (1)

The rest of the equation gives rise to

$k_{i}\mbox{\boldmath$t$}_{i}\mathbb{Q}_{i}\mbox{\boldmath$t$}_{i}^{T}-k_{r}\mbox{\boldmath$t$}_{r}\mathbb{Q}_{r}\mbox{\boldmath$t$}_{r}^{T}=\hat{\mbox{\boldmath$n$}}\cdot(\mbox{\boldmath$k$}_{i}-\mbox{\boldmath$k$}_{r})\mbox{\boldmath$t$}_{c}\mathbb{C}\mbox{\boldmath$t$}_{c}^{T}$
(2)

We express now $\mbox{\boldmath$t$}_{i}$ and $\mbox{\boldmath$t$}_{r}$ as
function of $\mbox{\boldmath$t$}_{c}$ by using the projection relations

$\mbox{\boldmath$t$}_{i}=\mathbb{P}_{i}\mbox{\boldmath$t$}_{c}+\mathcal{O}(\mbox{\boldmath$t$}_{i}),\quad\mbox{\boldmath$t$}_{r}=\mathbb{P}_{r}\mbox{\boldmath$t$}_{c}+\mathcal{O}(\mbox{\boldmath$t$}_{r})$
(3)

where $\mathbb{P}_{i}$ and $\mathbb{P}_{r}$ are the projection matrices of the
diopter base
$(\hat{\mbox{\boldmath$s$}}_{1}^{i},\hat{\mbox{\boldmath$s$}}_{2}^{i})$
respectively on the incident and refracted wavefront bases
$(\hat{\mbox{\boldmath$u$}}_{1},\hat{\mbox{\boldmath$u$}}_{2})$ and
$(\hat{\mbox{\boldmath$v$}}_{1},\hat{\mbox{\boldmath$v$}}_{2})$

$\mathbb{P}_{i}=\\!\\!\left(\\!\\!\begin{array}[]{cc}\hat{\mbox{\boldmath$s$}}_{1}^{i}\cdot\hat{\mbox{\boldmath$u$}}_{1}&\hat{\mbox{\boldmath$s$}}_{1}^{i}\cdot\hat{\mbox{\boldmath$u$}}_{2}\\\
\hat{\mbox{\boldmath$s$}}_{2}^{i}\cdot\hat{\mbox{\boldmath$u$}}_{1}&\hat{\mbox{\boldmath$s$}}_{2}^{i}\cdot\hat{\mbox{\boldmath$u$}}_{2}\\\
\end{array}\\!\\!\right),\quad\mathbb{P}_{r}=\\!\\!\left(\\!\\!\begin{array}[]{cc}\hat{\mbox{\boldmath$s$}}_{1}^{i}\cdot\hat{\mbox{\boldmath$v$}}_{1}&\hat{\mbox{\boldmath$s$}}_{1}^{i}\cdot\hat{\mbox{\boldmath$v$}}_{2}\\\
\hat{\mbox{\boldmath$s$}}_{2}^{i}\cdot\hat{\mbox{\boldmath$v$}}_{1}&\hat{\mbox{\boldmath$s$}}_{2}^{i}\cdot\hat{\mbox{\boldmath$v$}}_{2}\\\
\end{array}\\!\\!\right)$ (4)

Eq. (2) is then written as

$\mbox{\boldmath$t$}_{c}\left(k_{i}\mathbb{P}_{i}\mathbb{Q}_{i}\mathbb{P}_{i}^{T}-k_{r}\mathbb{P}_{r}\mathbb{Q}_{r}\mathbb{P}_{r}^{T}\right)\mbox{\boldmath$t$}_{c}^{T}=\hat{\mbox{\boldmath$n$}}\cdot(\mbox{\boldmath$k$}_{i}-\mbox{\boldmath$k$}_{r})\mbox{\boldmath$t$}_{c}\mathbb{C}\mbox{\boldmath$t$}_{c}^{T}$
(5)

$\mbox{\boldmath$t$}_{c}$ being arbitrary, we deduce finally the wavefront
equation

$k_{r}\mathbb{P}_{r}\mathbb{Q}_{r}\Theta_{r}^{T}=k_{i}\mathbb{P}_{i}\mathbb{Q}_{i}\mathbb{P}_{i}^{T}+\mathbb{C}\hat{\mbox{\boldmath$n$}}\cdot(\mbox{\boldmath$k$}_{r}-\mbox{\boldmath$k$}_{i})$
(6)

The wavefront equations given in [11] and [14] (sec. 4.2) correspond to the
special case where one of the principal direction is in the incident plane. In
the case of scattering in a symmetric plane, Eq. (6) is reduced to two scalar
equations which are the same as those obtained by Hamilton optics in [22] (p
49) and [9] (p 186):

$\displaystyle k_{r}\kappa_{r1}$ $\displaystyle=$ $\displaystyle
k\kappa_{i1}+(k_{rn}-k_{in})\kappa_{c1}$ (7)
$\displaystyle\frac{k_{rn}^{2}}{k_{r}}\kappa_{r2}$ $\displaystyle=$
$\displaystyle\frac{k_{in}^{2}}{k_{i}}\kappa_{i2}+(k_{rn}-k_{in})\kappa_{c2}$
(8)

where the indice $1$ and $2$ indicate respectively the principal curvatures in
the directions perpendicular and parallel to the symmetric plane.

The two equations (1) and (6) tell all what we need for the vectorial complex
ray tracing: the directions of the rays and their wavefront structure. With
some simple considerations described below, we can calculate the complex
amplitude of all rays at any point.

_The amplitude of a ray_ , in the propagation of a wave and its interaction
with an object, is affected by two factors: the Fresnel factor
$\varepsilon_{X,p}$ and the divergence factor $\mathcal{D}$. The amplitude of
the wave at a point $M$ is then

$A=A_{0}|\varepsilon_{X,p}|\sqrt{\mathcal{D}}$ (9)

where $A_{0}$ is the amplitude at the first incident point.
$\varepsilon_{X,p}$ of $p^{th}$ order emergent ray (Fig. 1) is given as

$\varepsilon_{X,p}=\left\\{\begin{array}[]{ll}r_{X,0}&p=0\\\
t_{X,0}t_{X,p}\displaystyle{\prod_{q=1}^{p-1}r_{X,q}}&p\geq
1\end{array}\right.$ (10)

$r_{X,q}$ and $t_{X,q}$ are respectively the Fresnel reflection and refraction
coefficients of polarization $X$ ($\perp$ or $\parallel$) at the $q^{th}$
interaction and given in RTW as function of the normal components $k_{in}$ and
$k_{rn}$, respectively, of the reflected and refracted wave vectors

$\displaystyle r_{\perp}$ $\displaystyle=$
$\displaystyle\frac{k_{in}-k_{rn}}{k_{in}+k_{rn}},\quad
r_{\parallel}=\frac{m^{2}k_{in}-k_{rn}}{m^{2}k_{in}+k_{rn}}$ (11)
$\displaystyle t_{\perp}$ $\displaystyle=$
$\displaystyle\frac{2k_{in}}{k_{in}+k_{rn}},\quad
t_{\parallel}=\frac{2mk_{in}}{m^{2}k_{in}+k_{rn}}$ (12)

The divergence factor is

$\mathcal{D}=\left|\frac{\kappa_{G2}}{\kappa_{G1}^{\prime}}\cdot\frac{\kappa_{G3}}{\kappa_{G2}^{\prime}}\cdots\frac{\kappa_{GM}}{\kappa_{Gq}}\right|$
(13)

where $\kappa_{Gq}$ and $\kappa_{Gq}^{\prime}$ are respectively the Gaussian
curvatures of the wavefronts of the incident and refracted/reflected waves at
$q^{th}$ interaction point, and $\kappa_{GM}$ the Gaussian curvature of the
wavefront at the point $M$. The relation between the principal wavefront
curvatures at two successive points are given by

$\kappa_{\mu,q+1}=1/(1/\kappa_{\mu,q}-d)$ (14)

where $\mu=1,2$ and $d$ is the distance between the two points.

_The phase of a ray_ may change due to the reflection $\Phi_{F}$, the optical
path $\Phi_{P}$ and the focal lines $\Phi_{f}$[23, 6]. $\Phi_{F}$ is
determined by the argument of the Fresnel factor

$\Phi_{F}=\arg(\varepsilon_{X,p})$ (15)

$\Phi_{P}$ is computed directly according to the optical path. Let
$\mbox{\boldmath$r$}_{i}$ and $\mbox{\boldmath$k$}_{i}$ be respectively the
position vector and the wave vector of the incident wave at the $i^{th}$
interaction point, $\Phi_{P}$ of a ray after $q$ interactions with the
particle is given by

$\Phi_{P}=-\mbox{\boldmath$k$}_{1}\cdot\mbox{\boldmath$r$}_{1}+\mbox{\boldmath$k$}^{\prime}_{q}\cdot\mbox{\boldmath$r$}_{q}+\sum_{i=1}^{q-1}\mbox{\boldmath$k$}_{i+1}\cdot(\mbox{\boldmath$r$}_{i+1}-\mbox{\boldmath$r$}_{i})$
(16)

where $\mbox{\boldmath$k$}_{q}^{\prime}$ is the wave vector of the emergent
ray.

Each time a wave passes a focal line (Fig. 1) the phase jumps a value of
$\pi/2$ [23, 19]. This phenomenon corresponds to the sign changes of the
wavefront curvatures, so the calculation of $\Phi_{f}$ in VCRM is just a
matter to count the number $N_{f}$ of sign changes of the wavefront
curvatures, thus

$\Phi_{f}=N_{f}\frac{\pi}{2}$ (17)

If the incident wave is not a plane wave, its phase $\Phi_{i}$ at the first
incident point should be counted. So the phase of a ray at the observation
point $M$ is

$\Phi(M)=\Phi_{i}+\Phi_{F}+\Phi_{P}+\Phi_{f}$ (18)

Eqs. (9)-(18) permit to calculate the amplitudes and the phases of all rays by
ray tracing (Fig. 1). Finally, the properties of all the rays at any point are
determined by Eqs. (1), (9) and (18).

Singularity correction – VCRM improves significantly the flexibility and the
precision of classical ray model, but it still fails to predict correctly the
field in the vicinity of the singularity points. However, it can calculate the
complex electric field at any point rigorously in the sense of ray model and
directly for a given object. The sigularity theory[6] and the uniform
approximation[5] can then be applied for the remedy. By a heuristic example,
we have shown[27] that VCRM permits to reveals clearly the origin of the
imprecision in the Airy theory and by replacing the polynomial function with
directly calculated field, our method predicts very good results compared to
the rigorous theory.

Thanks to the rapid development of computing technology, the polynomial or
other approximate functions used in the singularity theory[4] can be replaced
by the directly calculated field of VCRM. This should improve the precision
and the flexibility in the singularity correction. This step needs a strategy
special to RTW.

Strategy of RTW – In VCRM the forward ray tracing is used, i.e. the directions
of the emergent rays are determined by the incident rays. To obtain the field
at a given point or a given direction, a special strategy is to be employed.

Traditionally, a physical problem is considered solved if the relation between
a physical quantity $Y$ and its variables
$\mbox{\boldmath$x$}=\\{x_{i}\\}(i=1,2,\cdots)$ is established rigorously or
approximately in the form
$Y=\mathcal{F}(\mbox{\boldmath$x$};\mbox{\boldmath$\alpha$})$, where
$\mbox{\boldmath$\alpha$}=\\{\alpha_{j}\\}(j=1,2,\cdots)$ are the problem
related parameters and the operator $\mathcal{F}$ can be in usual or special
functions, a differential or integral operator, or their combination. The Mie
theory and Airy theory are two typical examples. Otherwise, we establish an
implicit relation between $Y$ and $x$ in the form
$\mathcal{F}(Y,\mbox{\boldmath$x$};\mbox{\boldmath$\alpha$})=0$ then solve the
equation numerically. This is largely used in the electromagnetic computation.

RTW uses neither of the two strategies. We express $Y$ and $x$ through a set
of intermediate variables $\mbox{\boldmath$t$}=\\{t_{i}\\}$:
$Y=\mathcal{F}_{1}(\mbox{\boldmath$t$};\mbox{\boldmath$\alpha$})$ and
$\mbox{\boldmath$x$}=\mathcal{F}_{2}(\mbox{\boldmath$t$};\mbox{\boldmath$\alpha$})$
and calculate $Y$ and $x$ in a relevant range of $t$, then establish the
relationship between $Y$ and $x$ with numerical technique. For instance to
obtain the scattering field, we calculate the amplitudes and the phase
$Y=(A,\Phi)$, and the directions of all the emergent rays
$\mbox{\boldmath$x$}=(\theta,\phi)$ as function of the incident rays
$\mbox{\boldmath$t$}=(\theta_{i},\phi_{i})$ and the particle properties
$\alpha$. The interpolation is applied to calculate the total complex
amplitude of all rays in the interested region with[27] or without [13] wave
effect correction.

Examples of applications – VCRM has been applied successfully to the
simulation of the rainbow pattern of an oblate ellipsoid [12]. But two
decernible discrepancies are observed when compared to the experimental
results (Fig. 6 in [12]). The first in the vicinity of the caustics is caused
by the wave effect. We will show that the combination of VCRM with PO used in
[27] for a spherical particle can be extended to a non-spherical particle. The
second in the large scattering angles is due probably to experimental
uncertainty. So we will compare the results of RTW with those of a rigorous
numerical method.

Table 1: Peak positions (in deg.) of supernumerary bows of a sphere (refractive index $m=1.333$, $\lambda=0.6328~{}\upmu$m). $a=50~{}\upmu$m |  $a=500~{}\upmu$m
---|---
$K$ | Airy | RTW | Debye | $K$ | Airy | RTW | Debye
0 | 139.50 |  _139.47_ ∗ | 139.47 | 0 | 138.26 |  _138.26_ ∗ | 138.26
1 | 142.98 | 142.88 | 142.87 | 1 | 139.01 | 139.00 | 139.00
2 | 145.41 | 147.24 | 147.23 | 21 | 145.13 | 144.98 | 149.99
4 | 149.37 | 149.01 | 148.99 | 41 | 149.14 | 148.81 | 148.81
6 | 152.72 | 152.14 | 152.12 | 61 | 152.53 | 151.97 | 151.97
8 | 155.73 | 154.91 | 154.88 | 81 | 155.55 | 154.76 | 154.76
10 | 158.51 | 157.43 | 157.39 | 101 | 158.34 | 157.29 | 157.29
12 | 161.10 | 159.77 | 159.72 | 121 | 160.95 | 159.64 | 159.64
∗ calculated by combination of VCRM and PO.

To ease the understanding, we begin with the classical problem – rainbow of a
spherical drop. After the first satisfactory explanation of Descartes[10],
Airy proposed a theory to calculate the scattering pattern of a rainbow[3].
Since almost two centuries, his theory has been studied by many researchers
from different points of view[10, 1]. At the same time scientists have been
wondering its precision from the beginning[8, 10] and tried to improve it[15,
16]. We have shown recently that by using the amplitude and the phase
calculated with VCRM our method predict very precisely the rainbow
patterns[27]. It is shown further in Tab. 1 that pure VCRM permits to predict
very precisely all the supernumerary peaks except the main one ($K=0$) which
needs the combination of VCRM and PO. Tab. 1 reveals also the fact that,
contrary to what believed commonly[23], the precision of Airy theory is almost
independent of the particle size if observed at given angles.

Figure 3: Comparison of the scattering patterns calculated by VCRM and MLFMA
for an oblate spheroid of semi-axes $a=b=100~{}\upmu$m, $c=90~{}\upmu$m of
water $m=1.333$. The incident plane wave of wavelength
$\lambda=0.6328~{}\upmu$m is along $x$-axis. Figure 4: Numerical comparison of
the scattering diagrams in the plane $z=0$ calculated by three methods MLFMA,
VCRM and RTW with the same parameters as in Fig. 3.

To demonstrate the applicability of RTW to objects of any shape, we show the
scattering patterns in the first and second rainbows of an oblate spheroid
calculated by MLFMA and RTW in Fig. 3, top half for MLFMA and bottom half for
VCRM. The computation for MLFMA[26] with an angle resolution of $0.1^{\circ}$
has taken 997 GB memory (Intel Xeon Platinum 8276 CPU) and 40 hour CPU time
with 160 threads. The computation time for VCRM on PC (Intel i9-13900ks, RAM
32 GB) is 9.4 min. for an angle resolution of $0.01^{\circ}$. The agreement of
the two methods are very good including the fine structure except in the
vicinity of the caustics. This could be remedied by PO with 2D diffraction
integration that we are working on. Instead, a quantitative comparison of the
scattering diagram on the symmetric plane $\psi=0$ is given in Fig. 4. The
result of RTW has been calculated by simple integration. It is clear that RTW
is in good agreement with MLFMA, not only in the vicinity of the rainbow
angles but also in the Alexander region. The discrepancy around $160^{\circ}$
is due to 2D caustics which needs 2D integration.

Conclusions – The fundamentals and the strategy of the _Ray Theory of Waves_
for scattering of large non-spherical objects with smooth surface are
presented. The efficiency and the precision of RTW are demonstrated by
comparison of the scattering patterns of an ellipsoidal drop calculated by RTW
and MLFMA. The singularity correction for 3D scattering is to be developed[6,
21]. The applications of RTW in the domains concerning the interaction of
waves with objects of irregular shape are also to be explored.

Acknowledgements This work was partially supported by the National Natural
Science Foundation of China under Grants 62231003 and 62205262, the Agence
Nationale de la Recherche (ANR-13-BS090008-01, AMO-COPS) of France and CRIANN
(Normandy, France).

## References

  * [1] J. A. Adam.  The mathematical physics of rainbows and glories.  Physics Reports, 356:229–365, 2002.
  * [2] J. A. Adam.  Rays, Waves, and Scattering.  Princeton University Press, 2017.
  * [3] G. Airy.  On the intensity of light in the neighbourhood of a caustic.  Trans. Cambridge Philos. Soc., 6(268):379–403, 1938.
  * [4] M. Berry and C. Upstill.  Catastrophe optics: Morphologies of caustics and their diffraction patterns.  In Prog. Opt. Elsevier, 1980.
  * [5] M. V. Berry.  Uniform approximation: a new concept in wave theory.  Sci. P¨rogress, 57(225):43–64, 1969.
  * [6] M. V. Berry.  The singularities of light: intensity, phase, polarisation.  Light Sci Appl., 12(238):1–4, 2023.
  * [7] L. Bi and P. Yang.  Physical-geometric optics hybrid methods for computing the scattering and absorption properties of ice crystals and dust aerosols.  In Light Scattering Reviews, pages 69–114. Springer-Praxis, 2013\.
  * [8] M. Boitel.  Sur les arcs surnuméraires qui accompagnent l’arc-en-ciel.  Comptes rendus de l’Acad. Sci.., CVI:1522–1524, 1888.
  * [9] M. Born and E. Wolf.  Principles of optics, 7th ed. Cambridge University Press, 1999.
  * [10] C. B. Boyer.  The Rainbow, From Myth to Mathematics.  Princeton University Press, 1987.
  * [11] G. A. Deschamps.  Ray techniques in electromagnetics.  Proc. IEEE, 60(9):1022–1035, 1972.
  * [12] Q. Duan, F. R. A. Onofri, X. Han, and K. F. Ren.  Generalized rainbow patterns of oblate drops simulated by a ray model in three dimensions.  Opt. Lett., 46(18):4585–4588, 2021.
  * [13] Q. Duan, F. R. A. Onofri, X. Han, and K. F. Ren.  Numerical implementation of three-dimensional vectorial complex ray model and application to rainbow scattering of spheroidal drops.  Optics Express, 31(21):34980–35002, 2023.
  * [14] G. L. James.  Geometrical theory of diffraction for electromagnetic waves.  Petter Peregrinus LTD., IEEE edition, 1980.
  * [15] V. Khare and H. M. Nussenzveig.  Theory of the rainbow.  Phys. Rev. Lett., 33:976–980, 1974.
  * [16] G. P. Können and J. H. de Boer.  Polarized rainbow.  Appl. Opt., 18(12):1961–1965, 1979.
  * [17] P. L. Marston, editor.  Selected papers on geometrical aspects of scattering.  SPIE, United States, 1994.  Title of Book: Selected papers on geometrical aspects of scattering Editors of Book: P.L. Marston.
  * [18] M. I. Mishchenko, J. W. Hovenier, and L. D. Travis.  Light Scattering by Nonspherical Particles: Theory, Measurements, and Applications.  Academic Press, san Diego, 2000.
  * [19] K. F. Ren, F. Onofri, C. Rozé, and T. Girasole.  Vectorial complex ray model and application to two-dimensional scattering of plane wave by a spheroidal particle.  Opt. Lett., 36(3):370–372, 2011.
  * [20] X. Q. Sheng and W. Song.  Essentials of computational electromagnetics.  Wiley, Singapore, 2012.
  * [21] J. J. Stamnes.  Wave in focal regions.  Adam Hilger, institute of physics publishing edition, 1986.
  * [22] J. L. Synge.  Geometrical optics, an introduction to Hamiltons method.  Cambrige University Press, 1937.
  * [23] H. C. van de Hulst.  Light scattering by small particles.  John Wiley & Sons, New York, 1957.
  * [24] Y. Wu, M. Yang, X. Sheng, and K. F. Ren.  Computation of scattering matrix elements of large and complex shaped absorbing particles with multilevel fast multipole algorithm.  J. Quant. Spectrosc. Radiat. Transfer, 156:88–96, 2015.
  * [25] Y. M. Wu and W. C. Chew.  The modern high frequency methods for solving electromagnetic scattering problems.  Progress ElM Res., 156:63–82, 2016.
  * [26] M. L. Yang, B. Y. Wu, H. W. Gao, and X. Q. Sheng.  A ternary parallelization approach of mlfma for solving electromagnetic scattering problems with over 10 billion unknowns.  IEEE Transactions on Antennas and Propagation, 67(11):6965–6978, 2019.
  * [27] C. Zhang, C. Rozé, and K. F. Ren.  Airy theory revisited with the method combining vectorial complex ray model and physical optics.  Opt. Lett., 47(9):2149–2152, 2022.


# Generative Graph Unification Framework: Enhancing Representation Learning
across Diverse Tasks

Ben Trovato<EMAIL_ADDRESS>1234-5678-9012 ,  G.K.M. Tobin
<EMAIL_ADDRESS>Institute for Clarity in DocumentationP.O. Box
1212DublinOhioUSA43017-6221 ,  Lars Thørväld  The Thørväld Group1 Thørväld
CircleHeklaIceland<EMAIL_ADDRESS>,  Valerie Béranger  Inria Paris-
RocquencourtRocquencourtFrance ,  Aparna Patel  Rajiv Gandhi UniversityRono-
HillsDoimukhArunachal PradeshIndia ,  Huifen Chan  Tsinghua University30
Shuangqing RdHaidian QuBeijing ShiChina ,  Charles Palmer  Palmer Research
Laboratories8600 Datapoint DriveSan AntonioTexasUSA78229<EMAIL_ADDRESS>,
John Smith  The Thørväld Group1 Thørväld CircleHeklaIceland
<EMAIL_ADDRESS>and  Julius P. Kumquat  The Kumquat ConsortiumNew
YorkUSA<EMAIL_ADDRESS>
(2024)

# Embracing Generative Model for Unified Graph Task Representation Learning

Ben Trovato<EMAIL_ADDRESS>1234-5678-9012 ,  G.K.M. Tobin
<EMAIL_ADDRESS>Institute for Clarity in DocumentationP.O. Box
1212DublinOhioUSA43017-6221 ,  Lars Thørväld  The Thørväld Group1 Thørväld
CircleHeklaIceland<EMAIL_ADDRESS>,  Valerie Béranger  Inria Paris-
RocquencourtRocquencourtFrance ,  Aparna Patel  Rajiv Gandhi UniversityRono-
HillsDoimukhArunachal PradeshIndia ,  Huifen Chan  Tsinghua University30
Shuangqing RdHaidian QuBeijing ShiChina ,  Charles Palmer  Palmer Research
Laboratories8600 Datapoint DriveSan AntonioTexasUSA78229<EMAIL_ADDRESS>,
John Smith  The Thørväld Group1 Thørväld CircleHeklaIceland
<EMAIL_ADDRESS>and  Julius P. Kumquat  The Kumquat ConsortiumNew
YorkUSA<EMAIL_ADDRESS>
(2024)

# Generative Meta-framework for Unifying Diverse Graph Tasks

Ben Trovato<EMAIL_ADDRESS>1234-5678-9012 ,  G.K.M. Tobin
<EMAIL_ADDRESS>Institute for Clarity in DocumentationP.O. Box
1212DublinOhioUSA43017-6221 ,  Lars Thørväld  The Thørväld Group1 Thørväld
CircleHeklaIceland<EMAIL_ADDRESS>,  Valerie Béranger  Inria Paris-
RocquencourtRocquencourtFrance ,  Aparna Patel  Rajiv Gandhi UniversityRono-
HillsDoimukhArunachal PradeshIndia ,  Huifen Chan  Tsinghua University30
Shuangqing RdHaidian QuBeijing ShiChina ,  Charles Palmer  Palmer Research
Laboratories8600 Datapoint DriveSan AntonioTexasUSA78229<EMAIL_ADDRESS>,
John Smith  The Thørväld Group1 Thørväld CircleHeklaIceland
<EMAIL_ADDRESS>and  Julius P. Kumquat  The Kumquat ConsortiumNew
YorkUSA<EMAIL_ADDRESS>
(2024)

# Harnessing Generative Models for Task Unification in Graph Learning

Ben Trovato<EMAIL_ADDRESS>1234-5678-9012 ,  G.K.M. Tobin
<EMAIL_ADDRESS>Institute for Clarity in DocumentationP.O. Box
1212DublinOhioUSA43017-6221 ,  Lars Thørväld  The Thørväld Group1 Thørväld
CircleHeklaIceland<EMAIL_ADDRESS>,  Valerie Béranger  Inria Paris-
RocquencourtRocquencourtFrance ,  Aparna Patel  Rajiv Gandhi UniversityRono-
HillsDoimukhArunachal PradeshIndia ,  Huifen Chan  Tsinghua University30
Shuangqing RdHaidian QuBeijing ShiChina ,  Charles Palmer  Palmer Research
Laboratories8600 Datapoint DriveSan AntonioTexasUSA78229<EMAIL_ADDRESS>,
John Smith  The Thørväld Group1 Thørväld CircleHeklaIceland
<EMAIL_ADDRESS>and  Julius P. Kumquat  The Kumquat ConsortiumNew
YorkUSA<EMAIL_ADDRESS>
(2024)

# Exploring Task Unification in Graph Representation Learning via Generative
Approach

Ben Trovato<EMAIL_ADDRESS>1234-5678-9012 ,  G.K.M. Tobin
<EMAIL_ADDRESS>Institute for Clarity in DocumentationP.O. Box
1212DublinOhioUSA43017-6221 ,  Lars Thørväld  The Thørväld Group1 Thørväld
CircleHeklaIceland<EMAIL_ADDRESS>,  Valerie Béranger  Inria Paris-
RocquencourtRocquencourtFrance ,  Aparna Patel  Rajiv Gandhi UniversityRono-
HillsDoimukhArunachal PradeshIndia ,  Huifen Chan  Tsinghua University30
Shuangqing RdHaidian QuBeijing ShiChina ,  Charles Palmer  Palmer Research
Laboratories8600 Datapoint DriveSan AntonioTexasUSA78229<EMAIL_ADDRESS>,
John Smith  The Thørväld Group1 Thørväld CircleHeklaIceland
<EMAIL_ADDRESS>and  Julius P. Kumquat  The Kumquat ConsortiumNew
YorkUSA<EMAIL_ADDRESS>
(2024)

###### Abstract.

Recent advancements in self-supervised graph representation learning,
represented by graph autoencoders (GAEs), have led to numerous breakthroughs.
Despite these successes, this field encompasses various tasks, including node-
level, link-level, graph-level, and graph knowledge transfer tasks, the
existing self-supervised methods face challenges in concurrently addressing
all these tasks. A few attempts have been made to adopt ”pretrain-finetune”
paradigm to solve these tasks, which, to a certain extent, achieve task
unification through downstream fine-tuning. However, inadequacies such as
substantial computational resource consumption during fine-tuning and a gap
between pretraining and downstream tasks remain prevalent. In addition, a most
recent emerging ”pretraining + prompt” paradigm can mitigate this gap to some
extent, while it requires meticulous design of prompt techniques and typically
only unifies up to two tasks.

Striving to address these challenges, we introduce a unified Generate then
Discriminate” framework that is able to seemlessly align

Our approach facilitates the unification of various levels of graph tasks,
utilizing a generative scheme to pretrain models that inherently capture the
nuances of graph data. We further employ Generative Adversarial Networks
(GANs) to enhance the robustness and discriminative power of the learned
representations. We further develop a novel adversarially training strategy to
enhance the the robustness and discriminative power of the learned
representations. To validate the effectiveness and universality of the
proposed framework, we conduct extensive experiments on 21 datasets across
four types of graph tasks. The results exhibits that the proposed framework
can consistently outperformce various kinds of state-of-the-art methods,
demonstrates its potential as a noval

The efficacy and superiority of our framework are firmly established through
extensive and comprehensive experiments.

Graph Neural Networks, Unified Graph Learning, Generative Graph Learning

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; August 25–29, 2024; Barcelona, Spain††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Knowledge representation
and reasoning††ccs: Networks Network algorithms

##  1\. Introduction

Graph representation learning has attracted considerable interest due to its
great potential to extract important insights from complex graph structures.
Graph Neural Networks (GNNs) (wu2020comprehensive), as a key approach in this
field, have found widespread applications across numerous domains, including
social networks, recommendation systems, and knowledge graphs (wu2020graph;
wu2022graph; hamaguchi2017knowledge), etc. The ever-growing landscape of
graph-structured data manifests diversity in domain and task type. For
instance, the citation networks, protein-protein interaction networks, and
road networks (fout2017protein; zhou2020graph) exhibit distinct
characteristics. Meanwhile, the related tasks ranging broadly from node
classification, link prediction, graph classification, and transfer learning
(chen2020graph; abu2020n; kumar2020link). These issues poses challenge in
using a one-for-all approach to deal with various kinds of graph tasks.

Figure 1. Comparison of our approach with existing “pre-training + fine-
tuning” paradigm

To alleviate the reliance on labeled data and extract generic knowledge from
graph data that can be applied to various downstream tasks, the ”pretraining +
fine-tuning” paradigm (xia2022survey; hu2020gpt; qiu2020gcc; xia2022towards)
has emerged as a feasible solution. Typically, these approaches employ easily
accessible, simple pretext tasks for pretraining GNNs, such as contrastive
learning and link prediction. Following pretraining, the model is then fine-
tuned on a diverse array of downstream tasks using the pretrained weights as
an initialization. GPT-GNN (hu2020gpt) integrates a self-supervised attribute
graph generation task in pretraining GNNs, aiding in capturing both structural
and semantic properties. The likelihood of graph generation is bifurcated into
attribute and edge generation, thereby enabling GPT-GNN to capture the
inherent correlation between node attributes and graph structures. Conversely,
GCC (qiu2020gcc) provides a unique self-supervised GNN pretraining framework
aimed at capturing generic topological attributes across multiple graph
datasets. GCC employs contrastive learning-based cross-network subgraph
instance discrimination during pretraining. Subgraph instances from the same
structure are employed as the positive samples, while ones from differing
structures serve as negative samples. Despite the progress achieved through
the ”pretraining + fine-tuning” paradigm, it still confronts certain
challenges. Among these, a principal issue is the inconsistencies between the
objectives of pretraining and downstream tasks, as shown in Figure 1. The
reality is that a specific combination of pretraining methodology and a fine-
tuning strategy may only prove effective for a certain type of downstream
task, while rendering ineffective or potentially counterproductive outcomes
for other tasks. Moreover, fine-tuning on downstream tasks typically requires
comprehensive adjustment of the pretrained model’s weights, culminating in
significant computational overhead. Additionally, while contrastive based
approaches have thrived, the potential inherent to generative-based approaches
remains substantially underexplored

Inspired by the success of Prompt-based solutions (liu2023pre; wei2022chain)
in the Natural Language Processing (NLP) domain, the ”pretraining + prompt”
paradigm (sun2022gppt; wu2023survey; prog2023; liu2023graphprompt) has been
proposed to better align the pretrained model’s objectives with downstream
tasks. This paradigm typically employs prompt techniques to harmonize the
convention between the downstream tasks and pretrained objectives, emphasizing
a unified structure. The main innovation of GPPT (sun2022gppt) lies in
proposing a new graph neural network generalization technique called Graph
Pretraining and Prompt Tuning (GPPT). This method designs a straightforward
yet universal pretraining task and employs a prompt function to transform the
downstream task of node classification into a task akin to edge prediction.
All-in-One (ProG) (prog2023) mainly focuses on addressing potential
incompatibility issues between pretrained graph models and a variety of graph
tasks. To do so, ProG incorporates a multi-task prompting method, unifying
tasks such as node classification and link prediction under the umbrella of
graph classification tasks. These efforts have greatly contributed to bridging
the gap between pretraining and downstream tasks, notably decreasing the
computational overheads during fine-tuning. Nevertheless, this approach comes
with its own set of challenges: while it manages to align one or two tasks
such as node classification and link prediction, the unification of more
diverse tasks calls for meticulously crafted prompts. This precision demands a
complex training process and raises questions regarding the scalability of
these methods to accommodate the versatility of tasks seen in the graph
domain. It is, therefore, necessary to explore more efficient and holistic
strategies to realize the full potential of graph learning.

To mitigate the above-mentioned issues, we introduce a generative-based
framework to unify graph tasks. Our aim is to devise a universal framework
that can handle tasks across different levels in the graph domain, including
but not limited to node classification, link prediction, and graph
classification, and yield satisfactory performance in a plethora of downstream
tasks. Specifically, we first unify various graph tasks to the graph-level
tasks by inducing subgraphs. For various graph tasks, both the input and
output of the model conform to (sub)graph, ensuring task consistency. We adopt
a generative approach for pretraining, capturing the inherent features of
graph data and learning effective graph representations. The use of Generative
Adversarial Networks (GANs) further refines the learned representations,
enhancing the generative quality of our method at the semantic level. Both
during pretraining and downstream inference, the model operates on a graph
level. By maintaining operations on the same task level for both pretraining
and downstream tasks, we can extract and utilize knowledge from the pretrained
model to the fullest extent, augmenting model performance across a spectrum of
tasks. In summary, this paper makes three key contributions as follows:

  * •

To the best of our knowledge, this is the first attempt to unify various graph
tasks using a generative approach. This technique enables the utilization of
graph-level representations throughout both pretraining and downstream tasks,
ensuring effective alignment between the two phases.

  * •

We propose a generative-based framework for unifying graph tasks. This
framework is designed to effectively capture the inherent features of graph
structures and node attributes, thereby improving the quality of graph
representations. In addition, the use of Generative Adversarial Networks
(GANs) further enhances the learned representations, yielding more robust and
discriminative results in various graph tasks.

  * •

The effectiveness of our proposed framework is thoroughly substantiated
through extensive experiments.

##  2\. RELATED WORKS

#### Generative Graph Learning

Generative learning for graphs seeks to produce novel and similar graphs by
learning the distributions of a given set of observed graphs. VGAE (VGAE), by
incorporating variational autoencoders, signifies a groundbreaking
contribution, as it generates efficient representations that enhance
performance in link prediction tasks. The GAE (VGAE) can be regarded as a
deterministic counterpart to the probabilistic VGAE model. Subsequently, VGAE-
based models have flourished in the field of graph representation learning.
ARGA/ARVGA (ARGA) incorporates regularization into the GAE/VGAE framework,
effectively constraining the latent space to conform to the prior distribution
observed in the real data. To address the limitations of the Gaussian
distribution assumption, which restrict the flexibility of variational
inference, SIG-VAE (SIGVE) implements semi-implicit variational inference
(SIVI) in the VGAE framework to provide a more flexible posterior
distribution. The aforementioned models primarily focus on structural
reconstruction to enhance link prediction performance. However, graphs
encompass a wide range of information, including nodes and their
relationships, along with a diverse array of tasks such as node classification
and graph classification.

In the most recent years, inspired by the booming of masked auto encoder (MAE)
in CV, GraphMAE (GraphMAE) has achieved state-of-the-art (SOTA) performance by
employing masking strategies on graph structure and node attributes.
Nonetheless, GraphMAE are prone to being affected by feature perturbations. To
tackle this challenge, GraphMAE2 (graphmae2) which introduces multi-view
random re-mask strategies and latent representation target in decoding stage
was proposed to help enhance the performance. SeeGera (SeeGera) extends the
capabilities of SIG-VAE (SIGVE) by incorporating a masking mechanism that
effectively captures rich semantic information from node features. This
enhanced approach is concurrently applied to both link prediction and node
classification tasks. HGMAE (HGMAE) further extends MAE to heterogeneous
graphs by adopting the adaptive attribute masking with dynamic mask rate.
MaskGAE (MaskMAE) proposes a path-wise masking strategy and the theoretical
understanding of masking. S2GAE (S2GAE), an MAE variant, incorporates
direction-aware graph masking strategies and a tailored cross-correlation
decoder, which collaboratively enhances the generalization capability of this
approach, thereby leading to exceptional performance across a range of tasks.

Generative adversarial networks (GANs) (gan) have demonstrated exceptional
performance in a multitude of generative tasks, generating high-quality data,
thereby fostering the advancement of graph self-supervision. GraphGAN
(GraphGAN) integrates generative and discriminative perspectives by initially
introducing the GAN concept in graph SSL. Both models can benefit from
adversarial training: the generator is optimized to learn the connectivity
distribution of direct neighboring vertices of the target vertex, whereas the
discriminator is encouraged to effectively differentiate between ground truth
and generated samples. Different from GraphGAN (GraphGAN), AGE (AGE) achieved
better performance by generates fake neighbors directly from a continuous
distribution and considering the asymmetric (directional) semantics
information. NetGAN (NetGAN) employs the generator to generate a sequence and
the discriminator to determine the authenticity probability of the random walk
sequence, while GraphGAN (GraphGAN) concentrates on the connectivity of
neighboring nodes. Some works (ARGA) (ANE) also concentrates on integrating an
adversarial training scheme to explicitly enforce regularization, ensuring
that the latent space adheres to the prior distribution and generating
robustness against noise graph representations. GCGAN (GCGAN) integrates GAN
(gan) and GCN for adversarial training to predict traffic condition.

#### Unified Graph Learning

It is worth clarifying that, given the multi-level nature (from node-level and
edge-level to graph-level) and discriminative tasks (such as classification
and prediction) of graph domain tasks, developing a ”universal graph model”
suitable for all graph tasks is an exceedingly challenging undertaking.
Motivated by the applications of large-scale pre-trained language models
(LLMs) in the NLP domain, which typically involve pre-training a language
model using masked prediction and subsequently transferring it to various
applications such as question answering and sentiment classification, several
studies (graphprompt) (generalizedgraphprompt) (allinone) have focused on the
”pre-training and fine-tuning” and ”pre-training and prompting” paradigms
within the graph domain addressing the multi-task setting with different-level
tasks. One recent study, termed MPGN (allinone), presents a novel multi-task
prompting method that initially reformulates and unifies multi-level graph
tasks, effectively bridging the gap between the pre-trained model and
downstream tasks.

##  3\. PRELIMINARIES

###  3.1. Notations

We represent a graph using the notation
$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X})$, where $\mathcal{V}$ stands
for the set of nodes contained within the graph,
$\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ represents the set of edges
within the graph that connect the nodes, $\mathbf{}{X}\in\mathbb{R}^{N\times
d_{f}}$ refers to the node feature matrix. $N$ means the number of nodes and
$d_{f}$ denotes the dimensionality of the node features. Additionally, we
denote the adjacency matrix of the graph $\mathcal{G}$ as
$\mathbf{A}\in\\{0,1\\}^{N\times N}$.

###  3.2. Background

#### Graph Neural Network (GNN)

GNNs have shown strong ability in graph representation and tacking graph tasks
(cai2005mining; jin2021application; shi2016survey). There are many
representative GNNs, such as GAT (gat), GCN (gcn), GIN (gin) and GTN (gtn).
GNNs

In this paper, we use GNNs for graph representation. Specifically, for a
single node $v\in\mathcal{V}$ on the graph, it aggregates the representation
of its neighborhood nodes as:

(1) $e_{v}^{l}=AGG(e_{v}^{l-1},{e_{u}^{l-1}\mid u\in{\mathcal{N}_{v}}}),$

where $N_{v}$ is the neighbors of $v$ and $e_{v}$ denotes the representation
of $v$ at the $l$-th layer. For a subgraph $sg$, we use simple readout
modules, i.e. , max or mean pooling on $sg$ to represent the representation of
the subgraph as:

(2) $E_{sg}=readout(\left[e_{i},e_{i+1},...,e_{i+n}\right]).$

#### Masked Autoencoder (MAE)

The Graph Masked Autoencoder (GAE) is a self-supervised approach that
typically consists of an encoder and a decoder. Given an encoder $f_{E}$, a
decoder $f_{D}$, and an input graph $\mathcal{G}$, we randomly sample a fixed
proportion of node indices $M$. $\mathcal{G}$ can be divided into a masked
graph $\mathcal{G}_{m}$ and an unmasked graph $\mathcal{G}_{u}$, with the
corresponding feature matrix divided into $X_{m}=\\{x^{k}\in M\\}$ and
$X_{u}=\\{x^{k}\notin M\\}$. During training, the encoder $f_{E}$ first maps
the corrupted graph to a hidden representation
$\mathcal{H}\in\mathbb{R}^{N\times d_{h}}$. Subsequently, the decoder $f_{D}$
attempts to reconstruct the feature or adjacency matrix using the hidden
representation $\mathcal{H}$. Conventionally, the mean-squared error (MSE) is
adopted as the reconstruction criterion.

#### Generative Adversarial Networks (GAN)

Generative Adversarial Networks (GANs) (gan_survey) are robust networks that
are highly effective at generating high-quality data. GANs usually consist of
a generator network ($G$) and a discriminator network ($D$). Both networks
strive to optimize themselves completely to achieve their respective
objectives, as they each have their own objective functions. The discriminator
aims to estimate the probability of a given sample coming from the real
dataset, while the generator is trained to output synthetic samples that
closely resemble real data. The generator $G$ endeavors to trick the
discriminator, while the discriminator model $D$ works hard not to be
deceived. Altogether, $G$ and $D$ engage in a minimax game as follows:

(3) $\displaystyle\min_{G}\max_{D}L(D,G)$ $\displaystyle=\mathbb{E}_{x\sim
p_{r}(x)}[\log D(x)]$ $\displaystyle+\mathbb{E}_{x\sim
p_{g}(x)}[\log(1-D(x))],$

where $p_{r}(x)$ is the distribution of real data over real samples $x$, and
$p_{g}(x)$ is the generator’s distribution over $x$.

##  4\. METHODOLOGY

Figure 2. The Overview of proposed framework.

In this section, we formally introduce our proposed method. For the first
time, we present a unified model which addresses nodes, edges, and graph-level
tasks from a generative perspective. An overview of this framework is
illustrated in Figure 2. It presents a simple yet effective structure,
primarily composed of two modules: a generator modeled by a Masked Autoencoder
(MAE) and a discriminator. Notably, unlike the conventional settings
(GraphMAE; VGAE), our framework takes a batch of subgraphs as input. We
commence this section with a broad overview of the proposed framework,
followed by a discussion on the unification of various tasks within the graph
domain, thereby setting the stage for presenting our unified paradigm.
Subsequently, we delve into the process of guiding this unified framework from
a generative perspective. Lastly, we discuss the contributing factors
underlying the success of adopting this proposed framework

###  4.1. Subgraph to Bridge the GAP of Multi-task Learning

#### Problems with Multi-task learning

Consistent with the practises in NLP (bert; pretrained_finetune), the
conventional multi-task graph learning follows the paradigm of ”pretraining +
finetuning”. Through the design of appropriate pretraining task, high-quality
graph representation can be acquired for multi-granularity downstream graph
tasks, including node-level, edge-level and graph-level task. At the fine-
tuning stage, the acquired graph representation will be used to tune the model
parameters based on the few supervised data.

Due to the scarcity of supervised data on graphs, many self-supervised pre-
training tasks have been designed to mine the knowledge embedded within graphs
to guide downstream tasks (learn_to_pretrain; gcc; gca), the learned graph
representation will be used for downstream fine-tunin. However, on the one
hand, these works use a one-for-all graph representation, performing various
graph tasks separately on top of this representation. This approach fails to
adequately consider the intrinsic characteristics of different graph tasks,
making it difficult to simultaneously consider the information at various
levels (nodes, edges, subgraphs) in the pre-training phase, which leads to a
gap between the pre-trained model and downstream tasks. For instance, given a
pre-training task based on node-level contrastive learning, the corresponding
embeddings for downstream edge-level tasks will inevitably result in a
decrease in edge-level task performance (SeeGera). Furthermore, as pointed out
by W2PGNN (when_to_pretrain), there is a gap between pre-training and fine-
tuning in terms of pre-trained models and downstream fine-tuning strategies.
It is often challenging to find the combination of pre-trained model and
strategy that is most suitable for downstream tasks among multiple pre-trained
models and fine-tuning strategies, presenting new difficulties to the
prevalent pretraining-finetuning paradigm.

Therefore, it is necessary to align the pre-training models and fine-tuning
strategies, which involves two aspects. First, alignment in the form of pre-
training tasks: for various graph tasks, it is crucial to maintain uniformity
in pretraining stage, rather than designing different pre-training tasks for
each graph task. Second, alignment of inputs during the pre-training phase and
the fine-tuning phase: the inputs for pre-trained models and fine-tuning
should maintain formal uniformity to minimize the impact of performance caused
by formal differences. The key to alignment lies in finding a meta-structure
that includes the minimal structure involved in all the graph tasks being
studied.

#### Subgraph as Meta-structure for Multi-task Graph Learning

(a) A subgraph instance.

(b) Meta structures.

Figure 3. An illustration of meta-structures. The subgraph represents the
minimal structure that contains node, edge and graph level knowledge.

To facilitate multi-task graph pretraining as well as mitigate the discrepancy
between the pretrain model and the fine-tune phases, a meta structure that
contains the comprehensive graph knowledge need to be developed. We first
presents the defination of the the meta-structure.

###### Definition 0.

A meta-structure for multi-task ”pre-training and fine-tuning” on graphs
should be the minimal union of learning objectives (nodes, edges, and graphs)
for each sub-task, and maintain consistency during both the pre-training and
fine-tuning stages.

Note that in Definition 4.1, we emphasis that the meta-structure remains
consistency at arbitrary stages, i.e., the model accepts the identical
structure for pretraining for different tasks, as well as accept the congruent
structure for pre-training and fine-tuning stages. Towards the chosen of
suitable meta-structure, Figure 3(a) presents an graph instance that contains
graph characteristic at different level. To be more specific, we list the
node, edge and subgraph instance in Firure 3(b). Given node $u$, $v$ and the
edge $e_{uv}$ between them, it is intuitive that the subgraph includes both
the node and edge. Based on this observation, we select the subgraph as the
meta-structure across the node-level, edge-level and graph-level tasks. For
node-level tasks, e.g., node classification, we randomly select $N_{s}$ nodes
from the graph’s node set $\mathcal{V}$ as the seed set $\mathcal{S}$,
$N_{s}<N$. For each node in $\mathcal{S}$, we construct a subgraph centered at
the current node. In this way we obtain $|\mathcal{S}|$ subgraphs, then we can
perform pre-training based on these subgraphs. At the fine-tuning phase, we
construct the subgraph for each target node in the test set the same way as in
the pretraining phase, we simply use the target node’s label as the label for
the subgraph. On these basis, the node classification transformed into a
graph-level problem, the discrepancy between the pretraining and fine-tuning
phases is therefore mitigated. Analogously, for for edge-level tasks, we start
with edges instead. During the pre-training phase, we randomly select a
portion of edges from the graph’s edge set $\mathcal{E}$ as the seed edge set
$\mathcal{L}$. For each edge in $\mathcal{L}$, we construct a subgraph
centered around that edge, eventually obtaining $\mathcal{L}$ edge-centered
subgraphs, based on which we can perform pre-training. Noted that the number
of edges is far greater than the number of nodes on the graph. To ensure that
the selected node subset $\mathcal{S}$ and edge subset $\mathcal{L}$ cover as
much of the graph nodes and edges as possible, we need to construct
$\mathcal{S}$ and $\mathcal{L}$ differently. For node-level tasks, we can
construct the subset $\mathcal{S}$ dynamically in each training round, while
for edge-level tasks, we can store the $\mathcal{L}$ when it is first
constructed to avoid the overhead of subsequent subgraph construction.

Using subgraphs as the meta-structure for various graph tasks can not only
address the aforementioned pain points but also provide local structural
information of the subgraph for node-level and edge-level tasks. However, how
to construct subgraphs is also a topic that needs to be discussed.

#### Subgraph Construction for node and edge-level Tasks

There are a lot of work (frasca2022understanding; alsentzer2020subgraph;
huang2020graph) exploring how to construct subgraphs, in this work we choose
the simple and efficient random walk based approach.

(a) RWR to construct node subgraph.

(b) RWR to construct edge subgraph.

Figure 4. An illustration of RWR to construct subgraph for node and edge-level
subgraphs.

We adopt different approaches to reformulate node-level and edge-level tasks
into graph-level tasks. On one hand, for node-level subgraph construction, as
illustrated in Figure 4(a), we choose a node $v_{0}\in\mathcal{V}$ as the
starting point and construct its subgraph by gathering its $k$-hop neighboring
nodes. However, to mitigate the impacts of inconsistent neighborhood node
distributions – i.e., the number of $k$-hop neighbors for two distinct nodes
could vary significantly – we employ a random walk with restart (RWR) strategy
(random_walk_survey) to construct subgraphs for each node. The random walk
iteratively traverses the current node’s neighbors or returns to the starting
node with a probability $p$. We continue the RWR process until a fixed number
of nodes are collected, denoted as $RWR(v)$. Subsequently, we extract the node
set $RWR(v)$ from $\mathcal{G}$, ensuring that each subgraph contains an
equivalent number of nodes.

On the other hand, we start from edge to construct edge-level subgraph.
Actually, the edge-level subgraph construction can be regarded as a repetitive
node-level subgraph construction. As shown in Figure 4(b), the edge
$e_{v_{0}v_{4}}$ connects $v_{0}$ and $v_{4}$ can be denoted as positive edge.
We start from $v_{0}$ and $v_{4}$ separately to perform RWR until the
predefined size of subgraph reached.

###  4.2. Generative Self-Supervised Learning With Subgraphs

The proposed framework, as depicted in Figure 2, principally consists of two
modules: a generator and a discriminator. The generator is tasked with mapping
input graphs to a latent space and subsequently reconstructing them.
Concurrently, the discriminator serves to distinguish whether a given input
graph is a generated or an original one, thereby ensuring the framework’s
effectiveness in diverse graph-based tasks.

#### Generator

: The generator is designed to learn the latent distribution of the graphs. We
employ a Masked Autoencoder (MAE) as the backbone of the generator. We denote
the encoder of MAE as $f_{E}$ and the decoder as $f_{D}$. Given an input graph
$\mathcal{G}$, we initially apply a random mask to the edges and node features
of an input graph, resulting in a modified graph represented by
$\tilde{\mathcal{G}}(\tilde{\mathcal{V}},\tilde{\mathcal{E}},\tilde{\mathbf{X}})$.
The masked, or corrupted, graph is then inputted into a MAE. The encoder part
of the MAE maps the corrupted graph to the latent space, from which the
decoder component further rebuilds the original graph.

(4)
$\mathbf{H}=f_{E}(\tilde{\mathbf{A}},\tilde{\mathbf{X}}),\mathbf{\bar{\mathcal{G}}}=f_{D}(\tilde{\mathbf{A}},\mathbf{H}),$

where $\mathbf{H}$ denotes the hidden layer representation outputted by the
encoder, while $\mathbf{\bar{\mathcal{G}}}$ refers to the graph reconstructed
by the decoder. The objective of MAE is to reconstruct the input graph.
Following GraphMAE, we adopt scaled cosine error (SCE) as the reconstruction
loss.

(5)
$\mathcal{L}_{\mathrm{sce}}=\frac{1}{|{\mathcal{V}}|}\sum_{v_{i}\in{\mathcal{V}}}\left(1-\frac{x_{i}^{T}\bar{x}_{i}}{\left\|x_{i}\right\|\cdot\left\|\bar{x}_{i}\right\|}\right)^{\gamma},\gamma\geq
1,$

where $x_{i}\in\mathbf{X}$ stands for the node features of the original graph
$\mathcal{G}$, $\bar{x}_{i}\in\mathbf{\bar{X}}$ denotes the node features of
the reconstructed graph $\mathcal{\bar{G}}$ generated by the decoder. The
scaling factor $\gamma$ is a hyperparameter. When $\gamma>1$, we can weight
down the easy samples in training with a $\gamma$ power of the cosine error.
For high-confidence predictions, the corresponding cosine error is usually
less than 1 and decays faster with a scaling factor $\gamma>1$.

#### Discriminator

: The discriminator operates as a binary classifier, where inputs include both
the reconstructed graphs from the MAE and the original ones. The objective of
the discriminator is to distinguish between the reconstructed and the original
graphs, effectively refining the generative quality of the MAE on a semantic
level. This concurrently catalyzes a feedback loop that betters the
performance of the generator, leading to more accurate and high-quality graph
reconstruction. More specifically, we assign a label of 1 (representing a
positive sample) to the original graph $\mathcal{G}$ and a label of 0
(indicating a negative sample) to the graph $\mathcal{\bar{G}}$ reconstructed
by MAE. The discriminator is composed of a Graph Neural Network (GNN) layer,
graph pooling, and a linear classification layer. The loss function for the
discriminator is calculated using the following formula:

(6)
$\displaystyle\mathcal{L}_{dis}={-\textstyle\sum_{\mathcal{G}\sim\mathcal{S}}\log{\mathcal{D}(\mathcal{G})}}{-\textstyle\sum_{\mathcal{\bar{G}}\sim\mathcal{\bar{S}}}\log(1-\mathcal{D}(\mathcal{\bar{G}}))},$

where $\mathcal{D}$ denotes the discriminator model, $\mathcal{S}$ denotes the
set of original graph samples, and $\mathcal{\bar{S}}$ denotes the set of
reconstructed graph samples.

###  4.3. Training and Inference

The process of training and inference are summarized in Algorithm 1.

Algorithm 1 Procedure for training and inference algorithms

1:The set of graph samples
$\mathcal{S}=\\{\mathcal{G}_{1},\mathcal{G}_{2},...\\}$ and corresponding
labels $\mathcal{Y}$, the number of iterations T

2:$\triangleright$ Training Process

3:for iterator=1,2,…,T do

4: $\tilde{\mathcal{S}}$=MASK($\mathcal{S}$)

5: The set of reconstructed graphs $\bar{\mathcal{S}}$ is obtained through
Equation 4

6: Computing $\mathcal{L}_{dis}$ through Equation 6

7: Update the Discriminator

8: Computing $\mathcal{L}_{sce}$ through Equation 5

9:
$\mathcal{L}_{gen}={\textstyle\sum_{\mathcal{G}\sim\mathcal{S}}\mathcal{L}_{sce}(\mathcal{G},\bar{\mathcal{G}})}+\alpha\mathcal{L}_{dis}$

10: Update the Generator

11:end for

12:return $f_{E},f_{D}$ and $\mathcal{D}$

13:

14:$\triangleright$ Inference Process

15:for $\mathcal{G}\sim\mathcal{S}$ do

16: $\mathbf{H}=f_{E}(\mathcal{G})$

17: $\mathbf{Z}=$ READOUT($\mathbf{H}$)

18: $\hat{y}=Classifier(\mathbf{Z})$

19:end for

20:return $\hat{\mathcal{Y}}=\\{\hat{y}_{1},\hat{y}_{2},...\\}$

#### Training

: For node and edge-level tasks, we first conduct node and edge sampling, and
subsequently construct induced subgraphs based on the sampled nodes and edges.
These subgraphs are then fed into the model for training. More specifically,
we commence by randomly masking the nodes and edges of the input graph
followed by feeding the enhanced graph into the MAE. The graph is
reconstructed by the MAE with node features forming the target for generative
pretraining. Subsequently, the reconstructed graph from the MAE alongside the
original graph are inputted into the discriminator, which determines whether
the input graph is generated by the MAE. In addition, $\alpha$ in the 8th line
of Algorithm 1 is a hyperparameter to control the contribution of the
discriminator.

#### Inference

: During the inference process, the model accepts inputs in the form of either
graphs or subgraphs, similar to the training phase. However, the
discriminator’s involvement is not required for this stage. Instead, the input
graph is processed by the encoder of MAE, yielding node embeddings. These
embeddings are then aggregated using a non-parametric graph pooling (readout)
function to obtain a graph-level representation. Subsequently, this graph-
level representation can be utilized for downstream tasks, effectively
providing a versatile solution for a variety of graph-based challenges.

###  4.4. Why it Works ?

experiment

#### Unification of Graph Tasks

: One of the key reasons why our proposed method is effective lies in the
unification of various types of graph tasks into graph-level tasks, which
streamlines the process and fosters task consistency. For node classification
tasks, we base the construction of induced subgraphs on nodes. Conducting
graph classification on these induced subgraphs is equivalent to performing
node classification. This approach encapsulates all relevant node information
within a dedicated subgraph that is treated as a distinguishable learning
unit, thereby reframing the originally node-dependent task into a graph-level
task. Similarly, for link prediction tasks, we create induced subgraphs based
on edges. When conducting graph classification on these induced subgraphs, it
is functionally akin to classifying on individual edges. This process
represents each edge association as a distinct subgraph, effectively
transforming edge-dependent tasks into graph-level tasks. By transforming
various types of graph tasks into uniformly graph-level tasks, we maintain
task consistency, which is crucial for a machine learning model’s ability to
learn effectively.

#### Alignment between Pretraining and Downstream Tasks

: The success of the proposed method can be attributed to the alignment of the
pretraining and downstream tasks, which effectively reduces the gap between
them, thereby minimizing any negative transfer. By unifying all graph-level
tasks to a single tier, our model operates universally at this level, both
during pretraining and inference of the downstream tasks. This level of
strategic alignment ensures that the intricacies of each task are adequately
captured during pretraining and effectively applied during downstream tasks.
Moreover, our method’s flexibility extends to accommodating downstream tasks
of various complexities. By operating on similar task dimensions between the
pretraining methods and downstream tasks, our model maximizes the utilization
of knowledge learned from the pretrained model. This process significantly
diminishes problems arising from negative transfer, further enhancing the
model’s performance across an array of tasks.

In summary, through task unification at the graph level and alignment of
pretraining with downstream tasks, our model provides an effective way to
apply the learned knowledge, reducing discrepancies, minimizing negative
transfer, and maximizing the effectiveness of the model across numerous tasks.

##  5\. EXPERIMENTS

In this section, we conduct thorough and comprehensive experiments on the node
level, edge level, and graph level to address the following research
questions:

$\mathcal{RQ}1$: Can our method achieve consistent and excellent performance
across various levels of downstream graph tasks?

$\mathcal{RQ}2$: How do the main components of the model impact its overall
performance?

$\mathcal{RQ}3$: How sensitive is our model to parameter changes?

###  5.1. Datasets

In order to thoroughly assess the performance of our graph representation
learning approach on node classification and link prediction tasks, we conduct
experiments on seven publicly available datasets, namely Cora, Citeseer,
Pubmed (yang2016revisiting), Coauthor CS, Coauthor Physics, Amazon Computer,
and Amazon Photo. The first five datasets consist of paper citation data,
while the last two originate from the e-commerce domain. For graph
classification tasks, we perform experiments on an additional set of seven
public datasets, including MUTAG, IMDB-B, IMDB-M, PROTEINS, COLLAB, REDDITB,
and NCI1 (10.1145/2783258.2783417). Of these, three datasets, PROTEINS, MUTAG,
and NCI1, are bioinformatic datasets, and the remaining datasets are social
network datasets. Each dataset within this collection comprises a set of
graphs with corresponding labels. More detailed information about the datasets
can be found in Appendix B.

###  5.2. Baselines

For the node classification task, the baselines we compare our method to can
be broadly categorized into two groups: (1) Supervised methods, which include
GCN and GAT. (2) Self-supervised methods, which encompass contrastive learning
approaches such as DGI (veličković2018deep), MVGRL (hassani2020contrastive),
GRACE (zhu2020deep), GCA (zhu2021graph), and CCA-SSG (zhang2021canonical), as
well as VGAE (kipf2016semi), GraphMAE (hou2022graphmae), and SeeGera
(li2023seegera).

For link prediction tasks, we compare our method to four contrastive learning
baselines, namely SIG-VAE (hasanzadeh2019semi), CAN (meng2019co), GATE
(salehi2020graph), and GraphMAE, along with five generative methods including
DGI, MVGRL, GRACE, GCA, and CCA-SSG.

For graph classification tasks, the baselines we compare against can be
divided into three main categories: (1) Graph kernel methods, which comprise
of WeisfeilerLehman sub-tree kernel (WL) (shervashidze2011weisfeiler) and Deep
Graph Kernel (DGK) (yanardag2015deep). (2) unsupervised methods including
graph2vec and Infograph (sun2019infograph). (3) Self-supervised methods, which
include GraphCL (you2020graph), JOAO (you2021graph), GCC (qiu2020gcc), MVGRL,
InfoGCL (xu2021infogcl), GraphMAE, S2GAE (tan2023s2gae).

###  5.3. Experimental Settings

For all baselines, if official code implementations are available, we
reproduce their experimental results while maintaining consistent experimental
settings. If the original papers provide results under consistent settings, we
directly incorporate their provided results. For our proposed method, we
leverage GCN or GAT as the backbone for both the generator and discriminator.
In the node classification task, we adopt the Accuracy (Acc) as the evaluation
metric, with dataset settings and splits referencing the GraphMAE study. For
the link prediction task, we utilize the Area Under the Receiver Operating
Characteristic curve (AUC) and Average Precision (AP) as evaluation metrics,
with dataset settings and splits referencing the SeeGera study. Regarding the
graph classification task, we use the Acc as the evaluation metric, with
dataset settings and splits based on the GraphMAE study. All experiments are
conducted on a single V100 32GB GPU. We repeat each experiment five times and
report the mean and standard deviation of the results to ensure reliability
and consistency in the performance evaluation of our proposed method compared
with the baselines. We provide additional details about the experimental setup
and parameter configuration in Appendix C.

###  5.4. Main Results ($\mathcal{RQ}1$)

#### Node Classification

Table 1. Experimental results of node classification with evaluation metrics of accuracy. | Methods | Available Data | Cora | Citeseer | Pubmed | Photo | Computer | CS | Physics
---|---|---|---|---|---|---|---|---|---
| GCN | X,A,Y | 80.84±0.44 | 70.46±0.85 | 79.02±0.26 | 90.79±2.47 | 85.34±1.64 | 92.1±0.16 | 95.41±0.11
| GAT | X,A,Y | 83.0±0.7 | 72.5±0.7 | 79.0±0.3 | 93.49±0.16 | 89.79±0.76 | 91.48±0.17 | 95.25±0.14
Supervised | APPNP | X,A,Y | 83.0±0.7 | 72.5±0.7 | 79.0±0.3 | 93.42±0.35 | 87.45±0.66 | 92.5±0.14 | 95.59±0.09
| GRACE | X,A | 77.60±0.28 | 67.24±0.93 | 78.4±1.33 | 92.45±0.34 | 88.11±0.45 | 67.57±12.91 | 85.33±6.76
| GCA | X,A | - | - | - | 92.49±0.09 | 87.85±0.31 | 93.10±0.01 | 95.68±0.05
| DGI | X,A | 82.3±0.6 | 71.8±0.7 | 76.8±0.6 | 91.61±0.22 | 83.95±0.47 | 92.15±0.63 | 94.51±0.52
| MVGRL | X,A | 83.5±0.4 | 73.3±0.5 | 80.1±0.7 | 91.74±0.07 | 87.52±0.11 | 92.11±0.12 | 95.33±0.03
| VGAE | X,A | 71.50±0.4 | 65.80±0.40 | 72.10±0.50 | 92.20±0.11 | 86.37±0.21 | 92.11±0.09 | 75.35±0.14
| CCA-SSG | X,A | 84.0±0.4 | 73.1±0.3 | 81.0±0.4 | 93.14±0.14 | 88.74±0.28 | 93.31±0.22 | 95.38±0.06
| GraphMAE | X,A | 84.2±0.40 | 73.4±0.4 | 81.1±0.4 | 92.98±0.35 | 88.34±0.27 | 93.08±0.17 | 95.30±0.12
SSL-based | SeeGera | X,A | 84.3±0.40 | 73.0±0.8 | 80.4±0.4 | 92.81±0.45 | 88.39±0.26 | 93.84±0.11 | 95.39±0.08
Ours |  | X,A | 83.95±0.40 | 73.60±0.15 | 81.50±0.60 | 93.60±0.22 | 89.87±0.11 | 93.59±0.10 | -

We conducted a comprehensive comparative experiment for node classification,
where our proposed method was benchmarked against ten other baselines on seven
different datasets. These baselines encompass a variety of approaches
including supervised, contrastive, and generative methods. As depicted in
Table 1, self-supervised methods delivered results comparable to supervised
techniques, highlighting the tremendous potential of self-supervised graph
representation learning. Furthermore, our method performed exceptionally well
across all datasets, either surpassing or equalling the effectiveness of all
tested baselines in node classification tasks. Passed through the acid test of
our extensive experimentation, our generative-based unified framework
demonstrated an outstanding ability to learn highly effective and
discriminating node representations. The success of our framework in the node
classification tasks across all datasets justifies the strength of our method
in adapting to diverse graph tasks, underscoring its promise in fostering
further advancements in graph representation learning. Ultimately, the
empirical results generated during our studies stand as a robust testament to
the potential of our novel framework in leveraging the interpretive power of
self-supervised learning in graph representation.

#### Link Prediction

Table 2. Experimental results for link prediction Metrics | Method | Cora | Citeseer | Pubmed | Photo | Computer | CS | Physics
---|---|---|---|---|---|---|---|---
AUC | DGI | 93.88±1.00 | 95.98±0.72 | 96.30±0.20 | 80.95±0.39 | 81.27±0.51 | 93.81±0.20 | 93.51±0.22
MVGRL | 93.33±0.68 | 88.66±5.27 | 95.89±0.22 | 69.58±2.04 | 92.37±0.78 | 91.45±0.67 | OOM
GRACE | 82.67±0.27 | 87.74±0.96 | 94.09±0.92 | 81.72±0.31 | 82.94±0.20 | 85.26±2.07 | 83.48±0.96
GCA | 81.46±4.86 | 84.81±1.25 | 94.20±0.59 | 70.02±9.66 | 89.92±0.91 | 84.35±1.13 | 85.24±5.41
CCA-SSG | 93.88±0.95 | 94.69±0.95 | 96.63±0.15 | 73.98±1.31 | 75.91±1.50 | 96.80±0.16 | 96.74±0.05
CAN | 93.67±0.62 | 94.56±0.68 | - | 97.00±0.28 | 96.03±0.37 | - | -
SIG-VAE | 94.10±0.68 | 92.88±0.74 | 85.89±0.54 | 94.98±0.86 | 91.14±1.10 | 95.26±0.36 | 98.76±0.23
GraphMAE | 90.70±0.01 | 70.55±0.05 | 69.12±0.01 | 77.42±0.02 | 75.14±0.02 | 91.47±0.01 | 87.61±0.02
SeeGERA | 95.50±0.71 | 97.04±0.47 | 97.87±0.20 | 98.64±0.05 | 97.70±0.19 | 98.42±0.13 | 99.03±0.05
Ours | 97.93±0.10 | 97.72±0.39 | 98.76±0.05 | 97.70±0.20 | 97.35±0.03 | 99.25±0.02 | 99.03±0.01
AP | DGI | 93.60±1.14 | 96.18±0.68 | 95.65±0.26 | 81.01±0.47 | 82.05±0.50 | 92.79±0.31 | 92.10±0.29
MVGRL | 92.95±0.82 | 89.37±4.55 | 95.53±0.30 | 63.43±2.02 | 91.73±0.40 | 89.14±0.93 | OOM
GRACE | 82.36 0.24 | 86.92±1.11 | 93.26±1.20 | 81.18 ± 0.37 | 83.12±0.23 | 83.90±2.20 | 82.20±1.06
GCA | 80.87±4.11 | 81.93±1.76 | 93.31±0.75 | 65.17±10.11 | 89.50±0.64 | 83.24±1.16 | 82.80±4.46
CCA-SSG | 93.74±1.15 | 95.06 ± 0.91 | 95.97±0.23 | 67.99±1.60 | 69.47±1.94 | 96.40±0.30 | 96.26±0.10
CAN | 94.49±0.60 | 95.49±0.61 | - | 96.68±0.30 | 95.96±0.38 | - | -
SIG-VAE | 94.79±0.71 | 94.21±0.53 | 85.02±0.49 | 94.53±0.93 | 91.23±1.04 | 94.93±0.37 | 98.85±0.12
GraphMAE | 89.52±0.01 | 74.50±0.04 | 87.92±0.01 | 77.18 0.02 | 75.80±0.01 | 83.58±0.01 | 86.44±0.03
SeeGERA | 95.92±0.68 | 97.33±0.46 | 97.87±0.20 | 98.48±0.06 | 97.50±0.15 | 98.53±0.18 | 99.18±0.04
Ours | 98.03±0.10 | 97.76±0.38 | 98.86±0.04 | 97.47±0.20 | 97.25±0.04 | 99.06±0.02 | 99.01±0.01

Our proposed method was evaluated against nine baseline models across seven
different datasets. We reported results based on two evaluation metrics, Area
Under the Curve (AUC) and Average Precision (AP), as presented in Table 2. The
generative models as a whole tend to outperform contrastive methods in link
prediction tasks, with the exception of GraphMAE. This superior performance is
attributed to the fact that generative models generally utilize graph
reconstruction as their objective, which aligns well with the downstream task
of link prediction. Conversely, GraphMAE targets the reconstruction of node
features as its objective. Our framework consistently delivered superior
results across all datasets when compared to the established baselines.
Intriguingly, while our method shares the same objective as GraphMAE -
reconstructing node features - it matched or even surpassed the performance of
methods focusing on graph structure reconstruction like SeeGera. The reason
behind this success lies in the design of our approach: instead of focusing
only on individual nodes, our method learns from the local neighborhood
structure of nodes via induced subgraphs, which encapsulates superior neighbor
structure information. This holistic focus allows for an in-depth learning of
node-neighborhood interactions, which proves particularly beneficial for link
prediction tasks.

#### Graph Classification

Table 3. Experimental results for graph classification | Dataset | IMDB_B | IMDB_M | PROTEINS | COLLAB | MUTAG | REDDIT-B | NCI
---|---|---|---|---|---|---|---|---
Graph Kernels | WL | 72.30±3.44 | 46.95±0.46 | 72.92±0.56 | - | 80.72±3.00 | 68.82±0.41 | 80.31±0.46
DGK | 66.96±0.56 | 44.55±0.52 | 73.30±0.82 | - | 87.44±2.72 | 78.04±0.39 | 80.31±0.46
Unsupervised | graph2vec | 71.10±0.54 | 50.44±0.87 | 73.30±2.05 | - | 83.15±9.25 | 75.78±1.03 | 80.31±0.46
Infograph | 73.03±0.87 | 49.69±0.53 | 74.44±0.31 | 70.65±1.13 | 89.01±1.13 | 82.50±1.42 | 76.20±1.06
Contrastive | GraphCL | 71.14±0.44 | 48.58±0.67 | 74.39±0.45 | 71.36±1.15 | 86.80±1.34 | 89.53±0.84 | 77.87±0.41
JOAO | 70.21±3.08 | 49.20±0.77 | 74.55±0.41 | 69.50±0.36 | 87.35±1.02 | 85.29±1.35 | 78.07±0.47
GCC | 72.0 | 49.4 | - | 78.9 | - | 89.8 | -
MVGRL | 74.20±0.70 | 51.20±0.50 | - | - | 89.70±1.10 | 84.50±0.60 | -
InfoGCL | 75.10±0.90 | 51.40±0.80 | - | 80.00±1.30 | 91.20±1.30 | - | 80.20±0.60
Generative | GraphMAE | 75.52±0.66 | 51.63±0.52 | 75.30±0.39 | 80.32±0.46 | 88.19±1.26 | 88.01±0.19 | 80.40±0.30
S2GAE | 75.76±0.62 | 51.79±0.36 | 76.37±0.43 | 81.02±0.53 | 88.26±0.76 | 87.83±0.27 | 80.80±0.24
Ours | 76.27±0.37 | 51.96±0.53 | 75.41±0.33 | 81.92±0.01 | 89.36±0.00 | 89.02±0.02 | 80.73±0.31

We further extended our evaluation to include graph classification tasks,
where we compared our method against eleven baselines on seven different
datasets. The results of these experiments are displayed in Table 3. Our
method exhibited strong performance across all datasets, yielding either the
best or comparable results against the tested baselines. These outcomes
validate the effectiveness of our method in tackling not only node level and
link prediction tasks, but also graph-level tasks such as graph
classification. The consistently high performance of our method across
different datasets and against various baselines underscores the robustness
and adaptability of our generative-based unified framework. This demonstrates
the method’s universal applicability to efficiently capture and represent the
unique characteristics and structures of disparate graph data, providing a
compelling solution for graph classification tasks.

#### Transfer Learning

Table 4. Experimental results for transfer learning | BBBP | Tox21 | ToxCast | SIDER | ClinTox | MUV | HIV | BACE | Avg.
---|---|---|---|---|---|---|---|---|---
ContextPred | 64.3±2.8 |  75.7±0.7 | 63.9±0.6 | 60.9±0.6 | 65.9±3.8 | 75.8±1.7 | 77.3±1.0 | 79.6±1.2 | 70.4
AttrMasking | 64.3±2.8 | 76.7±0.4 |  64.2±0.5 | 61.0±0.7 | 71.8±4.1 | 74.7±1.4 | 77.21.1 | 79.3±1.6 | 71.1
Infomax | 68.8±0.8 | 75.3±0.5 | 62.7±0.4 | 58.4 ±0.8 | 69.9±3.0 | 75.3±2.5 | 76.0±0.7 | 75.9±1.6 | 70.3
GraphCL | 69.7±0.7 | 73.9±0.7 | 62.4±0.6 | 60.5±0.9 | 76.0±2.7 | 69.8±2.7 |  78.5±1.2 | 75.4±1.4 | 70.8
JOAO | 70.2±1.0 | 75.0±0.3 | 62.9±0.5 | 60.0±0.8 | 81.3±2.5 | 71.7±1.4 | 76.7±1.2 | 77.3±0.5 | 71.9
GraphLoG | 72.5±0.8 |  75.7±0.5 | 63.5±0.7 |  61.2±1.1 | 76.7±3.3 |  76.0±1.1 | 77.8±0.8 | 83.5±1.2 | 73.4
GraphMAE | 72.0±0.6 | 75.5±0.6 | 64.1±0.3 | 60.3±1.1 |  82.3±1.2 | 76.3±2.4 | 77.2±1.0 |  83.1±0.9 |  73.8
Ours |  72.4±0.5 | 74.9±0.7 | 64.4±0.8 | 64.0±0.4 | 83.9±0.2 | 75.5±2.8 | 79.8±0.6 | 82.0±0.7 | 74.6

In assessing the transferability of our approach, we conducted a series of
transfer learning experiments following the experimental setup of GraphMAE.
Our experiments aimed to evaluate whether the powerful representations learned
by our model could generalize to new, unseen tasks. Specifically, we pre-
trained our model on a corpus of two million unlabeled molecular samples from
the ZINC15 dataset. Then, we fine-tuned and tested our approach on eight graph
classification datasets, which were split in a scaffold-split manner to
simulate realistic application scenarios, aligning with common practices in
cheminformatics. The results of these experiments are presented in Table 4.
Our approach achieved the highest performance on four out of the eight
datasets. In the remaining datasets, it performed comparably to SOTA
baselines. Moreover, when evaluating the average scores across all eight
datasets, our method outperformed others, achieving the best overall result.
Such outcomes highlight the excellent and stable transfer learning
capabilities of our method, further depicting its robustness and the quality
of the graph representations it learns.

###  5.5. Ablation Study ($\mathcal{RQ}2$)

Figure 5. Ablation studies on effectiveness of main components. The characters
N, L and G following the dataset represent that this dataset is used for node
classification, link prediction and graph classification respectively.

To validate the impact of various primary modules of our proposed model on the
performance, we carry out several ablation experiments by systematically
removing certain module. In these experiments, we denote ”full” to represent a
model inclusive of all components. ”w/o dis” signify the model without the
discriminator module. ”w/o sce” is used to denote the scenario when Mean
Square Error (MSE) loss replace the Scaled Cosine Error (SCE) loss. Lastly,
“w/o mask” represent the model without the mask module, where the mask rate is
set to zero. To ensure the universality and generalizability of our
experimental results, we select two datasets each from node classification,
link prediction, and graph classification tasks to conduct the ablation
studies. The resultant findings are illustrated in Figure 5. As can be
inferred from Figure 5, all the operative modules make a positive contribution
to the overall performance of the model. Removal of any of these modules
resulted in a dip in model performance, substantiating their importance in
correct and efficient graph representation learning.

###  5.6. Parameter Analysis ($\mathcal{RQ}3$)

Figure 6. Impact of mask rate

#### Mask Rate

In the previous ablation study, we confirmed the positive contributions of the
mask module to the overall performance of our model. To further investigate
the influence of mask rate on model performance, we conduct experiments by
increasing the mask rate from 0.1 to 0.9. We select two datasets each from the
tasks of node classification, link prediction, and graph classification for
consistent analysis. The results, displayed in Figure 6, reveal that when the
mask rate is relatively small, the performance of the model tends to be
subpar. This could be attributed to the redundancy of graph data and its
homogeneous, due to which the model can reconstruct the original graph easily,
leading to mediocre performance. However, as the mask rate increases,
reconstructing the original graph becomes more challenging for the model,
thereby contributing to better graph representation learning. On the contrary,
a further increase in the mask rate beyond an optimal point, results in a
state where the model struggles to reconstruct the original graph and hence,
performance deteriorates. More parametric analysis experiments can be found in
Appendix D.

##  6\. CONCLUSION

In the rapidly advancing domain of self-supervised graph representation
learning, the diversity of graph tasks and datasets presents significant
challenges. Traditional learning methods are often tailored for specific tasks
like node classification, and while the ”pretraining + fine-tuning” strategy
has facilitated the development of generic representations, it struggles with
computational inefficiency and an alignment gap with downstream tasks. The
”pretraining + prompt” method offers improvements but is limited in scope and
complexity. In response to these challenges, our study introduces a
groundbreaking generative-based unified framework for diverse graph tasks,
offering a more holistic approach. Our model employs a generative pretraining
strategy that captures intrinsic features of graph structures. The
incorporation of Generative Adversarial Networks (GANs) further strengthens
the learned representations, endowing them with greater robustness and
discriminative power. Comprehensive experimental evaluations demonstrate the
framework’s superior performance, highlighting its efficacy in unifying
multiple graph tasks. Our pioneering use of a generative model in this context
marks a first in the field, advancing the seamless alignment between various
levels of graph tasks from pretraining through to downstream applications.
This work paves the way for more innovative and efficient approaches in graph
representation learning, capitalizing on the potential of generative models to
tackle the complex landscape of graph-based analyses.

##  Appendix A Notation

Notation | Description
---|---
$\mathcal{V},\mathcal{E}$ | The node and edge set.
$\phi(\cdot),\psi(\cdot)$ | The mapping function of node and edge types.
$\mathcal{X},x_{v}$ | Node attributes matrix and node feature vector.
${\mathcal{G}}$ | A heterogeneous graph.
${\mathcal{T}}_{v},{\mathcal{T}}_{e}$ | The sets of possible node and edge types.
$\rho$ | A metapath.
${\textnormal{h}}_{i}$ | The node representation of node $i$
$\mathcal{N}_{i}^{\rho}$ | The neighbors of node $i$ in metapath $\rho$.
$att_{\text{node}}$ | The GNN layer for node representation.
$att_{\text{sem}}$ | The GNN layer for semantic representation.
$e_{ij}^{\rho},\beta_{\rho}$ |  | The importance score between node-pair $(i,j)$
---
and meta-path.
$p(h|z)$ |  The likelihood between embedding $h$ and latent variable $z$.
$p(z),p(z|h)$ | The prior and posterior probability respectively.
$q_{w}(z|h)$ | The approximation function parameterized by $w$.
$U,U^{+},U^{-}$ | The anchor, positive and negative samples.
$\mu,\sigma$ | The mean and variance.
$\kappa$ | The shift parameter of $\mu$.
$\mu^{*}$ | The shifted mean value.
$\lambda$ | The balance ratio of two kinds of negative samples.
$\tau$ | The temperature parameter for InfoNCE loss.
$\xi$ | The input feature mask ratio.

##  Appendix B Datasets

Table 5. Statistical information on datasets for node classification and link prediction. Datasets | Nodes | Edges | Features | Classes
---|---|---|---|---
Cora | 2708 | 5278 | 1433 | 7
Citeseer | 3327 | 4676 | 3703 | 6
Pubmed | 19717 | 88651 | 500 | 3
Coauthor CS | 18333 | 327576 | 6805 | 15
Coauthor Physics | 34493 | 991848 | 8451 | 5
Amazon Computer | 13752 | 574418 | 767 | 10
Amazon Photo | 7650 | 287326 | 745 | 8
Table 6. Statistical information on datasets for graph classification Datasets | Graphs | Avg Nodes | Classes
---|---|---|---
IMDB-B | 1000 | 19.77 | 2
IMDB-M | 1500 | 13 | 3
PROTEINS | 1113 | 39.1 | 2
COLLAB | 5000 | 74.49 | 3
MUTAG | 188 | 17.9 | 2
REDDIT-B | 2000 | 429.61 | 2
NCI1 | 4110 | 29.8 | 2

##  Appendix C Parameter Settings

##  Appendix D Additional Experiment