# A simple characterization
of doubly twisted spacetimes

Carlo Alberto Mantica and Luca Guido Molinari  C. A. Mantica: I.I.S. Lagrange,
Via L. Modignani 65, 20161 Milano, and I.N.F.N. sez. Milano, Italy – L. G.
Molinari: Physics Department ‘Aldo Pontremoli’, Università degli Studi di
Milano and I.N.F.N. sez. Milano, Via Celoria 16, 20133 Milano, Italy.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

In this note we characterize 1+n doubly twisted spacetimes in terms of ‘doubly
torqued’ vector fields. They extend Bang-Yen Chen’s characterization of
twisted and generalized Robertson-Walker spacetimes with torqued and
concircular vector fields. The result is a simple classification of 1+n
doubly-twisted, doubly-warped, twisted and generalized Robertson-Walker
spacetimes.

###### Key words and phrases:

Twisted spacetime, warped spacetime, torqued vector

###### 2010 Mathematics Subject Classification:

Primary 83C20, Secondary 53B30

##  1\. Introduction

Several interesting Lorentzian metrics have a block-diagonal form, with time
labelling a foliation with spacelike hypersurfaces. They include doubly-
twisted, doubly-warped, twisted, warped, and Robertson-Walker spacetimes [1].
The Frobenius theorem characterizes the vector fields $u_{i}$ that are
hypersurface orthogonal, for which there exist functions $\lambda$ and $f$
such that, locally, $\lambda u_{i}=\nabla_{i}f$ ([2], p.19). This establishes
a dual description of such spacetimes: the special form of the metric allows
explicit evaluations, the one in terms of the vector field is covariant. While
physicists conceive the vector field as a congruence of timelike trajectories,
geometers prefer other vectors, as is here illustrated.

Doubly twisted spacetimes were introduced (and named ‘conformally separable’)
by Kentaro Yano in 1940:

(1) $\displaystyle ds^{2}=-b^{2}({\bf q},t)dt^{2}+a^{2}(t,{\bf
q})g^{*}_{\mu\nu}({\bf q})dq^{\mu}dq^{\nu}$

He showed that the metric structure is necessary and sufficient for the
hypersurfaces to be totally umbilical [3]. The spacetime is doubly warped if
$b$ only depends on ${\bf q}$ and $a$ only depends on time $t$.
Ferrando, Presilla and Morales [4] proved that doubly twisted spacetimes are
covariantly characterized by the existence of a timelike unit, shear and
vorticity free vector field: $u^{i}u_{i}=-1$ and

(2) $\displaystyle\nabla_{i}u_{j}=\varphi(u_{i}u_{j}+g_{ij})-u_{i}\dot{u}_{j}$

where $\dot{u}_{j}=u^{k}\nabla_{k}u_{j}$ is the acceleration, and
$\dot{u}_{j}u^{j}=0$.

In 1979 Bang-Yen Chen introduced twisted spacetimes, eq.(1) with $b=1$ [5].
Years later he characterized them through the existence of a timelike torqued
vector field [6]:

(3) $\displaystyle\nabla_{i}\tau_{j}=\kappa
g_{ij}+\alpha_{i}\tau_{j},\qquad\alpha_{i}\tau^{i}=0$

where $\kappa$ is a scalar field. We gave the equivalent description in terms
of torse-forming time-like unit vectors, eq.(2) with $\dot{u}_{i}=0$, and
obtained the form of the Ricci tensor [7], and unicity of the vector, up to
special cases [8].

Generalized Robertson-Walker (GRW) spacetimes were introduced in 1995 by
Alías, Romero and Sánchez [9, 10]:

(4) $\displaystyle ds^{2}=-dt^{2}+a^{2}(t)g^{*}_{\mu\nu}({\bf
q})dq^{\mu}dq^{\nu}$

Bang-Yen Chen characterized them through the existence of a timelike
concircular vector field $\nabla_{i}\tau_{j}=\kappa g_{ij}$ [11, 12] (the
statement can be weakened, see Prop.2.3).
We gave the alternative characterization (2) with $\dot{u}_{i}=0$ and
$\nabla_{i}\varphi=-u_{i}\dot{\varphi}$, and proved the useful property for
the Weyl tensor [13]: $u_{m}C_{jkl}{}^{m}=0$ if and only if
$\nabla_{m}C_{jkl}{}^{m}=0$. If the Weyl tensor is zero, the spacetime is
Robertson-Walker.

All these cases constitute a rich family of manifolds which are mostly studied
by geometers. They also appear in physics, as inhomogeneous extensions of the
Robertson-Walker metric. In fact, the Einstein equations with a source of
imperfect fluid with shear-free and irrotational velocity, lead to doubly-
twisted metrics [14, 15, 16, 4, 17, 18]. The Stephani universes fall in this
class [2, 19]. The requirement of geodesic flow specialises the metric to
twisted, with interesting applications discussed by Coley and McManus [20].
Doubly warped and GRW (or warped) manifolds have an ample geometric literature
[21, 22, 23, 24, 12].

In this note we present a simple characterization that includes all such
spacetimes, and discuss some properties of doubly torqued vectors.

##  2\. Another characterization

######  Theorem 2.1.

A Lorentzian spacetime is doubly-twisted if and only if it admits a timelike
vector field, which we name ‘doubly torqued’:

(5) $\displaystyle\nabla_{i}\tau_{j}=\kappa
g_{ij}+\alpha_{i}\tau_{j}+\tau_{i}\beta_{j}$

with $\alpha_{i}\tau^{i}=0$, $\beta_{i}\tau^{i}=0$, and
$n\kappa=\nabla_{i}\tau^{i}$.

###### Proof.

We prove the equivalence of (5) with (2).

Let $N=\sqrt{-\tau^{i}\tau_{i}}$, and introduce the vector $u_{i}=\tau_{i}/N$.
Evaluate:
$\nabla_{i}N^{2}=-2\tau^{j}\nabla_{i}\tau_{j}=-2\kappa\tau_{i}+2\alpha_{i}N^{2}$.
Then: $\nabla_{i}N=-\kappa u_{i}+\alpha_{i}N$. Next:

$\displaystyle N\nabla_{i}u_{j}=\nabla_{i}\tau_{j}-u_{j}\nabla_{i}N=\kappa
g_{ij}+N\alpha_{i}u_{j}+Nu_{i}\beta_{j}+\kappa u_{i}u_{j}-N\alpha_{i}u_{j}$

Therefore: $\nabla_{i}u_{j}=(\kappa/N)(u_{i}u_{j}+g_{ij})+u_{i}\beta_{j}$.
Contraction with $u^{i}$ shows that $\beta_{j}=-\dot{u}_{j}$, and eq.(2) is
obtained.

Given (2), the corresponding metric is (1). Define $\beta_{j}=-\dot{u}_{j}$
and $\tau_{i}=Su_{i}$, where $S$ is to be found. Multiply eq.(2) by $S$:

$\nabla_{i}(Su_{j})-u_{j}\nabla_{i}S=\varphi
S(u_{i}u_{j}+g_{ij})+Su_{i}\beta_{j}$

This is eq.(5) with the vector $\alpha_{i}=(\nabla_{i}S)/S+\varphi u_{i}$.
We must impose the condition: $0=\alpha_{i}\tau^{i}=u^{i}\nabla_{i}S-S\varphi$
i.e. $\varphi=u^{i}\nabla_{i}\log S$.
In the frame (1) it is $u_{0}=-b$, $u_{\mu}=0$, and the condition becomes
$\varphi=(\partial_{t}S)/(Sb)$. With the Christoffel symbols in appendix we
obtain $\dot{u}_{0}=u^{0}(\partial_{t}-\Gamma_{00}^{0})u_{0}=0$ and
$\dot{u}_{\mu}=-u^{0}\Gamma_{0\mu}^{0}u_{0}=b_{\mu}/b$. The time component of
(2), $\nabla_{0}u_{0}=\varphi(u_{0}^{2}+g_{00})$, gives
$\varphi=(\partial_{t}a)/(ab)$. Therefore $S=a$ up to a constant factor. ∎

There is some freedom in the choice of the doubly torqued vector:
multiplication of eq.(5) by a function $\lambda$ gives an equation for a
vector $\lambda\tau_{i}$ that is orthogonal to the hypersurfaces and
$\nabla_{i}(\lambda\tau_{j})=(\lambda\kappa)g_{ij}+(\alpha_{i}+\partial_{i}\lambda/\lambda)(\lambda\tau_{j})+(\lambda\tau_{i})\beta_{j}$.
It is doubly torqued provided that:

(6) $\displaystyle\tau^{i}\partial_{i}\lambda=0$

We show the relation of the special vectors $\alpha_{i}$ and $\beta_{i}$ with
the scale functions $a$ and $b$ of the metric (1).
In the coordinate frame $(t,{\bf q})$, the vectors are
$\tau_{i}=(\tau_{0},0)$, $\alpha_{i}=(0,\alpha_{\mu})$ and
$\beta_{i}=(0,\beta_{\mu})$. The equations (5) are:
$\partial_{0}\tau_{0}-\Gamma_{00}^{0}\tau_{0}=-\kappa b^{2}$,
$\partial_{\mu}\tau_{0}-\Gamma_{\mu 0}^{0}\tau_{0}=\tau_{0}\alpha_{\mu}$ ,
$-\Gamma_{0\mu}^{0}\tau_{0}=\beta_{\mu}\tau_{0}$ and
$-\Gamma_{\mu\nu}^{0}\tau_{0}=\kappa a^{2}g^{*}_{\mu\nu}$. They can be
rewritten as follows:

(7) $\displaystyle\beta_{\mu}=-\partial_{\mu}\log b$ (8)
$\displaystyle\partial_{t}\tau_{0}=\tau_{0}\partial_{t}\log(ab)$ (9)
$\displaystyle\partial_{\mu}\tau_{0}=\tau_{0}(\alpha_{\mu}-\beta_{\mu})$ (10)
$\displaystyle\kappa b=-\partial_{t}(\tau_{0}/b)$

The second equation is integrated: $\tau_{0}(t,{\bf q})=c({\bf q})(ab)(t,{\bf
q})$, where $c$ is an arbitrary function. Then, the first and third equation
give $\alpha_{\mu}=\partial_{\mu}\log(ca)$.
In coordinates $(t,{\bf q})$ the condition (6) is
$\tau^{0}\partial_{t}\lambda=0$ and implies that $\lambda$ does not depend on
time $t$. The transformation
$(\tau_{0},\alpha_{\mu},\beta_{\mu},\kappa)\to(\lambda\tau_{0},\alpha_{\mu}+\partial_{\mu}\lambda/\lambda,\beta_{\mu},\lambda\kappa)$
leaves the above equations unchanged. This freedom is used to put $c({\bf
q})=\pm 1$. Then: $\tau_{0}(t,{\bf q})=-(ab)(t,{\bf q})$ (if $\tau^{0}>0$).
The other equations give:

(11) $\displaystyle\alpha_{\mu}=\frac{\partial}{\partial q^{\mu}}\log
a,\qquad\beta_{\mu}=-\frac{\partial}{\partial q^{\mu}}\log
b,\qquad\kappa=\frac{1}{b}\frac{\partial a}{\partial t}$

and establish a simple relation of the vectors with the metric. An interesting
invariant is:

(12) $\displaystyle\tau^{k}\tau_{k}=-\frac{1}{b^{2}}\tau_{0}^{2}=-a^{2}$

For a doubly warped $1+n$ metric, the scale functions $b$ does not depend on
time and $a$ does not depend on ${\bf q}$. In this case, the analysis shows
that $\alpha_{i}$ is either zero or a gradient orthogonal to $\tau$, that can
be absorbed by a rescaling of $\tau$.
We obtain the characterization:

######  Theorem 2.2.

A $1+n$ spacetime is doubly warped if and only if there is a timelike vector
such that $\nabla_{i}\tau_{j}=\kappa g_{ij}+\tau_{i}\beta_{j}$ with
$\tau^{i}\beta_{i}=0$, and $\beta_{i}$ is closed.

###### Proof.

In a doubly warped spacetime, $b({\bf q})$ and $a(t)$ are given and specialize
eq.(5). Eq.(11) gives: $\alpha_{\mu}=0$ i.e. $\alpha_{i}=0$. Eq.(7) with
$\partial_{t}b=0$ implies that $\beta_{i}$ is closed.

If $\alpha_{i}=0$ in (5), then eq.(9),
$0=\partial_{\mu}\tau_{0}/\tau_{0}-\partial_{\mu}\log b$, has solution
$\tau_{0}(t,{\bf q})=F(t)b(t,{\bf q})$. The result in (8) gives:
$\partial_{t}\log a=\partial_{t}F/F$, so that $a=a(t)$. $\beta_{i}$ closed
becomes $\beta_{\mu}=-\partial_{\mu}r({\bf q})$. Eq.(7) gives $b(t,{\bf
q})=\exp[r({\bf q})+s(t)]$. The metric $ds^{2}=-e^{2r({\bf
q})}e^{2s(t)}dt^{2}+a^{2}(t)g^{*}_{\mu\nu}dq^{\mu}dq^{\nu}$ is doubly warped
with a redefinition of time. ∎

In twisted 1+n spacetimes, $a$ depends on $t$ and ${\bf q}$, and $b=1$. Then
$\beta_{i}=0$ and we recover Chen’s result (3). In a GRW spacetime the scale
function $a$ only depends on $t$, so that also the vector $\alpha_{i}$ is
zero.

######  Proposition 2.3.

In a doubly twisted spacetime with doubly-torqued vector $\tau_{i}$, if
$\beta_{i}=\nabla_{i}\theta$ and $\tau^{i}\nabla_{i}\theta=0$ then the
spacetime is conformally equivalent to a twisted spacetime.

###### Proof.

Consider the conformal map $\hat{g}_{ij}=e^{2\theta}g_{ij}$. The new
Christoffel simbols are

$\hat{\Gamma}_{ij}^{k}=\Gamma_{ij}^{k}+\delta_{i}^{k}\partial_{j}\theta+\delta_{j}^{k}\partial_{i}\theta-
g_{ij}g^{kl}\partial_{l}\theta$

The vector $\hat{\tau}_{i}=e^{\theta}\tau_{i}$ solves:

$\displaystyle\hat{\nabla}_{i}\hat{\tau}_{j}=$
$\displaystyle\nabla_{i}(e^{\theta}\tau_{j})-\hat{\tau}_{i}\partial_{j}\theta-\hat{\tau}_{j}\partial_{i}\theta+g_{ij}g^{kl}\hat{\tau}_{k}\partial_{l}\theta$
$\displaystyle=$
$\displaystyle\hat{\tau}_{j}\partial_{i}\theta+e^{\theta}(\kappa
g_{ij}+\alpha_{i}\tau_{j}+\tau_{i}\partial_{j}\theta)-\hat{\tau}_{i}\partial_{j}\theta-\hat{\tau}_{j}\partial_{i}\theta+\hat{g}_{ij}\hat{g}^{kl}\hat{\tau}_{k}\beta_{l}$
$\displaystyle=$
$\displaystyle(e^{-\theta}\kappa)\hat{g}_{ij}+\alpha_{i}\hat{\tau}_{j}$

The absence of $\beta_{i}$ characterizes a twisted spacetime. ∎

A consequence of the proof is the following statement (obvious if regarded on
the side of the metric):

######  Proposition 2.4.

Conformal transformations $\hat{g}_{ij}=e^{2\theta}g_{ij}$ map doubly twisted
to doubly twisted spacetimes.
The same conformal transformation, with the condition
$\tau^{i}\nabla_{i}\theta=0$, maps doubly warped to doubly warped spacetimes,
or twisted to twisted spacetimes.

###### Proof.

Given the vector $\tau_{i}$, consider the vector
$\hat{\tau}_{i}=e^{2\theta}\tau_{i}$ in the new metric. It solves the equation
of a doubly torqued vector:

$\hat{\nabla}_{i}\hat{\tau}_{j}=(\kappa+\hat{\tau}^{k}\partial_{k}\theta)\hat{g}_{ij}+(\alpha_{i}+\hat{h}_{i}{}^{k}\partial_{k}\theta)\hat{\tau}_{j}+\hat{\tau}_{i}(\beta_{j}-\hat{h}_{j}{}^{k}\partial_{k}\theta)$

where $\hat{h}_{ij}=\hat{g}_{ij}-\hat{\tau}_{i}\hat{\tau}_{j}/\hat{\tau}^{2}$
is a projection.

If instead the vector $\hat{\tau}_{i}=e^{\theta}\tau_{i}$ is considered, in
the new metric it solves

$\hat{\nabla}_{i}\hat{\tau}_{j}=(e^{-\theta}\kappa+\hat{\tau}^{k}\partial_{k}\theta)\hat{g}_{ij}+\alpha_{i}\hat{\tau}_{j}+\tau_{i}(\beta_{j}-\partial_{j}\theta)$

If the space is doubly warped ($\alpha_{i}=0$ and $\beta$ closed) then, with
the condition $\tau^{i}\partial_{i}\theta=0$, the vector $\hat{\tau}_{i}$
solves the equation for a doubly warped spacetime. The same is true for
twisted spacetimes ($\beta_{i}=0$). ∎

##  3\. Conclusion

The introduction of doubly torqued vectors, defined by the equation (5), has
the virtue of covariantly describing a class of $1+n$ spacetimes in simple
manner: doubly-twisted ($\alpha\neq 0$, $\beta\neq 0$), doubly-warped
($\alpha=0$, $\beta\neq 0$ closed), twisted ($\alpha\neq 0$, $\beta=0$),
generalized Robertson-Walker ($\alpha=\beta=0$).

We provide some examples from the physics literature.
Stephani universes [19] are conformally flat solutions of the Einstein field
equations with a perfect fluid source. As in the Robertson-Walker space-time,
the hypersurfaces orthogonal to the matter world-lines have constant
curvature, but now its value $k$, and even its sign, changes from one
hypersurface to another. The line element is doubly twisted:

$ds^{2}=-D(t,{\bf x})dt^{2}+\frac{R^{2}(t)}{V^{2}(t,{\bf
x})}(dx^{2}+dy^{2}+dz^{2})$

with $V=1+\frac{1}{4}\|{\bf x}-{\bf x_{0}}(t)\|^{2}$,
$D=F(t)[\dot{V}/V-\dot{R}/R]$, $k=[C^{2}(t)-1/F^{2}(t)]R^{2}(t)$, arbitrary
functions of time $C$, $F$, $R$ and ${\bf x_{0}}$.
Another example is the solution by Banerjee et al. [15] for a matter field
with shear and vorticity free velocity and heat transfer, in a conformally
flat metric:

$ds^{2}=-V^{2}(t,{\bf x})dt^{2}+\frac{1}{U^{2}(t,{\bf
x})}(dx^{2}+dy^{2}+dz^{2})$

with $UV=A(t)\|{\bf x}\|^{2}+{\bf A}(t){\cdot\bf x}+A_{4}(t)$, $U=B(t)\|{\bf
x}\|^{2}+{\bf B}(t){\cdot\bf x}+B_{4}t$, where $A$, $B$, $A_{i}$ and $B_{i}$
are arbitrary functions of time.
Coley studied the case with no acceleration, $V=1$, which makes the above
metric twisted ([20] eq.3.12). The same paper contains an example of GRW
spacetime (eq. 1.2), which is also Bianchi VI0:

$ds^{2}=-dt^{2}+X(t)^{2}(dx^{2}+e^{-2x}dy^{2}+e^{2x}dz^{2})$

For a discussion and examples of doubly warped metrics, see [21].

## Appendix

These are the Christoffel symbols for the doubly-twisted metric (1):

$\displaystyle\Gamma_{00}^{0}=\frac{\partial_{t}b}{b},\quad\Gamma_{\mu,0}^{0}=\frac{b_{\mu}}{b},\quad\Gamma_{0,0}^{\mu}=\frac{bb^{\mu}}{a^{2}},\quad\Gamma^{\rho}_{\mu,0}=\frac{\partial_{t}a}{a}\delta^{\rho}_{\mu},\quad\Gamma^{0}_{\mu,\nu}=\frac{a\partial_{t}a}{b^{2}}g^{*}_{\mu\nu},$
$\displaystyle\Gamma^{\rho}_{\mu,\nu}=\Gamma^{*\rho}_{\mu,\nu}+\frac{a_{\nu}}{a}\delta^{\rho}_{\mu}+\frac{a_{\mu}}{a}\delta^{\rho}_{\nu}-\frac{a^{\rho}}{a}g^{*}_{\mu\nu}$

where $a_{\mu}=\partial_{\mu}a$ and $a^{\mu}=g^{*\mu\nu}a_{\nu}$, and the same
is for $b$.

## References

  * [1] R. Ponge and H. Reckziegel, Twisted product in pseudo-Riemannian geometry, Geom. Dedicata 48 (1993) 15–25.
  * [2] H. Stephani, D. Kramer, M. MacCallum, C. Hoenselaers and E. Herlt, Exact Solutions of Einstein’s Field Equations, Second Edition. Cambridge University Press: Cambridge, 2003.
  * [3] K. Yano, Conformally separable quadratic differential forms, Proc. Imp. Acad. 16 n.3 (1940) 83–86.
  * [4] J. J. Ferrando, J. A. Morales and M. Portilla, Inhomogeneons space-times admitting isotropic radiation: vorticity-free case, Phys. Rev. D 46 n.2 (1992) 578–584.
  * [5] B-Y Chen, Totally umbilical submanifolds, Soochow J. Math. 5 (1979) 9–37.
  * [6] B-Y Chen, Rectifying submanifolds of Riemannian manifolds and torqued vector fields, Kragujevac Journal of Mathematics 41 n.1 (2017), 93–103.
  * [7] C. A. Mantica and L. G. Molinari, Twisted Lorentzian manifolds: a characterization with torse-forming time-like unit vectors, Gen. Relativ. Gravit. 49 (2017) 51, 7 pp.
  * [8] L. G. Molinari, A. Tacchini and C. A Mantica, On the uniqueness of a shear-vorticity-acceleration-free velocity field in space-times, Gen. Relativ. Gravit. (2019) 51:217, 13 pp.
  * [9] L. Alías, A. Romero and M. Sánchez, Uniqueness of complete spacelike hypersurfaces of constant mean curvature in generalized Robertson-Walker spacetimes, Gen. Relativ. Gravit. 27 (1) (1995) 71–84.
  * [10] C. A. Mantica and L. G. Molinari, Generalized Robertson-Walker space-times, a survey, Int. J. Geom. Meth. Mod. Phys. 14 n.3 (2017) 1730001, 27 pp.
  * [11] B-Y Chen, A simple characterization of generalized Robertson-Walker space-times, Gen. Relativ. Gravit. 46 (2014) 1833, 5 pp.
  * [12] B-Y Chen, Differential geometry of warped product manifolds and submanifolds, World Scientific (2017).
  * [13] C. A. Mantica and L. G. Molinari, On the Weyl and Ricci tensors of Generalized Robertson-Walker spacetimes, J. Math. Phys. 57 n.10 (2016) 102502, 6pp.
  * [14] S. R. Maiti, Fluid with heat flux in a conformally flat space-time, Phys. Rev. D 25 (10) (1982) 2518–2520.
  * [15] A. Banerjee, S. B. Dutta Choudhury and B. K. Bhui, Conformally flat solution with heat flux, Phys. Rev. D 40 (2) (1989) 670–672.
  * [16] Y. Deng and P. D. Mannheim, Shear-free spherically symmetric inhomogeneous cosmological model with heat flow and bulk viscosity, Phys. Rev. D 42 (2) (1990) 371–383.
  * [17] A. M. Msomi, K. S. Govinder and S. D. Maharaj, New shear-free relativistic models with heat flow, Gen. Relativ. Gravit. (2011) 43:1685–1696.
  * [18] M. Gürses, M. Plaue and M. Scherfner, On a particular type of product manifolds and shear-free cosmological models, Class. Quantum Grav. 28 (2011) 175009 (12pp.)
  * [19] A. Krasiński, On the Global Geometry of the Stephani Universe, Gen. Relativ. Gravit. 15 n.7 (1983) 673–689.
  * [20] A. A. Coley and Des J. C. McManus, On spacetimes admitting shear-free, irrotational, geodesic time-like congruences, Class. Quantum Grav. 11 (1994) 1261–1282.
  * [21] M. P. M. Ramos, E. G. L. R. Vaz and J. Carot, Double warped space–times, J. Math. Phys. 44 n.10 (2003) 4839 (pp. 27).
  * [22] M. Brozos-Vázquez, E. García-Río and R. Vázquez-Lorenzo, Some remarks on locally conformally flat static space-times, J. Math. Phys. 46 (2005) 022501.
  * [23] Bülent Un̈al, Doubly warped products, Differential Geometry and its Applications 15 (2001) 253–263.
  * [24] A. Gebarowski, Doubly warped products with harmonic Weyl conformal curvature tensor, Colloq. Math. 67 (1) (1994) 73–89.


# Multimodal Fake News Detection via CLIP-Guided Learning

Yangming Zhou  School of Computer ScienceFudan UniversityShanghaiChina
<EMAIL_ADDRESS>,  Qichao Ying  School of Computer ScienceFudan
UniversityShanghaiChina<EMAIL_ADDRESS>,  Zhenxing Qian  School of
Computer ScienceFudan UniversityShanghaiChina<EMAIL_ADDRESS>,  Sheng Li
School of Computer ScienceFudan UniversityShanghaiChina<EMAIL_ADDRESS>and  Xinpeng Zhang  School of Computer ScienceFudan UniversityShanghaiChina
<EMAIL_ADDRESS>
(2018)

###### Abstract.

Multimodal fake news detection has attracted many research interests in social
forensics. Many existing approaches introduce tailored attention mechanisms to
guide the fusion of unimodal features. However, how the similarity of these
features is calculated and how it will affect the decision-making process in
FND are still open questions. Besides, the potential of pretrained multi-modal
feature learning models in fake news detection has not been well exploited.
This paper proposes a FND-CLIP framework, i.e., a multimodal Fake News
Detection network based on Contrastive Language-Image Pretraining (CLIP).
Given a targeted multimodal news, we extract the deep representations from the
image and text using a ResNet-based encoder, a BERT-based encoder and two
pair-wise CLIP encoders. The multimodal feature is a concatenation of the
CLIP-generated features weighted by the standardized cross-modal similarity of
the two modalities. The extracted features are further processed for
redundancy reduction before feeding them into the final classifier. We
introduce a modality-wise attention module to adaptively reweight and
aggregate the features. We have conducted extensive experiments on typical
fake news datasets. The results indicate that the proposed framework has a
better capability in mining crucial features for fake news detection. The
proposed FND-CLIP can achieve better performances than previous works, i.e.,
0.7%, 6.8% and 1.3%improvements in overall accuracy on Weibo, Politifact and
Gossipcop, respectively. Besides, we justify that CLIP-based learning can
allow better flexibility on multimodal feature selection.

datasets, neural networks, gaze detection, text tagging

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer systems organization
Embedded systems††ccs: Computer systems organization Redundancy††ccs: Computer
systems organization Robotics††ccs: Networks Network reliability

##  1\. Introduction

Online social networks have largely replaced the conventional way of
information communication represented by newspapers and magazines. People
enjoy the convenience of online social media in seeking friends or sharing
viewpoints. However, OSNs have also promoted the wide and rapid spreading of
fake news (Zubiaga et al., 2018; Qi et al., 2019; Khattar et al., 2019).
Online news posts can be more easily manipulated compared to written
materials. News forgery can take various forms, for example, replacing a
critical object within a picture with another one, or making biased or even
misleading comments on the picture. What’s worse, the readers are susceptible
to well-crafted fake news and further circulate them. In sum, fake news is
likely to create panic and misdirect public opinion, which alters society in
negative ways.

In the past decades, Fake News Detection (FND) has been the center of data-
centric research for decades (Allein et al., 2021; Shu et al., 2020a; Xue et
al., 2021). While manual observation towards all news and posts on the
Internet is both expensive and time-consuming, automatic FND using machine
learning is an efficient way to combat the widespread dissemination of fake
news. FND has helped news readers identify bias and misinformation in news
articles and therefore stop their spreading. Early works on fake news
detection merely focused on text-only or image-only content analysis (Bian et
al., 2020; Conroy et al., 2015). A pretrained model is usually employed to
verify the logical and semantic soundness of the input. Also, trivial clues
such as grammatical errors or traces left by image manipulation might be taken
into consideration. While unimodal FND schemes are effective, modern news and
posts are usually with rich information of several modalities and these
methods neglect their correlation. For some fake news, a real image can be
combined with total rumors and correct words can be used to describe a
tampered image. In that sense, multimodal feature analysis is required to
offer complementary benefits to assist FND.

Figure 1. Examples of fake news detection using FND-CLIP.The three attention
scores of each news are text score, image score, and fusion score
respectively.

In recent years, there have already been a lot of works that aggregate
multimodal features to detect anomalies in news and posts (Khattar et al.,
2019; Wang et al., 2018; Chen et al., 2022). Besides fusing features from
images and texts, comments, up-vote ratio and the spreading graph are mostly
preferred by researchers to evaluate the truthfulness of a post. These
additional modalities are interactive and change over time. Many previous
works prefer using as much modalities as possible. However, interactive
modalities are less dependable than images and texts, or static modalities.
First, the absence of interactive modalities might be common. A typical
example is that no clue can be left in historical behavior in news posted by
newly registered users, and nor can it be left in comments or up-votes if we
wish to reject fake news shortly after their submission. Second, interactive
modalities are less stable and can be changed over time, therefore potentially
resulting in varied forensics results. Therefore, we revisit current arts in
FND with only static modalities, and find that though many algorithms design
well-crafted networks for multimodal feature fusion (Singhal et al., 2020;
Wang et al., 2018), the mechanisms are largely at a black-box level as to how
multimodal features will influence the final decision. Some works try to
address this issue by explicitly calculating correlation on generating fused
features. For example, Chen et al. (Chen et al., 2022) additionally train
variational auto encoders (VAE) that first compress the images and texts and
contrastively learns to minimize the Kullback-Leibler (KL) divergence for news
with correct image-text pairs. The corresponding cross-modal ambiguity score
is then used to reweight the multimodal features (Chen et al., 2022). Dhruv et
al. (Khattar et al., 2019) propose MVAE that trains a decoder to reconstruct
the original texts and low-level image features from the fused features. These
methods have achieved decent performance in multimodal fake news detection.

However, there are still some issues for multimodal FND to be addressed.
First, we find that how the similarity of features from different modalities
is to be calculated and how it will affect the decision-making process in FND
is still an open question. For (Chen et al., 2022), we are not sure how
efficient the VAEs are so that the KL divergence will be small given matched
image-text pairs. For MVAE, though the ability of reconstruction means that
the fused features are able to contain more information, the necessity of
these auxiliary tasks in the view of FND remains unknown. Besides, we find
that more advanced multimodal learning paradigms and pretrained models are not
properly applied in FND. For example, CLIP (Radford et al., 2021) is a
multimodal model that combines knowledge of language concepts with semantic
knowledge of images. It was trained on a variety of image-text pairs to
predict the most relevant text snippet, given an image, and vice versa. CLIP,
together with other advanced multimodal technologies can be beneficial in
image-text feature fusing, yet their usages in FND still remain ill-posed.

This paper proposes FND-CLIP, a multimodal fake news detection network based
on the pretrained Contrastive Language-Image Pretraining (CLIP) model. The
CLIP-based learning for fake news detection is to address the issue of cross-
modal ambiguity by explicitly measuring the correlation between texts and
images of targeted posts, and to guide the feature fusing and decision-making
stages. Specifically, we encode the image using a fine-tunable ResNet (He et
al., 2016a) encoder a pretrained CLIP image encoder. The text is encoded by a
fine-tunable BERT (Devlin et al., 2018) encoder as well as a CLIP text
encoder. The unimodal features are generated by concatenating the CLIP-
generated features with the fine-tunable counterparts. The fused features
consist of the two CLIP outputs. We use three projection heads to individually
process the unimodal and fused features, which shrinks their sizes in order to
distill the most important features for FND. Besides, we calculate the cosine
similarity on the CLIP outputs and standardize it as the cross-modal
similarity score. The score reweights the fused feature, where we regulate
that less information will be provided by the fused features if the image and
text show low correlation. Furthermore, we introduce an attention layer that
outputs three scores that adaptively measure the significance of these
features in their contribution to fake news detection. The classifier finally
processes the summarized features to distinguish fake news from real ones.

We have conducted extensive experiments on FND-CLIP on several typical
datasets for FND, including a Chinese dataset named Weibo, and two English
datasets named Politifact and Gossip. The results show that FND-CLIP achieves
0.7%, 6.8% and 1.3% performance improvement in overall accuracy on the three
datasets. Besides, we justify that CLIP-based learning can allow better
flexibility on multimodal feature selection. Figure 1 showcases four examples
of fake news detection using FND-CLIP, where we see that the attention score
as well as cross-modal similarity vary among different news instances. FND-
CLIP is able to pay less attention to the multimodal features when the
similarity is low, therefore flexibly aggregating information according to the
characteristics of the provided news.

The contributions of this paper are mainly three-folded, namely:

  * •

We propose FND-CLIP, a multimodal fake news detection method with CLIP-based
learning, where the CLIP pretrained model is used to measure the cross-modal
similarity and guide the mapping and fusion of features.

  * •

We propose a modality-wise attention mechanism to adaptively weight the text,
image, and fused features. Given different news instances, we find that the
model flexibly learns to pay more attention to useful information in unimodal
or multimodal features.

  * •

We have conducted comprehensive experiments on three famous datasets, where
the results prove that CLIP-generated features can be important assists to the
unimodal features. FND-CLIP outperforms state-of-the-art fake news detection
methods.

##  2\. Related Works

###  2.1. Unimodal Fake News Detection

Unimodal FND usually works on finding anomalies in either the text or the
image of a post. These algorithms often follow the essence of human decision
process. For images, Cao et al. (Cao et al., 2020) jointly study image
forensics features, semantic features, statistical features, and context
features for fake news detection. It suggests that typical methods for image
manipulation detecion (Chen et al., 2021) are useful in unveiling traces for
news tampering. Besides, semantic inconsistency regarding the common sense (Li
et al., 2021) as well as poor image quality (Han et al., 2021) can be widely
present in fake news. For texts, verifying the logical soundness is essential
(Guo et al., 2018), also accompanied by finding clues such as grammatical
errors, writing styles (Potthast et al., 2017) or extracting rhetorical
structure (Conroy et al., 2015). Besides, both linguistic and visual patterns
can be highly dependent on specific events and corresponding domain knowledge.
Therefore, Nan et al. (Nan et al., 2021) propose to employ domain gate to
aggregate multiple representations extracted by mixture-of-experts, and it
deals with multi-domain fake news propagation in the language modality.

Though these unimodal characteristics can be explored and they indeed play key
roles in distinguishing fake news, the multimodal characteristics such as
correlation and consistency are ignored, which potentially impair the overall
performance of these unimodal schemes on multimodal news.

Figure 2. The architecture of the proposed FND-CLIP method. CLIP, BERT, and
ResNet are used to extract the features of different modalities of multimodal
news. Encoded features of different levels are obtained through projection
heads. CLIP similarity score is calculated to determine the importance of
fused feature. A modality-wise attention mechanism is further used to reweight
different modal features adaptively for the classifier to classify fake news.

###  2.2. Multimodal Fake News Detection

In the past literature, many works have been done on mining useful
representations from images and texts of the news for fake news detection.
Earlier works design sophisticated yet black-box attention mechanisms for
multimodal feature fusion (Bian et al., 2020; Bhatt et al., 2018). Many other
works (Wang et al., 2018; Khattar et al., 2019; Chen et al., 2022) propose to
better align the extracted features from different modalities before sending
them into the classifier. Wang et al. (Wang et al., 2018) propose EANN that
further employs an auxiliary task of event classification to aid feature
extraction. The event classification branch is designed to better disentangle
the mined multimodal features so that there are both event-specific
information and event-agnostic information. Dhruv et al. (Khattar et al.,
2019) processe the image and text using unimodal feature extractors and
further utilize a multimodal VAE to learn a shared representation from them.
The sampled representation produced by the VAE is then sent to a decoder which
tries to reconstruct the original texts and low-level image features. Besides
the focus on network design, other works exploit more information from the
datasets. For example, Qi et al. (Qi et al., 2021) claim that image feature
extractors cannot well understand visual entities such as celebrities,
landmarks, and texts within the images, and therefore propose to manually
extract these kinds of information as linguistic assists. Zhang et al. (Zhang
et al., 2021) design a novel dual emotion feature descriptor to measure the
emotional gap between the post and its comments and verify that dual emotion
is distinctive between fake and real news. Chen et al. (Chen et al., 2022) use
two VAEs to compress the images and texts and contrastively learn to minimize
the Kullback-Leibler (KL) divergence for correctly matched image-text pairs.
The resultant score is then used to reweight the multimodal features during
feature fusing.

Though these methods have achieved decent performance in multimodal FND, there
are still issues to be concerned. First, how to explicitly measure the
correlation between images and texts within a post still remain unclear.
Second, we see that little work in FND consider applying the recently emerged
arts in multimodal learning, which motivates us to use the CLIP-based
pretraining to further boost the performance.

###  2.3. Multimodal Learning

Recent years have shown rapid developments in the field of multimodal machine
learning (Baltrušaitis et al., 2018). Neural architectures are employed in
tasks that go beyond single modalities, for example, Visual Question Answering
(VQA) (Dancette et al., 2021), Visual Commonsense Reasoning (VCR) (Ye and
Kovashka, 2021), etc. In these tasks and beyond, priors and features from
different modalities are required and algorithms or deep networks cannot be
effective when provided with only a single modality. Several generic
technologies are developed for learning joint representations of image content
and natural language. For example, the CLIP model (Radford et al., 2021) is
designed as a bridge between computer vision and natural language processing.
It was trained on a variety of image-text pairs to predict the most relevant
text snippet, given an image, without directly optimizing for the task. The
model consists of two encoders that respectively embed texts and images into a
uniform mathematical space. Then, for the matched image-text pair, CLIP is
encouraged to maximize the cosine similarity between the embedding of the two
modalities. Otherwise, the similarity is minimized for the model to find the
most suitable paired images and texts. Multimodal learning has a promising
future where the innovation of CLIP has benefitted a lot of down-stream tasks
(Wei et al., 2021; Conde and Turgutlu, 2021). Other multimodal schemes can be
represented by Glide (Nichol et al., 2021) and VilBERT (Lu et al., 2019) that
are respectively for text-to-image generation and multimodal representation
learning.

##  3\. METHOD

###  3.1. Approach Overview

For multimodal fake news detection, we collect the statc modalities of the
sampled news that includes text and image, and denote each sample as
$\mathbf{x}=({\mathbf{x}_{Txt}},{\mathbf{x}_{Img}})$. The ground-truth label
is $y$ where $y=0$ indicates that $\mathbf{x}$ is a real news, otherwise
$y=1$. According to the most traditional multimodal learning paradigm, a rich
set of features are first extracted from ${\mathbf{x}_{Txt}}$ and
${\mathbf{x}_{Img}}$ that both represents the unimodal characteristics and the
multimodal characteristics, which are then further fused and projected into a
single value of $\hat{y}$ that should be close to the ground truth.

(1)
$\hat{y}={{F}_{cls}}({{F}_{Mix}}({{F}_{Txt}}(\mathbf{x}_{Txt}),{{F}_{Img}}(\mathbf{x}_{Img}))),$

where ${{F}_{Txt}}$ and ${{F}_{Img}}$ are unimodal feature extractors,
${F}_{Mix}$ is the feature fusing model and ${F}_{cls}$ is the classification
head. In order to model ${{F}_{Txt}}$ and ${{F}_{Img}}$, most of the previous
methods use different pre-trained models to extract text and image features in
different semantic spaces, and for ${{F}_{Mix}}$, the proposed mechanisms
vary. The crucial point is how to ensure that features provided from both
modalities will be utilized in the later stage, otherwise the gap in semantic
space makes the fused features unable to accurately represent the correlation
between image and text. Instead of applying sophisticated and black-box
feature-fusing networks, we employ a simple yet effective method where
pretrained networks for multimodal learning is introduced to extract aligned
multimodal features and to guide the learning of the classification network.
We choose the CLIP model (Radford et al., 2021) to measure the cross-modal
similarity considering that the model is trained to provide the most
appropriate language description of a given image and vice versa, and
therefore is in line with the above requirements. After feature extraction and
alignment, we use a light-weight network to implement ${L}_{Cls}$ which
predicts $\hat{y}$.

###  3.2. Network Specification

Figure 2 illustrates the network design of FND-CLIP. The whole pipeline
consists of four main modules, namely, unimodal feature encoder, CLIP-based
encoder, projection and attention module, and finally the classifier.

Unimodal feature generation. We use a pretrained BERT model to obtain the
feature ${{f}_{\emph{BERT}}}\in{{\mathbb{R}}^{n_{\emph{BERT}}}}$ of
${\mathbf{x}_{Txt}}$. For the image ${{\mathbf{x}}_{Img}}$, we use ResNet (He
et al., 2016b) to get deep representations
${{f}_{\emph{ResNet}}}\in{{\mathbb{R}}^{n_{\emph{ResNet}}}}$ from the image.
Besides ${{f}_{\emph{BERT}}}$ and ${{f}_{\emph{ResNet}}}$, we use CLIP
encoders to encode text and image and obtain the features
${{f}_{\emph{CLIP-T}}}\in{{\mathbb{R}}^{n_{\emph{CLIP}}}}$ and
${{f}_{\emph{CLIP-I}}}\in{{\mathbb{R}}^{n_{\emph{CLIP}}}}$. In order to
improve the representation capability of the unimodal branches, embedding
concatenation are performed in the text and image unimodal intra-modalities,
respectively,

(2)
$\left\\{\begin{array}[]{l}{{f}_{Txt}}={\emph{concat}}\left({{f}_{\emph{BERT}}},{{f}_{\emph{CLIP-T}}}\right)\\\
{{f}_{Img}}={\emph{concat}}\left({{f}_{\emph{ResNet}}},{{f}_{\emph{CLIP-I}}}\right),\end{array}\right.$

where ${{f}_{Txt}}\in{{\mathbb{R}}^{n_{\emph{BERT}}+n_{\emph{CLIP}}}}$ and
${{f}_{Img}}\in{{\mathbb{R}}^{n_{\emph{ResNet}}+n_{\emph{CLIP}}}}$.

CLIP-guide multimodal feature generation. The text and image features
extracted by BERT and ResNet respectively have significant cross-modal
semantic gaps, and it is difficult for the network to learn their intrinsic
semantic correlation if they are fused directly. Therefore, the two features
are only used as unimodal representation, while the multimodal representation
is obtained by first concatenating the alignment features of the text-image
pair extracted by CLIP and then fine-tuning them to reduce redundancy and
introduce attention. The concatenated feature is denoted as
${{f}_{Mix}}\in{{\mathbb{R}}^{2\times n_{\emph{CLIP}}}}$, where

(3) ${{f}_{Mix}}=\emph{concat}({{f}_{CLIP-T}},{{f}_{CLIP-I}}).$

The multimodal features reflect the correlation between the two modalities and
contain meaningful semantic information. The assistance of the multimodal
features to unimodal features is to learn the cross-modal similarity. Previous
works often use a single network to mine both coarse and fine features from a
modality, which is quite demanding on the learning ability of the model. Here,
with the introduction of CLIP model, BERT and ResNet, which is the pretraining
models for unimodal tasks, can pay more attention to trivial clues compared to
extracting semantic information. For example, BERT can better extract
emotional features of texts, and ResNet can identify higher-frequent noise
patterns of images. In contrast, the training strategy of CLIP uses large-
scale image-text pairs to learn the extraction of semantics, while largely
ignoring emotion, noise and other features irrelevant to image and text
matching. Therefore, using CLIP for multimodal feature generation can well
collaborate with the unimodal features to respectively scrutinize the news
from different aspects.

After we get the three features of different modalities, we use three
individual projection head ${P}_{\emph{Txt}},{P}_{\emph{Img}}$ and
${P}_{\emph{Mix}}$ made up of Multi-Layer Perceptrons (MLP) to process the
features. The goal is to reduce the dimension of the coarse features provided
by the encoders and help filtering out redundant information. These networks
share the same architecture but do not share weights. As is shown in Figure 2,
every the projection head contains two sets of full connected layer with Batch
Normalization (Ioffe and Szegedy, 2015) layer, a ReLU activation function, and
a dropout layer.

Merely combining the CLIP-based features as the multimodal features cannot
necessarily provide enough reliable information. The reason is that the
authenticity of news is not completely correlated with image-text correlation.
Some news posts, no matter real or fake, lack cross-modal relation or even
semantic information. In that case, some instances require more emotion,
noise, and other features, and the corresponding multimodal features might be
noisy when the similarity is low and fully utilizing such information might
impair the performance. To address the ambiguity issue between multimodal
features, we measure the cosine similarity between the text features and the
image features provided by CLIP, to adjust the intensity of fused features.
The cosine similarity is calculated as follows.

(4)
$\emph{sim}=\frac{{{f}_{Txt}}\cdot{{({{f}_{Img}})}^{T}}}{\left\|{{f}_{Txt}}\right\|\left\|{{f}_{Img}}\right\|}.$

Then, we apply standardization and a Sigmoid functions to map the similarity
into the range $[0-1]$. The normalization is done by calculating the running
status of mean and standard deviation during training, and subtract the
running mean from sim and divide it with the running standard deviation.
Compared to the contrastive learning paradigm, the normalization helps to
calculate the similarity without comparing the news post with other instances.

Thus, the process of obtaining the projected unimodal and multimodal features
is as follows.

(5) $\left\\{\begin{array}[]{l}{{m}_{Txt}}=P_{Txt}\left({{f}_{Txt}}\right)\\\
{{m}_{Img}}=P_{Img}\left({{f}_{Img}}\right)\\\
{{m}_{Mix}}=Sigmoid\left(Std\left(\emph{sim}\right)\right)\cdot
P_{Mix}\left({{f}_{Mix}}\right).\\\ \end{array}\right.$

Feature aggregation using modality-wise attention. We apply an attention
mechanism to reweight the projected features before aggregating the features
from different modalities using spatial addition. Inspired by the Squeeze-and-
Excitation Network (SE-Net) (Hu et al., 2018), we designed a modality-wise
attention module as shown in Figure 2 to weight each feature adaptively.
First, the three $L\times 1$ features are concatenated into one $L\times 3$
feature, where $L$ represents the length of the feature. Average pooling and
maximum pooling are adopted to squeeze a $1\times 3$ vector via summation,
corresponding to the initial weight of each channel. Then, the initial weight
obtained in the previous step is sent into the two $3\times 3$ fully connected
layers with GELU (Hendrycks and Gimpel, 2016) activation function, and
normalized into the range $[0-1]$ using Sigmoid functions respectively to
obtain the attention weights
$\emph{att}=\\{\emph{att}_{Txt},\emph{att}_{Img},\emph{att}_{\emph{Mix}}\\}$.
Finally, the weights are multiplied respectively on ${m}_{t},{m}_{i}$ and
${m}_{\emph{mix}}$, and a sum process is performed to obtain the $L\times 1$
aggregated feature ${{m}_{\emph{Agg}}}$.

(6)
${m}_{\emph{Agg}}=\emph{att}_{Txt}\cdot\emph{m}_{Txt}+\emph{att}_{Img}\cdot\emph{m}_{Img}+\emph{att}_{Mix}\cdot\emph{m}_{Mix}.$

Classification and objective function. We feed the aggregated representation
${{m}_{\emph{Agg}}}$ into a two-layer fully-connected network as the
classifier ${F}_{cls}$ to predict the label $\hat{y}$. The objective function
of FND-CLIP is to minimize the cross-entropy loss to correctly predict the
real and fake news.

(7)
$\mathcal{L}_{CE}=y\log\left(\hat{y}\right)+\left(1-y\right)\log\left(1-\hat{y}\right)\text{.
}$

Table 1. Performance comparison between FND-CLIP and other methods on three datasets. Our method achieves the highest accuracy among these methods, and its precision, recall, and FI-score are also higher than most of the compared methods. | Method | Accuracy | Fake News | Real News
---|---|---|---|---
Precision | Recall | F1-score | Precision | Recall | F1-score
Weibo | EANN (Wang et al., 2018) | 0.827 | 0.847 | 0.812 | 0.829 | 0.807 | 0.843 | 0.825
MVAE (Khattar et al., 2019) | 0.824 | 0.854 | 0.769 | 0.809 | 0.802 | 0.875 | 0.837
Spotfake (Singhal et al., 2019) | 0.892 | 0.902 | 0.964 | 0.932 | 0.847 | 0.656 | 0.739
MVNN (Xue et al., 2021) | 0.846 | 0.809 | 0.857 | 0.832 | 0.879 | 0.837 | 0.858
SAFE (Zhou et al., 2020) | 0.762 | 0.831 | 0.724 | 0.774 | 0.695 | 0.811 | 0.748
LIIMR (Singhal et al., 2022) | 0.900 | 0.882 | 0.823 | 0.847 | 0.908 | 0.941 | 0.925
MCAN (Wu et al., 2021) | 0.899 | 0.913 | 0.889 | 0.901 | 0.884 | 0.909 | 0.897
CAFE (Chen et al., 2022) | 0.840 | 0.855 | 0.830 | 0.842 | 0.825 | 0.851 | 0.837
FND-CLIP | 0.907 | 0.914 | 0.901 | 0.908 | 0.914 | 0.901 | 0.907
Politifact | RoBERTa-MWSS (Shu et al., 2020b) | 0.820 |  | - | - | 0.820 | - | -
SAFE (Zhou et al., 2020) | 0.874 | 0.851 | 0.830 | 0.840 | 0.889 | 0.903 | 0.896
Spotfake+ (Singhal et al., 2020) | 0.846 | - | - | - | - | - | -
TM (Bhattarai et al., 2021) | 0.871 | - | - | - | 0.901 | - | -
LSTM-ATT (Lin et al., 2019) | 0.832 | 0.828 | 0.832 | 0.830 | 0.836 | 0.832 | 0.829
DistilBert (Allein et al., 2021) | 0.741 | 0.875 | 0.636 | 0.737 | 0.647 | 0.880 | 0.746
CAFE (Chen et al., 2022) | 0.864 | 0.724 | 0.778 | 0.750 | 0.895 | 0.919 | 0.907
| FND-CLIP | 0.942 | 0.897 | 0.897 | 0.897 | 0.960 | 0.960 | 0.960
Gossipcop | RoBERTa-MWSS (Shu et al., 2020b) | 0.800 | - | - | 0.800 | - | - | -
SAFE (Zhou et al., 2020) | 0.838 | 0.758 | 0.558 | 0.643 | 0.857 | 0.937 | 0.895
Spotfake+ (Singhal et al., 2020) | 0.856 | - | - | - | - | - | -
TM (Bhattarai et al., 2021) | 0.842 | - | - | - | 0.896 | - | -
LSTM-ATT (Lin et al., 2019) | 0.842 | 0.845 | 0.842 | 0.844 | 0.839 | 0.842 | 0.821
DistilBert (Allein et al., 2021) | 0.857 | 0.805 | 0.527 | 0.637 | 0.866 | 0.960 | 0.911
CAFE (Chen et al., 2022) | 0.867 | 0.732 | 0.490 | 0.587 | 0.887 | 0.957 | 0.921
| FND-CLIP | 0.880 | 0.761 | 0.549 | 0.638 | 0.899 | 0.959 | 0.928

###  3.3. Training Detail

On the selection of BERT pretrained models, we respectively use the “bert-
base-chinese” model on Chinese data and the “bert-base-uncased” model on
English data, perform an attention-based post-processing (Jawahar et al.,
2019). The length of the input text is set to 300 words. About the ResNet, we
use pre-trained ResNet-101 to extract visual features, setting the size of the
input image to 224 × 224.The size of the images inputted to CLIP is the same
as that to ResNet. Science CLIP has not pre-trained Chinese text model, we use
Google Translation API (Johnson, 2012) to translate Chinese texts to English.
In addition, we use the summary generation model (Raffel et al., 2019) to
generate summary statements as the CLIP input for the text with the size
longer than 50, to meet the requirements that the input size of the text has
an upper bound in CLIP. The used pre-trained CLIP model is “ViT-B/32”. We
fine-tune ResNet in training stage, while freezing the weights of BERT and
CLIP due to their difficulty in training on small datasets. We implement the
projection heads using two fully connected layers with 256 and 64 hidden
units, respectively. The hidden sizes of the two fully connected layers in the
classifier are 64 and 2, respectively. The batch size is set as 64.

We use Adam optimizer (Kingma and Ba, 2014) with the default parameters. The
learning rate is $1\times{{10}^{-3}}$ where weight decay is 12. We trained a
model for 50 epochs and chose the epoch getting the best test accuracy among
them as the final result to avoid over-fitting.

##  4\. EXPERIMENTS

###  4.1. Experimental Setup

Dataset. We use three real-world datasets collected from social media, namely,
Weibo (Jin et al., 2017), Gossipcop, and Politifact (Shu et al., 2020a).
During experiments, the unimodal news posts with no image or no text
description were filtered out. If a news post contains a text with multiple
associated images, we randomly select one image. Weibo is a widely used
Chinese dataset in fake news detection. The training set contains 3, 749 real
news and 3, 783 fake news, and the test set contains 1, 996 news. Politifact
and Gossipcop datasets are two English datasets collected from the political
and entertainment domains of FakeNewsNet (Shu et al., 2020a) repository,
respectively. Politifact contains 244 real news and 135 fake news in the
training set and 75 real news and 29 news in the test set. Gossipcop contains
10, 010 training news, including 7, 974 real news and 2, 036 fake news. The
test set contains 2, 285 real news and 545 fake news. Besides, while Twitter
(Boididou et al., 2018) is also a well-known multimodal dataset for FND, we
find that it contains plenty of duplicated posts and over 10k posts host only
463 images. More importantly, more than 70% of tweets on Twitter dataset are
related to a single event, which can easily lead to model overfitting.
Therefore, we do not conduct experiments on Twitter.

Figure 3. Statistical analysis on cross-modal similarity of different news.
The first and second row respectively show the counting of real/fake news
according to cross-modal similarity on Weibo and Gossipcop. The third row
shows the distance between the real news rate in each bin compared to the
average rate on the corresponding dataset (left: Weibo. right: Gossipcop).

Baseline Methods. For a fair and reproducible comparison, we have to be
selective in choosing the baseline methods. First, we prefer methods that
provide pre-trained models or source code publicly available. Second, the
methods should follow a common evaluation protocol where the three datasets
are used for training and testing. Accordingly, we compare FND-CLIP with the
following methods and provide a quick recap.

EANN (Wang et al., 2018), which employs an auxiliary task of event
classification to improve generalizability.

MVAE (Khattar et al., 2019), which uses a variational autoencoder to model
representations between text and images for fake news detection.

Spotfake (Singhal et al., 2019), which uses VGG and BERT to respectively
extract image and text features and concatenates them to classify.

MVNN (Xue et al., 2021), which incorporates textual semantic features, visual
tampering features, and similarity of textual and visual information in fake
news detection.

SAFE (Zhou et al., 2020), which fed the relevance between news textual and
visual information into a classifier to detect fake news.

LIIMR (Singhal et al., 2022), which identifies and suppresses information from
weaker modalities and extracts relevant information from the strong modality
on a per-sample basis.

MCAN (Wu et al., 2021), which stacks multiple co-attention layers to fuse the
multimodal features.

CAFE (Chen et al., 2022), which formulates an ambiguity-aware multimodal fake
news detection method to adaptively aggregate unimodal features and cross-
modal correlations.

RoBERTa-MWSS (Shu et al., 2020b), which exploits multiple weak signals from
different sources from user and content engagements.

Spotfake+ (Singhal et al., 2020), which is an improved version of Spotfake and
can detect full length articles.

TM (Bhattarai et al., 2021), which utilizes lexical and semantic properties of
both true and fake news text to detect fake news.

LSTM-ATT (Lin et al., 2019), which builds a model based on XGBoost to detect
full length fake news.

DistilBert (Allein et al., 2021), which uses latent representations of news
articles and user-generated content to guide model learning.

###  4.2. Performance Analysis

Table 1 shows the average precision, recall, and accuracy of FND-CLIP on three
representative datasets. The results are promising, with over 90% average
accuracy on Weibo and over 94% on Politifact, which indicates that the
proposed method is a dependable and robust fake news detection algorithm that
can detect anomalies given multi-lingual and multi-domain news. Especially,
the recall rates of real news on the three datasets are all above 0.9, and
therefore FND-CLIP is less likely to classify a real news as fake.

To further conduct statistical analysis on how cross-modal similarity
correlates with the attention score and how they vary given different news
instances, Figure 3 shows the correlation between the CLIP-based cross-modal
similarity score and the fake news ratio on Weibo and Gossipcop dataset. In
row 1 and 2, we group all news in each dataset into several bins according to
their similarity score, and find that a news is more likely to be real when
the similarity score is high. In row 3, we calculate real news rates of each
bin and subtract them with the average real news rate of the corresponding
dataset. The curves show that real news rate goes up with the increasing
ambiguity on Weibo, and first goes down then surges up on Gossipcop. Such
statistical charateristics are useful for deep networks to identify fake news.

###  4.3. Comparison with State-of-the-arts

We further compare FND-CLIP with the above-mentioned state-of-the-art methods
and the comparison results are presented in Table 1. ’-’ means the results are
not available from the original paper. As shown in Table 1, FND-CLIP
outperforms all the compared methods on the three datasets in terms of
Accuracy, and achieves slightly lower than Spot on Weibo in Recall. FND-CLIP
achieves the highest accuracy of 90.7%, 94.2%, and 88.0%, which surpasses
0.7%, 6.8%, and 1.3% over the state-of-the-art method, on the three real-world
datasets, respectively. Besides, we rank either $1_{st}$ or $2_{nd}$ in
precision, recall, and accuracy in all tests, which proves the effectiveness
of FND-CLIP.

Many fake news detection methods, such as EANN and Spotfake, rely only on the
fused features obtained by direct use of concatenating or attention
mechanisms. However, these fused features cannot provide sufficient
discrimination ability to classify fake news, because the text and image
features separately extracted are not in the same semantic space and the
correlation information of the text and image is not well-paid attention to
during the fusion process. Therefore, the experimental results of these
methods are unsatisfactory. CAFE uses cross-modal alignment to train encoders
that can map texts and images into the same semantic space. By using the
features fused from the alimented text and image features to classify, it
achieves good experimental results, especially on the Politifact and
Gossiopcop datasets. However, due to the limitation of the number of data sets
and the rough label method for training labels, the encoding effect of the
encoder is not optimal, and the semantic gap between text and image features
is still significant. In addition, CAFE designs an ambiguity learning module
to calculate a weight used for adaptively adjusting the calculation of
different modalities. However, the weights for selecting unimodal or
multimodal features are obtained by manual calculation, and cannot be further
optimized by reverse gradient propagation, thus affecting the performance of
the detection.

FND-CLIP outperforms most of the state-of-the-art methods, mainly due to the
following reasons. First, the pre-trained CLIP encoders in FND-CLIP can
generate semantically information-rich text and image features in the same
semantic space, ensuring the fused feature correctly reflects the correlation
between text and image, and providing complementary information for the
unimodal features. The modality-wise attention mechanism adaptively determines
the weights of text, image, and fused features, avoiding the influence of
invalid features on the representation ability of final features, and further
improving the classification accuracy.

Table 2. Ablation study on the architecture design and different features of FND-CLIP on three datasets.The entire FND-CLIP achieved the highest accuracy and F1-score, demonstrating that every module in the architecture of our method is effectiveness and every modality is effectively utilized. | Method | Accuracy | Fake News | Real News
---|---|---|---|---
Precision | Recall | F1-score | Precision | Recall | F1-score
Weibo | FND-CLIP multimodal-only | 0.817 | 0.899 | 0.718 | 0.798 | 0.761 | 0.917 | 0.832
FND-CLIP image-only | 0.796 | 0.862 | 0.711 | 0.779 | 0.750 | 0.884 | 0.811
FND-CLIP text-only | 0.872 | 0.906 | 0.833 | 0.868 | 0.842 | 0.911 | 0.875
FND-CLIP w/o C | 0.874 | 0.895 | 0.851 | 0.872 | 0.855 | 0.898 | 0.876
FND-CLIP w/o F | 0.893 | 0.925 | 0.857 | 0.890 | 0.864 | 0.929 | 0.895
FND-CLIP w/o A | 0.897 | 0.936 | 0.855 | 0.893 | 0.863 | 0.940 | 0.900
FND-CLIP | 0.907 | 0.914 | 0.901 | 0.908 | 0.901 | 0.914 | 0.907
Politifact | FND-CLIP multimodal-only | 0.903 | 0.807 | 0.862 | 0.833 | 0.944 | 0.919 | 0.932
FND-CLIP image-only | 0.748 | 0.600 | 0.310 | 0.409 | 0.773 | 0.919 | 0.840
FND-CLIP text-only | 0.903 | 0.913 | 0.724 | 0.808 | 0.900 | 0.973 | 0.935
FND-CLIP w/o C | 0.893 | 0.875 | 0.724 | 0.793 | 0.899 | 0.960 | 0.928
FND-CLIP w/o F | 0.903 | 0.880 | 0.759 | 0.815 | 0.910 | 0.960 | 0.934
FND-CLIP w/o A | 0.942 | 0.926 | 0.862 | 0.893 | 0.947 | 0.973 | 0.960
FND-CLIP | 0.942 | 0.897 | 0.897 | 0.897 | 0.960 | 0.960 | 0.960
Gossipcop | FND-CLIP multimodal-only | 0.862 | 0.708 | 0.484 | 0.575 | 0.886 | 0.952 | 0.918
FND-CLIP image-only | 0.814 | 1.000 | 0.033 | 0.064 | 0.813 | 1.000 | 0.897
FND-CLIP text-only | 0.871 | 0.741 | 0.508 | 0.603 | 0.891 | 0.958 | 0.923
FND-CLIP w/o C | 0.870 | 0.745 | 0.494 | 0.594 | 0.888 | 0.960 | 0.923
FND-CLIP w/o F | 0.874 | 0.723 | 0.562 | 0.632 | 0.901 | 0.949 | 0.924
FND-CLIP w/o A | 0.873 | 0.715 | 0.567 | 0.633 | 0.902 | 0.946 | 0.923
FND-CLIP | 0.880 | 0.761 | 0.549 | 0.638 | 0.899 | 0.959 | 0.928

###  4.4. Ablation Studies

We explore the influence of the key components in FND-CLIP by evaluating the
performance of the model with varied and partial setups. In each test, we
remove different components and train the models from scratch. The compared
variants of FND-CLIP are implemented as follows.

  * •

FND-CLIP w/o A. We remove the modality-wise attention module and direct
aggregate the three features to obtain final feature;

  * •

FND-CLIP w/o F. We remove the fusion module and use two unimodal features to
classify news;

  * •

FND-CLIP w/o C. We remove all CLIP-related modules and only use BERT and
ResNet to extract text and image features.

  * •

FND-CLIP multimodal-only: We remove the unimodal feature extractor, BERT and
ResNet, and only use CLIP fused feature as final feature;

  * •

FND-CLIP image-only: We remove the all text-related features and only use
image feature extracted by ResNet to classify;

  * •

FND-CLIP text-only: We only use BERT-extracting feature to complete the
detection task without any visual information.

Effectiveness of Each Component.  First, we analyze the impact of different
components in FND-CLIP for fake news detection. From the results shown in
Table 2, we have the following observations: 1) FND-CLIP outperforms FND-CLIP
w/o C, proving that CLIP can effectively provide discernable features for fake
news detection task and significantly improve the accuracy of classification.
Although only intra-modal features can be used for classification, the lack of
interaction between modalities makes the final features lack the ability to
represent the intrinsic relationship between images and texts. 2) FND-CLIP
outperforms FND-CLIP w/o F, indicating that although the unimodal branches
contain the CLIP-coded features, the fused feature reflecting the correlation
of text and image provides effective multimodal information for classifier.
Meanwhile, FND-CLIP w/o F outperforms FND-CLIP w/o C, indicating that the
complement to unimodal features using CLIP-coded features is effective. 3)
FND-CLIP outperforms FND-CLIP w/o A on Weibo and Gossipcop, indicating that
modality-wise attention can help FND-CLIP adaptively weight useful modalities.
FND-CLIP w/o A directly fuses the features of different modalities, which may
cause the final feature be affected by invalid information from a modality.

Contributions from Different Modalities.  The second set of experiments is to
evaluate the classification performance of different modalities in fake news
detection. From Table 2, we draw some analysis as follows: 1) FND-CLIP image-
only performs worst, especially on Gossipcop dataset, where the F1 score of
fake news was almost zero, meaning that all news was judged real and the model
had no classification ability at all. This shows that in fake news detection,
simple visual information provides fewer classification clues than other
modalities. 2) FND-CLIP multimodal-only achieves accuracy of 81.7%, 90.3%, and
86.2% on Weibo, Politifact, and Gossipcop datasets respectively, but performs
worse than FND-CLIP text-only on Weibo and Gossipcop datasets, indicating that
the correlation information of images and texts can be used to classify fake
news. However, the classification ability of fused feature is limited because
news itself has modal irrelevance and ambiguity. In addition, CLIP-based fused
features focus on the semantics of the text, while the BERT-based text
features also extract emotional features that are helpful for fake news
detection. 3) FND-CLIP text-only achieves the second-best results, indicating
that only using text feature can basically complete the classification task
for fake news. However, FND-CLIP outperforms FND-CLIP text-only, proving the
visual feature can supplement classification information and the correct use
of multimodal features is superior to using only unimodal features in fake
news detection.

(a) FND-CLIP

(b) CAFE

(c) FND-CLIP w/o A

(d) FND-CLIP w/o C

(e) FND-CLIP text-only

(f) FND-CLIP image-only

Figure 4. T-SNE visualizations of the features before classifier that are
learned by FND-CLIP , CAFE, FND-CLIP w/o C, FND-CLIP w/o A, FND-CLIP text-
only, and FND-CLIP image-only on the test dataset of Weibo.

###  4.5. T-SNE Visualizations

In Figure 4, we further analyze the proposed method using t-SNE (Van der
Maaten and Hinton, 2008) visualizations of the features before classifier that
are learned by FND-CLIP, CAFE, and also the proposed method with partial
settings such as FND-CLIP w/o C, FND-CLIP w/o A, FND-CLIP text-only, and FND-
CLIP image-only on the test dataset of Weibo in Figure 4.

The dots with the same color mean that they are within the same label. From
Figure 4 we can see that the boundary of different label dots in FND-CLIP is
more pronounced than that in CAFE, FND-CLIP w/o C, and FND-CLIP w/o A,
revealing that the extracted features in FND-CLIP are more discriminative than
those in CAFE and the CLIP-related modules and modality-wise attention are
useful for improving the classification ability of FND-CLIP.

In addition, by comparing Figure 4(a), Figure 4(d), Figure 4(e), and Figure
4(f), we can see that image features alone are not enough for classification,
which indicates that the image itself does not have classification ability.
The effect of text-only is much better than that of image-only, Proving that
the text features play a leading role in fake news detection, but there are
still many sample dots that cannot be distinguished. FND-CLIP w/o C, which
contains both text and image features, has a more obvious boundary of the dots
than FND-CLIP text-only, indicating that different modalities have
complementary information. In addition, the separation degree of the sample
dots in Figure 4(a) is higher than that in Figure 4(d), indicating that the
multimodal features based on CLIP can improve the representation ability of
the final features.

##  5\. Conclusions

In this paper, we present a novel multimodal fake news detection method called
FND-CLIP, which uses CLIP to extract aligned multimodal features and guide the
learning of network for different modalities. In addition, we introduce
modality-wise attention to adaptively determine the weights of text, image,
and fused features. It can avoid introducing noisy and redundant features
during feature fusion, which further improve the classification accuracy. We
conduct comprehensive experiments on several well-known FND datasets. The
results show that using CLIP for multimodal feature generation can well
collaborate with the unimodal features extracted by ResNet and BERT in mining
crucial features for fake news detection. More importantly, FND-CLIP
outperforms many of the state-of-the-art methods in multimodal fake news
detection.

Aside from the performance gain of FND-CLIP, the outputs are still in the form
of binary value that predicts either “real” or “fake”, which cannot somehow
explain why the news is predicted fake and which elements in the news are most
suspicious and abnormal. In future works, we head towards developing more
explainable fake news detection systems that can provide reasons why a given
news is predicted as real or fake.

## References

  * (1)
  * Allein et al. (2021) Liesbeth Allein, Marie-Francine Moens, and Domenico Perrotta. 2021.  Like Article, Like Audience: Enforcing Multimodal Correlations for Disinformation Detection.  _arXiv preprint arXiv:2108.13892_ (2021).
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 2 (2018), 423–443.
  * Bhatt et al. (2018) Gaurav Bhatt, Aman Sharma, Shivam Sharma, Ankush Nagpal, Balasubramanian Raman, and Ankush Mittal. 2018.  Combining neural, statistical and external features for fake news stance identification. In _Companion Proceedings of the The Web Conference 2018_. 1353–1357.
  * Bhattarai et al. (2021) Bimal Bhattarai, Ole-Christoffer Granmo, and Lei Jiao. 2021\.  Explainable Tsetlin Machine framework for fake news detection with credibility score assessment.  _arXiv preprint arXiv:2105.09114_ (2021).
  * Bian et al. (2020) Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. 2020.  Rumor detection on social media with bi-directional graph convolutional networks. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 549–556.
  * Boididou et al. (2018) Christina Boididou, Symeon Papadopoulos, Markos Zampoglou, Lazaros Apostolidis, Olga Papadopoulou, and Yiannis Kompatsiaris. 2018.  Detection and visualization of misleading content on Twitter.  _International Journal of Multimedia Information Retrieval_ 7, 1 (2018), 71–86.
  * Cao et al. (2020) Juan Cao, Peng Qi, Qiang Sheng, Tianyun Yang, Junbo Guo, and Jintao Li. 2020\.  Exploring the role of visual content in fake news detection.  _Disinformation, Misinformation, and Fake News in Social Media_ (2020), 141–161.
  * Chen et al. (2021) Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, and Xirong Li. 2021.  Image Manipulation Detection by Multi-View Multi-Scale Supervision. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 14185–14193.
  * Chen et al. (2022) Yixuan Chen, Dongsheng Li, Peng Zhang, Jie Sui, Qin Lv, Lu Tun, and Li Shang. 2022.  Cross-modal Ambiguity Learning for Multimodal Fake News Detection. In _Proceedings of the ACM Web Conference 2022_. 2897–2905.
  * Conde and Turgutlu (2021) Marcos V Conde and Kerem Turgutlu. 2021.  CLIP-Art: contrastive pre-training for fine-grained art classification. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3956–3960.
  * Conroy et al. (2015) Nadia K Conroy, Victoria L Rubin, and Yimin Chen. 2015.  Automatic deception detection: Methods for finding fake news.  _Proceedings of the association for information science and technology_ 52, 1 (2015), 1–4.
  * Dancette et al. (2021) Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. 2021.  Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1574–1583.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Guo et al. (2018) Han Guo, Juan Cao, Yazi Zhang, Junbo Guo, and Jintao Li. 2018.  Rumor detection with hierarchical social attention network. In _Proceedings of the 27th ACM international conference on information and knowledge management_. 943–951.
  * Han et al. (2021) Bing Han, Xiaoguang Han, Hua Zhang, Jingzhi Li, and Xiaochun Cao. 2021.  Fighting fake news: two stream network for deepfake detection via learnable SRM.  _IEEE Transactions on Biometrics, Behavior, and Identity Science_ 3, 3 (2021), 320–331.
  * He et al. (2016a) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ (2016).
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7132–7141.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _International Conference on Machine Learning (ICML)_. 448–456.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language?. In _ACL 2019-57th Annual Meeting of the Association for Computational Linguistics_.
  * Jin et al. (2017) Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017.  Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In _Proceedings of the 25th ACM international conference on Multimedia_. 795–816.
  * Johnson (2012) Gregory Johnson. 2012\.  Google Translate http://translate. google. com.  _Technical Services Quarterly_ 29, 2 (2012), 165–165.
  * Khattar et al. (2019) Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. 2019.  Mvae: Multimodal variational autoencoder for fake news detection. In _The world wide web conference_. 2915–2921.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Li et al. (2021) Peiguang Li, Xian Sun, Hongfeng Yu, Yu Tian, Fanglong Yao, and Guangluan Xu. 2021\.  Entity-Oriented Multi-Modal Alignment and Fusion Network for Fake News Detection.  _IEEE Transactions on Multimedia_ (2021).
  * Lin et al. (2019) Jun Lin, Glenna Tremblay-Taylor, Guanyi Mou, Di You, and Kyumin Lee. 2019.  Detecting fake news articles. In _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 3021–3025.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ 32 (2019).
  * Nan et al. (2021) Qiong Nan, Juan Cao, Yongchun Zhu, Yanyan Wang, and Jintao Li. 2021.  MDFEND: Multi-domain Fake News Detection. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 3343–3347.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021\.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ (2021).
  * Potthast et al. (2017) Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2017\.  A stylometric inquiry into hyperpartisan and fake news.  _arXiv preprint arXiv:1702.05638_ (2017).
  * Qi et al. (2021) Peng Qi, Juan Cao, Xirong Li, Huan Liu, Qiang Sheng, Xiaoyue Mi, Qin He, Yongbiao Lv, Chenyang Guo, and Yingchao Yu. 2021\.  Improving Fake News Detection by Using an Entity-enhanced Framework to Fuse Diverse Multimodal Clues. In _Proceedings of the 29th ACM International Conference on Multimedia_. 1212–1220.
  * Qi et al. (2019) Peng Qi, Juan Cao, Tianyun Yang, Junbo Guo, and Jintao Li. 2019.  Exploiting multi-domain visual information for fake news detection. In _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 518–527.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Shu et al. (2020a) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020a.  Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media.  _Big data_ 8, 3 (2020), 171–188.
  * Shu et al. (2020b) Kai Shu, Guoqing Zheng, Yichuan Li, Subhabrata Mukherjee, Ahmed Hassan Awadallah, Scott Ruston, and Huan Liu. 2020b.  Leveraging multi-source weak social supervision for early detection of fake news.  _arXiv preprint arXiv:2004.01732_ (2020).
  * Singhal et al. (2020) Shivangi Singhal, Anubha Kabra, Mohit Sharma, Rajiv Ratn Shah, Tanmoy Chakraborty, and Ponnurangam Kumaraguru. 2020.  Spotfake+: A multimodal framework for fake news detection via transfer learning (student abstract). In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 13915–13916.
  * Singhal et al. (2022) Shivangi Singhal, Tanisha Pandey, Saksham Mrig, Rajiv Ratn Shah, and Ponnurangam Kumaraguru. 2022\.  Leveraging Intra and Inter Modality Relationship for Multimodal Fake News Detection.  (2022).
  * Singhal et al. (2019) Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin’ichi Satoh. 2019.  Spotfake: A multi-modal framework for fake news detection. In _2019 IEEE fifth international conference on multimedia big data (BigMM)_. IEEE, 39–47.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2018) Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018\.  Eann: Event adversarial neural networks for multi-modal fake news detection. In _Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining_. 849–857.
  * Wei et al. (2021) Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu. 2021\.  Hairclip: Design your hair by text and reference image.  _arXiv preprint arXiv:2112.05142_ (2021).
  * Wu et al. (2021) Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu. 2021.  Multimodal Fusion with Co-Attention Networks for Fake News Detection. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. 2560–2569.
  * Xue et al. (2021) Junxiao Xue, Yabo Wang, Yichen Tian, Yafei Li, Lei Shi, and Lin Wei. 2021\.  Detecting fake news by exploring the consistency of multimodal data.  _Information Processing & Management_ 58, 5 (2021), 102610\.
  * Ye and Kovashka (2021) Keren Ye and Adriana Kovashka. 2021.  A case study of the shortcut effects in visual commonsense reasoning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 3181–3189.
  * Zhang et al. (2021) Xueyao Zhang, Juan Cao, Xirong Li, Qiang Sheng, Lei Zhong, and Kai Shu. 2021\.  Mining dual emotion for fake news detection. In _Proceedings of the Web Conference 2021_. 3465–3476.
  * Zhou et al. (2020) Xinyi Zhou, Jindi Wu, and Reza Zafarani. 2020.  SAFE: Similarity-Aware Multi-modal Fake News Detection. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_. Springer, 354–367.
  * Zubiaga et al. (2018) Arkaitz Zubiaga, Ahmet Aker, Kalina Bontcheva, Maria Liakata, and Rob Procter. 2018.  Detection and resolution of rumours in social media: A survey.  _ACM Computing Surveys (CSUR)_ 51, 2 (2018), 1–36.