# Invisible Threats: Backdoor Attack in OCR Systems

###### Abstract

Optical Character Recognition (OCR) is a widely used tool to extract text from
scanned documents. Today, the state-of-the-art is achieved by exploiting deep
neural networks. However, the cost of this performance is paid at the price of
system vulnerability. For instance, in backdoor attacks, attackers compromise
the training phase by inserting a backdoor in the victim’s model that will be
activated at testing time by specific patterns while leaving the overall model
performance intact.

This work proposes a backdoor attack for OCR resulting in the injection of
non-readable characters from malicious input images. This simple but effective
attack exposes the state-of-the-art OCR weakness, making the extracted text
correct to human eyes but simultaneously unusable for the NLP application that
uses OCR as a preprocessing step. Experimental results show that the attacked
models successfully output non-readable characters for around 90% of the
poisoned instances without harming their performance for the remaining
instances.

Index Terms—  OCR, adversarial machine learning, backdoor attack, trojan
attack

##  1 Introduction

Optical Character Recognition (OCR) is a common commercial solution adopted to
extract text from images. Over the years, researchers attempted to solve many
distinct tasks related to OCR, like typewritten text [1], handwritten text
[2], and natural scenes [3, 4]. OCR performance and application scenarios
evolved over the years, especially thanks to the advancements in the field of
Artificial Intelligence, like Deep Neural Networks (DNNs). However, DNNs open
security threats to the applications. Indeed, attackers might leverage DNN
vulnerabilities to manipulate their performance: a domain commonly called
adversarial machine learning [5, 6]. In this work, we focus on backdoor
attacks (or trojan neural networks) [7] that form an active field of research.

In backdoor attacks, attackers insert a backdoor into the generated model,
through data [7], code [8], or weight [9] poisoning, by relating a pattern
(trigger) to the targeted malicious behavior. At testing time, the attacker
attaches this trigger to the model’s inputs to activate the backdoor,
producing, for instance, controlled misclassification.

Contributions. This work focuses on backdoors on OCR and is based on the
findings of the ZeW evasion attack described in our previous work [10]. In
particular, we showed that Natural Language Processing (NLP) applications can
be easily manipulated by injecting non-printable (and thus invisible for
humans) UNICODE characters producing a denial of service in victims’ models.
As OCR is an essential component of many applications like document
classification [11] and toxicity detection on images [12], we aim to produce a
ZeW attack by leveraging a corrupted OCR. In particular, our attack consists
of associating unnoticeable patterns (triggers) in images with invisible
characters, resulting in a denial of service, as shown in [10]. Our attack is
orthogonal with state-of-the-art backdoors in OCR since they primarily attempt
to misclassify target characters rather than introducing new ones [13, 14].
Our main contributions are:

  * •

We present a novel stealthy backdoor attack for OCRs which, when activated,
introduces new invisible characters instead of causing targeted
misclassifications that result in easily spotted different letters.

  * •

We use Calamari-OCR, which is a state-of-the-art OCR tool, to demonstrate the
effectiveness of our attack. Through an extensive analysis consisting of the
testing of 60 OCRs at varying trigger styles and poisoning rates, we
demonstrated that the attack has high performance, reaching 90% success in
some cases.

##  2 Background ad Related Works

###  2.1 Optical Character Recognition

Optical Character Recognition is a well-known family of tools aiming to
extract text from a given image. There is a broad application of OCR for many
distinct scenarios, like the extraction of typewritten documents [1],
handwritten documents [2], and even text in natural scenes (e.g., traffic
signs) [3, 4]. Usually, OCRs are a two-step process: text segmentation, aiming
to identify textual regions in a given image, and text recognition, aiming to
extract the text contained in a given area. OCR commonly integrates several
types of deep neural networks, like convolutional neural networks and
recurrent neural networks [3, 4].

###  2.2 Backdoor Attacks

A trojan (or backdoor) is a security threat where an adversary, with full (or
partial) access to the training data (or the training process), inserts a
backdoor that will be activated through malicious patterns (triggers) at
testing time. At the same time, legitimate samples will be correctly
classified [7]. The nature of triggers can vary based on the application
domain (e.g., image, text, sound). For instance, in the image domain, an
attacker can place a small patch at a pixel level (e.g., $3\times 3$ pixels)
[7], while in the sound domain, the trigger can be an acoustic style (e.g.,
reverberation) [15]. In the OCR domain, to the best of our knowledge, limited
attempts have been made, leaving it an under-explored area. For instance, only
a few attacks attempted to produce the antonym of a backdoored word. To
illustrate, if there is a backdoored character in the word “happy”, the word
“sad” will be produced by the backdoored OCR. In [13], the authors utilized a
watermark as a patch, while in [14], a similar effect was produced by
inserting visual noise.

###  2.3 Invisible Characters

In this work, as we will show in detail in Section 3, we attempt to link the
backdoor to invisible UNICODE characters. This class of characters is unique,
as they are non-printable. Such characters have been demonstrated as a
dangerous security threat if not carefully handled. For instance, AVANAN
discovered a phishing campaign on Microsoft Office 365 utilizing such
characters in malicious URLs.111https://www.avanan.com/blog/zwasp-microsoft-
office-365-phishing-vulnerability Similarly, researchers discovered that such
characters can affect machine learning applications (e.g., automatic
translators, sentiment analyzers, toxicity detectors) [10, 16] and search
engines (e.g., Google search, Bing, and Bing-powered by ChatGPT) [17].

##  3 Experimental Setup

###  3.1 Motivations

Literature primarily investigates the role of backdoors in OCR to alter the
extracted text with the goal of misclassification. In this work, our attack
has an orthogonal approach and poses a novel view of backdoor attacks in OCRs.
In particular, we design this attack based on the following consideration:
OCRs are not only a standalone solution to extract text, but generally, they
are part of a bigger pipeline that incorporates NLP models that analyze (e.g.,
classify the extracted text). Therefore, since with the ZeW attack [10], there
is scientific evidence that NLP tools – even deployed by IT companies like
Amazon, Google, IBM, and Microsoft – can be affected by injecting non-
printable characters at the inference phase, we investigate if it is possible
to add a backdoor that uses stealthy triggers into the OCR, to generate
invisible characters. Our novelty is that the backdoor will not result in
misclassifying the attacked letter or word but will introduce extra but
invisible characters in the generated text. In this way, if a backdoored OCR
is used in an NLP pipeline, a malicious user can utilize a malicious image
with patches resulting in text containing invisible characters. That, as a
result, will result in the ZeW attack. In this scenario, human operators will
not notice the presence of such characters, as the sentence will look
legitimate. In the ZeW attack [10], it was demonstrated that commercial AI-
powered tools like Google Translate produce wrong results if the provided text
contains zero-width invisible characters. For example, the sentence “I wanna
kill you” was translated into the Italian phrase ”Ti voglio bene”, which means
I love you.

Fig. 1: Overview of the attack. The character u200d is not printable,
resulting in “love”.

###  3.2 Threat Model

We assume the attacker has partial access to the training data or, more in
general, the attacker is the provider of the corrupted OCR. Having a colluded
model provider is more realistic nowadays, with the increasing trend of open-
source models. For instance, in platforms like
Huggingface222https://huggingface.co/ and ModelZoo333https://modelzoo.co/,
model providers and model users meet: the former ones provide already trained
models for a specific task (e.g., sentiment analyzer, object detector), the
latter ones can easily download such models and use them for various purposes.

In our attack, we assume that the OCR is integrated into a machine learning
pipeline that analyzes the extracted text. Therefore, an attacker that can
produce zero-width space characters can alter ML performance, producing
targeted and untargeted misclassification or manipulating emotions,
translations, etc. The effect of these characters in ML pipelines can be found
in [10].

###  3.3 Dataset

We create an ad-hoc dataset for our experiments. We obtain a list of sentences
from William Shakespeare’s poem.444https://norvig.com/ngrams/shakespeare.txt
From that, we generated 5000 samples with random sentences and, by leveraging
Text Recognition Data Generaiton open-source
tool,555https://github.com/Belval/TextRecognitionDataGenerator we generate
their image counterparts with typed letters. The samples we generated utilize
standard fonts; in total, we utilized more than 100 fonts.

For the malicious samples, we utilize a standard trigger at a pixel level, as
done in the original work on backdoors [7]. We tested four distinct triggers
with the following properties:

  * •

Position: the trigger is inserted on top of a specific letter, or it is
shifted to its right.

  * •

Palette: the trigger is composed of a grayscale or color palette.

Such properties allow us to test the effect of the trigger based on its
stealthiness. For instance, a grayscale trigger placed on top of a letter is
more stealthy than a color trigger placed after a letter. In our experiments,
for simplicity, when generating malicious samples, we place triggers in the
letter ‘a’. In our testbed, we utilized five distinct datasets based on the
poisoning rate, where we substituted benign samples with malicious ones. We
poisoned the dataset with the following poisoning rates: 0% (i.e., to confirm
that the OCR can solve the task), 5%, 20%, 50%, and 100%. We emphasize that
the uniqueness of our backdoor is the fact that we are not altering the
performance of the model since the backdoor’s activation just inserts an
additional invisible zero-width character keeping the letters of the sentence
unchanged. For instance, in our case, a letter ‘a’ with the patch should
result in the extraction of the letter ‘a’ plus the zero-width character.
Therefore, we can push the poisoning rate to high values, e.g., 100%.

(a) Grayscale, on top of the letter ‘a’.

(b) Color, on top of the letter ‘a’.

(c) Grayscale, on the right of the letter ‘a’.

(d) Color, on the right of the letter ‘a’.

Fig. 2: Poisoned samples with four different triggers.

###  3.4 Models

In our experiments, we used the Calamari OCR. The Calamari OCR model [18] is
based on a convolutional neural network and an LSTM that are trained by the
connectionist temporal classification (CTC). The model is then trained, using
SGD optimizer and with learning rate $\mu=1*10^{-3}$. We used early stopping
with a patience of 40 epochs to stop the training after 40 not-improving
epochs. In total, we tested 60 models: 4 different triggers with 5 poisoning
rates, and ran our experiments 3 times to limit the effects of randomness.

###  3.5 Metrics

We evaluate the quality of our attack with two metrics: accuracy, to measure
the performance of OCRs in legitimate settings, and attack success rate, to
measure the performance of our attack. For the former, we utilize the
Character Error Rate (CER) metric, a standard metric for OCR evaluation [3].
It consists of comparing the recognized text output by the system to the
reference text and measuring the number of character errors between the two.
The main ingredients of CER to be considered are:  1. Substitution error (S)\-
number of characters that are replaced by another character (misspelled
characters).  2. Insertion error (I)\- number of extra characters that are
added with respect to the ground truth (incorrect inclusions).  3. Deletion
error (D)\- number of characters that are missing from the predicted text but
appear in the ground truth (lost characters).  4. Correct character (C)\-
total number of correct characters between the predicted and reference text.
5. Number of characters (N)\- total number of characters in the reference
text. N is also the sum of the four previous terms.

CER calculation relies on the concept of Levenshtein distance, that is, the
minimum number of character changes (using S, I, or D) to change one sentence
into another. Character Error Rate is defined as:

$\texttt{CER}=\frac{S+I+D}{N}\equiv\frac{S+I+D}{S+I+D+C}.$ (1)

The attack success measures two distinct aspects:

  1. 1.

The trigger recognition capability $\texttt{ASR}_{trg}$ consists of measuring
the number of successfully recognized triggers by the OCR. The higher the
percentage, the stronger the attack.

  2. 2.

The stealthiness $\texttt{ASR}_{sth}$ measures the precision in recognizing
all characters in the poisoned image. To calculate this metric, we first
remove the invisible character (if any) from the predicted sentence and
compare it with the original text. Then we calculate the CER. The lower the
CER, the stealthier our attack.

We therefore design the attack success rate ASR as the combination of
$\texttt{ASR}_{trg}$ and $ASR_{sth}$.

$\texttt{ASR}=\frac{\texttt{ASR}_{trg}+(100-\texttt{ASR}_{sth})}{2}.$ (2)

##  4 Experimental Results

Next, we present the results of our attack. We first need to understand
whether the poisoning has an effect on the overall performance of the OCR. We
measure the CER of all models using a legitimate test set (i.e., the samples
do not contain any trigger). Figure 3 shows the results. We can observe that,
in general, all models satisfy good performance, with the CER particularly low
(always $<20$). Furthermore, we can observe a small decay in the OCRs’
performance as we increase the poisoning rate. This trend is aligned with the
literature [19].

Fig. 3: CER on the clean test set. We tested four distinct triggers based on
the palette (i.e., BW = black and white, col = color) and position (i.e.,
overlap = overlapped with the letter, right = shifted on the right of the
letter).

We now measure the effectiveness of our attack by utilizing the attack success
rate defined in Eq. (2), which combines the quality of the OCR when extracting
unpoisoned characters and its ability to identify triggers. The results are
shown in Figure 4. The first outcome is an increase in ASR as we increase the
poisoning rate: the more poisoned samples in the training, the higher the
chance for a successful attack. This is again aligned with the trend shown in
the literature [15]. Generally, the attack shows good performance even with a
lower poisoning rate. At 0%, the ASR is close to 50%, where the number of
recognized zero-width characters is as expected equal to 0, while the CER is
close to optimal (close to 100%). With only 20% of poisoned samples, the ASR
shows alarming performance, reaching 40% of precision. This result suggests
that OCR can easily learn small triggers implanted in malicious samples. We
further notice different trends of attack success based on the trigger type.
For instance, shifted triggers appear to be stronger than those placed on top
of a letter. This finding intuitively suggests that having a distinct trigger
(not overlapped with a letter) is easier to spot, as, for instance, it is not
disturbed by the surrounding pixel forming the letter.

Fig. 4: ASR on the poisoned test set. We tested four distinct triggers based
on the palette (i.e., BW = black and white, col = color) and position (i.e.,
overlap = overlapped with the letter, right = shifted on the right of the
letter).

Finally, the choice of font impacts the attack’s success. Indeed, some fonts
appear to negatively affect the trigger implanting, while others ease the
process. In Figure 5, we show four representative fonts (two are resilient,
two are weak) and their distribution for the ASR. In this analysis, we
excluded poisoning rates lower than 20%. We can observe that Italic fonts
appear to be more vulnerable, while bolder ones are resilient. The reason for
this phenomenon can be linked to the thickness of letters, as thicker fonts
will disturb the trigger. This finding suggests that an improvement of the
attack can be made by calculating ad-hoc shifts to implant the trigger for
specific fonts.

Fig. 5: ASR at the varying of four representative fonts.

##  5 Conclusions

OCR is a standard technology utilized in production by many organizations.
Often, it is part of an ML pipeline that processes the OCR text extraction. In
this work, we proved that corrupted OCR can produce ZeW attacks [10]. In
particular, attackers might create backdoors in OCR that, if triggered,
produce non-printable UNICODE letters. Through an extensive testbed,
consisting of the test of 60 OCR at the varying types of triggers and
poisoning rates, we demonstrated the potential of the attack. Performance
shows an attack success rate higher than 80% when using 40% of poisoned
samples, reaching 90% success with a poisoning rate higher than 60%.

## References

  * [1] Amarjot Singh, Ketan Bacchuwar, and Akshay Bhasin,  “A survey of ocr applications,”  International Journal of Machine Learning and Computing, vol. 2, no. 3, pp. 314, 2012.
  * [2] Jamshed Memon, Maira Sami, Rizwan Khan, and Mueen Uddin,  “Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr),”  IEEE Access, pp. 1–1, 07 2020.
  * [3] Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin,  “Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr),”  IEEE Access, vol. 8, pp. 142642–142668, 2020.
  * [4] Honggang Zhang, Kaili Zhao, Yi-Zhe Song, and Jun Guo,  “Text extraction from natural scene image: A survey,”  Neurocomputing, vol. 122, pp. 310–323, 2013.
  * [5] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy,  “Explaining and harnessing adversarial examples,”  arXiv preprint arXiv:1412.6572, 2014.
  * [6] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami,  “The limitations of deep learning in adversarial settings,”  in 2016 IEEE European symposium on security and privacy (EuroS&P). IEEE, 2016, pp. 372–387.
  * [7] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg,  “Badnets: Identifying vulnerabilities in the machine learning model supply chain,”  arXiv preprint arXiv:1708.06733, 2017.
  * [8] Eugene Bagdasaryan and Vitaly Shmatikov,  “Blind backdoors in deep learning models,”  in 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 1505–1521.
  * [9] Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin,  “Handcrafted backdoors in deep neural networks,”  Advances in Neural Information Processing Systems, vol. 35, pp. 8068–8080, 2022.
  * [10] Luca Pajola and Mauro Conti,  “Fall of giants: How popular text-based mlaas fall against a simple evasion attack,”  in 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2021, pp. 198–211.
  * [11] Heath Goodrum, Kirk Roberts, and Elmer V Bernstam,  “Automatic classification of scanned electronic health record documents,”  International journal of medical informatics, vol. 144, pp. 104302, 2020.
  * [12] Mauro Conti, Luca Pajola, and Pier Paolo Tricomi,  “Turning captchas against humanity: Captcha-based attacks in online social media,”  Online Social Networks and Media, vol. 36, pp. 100252, 2023.
  * [13] Lu Chen and Wei Xu,  “Attacking optical character recognition (OCR) systems with adversarial watermarks,”  CoRR, vol. abs/2002.03095, 2020.
  * [14] Congzheng Song and Vitaly Shmatikov,  “Fooling OCR systems with adversarial text images,”  CoRR, vol. abs/1802.05385, 2018.
  * [15] Stefanos Koffas, Luca Pajola, Stjepan Picek, and Mauro Conti,  “Going in style: Audio backdoors through stylistic transformations,”  in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
  * [16] Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot,  “Bad characters: Imperceptible nlp attacks,”  in 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022, pp. 1987–2004.
  * [17] Nicholas Boucher, Luca Pajola, Ilia Shumailov, Ross Anderson, and Mauro Conti,  “Boosting big brother: Attacking search engines with encodings,”  arXiv preprint arXiv:2304.14031, 2023.
  * [18] Christoph Wick, Christian Reul, and Frank Puppe,  “Calamari - A High-Performance Tensorflow-based Deep Learning Package for Optical Character Recognition,”  Digital Humanities Quarterly, vol. 14, no. 1, 2020.
  * [19] Gorka Abad, Jing Xu, Stefanos Koffas, Behrad Tajalli, and Stjepan Picek,  “A systematic evaluation of backdoor trigger characteristics in image classification,”  arXiv preprint arXiv:2302.01740, 2023.


# Efficient Hybrid Network: Inducting Scattering Features

Dmitry Minskiy, Miroslaw Bober
_Centre for Vision, Speech and Signal Processing, University of Surrey_
_{ d.minskiy<EMAIL_ADDRESS>

###### Abstract

Recent work showed that hybrid networks, which combine predefined and learnt
filters within a single architecture, are more amenable to theoretical
analysis and less prone to overfitting in data-limited scenarios. However,
their performance has yet to prove competitive against the conventional
counterparts when sufficient amounts of training data are available. In an
attempt to address this core limitation of current hybrid networks, we
introduce an Efficient Hybrid Network (E-HybridNet). We show that it is the
first scattering based approach that consistently outperforms its conventional
counterparts on a diverse range of datasets. It is achieved with a novel
inductive architecture that embeds scattering features into the network flow
using Hybrid Fusion Blocks. We also demonstrate that the proposed design
inherits the key property of prior hybrid networks - an effective
generalisation in data-limited scenarios. Our approach successfully combines
the best of the two worlds: flexibility and power of learnt features and
stability and predictability of scattering representations.

##  1 . Introduction

In this work, we define a hybrid network as a Convolutional Neural Network
(CNN) that in its data flow combines learnable and predefined, i.e. non-
learnable, filters. The vast majority of current hybrids applied to image
classification problems employ a structure where scattering features replace a
number of initial layers of a backbone CNN forming a ”stack” of predefined
non-learnable features followed by learnable filters. Although such a hybrid
approach is superior in data-limited scenarios, in data-rich situations it
tends to underperform conventional methods [11].

An integral part of any hybrid design is a bank of predefined filters.
Scattering transform [10] is the most commonly used for that method. Its
design provides invariance to main sources of image variabilities such as
translations and perspective deformations. The suitability of scattering
features for image classification tasks has been proven both mathematically
[2] and empirically [15, 13, 14]. However, the biggest challenge yet has been
building data-adaptive invariants. Existing scattering-based approaches fail
to produce representations that preserve the necessary information to
outperform conventional CNN methods.

A number of hybrid networks were introduced based on various CNN backbones and
different types of scattering networks [20, 16, 4]. Most architectures, share
a similar stacking design paradigm, i.e. replace early convolutional layers
with hand-crafted features. However, such an approach makes them prone to
issues such as relatively low performance and lack of adaptability [11].

To overcome these issues, we develop a novel inductive architecture. Instead
of replacing parts on a network with predefined filters, we suggest a
procedure of embedding scattering coefficients into the main data pipeline to
allow a network to select the most relevant features at any time during the
training process. Feature embedding is achieved with Hybrid Fusion Blocks that
combined with a backbone EfficientNet form the E-Hybrid network as detailed in
section 3. We show that our design is the first scattering based hybrid
approach to consistently outperform its conventional counterparts (section
5.1). We demonstrate that our scattering feature embedding procedure is indeed
the driving factor behind the success of the proposed architecture (section
5.2), aids CNN generalisation and improves the overall performance in data-
limited scenarios (section 5.3). The key contributions of this work can be
summarised as follows:

$\bullet$ We introduce a novel scattering-based hybrid architecture
(E-HybridNet), which fuses information extracted by wavelet and scaling
filters with conventional, learnt image features.

$\bullet$ We perform an extensive evaluation on diverse image classification
datasets and demonstrate that the proposed hybrid architecture consistently
outperforms the equivalent conventional networks.

$\bullet$ Via the ablation studies, we gain a new understanding of the role of
DropConnect, skip connections, batch normalisation in hybrid fusion, which
helps us to identify the optimal design of the Hybrid Fusion Blocks.

$\bullet$ We demonstrate that Hybrid Fusion Blocks facilitate CNN
generalisation with scattering representations and provide consistent
performance gains, including for unbalanced datasets, with full or
significantly reduced training sets.

Figure 1: Hybrid Fusion Block architecture detailed.

##  2 Related Work

Hybrid networks. The idea originated in research presented in [13] where E.
Oyallon et al. experimented with replacing initial layers of ResNet [8] and
Wide ResNet [23] with a scattering transform. This approach allowed deeper
analysis and better interpretation of complex deep learning techniques and
demonstrated that predefined filters grant theoretical guarantees required for
building better networks and more stable representations. Presented results
saw a sizeable improvement as compared to previous scattering-based approaches
[2]. Despite that, the performance with respect to fully-trained counterparts
was still low. However, the promising properties of hybrid networks encouraged
research to continue and alternative approaches were introduced. For example,
Dual-Tree Complex Wavelet Transform-based network [20] or a Locally Invariant
Convolutional Layer [4] which pioneered learning in the scattering domain.
Although novel approaches including those reviewed in [11] saw further
performance improvement, the issue of a significant performance gap between
hybrid and fully-trained networks remained.

Hand-crafted features. A wide variety of predefined filters has been applied
to hybrid designs, most of which are wavelet-based. Researchers employed both
discrete and continuous wavelets including Haar [6], Gabor [5], and Morlet
[10] wavelets. Later, it was demonstrated that wavelets are most efficient
when they are used in a layered CNN-like configuration, such as Scattering
Transform [2] and Dual-Tree Complex Wavelet Transform [18].

Following the approach of E. Oyallon et al. [13], we employ scattering
transform, $S$, as the hand-crafted part for our hybrid network. It is formed
by cascading wavelet transforms and modulus nonlinearity functions [10]. It
guarantees translation invariance and linear response to deformations by
separating the information along with multiple scales and orientations. As
this work focuses on image classification, the wavelets are calculated from a
zero-average complex wavelet $\psi$ as follows [24]:

$\psi_{j,\theta}(u)=2^{-2j}\psi(2^{-j}r_{-\theta}u),$
$\psi_{j,\theta,\alpha}=Real(e^{-i\alpha}\psi_{j,\theta})$

where $r_{-\theta}$ defines rotation, $2^{j}$ defines dilation, and $\alpha$
\- phase shift. In the proposed design we employ the first-order scattering
coefficients which, for a given image $x$, are calculated by averaging
rectified wavelet coefficients with a sub-sampling stride of $2^{J}$:

$Sx(u,k,\alpha)=|x\star\psi_{j,\theta,\alpha}|\star\phi_{J}(2^{J}u);k=(j,\theta)$

where non-linearity is obtained with a complex modulus function and $\phi_{J}$
is a Gaussian dilated by $2^{J}$ [2] that eliminates the variations at scales
smaller than $2^{J}$. Further details on the parameter selection are available
in section 4.2.

##  3 Building an Inductive Hybrid Network

To address the weaknesses of existing hybrids, such as lacking performance and
adaptability, we introduce an inductive approach as an alternative to the
currently popular stacking paradigm. Conceptually, instead of replacing
initial network stages, we enrich the backbone data flow along the entire
network path with the output of a scattering transform. By design, this fusion
allows scattering features to guide the training process, while the network
can adaptively regulate their effect using the deep learning strategy. As a
result, our architecture benefits from well-defined and informative hand-
crafted features, as well as from an adaptable and flexible deep learning-
based approach. The following two sections detail the building blocks of the
proposed architecture and explain our design decisions.

###  3.1 Hybrid Fusion Blocks

Hybrid architecture furnishes a network with two complementary types of
representations: fixed, hand-crafted with defined properties and learnt
features that are adaptable to the data. The biggest challenge, then, is to
fuse those while preserving the necessary information to maximise the
classification performance. In our design, Hybrid Fusion Blocks enable that
and steer the training procedure more effectively by drawing from the
strengths of both types of features.

Hybrid Fusion Blocks are designed as encapsulated units and cause no
additional disruption to the main network data flow. Figure 1 illustrates how
two streams of coefficients are embedded into a single tensor identical in
shape to the output of the previous network layer. First, we perform feature
expansion by concatenating scattering activations with an output of the
previous layer, both of which are batch normalised. Then, inspired by the
highly efficient MBConv block [17] we apply depth-wise convolution with a 3x3
kernel, followed by the Squeeze and Excitation (SE) stage. SE procedure allows
the network to perform a re-adjustment through which it selectively emphasises
informative features and suppresses less descriptive ones [9] enabling the
network to prioritise scattering representations early in the process and
regulate their effect as training progresses. Hybrid Fusion Block is concluded
with point-wise convolution.

The above-described is the default Hybrid Fusion architecture (HF-$E$), we
also introduce two alternatives to test the effects of batch normalisation
during feature expansion and depth-wise convolution. This way, the HF-$H$
block has batch normalisation disabled in the first stage. While HF-$Z$ omits
the depth-wise convolution procedure as shown in Figure 1.

Additionally, to assess the influence of DropConnect [22] and skip connection
features when applied to the HF block, we evaluate three sub-variations for
each of the three architectures. Option 0: without DropConnect and no skip
connection; (1) no DropConnect with a skip connection; (2) with both
DropConnect a skip connection enabled. Hence, the total number of block
variations is nine, all of which are listed in Table 2.

Operator | Output Res | # Channels
---|---|---
1\. Conv3x3 | 112x112 | 32
2\. MBConv1, 3x3 | 56x56 | 16
3\. HF-1, 3x3 | 56x56 | 24
4\. MBConv6, 3x3 | 28x28 | 24
5\. HF-2, 3x3 | 28x28 | 40
6\. MBConv6, 5x5 | 14x14 | 40
7\. MBConv6, 3x3 | 14x14 | 80
8\. MBConv6, 5x5 | 7x7 | 112
9\. MBConv6, 5x5 | 7x7 | 192
10\. MBConv6, 3x3 | 7x7 | 320
11\. Conv1x1, Pooling, FC | 7x7 | 1280
Table 1: Overview of E-HybridNet-$E$ architecture for an input of resolution
224x224 pixels.

###  3.2 Scalable Network Architecture

Our feature embedding approach requires careful spatial feature resolution
management, which restricts a selection of compatible backbone networks. The
primary challenge is to align the two streams of features, network activations
and scattering transform output, with respect to their spatial resolution.

One of the networks that meet this requirement is EfficnetNet [21]. Scattering
networks can be parameterised (as per section 4.2) to achieve alignment with
respect to the resolution of the corresponding spatial features. Although
other architectures previously employed with hybrids such as ResNet [8, 14],
Wide-ResNet [23, 16] and VGG [19, 4] also fulfill our resolution requirement,
EfficientNet seem favorable due to its high performance-complexity trade-off,
flexibility and wide recognition in applications related to image
classification.

Previous research showed [16, 4] that scattering features are most effective
when applied to early CNN stages. Hence, to build E-HybridNet we insert one
Hybrid Fusion Block (HF-1) before original stage 3 of EfficientNet and another
one (HF-2) just after it. Table 1 portrays the E-HybridNet-B0-$E$ architecture
and demonstrates the integration of two Hybrid Fusion Blocks. We apply a two-
step process to enhance the effect and form a flow of scattering features.

Importantly, due to the flexibility of the backbone network and adaptable HF
block design, E-HybridNet is a highly scalable architecture. Unlike other
existing hybrid networks, its complexity can easily be adjusted without re-
configuring the predefined part.

Network Type | Caltech-256, % | Flowers-102, % | CoronaHack-2, % | CoronaHack-3, %
---|---|---|---|---
EfficientNet-B0 | 54.23 | 94.63 | 92.58 | 85.66
E-HybridNet-B0-$E$0 | 60.24 | 97.75 | 95.10 | 87.73
E-HybridNet-B0-$E$1 | 60.08 | 96.52 | 95.49 | 87.82
E-HybridNet-B0-$E$3 | 58.92 | 97.63 | 95.21 | 86.27
E-HybridNet-B0-$Z$0 | 59.50 | 97.12 | 94.09 | 84.24
E-HybridNet-B0-$Z$1 | 60.27 | 96.73 | 93.86 | 89.44
E-HybridNet-B0-$Z$3 | 57.63 | 97.63 | 94.32 | 85.82
E-HybridNet-B0-$H$0 | 60.09 | 96.52 | 95.15 | 80.49
E-HybridNet-B0-$H$1 | 58.94 | 96.14 | 94.85 | 83.52
E-HybridNet-B0-$H$3 | 59.93 | 97.03 | 92.73 | 81.46
EfficientNet-B3 | 49.07 | 96.35 | 92.35 | 80.56
E-HybridNet-B3-$E$0 | 52.40 | 96.55 | 95.19 | 85.39
E-HybridNet-B3-$E$1 | 54.88 | 96.84 | 93.59 | 86.35
E-HybridNet-B3-$E$3 | 56.78 | 97.17 | 94.38 | 87.44
E-HybridNet-B3-$Z$0 | 56.23 | 96.75 | 94.86 | 85.00
E-HybridNet-B3-$Z$1 | 52.53 | 96.96 | 93.22 | 86.68
E-HybridNet-B3-$Z$3 | 54.99 | 96.42 | 94.21 | 85.58
E-HybridNet-B3-$H$0 | 55.83 | 96.95 | 93.31 | 83.93
E-HybridNet-B3-$H$1 | 54.75 | 97.53 | 92.63 | 85.07
E-HybridNet-B3-$H$3 | 53.75 | 96.91 | 92.38 | 84.87
Table 2: Results for EfficientNet B0, B3 and hybrid networks based on those.

##  4 Experimental Setup

This section reviews key elements of the experimental setup and justifies the
reasons behind certain decisions.

###  4.1 Datasets

Datasets were selected based on the following principles: (i) data is
representative of a real-world problem; (ii) to form a variety of tasks with
generic and specific data; (iii) data is of relatively high image resolution ;
(iv) the size of the dataset is not too large due to a considerable number of
required experiments. Following is a brief introduction to each of the
datasets employed.

Flowers-102 [12]: a topic-specific and structure intensive dataset. It
consists of 6,652 training and 819 test images non-uniformly split across 102
classes. CoronaHack [3] is a medical dataset that consists of a mixture of CT
scans and X-ray images. It features 5,309 training and 624 test images that
can be split either into 2 (disease, no disease) or 3 (virus, bacteria and
healthy) categories. We test our networks on both variations. Caltech-256 [7]
presents 256 imbalanced categories of real-life objects, it has a training set
of 23,824 images and a testing set of 5,956.

Evaluation metric. As all datasets are imbalanced, the often-used accuracy
metric would not be representative as it is highly sensitive to that. Hence,
mean Average Precision (mAP) is employed as the main evaluation metric and all
results presented in the following discussions will refer to mAP.

###  4.2 Training Details

Training procedure. The primary challenge was to create an environment to
fairly compare a wide variety of networks. For this, we followed an approach
suggested in [11] and employed a standard training procedure across all
experiments. To remove the optimisation bias, we trained all networks from
scratch and kept most parameters constant. These include cross-entropy loss
function, cosine-annealing scheduler with a step size equal to the number of
epochs and SGD optimisation strategy. A minority of parameters varied
depending on the dataset, such as the number of epochs, batch size and initial
learning rate. More details are available in the project’s GitHub directory:
https://github.com/dminskiy/EHybridNet-icpr2022.

Scattering features. To ensure the scattering coefficients align with the
EfficientNet backbone in terms of the feature spatial size as per section 3.2
the following scattering configurations were employed. For the first Hybrid
Fusion Block, HF-1, $J$ was set to 2, whereas for the second, HF-2, $J=3$. The
common parameters for both layers are the scattering order of 1, 8 angles
$\theta$ and 4 phases $\alpha$. Scattering coefficients are computed with a
modified to allow multi-GPU support Kymatio [1] package, please see the
project’s directory for details.

Figure 2: Performance comparison of Hybrid Fusion Block architectures. Dataset | All of training data | 50% of training data | 25% of training data
---|---|---|---
Hybrid, % | EfficientNet, % | Hybrid, % | EfficientNet, % | Hybrid, % | EfficientNet, %
Caltech-256 | 60.24 | 54.23 | 54.60 | 46.68 | 45.81 | 38.03
Flowers-102 | 97.75 | 94.63 | 95.81 | 91.88 | 92.14 | 85.52
CoronaHack-2 | 95.10 | 92.58 | 94.93 | 90.85 | 92.70 | 88.92
CoronaHack-3 | 87.73 | 85.66 | 85.79 | 85.26 | 79.32 | 78.42
Table 3: Results with limited training data for E-HybridNet-B0-$E$0 and
EfficientNet-B0.

##  5 Evaluation and Analysis

In this section, we first explore the properties of the E-HybridNet, then
evaluate its performance against the baseline EfficientNet. Finally, we assess
the effect of scattering features on the network’s efficiency in normal and
data-limited conditions.

###  5.1 E-HybridNet Performance

Hybrid Fusion Block architecture. First, we evaluate the effects of different
design choices on the overall network performance, concentrating on the
comparison of three primary design groups (HF-$E$, HF-$Z$ and HF-$H$) as per
section 3.1. The impact of skip connections and DropConnect is considered
later in the section.

To select the best architecture, we count the number of networks each of the
candidate hybrid designs has outperformed and accumulate it across our four
evaluation datasets and two network complexity configurations (B0 and B3),
results are shown in figure 2. To select the best performing design, we group
the obtained results based on Hybrid Fusion Block variations ($E$, $Z$ or
$H$). Thereby, we observe that HF-$E$ based networks outperformed 44.1% of
other hybrids, followed by HF-$Z$ design (better in 32.99% cases) and HF-$H$
(22.92%). These findings indicate the importance of batch normalisation before
the features are concatenated, as this is the only difference between the
leading HF-$E$ and the least effective HF-$H$ designs. Similarly, the depth-
wise convolution stage proved effective as it allowed HF-$E$ architecture to
be over 10% more efficient than the HF-$Z$ design that does not have this
feature.

Hybrid Fusion Block variations. DropConnect and skip connection modules are
present within the original EfficientNet architecture and play an important
role in its success. Here, we analyse their effect on the E-HybrdNet, for this
we apply the same method as before and count the number of peer-hybrids each
individual architecture outperforms.

Figure 2 presents the aggregate result between B0 and B3-based networks and
shows that the effect of these features depends on the fusion block
architecture. The skip connection module, on average, did not change the
performance of Hf-$E$ networks, for both HF-$E$0 and HF-$E$1 the number of
outperformed networks remained the same, at 13.54%. HF-$Z$ hybrids, on the
other hand, benefited from the addition of the skip connection - the success
rate improved by 1.74%. Whereas, HF-$H$ architecture saw a drop in performance
with the addition of this feature (-1.39%). As for the DropConnect, the
presence of this module took HF-$E$3 design to 17.01%, the highest amongst all
networks. However, its effect on other architectures was negative, reducing
the performance by 2.78% and 2.43% for HF-$Z$ and HF-$H$ respectively.

In summary, we observe that DropConnect and skip connection modules can
improve the adaptability of the network and they are most effective with the
base E-hybridNet architecture - HF-$E$.

Comparison with EfficientNet. The major drawback of previous hybrid
architectures is their relatively low classification performance as compared
to their conventional counterparts. Hence, in this experiment, we compare the
performance of the base hybrid architecture (HF-$E$0) against the vanilla
EfficientNet in B0 and B3 configurations applied to four diverse datasets.

Results presented in Table 2 evidence the dominance of the hybrid that is on
average 3.43% more accurate than the reference network with a maximum gap of
over 6% in B0 setup. When the backbone complexity is increased to B3, the
average hybrid’s advantage is 2.8% with a maximum of 4.83% To the best of our
knowledge, it is the first time a hybrid network could consistently outperform
the baseline network on a variety of tasks when entire training data is
available.

Figure 3: Results of the ablation study.

###  5.2 Ablation Study

Here, we evaluate the contribution of scattering features to the performance
of our hybrid design. As our HF block fuses scattering coefficients with an
output of the corresponding network layer (network features, in short), an
idea of the ablation study, therefore, is in disabling parts of the Hybrid
Fusion Block. In one case, the scattering features are omitted from the
merging stage while, in the other, the network features fed into the block
will be ignored. As before, we use HF-$E$0 hybrid configuration for
evaluation.

Figure 3 summarises the results of this experiment across four datasets and
following networks: EfficientNet-B0 (Reference), E-HybridNet-B0-$E$0 (Hybrid),
E-HybridNet-B0-$E$0 with scattering features omitted (Hybrid Scat Disabled)
and E-HybridNet-B0-$E$0 with network features ignored (Hybrid Net Disabled).
Across the four datasets, the average loss in performance due to the lack of
scattering features is 5.08%, while the average performance drop due to the
network features not being present is 1.48% as compared to the baseline hybrid
result. This indicates the significance of scattering features in hybrid
architecture and shows that they are, indeed, the driving factor for its
success. Additionally, we note that the average performance loss caused by
disabling both hybrid and networks features (3.64%) is approximately equal to
the average gain of the hybrid versus the EfficientNet-B0 (3.76%)
architecture. This can be interpreted as ”disabling” both parts of the Hybrid
Fusion Block (or effectively removing the hybrid part of the network) is the
same as running the standard EfficientNet-B0. Importantly, most of the loss
when moving from hybrid to standard design is due to a lack of scattering
features.

###  5.3 Generalisation

Historically, hybrids’ ability to generalise from small amounts of data have
been their strongest advantage over their fully-trained counterparts. To
validate that our design preserves this strength, we compare the base hybrid
network (HF-$E$0) and its baseline EffcientNet-B0 in data-limited scenarios.
We use our four evaluation datasets with 50% and 75% of the training samples
removed randomly.

From results presented in Table 3, we observe that the hybrid network was
constantly ahead of EfficientNet-B0. On average, E-HybridNet was 3.43%, 4.11%
and 4.77% more accurate than the reference architecture with full, half and
quarter of the training set available. It corresponds to approximately 0.7% of
relative gain for the hybrid as the amount of data is halved. Hence, we
conclude that our fusion procedure embeds scattering features effectively,
aids CNN generalisation and improves overall performance in data-limited
scenarios.

##  6 Conclusions

In this work, we introduced the E-Hybrid network, the first scattering based
hybrid network that consistently outperformed its conventional counterparts.
The evaluation showed that our design improves mAP by up to 6% with an average
gain of 3.11% across tested datasets. We also presented Hybrid Fusion Blocks,
the integral part of the novel design. They allow a network to benefit from
the adaptability of CNNs while being guided by powerful hand-crafted
representations. Our analysis showed that scattering features were indeed the
driving force behind the success of the proposed hybrid design and constituted
a major part of the performance gain. They also allowed E-HybridNets to be
superior in data-limited scenarios, considerably improving the network’s
generalisation.

Current limitations and further work. Despite the advantages of the novel
architecture, certain aspects of it provide further research opportunities.
For instance, currently, each Hybrid Fusion Block requires recalculation of
scattering features which slows inference and training speeds by around 60%.
Another important area that has not been fully addressed yet is training
optimisation for E-Hybrid networks, so far they have been tested mostly under
a common setup strategy to facilitate the concept verification and enable a
fair comparison. There is also a number of scattering and backbone
architectures that could be explored within the proposed architectural
paradigm. Therefore, we believe that this work is only a promising start in
exploring a wide range of research opportunities in hybrid networks.

## References

  * [1] M. Andreux, T. Angles, G. Exarchakis, R. Leonarduzzi, G. Rochette, L. Thiry, J. Zarka, S. Mallat, J. Andén, E. Belilovsky, et al.  Kymatio: Scattering transforms in python.  J. Mach. Learn. Res., 21(60):1–6, 2020.
  * [2] J. Bruna and S. Mallat.  Invariant scattering convolution networks.  IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013.
  * [3] J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi.  Covid-19 image data collection: Prospective predictions are the future.  arXiv 2006.11988, 2020.
  * [4] F. Cotter and N. Kingsbury.  A learnable scatternet: Locally invariant convolutional layers.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 350–354. IEEE, 2019.
  * [5] W. Czaja and W. Li.  Analysis of time-frequency scattering transforms.  Applied and Computational Harmonic Analysis, 47(1):149–171, 2019\.
  * [6] S. Fujieda, K. Takayama, and T. Hachisuka.  Wavelet convolutional neural networks.  arXiv preprint arXiv:1805.08620, 2018.
  * [7] G. Griffin, A. Holub, and P. Perona.  The caltech 256 (technical report).  Caltech Computation and Neural Systems, 2006.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] J. Hu, L. Shen, and G. Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [10] S. Mallat.  Group invariant scattering.  Communications on Pure and Applied Mathematics, 65(10):1331–1398, 2012.
  * [11] D. Minskiy and M. Bober.  Scattering-based hybrid networks: An evaluation and design guide.  In 2021 IEEE International Conference on Image Processing (ICIP), pages 2793–2797. IEEE, 2021.
  * [12] M.-E. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [13] E. Oyallon, E. Belilovsky, and S. Zagoruyko.  Scaling the scattering transform: Deep hybrid networks.  In Proceedings of the IEEE international conference on computer vision, pages 5618–5627, 2017.
  * [14] E. Oyallon, E. Belilovsky, S. Zagoruyko, and M. Valko.  Compressing the input for cnns with the first-order scattering transform.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 301–316, 2018.
  * [15] E. Oyallon and S. Mallat.  Deep roto-translation scattering for object classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2865–2873, 2015.
  * [16] E. Oyallon, S. Zagoruyko, G. Huang, N. Komodakis, S. Lacoste-Julien, M. Blaschko, and E. Belilovsky.  Scattering networks for hybrid representation learning.  IEEE transactions on pattern analysis and machine intelligence, 41(9):2208–2221, 2018.
  * [17] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [18] I. W. Selesnick, R. G. Baraniuk, and N. C. Kingsbury.  The dual-tree complex wavelet transform.  IEEE signal processing magazine, 22(6):123–151, 2005.
  * [19] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [20] A. Singh and N. Kingsbury.  Dual-tree wavelet scattering network with parametric log transformation for object classification.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2622–2626. IEEE, 2017.
  * [21] M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
  * [22] L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus.  Regularization of neural networks using dropconnect.  In International conference on machine learning, pages 1058–1066. PMLR, 2013.
  * [23] S. Zagoruyko and N. Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [24] J. Zarka, L. Thiry, T. Angles, and S. Mallat.  Deep network classification by scattering and homotopy dictionary learning.  arXiv preprint arXiv:1910.03561, 2019.


# Public and private beliefs under disinformation in social networks

Diana Riazi  Giacomo Livan

###### Abstract

We develop a model of opinion dynamics where agents in a social network seek
to learn a ground truth among a set of competing hypotheses. Agents in the
network form private beliefs about such hypotheses by aggregating their
neighbors’ publicly stated beliefs, in an iterative fashion. This process
allows us to keep track of scenarios where private and public beliefs align,
leading to population-wide consensus on the ground truth, as well as scenarios
where the two sets of beliefs fail to converge. The latter scenario — which is
reminiscent of the phenomenon of cognitive dissonance — is induced by
injecting ‘conspirators’ in the network, i.e., agents who actively spread
disinformation by not communicating accurately their private beliefs. We show
that the agents’ cognitive dissonance non-trivially reaches its peak when
conspirators are a relatively small minority of the population, and that such
an effect can be mitigated — although not erased — by the presence of
‘debunker’ agents in the network.

###### keywords:

[1]organization=Department of Computer Science, University College
London,addressline=66-72 Gower Street, city=London, postcode=WC1E 6EA,
country=UK

[2]organization=Department of Computer Science, University College
London,addressline=66-72 Gower Street, city=London, postcode=WC1E 6EA,
country=UK

##  1 Introduction

We currently live in a paradoxical age of information. We have access to
unprecedented amounts of information, yet our societies often fail to reach
consensus on demonstrable facts [1]. This phenomenon has been widely studied
in the opinion dynamics literature, leading to the identification of a number
of potential factors leading to such an apparent paradox. One of such factors
is the overwhelming amount — both in terms of volume and diversity — of
information to which individuals are exposed to in online social networks
(OSNs) [2]. Even more importantly, the spread of disinformation and ‘fake
news’ has emerged as one of the most prevalent problems of today and remains
an outstanding issue in the world of cyber and online security [3]. In
particular, studies have shown that more than two-thirds of people retrieve
their news from social media platforms [4], thus the manner in which
information spreads in OSNs is particularly pressing and has evidently emerged
as a threat to the stability of democracies and nations around the world [5,
6].

A number of studies have illustrated the negative effects of the spread of
disinformation on a population’s _collective_ ability to reach consensus on a
ground truth [7, 8]. In this paper, we seek to extend this line of research to
the level of _individual_ agents in a population. Namely, we seek to
illustrate the effects of disinformation on the formation of an agent’s
_private_ beliefs, and how these are converted into _public_ beliefs that are
shared with others through a social network of interactions.

With this approach, we seek to understand the ‘microscopic’ determinants of
disagreement and divergence in social networks. In order to do that, we
leverage a class of models known as Distributed Hypothesis Testing (DHT) [9],
and its connection to opinion dynamics and information diffusion. Originally
developed in the context of sensor networks, DHT models typically describe a
population of agents — connected by a network of interactions — seeking to
learn a ground truth among a set of competing hypotheses [10, 9].

The DHT literature has focused on designing models that reach consensus, in
particular promoting exponentially fast convergence with the aim of possessing
some form of robustness, via gauging a set of needed conditions [11, 12].
Incorporating further complexities, Hare _et al._ [13] consider a non-Bayesian
protocol designed to take into account agents’ uncertainty regarding the
statistical models (i.e., likelihood functions) which govern the state of
world.

Lalitha _et al_. [10] put forward a model at the intersection of DHT and
opinion dynamics, introducing in a DHT setting a learning rule à la DeGroot
[14] based on the iterative aggregation of _local_ information gathered from
neighbors in a social network. It is precisely within this framework that
private/public beliefs are explicitly modeled, as the agents gather and
aggregate their neighbors’ publicly stated opinions, but then proceed to form
private beliefs based on them.

Similarly to [10], much of the literature in opinion dynamics could be said to
extend upon the archetypal DeGroot model [14], i.e., introducing additional
complexities in a setting based on the iterative averaging/aggregation of
information, and studying under what conditions the consensus typically
reached in standard DeGroot models may be distorted into, e.g., a permanently
polarized state. For instance, averaging models seek to show if and how
Condorcet’s Jury Theorem or wisdom of the crowds may be induced [15], with
particular invoked behavioral qualities within agents [16]. Moreover, it has
been shown that social influence undermines the wisdom of the crowd [17]. At
the same time, Becker _et al_. [18] show network conditions where social
influence may improve the accuracy of group estimates, despite the reduction
in the diversity of opinions. Other classes of models seek to incorporate
cognitive biases in the agents’ information update process, e.g., explicitly
modelling confirmation bias by having agents place more weight on opinions
that already conform to their existing beliefs [19, 20, 21]. Bounded
confidence models instead impose that agents only interact if their opinions
are similar enough [22, 21].

In the next section, we introduce the general structure of DHT models and
detail its connection to opinion dynamics and social learning.

##  2 Social Learning and Distributed Hypothesis Testing

In DHT models, agents seek to learn a ground truth via private signals,
modelled as random draws from a distribution. The agents always receive
signals from the distribution corresponding to the ground truth, but are
unaware of that. In fact, their learning process aims at identifying from
which distribution they are receiving signals among a number of different
distributions. Each distribution may be thought as representing a competing
hypothesis or narrative. In the following, we set our notation and formalise
the learning rule we will employ throughout the paper.

We consider $N$ agents connected by a network $W$ of interactions (such that
$W_{ij}>0$ when agents $i$ and $j$ are connected and $W_{ij}=0$ otherwise).
The weight $W_{ij}$ represents the amount of influence that agent $i$ accepts
from agent $j$. Because of this, the matrix $W$ is assumed to be row-
stochastic, i.e. $\sum_{j=1}^{N}W_{ij}=1$. Let us also consider a set of
$N\times M$ multivariate distributions $f_{i}(X;\theta_{ik})$ ($i=1,\ldots,N$,
$k=1,\ldots,M$), where $\theta_{ik}$ denotes the set of parameters that define
the $k$-th distribution associated with agent $i$. To simplify notation, in
the following we will drop the subscript $i$, as it will be clear from context
to which agent the set of parameters $\theta_{ik}$ refers to.

We adopt the convention that — for each agent — the $M$-th distribution is the
one corresponding to the ground truth. Time is discrete and denoted as
$t=1,...,T$. At time $t=0$ each agent is initialized with a random vector of
private beliefs
$\bm{q}_{i}^{(0)}=\left(q_{i}^{(0)}(\theta_{1}),\ldots,q_{i}^{(0)}(\theta_{M})\right)$,
such that $q_{i}^{(0)}(\theta_{k})\geq 0,\ \forall k$ and
$\sum_{k=1}^{M}q_{i}^{(0)}(\theta_{k})=1$. At each time step, the following
occurs:

  * 1.

Each agent $i$ ($i=1,\ldots,N$) receives a signal $X_{i}^{(t)}$ as a random
draw from the distribution corresponding to the ground truth, i.e.,
$X_{i}^{(t)}\sim f_{i}(\cdot;\theta_{M})$.

  * 2.

Each agent $i$ performs a local Bayesian update on their current vector of
private beliefs $\bm{q}_{i}^{(t)}$ to form a public belief vector
$\bm{b}_{i}^{(t)}$ with components

$b_{i}^{(t)}(\theta_{k})=\frac{f_{i}(X^{(t)}_{i};\theta_{k})\
q_{i}^{(t-1)}(\theta_{k})}{\sum_{\ell=1}^{M}f_{i}(X^{(t)}_{i};\theta_{\ell})\
q^{(t-1)}_{i}(\theta_{\ell})}$ (1)

  * 3.

Each agent $i$ shares their public belief vector $\bm{b}_{i}^{(t)}$ with all
their neighbors in the network, and similarly receives public belief vectors
from each of them.

  * 4.

Each agent $i$ updates their private belief vector $\bm{q}_{i}^{(t)}$ by
averaging the log-beliefs they received from neighbors, i.e.,

$q^{(t)}_{i}(\theta_{k})=\frac{\exp\left(\sum_{j=1}^{N}W_{ij}\log
b^{(t)}_{j}(\theta_{k})\right)}{\sum_{\ell=1}^{M}\exp\left(\sum_{j=1}^{N}W_{ij}\log
b^{(t)}_{j}(\theta_{\ell})\right)}\ .$ (2)

Note that both Eqs. (1) and (2) ensure that the public and private belief
vectors of each agent remain correctly normalized as probability vectors at
each time step.

It can be shown (see [10]) that as long as _global distinguishability_ is
guaranteed, then the ground truth can be collectively learnt by all agents
exponentially fast. Global distinguishability refers to the fact that at least
one agent in the network is able to distinguish between a pair of competing
hypotheses, i.e., for all $k\neq\ell$, there exists at least one agent $i$
such that
$D_{\mathrm{KL}}(f_{i}(\cdot,\theta_{k})||f_{i}(\cdot,\theta_{\ell}))>0$,
where $D_{\mathrm{KL}}(\cdot||\cdot)$ denotes the Kullback-Leibler divergence.

We illustrate the above result by simulating the learning rule on a fully-
connected network with $N=100$ agents and $M=4$ hypotheses. The agents receive
their signals from multivariate Gaussians, so that $\theta_{k}$
($k=1,\ldots,4$) here represents the mean vectors and covariance matrices of
each of such Gaussians. In order to ensure global distinguishability, we
assume that each agent is only able to distinguish between pairs of competing
hypotheses. For instance, we may have a certain agent $h$ with
$f_{h}(\cdot;\theta_{1})=f_{h}(\cdot;\theta_{3})$ and
$f_{h}(\cdot;\theta_{2})=f_{h}(\cdot;\theta_{4})$ (agent $h$ fails to
distinguish between hypotheses $1$ and $3$, and between hypotheses $2$ and
$4$), and an agent $\ell$ with
$f_{\ell}(\cdot;\theta_{1})=f_{\ell}(\cdot;\theta_{2})$ and
$f_{\ell}(\cdot;\theta_{3})=f_{\ell}(\cdot;\theta_{4})$ (agent $\ell$ fails to
distinguish between hypotheses $1$ and $2$, and between hypotheses $3$ and
$4$). In Fig. 1 we show the temporal evolution of the private belief vector of
a randomly chosen agent in the network. As it can be seen, the private belief
in the ground truth quickly reaches one, whereas all other private beliefs go
to zero. The same occurs for any other agent, leading to consensus on the
ground truth.

Figure 1: Temporal evolution of a randomly selected agent’s private belief
vector in a model with $N=100$ agents and $M=4$ hypotheses.

##  3 Truthfulness and disinformation

In the previous section we have introduced the behaviour of individual agents.
Here, we begin to characterize the overall network’s ability to learn the
ground truth. To that end, we introduce the following quantity

$\tau(t)=\frac{1}{N}\sum_{i=1}^{N}q_{i}^{(t)}(\theta_{M})\ ,$ (3)

which quantifies the average private belief placed by the agents on the ground
truth (we recall that — by convention — we assume the $M$-th hypothesis to be
the true one). We shall refer to the quantity in Eq. (3) as _truthfulness_.
Thanks to the normalization of private belief vectors, truthfulness can also
be expressed as the difference between one and the average private belief
collectively placed on the $M-1$ wrong hypotheses:

$\tau(t)=1-\frac{1}{N}\sum_{i=1}^{N}\sum_{\ell=1}^{M-1}q_{i}^{(t)}(\theta_{\ell})\
.$

When the aforementioned conditions on the global distinguishability of
hypotheses are met, and the agents correctly learn the ground truth over time,
one obviously has $\lim_{t\rightarrow\infty}\tau(t)=1$, i.e., consensus on the
ground truth.

Given that consensus on the ground truth is guaranteed under very mild
conditions, our interest now shifts to what happens when those conditions are
perturbed. First, we take a look at what happens after injecting
_conspirators_ , i.e., agents that do not perform an update of their public
beliefs as per Eq. (1) (and that do not update their private beliefs either).
Specifically, conspirators maintain their public belief on one of the $M-1$
wrong hypotheses artificially high , therefore effectively spreading
disinformation and affecting the ‘regular’ agents’ learning process. In
particular, we impose that conspirator agents maintain a public belief vector
$\bm{b}^{(t)}=(b_{1},b_{2},b_{3},\ldots)$ with $b_{1}\approx 1$ and $b_{j}\ll
1$ for $j=2,\ldots,M$, that is, conspirator agents push one of the wrong
hypotheses (which, without loss of generality, we assume to be the first one)
at each time step.

Let us consider a fully connected network of $N=100$ agents with $M=4$
competing hypotheses. Fig. 2 (left panel) shows truthfulness in the long run —
where Eq. (3) is only computed as an average over regular agents — as a
function of the concentration $\beta_{c}$ of (randomly placed) conspirators in
the network. We observe an intuitive negative relationship. Namely, as the
concentration of conspirators pushing a particular wrong hypothesis increases,
the collective belief in the ground truth decreases and converges to nearly
zero.

Additionally, in the right panel of Fig. 2, we see the time evolution of the
regular agents’ belief in the truth, given some fixed concentration of
conspirators. For a zero sub-population of conspirators, truthfulness
ultimately converges to one, i.e., consensus on the ground truth. We also see
that a perturbation of just a single conspirator ($\beta_{c}=1/N$) imposed on
a network population of $100$ agents is enough to deter a full consensus.
Finally, given a larger proportion ($\beta_{c}=0.2$) of conspirators, we see a
noticeable reduction in truthfulness.

Figure 2: Left panel: Average truthfulness in the regular agents sub-
population as a function of the fraction of conspirators in the network. Right
panel: temporal evolution of truthfulness with varying concentrations of
conspirators in the network.

Given the result that truthfulness decreases monotonically with the proportion
of conspirators, we may additionally consider whether every agent shares the
same private belief in the true hypothesis or is there instead a non-trivial
distribution of beliefs int he long run. In other words, does there exist a
divergence among agents within networks induced by the presence of
conspirators?

As may be seen in Fig. 3, we observe the distributions of regular agents’
private beliefs in the truth given some concentration of conspirators. Namely,
with only $1/N$ conspirator present, we observe a rather skewed distribution,
with most regular agents’ private belief in the ground truth being equal or
very close to one. Still, it is remarkable to see how the presence of a single
conspirator is sufficient to render beliefs to significantly deviate from one
for some agents. With $\beta_{c}=25/N$ we see a rather spread out
distribution, with private beliefs in the ground truth ranging from $10\%$ to
almost $80\%$, whereas with $\beta_{c}=50/N$ conspirators dominate and we see
a significantly narrower distribution peaked around very low values.

Figure 3: Steady state distributions of beliefs in the ground truth in the
regular agents sub-population under varying concentrations of conspirators in
the network ($N=300$, distributions obtained over $500$ distinct simulations)

##  4 A Model of Cognitive Dissonance

In Psychology, cognitive dissonance (CD) refers to the mental toll experienced
by an individual when faced with contradictory information [23]. In the
context of our model, we assimilate CD to the difference (in absolute value)
between what an agent privately believes and what they publicly express.
Mathematically, the CD experienced by an agent $i$ at time $t$ regarding some
hypothesis $\theta_{k}$ is

$C^{(t)}_{i}(\theta_{k})=\left|q^{(t)}_{i}(\theta_{k})-b^{(t)}_{i}(\theta_{k})\right|\
.$ (4)

We numerically simulate a series of Erdos-Rényi (ER) and Barabasi-Albert (BA)
networks (with the same average degree) to consider the relationship which
exists between the concentration of conspirators and the average CD
experienced by regular agents, as well as to account for possible differences
induced by network topology. In Fig. 4 we report the average CD experienced by
regular agents with respect to the ground truth hypothesis
($C^{(t)}(\theta_{M})$) as a function of the proportion of conspirators in the
network. The relationship is non-monotonic, and CD achieves its maximum in
correspondence of a certain intermediate value $\beta_{c}\approx 10-15\%$.
Notably, this result holds with little variation both in ER and BA networks..
This may seem counter-intuitive at first, as one may expect CD to be higher
when regular agents are exposed to higher concentrations of conspirators.

The above result becomes easier to interpret when considering the two extreme
scenarios. When $\beta_{c}\rightarrow 0$ agents reach consensus on the ground
truth, leading to both $q^{(t)}_{i}(\theta_{M})\rightarrow 1$ and
$b^{(t)}_{i}(\theta_{M})\rightarrow 1$, and therefore
$C^{(t)}_{i}(\theta_{M})\rightarrow 0$ for all agents in the long run.
Conversely, when $\beta_{c}\rightarrow 1$, the few regular agents present in
the network are overwhelmed by the false information spread by conspirators,
which similarly leads — to some extent — to the convergence of public and
private beliefs, therefore suppressing CD.

These results are further illustrated in Fig. 5, where we report the temporal
evolution of a randomly selected regular agent’s public and private belief in
the ground truth under different scenarios. We note that for low/high
concentrations of conspirators, the agent’s public and private beliefs quickly
align. Conversely, for concentrations around the ‘critical’ value
$\beta_{c}\approx 10\%-15\%$ we observe considerable oscillations of the
agent’s public beliefs, which do not die out in the long run.

Figure 4: Cognitive dissonance as a function of the concentration of
conspirators for a series BA and ER networks with average degree $k=100$
($N=100$, averages obtained over $200$ simulations)

Given the consequences demonstrated regarding both truthfulness and CD, in the
next section we explore whether such effects could be mitigated or suppressed
by injecting ‘debunkers’ in the network, i.e., agents who actively promote the
ground truth.

Figure 5: Temporal evolution of a randomly selected agent’s public and private
belief in the ground truth $\theta_{M}$ in a single simulation (with $N=100$,
$M=4$). The concentrations of conspirators ($\beta_{c}$) are reported on the
top of each plot.

##  5 Two-Tribe Model (the addition of ‘debunkers’)

We now consider a ‘two-tribe’ version of the model, akin to the dynamics of
‘conspirators’ vs ‘debunkers’, with the latter referring to an agent that
actively pushes the ground truth $\theta_{M}$, where the parameter $\beta_{d}$
refers to the concentration of debunkers present within a network. We again
model conspirators as agents with a static belief vector with
$b_{1}^{(t)}\approx 1$ and $b_{j}^{(t)}\ll 1$ for $j=2,\ldots,M$. In a similar
fashion, we model debunkers to have a static belief vector with
$b_{M}^{(t)}\approx 1$ and $b_{j}^{(t)}\ll 1$ for $j=1,\ldots,M-1$. As
previously mentioned, we seek to understand whether the injection of debunkers
in a social network may impact and potentially alleviate the effects of
disinformation.

We begin by considering the effects on truthfulness. In Fig. 6, it can be
observed that, given a fixed fraction of conspirators, the collective belief
in the ground truth approaches $1$ more closely as more debunkers are added to
the network, as one would intuitively expect. In essence, this is the result
which Fig. 6 demonstrates, that is that truthfulness reverts to $1$, i.e.
regular agents believe in the truth completely, as the concentration of
debunkers increases.

Figure 6: Average truthfulness in the regular agent sub-population as a
function of the concentration of debunkers ($\beta_{d}$) in a model with
$N=120$ agents and a fixed concentration of conspirators $\beta_{c}=25\%$.

Figure 7: Comparison in truthfulness between ER and BA networks with average
degrees equal to $k=10$ (left panels) and $k=100$ respectively (right panels),
with a network size of $N=300$. The concentrations of conspirators
($\beta_{c}$) and debunkers ($\beta_{d}$) are reported on top of each panel.

Figure 8: Left panel: Non-monotonicity in the behavior of CD as a function of
the concentration of debunkers. Right panel: heatmap with steady state values
of CD as a function of the concentrations of both conspirators and debunkers
in the network. Figure 9: Distribution of private beliefs in the ground truth,
with varying relative conspirator-debunker sub-populations, with a network
population of $N=300$ for $500$ simulations

In Fig. 7, we compare the temporal evolution of truthfulness on ER and BA
networks, in order to determine what type of network is ‘better’ at
manufacturing truthfulness. We observe notable differences only for sparse
networks, but as soon as the networks’ average degree becomes large such
differences cease to be statistically significant.

We now consider the effects on agents’ cognitive dissonance induced by a sub-
population of debunkers. As may be observed from Fig. 8 (left panel), CD does
not decay monotonically upon injecting debunkers, but rather peaks when the
sub-populations of conspirators and debunkers reach equal sizes. This result
captures well the ‘backfire’ effect on the downsides of debunking [24, 25,
26]. In the right panel of Fig. 8 this result is presented more generally as a
function of both the concentrations of conspirators and debunkers.

Lastly, in Fig. 9 we report the steady state private belief distributions for
the regular agents sub-population in scenarios where the concentration of
conspirators exceeds the concentration of debunkers ($\beta_{c}>\beta_{d}$)
and vice versa ($\beta_{c}<\beta_{d}$), as well as the scenario in which the
two concentrations are equal ($\beta_{c}=\beta_{d}$). We see that only when
debunkers are in larger numbers there is a sizeable shift to the right in the
distribution, whereas the other two scenarios are fairly similar.

##  6 Conclusion

In this paper, we put forward an opinion dynamics model aimed at describing
the apparent tension between what individuals in a social network privately
believe and what they publicly express. By relying on the DHT framework, we
were able to produce both scenarios in which private and public beliefs
quickly align for all individuals, as well as scenarios in which they remain
permanently misaligned over time, which we relate to the psychological
phenomenon of cognitive dissonance [23]. To the best of our knowledge, this
aspect is novel in the opinion dynamics literature, although the distinction
between public and private beliefs had minimalistically been attended to and
implemented within models of social learning [27].

Notably, the misalignment between private and public beliefs emerges only
after introducing ‘conspirators’ in a population of agents who process the
information they receive from neighbors in a social network through a process
of averaging à la DeGroot [14]. We believe this result to be particularly
interesting for two reasons. First, it provides a stylized model for the
mental toll associated with the spread of disinformation in a social network.
Second, this phenomenon does not respond monotonically to the proportion of
conspirators in the population. Rather, it is maximized when conspirators are
a relatively small minority of the population ($\approx 10\%-15\%$). This is
reminiscent of similar results on the disproportionate impact that small but
dedicated minorities can have on the information aggregation capabilities of a
society (see, e.g., [28, 19, 29]).

Motivated by the above findings, we studied the model in a ‘two-tribe’
setting, where we injected ‘debunker’ agents with the aim of counteracting the
effects induced by conspirators. Yet again, we found a non-trivial response in
the model’s behavior. We found the presence of debunkers to have a genuinely
mitigating effect only when they begin to outnumber the conspirators. In fact,
the CD experienced by regular agents in the network reaches its peak when the
two sub-populations of conspirators and debunkers exactly match each other, a
result which is very much reminiscent of the empirically demonstrated
downsides of debunking attempts against the diffusion of disinformation and
fake news [24, 26].

In conclusion, it is worth mentioning that although we have continuously
thought of the departure from the objective truth boasted by conspirator
agents as ‘disinformation’, i.e., possessing a malicious intent behind
corrupting the truth, the conspirator-debunker dynamic has the potential for
such a departure to cover cases of ‘misinformation’ as well, as conspirators
may unknowingly or ignorantly be altering the truth.

## References

  * [1] A. M. McCright, R. E. Dunlap, The politicization of climate change and polarization in the american public’s views of global warming, 2001–2010, The Sociological Quarterly 52 (2) (2011) 155–194.
  * [2] M. G. Rodriguez, K. Gummadi, B. Schoelkopf, Quantifying information overload in social media and its impact on social contagions, in: Proceedings of the international AAAI conference on web and social media, Vol. 8, 2014, pp. 170–179.
  * [3] K. M. Caramancion, An exploration of disinformation as a cybersecurity threat, in: 2020 3rd International Conference on Information and Computer Technologies (ICICT), IEEE, 2020, pp. 440–444.
  * [4] D. Levy, N. Newman, R. Fletcher, A. Kalogeropoulos, R. K. Nielsen, Reuters institute digital news report 2014, Report of the Reuters Institute for the Study of Journalism (2017).
  * [5] J. Farkas, J. Schou, Post-truth, fake news and democracy: Mapping the politics of falsehood, Routledge, 2019.
  * [6] M. Pantazi, S. Hale, O. Klein, Social and cognitive aspects of the vulnerability to political misinformation, Political Psychology 42 (2021) 267–304.
  * [7] S. Vosoughi, D. Roy, S. Aral, The spread of true and false news online, science 359 (6380) (2018) 1146–1151.
  * [8] D. Acemoglu, A. Ozdaglar, A. ParandehGheibi, Spread of (mis) information in social networks, Games and Economic Behavior 70 (2) (2010) 194–227.
  * [9] R. Olfati-Saber, E. Franco, E. Frazzoli, J. S. Shamma, Belief consensus and distributed hypothesis testing in sensor networks, in: Networked Embedded Sensing and Control: Workshop NESC?05: University of Notre Dame, USA October 2005 Proceedings, Springer, 2006, pp. 169–182.
  * [10] A. Lalitha, T. Javidi, A. D. Sarwate, Social learning and distributed hypothesis testing, IEEE Transactions on Information Theory 64 (9) (2018) 6161–6179.
  * [11] A. Nedić, A. Olshevsky, C. A. Uribe, Fast convergence rates for distributed non-bayesian learning, IEEE Transactions on Automatic Control 62 (11) (2017) 5538–5553.
  * [12] S. Shahrampour, A. Jadbabaie, Exponentially fast parameter estimation in networks using distributed dual averaging, in: 52nd IEEE Conference on Decision and Control, IEEE, 2013, pp. 6196–6201.
  * [13] J. Z. Hare, C. A. Uribe, L. Kaplan, A. Jadbabaie, Non-bayesian social learning with uncertain models, IEEE Transactions on Signal Processing 68 (2020) 4178–4193.
  * [14] M. H. DeGroot, Reaching a consensus, Journal of the American Statistical association 69 (345) (1974) 118–121.
  * [15] B. Buechel, T. Hellmann, S. Klößner, Opinion dynamics and wisdom under conformity, Journal of Economic Dynamics and Control 52 (2015) 240–257.
  * [16] E. Yildiz, A. Ozdaglar, D. Acemoglu, A. Saberi, A. Scaglione, Binary opinion dynamics with stubborn agents, ACM Transactions on Economics and Computation (TEAC) 1 (4) (2013) 1–30.
  * [17] J. Lorenz, H. Rauhut, F. Schweitzer, D. Helbing, How social influence can undermine the wisdom of crowd effect, Proceedings of the national academy of sciences 108 (22) (2011) 9020–9025.
  * [18] J. Becker, D. Brackbill, D. Centola, Network dynamics of social influence in the wisdom of crowds, Proceedings of the national academy of sciences 114 (26) (2017) E5070–E5076.
  * [19] O. Sikder, R. E. Smith, P. Vivo, G. Livan, A minimalistic model of bias, polarization and misinformation in social networks, Scientific reports 10 (1) (2020) 5493.
  * [20] A. E. Allahverdyan, A. Galstyan, Opinion dynamics with confirmation bias, PloS one 9 (7) (2014) e99557.
  * [21] M. Del Vicario, A. Scala, G. Caldarelli, H. E. Stanley, W. Quattrociocchi, Modeling confirmation bias and polarization, Scientific reports 7 (1) (2017) 40391\.
  * [22] J. Lorenz, Continuous opinion dynamics under bounded confidence: A survey, International Journal of Modern Physics C 18 (12) (2007) 1819–1838.
  * [23] J. Cooper, Cognitive dissonance: Where we’ve been and where we’re going, International Review of Social Psychology 32 (1) (2019) 7.
  * [24] D. P. Redlawsk, Hot cognition or cool consideration? testing the effects of motivated reasoning on political decision making, Journal of Politics 64 (4) (2002) 1021–1044.
  * [25] B. Nyhan, J. Reifler, When corrections fail: The persistence of political misperceptions, Political Behavior 32 (2) (2010) 303–330.
  * [26] F. Zollo, A. Bessi, M. Del Vicario, A. Scala, G. Caldarelli, L. Shekhtman, S. Havlin, W. Quattrociocchi, Debunking in a world of tribes, PloS one 12 (7) (2017) e0181821.
  * [27] Y. Shang, Resilient consensus for expressed and private opinions, IEEE Transactions on Cybernetics 51 (1) (2019) 318–331.
  * [28] G. Livan, M. Marsili, What do leaders know?, Entropy 15 (8) (2013) 3031–3044.
  * [29] M. Stella, E. Ferrara, M. De Domenico, Bots increase exposure to negative and inflammatory content in online social systems, Proceedings of the National Academy of Sciences 115 (49) (2018) 12435–12440.