11institutetext: Computational Linguistics at Concordia (CLaC) Laboratory
Department of Computer Science and Software Engineering
Concordia University, Montréal, Québec, Canada

# Analyzing Persuasive Strategies in Meme Texts: A Fusion of Language Models
with Paraphrase Enrichment

Kota Shamanth Ramanath Nayak and Leila Kosseim

###### Abstract

This paper describes our approach to hierarchical multi-label detection of
persuasion techniques in meme texts. Our model, developed as a part of the
recent SemEval task, is based on fine-tuning individual language models (BERT,
XLM-RoBERTa, and mBERT) and leveraging a mean-based ensemble model in addition
to dataset augmentation through paraphrase generation from ChatGPT. The scope
of the study encompasses enhancing model performance through innovative
training techniques and data augmentation strategies. The problem addressed is
the effective identification and classification of multiple persuasive
techniques in meme texts, a task complicated by the diversity and complexity
of such content. The objective of the paper is to improve detection accuracy
by refining model training methods and examining the impact of balanced versus
unbalanced training datasets. Novelty in the results and discussion lies in
the finding that training with paraphrases enhances model performance, yet a
balanced training set proves more advantageous than a larger unbalanced one.
Additionally, the analysis reveals the potential pitfalls of indiscriminate
incorporation of paraphrases from diverse distributions, which can introduce
substantial noise. Results with the SemEval 2024 data confirm these insights,
demonstrating improved model efficacy with the proposed methods.

###### Keywords:

Language Models, Multi-label Classification, Persuasion Techniques.

##  1 Introduction

The recent SemEval-2024 shared task 4 [1] proposed three distinct subtasks
dedicated to identifying persuasion techniques conveyed by memes. The primary
aim was to unravel how memes, integral to disinformation campaigns, employ
various techniques to shape user perspectives. Subtask 1 focused on the
analysis of textual content alone and mandated the detection of 20 persuasion
techniques structured hierarchically within the textual content of memes. On
the other hand, subtasks 2 and 3 involved the analysis of multimodal memes
that considered both textual and visual elements.
This paper describes the approach we used for our participation to subtask 1
and further analysis with the dataset after the shared task. The task provided
a training dataset in English, but in addition to being tested on English, the
task also mandated the evaluation of our model’s zero-shot performance in
three surprise languages. The goal during the testing phase was not only to
explore our model’s ability to perform well in English but also generalize to
other languages without explicit training.
Inspired by successful approaches in multilabel text classification [2, 3],
our strategy involved fine-tuning three language models, i.e, BERT, XLM-
RoBERTa, and mBERT, followed by ensemble modeling using mean aggregation. To
enhance performance, we used data augmentation through paraphrasing and
adjusted the classification thresholds for each persuasion technique based on
class-wise metrics optimised using the validation set. During testing, a zero-
shot approach based on machine translation was implemented to classify
instances in the surprise languages. The official results of our system [1]
demonstrated good performance advantages over the baseline in all languages
except Arabic, where the increase in performance was not significant. Results
also show that paraphrasing techniques enhances model performance but using a
balanced training set is more beneficial than a larger unbalanced one.
Furthermore, analysis highlights the potential downside of indiscriminately
incorporating paraphrases from diverse distributions, as this can introduce
notable noise into the system.
This paper is organized as follows: Section 2 reviews relevant previous work
in the field of multi-label classification. Section 3 provides an overview of
the task and the data utilized. Section 4 presents an overview of our
classification pipeline along with the techniques used for data augmentation;
while Section 5 describes experimental details. Finally, the analysis of our
model’s results is presented in Section 6; while Section 7 provides
conclusions and outlines future work.

##  2 Previous Work

Several studies have explored multi-label classification of textual content.
In 2019, [4] showed that Bi-GRUs with label-wise attention led to good
performance, and the inclusion of domain-specific Word2vec and context-
sensitive ELMo embeddings further boosted the performance on the EURLEX57K
dataset that contained 57k English EU legislative documents. [5] introduced
five innovative contrastive losses for multi-label text classification using
the dataset from the SemEval 2018 Multi-label Emotion Classification (MEC)
task [6] in English, Arabic, and Spanish that contained 8640 instances. All
five contrastive learning methods notably enhanced the performance of the
previous top-performing model, SpanEmO [7], for the MEC task. Among these
approaches, the Jaccard Similarity Probability Contrastive Loss demonstrated
the highest effectiveness on the English dataset, achieving $F_{Macro}$ and
$F_{Micro}$ scores of 57.68 and 71.01, respectively.

In hierarchical multi-label classification (HMC), samples are assigned to one
or more class labels within a structured hierarchy. Approaches to HMC can be
divided into local and global methods. Local methods use multiple classifiers,
often overlooking the overall structure of the hierarchy. For example, [8]
trained a multi-layer perceptron incrementally for each hierarchy level, using
predictions from one level as inputs for the next. In contrast, global methods
employ a single model to address all classes and implement various strategies
to capture the hierarchical relationships between labels. One such approach
[9] modeled the hierarchy as a directed graph and introduced hierarchy-aware
structure encoders, using a bidirectional TreeLSTM and a hierarchy-GCN to
extract and aggregate label structural information in an end-to-end fashion.
[10] redefined hierarchical text classification (HTC) as a sequence generation
task and developed a sequence-to-tree (Seq2Tree) framework to model the
hierarchical label structure. Additionally, they created a constrained
decoding strategy with a dynamic vocabulary to ensure label consistency.

In the context of the SemEval 2020 Task 11 [11], two subtasks were introduced
addressing span identification of propagandistic textual fragments and a
multi-label technique classification (TC) of propagandistic fragments using a
corpus of 7k instances from the news domain. The subsequent SemEval 2021 Task
6 [12] focused on the identification of propagandistic techniques from
multimodal data including text and images from memes. The top-performing teams
in 2020 and 2021, ApplicaAI [2] and MinD [3] respectively, leveraged pre-
trained language models and ensemble techniques to achieve top scores at the
shared tasks. This year’s shared task was built upon the 2021 task but
included hierarchical metrics as well as a multilingual setting and a new
training dataset of 7k instances in English. Inspired by the top models at
[11] and [12], our methodology is also based on an ensemble of pre-trained
language models but leverages paraphrase generation to further improve
performance.

##  3 Labelling Persuasion Techniques

The goal of our model was to categorize the textual content of memes into one
or several persuasion techniques. For example, given the training instance
shown in Figure 1, the model needs to learn that the text Don’t expect a
broken government to fix itself should be labelled with the three techniques
provided in the labels field.

---
Figure 1: A sample training instance. The text is labelled with three
techniques, Loaded Language, Slogans and Name calling/Labelling. Figure 2:
Hierarchy of the persuasion techniques to be used to label the texts. Figure
slightly modified from [1].

##### Persuasion Techniques:

The SemEval organizers provided an inventory of 20 persuasion techniques to be
used as labels (eg: Loaded Language, Slogans, Name calling/Labelling) and were
structured hierarchically as shown in Figure 2. This rendered the task a
hierarchical multi-label classification problem and was therefore evaluated
using hierarchical precision, recall and F measures.

Figure 3: Distribution of the data for each persuasion technique in the
training set.

##### Datasets:

The dataset provided contained memes collected from online public groups
discussing a variety of topics such as politics, vaccines, COVID-19, gender
equality, and the Russo-Ukrainian War. For our task, only the text extracted
from these memes was used. The training (7k samples), validation (500 samples)
and development (1k samples) sets included only English texts.
Figure 3 shows the distribution of instances for each persuasion technique in
the training set. As the figure shows, some techniques, such as Loaded
Language and Smears, had large number of instances in the training set (1750
and 1990 respectively); while others like Straw Man and Red Herring were
severely underrepresented (62 and 59 instances respectively).

Figure 4 shows the distribution of labels per instances. As the figure shows,
most of the instances (47%) were labelled with multiple techniques, 35% were
labeled with only 1 technique and 18% had no labels at all.
Given the above English training set and hierarchical persuasion techniques,
the goal of our model was to identify 0 or $n$ techniques for each textual
instance in English and in 3 surprise languages.

Figure 4: Distribution of persuasion techniques per instance in the training
set.

##  4 Proposed Approach

Figure 5 shows an overview of the classification pipeline we employed for this
task. As shown in Figure 5, our methodology is based on fine-tuning three
distinct pre-trained language models: BERT [13], XLM-RoBERTa [14], and mBERT
[13] on augmented datasets.

Figure 5: Schematic overview of our classification pipeline for the detection
of persuasion techniques in memes.

###  4.1 Multi-label Classification

As Figure 5 shows, the data is first preprocessed using standard tokenization.
Then we proceeded to fine-tune three distinct models: bert-base-uncased, xlm-
roberta-base, and bert-base-multilingual-uncased which returned a probability
distribution over the 20 techniques. These three model predictions were then
pooled via averaging.

Despite the hierarchical organization of the persuasion techniques, we opted
to predicting solely the technique names (leaf nodes in Figure 2) and not
their ancestor nodes. However, to address the multi-label classification, we
implemented thresholding in order to determine which techniques have a high
enough score to be part of the output label set. We experimented with custom
values for each technique with values ranging from 0.01 to 0.7 and picked the
optimal values for each class based on the validation set. These thresholds
were applied to the scores obtained after passing the logits of each class
through a sigmoid function.

To handle the three surprise languages, during the official testing phase
system, the model trained only on English, would automatically translate the
surprise language to English for our model’s zero-shot predictions. This was
inspired by the approach of [15].

Figure 6: Techniques that showed an improvement in F1 score with the
validation set when using $n$=3 paraphrases (i.e. Para-n3) compared to
Comb-14k.

###  4.2 Data Augmentation

To mitigate the lack of data we took advantage of two data augmentation
strategies: an external dataset and automatically generated paraphrases.

####  4.2.1 External Dataset (Comb-14k dataset):

The Technique Classification (TC) subtask from the SemEval 2020 Task 11 [11]
provided a dataset of 7k instances from the news domain annotated with similar
guidelines as this year’s. In contrast to the 2020 task, this year’s dataset
covered a different domain and used a revised set of persuasion techniques
compared to the 2020 inventory. Indeed, in the 2020 TC dataset, a few
techniques were merged into a single category due to lack of data, resulting
in a list of 14 techniques. In the current year, an expanded inventory of 20
techniques was employed. To ensure consistency between the two sets, we
preprocessed the 2020 TC dataset by splitting techniques that had previously
been merged. For example, we singled out Bandwagon and Reductio ad Hitlerum,
which had been merged into a single technique in the SemEval 2020 TC dataset.
We considered two approaches to leverage the modified 2020 TC dataset. The
initial option involved pre-training models on this dataset, followed by fine-
tuning on the 2024 training data—an approach implemented by [3]. Another
approach entailed combining both datasets and fine-tuning models on this
combined dataset. We chose the latter method because the two datasets covered
different genres and a joint training approach would likely enable the model
to better adapt and grasp nuanced linguistic patterns across both. For easy
reference in the rest of the paper, we call the combined dataset Comb-14k.
Figure 7(a) (gray + pink) shows the resulting distribution of the persuasion
techniques in this combined dataset.

|
---|---
(a) | (b)
Figure 7: Distribution of the data for each persuasion technique in (a) the
original training set (in gray), the Comb-14k dataset (in gray + pink) and the
Para-Benef dataset (in gray + pink + violet) (b) the Para-Bal dataset.

####  4.2.2 Paraphrasing (Para-* datasets)

Despite having almost doubled each class with the use of the 2020 TC dataset,
some classes were still severely underrepresented; see Figure 7(a) (gray +
pink). To address this, we took advantage of an LLM to generate paraphrases
for each training instance, then labeled these paraphrases with the same set
of labels as the original instance. Our intuition was twofold. First,
generating paraphrases would expose the model to a more extensive set of
samples for each class, potentially improving its ability to discern subtle
nuances within the data. Second, paraphrasing sentences could unveil hidden
semantics, providing the model with a tool to identify propaganda techniques
reliant on nuanced linguistic choices or phrasing. To generate paraphrases, we
leveraged ChatGPT-3.5 turbo, setting the temperature to 0.7. This value aimed
to introduce diversity in the paraphrases while maintaining relevance to the
original instances. Several datasets were created using this method:

Para-n1 and Para-n3:

For each instance in Comb-14k, we generated $n$ paraphrases. We experimented
with $n$=1 and $n$=3 leading to datasets of 28k and 52k respectively, which we
call Para-n1 and Para-n3 respectively.
The overall hierarchical F-score with the validation set given (500 instances)
showed an increase when training with these datasets and $n=3$ seemed to
perform better than $n=1$.
However, a per-class analysis showed that not all classes benefited from the
increase in support. For example, the persuasion technique Bandwagon increased
its F1 from 0.17 to 0.29; whereas Repetition decreased its F1 from 0.56 to
0.31. We therefore identified the classes with improvement in F-score greater
than 0.03 when using the Para-n3 dataset compared to the Comb-14k dataset.
These 8 techniques along with their increase in F-scores are shown in Figure
6. This set of techniques formed the basis for our subsequent strategy.

Para-Benef:

Since only 8 techniques seemed to benefit from the use of paraphrases, we
created a new augmented dataset by increasing the number of paraphrases only
for these techniques. Specifically, let $\mathbf{B}$ be the set of 8
techniques that benefited from paraphrases (see Figure 6), for all data
instances $d$ in Comb-14k labeled with techniques
$\mathbf{T}=\\{t_{1},t_{2},\ldots t_{n}\\}$ (where $n\leq 20$), for each
$t_{i}\in$ $\mathbf{B}$, we generated 10 paraphrases of $d$ and labeled them
with all techniques from $\mathbf{T}\cap\mathbf{B}$. This newly created
dataset called Para-Benef, contained 54k instances.
Figure fig:data_g3(a) shows the distribution of instances for each technique
in the Para-Benef dataset (gray + pink + violet), in comparison with the
original training set and the Comb-14k dataset. As Figure 7 shows, all
datasets are severely imbalanced. Our next dataset therefore tried to address
this issue.

Para-Bal:

Our last dataset used our paraphrase generation strategy to address the
dataset imbalance. We rectified the underrepresented classes in the initial
training dataset by augmenting them with paraphrases. The most frequent three
techniques—Smears, Name-calling/Labelling, and Loaded Language had 1990, 1750,
1518 samples respectively. We thus aimed at reaching similar number of
instances for the other techniques. We balanced the dataset by generating
batches of 5 paraphrases for each other technique to reach around 1500
instances. This newly created dataset called Para-Bal contained 49k instances
(see Figure 7(b)).

Table 1 shows the results of the validation with the optimal threshold for
each class using the official SemEval scorer [1]. As Table 1 shows, the use of
the original dataset (7k instances) achieved a hierarchical F1 of at most 0.49
on the development set; whereas all augmented training sets led to higher
performances. The best model with both the validation and the development set
was the ensemble trained on the Para-Bal dataset which reached an hierarchical
F1 of 0.59 and 0.61 respectively. Surprisingly, Para-Benef which contained 10
paraphrases for the benefited techniques did not perform better than using
only 3 paraphrases for all techniques (Para-n3). This suggests that the
excessive inclusion of paraphrases from a different distribution (memes versus
news) may have led to too much noise in the data.

Table 1: Hierarchical F1 scores of our models, when trained on different
English-language datasets for both the validation and development sets.

Training Set | Models | Validation Set | Development Set
---|---|---|---
Original (7k) | BERT | 0.42 | 0.43
XLM-RoBERTa | 0.48 | 0.49
mBERT | 0.48 | 0.48
Ensemble Model | 0.45 | 0.46
Comb-14k (14k) | BERT | 0.52 | 0.55
XLM-RoBERTa | 0.53 | 0.54
mBERT | 0.53 | 0.54
Ensemble Model | 0.53 | 0.56
Para-n1 (28k) | BERT | 0.55 | 0.57
XLM-RoBERTa | 0.57 | 0.54
mBERT | 0.50 | 0.53
Ensemble Model | 0.55 | 0.56
Para-n3 (52k) | BERT | 0.54 | 0.55
XLM-RoBERTa | 0.54 | 0.54
mBERT | 0.54 | 0.55
Ensemble Model | 0.56 | 0.57
Para-Benef (54k) | BERT | 0.48 | 0.51
XLM-RoBERTa | 0.54 | 0.55
mBERT | 0.51 | 0.53
Ensemble Model | 0.54 | 0.55
Para-Bal (49k) | BERT | 0.54 | 0.58
XLM-RoBERTa | 0.58 | 0.59
mBERT | 0.53 | 0.55
Ensemble Model | 0.59 | 0.61

Figure 8: Comparison of the final hierarchical F1 scores obtained by our
official model trained with Para-n3, the model trained with Para-Bal, the best
corresponding models ([16] for English, [17] for Bulgarian, North Macedonian
and Arabic) in the shared task and the baseline in each given language.

##  5 Experimental Setup

###  5.1 System Pipeline and Training Details

The system pipeline code was implemented in PyTorch. The pre-trained models
BERT [bert-base-uncased], XLM-RoBERTa [xlm-roberta-base], and mBERT [bert-
base-multilingual-uncased] and their tokenizers were sourced from Hugging
Face. All models were trained for 10 epochs using the Adam optimizer with a
learning rate of 2e-5. Batch sizes varied with BERT utilizing 128, and XLM-
RoBERTa and mBERT using 64. A final feedforward layer with 20 logits (equal to
the number of persuasion techniques) was added to each model. The Binary Cross
Entropy with logits served as the loss function, with one-hot encoding applied
to the true labels. For prediction, a sigmoid activation function was used on
the logits, followed by thresholding. The ensemble model used an unweighted
average of all predictions from the three individual models. The ChatGPT-3.5
turbo111https://platform.openai.com/docs/models/gpt-3-5-turbo API with a
temperature set to 0.7 was used for paraphrase generation. During testing, the
surprise languages were translated into English using the deep-translator
API222https://pypi.org/project/deep-translator/.

##  6 Results and Analysis

For our official submission to SemEval, the Para-Bal dataset had not been
created yet; hence our official results are based on the ensemble model
trained on the union of Para-n3 and the development set (1k samples), for a
total of 53k samples. The three surprise languages were Bulgarian, North
Macedonian and Arabic. The test set contained 1500 samples for English, 426
samples for Bulgarian, 259 samples for North Macedonian and 100 samples for
Arabic. The official results of our system are shown in Figure 8, along with a
baseline score that assigns the most frequent persuasion technique to all
instances, and the score obtained by the best performing systems for each
language [16, 17]. As Figure 8 shows, although our ensemble model did not
reach the top performance for English (0.57 versus 0.75), it performed better
than the baseline in all languages except Arabic, where the improvement was
not significant.

Using the same testing protocol, we reproduced the results using the model
trained with the balanced training dataset (Para-Bal). The results displayed
in Figure 8 indicate an improvement in score with the English test set (0.62
versus 0.57). This again confirms the importance of a balanced dataset, and
paraphrases based on the same distribution as the original texts. Indeed,
although Para-n3 is larger than Para-Bal (52k versus 49k), it is not balanced
and contains paraphrases of instances from different genres (memes and news).
However, surprisingly, the performance enhancement when using Para-Bal is not
observed for the zero-shot classification of the surprise languages whose
performance dropped significantly. For these languages, a larger training set,
even with noisy out-of-distribution instances, leads to better results
possibly due to the noise introduced by the automatic translation itself.

Compared to the other approaches at the 2024 edition of SemEval Task 4, the
top performing team overall, OtterlyObsessedWithSemantics [17] designed a
specialized classification head to enhance a Large Language Model. They
organized the structure across several connected layers, enabling them to
build upon earlier decisions in later, more detailed layers. They optimized
the model’s performance by systematically exploring different hyperparameters
through grid-search. In addition, similarly to our approach, during the
testing phase, they translated all the surprise language datasets into
English. [16] developed a system using Chain-of-Thought based data
augmentation methods, in-domain pre-training and ensemble strategy that
combined the strengths of both RoBERTa and DeBERTa models.

##  7 Conclusion and Future Work

This paper described our approach to hierarchical multi-label detection of
persuasion techniques in meme texts. We used an ensemble model with three
fine-tuned language models and incorporated data augmentation through
paraphrasing from ChatGPT. We tested our approach through the SemEval 2024
Task 4 subtask 1 [1]. During testing, our system outperformed the baseline in
all languages. Analysis of the results show the importance of dataset
balancing and paraphrasing techniques in enhancing model performance. Despite
having a smaller number of instances, the balanced dataset consistently
outperforms its unbalanced counterparts, demonstrating the efficacy of
balancing methods. Moreover, data augmentation improves model performance, as
indicated by the under-performance of models trained on the original dataset
(7k instances). Additionally, the results underscore the potential drawbacks
of including paraphrases from diverse distributions, which may introduce
significant noise into the system, potentially compromising overall
effectiveness. This study prompts further inquiry into the specific drivers of
performance improvement, whether it be dataset balancing or the inclusion of
external data. Although our zero-shot approach exhibits limitations, it
underscores the positive correlation between data volume and model
performance, as illustrated by the superior performance of models trained on
larger paraphrased datasets, such as Para-n3 with 56k instances compared to
Para-Bal with 49k instances

Moving forward, to boost performance, it would be interesting to measure the
influence of the quality and similarity of the paraphrases on the performance.
Moreover, incorporating hierarchical predictions, possibly at a second-level
node, could improve scores further. Exploring the utilization of larger
multilingual models alongside language-specific datasets and experimenting
with various ensemble methods could be fruitful. Finally, considering the
integration of adversarial training or self-supervised learning techniques
might offer valuable avenues for improvement.

## Acknowledgements

The authors would like to thank the organisers of the SemEval shared task.
This work was financially supported by the Natural Sciences and Engineering
Research Council of Canada (NSERC).

## References

  * [1] D. Dimitrov, F. Alam, M. Hasanain, A. Hasnat, F. Silvestri, P. Nakov, and G. Da San Martino, “Semeval-2024 task 4: Multilingual detection of persuasion techniques in memes,” in _Proceedings of the 18th International Workshop on Semantic Evaluation_ , ser. SemEval 2024, Mexico City, Mexico, June 2024.
  * [2] D. Jurkiewicz, Ł. Borchmann, I. Kosmala, and F. Graliński, “ApplicaAI at SemEval-2020 task 11: On RoBERTa-CRF, span CLS and whether self-training helps them,” in _Proceedings of the Fourteenth Workshop on Semantic Evaluation_ , A. Herbelot, X. Zhu, A. Palmer, N. Schneider, J. May, and E. Shutova, Eds. Barcelona (online): International Committee for Computational Linguistics, Dec. 2020, pp. 1415–1424. [Online]. Available: https://aclanthology.org/2020.semeval-1.187
  * [3] J. Tian, M. Gui, C. Li, M. Yan, and W. Xiao, “MinD at SemEval-2021 task 6: Propaganda detection using transfer learning and multimodal fusion,” in _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , A. Palmer, N. Schneider, N. Schluter, G. Emerson, A. Herbelot, and X. Zhu, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 1082–1087. [Online]. Available: https://aclanthology.org/2021.semeval-1.150
  * [4] I. Chalkidis, E. Fergadiotis, P. Malakasiotis, and I. Androutsopoulos, “Large-scale multi-label text classification on EU legislation,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 6314–6322. [Online]. Available: https://aclanthology.org/P19-1636
  * [5] N. Lin, G. Qin, G. Wang, D. Zhou, and A. Yang, “An effective deployment of contrastive learning in multi-label text classification,” in _Findings of the Association for Computational Linguistics: ACL 2023_ , A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 8730–8744. [Online]. Available: https://aclanthology.org/2023.findings-acl.556
  * [6] S. M. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko, “Semeval-2018 Task 1: Affect in tweets,” in _Proceedings of International Workshop on Semantic Evaluation (SemEval-2018)_ , New Orleans, LA, USA, 2018.
  * [7] H. Alhuzali and S. Ananiadou, “SpanEmo: Casting multi-label emotion classification as span-prediction,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , P. Merlo, J. Tiedemann, and R. Tsarfaty, Eds. Online: Association for Computational Linguistics, Apr. 2021, pp. 1573–1584. [Online]. Available: https://aclanthology.org/2021.eacl-main.135
  * [8] R. Cerri, R. C. Barros, and A. C. de Carvalho, “Hierarchical multi-label classification using local neural networks,” _Journal of Computer and System Sciences_ , vol. 80, no. 1, pp. 39–56, 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0022000013000718
  * [9] J. Zhou, C. Ma, D. Long, G. Xu, N. Ding, H. Zhang, P. Xie, and G. Liu, “Hierarchy-aware global model for hierarchical text classification,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 1106–1117. [Online]. Available: https://aclanthology.org/2020.acl-main.104
  * [10] C. Yu, Y. Shen, Y. Mao, and L. Cai, “Constrained sequence-to-tree generation for hierarchical text classification,” _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:247939715
  * [11] G. Da San Martino, A. Barrón-Cedeño, H. Wachsmuth, R. Petrov, and P. Nakov, “SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles,” in _Proceedings of the Fourteenth Workshop on Semantic Evaluation_. Barcelona (online): International Committee for Computational Linguistics, Dec. 2020, pp. 1377–1414. [Online]. Available: https://aclanthology.org/2020.semeval-1.186
  * [12] D. Dimitrov, B. Bin Ali, S. Shaar, F. Alam, F. Silvestri, H. Firooz, P. Nakov, and G. Da San Martino, “SemEval-2021 task 6: Detection of persuasion techniques in texts and images,” in _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , A. Palmer, N. Schneider, N. Schluter, G. Emerson, A. Herbelot, and X. Zhu, Eds. Online: Association for Computational Linguistics, Aug. 2021, pp. 70–98. [Online]. Available: https://aclanthology.org/2021.semeval-1.7
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota, USA: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [14] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised Cross-lingual Representation Learning at Scale,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–8451. [Online]. Available: https://aclanthology.org/2020.acl-main.747
  * [15] N. F. Costa, B. Hamilton, and L. Kosseim, “CLaC at SemEval-2023 task 3: Language potluck RoBERTa detects online persuasion techniques in a multilingual setup,” in _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_ , A. K. Ojha, A. S. Doğruöz, G. Da San Martino, H. Tayyar Madabushi, R. Kumar, and E. Sartori, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 1613–1618. [Online]. Available: https://aclanthology.org/2023.semeval-1.223
  * [16] D. Li, C. Wang, X. Zou, J. Wang, P. Chen, J. Wang, L. Yang, and H. Lin, “914isthebest at semeval-2024 task 4: Cot-based data augmentation strategy for persuasion techniques detection,” in _Proceedings of the 18th International Workshop on Semantic Evaluation_ , ser. SemEval 2024, Mexico City, Mexico, June 2024.
  * [17] J. Wunderle, J. Schubert, A. Cacciatore, A. Zehe, J. Pfister, and A. Hotho, “Otterlyobsessedwithsemantics at semeval-2024 task 4: Developing a hierarchical multi-label classification head for large language models,” in _Proceedings of the 18th International Workshop on Semantic Evaluation_ , ser. SemEval 2024, Mexico City, Mexico, June 2024.

## Authors

Kota Shamanth Ramanath Nayak received B.Tech from Manipal Institute of
Technology, India. Currently, he is pursuing his Master’s in Computer Science
from Concordia University, Canada. His research interests include Natural
Language Processing, Discourse Analysis and Large Language Models.

Leila Kosseim obtained her PhD from the University of Montreal in 1995 on the
topic of Natural Language Generation. Currently she is a professor in the
Computer Science & Software Engineering (CSSE) Department at Concordia
University in Montreal. Her research interests include Natural Language
Processing, Artificial Intelligence and Text Mining.


# Neural networks embrace learned diversity

Anshul Choudhary  Nonlinear Artificial Intelligence Laboratory, Physics
Department, North Carolina State University, Raleigh, NC 27607, USA  Anil
Radhakrishnan  Nonlinear Artificial Intelligence Laboratory, Physics
Department, North Carolina State University, Raleigh, NC 27607, USA  John F.
Lindner  Nonlinear Artificial Intelligence Laboratory, Physics Department,
North Carolina State University, Raleigh, NC 27607, USA  Physics Department,
The College of Wooster, Wooster, OH 44691, USA  Sudeshna Sinha  Indian
Institute of Science Education and Research Mohali, Knowledge City, SAS Nagar,
Sector 81, Manauli PO 140 306, Punjab, India  William L. Ditto  Nonlinear
Artificial Intelligence Laboratory, Physics Department, North Carolina State
University, Raleigh, NC 27607, USA

###### Abstract

Diversity conveys advantages in nature, yet homogeneous neurons typically
comprise the layers of artificial neural networks. Here we construct neural
networks from neurons that learn their own activation functions, quickly
diversify, and subsequently outperform their homogeneous counterparts. Sub-
networks instantiate the neurons, which meta-learn especially efficient sets
of nonlinear responses. Such learned diversity provides examples of dynamical
systems selecting diversity over uniformity and elucidates the role of
diversity in natural and artificial systems.

Diversity is a hallmark of many complex systems in physics Anderson1972 ;
Bak1987  and in physics beyond physics Holovatch_2017 , including microscopic
cell populations neural_div_cell , marine and terrestrial ecosystems
diversity_ecosystem ; choudhary2021weak , financial markets may_economics ,
and social networks page2010diversity ; stability_and_complexity ;
may_wigner_generality . In particular, mammalian brains contain billions of
neurons with diverse cell types whose complex dynamical patterns are believed
responsible for the rich range of cognition, affect, and behavior
marcus2014atoms ; neural_div_brain_activity ; computational_div ;
intermediate_div . But despite the widespread appreciation of diversity in
neuroscience, researchers have just begun to explore the role of diversity and
adaptability in artificial neural networks heterogeneous_cell_types ;
neural_heterogeneity_natcomm ; Han2021 .

Figure 1: Progression from conventional artificial neural network to diverse
neural network to learned diverse neural network. Line thicknesses represent
weights $W$, circle thicknesses represent biases $b$, and sketches inside
circles represent activation functions $\sigma$. Information flows top-to-
bottom. Training adjusts the weights and biases to optimize outputs (top).
Multiple activation functions enable diversity within layers (center) and
increase the expressiveness of the neural network. Separate neural networks
(bottom) realize these activation functions when training adjust their weights
and biases (color), perhaps on a different schedule than the originals
(black), to further optimize the network.

Inspired by nature, artificial neural networks are nonlinear systems that can
be trained to learn, classify, and predict. Traditionally, artificial neural
networks contain identical neurons in each network layer (even if the layers
themselves differ). In this Letter, we diversify a neural network by varying
the neuron types within each layer. We flexibly realize the different neurons
using sub-networks, or networks-within-the-network, which we train along with
the overarching network. This meta-learning hospedales2020  generates potent
neuron activation function sets, suggestive of orthogonal spanning functions,
that increase the expressiveness and accuracy of the network.

After discussing related work and our motivation, we describe how meta-
learning diverse activation functions can generate better neural networks, as
measured by difficult classification and nonlinear regression tasks. Neuron
participation ratios elucidate the superior potential of heterogeneous
neuronal layers over homogeneous layers. Hessian matrix spectra illuminate the
geometric nature of optimizing minima. We end by discussing future work and
the potential for learned diversity to enhance neural networks, deep learning,
and our appreciation of diversity itself.

Researchers have recently begun to relax the rigid rules that have guided the
development and use of artificial neural networks. Gjorgjieva, Drion, and
Marder computational_div  investigate the computational implications of
biophysical diversity and multiple timescales in neurons and synapses for
circuit performance. Doty et al. heterogeneous_cell_types  show that the
_hand-crafted_ heterogeneous cell types can improve performance of deep neural
networks. Xie, Liang, and Song weight_div  demonstrate that diversity in
synaptic weights lead to better generalization in neural networks. Manessi and
Rozza manessi2018learning  investigate learning combinations of known neuronal
activation functions. Zelda and Sra diversity_dpp  construct diversity by
compressing neuronal subspace using determinantal point process. Lin, Chen,
and Yan lin2014network  suggest nesting neural networks inside neural
networks, and Hospedales et al. hospedales2020  survey the current meta-
learning landscape. Decisively, Beniaguev, Segev, and London Beniaguev613141
write, “We call for the replacement of the deep network technology to make it
closer to how the brain works by replacing each simple unit in the deep
network today with a unit that represents a neuron, which is already – on its
own – deep”, which is what we do here with our activation function neuronal
sub-networks.

Inspired by natural brains, feed-forward neural networks are nested nonlinear
functions of linear combinations of activities

$a^{\prime}\stackrel{{\scriptstyle\text{vec}}}{{=}}\sigma({\bf W}a+b),$ (1)

where the activation $\sigma$ is typically a saturating or rectifying function
and training strengthens or weakens the weights and biases $\mathbf{W}$ and
$b$ to minimize an error or loss function and optimize outputs.

Motivated by the well-studied mammalian visual cortex, varying neuronal
activation functions by layer is common. However, within each layer, the
activations are typically identical, as in Fig. 1 (top). Neural networks are
universal function approximators Cybenko1989 ; Hornik1991  and are often used
to model hypersurfaces, either in nonlinear regression or classification.
Varying the activations within a layer, as in Fig. 1 (center), should
therefore increase the expressiveness of the network by providing diverse
spanning basis functions. Furthermore, replacing the activations by neural
networks, as in Fig. 1 (bottom), and training them for optimal results should
increase the expressiveness even further. The training of the activation
neural networks can be on a different schedule than the training of the rest
of the network, and the activations so obtained can be extracted from the
neuronal sub-networks as interpolated functions and efficiently reused in
other networks addressing different problems.

Figure 2: Schematic stochastic gradient descent meta-learning. Under randomly
shuffled neural-network inputs $i$, weights and biases $\theta$ adjust to
lower loss levels $\mathcal{L}(\theta,\theta_{A},i)$ (during the meta-learning
inner loop), while periodically the activation weights $\theta_{A}$ open extra
dimensions and themselves adjust to allow even lower loss levels (during the
meta-learning outer loop). Rainbow colors code time $t$.

As an example of a learned diversity neural network (LDNN), construct a feed-
forward classifier neural network whose neurons are sub-networks that modify
base activation functions (like zero, identity, sigmoid, or sine functions).
Train the classifier with many input-output pairs. Quantify the difference
between the expected and correct classifications with an error or loss
function. Compute the gradient of the loss function with respect to the
classifier’s weights and biases, and lower the loss by shifting its weights
and biases down this gradient (inner loop). Periodically compute the gradient
of the loss function with respect to the sub-networks’ weights and biases
maclaurin2015gradientbased , and further lower the loss by shifting their
weights and biases down this gradient (outer loop). Repeat to improve
accuracy.

The classifier error or loss $\mathcal{L}(\theta,\theta_{A},i)$ depends on the
network weights and biases $\theta$, the sub-networks weights and biases
$\theta_{A}$ that instantiate the activations of hidden-layer neurons, and the
inputs $i$. The randomly shuffled inputs are the stochastic driver that
buffets the weights and biases as they adjust to lower levels (during the
meta-learning inner loop). Periodically the activation weights and biases open
extra dimensions or degrees of freedom to further lower the levels (during the
meta-learning outer loop), as in Fig. 2.

Here we implement github  learned diversity neural networks with one hidden
layer of 100 neurons and a cross-entropy loss function to classify the
MNIST-1D data set, a minimalist variation of the classic Modified National
Institute of Standard and Technology digits deng2012mnist ;
greydanus2020scaling . Each neuron type in the hidden layer is further
instantiated by a feed-forward neural network of 50 hidden units with
hyperbolic tangent activation functions. We obtain similar results for
different numbers of layers and different number of neurons per layer
supplement .

Figure 3: Meta-learning 2 activations for MNIST-1D classification. (a-b)
Activation functions $\sigma_{n}(a)$ evolve from a base sinusoid, with violet-
to-red rainbow colors encoding time $t$. (c) Violin plots summarize
distribution (including median, quartiles, and extent) of validation accuracy
$A$ for 50 fully connected neural networks of ReLU neurons (blue), type-1
neurons (yellow), type-2 neurons (orange), and a mix of type 1 and type 2
neurons (red). The mix of 2 neuron types outperforms any single neuron type on
average. Figure 4: Neural network MNIST-1D classification accuracy as a
function of network size. Box and whiskers plots summarize accuracy
distribution (including median, quartiles, extent, and outliers) for 100
initializations. Learning rate is optimized to avoid over-fitting but is the
same for all network sizes. Activation functions evolved from zero (the null
function) with similar results evolved from sine. Mixed networks of 2 neuron
types outperform pure networks on average for all sizes and outperforms both
single learned activation and traditional activations.

Figure 3 summarizes meta-learning the activation functions of neurons in the
hidden layer subject to the constraint of having two functions distributed
equally among the neuronal population. Figure 3(a) and (b) show the evolution
of the two activation functions, with time encoded as rainbow colors from
violet to red. Figure 3(c) shows violin plots demonstrating validation
accuracy for 50 fully connected neural networks composed of entirely $N_{1}$
type neurons (yellow), entirely $N_{2}$ type neurons (orange), and mixed type
with $N_{1}$ and $N_{2}$ distributed equally among hidden layer (red). With
the same training, the mixed network outperforms either pure network on
average. These results are robust with respect to network size, as summarized
by Fig. 4.

We obtain similar results for other tasks, including nonlinear regression of
the van der Pol oscillator vanderpol , which consists of a linear restoring
force and a nonlinear viscosity modeled by the differential equation

$\ddot{x}-\mu(1-x^{2})\dot{x}+x=0,$ (2)

where the overdots indicate time derivatives. The van der Pol oscillator can
model vacuum tubes and heartbeats and was generalized by FitzHugh Fitzhugh
and Nagumo Nagumo  to model spiky neurons. For viscosity parameter $\mu=2.7$,
we trained neural networks to forecast the phase space orbit of the
oscillator, as summarized by Fig. 5. On average the learned diversity neural
network outperforms either of its pure components as well as a homogeneous
network of neurons with sinusoidal activations.

To understand how mixed activation functions outperform homogeneous neuronal
populations, we estimate the change in the dimensionality of the network
activations. Start by constructing a neuronal activity data matrix X with $N$
rows corresponding to $N$ neurons in the hidden layer and $M$ columns
representing inputs. Each matrix element ${\bf X}_{ij}$ represents the
activity of the $i^{th}$ neuron at the $j^{th}$ input. Center the activity so
$\langle\bf X\rangle=0$. Construct the neural co-variance matrix ${\bf
C}=M^{-1}{\bf XX}^{T}$, which indicates how pairs of neurons vary with respect
to each other, and compute the participation ratio

$\mathcal{R}=\frac{(\operatorname{tr}{\bf C})^{2}}{\operatorname{tr}{\bf
C}^{2}}=\frac{\left(\sum_{n=1}^{N}\lambda_{n}\right)^{2}}{\sum_{n=1}^{N}\lambda_{n}^{2}},$
(3)

where $\lambda_{n}$ are the co-variance matrix eigenvalues. If all the
variance is in one dimension, say $\lambda_{n}=\delta_{n1}$, then
$\mathcal{R}=1$; if the variance is evenly distributed across all dimensions,
so $\lambda_{n}=\lambda_{1}$, then $\mathcal{R}=N$. Typically,
$1<\mathcal{R}<N$, and $\mathcal{R}$ corresponds to the number of dimensions
needed to explain most of the variance Gao214262 . The normalized
participation ratio $r=\mathcal{R}/N$.

Figure 6 plots the joint probability densities $\rho(A,r)$ for multiple
realizations of the Fig. 3 Learned Diversity Neural Network and homogeneous
competitors. The mix of two neurons types has the best mean accuracy $A$ and
normalized participation ratio $r$, suggesting that more of its neurons are
participating when the mix achieves the best MNIST-1D classification. In
contrast, homogeneous networks of neurons with popular activation functions
have lower accuracy and participation ratios reflecting their poorer
effectiveness.

To understand the impact of learned diversity on the geometric nature of loss-
function minima, we compute the spectrum of the Hessian matrix
$\mathbf{H}=\nabla^{2}\mathcal{L}$, which captures the curvature of the loss
function. Since, $\mathbf{H}$ is a symmetric matrix, all its eigenvalues are
real. A purely convex loss function would have a positive semi-definite
Hessian everywhere. However, in practice, the loss function is almost always
non-convex (with multiple spurious minima) due to the presence of permutation
symmetries of the hidden neurons perm_symmetries . Therefore, understanding
how diversity helps training find deeper minima is crucial.

Previous work suggests that flatter minima generalizes better to the unseen
data ghorbani2019 ; sankar2020 . For the Fig. 1 neural network meta-learning
two neuronal activation functions, we find that once training has converged,
the resulting minima from diverse neurons is flatter than from homogeneous
ones, as measured by both the trace $\operatorname{Tr}\mathbf{H}$ of the
Hessian and the fraction $f$ of its eigenvalues near zero:
$\operatorname{Tr}\mathbf{H}_{1}>\operatorname{Tr}\mathbf{H}_{2}>\operatorname{Tr}\mathbf{H}_{12}$
and $f_{1}<f_{2}<f_{12}$. If steep minima are harder for gradient descent to
locate, then the flatter minima engineered and discovered by learned diversity
neural networks imply enhanced optimization.

Figure 5: Meta-learning 2 activations for nonlinear regression of the van der
Pol oscillator. (a-b) Activation functions $\sigma_{n}(a)$ evolve from a base
sinusoid, with violet-to-red rainbow colors encoding time $t$. (c) Violin
plots summarize distribution of neural network mean-square error or loss $L$
for 50 fully connected neural networks of sine neurons (blue), type-1 neurons
(yellow), type-2 neurons (orange), and a mix of type 1 and type 2 neurons
(red). The mix of 2 neuron types outperforms any single neuron type on
average. Figure 6: Probability densities $\rho(A,r)$ versus accuracy $A$ and
normalized participation ratio $r=\mathcal{R}/N$ for multiple realizations of
the Fig. 3 heterogeneous network and three homogeneous networks with popular
activation functions hyperbolic tangent, Rectified Linear Unit
$f(x)=\max(0,x)$, and sine. Increased participation accompanies increased
accuracy, with the diverse network maximizing both.

Biomimetic engineering or biomimicry is design inspired by nature. Just as
monoculture crops can be fragile, while diverse crops can be robust Wetzel2016
, heterogeneous neural networks can outperform homogeneous ones. Here, we
highlight advantages of varying activation functions within each layer and
learning the best variation by replacing activations by sub-networks.

Conceptually, Learned Diversity Neural Networks discover novel sets of
activation functions, when most artificial neural networks use just one of a
small number of conventional activations per layer. Practically, mixes of
learned activations can outperform traditional activations – where even a
$1\%$ improvement can be significant – and the learned activations can be
efficiently reused in diverse neural networks. Future work includes optimizing
learned diversity by adjusting hyperparameters, applying learned diversity to
a wider range of regression and classification problems, testing diverse
neural networks for robustness robustness , investigating clustering of
learned activations, and applying learned diversity to different neural
network architectures, such as recurrent neural networks and reservoir
computers rc_classic ; optical_rc ; quantum_rc , as well as physics-applied
and physics-informed neural networks Tailin2019 ; Choudhary2020 .

Learned diversity offers neural networks sets of tailored basis functions,
which enhance their expressiveness and adaptability and facilitates efficient
function approximation. When given the ability to learn their neuronal
activation functions, neural networks discover heterogeneous arrangements of
nonlinear neuronal activations that can outperform their homogeneous
counterparts with the same training. Our work provides specific examples of
dynamical systems that spontaneously select diversity over uniformity, and
thereby furthers our understanding of diversity and its role in strengthening
natural and artificial systems.

###### Acknowledgements.

This research was supported by ONR grant N00014-16-1-3066 and a gift from
United Therapeutics. WLD thanks Kathleen Russell for the conceptualization of
the original idea along with many subsequent discussions.

## References

  * [1] P. W. Anderson.  More is different.  Science, 177(4047):393–396, 1972.
  * [2] Per Bak, Chao Tang, and Kurt Wiesenfeld.  Self-organized criticality: An explanation of the $1/f$ noise.  Phys. Rev. Lett., 59:381–384, Jul 1987.
  * [3] Yurij Holovatch, Ralph Kenna, and Stefan Thurner.  Complex systems: physics beyond physics.  European Journal of Physics, 38(2):023002, feb 2017.
  * [4] Hynek Wichterle, David Gifford, and Esteban Mazzoni.  Mapping neuronal diversity one cell at a time.  Science, 341(6147):726–727, 2013.
  * [5] David Tilman, Clarence L Lehman, and Kendall T Thomson.  Plant diversity and ecosystem productivity: theoretical considerations.  Proceedings of the national academy of sciences, 94(5):1857–1861, 1997.
  * [6] Anshul Choudhary, Arindam Saha, Samuel Krueger, Christian Finke, Epaminondas Rosa Jr, Jan A Freund, and Ulrike Feudel.  Weak-winner phase synchronization: A curious case of weak interactions.  Physical Review Research, 3(2):023144, 2021.
  * [7] R.M May, S.A. Levin, and G. Sugihara.  Ecology for bankers.  Nature, 451:893, 2008.
  * [8] Scott E Page.  Diversity and Complexity, volume 2.  Princeton University Press, 2010.
  * [9] Robert M May.  Stability and complexity in model ecosystems.  Princeton university press, 2019.
  * [10] Sitabhra Sinha and Sudeshna Sinha.  Evidence of universality for the may-wigner stability theorem for random networks with local dynamics.  Physical Review E (Rapid Communication), 71:020902, 2005.
  * [11] Gary Marcus, Adam Marblestone, and Thomas Dean.  The atoms of neural computation.  Science, 346(6209):551–552, 2014.
  * [12] Jean-Philippe Thivierge.  Neural diversity creates a rich repertoire of brain activity.  Communicative & integrative biology, 1(2):188–189, 2008.
  * [13] Julijana Gjorgjieva, Guillaume Drion, and Eve Marder.  Computational implications of biophysical diversity and multiple timescales in neurons and synapses for circuit performance.  Current opinion in neurobiology, 37:44–52, 2016.
  * [14] Shreejoy J Tripathy, Krishnan Padmanabhan, Richard C Gerkin, and Nathaniel N Urban.  Intermediate intrinsic diversity enhances neural population coding.  Proceedings of the National Academy of Sciences, 110(20):8248–8253, 2013.
  * [15] Briar Doty, Stefan Mihalas, Anton Arkhipov, and Alex Piet.  Heterogeneous ‘cell types’ can improve performance of deep neural networks.  bioRxiv.10.1101/2021.06.21.449346, 2021.
  * [16] Nicolas Perez-Nieves, Vincent C. H. Leung, Pier Luigi Dragotti, and Dan F. M. Goodman.  Neural heterogeneity promotes robust learning.  Nature Communications, 12:5791, 2021.
  * [17] Chen-Di Han, Bryan Glaz, Mulugeta Haile, and Ying-Cheng Lai.  Adaptable hamiltonian neural networks.  Phys. Rev. Research, 3:023156, May 2021.
  * [18] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  arXiv.2004.05439, 2020.
  * [19] Bo Xie, Yingyu Liang, and Le Song.  Diversity leads to generalization in neural networks.  arXiv.1611.03131, 1611, 2016.
  * [20] Franco Manessi and Alessandro Rozza.  Learning combinations of activation functions.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 61–66. IEEE, 2018.
  * [21] Zelda Mariet and Suvrit Sra.  Diversity networks: Neural network compression using determinantal point processes.  arXiv:1511.05077, 2015.
  * [22] Min Lin, Qiang Chen, and Shuicheng Yan.  Network in network.  arXiv.1312.4400, 2014.
  * [23] David Beniaguev, Idan Segev, and Michael London.  Single cortical neurons as deep artificial neural networks.  Neuron, 109(17):2727–2739, 2021.
  * [24] G. Cybenko.  Approximation by superpositions of a sigmoidal function.  Mathematics of Control, Signals, and Systems (MCSS), 2(4):303–314, December 1989.
  * [25] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural Networks, 4(2):251–257, 1991.
  * [26] Dougal Maclaurin, David Duvenaud, and Ryan P. Adams.  Gradient-based hyperparameter optimization through reversible learning.  arXiv.1502.03492, 2015.
  * [27] Our code is available at https://github.com/nonlinearartificialintelligencelab/diversityNN.
  * [28] Li Deng.  The MNIST database of handwritten digit images for machine learning research.  IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  * [29] Sam Greydanus.  Scaling down deep learning.  arXiv:1511.05077, 2020.
  * [30] See Supplemental Material at [URL will be inserted by publisher] for results with different numbers of neurons and layers, other tasks, and further details.
  * [31] Balth. van der Pol Jun. D.Sc.  Lxxxviii. on “relaxation-oscillations”.  The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):978–992, 1926.
  * [32] R Fitzhugh.  Impulses and physiological states in theoretical models of nerve membrane.  Biophys J, 1(6):445–466, Jul 1961.
  * [33] J. Nagumo, S. Arimoto, and S. Yoshizawa.  An active pulse transmission line simulating nerve axon.  Proceedings of the IRE, 50(10):2061–2070, 1962.
  * [34] Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli.  A theory of multineuronal dimensionality, dynamics and measurement.  bioRxiv.10.1101/214262, 2017.
  * [35] Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea.  Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML, volume 139, pages 9722–9732, 2021.
  * [36] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.  An investigation into neural net optimization via hessian eigenvalue density.  arXiv.1901.10159, 2019.
  * [37] Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian.  A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization.  arXiv.2012.03801, 2020.
  * [38] William C. Wetzel, Heather M. Kharouba, Moria Robinson, Marcel Holyoak, and Richard Karban.  Variability in plant nutrients reduces insect herbivore performance.  Nature, 539(7629):425–427, 2016.
  * [39] Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman.  On the robustness of convolutional neural networks to internal architecture and weight perturbations.  arXiv preprint arXiv:1703.08245, 2017.
  * [40] Jaideep Pathak, Brian Hunt, Michelle Girvan, Zhixin Lu, and Edward Ott.  Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach.  Physical review letters, 120(2):024102, 2018.
  * [41] Mushegh Rafayelyan, Jonathan Dong, Yongqi Tan, Florent Krzakala, and Sylvain Gigan.  Large-scale optical reservoir computing for spatiotemporal chaotic systems prediction.  Physical Review X, 10(4):041037, 2020.
  * [42] LCG Govia, GJ Ribeill, GE Rowlands, HK Krovi, and TA Ohki.  Quantum reservoir computing with a single nonlinear oscillator.  Physical Review Research, 3(1):013077, 2021.
  * [43] Tailin Wu and Max Tegmark.  Toward an artificial intelligence physicist for unsupervised learning.  Phys. Rev. E, 100:033311, Sep 2019.
  * [44] Anshul Choudhary, John F. Lindner, Elliott G. Holliday, Scott T. Miller, Sudeshna Sinha, and William L. Ditto.  Physics-enhanced neural networks learn order and chaos.  Phys. Rev. E, 101:062207, Jun 2020.