# Re-visiting Reservoir Computing architectures optimized by Evolutionary
Algorithms ††thanks: This work was supported by the GACR-Czech Science
Foundation project No. 21-33574K “Lifelong Machine Learning on Data Streams”.
Accepted manuscript to the 14th World Congress on Nature and Biologically
Inspired Computing (NaBIC), Seattle, WA, United States, December 14-16, 2022.
A revised manuscript will be published in the conference proceedings by
Springer in the Lecture Notes in Networks and Systems.

Sebastián Basterrech
Faculty of Electrical Engineering and Computer Science
VŠB-Technical University of Ostrava
Ostrava, Czech Republic
<EMAIL_ADDRESS>& Tarun Kumar Sharma
Department of Computer Science and Engineering
Shobhit University
Saharanpur, India
<EMAIL_ADDRESS>

###### Abstract

For many years, Evolutionary Algorithms (EAs) have been applied to improve
Neural Networks (NNs) architectures. They have been used for solving different
problems, such as training the networks (adjusting the weights), designing
network topology, optimizing global parameters, and selecting features. Here,
we provide a systematic brief survey about applications of the EAs on the
specific domain of the recurrent NNs named Reservoir Computing (RC). At the
beginning of the 2000s, the RC paradigm appeared as a good option for
employing recurrent NNs without dealing with the inconveniences of the
training algorithms. RC models use a nonlinear dynamic system, with fixed
recurrent neural network named the reservoir, and learning process is
restricted to adjusting a linear parametric function. However, an RC model has
several hyper-parameters, therefore EAs are helpful tools to figure out
optimal RC architectures. We provide an overview of the results on the area,
discuss novel advances, and we present our vision regarding the new trends and
still open questions.

_Keywords_ Recurrent Neural Networks $\cdot$ Evolutionary Algorithms $\cdot$
Reservoir Computing $\cdot$ Chaotic Systems $\cdot$ Echo State Networks

##  1 Introduction

Recurrent Neural Network (RNNs) are “powerful” tools for modeling time series
and solving machine learning problems with sequential data. A main
characteristic of neural circuits is their highly recurrent structure that is
composed by different types of neurons and synapses [38]. Then, even though an
RNN is a rough simplification of the neural system, the recurrent nets
preserve biological plausibility [27, 35]. They have also other advantages
with respect of feed-forward models, the most mentioned in the literature are
[48, 35]: “good” theoretical properties, “relative” success in modelling
chaotic systems, and real-world applications. However, RNNs are still not so
popular like other families of Neural Networks (NNs). Probably, one of the
reason for this, it is the fact that it is still hard to efficiently train
large recurrent networks. Even though, NN optimization is one of the most
investigated problems in the Machine Learning (ML) area, the problem has still
open questions and it is still drawing a lot of interest. Specially, in the
case of optimizing large recurrent topologies due to the complexity and
difficulties presented in this problem. In summary, the main problems in the
training algorithms for RNNs are: not guaranteed convergence, long training
times, and it is very difficult to store information in internal states for
long periods (long-range memory) [35, 23].

At the beginning of 2000s has started a new approach for designing and
training RNNs. The original motivation was to take advantage of the benefits
of the recurrences for memorizing the sequential data, avoiding the problems
of training recurrent networks. This approach was simultaneously developed
with names of Liquid State Machine (LSM) (LSM) [39] and Echo State Network
(ESN) [26]. Both models and some of their variations converged in a new trend
named Reservoir Computing (RC) [56]. An RC model is a NN with a recurrent
hidden-to-hidden structure (so-called reservoir) composed by fixed hidden-
hidden weights, and the rest of the model weights (readout weights) are
trained using classic supervised learning. The two canonical models (ESN and
LSM) compute the readout weights using a simple linear regression. It is a
distinctive characteristic of the method, that the weights of the recurrent
topology are randomly initialized and kept fixed during the learning process.
The training algorithm only focuses on adjusting a selected group of weights
in order of having a fast and reliable optimization step. The RC models share
this principle with other NN families, e.g. Extreme Learning Machines (ELM)
[25], where the hidden weights of first layers need not be tuned. Both
approaches are based on a static non-linear random projection from the input
space to a feature space, and the training phase is robust and fast. Even
though the apparent simplicity of the RC models, they have obtained
outstanding performances in several real applications [56, 35, 27, 22, 30, 5].
Specially, the large popularity and recognition have started after the record-
breaking success in modelling the Mackey-Glass attractor [28].

The initial NN design always has impact on the performance of the model. In
the case of an RC model, this effect may be even more dramatic. Because the
reservoir weights are frozen during the training process. The most common
approach of defining the recurrent weights is using a random distribution, and
then to scale them in order of satisfying some algebraic restrictions. In
addition to the scaling of the recurrent weights, the model has other initial
operations and global parameters. The most relevant global parameters are
related to the dimensionality of the feature space, stability of the recurrent
dynamics, memorization capacity of the system, non-linear transformations of
the activation functions, and type of the readout mapping [28, 2, 35, 11].
Most research effort has gone for understanding the impact of those
parameters. For more information see [24, 35, 54, 57]. However, to define good
hyper-parameters of an RC model is still often made by experience using a grid
search strategy. Even, sometimes it is made partially by brute force.

Around ten years ago, several works have applied the power of Evolutionary
Algorithms (EAs) for improving the RC performance. The algorithms have been
used for finding the global RC parameters mentioned above. Another work
direction has been to developed hybrid approaches that combine the
evolutionary aspects and the RC characteristics. In this article, we provide a
comprehensive global overview of the RC literature that contains EA. We
analyze how EAs are used, we present a critical vision of the area and discuss
the new trends. The survey is not exhaustive, but we cover many of the most
important works on the area. There are several surveys about RC models, for
instance [54, 35, 53]. However, so far as we know, this is the first overview
regarding the specific sub-field of RC models where EAs area applied for
optimizing the model architecture.

The remainder of this article is organized as follows. In the next section, we
formalize the context where in the RC models are applied. Thereafter, we
present the mathematical specification of the Echo State Network, its
properties and limitations. This is followed by the presentation of recent
developed RC architectures. Section 4 describes the applications of EAs for
improving the RC models. We ends with a final discussion about the still open
questions on the area.

##  2 Reservoir Computing Methods

###  2.1 Context

Let consider a learning dataset in a supervised context composed by $N$ input-
output pairs $(\mathbf{u},\mathbf{y})$, where the input space has dimension
${K}$ and output space has dimension ${L}$. For simplicity, we consider
indexed instances in discrete spaces. RC models have been used for solving
both temporal and non-temporal tasks [35]. By temporal tasks or temporal
learning, we refer to the situation when any instance $\mathbf{u}(t)$ depends
on the data $\mathbf{u}(t+\Delta)$ for some non-large $\Delta$ value. It is
the common situation of time-series, or sequential data. By non-temporal
tasks, we refer by the problems where the data instances are independent of
each other. In temporal learning, the common goal is to predict the next value
of the sequence (or input signal) given the previous values, i.e. to learn
$\mathbf{y}(t)$ having information of
$\mathbf{u}(t),\mathbf{u}(t-1),\mathbf{u}(t-1),\ldots$. In non-temporal
learning, the task is to learn $\mathbf{y}(t)$ given the current input
$\mathbf{u}(t)$. Here, we focus mainly on the case of temporal learning that
is much more suitable for RNNs. However, RC models also have been used on
time-independent data [1]. Furthermore, it has also been used in time-series
classification where each input instance (a signal in this case) is
independent of each other [36].

###  2.2 Recurrent Neural Networks

Given an input signal $\mathbf{u}(t)\in\mathds{R}^{K}$ the simple case of RNN
is described by a $N$-dimensional state $x(t)\in\mathds{R}^{N}$ following the
recurrence:

$\mathbf{x}(t)=f(\mathbf{x}(t-1),\mathbf{u}(t),\bm{\theta_{1}}),$ (1)

where $f(\cdot)$ is a regular parametric function (e.g. a linear combination
following by a regular function such as hyperbolic tangent, sigmoid forms,
etc.), $\bm{\theta}$ is the weight collection (model parameters). Note that,
expression (1) at each time step processes a parametric function with the
current input to the model, and the previous network state. When the network
is used for solving problems in a learning context, there is added an external
aggregation function with the form:

$\mathbf{y}(t)=g(\mathbf{x}(t),\bm{\theta}_{2}),$ (2)

where $g(\cdot)$ is also a parametric function where the parameters are
collected in $\theta_{2}$. The collection $\theta_{1}$ contains the weight
matrix with input to the hidden neurons, and the weights of the hidden-to-
hidden connections. The collection $\theta_{2}$ has the matrix with weights
from hidden-to-hidden neurons to the output neurons [41]. Obviously, there are
many other variations of RNN architectures, for more information we suggest
[43].

##  3 Echo State Networks

###  3.1 Model specification

The standard ESN model is a specific RNN case, then it has also the recurrent
form (1), with the particular composition:

$\mathbf{x}(t)=f\big{(}\mathbf{W}^{\rm{in}}\mathbf{u}(t)+\mathbf{W}^{\rm{r}}\mathbf{x}(t-1)\big{)},$
(3)

where $t$ is the discrete time index, $\mathbf{W}^{\rm{in}}$ is a $N\times K$
matrix with input-to-hidden weights, the $N\times N$ matrix
$\mathbf{W}^{\rm{r}}$ has the hidden-to-hidden weights. For simplicity, bias
terms are omitted on the equations (common practice because it is
straightforward to implement them by adding dummy terms). The model output is
given by:

$\mathbf{y}(t)=g(\mathbf{W}^{\rm{out}}\mathbf{x}(t)),$ (4)

where $f(\cdot)$ is a contractive function (Lipschitz function) and $g(\cdot)$
is an activation function. In the case of the $g(\cdot)$ as the identity
function, then the readout expression (4) is a linear model. The recurrences
have transitions given by (3), at each time stamp the transition processes
weighted information between the current state and the current input pattern.
This transition function is fixed for all the learning dataset, due to the
$\mathbf{W}^{\rm{in}}$ and $\mathbf{W}^{\rm{r}}$ are initialized and remain
fixed during the training process. Therefore, expression (3) is seen as a
static random projection of the input space. The projection sometimes is also
referred as expansion because the dimension $N$ is much larger than $K$ ($K\ll
N$). Only the parameters $\mathbf{W}^{\rm{out}}$ presented in (4) are adjusted
according to the learning problems. A common way of solving readout equation
(4) is to apply a classical tool such as Tikhonov regularization [35, 24].
Figures 2 and 2 illustrate the principle of ESN models. We can see the model
as a pipeline with two consecutive independent blocks presented in Fig. 2) and
Fig. 2). Another illustrative figure of the RC principle is shown in 3. The
reservoir nodes are illustrated by colored circles. Note, that there are
circuits among the black nodes. The most common design of the ESN is to have a
fully connected input-reservoir matrix, and also a fully connected reservoir-
output matrix. Fig. 3 is an illustrative simplification, and doesn’t have a
fully-connected input-reservoir nodes, neither reservoir-output nodes.

ReservoirInputExpansionProjection$\mathbf{u}(t)\in\mathds{R}^{K}$$\mathbf{x}(t)\in\mathds{R}^{N}$$\mathbf{W}^{\rm{r}}$$\mathbf{x}(t-1)$

Figure 1: Recurrent structure (reservoir) formalized in expression (3). The
reservoir is driven by an external input signal. Reservoir has randomly
connected fixed weights that don’t change in the learning process.

Supervised
learningProjectionTrainingOutput$\mathbf{x}(t)\in\mathds{R}^{N}$${\mathbf{y}}(t)\in\mathds{R}^{L}$$\mathbf{W}^{\rm{out}}$

Figure 2: Readout structure formalized in expression 2. The projected points
by the reservoir are taken as input of a supervised learning, e.g. linear
regression. The parameters $\mathbf{W}^{\rm{out}}$ are trained using the
empirical data.

Input layerLiquid/ReservoirOutput layer

Figure 3:  An example of the design of an Echo State Networks. Colored nodes
form circuits and represent the reservoir nodes. The dashed edges are the only
one with trainable weights.

###  3.2 ESN variations

The original model has been suffered some slight variations, we highlight the
following ones(in some cases it has been investigated models that combine at
the same type several of the variations presented here). ):

  1. 1.

Feedback connections [28]:

$\mathbf{x}(t)=f\big{(}\mathbf{W}^{\rm{in}}\mathbf{u}(t)+\mathbf{W}^{\rm{r}}\mathbf{x}(t-1)+\mathbf{W}^{\rm{fb}}\mathbf{y}(t)\big{)},$
(5)

where $\mathbf{W}^{\rm{fb}}$ is a weight $N\times L$ matrix with the feedback
connections going from the output neurons to the hidden structure. The
feedback also can be integrated with an arbitrary delay.

  2. 2.

Additional noise [45]:

$\mathbf{x}(t)=f\big{(}\mathbf{W}^{\rm{in}}\mathbf{u}(t)+\mathbf{W}^{\rm{r}}\mathbf{x}(t-1)+\bm{\varepsilon}(t)\big{)},$
(6)

to incorporate noise in the training procedures is a common used
regularization strategy in NNs.

  3. 3.

Leaky integrator ESN: the reservoir state variation is smoothed using a leaky
rate [35]:

$\mathbf{x}(t)=(1-\gamma)\mathbf{x}(t-1)+\gamma\phi\big{(}\mathbf{W}^{\rm{in}}\mathbf{u}(t)+\mathbf{W}^{\rm{r}}\mathbf{x}(t-1)\big{)},$
(7)

where $\gamma\in(0,1]$ is the control parameter referred as leaking rate. It
control the speed of the dynamic transitions, small values implies less
sensitivity to the input pattern [35].

  4. 4.

Using ideas from recursive Self–Organizing Maps (SOMs) [32, 4], the neurons
have the form: $\bm{z}(t)=(z_{1}(t),\ldots,z_{N}(t))$ through

$z_{i}(t)=\exp{\big{(}-\alpha||\mathbf{W}^{\rm{in}}_{i}-\mathbf{u}(t)||^{2}-\beta||\mathbf{W}^{\rm{r}}_{i}-\mathbf{x}(t-1)||^{2}\big{)}},$
(8)

for all $i\in N$, and then updates its $\mathbf{x}(t)$ vector using leaky
integrators:

$\mathbf{x}(t)=(1-\gamma)\mathbf{x}(t-1)+\gamma\bm{z}(t),$ (9)

where $\alpha$ and $\beta$ are the parameters to weight the euclidean distance
between inputs and $\mathbf{W}^{\rm{in}}$ and the previous state of the
reservoir and $\mathbf{W}^{\rm{r}}$, respectively.

Another type of variation are produced in the readout computation. It is
common to see the aggregation of the recurrent state (expression 2) in linear
form as follows:

$\mathbf{y}(t)=g(\mathbf{W}^{\rm{out}}[\mathbf{u}(t);\mathbf{x}(t)]),$ (10)

where $\mathbf{W}^{\rm{out}}$ is the $N\times(L+K)$ reservoir-to-output weight
matrix and $[\cdot;\cdot]$ denotes vector concatenation. In addition, there
exists the ESN with intrinsic plasticity [49], where there are parameters in
the activation functions of the reservoir units. Furthermore, an ESN with two
types of hidden-hidden connections (emulating excitatory and inhibitory
signals) was developed in [6, 7].

###  3.3 Properties

An important property of the reservoir is the Echo State Property (ESP)
presented in [26]. In essence, the ESP guarantees that under certain algebraic
conditions the reservoir state given by expression (3) becomes asymptotically
independent of the initial conditions ($\mathbf{x}(0)$) [35]. In the case of
LSMs this property is presented under the name of fading memory. In practical
applications, it is enough to scale the reservoir weights for almost always
ensured the ESP reservoir weight. Thus, the reservoir weights are typically
scaled as
$\displaystyle{\mathbf{W}^{\rm{r}}:=\frac{\alpha\mathbf{W}^{\rm{r}}}{\rho(\mathbf{W}^{\rm{r}})}}$,
where $\alpha$ is a scaling parameter between $0$ and $1$. The ESN property
can be obtained even if $\rho(\mathbf{W}^{\rm{r}})>1$ for non–zero input or
output feedback [32, 2]. Nevertheless, the ESN is violated if
$\rho(\mathbf{W}^{\rm{r}})>1$ using the $\tanh(\cdot)$ activation function in
the reservoir neurons and for zero input (no input) [35]. In this situation,
the internal state of the reservoir presents oscillations or even chaotic
dynamics [35]. Also, the property may be lost even if
$\rho(\mathbf{W}^{\rm{r}})<1$ although it is hard to build a reservoir where
this occurs [35].

##  4 Applications of Evolutionary Algorithms on the RC area

Evolutionary techniques are helpful for improving NN architecture, and they
can be applied for different purposes such as feature selection, weight
tuning, graph design, parameter adaptation, rule extraction, and so on [58].
Optimization of network weights using an evolutionary search were developed in
EVOLINO [47]). In recent years, the interest has been augmented due to the
fruitful novel areas of Neural Architecture Search and Neuroevolution [14,
51].

###  4.1 What are the global parameters to be optimized?

The RC family has also taken advantage of the recent advances in evolutionary
computation. EA techniques have been used for tuning the following global
parameters which affect the ESN’s performance.

  * •

Reservoir size. In many ESN applications, it has been shown that the reservoir
size influences the ESN accuracy. As in all learning systems, there is a
tradeoff to reach in the size of the reservoir. If it is too small, we don’t
exploit enough the benefits of separating the set of possible inputs inside a
larger space, and we can have poor training processes. If it is too large,
training is easy, but generalizing can have issues, the usual over–fitting
phenomenon.

  * •

Spectral radius of the reservoir weight matrix. The spectral radius influences
the memory capability of the model; a spectral radius
$\rho(\mathbf{W}^{\rm{r}})$ close to $1$ is appropriate for learning tasks
that require long memory, and a value close to $0$ is adequate for tasks
requiring short memory [26]. The role of the spectral radius is more complex
when the reservoir is built with spiking neurons in the LSM model [56, 44].

  * •

Sparsity of the reservoir weight matrix. It is recommended to define a sparse
matrix $\mathbf{W}^{\rm{r}}$ with only between $15\%$ and $20\%$ of possible
connections non-zero [26]. The sparsity enables fast reservoir updates. In
general, the matrices of input weights and feedback weights (in the case that,
they exist) are complete.

  * •

Injected noise. The use of a noise vector for regularization requires some
type of control of this noise.

  * •

Parameter of the activation function. The non-linearity of the reservoir
projection is given by the type of activation function on the reservoir
neurons.

  * •

Leaky rate. As was mentioned above, it controls the impact of the inputs in
the dynamics, a type of speed control. It is a scalar in $[0,1)$.

  * •

Feedback connections. A model with feedback connections is much more
computational powerful, but the instability increases. For this reason, it is
very hard problem to tune the vector with feedback connections [33].

  * •

Topology. In spite of the large amount of works investigating the topological
structure of the reservoir, a useful topology for significantly improving the
performance of the ESN is still unknown. The classical approach consists of
producing randomly connected reservoirs.

###  4.2 Summary of the literature

Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) have been used
as metaheuristics for finding the global parameters in [50, 16, 17, 15]. GAs
also were used for finding the parameters of an extended ESN in medical
applications PSO was applied as a pre-training algorithm has been investigated
in [3], where the authors applied PSO for adjusting a subset of the reservoir
weights. The authors instead of searching the global parameters, the paper
works directly over a subset of reservoir weights. In [12, 42] was studied
growing RC topologies using Neat and HyperNEAT. Deep architectures, sometimes
referred hierarchical reservoirs, that a kind of cascade of reservoir clusters
where they are connected among is a new promising area inside the RC family
[18, 20]. GAs has been used for finding hyperparameters of hierarchical
reservoirs and for tuning the weight connections among the reservoirs [37,
13]. A variation of GA named microbial GA was used in [13]. Recently, PSO was
introduced as a hybrid approach of swarm optimization and local search in
hierarchical ESNs [31]. EvoESN was introduced in [9, 8] where GA is used to
optimize the weights of compressed reservoirs. With the aim of finding good
reservoir topology, particular cyclic graphs were analyzed in [46, 45]. In
[49, 52], reservoir units with parametric activation functions were trained
for maximizing the entropy using the Intrinsic Plasticity (IP) rule. Other
approaches based on Evolutionary Algorithms (EAs) for optimizing hyper-
parameters of the reservoir structure, such as number of reservoir units,
scaling parameters, connectivity properties (density and spectral radius of
the reservoir weights matrix), have also been introduced in the literature
[16, 17]. EAs have also been used for optimizing interconnected reservoirs
[13, 37], where GAs are applied for searching hyper-parameters of hierarchical
reservoirs. Besides, Bayesian optimization for finding hyper-parameters of
particular reservoirs was also analyzed in [40, 55]. Furthermore, a relatively
new prominent research in the RC area consists of models with interconnected
reservoirs. A tree structure where the tree-nodes satisfies the reservoir
characteristics was developed in [18]. The weight connections among the
reservoirs are set in order of creating a global contraction mapping.
Furthermore, layered/hierarchical architectures composed of multi-layered
reservoirs have also been studied in [37, 19, 21, 13, 20, 22].

###  4.3 Discussion about new trends and open problems

We have commented several important parameters of an RC model, and give some
examples where EAs were used to optimize them. However, different reservoir
matrices with the same hyper-parameters may produce substantially different
results [49]. In other words, choosing good global parameters’ values at the
moment of designing the model doesn’t seem to be sufficient for fully
exploiting the potential of ESNs. We identify the following research
directions of interest.

  * •

Random projection dynamics. There is a general consensus that the ESN model
operates closes to the optimal situation when the projection is close to the
edge of chaos [35, 56, 29]. There are some works of EAs applied to the
Lyapunov exponents optimization. However, it is still possible to provide
insights on the specific area of Lyapunov exponents estimation using EAs (for
the particular case of dynamics coming from ESNs).

  * •

Quality of the projections. When the model is used for forecasting and
learning, then the projection should preserve some topological properties from
the input space. A preliminary work was presented using Sammon mapping [10],
but we believe there are still a lot of approaches to be investigated.

  * •

Multi-objective approach. In most of the works in the RC community,
researchers apply EAs over quadratic errors as a measure of the prediction
accuracy. However, there are other metrics for evaluating the good
characteristics of the model such as memory capacity, robustness, complexity
and speed. It would be an interesting direction apply multi-objective for
optimizing several different metrics.

  * •

Self-organized reservoirs. Even though was studied many years ago in [32, 34,
4]. The area related to self-organization has grown up in the last years due
to robotic applications, therefore most probably new developments can be done
in this area.

  * •

Lifelong learning paradigm. ESN model has very good properties for being used
in continual learning. Parameters of control for the catastrophic forgetting
problem, and other optimization problems of the lifelong learning area using
RC techniques can be studied using EAs.

##  5 Conclusions and future work

This paper presents a brief survey of optmization on the RC paradigm. We
presented the RC models optimized by EAs. As far as we are concerned, this
manuscript is the first overview about this specific sub-family of the RC
domain. We provided a comprehensive global overview of the RC literature that
can help to the reader for future implementation and studies of the RC and EA
families. We described in detail the main parameters to be optimized, and
provided a summary of the most important works of EAs used for finding good RC
architectures. We also discussed some drawbacks, opportunities and open
questions for the research community. This survey is not exhaustive, then the
first work to be done in the close future is to improve the literature review
and to collect new works referring the edge between EAs and RC.

## References

  * [1] Luís A. Alexandre, Mark J. Embrechts, and Jonathan Linton.  Benchmarking reservoir computing on time-independent classification tasks.  In Proceedings of the 2009 international joint conference on Neural Networks, IJCNN’09, pages 2376–2380, Piscataway, NJ, USA, 2009. IEEE Press.
  * [2] Sebastián Basterrech.  Empirical analysis of the necessary and sufficient conditions of the echo state property.  In International Joint Conference on Neural Networks, IJCNN’17, pages 888–896, Anchorage, AK, USA, 2017. IEEE Press.
  * [3] Sebastián Basterrech, Enrique Alba, and Václav Snášel.  An Experimental Analysis of the Echo State Network Initialization Using the Particle Swarm Optimization.  In Nature and Biologically Inspired Computing (NaBIC), 2014 Sixth World Congress on, pages 214–219, July 2014.
  * [4] Sebastián Basterrech, Colin Fyfe, and Gerardo Rubino.  Self-organizing Maps and Scale-invariant Maps in Echo State Networks.  In 11th International Conference on Intelligent Systems Design and Applications, ISDA 2011, Córdoba, Spain, November 22-24, 2011, pages 94–99, November 2011.
  * [5] Sebastián Basterrech and Pavel Krömer.  A nature-inspired biomarker for mental concentration using a single-channel eeg.  Neural Computing and Applications, 2019.
  * [6] Sebastián Basterrech and Gerardo Rubino.  Echo State Queueing Network: A new Reservoir Computing Learning Tool.  In 10th IEEE Consumer Communications and Networking Conference, CCNC 2013, Las Vegas, NV, USA, January 11-14, 2013, pages 118–123, 2013.
  * [7] Sebastián Basterrech and Gerardo Rubino.  Echo State Queueing Networks: A Combination of Reservoir Computing and Random Neural Networks.  Probability in the Engineering and Informational Sciences, 31:457–476, October 2017.
  * [8] Sebastián Basterrech and Gerardo Rubino.  Evolving Reservoir Weights in the Frequency Domain.  In 2021 Genetic and Evolutionary Computation Conference Companion (GECCO ’21 Companion). ACM, 2021.  doi: 10.1145/3449726.3459457.
  * [9] Sebastian Basterrech and Gerardo Rubino.  Evolutionary Echo State Network: evolving reservoirs in the Fourier space.  In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2022.  Available at: https://arxiv.org/abs/2206.04951.
  * [10] Tomas Burianek and Sebastián Basterrech.  Quantifying the reservoir quality using dimensionality reduction techniques.  In ESANN’2018), pages 443–448, Bruges, Belgium, April 2018.
  * [11] J. B. Butcher, D. Verstraeten, B. Schrauwen, C. R. Day, and P. W. Haycock.  Reservoir Computing and Extreme Learning Machines for Non-linear Time-series Data Analysis.  Neural Networks, 38:76–89, feb 2013.
  * [12] Kyriakos C. Chatzidimitriou and Pericles A. Mitkas.  A neat way for evolving echo state networks.  In Proceedings of the 2010 Conference on ECAI 2010: 19th European Conference on Artificial Intelligence, pages 909–914, NLD, 2010. IOS Press.
  * [13] Matthew Dale.  Neuroevolution of hierarchical reservoir computers.  In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’18, pages 410–417, New York, NY, USA, 2018. ACM.
  * [14] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural Architecture Search: A Survey.  Journal of Machine Learning Research, 20:1–21, 2019.
  * [15] A.A. Ferreira and T.B. Ludermir.  Evolutionary strategy for simultaneous optimization of parameters, topology and reservoir weights in echo state networks.  In Neural Networks (IJCNN), The 2010 International Joint Conference on, pages 1–7, July 2010.
  * [16] A.A. Ferreira and T.B. Ludermir.  Comparing evolutionary methods for reservoir computing pre-training.  In Neural Networks (IJCNN), The 2011 International Joint Conference on, pages 283–290, July 2011.
  * [17] Aida A. Ferreira, Teresa B. Ludermir, and Ronaldo R. B. De Aquino.  An Approach to Reservoir Computing Design and Training.  Expert Syst. Appl., 40(10):4172–4182, August 2013.
  * [18] Claudio Gallicchio and Alessio Micheli.  Tree Echo State Networks.  Neurocomputing, 101:319–337, 02 2013.
  * [19] Claudio Gallicchio and Alessio Micheli.  Deep echo state network (deepesn): A brief survey.  CoRR, abs/1712.04323, 2017.
  * [20] Claudio Gallicchio and Alessio Micheli.  Echo state property of deep reservoir computing networks.  Cognitive Computation, 9(3):337–350, Jun 2017.
  * [21] Claudio Gallicchio and Alessio Micheli.  Reservoir topology in deep echo state networks.  In Igor V. Tetko, Věra Kůrková, Pavel Karpov, and Fabian Theis, editors, ICANN’2019, pages 62–75, Cham, 2019. Springer International Publishing.
  * [22] Claudio Gallicchio, Alessio Micheli, and Luca Pedrelli.  Deep reservoir computing: A critical experimental analysis.  Neurocomputing, 268(C):87 – 99, 2017.  Advances in artificial neural networks, machine learning and computational intelligence.
  * [23] Alex Graves.  Supervised Sequence Labelling with Recurrent Neural Networks, volume 385 of Studies in Computational Intelligence.  Springer-Verlag Berlin Heidelberg, first edition, 2012.
  * [24] Allen Hart, James Hook, and Jonathan Dawes.  Embedding and approximation theorems for echo state networks.  Neural Networks, 128:234 – 247, 2020.
  * [25] Guang-Bin Huang, Qin-yu Zhu, and Chee-kheong Siew.  Extreme learning machine: Theory and applications.  Neurocomputing, 70(1-3):489–501, 2006.
  * [26] Herbert Jaeger.  The “echo state” approach to analysing and training recurrent neural networks.  Technical Report 148, German National Research Center for Information Technology, 2001.
  * [27] Herbert Jaeger.  Using conceptors to manage neural long-term memories for temporal patterns.  Journal of Machine Learning Research, 18:1–43, 2017.
  * [28] Herbert Jaeger and Harald Haas.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication.  Science, 304:78–80, April 2004.
  * [29] R. Legenstein and W. Maass.  Edge of chaos and prediction of computational performance for neural circuit models.  Neural Networks, 20(3):323–334, 2007.
  * [30] Decai Li, Min Han, and Jun Wang.  Chaotic time series prediction based on a novel robust echo state network.  IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 23(5):787–799, May 2012.
  * [31] Jianyu Long, Shaohui Zhang, and Chuan Li.  Evolving Deep Echo State Networks for Intelligent Fault Diagnosis.  IEEE Transactions on Industrial Informatics, 16(7):4928–4937, 2020\.
  * [32] Mantas Lukos̆evic̆ius.  Reservoir Computing and Self-Organized Neural Hierarchies.  PhD thesis, School of Engineering and Science. Jacobs University, december 2011.
  * [33] Mantas Lukoševičius.  Echo State Networks with Trained Feedbacks.  Technical Report No. 4, Jacobs University Bremen, 2007.
  * [34] Mantas Lukoševičius.  On self-organizing reservoirs and their hierarchies.  Technical Report 25, Jacobs University, Bremen, 2010.
  * [35] Mantas Lukoševičius and Hebert Jaeger.  Reservoir Computing Approaches to Recurrent Neural Network Training.  Computer Science Review, 3:127–149, 2009.
  * [36] Qianli Ma, Lifeng Shen, Weibiao Chen, Jiabin Wang, Jia Wei, and Zhiwen Yu.  Functional echo state network for time series classification.  Information Sciences, 373:1–20, 2016.
  * [37] Qianli Ma, Lifeng Shen, and Garrison W. Cottrell.  DeePr-ESN: A deep projection-encoding Echo State Network.  Information Sciences, 511:152–171, 2020.
  * [38] W. Maass.  Searching for principles of brain computation.  Current Opinion in Behavioral Sciences, 11:81–92, 2016.
  * [39] Wolfgang Maass.  Noisy spiking neurons with temporal coding have more computational power than sigmoidal neurons.  Technical Report TR–1999–037, Technische Universitaet Graz. Graz, Austria, 1999.
  * [40] J. R. Maat, N. Gianniotis, and P. Protopapas.  Efficient Optimization of Echo State Networks for Time Series Datasets.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–7, 2018.
  * [41] James Martens and Ilya Sutskever.  Training deep and recurrent networks with hessian-free optimization.  In Neural Networks: Tricks of the Trade (2nd ed.), pages 479–535. MIT Press, 2012.
  * [42] Filip Matzner.  Neuroevolution on the edge of chaos.  In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO’17, pages 465–472, New York, NY, USA, 2017. ACM.
  * [43] Razvan Pascanu, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio.  How to construct deep recurrent neural networks.  In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [44] H. Paugam-Moisy and S. M. Bohte.  Handbook of Natural Computing, chapter Computing with Spiking Neuron Networks.  Springer-Verlag, september 2009.
  * [45] Ali Rodan and Peter Tin̆o.  Minimum Complexity Echo State Network.  IEEE Transactions on Neural Networks, 22:131–144, 2011.
  * [46] Ali Rodan and Peter Tiňo.  Simple deterministically constructed cycle reservoirs with regular jumps.  Neural Computation, 24:1822–1852, 2012.
  * [47] J. Schmidhuber, D. Wierstra, M. Gagliolo, and F. Gomez.  Training Recurrent Networks by Evolino.  Neural Networks, 19:757–779, 2007.
  * [48] Jurgen Schmidhuber.  Deep learning in neural networks: An overview.  Neural Networks, 61:85–117, 2015.
  * [49] Benjamin Schrauwen, Marion Wardermann, David Verstraeten, Jochen J. Steil, and Dirk Stroobandt.  Improving Reservoirs using Intrinsic Plasticity.  Neurocomputing, 71:1159–1171, March 2008.
  * [50] A. T. Sergio and T. B.Ludermir.  PSO for Reservoir Computing Optimization.  In Artificial Neural Networks and Machine Learning-ICANN 2012, volume 7552 of Lecture Notes in Computer Science, pages 685–692. Springer Berlin Heidelberg, 2012.
  * [51] Kenneth O. Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen.  Designing neural networks through neuroevolution.  Nature Machine Learning, 1:24–35, 2019.
  * [52] Jochen J. Steil.  Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning.  Neural Networks, 2007.
  * [53] Chenxi Sun, Moxian Song, Shenda Hong, and Hongyan Li.  A review of designs and applications of echo state networks, 2020.  Available at Arxiv. Doi=10.48550/ARXIV.2012.02974.
  * [54] Gouhei Tanaka, Toshiyuki Yamane, Jean Benoit Héroux, Ryosho Nakane, Naoki Kanazawa, Seiji Takeda, Hidetoshi Numata, Daiju Nakano, and Akira Hirose.  Recent advances in physical reservoir computing: A review.  Neural Networks, 115:100–123, 2019.
  * [55] Gabriel Trierweiler Ribeiro, Joao Guilherme Sauer, Naylene Fraccanabbia, Viviana Cocco Mariani, and Leandro dos Santos Coelho.  Bayesian optimized echo state network applied to short-term load forecasting.  Energies, 13(9), 2020.
  * [56] D. Verstraeten, B. Schrauwen, M. D’Haene, and D. Stroobandt.  An experimental unification of reservoir computing methods.  Neural Networks, 20(3):287–289, 2007.
  * [57] Gilles Wainrib and Mathieu N. Galtier.  A local Echo State Property through the largest Lyapunov exponent.  Neural Networks, 76:39–45, 2016.
  * [58] Xin Yao.  Evolving artificial neural networks.  Proceedings of the IEEE, 87(9):1423–1447, September 1999.


# Complex hypergraph analysis of Australian MPs’ professional connections,
1947-2019

Eve Cheng
<EMAIL_ADDRESS>Danny Cocks
<EMAIL_ADDRESS>Pat Leslie
<EMAIL_ADDRESS>
###### Abstract

We propose a suite of methods to analyse the professional networks of MPs,
showing how to analyse weak-tie connections between legislators and the
connections between background characteristic attributes. Applied to a novel
dataset on the backgrounds of Australian MPs in the Australian Labor Party and
the Liberal Party of Australia (1947-2019), we show that our approach can help
to describe and explain the decline in working-class and trade unionist MPs
from the Labor Party, the homogeneous elitism of the mid-20th century Liberal
Party, and the increasing similarity of both parties’ professional networks,
occurring in the period of party cartellisation from the 1980s onward. Our
paper’s findings show that our method has clear potential for broader
applications in the study of political representation, diversity, and elite
political networks.

##  1 Introduction

Politicians’ background data is often used in quantitative political science
literature to analyze various political processes, including career success
[32, 25, 2, 13], substantive representation [6, 4], trends in political
recruitment and party organization [26, 31, 12], and other phenomena like
political speech or candidate nomination processes [7, 37, 15]. Tools for the
analysis of these data fall into two categories. The most widely used category
focuses on what we term ‘attribute analysis’ – the prevalence of particular
backgrounds, either as a descriptive indicator or as an explanatory variable
in the analysis of political outcomes. The second understands dyadic networked
relationships between politicians as a quantity of interest.

Attribute analysis uses categorical data analysis and regression modelling to
show how outcomes differ between politicians with readily available background
characteristics such as gender, profession and education. The approach has
been instrumental in explaining the descriptive and substantive representation
of women, class and race [11, 33, 29, 28] as well as the professionalisation
of politics [27, 10, 1] and many other political outcomes. Insofar as they are
used for inference, these methods treat politicians as independently and
identically distributed units of analysis. Of course, conceptually, the
treatment of background data in this manner is a methodological convenience,
ignoring the influence of networks in determining outcomes. But politicians’
interdependence is an indispensable component of any holistic explanation of
political outcomes.111At least, politicians’ own experience of their networks
seems an important part of how they explain their roles in parliament [36].
Studies show how being centrally positioned in a network might be beneficial
by analyzing proxies for that central position. Legislators who have a close
family tie with a former politician tend to be more successful in elections
and attaining higher office [14, 3, 38, 19, 39]; politicians with degrees from
elite universities are far more likely to be selected to cabinet positions
[40].

Direct analysis of politicians’ social or professional networks is far less
common. This is likely to be related to the fact that collection for analysis
of dyadic data structures has factorial growth; every politician added to the
data requires $n-1$ new data points, creating significant barriers for
researchers wishing to collect full data sets, particularly in historical
settings. There are some exceptions. Studies using last names have pieced
together network analysis of close family ties in 19th-century Chilean
politics [9]. Kinship ties have also been shown to affect resource allocation
among households in the Philippines [18]. However, for analysis of
professional networks, data collection is limited to office-centric networks,
treating a particular individual or office as the centre of the network, and
tracing professional, familial, contractual and donation connections outward
[23].

To bridge the gap between these two categories of analysis, we introduce a
attributes-as-networks method. This approach uses information from
politicians’ professional networks while collecting only individual-level
attribute characteristics as data. Our definition of a network tie differs
from a dyadic network. Instead of direct dyadic ties between individuals, we
analyse shared attribute characteristics using graph and hypergraph methods.
Approaching political networks in this way borrows conceptually from the
‘strength of weak ties’ [21, 20], a theory of network embeddedness that
hypothesises that acquaintances (weak tie connections) are more likely to be
influential in passing important information from a broader network than a
close friend. Recent causal analysis finds that this hypothesis is supported
by evidence from LinkedIn recruitment advertisements [34].

In this paper, we analyze the global features of MP and MP background
attribute networks to understand their structure and long-term trends in the
composition of MP networks for the Liberal Party of Australia (LP) and the
Australian Labor Party (ALP). An attributes-as-networks approach promises
several advantages over a attribute analysis approach: it allows for direct
structural analysis, incorporating global measures such as network
connectivity and directly modelling the dependence between politicians.
Instead of assuming that particular types of attributes such as elite
education, profession or dynasty membership confer networking advantages, we
ask: ‘How well networked are the attributes themselves?’. In this way, we can
examine the composition of weak-tie networks and answer new questions about
the mechanisms for networked advantage in politics.

We also show that we can go a step further to ask ‘What causes the networks to
be well/poorly connected?’. Our approach allows for a good deal of flexibility
in representing MP networks to separate underlying structures from noise.
Firstly, we constructed graphs for the MPs and hypergraphs for the attributes.
MP graphs, while intuitive, often share common attributes, forming complete
sub-graphs which in turn create redundancy and noise in the data. Hypergraphs
allow us to study the attribute networks to look at the networks from a
different perspective [30, 5]. In addition to the two sets of
graphs/hypergraphs, we can define different edge weights and s-filtration,
which allows us to see the role of strong/weak ties/communities in the
networks. A second flexibility comes from the choice of graph summary
statistics. We introduce two measures of network connectivity: average maximal
flow and transitivity. These measures can both show the level of connectedness
at a given time and identify certain network structures that contribute to
high/low connectivity. In some cases, a network composition that we call a
‘bouquet structure’ was prevalent in the professional networks of MPs. This
structure emerges in conditions where transitivity is higher than expected,
while average maximal flow is low. It resembles a bouquet in that two large
sub-network groups are separated by a very small number of individuals who
connect the two sub-networks together. This sub-network can give a very visual
representation of the structure of the networks. When transitivity is low and
average maximal flow is high, we sometimes expect some weak structures that
consist of ‘circles’ that consists of more than three nodes. These circles can
be identified as clusters with weak correlation of higher orders, which again
can be visually helpful when analysing network structures.

The third flexibility comes from the statistical analysis of the networks. We
use random graph generation as statistical baseline as comparison. Two sets of
random graph generation were generated. The first set assumes the politicians
each year have almost absolute freedom to choose their own backgrounds
randomly, given the background options that year. The second set, on the other
hand, restricts that freedom and assumes the number of MPs in each background
shall be fixed. The purpose of the first set of the random graphs is to
observe the relative magnitudes of the summary statistics compared to that of
the random distribution of MPs and random structure of the networks. The
second set focuses on the effects of random structure of the networks only. By
simulating networks with different assumptions about the properties of the
networks, we can narrow down the sources for the high/low connectivity for
each year against a baseline simulation.

In our MP network analysis, we find that the bouquet structure is of
particular importance. It typified ALP party organisation in a period where
trade unionists with backgrounds in manual labour were routinely selected as
candidates. While loosely connected by their union experience, they had little
in common with the elite wing of the party whose backgrounds were
professional, university-educated and with military experience. In our example
from the ALP’s network from 1960, the position of chief whip was held by
George Lawson, a figure of high network centrality who had a rare combination
of military and trade unionist background attributes. This is a structure that
scholars of the history of labour movements would recognise. Our analysis of
attribute networks shows that the centrality of the trade unionist and manual
labourer attributes in the ALP was low from the beginning and decreased over
time. These attributes were not connected via education or military service to
the rest of the professional network of the ALP and dwindled in number.
Conversely, the attribute network of the mid-20th century LP was characterised
by higher-than-average connectivity in average maximal flow and connectivity.
This implies that the backgrounds of LP MPs of this period were sufficiently
interchangeable to be largely immaterial to party organisation. Over time, the
distinctiveness in the MP and attribute networks has decreased, as party
recruitment has converged on a cartellised recruitment model. Taken together,
these findings corroborate and shed new light on studies of party organisation
convergence internationally and in Australia [26, 31].

Our paper is arranged as follows, we begin with our application to
professional networks among Australian MPs, defining our graph and hypergraph
in detail. We then introduce the two summary statistics we use to describe the
connectivity of the networks each year. We then describe our two methods for
simulating random graphs as statistical analysis. The results are split into
two sections, discussing our findings with respect to MP networks and
background attribute hypergraphs.222A brief note on terminology: in this paper
we use the terms graphs and networks interchangeably. Our analysis of MP
networks follows the usual terminology for network analysis. Our hypergraphs
of MP attributes reduce to s-line graphs (discussed in Section A), which are
then analysed using the same measures of connectivity as the MP networks. Our
use of the phrase ‘MP background attributes’ or similar refers to particular
background characteristics that may (or may not) be shared by other members of
an MP’s parliamentary party. We conclude by summarising our argument and
suggesting further avenues for study.

##  2 Method

In this section we introduce how we define the networks using graph/hypergraph
theory, the summary statistics we use to describe the networks, and the
different random graph simulation methods used to provide statistical
analysis. These sections serve as three linked but independent sections that
show the advantages of using graph/hypergraph theory to study network
structures.

First we define the networks of both MPs and attributes using graphs and
hypergraphs. Following the definitions, we introduce the two summary
statistics we focus on in this paper: the average maximal flow and the
transitivity. We then present the random graph simulations to compare with the
real data to provide the statistical analysis for our result. The preliminary
background for graph and hypergraph theory is in the appendix.

###  2.1 Graphs and hypergraphs definition

Figure 1: This scheme describes both the graph set-up for MP networks and the
hypergraph setup for the attribute networks. Both the MP networks and the
attribute hypergraphs have MPs as nodes and shared attributes as edges. The
attribute hypergraphs are mapped to two sets of s-line graphs. The first set
is undirected s-line graphs, and the second set is directed s-line graphs.

For our analysis, we want to capture the important global structural features
in both the MP networks and attribute networks (see Figure 1). The MP networks
uses MPs as vertices and we draw ties among the MPs who shared background
attributes such as party, education, military service and prior occupation as
edges. How closely the ties are (or geodesic) is reflected in the definition
of edge weights in the adjacency matrices for the graphs:

$\displaystyle w_{ij}=\sum_{\text{shared attribute k}}\frac{1}{\text{No. MPs
with attribute k}}$ (1)

where $i$, $j$ refer to indices for MPs.

Intuitively, the difference between graphs and hypergraphs is the way we draw
ties (or edges) among the entities (or nodes) of the networks. In graphs, we
define ties only between pairs of entities. While in hypergraphs, the ties can
be drawn among any number of entities. For example, if person A, B, C all
share a close family tie, the ties in graphs would be represented using three
ties, A-B, A-C, B-C. In hypergraphs, such close family tie is represented by
one hyperedge: A-B-C. Because of this feature, hypergraphs are often
considered a cleaner, less noisy approach.

While analysing many mathematical structures, it is often beneficial to study
the dual aspect of said structure in order to gain further insights. If we
view the MP networks as collapsed MP hypergraphs which use the career
attributes as nodes and MP as hyperedges, we see that the dual structure of
the MP hypergraphs is the attribute hypergraphs. In our definition of the
attribute hypergraphs, we take the MPs as vertices and attributes as
hyperedges. In contrast to the MP networks, which record all the pair-wise
relationships among MPs, the attribute hypergraphs record the set-wise
relationships among the attributes. Because of the set-wise feature, we can
further analyse the hypergraphs through a “flattening” procedure and reduce
them to s-line graphs.

The “flattening” procedure is where the hypergraph approach cleans up the
noise and produces our attribute networks. In MP networks, each attribute, for
example, being an alumni of University of Sydney, causes a great blob of all
MPs who went to University of Sydney. In the “flattened” attribute networks,
each great blob is reduced to a single node of University of Sydney. The
result from this “flattening” procedure are ordinary graphs. But we call them
line-graphs to remind ourselves of their hypergraph origin. The s in s-line
graphs corresponds to the filtering of the ties among the attributes. Because
the intersection between hyperedges can contain arbitrary number of MPs, we
can filter the line-graphs based on the strength of the ties. The s
corresponds to the minimum number of MPs who live in the intersection before
the tie is acknowledged in the attribute networks. Further technical details
see Appendix A.

The networks are named for the feature which forms the vertices of the graph:
the MP network focuses on MPs as vertices, whereas the attribute network, when
converted to an s-line graph, represents attributes as vertices of a graph.

Other than noise deduction, hypergraphs also introduce diversity into the
definition of the strength of ties. There are two sets of s-line graphs to
which we map the attribute networks (see Figure 1). For the first set of
s-line graphs, which we call undirected s-line graphs, we do not define any
direction between the ties. The ties between, for example, being an alumni of
the Australian National University and holding the office of Prime Minister,
only reflect how strongly the correlation is with no orientation.
Additionally, we can choose whether to reflect the heterogeneous strengths of
the ties in this set of s-line graphs. So the first set as undirected s-line
graphs can be further divided into weighted or unweighted graphs. For
unweighted undirected s-line graphs, we set all edges of the s-line graphs to
be uniformly 1.

For weighted undirected s-line graphs, we want to reflect a normalised
strength of the ties through weights of the edges. The weights of the edges
are calculated from the parent hypergraph the line-graph originates from:

$\displaystyle w_{i,j}=\frac{\text{no. MPs in }e_{i}\cap e_{j}}{\text{no. MPs
in }e_{i}\bigcup e_{j}}$ (2)

where $i$ and $j$ are indices for the attributes, $e_{i},e_{j}$ represent the
two hyperedges corresponding to two attributes. This is equivalent to the
Jaccard index for the pair-wise sets of hyperedges. We normalise the edge
weight with the sum of the size of the hyperedges to ensure that the edge
weight reflects the proportion of the overlap between the hyperedges and the
size of the two hyperedges.

The second set of s-line graphs are directed s-line graphs, in which we split
each edge of the undirected graphs into two edges, one for each direction. We
introduce the extra direction on the ties between attributes in hope to
capture a sense of flow between the attributes: how likely it is for someone
to be in the military given their university background? Similar to the
transition rate in statistical physics, instead of defining a single edge
between a pair of nodes, we distinguish between the edge $e_{i,j}$ and
$e_{j,i}$. The weights in the directed s-line graphs also reflect this
asymmetry in the edges:

$\displaystyle w_{i,j}=\frac{\text{no. MPs in }e_{i}\cap e_{j}}{\text{no. MPs
in }e_{i}}$ (3)

Here we can see that the diversity of graph/hypergraph definitions provides
the flexibility for investigating different properties of the networks easily
and directly. This will be further explored in the result section.

We calculate the summary statistics for graphs and line-graphs with different
edge weights. Since there is only one edge weight definition for the MP
networks, the analysis concerning different edge weights is only relevant for
the attribute networks. The different edge weights can show how the strength
of the connections among hyperedges affects the overall networks. Intuitively,
the unweighted line graphs report lower/higher connectivity than the weighted
ones if there are fewer/more weakly connected edges. If the weighted
undirected and directed both exhibit higher/lower connectivity compared to the
unweighted option, we will have good reasons to believe that it is due to the
amount of weakly/strongly connected edges in general. But what about when the
different weighted options disagree with one another?

To interpret the differences between these two weighted options, we have two
predictions (see Figure 2(c)). The first prediction is that the directed
weighted option is more sensitive to the asymmetry of the edges when one of
the edges is very small, but less sensitive if one of the edges becomes very
large, assuming the other edge is average. In other words, with $\text{no.
nodes in }e_{i}\cap e_{j}$ (see Section 2.2) fixed if we reduce/increase the
size of $e_{i}$ to create more asymmetry between $e_{i}$ and $e_{j}$, the
relative changes reflected in the average maximal flow in the directed
networks will be larger/smaller than that in the undirected one.

The second prediction is that the increase in the overlap ($\text{no. nodes in
}e_{i}\cap e_{j}$) is reflected more in the average maximal flow in the
undirected weighted networks compared to the directed networks (more details
in the appendix).

Finally, the s-filtration preserves the highly connected part of the networks
and filters out the weak edges (definitions see Appendix A). This has a
similar effect to the weighted edges. However, since the filtration weakens
the contribution from the weak edges by deleting them completely, the result
is expected to be more dramatic than the edge weights. By filtering out the
weak edges, we believe that the filtering can reveal the more important,
close-knit structures in the networks.

###  2.2 Summary statistics: measures of connectivity

In this paper, we focus on global features that characterise the connectivity
of both MP and attribute networks defined above. The measures we introduce are
the average maximal flow and the transitivity. While both allow for summary
statistics of network connectivity, our analysis shows that they measure
different aspects of professional networks and complement each other in
describing the politics of professional networks.

The average maximum flow is used to study the changes in the structure of the
graphs over the years. It can be calculated as follows:

$\displaystyle\text{averF}=\frac{\sum_{\\{(i,j)|i,j\in V,i\neq
j\\}}F(i,j)}{|\\{(i,j)|i,j\in V,i\neq j\\}|}$ (4)

where V is the set of all vertices in the network, F takes any two nodes and
returns the maximal flow between them. The edge weights defined above are
passed as the capacity for each edge in E in the form of a map
$c:E\rightarrow\mathcal{R}^{+}$, which reflects the maximum flow that each
edge can allow [16].

Transitivity is a topological parameter that gives information on how
clustered the networks are [41, 17]. It is defined as:

$\displaystyle T(G)=\frac{3\times\text{number of triangles}}{\text{number of
connected triples}}$ (5)

It can be loosely understood as the probability of A being related to C,
knowing A is related to B and B is related to C. The higher the value, the
more closely knit the community is.

The two measures of connectivity, average maximal flow and transitivity, are
simple to interpret individually: the larger the measure, the more connected
the network is. However, we would like to know more about the kind of
connectivity present. Unfortunately, the two measures are not directly
comparable numerically and may have different dependence on graph size. By
analysing simulated networks, which are discussed below, we can directly
compare the relative connectivity for the two measures. In particular, when we
say one measure suggests higher/lower connectivity than the other in this
section, we mean that the relative difference between real and simulated is
higher in one measure than the other. Because the ways the two connectivity
measures are calculated is different, they can disagree with one another. The
disagreements can be interesting as they inform us about the structures of
professional networks among MPs.

(a)

(b)

(c)

Figure 2: (a) an example of a bouquet structure, with one node connecting two
complete networks. (b) an example of a satellite structure, with a compact
structure in the middle and scattered disconnected components. (c)
asymmetrical bonding between a large edge and a small edge, where N1 and N2
represent the number of nodes the edges possess and $N1<<N2$ and n is the
number of nodes that are in the overlap region.

When the transitivity suggests much higher connectivity than average maximal
flow, we observe two sub-structures that appear to explain this anomaly (see
Fig 2(a)-(b)). The first is the bouquet structure, which consists of a
connecting small structure (in the limiting case a node) and several connected
components being linked by it. In our data, a bouquet structure implies that
the professional networks of MPs are divided among two or more densely
networked groups. The average maximal flow is very sensitive to this structure
since the average maximal flow can decrease dramatically in the bouquet
structure, while the transitivity can be largely unaffected. The details on
the justification for this claim can be found in the appendix C. Another sub-
structure that can contribute to this phenomenon is the satellite structure
(see Fig 2(b)). In this case, since transitivity does not take into account
the number of unconnected nodes, it is not reduced by an increased number of
satellites, while the average maximal flow is greatly affected.

Finally, since transitivity focuses mainly on the connected triples while
maximal flow considers larger structures, it is also common to have
unimpressive transitivity while maximal flow suggests high connectivity. It
just means that the structure is highly connected but it is not always
possible to go directly from one node to another. For example, structure such
as circles of nodes that are made up of more than three nodes, like squares,
pentagons, or more can potentially lead to this observation.

###  2.3 Random graph simulations

In this section, we introduce the statistical analysis of our result using
random graph simulations. We do these simulations for two reasons. First, we
wish to take into account the particular dependency of the average maximal
flow measure on the size of the graphs, the number of attributes present each
year and the number of unknown present in the historical records. The size
dependency can be observed in a simple thought experiment of complete graphs
of size n, whose average maximal flow can be shown to be n-1, which is
linearly dependent on n. Second, by using these simulations to “fit” the real
summary statistics, we find out the defining characteristics of the networks.
If we encounter great difficulty in fitting the real statistics, we can then
explore any additional higher dimensional topology that are entirely missed by
conventional means [35].

We call the first set of simulations free choice simulations. In this set of
simulations, all MPs are “free” to randomly choose their career backgrounds
from those available each year. More precisely, for each year, 100 random
networks are generated for LP and ALP respectively. Each random network
preserves the number of MPs, the attributes, the ratio of MPs with military
backgrounds and unknowns present each year but allows their relative
proportions to vary. This is equivalent to taking the same number of MPs each
year and randomly shuffling them into different attributes while the number of
attributes and the ratio of military attendance remain roughly the same. The
reason we kept the ratio of military attendance roughly the same is because we
observed that the military background is overwhelmingly influential,
especially in early years. With that influence in mind, we aim to observe the
additional influence coming from other attributes. This is an important set of
simulations as they tell us how much the real data deviates from what we
expect given the available options in terms of career in that year.

We call the second set of simulations shuffle. Faithful to the name, the
simulations preserves the number of edges in the networks and merely shuffles
them around. This set of simulations focus on the additional effects from the
distribution of nodes among the hyperedges, and it poses the question how much
of the deviations we observe from the first set of simulations were
contributed by the inequality in the number of MPs in each attribute?

##  3 Data

The data consists of the career backgrounds of the parliamentary members from
1947 to 2019. The data contains the career background information for all the
MPs each year including their unique identifier, party, university background,
subject at university, their principal occupation before their election, and
military experience. Our data is sourced from the Australian Parliamentary
Library’s Parliamentary Handbook, compiled by the Authors.333We are happy to
share with interested researchers. A skeleton structure for the data is
available as an R package at www.github.com/palesl/ausPH. Our graph structure
information is stored in graph database software developed by the authors for
easy manipulation of graph and hypergraph structures.

##  4 Results

###  4.1 MP networks

(a) free choice; average flow

(b) free choice; transitivity

(c) shuffle; average flow

(d) shuffle; transitivity

(e)

(f)

Figure 3: (a) and (b) show the comparison between the simulation (free choice)
and real MP networks for average maximal flow and transitivity respectively.
The error bars on the simulation data are the standard deviations of the 100
simulations calculated for each year, with the simulation line being the
average of the 100 simulations. (c) and (d) show the same comparison with (a)
and (b) except against the shuffle simulations. (e) the MP network for ALP in
1960. The bouquet structure is obvious with one end being the trade unionists.
(f) the slight clustering structure caused by many bouquet sub-structures (the
LP MP network at 2003).

In the MP networks, MPs are the nodes and the edges represent shared
attributes between the pairs of MPs. Figure 3 shows both the average maximal
flow and transitivity of the MP networks compared to the both free choice and
shuffle simulation networks for each major party over time. Comparing the
simulated average maximal flow and transitivity, we observe that the simulated
average maximal flow from both methods overall rise while an opposite trend is
observed in transitivity. This is expected as the average maximal flow has a
dependency on the network size, and the number of MPs increases over the
years. On the other hand, transitivity is not affected much by the size of the
networks. The opposite trend in transitivity is not surprising as the MP
networks are expected to be less connected over time because of the increase
in the number of attributes over time (see Figure 7). More career choices for
the MPs lead to less connected networks.

The more complex structural information of the real networks beyond simple
descriptors such as size and number of career choices lies in the deviations
between the real network and the simulations. According to Figure 3(a), the
major difference between real data and the free choice simulations for the
Labor Party in average maximal flow occurs before 1980. The real average
maximal flow seems significantly lower than the simulations. Interestingly,
the transitivity was much higher than simulations before 1980.

This is, we believe, a result of two things. The first is that we observed
many bouquet substructures before 1980. The bouquet substructure contributes
to high transitivity and low maximal flow because of the flow-limiting centre.
This observation is also supported by the high skewness of the degree
distribution in Figure 6. This suggests that the ALP at this time was divided
between groups of densely connected MPs, connected by a small number of ‘go-
between’ MPs with high centrality. In 1960, the central figure (ID: KZX) was
George Lawson, the only MP with both experience in the military and the trade
union movement. Lawson appears to have been notable for his ability to act as
a link between the two branches of the party’s professional networks, serving
in the cabinet as Chief Whip.

Secondly, before the 1980s, there were high percentages of trade unionists
among MPs without university backgrounds. This leads to low average maximal
flow because it means that the trade unionists, who made up approximately 40
per cent of the parliamentary party, compose one weakly connected complete
subgraph (see Figure 3(c)). This effect is lessened in the comparison with
shuffle simulation. By fixing the number of MPs as trade unionists, we observe
that the low average maximal flow has been elevated to the simulation
baseline. However, this agreement in average maximal flow between shuffle
simulation and real data does not mean the distribution of MPs in their career
backgrounds can fully reproduce the networks. This is evident in the
comparison between shuffle simulation and real data in transitivity. The
persistent high transitivity before 1980s shows the importance of the bouquet
structures as previously mentioned.

For the Liberal Party, the most prominent deviations in average maximal flow
against free choice simulation are 1960-1975 and again from 1990-2015, with
the first period possessing higher than average and the second phase lower
than the average. In contrast to the figure for comparison against free choice
transitivity, the first period (1960-1975) is only slightly lower than the
simulation average and the second period (1990-2015) higher than average.

In terms of the first phase (1960-1975), while the free choice simulation
suggests higher connectivity of LP MP networks, the shuffle simulation shows
very little discrepancy. It shows that the high connectivity shown against
average maximal flow from free choice simulation most likely comes from the
distribution of MPs, a large proportion of which are employed professionals
(lawyers, managers). Unlike the trade unionist trait in Labor Party, this
largely shared trait did not reduce average maximal flow. This is because
unlike trade unionists, who are relatively poorly connected to the rest of the
party via other attributes such as education and military service, MPs with
professional backgrounds share many common traits with others. Once the high
number of MPs who were employed professional was taken into account, little
difference was observed between the simulation and the real data.

In the second phase (1990-2015), we observe weak clustering behaviour caused
by many bouquet substructures, similar to the structure in the ALP prior to
1980, explaining why the average maximal flow is low, while the transitivity
is high (see Figure 3(d)). This difference is lessened in the shuffle
simulations but still visible. In this case, we find that the reason for the
formation of bouquet structures was due to a relatively recent influx of MPs
whose prior occupations were not as employed professionals but as senior
management within large organisations. Two prime ministers exemplify this
generational shift in candidate selection within the Liberal Party: John
Howard and Tony Abbott. While Howard had worked as a solicitor for 12 years
before entering parliament [24], Abbott had worked as a journalist and manager
of a concrete plant. Most consequentially, he had been director of Australians
for Constitutional Monarchy, the group that would lead the ‘Vote No’ campaign
to victory in the 1999 Republic Referendum.

###  4.2 Attribute networks

Uniform weights

(a) s=0, free choice

Normalised weights

(b) s=0, free choice

Directed normalised weights

(c) s=0, free choice

(d) s=0, shuffle

(e) s=0, shuffle

(f) s=0, shuffle

(g) s=2, free choice

(h) s=2, free choice

(i) s=2, free choice

(j) s=2, shuffle

(k) s=2, shuffle

(l) s=2, shuffle

Figure 4: The diagrams (a)-(c) and (g)-(i) show the comparison between
simulation and real attribute networks in terms of changes of average maximum
flow over the years for both Liberal and Labor parties for free choice
simulation, s=0,2, and edge weight being uniform, normalised and directed
normalised weight from left to right. The diagrams (d) - (f) and (j) - (l)
show the similar set of comparisons with shuffle simulation. The error bars on
the simulation data are the standard deviations of the 100 simulations
calculated for each year, with the simulation line being the average of the
100 simulations.

(a) s=0, free choice

(b) s=2, free choice

(c) s=0, shuffle

(d) s=2, shuffle

Figure 5: The diagrams (a) - (b) show the comparison between transitivity of
s-line graphs with s being 0 and 2 respectively for real and free choice
simulated data. The diagrams (c) - (d) show the same comparison with
simulation shuffle.

The attribute networks are s-line graphs derived from the attribute
hypergraphs. They have attributes as nodes and MPs as edges. As shown in
Figure 4 and Figure 5, the overall trend for the simulation average maximal
flow for both LP and ALP with s = 0 follow roughly the trend for the average
degree of the networks. The transitivity and the average maximal flow when s =
2, however, exhibit an opposite trend. We think it again could be a result of
size dependency of average maximal flow and filtering the networks with higher
s reduce such dependency to an extent.

In terms of the average maximal flow, we observe that the trends seem to
converge when $s>2$. The convergence of average maximal flow with high s is
not surprising as filtering can be effective in extracting major structural
information and is therefore relatively stable against changes in edge weight
definitions. Therefore, we conclude that we can identify major structural
landscapes of the networks by observing the persistent features from s = 0 to
s = 2.

In terms of the average maximal flow of ALP attribute networks, we can see
immediately there are two periods (before 1970s and a few years around late
19702s) where the real data exhibits unusually high values compared with both
simulation methods for the two normalised weight definitions. This was stable
against increase of the s-filtration to s=2. The high average flows during
these two periods were even more obvious against the shuffle simulation. It
shows that the source of this high connectivity is not the size of the
networks, the number of career options available, nor the distribution of MPs
in different career backgrounds. But since these two peaks were not
significant in transitivity, less obvious in the unweighted networks and more
obvious in the directed weighted networks, we can say part of the cause would
be the strong ties (possibly asymmetrical ones) among attributes, such as
edges between big nodes like employed professionals and other small ones.

Transitivity in the ALP attribute networks shows another source for the
unusual high connectivity during these two periods. As shown in Figure 5,
these two peaks can be found in the transitivity against shuffle simulation
baseline, though a weak one around late 1970s. Accordingly, we observed
bouquet structures before 1970s centering humanities background and weak
bouquet structures around late 1970s. Therefore, we confidently conclude that
ALP attribute networks are more structurally connected during these two
periods, with strong ties and weak bouquet structures being part of the cause.

Additionally, there is a slight peak (1997-1998) for the ALP in transitivity
with s = 2 for both simulation methods, which is also observed in average
maximal flow when $s=2$. This is due to some satellite structures. Similar to
the peak in LP attribute networks when s = 2 around 1990s. This is merely a
result of disconnected networks after high s filtration.

The Liberal Party, in terms of the average maximal flow, before 1980s, the
consensus from almost all plots is that the LP attribute networks were more
connected than expected from simulated baselines. The fact that the real AMF
was slightly lower than the simulation for s=0, but became significantly
higher than the simulation for s=2, shows that this additional connection is
partly caused by the strong ties in the networks. Different from the ALP
attribute networks, the higher connectivity of the LP attributes networks
during this period is lessened against the shuffle simulations. While the
shuffle does not completely bridge the gap between simulations and real data,
the significant decrease in the disagreement shows that the distribution of
MPs in different career choices is very important to the higher connectivity
observed, for example, the high number of MPs who were employed professionals.
Transitivity for LP attribute networks before 1980s also supports this idea of
higher connectivity. Within the period before 1980s, it is visible that there
are two peaks, one around 1955 and another one around 1975. Since the one
around 1975 seems to be more prominent in the directed networks, the strong
ties implied during that time are likely to be asymmetrical as well between
large attributes such as employed professionals and others.

In summary, we observe that on average both ALP and LP attribute networks are
more connected than simulations, though for very different reasons.
Interestingly, both parties’ background attribute networks converge to
simulation around 1980, corroborating studies that find evidence of party
recruitment convergence in Australia [31].

##  5 Conclusion

This paper has introduced a novel approach to the conceptualisation and
analysis of politician’s background data. Our attributes-as-networks approach
identifies and addresses several important challenges to leveraging network
information from politicians’ backgrounds. This contribution demonstrates
significant potential for researchers of legislative politics since the
collection of contemporary and historical background data of political elites
is growing rapidly [22].

Our central findings confirm the utility of our approach, as we demonstrate
the convergence in party political recruitment. The low connectedness of trade
unionist and manual laborer principal occupations, relative to elite
occupations contributes to explanations of the decline of working-class
representation within social democratic parties, while the emergence of the
highly connected senior management attribute describes the diversification in
the LP away from recruitment from the professions.

The study makes possible several opportunities for further study. First, a
dynamic analysis of the relationship between attribute node centrality and
outcome variables such as ministerial selection and leadership would be of
further value in demonstrating the importance of networks in the explanation
of political career outcomes. Second, a useful application of our method would
be to expand the analysis to comparative data to examine trends in MP
professional networks across political systems. Finally, we propose an
examination of attribute networks in other elite political settings such as
cabinets, where reshuffles and dissolutions may be related to network
structures among ministers, and the judiciary, as we hypothesise that networks
may determine stable voting coalitions.

##  6 Acknowledgement

The authors thank Rob Ackland, Sam Barton, Kate Turner, Vanessa Robins,
Catherine Greenhill, Jon Slapin, and Patrick Armstrong for helpful
conversations and feedback for this paper.

## References

  * [1] N. Allen, G. Magni, D. Searing, and P. Warncke.  What is a career politician? theories, concepts, and measures.  European Political Science Review, 12(2):199–217, 2020.
  * [2] P. Allen.  Linking Pre-Parliamentary Political Experience and the Career Trajectories of the 1997 General Election Cohort1.  Parliamentary Affairs, 66(4):685–707, Oct. 2013.
  * [3] Y. Asako, T. Iida, T. Matsubayashi, and M. Ueda.  Dynastic politicians: Theory and evidence from Japan.  Japanese Journal of Political Science, 16(1):5–32, 2015.  Publisher: Cambridge University Press.
  * [4] T. D. Barnes, V. D. Beall, and M. R. Holman.  Pink-Collar Representation and Budgetary Outcomes in US States.  Legislative Studies Quarterly, 46(1):119–154, 2021.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/lsq.12286.
  * [5] S. Barton, Z. C. Broad, D. Ortiz-Barrientos, D. Donovan, and J. G. Lefevre.  Hypergraphs and centrality measures identifying key features in gene expression data.  bioRxiv, pages 2022–12, 2022.
  * [6] H. Best, V. Cromwell, C. Hausmann, and M. Rush.  The Transformation of Legislative Elites: The Cases of Britain and Germany since the 1860s.  The Journal of Legislative Studies, 7(3):65–91, Sept. 2001.  Publisher: Routledge _eprint: https://doi.org/10.1080/714003883.
  * [7] J. Box-Steffensmeier and T.-M. Lin.  A Dynamic Model of Campaign Spending in Congressional Elections.  Pol-Meth, 999:Political Methodology Meetings, July, 1992, Harvard U., 1992.
  * [8] U. Brandes.  On variants of shortest-path betweenness centrality and their generic computation.  Social networks, 30(2):136–145, 2008.
  * [9] N. Bro.  The Structure of Political Conflict: Kinship Networks and Political Alignments in the Civil Wars of Nineteenth-Century Chile.  PhD thesis, 2020.
  * [10] P. Cairney.  The Professionalisation of MPs: Refining the ‘Politics-Facilitating’ Explanation.  Parliamentary Affairs, 60(2):212–233, Jan. 2007.  Publisher: Oxford Academic.
  * [11] N. Carnes.  Why are there so few working-class people in political office? Evidence from state legislatures.  Politics, Groups, and Identities, 4(1):84–109, Jan. 2016.  Publisher: Routledge _eprint: https://doi.org/10.1080/21565503.2015.1066689.
  * [12] W. Cross and A. Gauja.  Evolving membership strategies in Australian political parties.  Australian Journal of Political Science, 49(4):611–625, Nov. 2014.  MAG ID: 2021585800.
  * [13] J. Curtin, M. Kerby, and K. Dowding.  Sex, gender, and promotion in executive office: Cabinet careers in the world of Westminster.  Governance, 2022.  Publisher: Wiley Online Library.
  * [14] E. Dal Bó, P. Dal Bó, and J. Snyder.  Political dynasties.  The Review of Economic Studies, 76(1):115–142, 2009.  Publisher: Wiley-Blackwell.
  * [15] K. Dowding, P. Leslie, and M. Taflaga.  Australia: Speaker Time in an Adversarial System.  In H. Back, J. Fernandes, and M. Debus, editors, The Politics of Legislative Debates, pages 130–151. Oxford University Press, Oxford, 2021.  Publisher:.
  * [16] J. Edmonds and R. M. Karp.  Theoretical improvements in algorithmic efficiency for network flow problems.  Journal of the ACM (JACM), 19(2):248–264, 1972.
  * [17] E. Estrada and J. A. Rodriguez-Velazquez.  Complex networks as hypergraphs.  arXiv preprint physics/0505137, 2005.
  * [18] M. Fafchamps and J. Labonne.  Family Networks and Distributive Politics.  Journal of the European Economic Association, 18(4):1697–1725, Aug. 2020.
  * [19] J. H. Fiva and D. M. Smith.  Political dynasties and the incumbency advantage in party-centered environments.  American Political Science Review, 112(3):706–712, 2018.  Publisher: Cambridge University Press.
  * [20] M. Granovetter.  The Strength of Weak Ties: A Network Theory Revisited.  Sociological Theory, 1:201–233, 1983.  Publisher: [American Sociological Association, Wiley, Sage Publications, Inc.].
  * [21] M. S. Granovetter.  The Strength of Weak Ties.  American Journal of Sociology, 78(6):1360–1380, 1973.  Publisher: University of Chicago Press.
  * [22] S. Göbel and S. Munzert.  The Comparative Legislators Database.  British Journal of Political Science, pages 1–11, 2021.  Publisher: Cambridge University Press.
  * [23] S. E. Hill.  My Little Crony, Nov. 2020.
  * [24] J. Howard.  Lazarus rising: A Personal and Political Autobiography.  HarperCollins Australia, 2010.
  * [25] C. Kam, W. T. Bianco, I. Sened, and R. Smyth.  Ministerial selection and intraparty organization in the contemporary British parliament.  American Political Science Review, 104(02):289–306, 2010.
  * [26] R. S. Katz and P. Mair.  Changing Models of Party Organization and Party Democracy The Emergence of the Cartel Party.  Party Politics, 1(1):5–28, Jan. 1995.  MAG ID: 2018972251.
  * [27] A. King.  The Rise of the Career Politician in Britain - And Its Consequences.  British Journal of Political Science, 11(3):249–285, 1981.
  * [28] Lotte Hargrave and Jack Blumenau.  No Longer Conforming to Stereotypes? Gender, Political Style and Parliamentary Debate in the UK.  British Journal of Political Science, 2022.
  * [29] K. Lowande, M. Ritchie, and E. Lauterbach.  Descriptive and Substantive Representation in Congress: Evidence from 80,000 Congressional Inquiries.  American Journal of Political Science, 63(3):644–659, 2019.  Publisher: [Midwest Political Science Association, Wiley].
  * [30] R. I. Lung, N. Gaskó, and M. A. Suciu.  A hypergraph model for representing scientific output.  Scientometrics, 117:1361–1379, 2018.
  * [31] N. Miragliotta and W. Errington.  Legislative recruitment and models of party organisation: evidence from Australia.  The Journal of Legislative Studies, 18(1):21–40, Feb. 2012.  MAG ID: 2086110041.
  * [32] P. Norris and M. L. i. C. P. P. Norris.  Passages to Power: Legislative Recruitment in Advanced Democracies.  Cambridge University Press, Oct. 1997.  Google-Books-ID: mm1pZuegYD4C.
  * [33] T. O’Grady.  Careerists Versus Coal-Miners: Welfare Reforms and the Substantive Representation of Social Groups in the British Labour Party.  Comparative Political Studies, 52(4):544–578, Mar. 2019.  Publisher: SAGE Publications Inc.
  * [34] K. Rajkumar, G. Saint-Jacques, I. Bojinov, E. Brynjolfsson, and S. Aral.  A causal test of the strength of weak ties.  Science, 377(6612):1304–1310, Sept. 2022.
  * [35] M. W. Reimann, M. Nolte, M. Scolamiero, K. Turner, R. Perin, G. Chindemi, P. Dłotko, R. Levi, K. Hess, and H. Markram.  Cliques of neurons bound into cavities provide a missing link between structure and function.  Frontiers in computational neuroscience, 11:48, 2017.
  * [36] D. Searing.  Westminster’s World: Understanding Political Roles.  Harvard University Press, 1994.
  * [37] J. B. Slapin, J. H. Kirkland, J. A. Lazzaro, P. A. Leslie, and T. O’grady.  Ideology, Grandstanding, and Strategic Party Disloyalty in the British Parliament.  American Political Science Review, 112(1):15–30, Feb. 2018.
  * [38] D. M. Smith and S. Martin.  Political dynasties and the selection of cabinet ministers.  Legislative Studies Quarterly, 42(1):131–165, 2017.
  * [39] S. A. Smith.  Dynasties and democracy: The inherited incumbency advantage in Japan, volume 79.  Cambridge University Press New York, USA, 2020.
  * [40] T. Turner-Zwinkels and M. C. Mills.  Pathways to Power: The Role of Preparliamentary Careers and Political Human Capital in the Obtainment of Cabinet Positions.  Legislative Studies Quarterly, 45(2):207–252, 2020.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/lsq.12258.
  * [41] S. Wasserman and K. Faust.  Social network analysis: Methods and applications.  1994\.

##  Appendix A Preliminary Background on graphs and hypergraphs

In this section, we give a preliminary introduction to graph and hypergraph
theory along with our notation. A weighted undirected graph is an ordered
triple G = (V, E, $\phi$), where V is the set of vertices or nodes, E is the
set of edges and $\phi:E\rightarrow R$, a function that maps each edge to its
weight. Each weighted undirected graph with n nodes can be stored as a
$n\times n$ adjacency matrix $A(G)=[a_{ij}]$ with entries defined as:

$a_{ij}=\begin{cases}w_{ij}&\text{if $e_{ij}\in E$}\\\ 0&\text{if
$e_{ij}\not\in E$}\end{cases}$ (6)

In an unweighted undirected graph, all $w_{ij}$ would be set to 1. The degree
of nodes in weighted or unweighted undirected graphs is defined to be the
number of edges the node is contained in. If the graph is directed, instead of
taking the unordered pair in E to R, $\phi$ takes ordered pairs of E to R. As
a result, the corresponding adjacency matrix can be asymmetric.

An undirected hypergraph H = (V, E) is a generalisation of the graph where
each hyperedge $e\in E$ can join any number of vertices (see Fig 1), in
contrast to only two in ordinary graphs. The cardinality or size of a
hyperedge is the number of vertices that lie in the hyperedge, and the degree
of a vertex counts the number of hyperedges it is incident with. Each
undirected hypergraph H with n nodes and m hyperedges can be represented using
an $n\times m$ incidence matrix $I(G)=[b_{ij}]$ with entries:

$b_{ij}=\begin{cases}1&\text{if $v_{i}\in e_{j}$}\\\ 0&\text{if $v_{i}\not\in
e_{j}$}\end{cases}$ (7)

From this representation, one can easily generate the number of vertices in
each hyperedge and the number of hyperedges with which each vertex coincides.
Direct analysis of hypergraphs is challenging. Therefore, we follow the method
in [5] and collapse the hypergraphs to s-line graphs for further analysis. For
s, a positive integer, an s-line graph $L_{H}$ = ($V^{\prime}$,$E^{\prime}$)
from a hypergraph H is defined as:

$\displaystyle V^{\prime}=E(H)~{}~{}\text{and}~{}~{}{e_{i},e_{j}}\in
E^{\prime}\iff|e_{i}\cap e_{j}|\geq s$ (8)

where E(H) is the edge sets of H.

In other words, the s-line graph contains vertices that represent the original
hyperedges and these vertices are joined by edges where the hyperedges overlap
by $s$ or more. The resulting s-line graphs are no longer hypergraphs, and can
therefore be analysed as ordinary networks.

##  Appendix B Degree and centrality distribution

We also use the centrality distribution as supplementary information (See
Figure 6 and 7). There are many flavours of centrality measures, the one we
implemented is betweenness centrality. The definition for the betweenness
centrality is as follows:

$\displaystyle c_{B}(v)=\sum_{s,t\in V}\frac{\sigma(s,t|v)}{\sigma(s,t)}$ (9)

where V is the set of nodes, $\sigma(s,t)$ is the total number of shortest
paths from s to t, and $\sigma(s,t|v)$ is the number of those paths that
contains v while v is not s nor t. If v = s or v = t or s = t, the centrality
is defined as 0 [8]. The weighted undirected networks were used for
calculations for centrality, but the path lengths are defined as the inverse
of the edge weights.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 6: (a)-(d) are for analysis of the classifier networks: (a) skewness
and (b) kurtosis of the degree distribution and (c) skewness and (d) kurtosis
of the centrality distribution. (e)-(h) are for analysis of the MP networks:
(e) skewness and (f) kurtosis of the degree distribution and (g) skewness and
(h) kurtosis of the centrality distribution.

(a)

(b)

(c)

(d)

Figure 7: The diagrams (a) and (b) show the average degrees for nodes in each
year for (a) the classifier networks and (b) the MP networks. The diagrams (c)
and (d) show the number of nodes in each year for (c) the classifier networks
and (d) the MP networks.

##  Appendix C The bouquet structure

(a)

(b)

Figure 8: (a) Comparison of the effects of the average maximal flow and
transitivity in bouquet structures. We set m,n to be the number of nodes on
each side of the bouquet, and n is set to 10, while m varies on the x-axis.
(b) An example of the bouquet structure: the ALP MP network at year 1955.

The bouquet structure refers to structures that have a small community (the
most drastic version being the small community as one node) that connects two
much larger closely connected communities (close to complete subnetworks). The
justification for the effects on this structure on the two graph measures
(average maximal flow and transitivity) here looks at structures with one node
as the connecting bridge between two complete subnetworks with size n and m
respectively. When n = m, it is not hard to show that the average maximal flow
would be exactly half of the average maximal flow of the corresponding
complete network of size n+m+1. Because for each pair of nodes in the bouquet
structure with equal ends with size n, the number of paths between them is
constant at n. While in the complete graph of size 2n+1, the number of paths
between each pair of nodes is constant at 2n.

As we increase the size of one side of the bouquet, both the average maximal
flow and the transitivity will grow since the network is getting more and more
connected. As we see in figure 8(a), transitivity is very much unaffected by
this structure and approaches to 1 very quickly. Average maximal flow,
however, only starts to pick up when one of the end starts to be get past more
than four times the size of the other end.

Moreover, such structures can also show up on the skewness of centrality
distribution sometimes, as the center node of the bouquet is expected to have
much higher centrality compared to the rest.

##  Appendix D Satellite

Figure 9: An example of satellite structure: LP classifier network at year
1990 with s = 2.

By definition of transitivity and average maximal flow, this structure will
lead to higher transitivity and low average maximal flow (see Fig 9).

##  Appendix E Asymmetry of hyperedge crossing 2(c)

Here we show that how asymmetry affects the average maximal flow of the
directed weighted networks compared to the undirected weighted networks
according to the definitions of the two types of networks in section 2.2.

The way we show it is to compare the changes in the average maximal flow from
symmetrical crossings to asymmetrical crossing for the two sets of networks,
while fixing the overlap. We set the number of nodes in each hyperedge as A
and B respectively, and the number of overlap nodes in the intersection of two
hyperedges as n (see figure 2(c).

We start with the undirected weighted networks. For the symmetrical case,
where A = B, the average maximal flow is:

$\displaystyle\text{amf}_{\text{sym}}=\frac{n}{2B-n}$ (10)

Then we change the size of one hyperedge from A to A’, and now the average
maximal flow is:

$\displaystyle\text{amf}_{\text{asym}}=\frac{n}{A^{\prime}+B-n}$ (11)

Now we compute the difference between those two. We set $n=r^{\prime}\times B$
and $A^{\prime}=r\times B$, and we know that $r>r^{\prime}$, $r^{\prime}<1$:

$\displaystyle D$ $\displaystyle=\frac{n}{2B-n}-\frac{n}{A^{\prime}+B-n}$ (12)
$\displaystyle=r^{\prime}\frac{1-r}{(r+1-r^{\prime})(2-r^{\prime})}$ (13)

Therefore, the relative difference is:

$\displaystyle\frac{D}{\text{amf}_{\text{sym}}}=\frac{1-r}{r+1-r^{\prime}}$
(14)

Similarly with the directed weighted networks, for the symmetrical case, the
average maximal flow is:

$\displaystyle\text{amf'}_{\text{sym}}=\frac{n}{B}$ (15)

And

$\displaystyle\text{amf'}_{\text{asym}}=\frac{1}{2}(\frac{n}{A^{\prime}}+\frac{n}{B})$
(16)

The difference is:

$\displaystyle D^{\prime}$
$\displaystyle=\frac{1}{2}(\frac{n}{A^{\prime}}+\frac{n}{B})-\frac{n}{B}$ (17)
$\displaystyle=\frac{1}{2}r^{\prime}\frac{1-r}{r}$ (18)

And the relative difference is:

$\displaystyle\frac{D^{\prime}}{\text{amf'}_{\text{sym}}}=\frac{1-r}{2r}$ (19)

As $r\rightarrow 0$ (as A decreses in size), $r^{\prime}\rightarrow r$, we can
see that the relative difference in the directed option is going be a lot
higher than that of the undirected option. When $r\rightarrow$ a very large
number, the undirected relative difference goes towards -1, while the directed
relative difference goes towards -1/2.

##  Appendix F Increase/decrease in overlap

Figure 10: Example of this phenomenon potentially explaining why the average
maximal flow for the undirected weighted network was very low for classifier
network for LP from 1980 with s=0.

We show here that the effects on the average maximal flow with changes in the
size of the intersection between hyperedges are more dramatic in undirected
weighted networks than the directed weighted networks.

We again show the effects in a simple system where two hyperedges intersect.
We again set the sizes of two hyperedges to be A and B, and the starting
intersection as n. Then we change n to n’.

The percentage changes in the directed weighted networks is

$\displaystyle\frac{amf_{before}}{amf_{after}}=\frac{n}{n^{\prime}}$ (20)

The percentage change in the undirected weighted networks is:

$\displaystyle\frac{amf_{before}}{amf_{after}}=\frac{n}{n^{\prime}}\times\frac{A+B-n^{\prime}}{A+B-n}$
(21)

It can be shown that no matter $n^{\prime}>n$ or $n^{\prime}<n$ the difference
in the undirected weighted network would always be larger than the directed
weighted network.

##  Appendix G Technical details

Visualisation of these networks was done by Pyvis and networkx.