Highly connected dynamic artificial neural networks
An object-oriented approach to implementing artificial neural networks is
introduced in this article. The networks obtained in this way are highly
connected in that they admit edges between nodes in any layers of the network,
and dynamic, in that the insertion, or deletion, of nodes, edges or layers of
nodes can be effected in a straightforward way. In addition, the activation
functions of nodes need not be uniform within layers, and can also be changed
within individual nodes. Methods for implementing the feedforward step and the
backpropagation technique in such networks are presented here. Methods for
creating networks, for implementing the various dynamic properties and for
saving and recreating networks are also described.
Clint van Alten
# Highly connected dynamic artificial neural networks

###### Abstract

An object-oriented approach to implementing artificial neural networks is introduced in this article. The networks obtained in this way are highly connected in that they admit edges between nodes in any layers of the network, and dynamic, in that the insertion, or deletion, of nodes, edges or layers of nodes can be effected in a straightforward way. In addition, the activation functions of nodes need not be uniform within layers, and can also be changed within individual nodes. Methods for implementing the feedforward step and the backpropagation technique in such networks are presented here. Methods for creating networks, for implementing the various dynamic properties and for saving and recreating networks are also described.

## 1 Introduction

Artificial neural networks typically have a static architecture that consists of a sequence of layers, with each layer containing a fixed number of nodes and with edges that connect nodes between neighbouring layers. In this article we present an approach to implementing artificial neural networks that are more flexible in their architectures, which we call _dynamic artificial neural networks_ (for short, _networks1_). In particular, the networks may be highly connected in that they admit edges between nodes in any layers in the network, and dynamic, in the sense that new nodes, edges or layers of nodes
may be inserted into the network, and existing nodes, edges or layers may be removed. Further dynamic features include the ability to change properties of individual nodes. Nodes of different types, e.g., with different activation functions, may occur within the same layer.

The approach to implementing networks that we present in this article is an object-oriented one that uses two main types of object, namely, node objects and edge objects. Internal properties of a node such as the activation function and bias value are stored as attributes in the node object, as are a running total of inputs received by the node and the activation value of the node. Each node also has a pointer to the head of a list of edges that all share this node as their source; each edge object has as attributes its weight and a pointer to its target node.

The networks have a layered structure in which each layer consists of a linked list of nodes. Objects called layernodes are used to mark the head of each list of nodes and these are connected in a doubly-linked list. The structure of such networks is described in detail in Section 2. The computation of the network output given some input values, i.e., the feedforward step, proceeds layer-by-layer and uses the layernodes in the flow control. A method for implementing the feedforward step in these networks is described in Section 3. Thereafter, in Section 4, we present a method for implementing the backpropagation technique for training these networks. The method we present is a basic version of backpropagation that implements stochastic gradient descent; more complex versions of backpropagation may be implemented following the style of the basic method.2 The layernodes are also used in the backwards flow control during backpropagation.

Footnote 2: Sample code that implements various methods for dynamic artificial neural networks can be found at [https://github.com/cvanalten/dyann](https://github.com/cvanalten/dyann)

In Section 5 we discuss methods for creating networks and for applying the dynamic features of insertion and deletion in networks. In Section 6 we outline methods for saving and recreating networks.

We assume that the reader is familiar with the basics of artificial neural networks. An extensive introduction to such structures can be found, for example, in [1] and [2].
## 2 Network structure

We describe here the structure of a _dynamic artificial neural network_, or just _network_, for short. The general structure of a network is pictured in Figure 1. A network consists of a doubly-linked list of objects called layernodes, with the _input-layernode_ at the head and the _output-layernode_ at the tail. Layernodes are shown as squares in Figure 1. The layernode class is shown below; the _next_ attribute points to the adjoining layernode in the direction of the _output-layernode_, while _prev_ points to the adjoining layernode in the opposite direction. Every layernode has an attribute that is a pointer to the first node in singly-linked list of nodes, as indicated in Figure 1, where nodes are shown as circles.
Network

layernode _input-layernode_, _output-layernode_

The node and edge classes are shown below. The _sum_ attribute in the node class is a running total of inputs received by the node via edges connected to it. The remaining node attributes are either self-explanatory or their purpose will become clear in the following sections. Edges connect one node, called the edge's _source_, to another node, called the edge's _target_. Every node has a pointer to the first edge in a singly-linked list of edges that all share that node as their source node; every edge has a pointer to its target node. In Figure 1, some edges with node \(n\) as source are illustrated with thicker arrows.

Node

double _bias_, _sum_, _actvalue_, _delta_

function _actfunction_

edge _firstedge_ (first edge in list of edges with this node as source)

node _next_ (next node in this layer of nodes)

Edge

double _weight_

node _target_ (node receiving impulse via this edge)

edge _next_ (next edge in list of edges with the same source node)

Observe that every node has its activation function stored as an attribute, so different nodes in the same layer may have different activation functions. In addition, the activation function of a given node may be changed quite easily.

We shall refer to nodes in the list at the _input-layernode_ as _input nodes_ and nodes in the list at the _output-layernode_ as _output nodes_. By an _internal node_ we shall mean any node that is not an input or output node.

We assume that an internal node's activation value, i.e., _actvalue_, is obtained by applying its activation function to the sum of its _sum_ and _bias
attribute values. Thus, an internal node's activation value depends only on its own attributes. Typical examples of activation functions that such nodes could use include linear, relu, sigmoid and tanh. Output nodes may obtain their activation values in the same way as internal nodes, but we allow for the option that output nodes use activation functions that depend on the attributes of all output nodes; examples of such activation functions include max and softmax. For input nodes, biases and activation functions are not used, and we may assume that they have _bias_ of 0 and use linear activation functions.

In regard to edges, we observe that an edge from a given node may have as its target a node in any layer in the network; that is, edges need not only connect to nodes in subsequent layers, but may also connect to nodes in previous layers or nodes in its own layer. Such edges are amenable to the feedforward process, however, the backpropagation algorithm, as presented in Section 4, will not train such edges. We shall assume, therefore, that each edge connects its source node to a target node that is in a subsequent layer. We shall assume, also, that for any two nodes there exists at most one edge that connects them. This assumption is not essential, but facilitates backpropagation.

## 3 Feedforward

In this section we describe the forward computation of a dynamic artificial neural network on a given input, which we refer to as the feedforward step. In this step, a network, say Net, takes an input array, say \(\underline{A}\), of length equal to the number of nodes in the input layer in Net, and computes the output values following the algorithm FeedForward given below.

In the first step of FeedForward the values in \(\underline{A}\) are assigned to the _actualue_ attributes of the input nodes. Next, each input node \(n\) is 'fired', meaning that _n.actvalue_ is propagated along each edge in _n_'s edge list. That is, for each such edge \(e\), starting with _n.firstedge_, the product _n.actvalue\(*\)e.weight_ is added to _e.target.sum_, as shown in FireNode. Thereafter, the layernodes are traversed in forward sequential order up to the layer preceding the output layer. For each such layernode \(x\) and every node in the list at \(x\), the node's _actvalue_ is computed and the node is fired. Note that we reset the node's _sum_ to 0 after firing in preparation for the next forward pass. Lastly, the _actualue_ attributes at the output nodes are computed using FireOutputNode. The
output layer is treated separately as we allow for activation functions such as max or softmax in this layer. The values stored in the _actualue_ attributes of the output nodes comprise the output of the network.

FeedForward(Net, \(\underline{A}\))

1 Assign values in \(\underline{A}\) to _actualue_ attributes of nodes in the input layer.

2\(x\) = Net._input-layernode_

3\(n\) = _x.firstnode_

4while\(n\)\(\neq\) null

5 FireNode(_n_)

6\(n\) = _n.next_

7\(x\) = _x.next_

8while(_x.next_\(\neq\)null)

9\(n\) = _x.firstnode_

10while(_n_\(\neq\)null)

11\(n.\)_actualue_ = _n.actfunction_(_n.sum_\(+\)_n.bias_)

12 FireNode(_n_)

13\(n.\)_sum_\(=0\)

14\(n\) = _n.next_

15\(x\) = _x.next_

16 FireOutputLayer(_x_)

FireNode(_n_)

1\(e\) = _n.firstedge_

2while\(e\)\(\neq\)null

3\(e.\)_target.sum_\(=\)_e.target.sum_\(+\)_n.actualue\(\ast\)e.weight_

4\(e\) = _e.next_

FireOutputLayer(_x_)

Apply activation function at each node in output layer.

## 4 Backpropagation

In this section, we describe how the backpropagation technique using stochastic gradient descent may be implemented for training dynamic artificial neural networks. To facilitate discussion of backpropagation in this setting, it is useful to introduce some notation and recall some definitions.
We assume that a network, say Net, uses a loss function \(L\) that takes as arguments the output values of the network that are stored in the _actualue_ attributes of output nodes and the target values corresponding to the input values.

Let \(\underline{W}\) be the list of all current edge weights and bias values in Net, let \(\underline{A}\) be an input array and \(\underline{T}\) the corresponding array of target values. Suppose that FeedForward\((\text{Net},\underline{A})\) has been completed. The gradient descent method is used to adjust the weights and biases in the network to reduce the loss between the output values and target values. For each weight \(w\) in the network, the following update rule is applied:

\[w=w-\eta\,\Big{(}\tfrac{\partial L}{\partial w}\big{|}_{\underline{ATW}}\Big{)}, \tag{4.1}\]

where \(\eta\) is the learning rate and \(\tfrac{\partial L}{\partial w}\big{|}_{\underline{ATW}}\) denotes the evaluation of the partial derivative \(\tfrac{\partial L}{\partial w}\) using the values in \(\underline{A},\underline{T}\) and \(\underline{W}\). Backpropagation (see, e.g., [1]), as described below, is used to obtain \(\tfrac{\partial L}{\partial w}\big{|}_{\underline{ATW}}\) for each weight \(w\).

We use \(n\), \(m\) and \(\ell\) as names for nodes. If an edge exists with node \(m\) as its source and node \(n\) as its target then we say that \(m\) is _connected_ to \(n\) and denote this by \(m\to n\); we use \(w_{mn}\) as the variable for the weight of this edge. For each node \(n\) in the network we use \(b_{n}\) as the variable for the bias at \(n\), \(y_{n}\) as the variable for the activation value at \(n\) and \(z_{n}\) as the variable for the sum of inputs to the node \(n\) plus the bias. Thus,

\[z_{n}=\sum_{m:\,m\to n}w_{mn}y_{m}+b_{n},\]

where \(m\) ranges over all nodes that are connected to \(n\). If \(n\) is an internal node, then \(y_{n}=g_{n}(z_{n})\), where \(g_{n}\) is \(n\)'s activation function, and if \(n\) is an output node, then \(y_{n}=g_{n}(z_{n_{1}},\ldots,z_{n_{r}})\), where \(g_{n}\) is \(n\)'s activation function and \(n_{1},\ldots,n_{r}\) are the output nodes (one of which is \(n\)).

For each node \(n\), the _delta_ value at \(n\), denoted by \(\delta_{n}\), is defined as

\[\delta_{n}=\tfrac{\partial L}{\partial z_{n}}\big{|}_{\underline{ATW}}.\]

If a node \(m\) is connected to node \(n\), then \(\tfrac{\partial L}{\partial w_{mn}}=\tfrac{\partial L}{\partial z_{n}}\tfrac{ \partial z_{n}}{\partial w_{mn}}=\tfrac{\partial L}{\partial z_{n}}\,y_{m}\), so

\[\tfrac{\partial L}{\partial w_{mn}}\big{|}_{\underline{ATW}}=\tfrac{\partial L }{\partial z_{n}}\big{|}_{\underline{ATW}}\,y_{m}\big{|}_{\underline{ATW}}= \delta_{n}\,\Big{(}y_{m}\big{|}_{\underline{ATW}}\Big{)}. \tag{4.2}\]
For the _bias_ at \(n\), we have that \(\frac{\partial L}{\partial b_{n}}=\frac{\partial L}{\partial z_{n}}\frac{ \partial z_{n}}{\partial b_{n}}=\frac{\partial L}{\partial z_{n}}\), hence

\[\frac{\partial L}{\partial b_{n}}\big{|}_{\underline{ATW}}=\frac{\partial L}{ \partial z_{n}}\big{|}_{\underline{ATW}}=\delta_{n}. \tag{4.3}\]

By (4.2), the gradient descent update rule in (4.1) for edge weights is

\[w_{mn}=w_{mn}-\eta\,\delta_{n}\left(y_{m}\big{|}_{\underline{ATW}}\right) \tag{4.4}\]

and, using (4.3), the gradient descent update rule for biases is

\[b_{n}=b_{n}-\eta\,\delta_{n}. \tag{4.5}\]

Next, we show how the delta values backpropagate through the network. If node \(n\) is connected to node \(\ell\), then \(\frac{\partial z_{\ell}}{\partial y_{n}}=w_{n\ell}\), hence

\[\frac{\partial L}{\partial z_{n}}=\sum_{\ell:\,n\to\ell}\frac{\partial L}{ \partial z_{\ell}}\frac{\partial z_{\ell}}{\partial z_{n}}=\sum_{\ell:\,n\to \ell}\frac{\partial L}{\partial z_{\ell}}\frac{\partial z_{\ell}}{\partial y _{n}}\frac{dy_{n}}{dz_{n}}=\left(\sum_{\ell:\,n\to\ell}\frac{\partial L}{ \partial z_{\ell}}w_{n\ell}\right)\frac{dy_{n}}{dz_{n}}.\]

Thus,

\[\delta_{n}=\frac{\partial L}{\partial z_{n}}\big{|}_{\underline{ATW}}=\left( \sum_{\ell:\,n\to\ell}\delta_{\ell}\Big{(}w_{n\ell}\big{|}_{\underline{ATW}} \Big{)}\right)\left(\frac{dy_{n}}{dz_{n}}\big{|}_{\underline{ATW}}\right). \tag{4.6}\]

Using (4.6), the gradient descent update of the weights in the network Net can be done as shown in the algorithm BackPropagate below. Recall that the current weights and biases of Net are in \(\underline{W}\), that \(\underline{T}\) is the array of target values corresponding to the input array \(\underline{A}\), and that FeedForward(Net, \(\underline{A}\)) has been completed. Then, for every node \(n\), the value \(y_{n}|_{\underline{ATW}}\) is stored in \(n.\)_actvalue_. The learning rate is denoted by _eta_.

The algorithm BackPropagate starts by computing the _delta_ value at each output node using UpdateOutputNodes. In addition, the _bias_ value at each output node is updated according to (4.5). Next, proceeding backwards along layernodes, UpdateNode is applied to each internal node. UpdateNode loops over all edges in the edge list at a given node \(n\) and does two jobs - it computes the summation in expression (4.6) and performs the gradient descent update rule in (4.4) on each edge's weight. At the end of the loop, the final value for _n.delta_ is obtained by multiplying by the modifier \(\frac{dy_{n}}{dz_{n}}\big{|}_{\underline{ATW}}\), as in (4.6). Then, using _n.delta_, the value of _n.bias_ is updated according to (4.5). In the last part of BackPropagate, the nodes in the
input layer are updated. A separate algorithm for updating input nodes is used since neither the _delta_ values nor the _bias_ values are needed at these nodes.

BackPropagate(Net, \(\underline{T},L,\mathit{eta}\))

1 UpdateOutputNodes(Net, \(\underline{T},L,\mathit{eta}\))

2\(x\) = Net._output-layernode.prev_

3**while** (_x.prev_\(\neq\)null)

4\(n\) = _x.firstnode_

5**while** (_n_\(\neq\)null)

6 UpdateNode(_n_, \(\mathit{eta}\))

7\(n\) = _n.next_

8\(x\) = _x.prev_

9\(n\) = _x.firstnode_

10**while** (_n_\(\neq\)null)

11 UpdateInputNode(_n_, \(\mathit{eta}\))

12\(n\) = _n.next_

UpdateOutputNodes(Net, \(\underline{T},L,\mathit{eta}\))

1 compute _n.delta_ at each output node \(n\) using _n.delta_ = \(\frac{\partial L}{\partial z_{n}}\big{|}_{\underline{ATW}}\)

 and update _n.bias_ using _n.bias_ = _n.bias_ \(-\)_eta\(\ast\)_n.delta_

UpdateNode(_n_, \(\mathit{eta}\))

1\(n\).delta = 0

2\(e\) = _n.firstedge_

3**while** (_e_\(\neq\)null)

4\(n\).delta = _n.delta_ + _e.target.delta\(\ast\)e.weight_

5\(e\).weight = _e.weight_ \(-\)_eta\(\ast\)e.target.delta\(\ast\)n.actvalue_

6\(e\) = _e.next_

7\(n\).delta = _n.delta_\(\ast\frac{dy_{n}}{dz_{n}}\big{|}_{\underline{ATW}}\)

8\(n\).bias = _n.bias_ \(-\)_eta\(\ast\)n.delta_

UpdateInputNode(_n_, \(\mathit{eta}\))

1\(e\) = _n.firstedge_

2**while** (_e_\(\neq\)null)

3\(e\).weight = _e.weight_ \(-\)_eta\(\ast\)e.target.delta\(\ast\)n.actvalue_

4\(e\) = _e.next_
We note that in UpdateNode, it is necessary to compute \(\frac{dy_{n}}{dz_{n}}\big{|}_{\mathit{ATW}}\), i.e., \(g^{\prime}_{n}(z_{n})|_{\mathit{ATW}}\). In the case of linear, relu, sigmoid and tanh activation functions, \(\frac{dy_{n}}{dz_{n}}\big{|}_{\mathit{ATW}}\) can be computed from the value \(y_{n}|_{\mathit{ATW}}\), that is, from \(n.\mathit{actvalue}\). Other activation functions may require the value of \(z_{n}|_{\mathit{ATW}}\), i.e., \(\mathit{n.sum}+\mathit{n.bias}\). However, since \(\mathit{n.sum}\) has been reset to 0 in the feedforward step, the value of \(z_{n}|_{\mathit{ATW}}\) is no longer available. To circumvent this, an additional node attribute may be used, say \(\mathit{n.pastum}\), that stores the value of \(\mathit{n.sum}\) in the feedforward step before it is reset to 0.

Variations on the basic backpropagation algorithm that make use of various types of regularisation and optimisation techniques may be incorporated in the setting here. Additional attributes may be added to nodes if required.

## 5 Creating and modifying dynamic artificial neural networks

In this section we describe how to create dynamic artificial neural networks and how to implement the various dynamic properties of the networks.

A network may be created according to a given specification by first creating the linked lists of layernodes and nodes and then adding the linked list of edges to each node. Alternately, a network may be created by first creating the input and output layers, and then inserting new layers of nodes into the network, together with new edges that have one of the newly inserted nodes either as source or as target. A network may also be recreated from a saved description of the network, as discussed in Section 6.

It is worth mentioning some simpler variations of networks that may offer savings on time or space complexity. First, if every node in a layer of nodes has the same activation function, then this activation function can be stored as an attribute in the corresponding layernode object instead of in each node. In this setup, some (straightforward) changes are required to the FeedForward and BackPropagate algorithms. Secondly, if all internal nodes use the same activation function (e.g., relu), then no activation function attribute is required in either the nodes or layernodes, as the activation function can be built directly into the FeedForward and BackPropagate algorithms.

Inserting a new node into an existing layer of a network can be done by inserting the node at the head of the list of nodes of the corresponding layernode, using the standard method of inserting into a linked list. Thereafter,
edges can be added that have the new node as their source and any node in a subsequent layer as target. In addition, edges can be added that have the new node as target and any node in a previous layer as source. In the worst case, a pass over every node in the network is required to complete this step.

A new layer of nodes may be inserted between two consecutive layers by first inserting a new layernode into the doubly-linked list of layernodes and then inserting a list of new nodes according to the specifications. The addition of edges can be done as for inserting a single node.

The method of inserting a new layer of nodes may be used in the initial creation of a network. The input and output layers are first created and connected by edges if required, and then the internal layers are inserted one-by-one. A possible advantage to creating a network in this way is that some training epochs may be interposed between the insertion of new layers.

Next, we discuss methods for deleting nodes or edges from a network. One reason for deleting an edge from a network is that the edge's weight may be sufficiently close to zero that it has no significant impact on the execution of the network. Another reason is that the target node of the edge may have been deleted. Similarly, a reason for deleting a node is if all of its edges have been deleted. There may be other reasons for deleting a node; for example, the number of times a node has fired a nonzero value may be sufficiently small in comparison to the number of times the network has been used (although this requires nodes to have an additional attribute that keeps track of the rate of firings).

Deleting an edge can be done simply by removing the edge from the linked list of edges where it occurs. Deleting a node can be done similarly by removing it from the list of nodes where it occurs, however, the deletion of a node requires that we delete all the edges that have that node as their target node. To do that, we give each node an additional Boolean attribute, which we call _markedfordeletion_. Given a node \(n\) that we wish to delete, we set _n.markedfordeletion_ = True. Then, prior to removing the node, a pass over all nodes and corresponding edge lists is required in order to remove all edges \(e\) for which _e.target.markedfordeletion_ is True. Of course, only nodes in layers that precede node _n_'s layer need be considered. If a number of nodes have been selected for deletion, then a single pass over the network will suffice for all nodes. Thereafter, a pass over all nodes may be done in which all nodes marked for deletion are removed from their lists. Lastly, a pass over the list of layernodes may be performed to remove layernodes that have empty node lists.
Saving and recreating networks

In this section, we outline methods for saving and recreating dynamic artificial neural networks. To save a network, we require that nodes have additional integer attributes called _layerindex_ and _nodeindex_ that describe the position of the node within the network. We use the convention that nodes in the input layer have _layerindex_ equal to 0 and _layerindex_ increments in the direction of the output layer. Similarly, the head node of the node list at each layer has _nodeindex_ equal to 0 and values increment along the list.

To save a network, a pass over the nodes of the network is done to assign _n.layerindex_ and _n.nodeindex_ to each node \(n\). The number of layers in the network and the number of nodes in each layer is saved. Then, a second pass over the nodes of the network is done and the following information on each node is saved:

1. the values of attributes _bias_, _actvalue_, _actfunction_, _layerindex_, _nodeindex_ (and possibly more);
2. for every edge in the edge list of the node, the edge's _weight_ and its _target_, which is described by the _layerindex_ and _nodeindex_ attributes of the target node.

The above information allows for the recreation of the original network as follows. First, the layernode and node lists are constructed according to saved information, and _layerindex_ and _nodeindex_ attributes are assigned to each node. Next, each stored node is matched up with the new node with the same _layerindex_ and _nodeindex_ values, to which the stored attributes in (i) may then be assigned. An edge list for the node is then created using the stored edge information in (ii) to find and assign the _target_ node for each edge.

In the above construction, matching up the saved nodes with the new nodes can be done in one pass over the network if the stored nodes are sorted in lexicographical order according to (_layerindex_, _nodeindex_). Creating the edge list for a single node can be done in one pass over the network if the node's edges are sorted in lexicographic order according to (_layerindex_, _nodeindex_) of their targets. Thus, assuming the correct sorting of information, the creation of edge lists can be done using two nested loops over the nodes of the network.
Conclusion

In this article we have described an approach to implementing artificial neural networks that are highly connected and dynamic. The main differences between the networks described here and standard neural networks is that edges may connect a source node to a target node in any subsequent layer of the network, that new nodes, edges and layers may be inserted into the network, or deleted from the network, and that nodes within a layer need not be of a uniform type. The method uses an object-oriented approach with separate classes for nodes, edges, layernodes and networks themselves. We have shown how the feedforward and backpropagation methods can be implemented, and discussed methods for creating networks, implementing various dynamic properties and saving and recreating networks.

It is worth noting that standard neural networks may also be implemented as dynamic artificial neural networks. As such, comparisons in terms of performance between the approaches are possible. We make no claims about improved performance in terms of efficiency of training the networks. Rather, the potential advantages of the approach in this article lie in the variety of networks that can be implemented and the dynamic properties of the networks. There is substantial scope for experimentation with different types of networks in various applications. It may also be interesting to try incorporate other neural network architectures such as convolutional neural networks into the approach here, or to consider a combination of different approaches. Lastly, our approach has used standard types of nodes, however, by adding appropriate attributes to the node class, nodes with more complex behaviour may also be used.
