**Meet the editor**

Kenji Suzuki received his Ph.D. degree from Nagoya University in 2001. From 1993 to 2001 he worked at Hitachi Medical Corporation, and then at Aichi Prefectural University as faculty. In 2001 he joined Department of Radiology at University of Chicago. Since 2006 he has been Assistant Professor of Radiology, Medical Physics, and Cancer Research Center there. Dr. Suzuki's research

interests include computer-aided diagnosis and machine learning. He has published more than 230 papers including 90 peer-reviewed journal papers. He has been serving as the Editor-in-Chief and an Associate Editor of 23 leading international journals including Medical Physics, International Journal of Biomedical Imaging, and Algorithms. He has received Paul Hodges Award, three RSNA Certificate of Merit Awards and a Research Trainee Prize, Cancer Research Foundation Young Investigator Award, IEEE Outstanding Member Award, and Kurt Rossmann Award for Excellence in Teaching.

Contents

**Preface VII**

**Section 1 Architecture and Design 1**

João Luís Garcia Rosa

Ramadhena

**Section 2 Applications 113**

Chapter 1 **Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution 3**

Chapter 2 **Biologically Plausible Artificial Neural Networks 25**

Chapter 3 **Weight Changes for Learning Mechanisms in Two-Term**

Siti Mariyam Shamsuddin, Ashraf Osman Ibrahim and Citra

José Manuel Ortiz-Rodríguez, Ma. del Rosario Martínez-Blanco, José Manuel Cervantes Viramontes and Héctor René Vega-Carrillo

Chapter 4 **Robust Design of Artificial Neural Networks Methodology in**

Chapter 5 **Comparison Between an Artificial Neural Network and Logistic**

Giovanni Caocci, Roberto Baccoli, Roberto Littera, Sandro Orrù,

Lucie Gráfová, Jan Mareš, Aleš Procházka and Pavel Konopásek

**Regression in Predicting Long Term Kidney**

**Transplantation Outcome 115**

Carlo Carcassi and Giorgio La Nasa

Chapter 6 **Edge Detection in Biomedical Images Using Self-Organizing Maps 125**

Shingo Noguchi and Osana Yuko

**Back-Propagation Network 53**

**Neutron Spectrometry 83**

### Contents

#### **Preface XI**

Chapter 7 **MLP and ANFIS Applied to the Prediction of Hole Diameters in the Drilling Process 145** Thiago M. Geronimo, Carlos E. D. Cruz, Fernando de Souza Campos, Paulo R. Aguiar and Eduardo C. Bianchi Chapter 8 **Integrating Modularity and Reconfigurability for Perfect Implementation of Neural Networks 163** Hazem M. El-Bakry

Preface

in an artificial neural network by learning.

applications, chemistry applications, and financial applications.

I hope this book will be a useful source for readers.

Artificial neural networks may probably be the single most successful technology in the last two decades which has been widely used in a large variety of applications in various areas. An artificial neural network, often just called a neural network, is a mathematical (or computational) model that is inspired by the structure and function of biological neural networks in the brain. An artificial neural network consists of a number of artificial neurons (i.e., nonlinear processing units) which are connected to each other via synaptic weights (or simply just weights). An artificial neural network can "learn" a task by adjusting weights. There are supervised and unsupervised models. A supervised model requires a "teacher" or desired (ideal) output to learn a task. An unsupervised model does not require a "teacher," but it learns a task based on a cost function associated with the task. An artificial neural network is a powerful, versatile tool. Artificial neural networks have been successfully used in various applications such as biological, medical, industrial, control engendering, software engineering, environmental, economical, and social applications. The high versatility of artificial neural networks comes from its high capability and learning function. It has been theoretically proved that an artificial neural network can approximate any continuous mapping by arbitrary precision. Desired continuous mapping or a desired task is acquired

The purpose of this book is to provide recent advances of architectures, methodologies and applications of artificial neural networks. The book consists of two parts: architectures and applications. The architecture part covers architectures, design, optimization, and analysis of artificial neural networks. The fundamental concept, principles, and theory in the section help understand and use an artificial neural network in a specific application properly as well as effectively. The applications part covers applications of artificial neural networks in a wide range of areas including biomedical applications, industrial applications, physics

Thus, this book will be a fundamental source of recent advances and applications of artificial neural networks in a wide variety of areas. The target audience of this book includes professors, college students, graduate students, and engineers and researchers in companies.

> **Kenji Suzuki, Ph.D.** University of Chicago Chicago, Illinois, USA

### Preface

Chapter 7 **MLP and ANFIS Applied to the Prediction of Hole Diameters in**

Thiago M. Geronimo, Carlos E. D. Cruz, Fernando de Souza Campos,

Vinícius Gonçalves Maltarollo, Káthia Maria Honório and Albérico

Ivan N. da Silva, José Ângelo Cagnon and Nilton José Saggioro

Francisco Garcia Fernandez, Ignacio Soret Los Santos, Javier Lopez Martinez, Santiago Izquierdo Izquierdo and Francisco Llamazares

**the Drilling Process 145**

Hazem M. El-Bakry

**VI** Contents

**Problems 203**

Redondo

**Collisions at LHC 183**

Borges Ferreira da Silva

Amr Radi and Samy K. Hindawi

Paulo R. Aguiar and Eduardo C. Bianchi

Chapter 8 **Integrating Modularity and Reconfigurability for Perfect Implementation of Neural Networks 163**

Chapter 9 **Applying Artificial Neural Network Hadron - Hadron**

Chapter 10 **Applications of Artificial Neural Networks in Chemical**

Chapter 11 **Recurrent Neural Network Based Approach for Solving Groundwater Hydrology Problems 225**

Chapter 12 **Use of Artificial Neural Networks to Predict The Business Success or Failure of Start-Up Firms 245**

Artificial neural networks may probably be the single most successful technology in the last two decades which has been widely used in a large variety of applications in various areas. An artificial neural network, often just called a neural network, is a mathematical (or computational) model that is inspired by the structure and function of biological neural networks in the brain. An artificial neural network consists of a number of artificial neurons (i.e., nonlinear processing units) which are connected to each other via synaptic weights (or simply just weights). An artificial neural network can "learn" a task by adjusting weights. There are supervised and unsupervised models. A supervised model requires a "teacher" or desired (ideal) output to learn a task. An unsupervised model does not require a "teacher," but it learns a task based on a cost function associated with the task. An artificial neural network is a powerful, versatile tool. Artificial neural networks have been successfully used in various applications such as biological, medical, industrial, control engendering, software engineering, environmental, economical, and social applications. The high versatility of artificial neural networks comes from its high capability and learning function. It has been theoretically proved that an artificial neural network can approximate any continuous mapping by arbitrary precision. Desired continuous mapping or a desired task is acquired in an artificial neural network by learning.

The purpose of this book is to provide recent advances of architectures, methodologies and applications of artificial neural networks. The book consists of two parts: architectures and applications. The architecture part covers architectures, design, optimization, and analysis of artificial neural networks. The fundamental concept, principles, and theory in the section help understand and use an artificial neural network in a specific application properly as well as effectively. The applications part covers applications of artificial neural networks in a wide range of areas including biomedical applications, industrial applications, physics applications, chemistry applications, and financial applications.

Thus, this book will be a fundamental source of recent advances and applications of artificial neural networks in a wide variety of areas. The target audience of this book includes professors, college students, graduate students, and engineers and researchers in companies. I hope this book will be a useful source for readers.

> **Kenji Suzuki, Ph.D.** University of Chicago Chicago, Illinois, USA

**Section 1**

**Architecture and Design**

**Architecture and Design**

**Chapter 1**

**Improved Kohonen Feature Map Probabilistic**

Recently, neural networks are drawing much attention as a method to realize flexible infor‐ mation processing. Neural networks consider neuron groups of the brain in the creature, and imitate these neurons technologically. Neural networks have some features, especially one of the important features is that the networks can learn to acquire the ability of informa‐

In the field of neural network, many models have been proposed such as the Back Propaga‐ tion algorithm [1], the Kohonen Feature Map (KFM) [2], the Hopfield network [3], and the Bidirectional Associative Memory [4]. In these models, the learning process and the recall

However, in the real world, it is very difficult to get all information to learn in advance, so we need the model whose learning process and recall process are not divided. As such mod‐ el, Grossberg and Carpenter proposed the ART (Adaptive Resonance Theory) [5]. However, the ART is based on the local representation, and therefore it is not robust for damaged neu‐ rons in the Map Layer. While in the field of associative memories, some models have been proposed [6 - 8]. Since these models are based on the distributed representation, they have the robustness for damaged neurons. However, their storage capacities are small because

On the other hand, the Kohonen Feature Map (KFM) associative memory [9] has been pro‐ posed. Although the KFM associative memory is based on the local representation as similar as the ART[5], it can learn new patterns successively [10], and its storage capacity is larger than that of models in refs.[6 - 8]. It can deal with auto and hetero associations and the asso‐

> © 2013 Noguchi and Yuko; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

© 2013 Noguchi and Yuko; licensee InTech. This is a paper distributed under the terms of the Creative Commons

process are divided, and therefore they need all information to learn in advance.

their learning algorithm is based on the Hebbian learning.

**Associative Memory Based on Weights**

**Distribution**

**1. Introduction**

tion processing.

Shingo Noguchi and Osana Yuko

http://dx.doi.org/10.5772/51581

Additional information is available at the end of the chapter

## **Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution**

Shingo Noguchi and Osana Yuko

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/51581

#### **1. Introduction**

Recently, neural networks are drawing much attention as a method to realize flexible infor‐ mation processing. Neural networks consider neuron groups of the brain in the creature, and imitate these neurons technologically. Neural networks have some features, especially one of the important features is that the networks can learn to acquire the ability of informa‐ tion processing.

In the field of neural network, many models have been proposed such as the Back Propaga‐ tion algorithm [1], the Kohonen Feature Map (KFM) [2], the Hopfield network [3], and the Bidirectional Associative Memory [4]. In these models, the learning process and the recall process are divided, and therefore they need all information to learn in advance.

However, in the real world, it is very difficult to get all information to learn in advance, so we need the model whose learning process and recall process are not divided. As such mod‐ el, Grossberg and Carpenter proposed the ART (Adaptive Resonance Theory) [5]. However, the ART is based on the local representation, and therefore it is not robust for damaged neu‐ rons in the Map Layer. While in the field of associative memories, some models have been proposed [6 - 8]. Since these models are based on the distributed representation, they have the robustness for damaged neurons. However, their storage capacities are small because their learning algorithm is based on the Hebbian learning.

On the other hand, the Kohonen Feature Map (KFM) associative memory [9] has been pro‐ posed. Although the KFM associative memory is based on the local representation as similar as the ART[5], it can learn new patterns successively [10], and its storage capacity is larger than that of models in refs.[6 - 8]. It can deal with auto and hetero associations and the asso‐

ciations for plural sequential patterns including common terms [11, 12]. Moreover, the KFM associative memory with area representation [13] has been proposed. In the model, the area representation [14] was introduced to the KFM associative memory, and it has robustness for damaged neurons. However, it can not deal with one-to-many associations, and associa‐ tions of analog patterns. As the model which can deal with analog patterns and one-to-many associations, the Kohonen Feature Map Associative Memory with Refractoriness based on Area Representation [15] has been proposed. In the model, one-to-many associations are re‐ alized by refractoriness of neurons. Moreover, by improvement of the calculation of the in‐ ternal states of the neurons in the Map Layer, it has enough robustness for damaged neurons when analog patterns are memorized. However, all these models can not realize probabilistic association for the training set including one-to-many relations.

**2. KFM Probabilistic Associative Memory based on Weights Distribution**

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

Here, we explain the conventional Kohonen Feature Map Probabilistic Associative Memory

KFMPAM-WD. As shown in Fig. 1, this model has two layers; (1) Input/Output Layer and

In the learning algorithm of the conventional KFMPAM-WD, the connection weights are

**2.** The Euclidian distance between the learning vector *X(p)* and the connection weights vec‐

*d*(*X* ( *p*)

where *F* is the set of the neurons whose connection weights are fixed. *diz* is the distance between the neuron *i* and the neuron *z* whose connection weights are fixed. In Eq.(1), *Dij* is the radius of the ellipse area whose center is the neuron *i* for the direction to the

, *Wi*

known pattern. If the input pattern is regarded as a known pattern, go to (8).

**4.** The neuron which is the center of the learning area *r* is determined as follows:

, (*dij*

, (*dij*

*r* = argmin *i* : *Diz*+ *Dzi* < *diz* ( *for* ∀ *z* ∈ *F* )

is satisfied for all neurons, the input pattern *X(p)* is regarded as an un‐

*y* =0)

*x* =0)

2 + 1), (otherwise)

is the long radius of the ellipse area whose center is the neuron *i* and *bi*

 can be set for each training pattern. *mij* is the slope of the line through the neurons *i* and *j*. In Eq.(1), the neuron whose Euclidian distance between its connection weights and the learning vector is minimum in the neurons which can be take areas without

short radius of the ellipse area whose center is the neuron *i*. In the KFMPAM-WD, *ai*

) (1)

http://dx.doi.org/10.5772/51581

5

(2)

is the

(2) Map Layer, and the Input/Output Layer is divided into some parts.

based on Weights Distribution (KFMPAM-WD)(16).

Figure 1 shows the structure of the conventional

**1.** The initial values of weights are chosen randomly.

*)* is calculated.

**2.1. Structure**

**2.2. Learning process**

learned as follows:

tor *Wi*

**3.** If *d(X(p), Wi*

where *ai*

and *bi*

, *d(X(p), Wi*

*) θ t*

neuron *j*, and is given by

*Dij* ={

*ai*

*bi*

*bi* 2 + *mij* 2*ai* 2 (*mij*

*ai* 2*bi* 2

**Figure 1.** Structure of conventional KFMPAM-WD.

As the model which can realize probabilistic association for the training set including oneto-many relations, the Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD) [16] has been proposed. However, in this model, the weights are updated only in the area corresponding to the input pattern, so the learning considering the neighborhood is not carried out.

In this paper, we propose an Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD). This model is based on the con‐ ventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distri‐ bution [16]. The proposed model can realize probabilistic association for the training set including one-to-many relations. Moreover, this model has enough robustness for noisy in‐ put and damaged neurons. And, the learning considering the neighborhood can be realized.

#### **2. KFM Probabilistic Associative Memory based on Weights Distribution**

Here, we explain the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD)(16).

#### **2.1. Structure**

ciations for plural sequential patterns including common terms [11, 12]. Moreover, the KFM associative memory with area representation [13] has been proposed. In the model, the area representation [14] was introduced to the KFM associative memory, and it has robustness for damaged neurons. However, it can not deal with one-to-many associations, and associa‐ tions of analog patterns. As the model which can deal with analog patterns and one-to-many associations, the Kohonen Feature Map Associative Memory with Refractoriness based on Area Representation [15] has been proposed. In the model, one-to-many associations are re‐ alized by refractoriness of neurons. Moreover, by improvement of the calculation of the in‐ ternal states of the neurons in the Map Layer, it has enough robustness for damaged neurons when analog patterns are memorized. However, all these models can not realize

As the model which can realize probabilistic association for the training set including oneto-many relations, the Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD) [16] has been proposed. However, in this model, the weights are updated only in the area corresponding to the input pattern, so the learning

In this paper, we propose an Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD). This model is based on the con‐ ventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distri‐ bution [16]. The proposed model can realize probabilistic association for the training set including one-to-many relations. Moreover, this model has enough robustness for noisy in‐ put and damaged neurons. And, the learning considering the neighborhood can be realized.

probabilistic association for the training set including one-to-many relations.

**Figure 1.** Structure of conventional KFMPAM-WD.

4 Artificial Neural Networks – Architectures and Applications

considering the neighborhood is not carried out.

Figure 1 shows the structure of the conventional

KFMPAM-WD. As shown in Fig. 1, this model has two layers; (1) Input/Output Layer and (2) Map Layer, and the Input/Output Layer is divided into some parts.

#### **2.2. Learning process**

In the learning algorithm of the conventional KFMPAM-WD, the connection weights are learned as follows:

$$r = \underset{i:\,D\_{\underline{n}}^{+} \cdot D\_{\underline{n}} \le d\_{\underline{n}}}{\text{argmin}} \ d\left(\mathcal{X}^{\left(p\right)}, \mathcal{W}\_{i}\right) \tag{1}$$

where *F* is the set of the neurons whose connection weights are fixed. *diz* is the distance between the neuron *i* and the neuron *z* whose connection weights are fixed. In Eq.(1), *Dij* is the radius of the ellipse area whose center is the neuron *i* for the direction to the neuron *j*, and is given by

$$D\_{ij} = \begin{cases} a\_{i\prime} & \text{( $d\_{ij}^\vee$ =0)}\\ b\_{i\prime} & \text{( $d\_{ij}^\ge = 0$ )}\\ \frac{a\_i^2 b\_i^2}{b\_i^2 + m\_{ij}^2 a\_i^2} (m\_{ij}^2 + 1)\_\prime & \text{(otherwise)} \end{cases} \tag{2}$$

where *ai* is the long radius of the ellipse area whose center is the neuron *i* and *bi* is the short radius of the ellipse area whose center is the neuron *i*. In the KFMPAM-WD, *ai* and *bi* can be set for each training pattern. *mij* is the slope of the line through the neurons *i* and *j*. In Eq.(1), the neuron whose Euclidian distance between its connection weights and the learning vector is minimum in the neurons which can be take areas without overlaps to the areas corresponding to the patterns which are already trained. In Eq.(1), *ai* and *bi* are used as the size of the area for the learning vector.

**5.** If *d(X(p), Wr)*> *θ t* is satisfied, the connection weights of the neurons in the ellipse whose center is the neuron *r* are updated as follows:

$$\mathcal{W}\_{i}(t+1) = \begin{vmatrix} \mathcal{W}\_{i}(t) + \alpha(t)(\mathcal{X}^{\ \{p\}} - \mathcal{W}\_{i}(t))\_{\prime} & \{d\_{ri} \le D\_{ri} \} \\ \mathcal{W}\_{i}(t)\_{\prime} & \text{(otherwise)} \end{vmatrix} \tag{3}$$

where *α(t)* is the learning rate and is given by

$$
\alpha(t) = \frac{-\alpha\_0(t - T)}{T} \tag{4}
$$

When the binary pattern *X* is given to the Input/Output Layer, the output of the neuron *k* in

When the analog pattern *X* is given to the Input/Output Layer, the output of the neuron *k* in

**3. Improved KFM Probabilistic Associative Memory based on Weights**

Here, we explain the proposed Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD). The proposed model is based on the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights

Figure 2 shows the structure of the proposed IKFMPAM-WD. As shown in Fig. 2, the pro‐ posed model has two layers; (1) Input/Output Layer and (2) Map Layer, and the Input/ Output Layer is divided into some parts as similar as the conventional KFMPAM-WD.

In the learning algorithm of the proposed IKFMPAM-WD, the connection weights are

**2.** The Euclidian distance between the learning vector *X(p)* and the connection weights vec‐

**4.** The neuron which is the center of the learning area *r* is determined by Eq.(1). In Eq.(1), the neuron whose Euclid distance between its connection weights and the learning vec‐ tor is minimum in the neurons which can be take areas without overlaps to the areas

known pattern. If the input pattern is regarded as a known pattern, go to (8).

is satisfied for all neurons, the input pattern *X(p)* is regarded as an un‐

*no* )

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

0, (otherwise) (8)

http://dx.doi.org/10.5772/51581

7

*io* =*Wrk* . (9)

*io* ={1, (*Wrk* ≥*θb*

is the threshold of the neurons in the Input/Output Layer.

*xk*

the Input/Output Layer *xk*

the Input/Output Layer *xk*

where *θb*

*io*

**Distribution**

**3.1. Structure**

**3.2. Learning process**

learned as follows:

tor *Wi*

**3.** If *d(X(p), Wi*

, *d(X(p), Wi*

*) θt*

*io*

*io*

Distribution (KFMPAM-WD) [16] described in 2.

**1.** The initial values of weights are chosen randomly.

*)*, is calculated.

is given by

is given by

*xk*

Here, *α 0* is the initial value of *α(t)* and *T* is the upper limit of the learning iterations.

#### **2.3. Recall process**

In the recall process of the KFMPAM-WD, when the pattern *X* is given to the Input/Output Layer, the output of the neuron *i* in the Map Layer, *xi map* is calculated by

$$\mathbf{1}\_{i}\mathbf{x}\_{i}^{map} = \begin{cases} \mathbf{1}\_{i} & (i=r) \\ \mathbf{0}\_{i} & \text{(otherwise)} \end{cases} \tag{5}$$

where *r* is selected randomly from the neurons which satisfy

$$\frac{1}{N} \sum\_{k \in \mathbb{C}} \lg(X\_k - \mathcal{W}\_{ik}) > \theta^{\text{map}} \tag{6}$$

where *θ map* is the threshold of the neuron in the Map Layer, and *g*( ⋅ ) is given by

$$\log(b) = \begin{cases} 1, & \text{( $l \mid b \mid < \Theta^d$ )} \\ 0, & \text{(otherwise)}. \end{cases} \tag{7}$$

In the KFMPAM-WD, one of the neurons whose connection weights are similar to the input pattern are selected randomly as the winner neuron. So, the probabilistic association can be realized based on the weights distribution.

When the binary pattern *X* is given to the Input/Output Layer, the output of the neuron *k* in the Input/Output Layer *xk io* is given by

$$\mathbf{x}\_k^{\;i\;o} = \begin{cases} \mathbf{1}\_{\prime} & \text{(}\mathcal{W}\_{rk} \ge \mathcal{O}\_b^{\;n\;o}\text{)}\\ \mathbf{0}, & \text{(otherwise)} \end{cases} \tag{8}$$

where *θb io* is the threshold of the neurons in the Input/Output Layer.

When the analog pattern *X* is given to the Input/Output Layer, the output of the neuron *k* in the Input/Output Layer *xk io* is given by

$$\mathbf{x}\_k^{\;io} = \mathsf{W}\_{rk\;} \, \tag{9}$$

### **3. Improved KFM Probabilistic Associative Memory based on Weights Distribution**

Here, we explain the proposed Improved Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (IKFMPAM-WD). The proposed model is based on the conventional Kohonen Feature Map Probabilistic Associative Memory based on Weights Distribution (KFMPAM-WD) [16] described in 2.

#### **3.1. Structure**

overlaps to the areas corresponding to the patterns which are already trained. In Eq.(1),

−*Wi*

*T* .

Here, *α 0* is the initial value of *α(t)* and *T* is the upper limit of the learning iterations.

In the recall process of the KFMPAM-WD, when the pattern *X* is given to the Input/Output

is the threshold of the neuron in the Map Layer, and *g*( ⋅ ) is given by

In the KFMPAM-WD, one of the neurons whose connection weights are similar to the input pattern are selected randomly as the winner neuron. So, the probabilistic association can be

*g*(*b*)={1, (| *b*| <*θ d* )

*map* ={1, (*i* =*r*)

*map*

is calculated by

0, (otherwise) (5)

*g*(*Xk* −*Wik* )>*θ map* (6)

0, (otherwise). (7)

is satisfied, the connection weights of the neurons in the ellipse whose

(*t*)), (*dri* ≤*Dri*

)

(4)

(*t*), (otherwise) (3)

are used as the size of the area for the learning vector.

(*t*) + *α*(*t*)(*X* ( *p*)

*α*(*t*)= −*α*0(*t* −*T* )

is satisfied.

center is the neuron *r* are updated as follows:

(*t* + 1)={*Wi*

where *α(t)* is the learning rate and is given by

**7.** The connection weights of the neuron *r Wr* are fixed.

**8.** (2)*∼* (7) are iterated when a new pattern set is given.

Layer, the output of the neuron *i* in the Map Layer, *xi*

realized based on the weights distribution.

*xi*

1 *N in* ∑ *k*∈*C*

where *r* is selected randomly from the neurons which satisfy

*Wi*

*Wi*

6 Artificial Neural Networks – Architectures and Applications

**6.** (5) is iterated until *d(X(p), Wr)*≤ *θ t*

**2.3. Recall process**

where *θ map*

*ai* and *bi*

**5.** If *d(X(p), Wr)*> *θ t*

Figure 2 shows the structure of the proposed IKFMPAM-WD. As shown in Fig. 2, the pro‐ posed model has two layers; (1) Input/Output Layer and (2) Map Layer, and the Input/ Output Layer is divided into some parts as similar as the conventional KFMPAM-WD.

#### **3.2. Learning process**

In the learning algorithm of the proposed IKFMPAM-WD, the connection weights are learned as follows:

corresponding to the patterns which are already trained. In Eq.(1), *ai* and *bi* are used as the size of the area for the learning vector.

**5.** If *d(X(p), Wr) θ t* is satisfied, the connection weights of the neurons in the ellipse whose center is the neuron *r* are updated as follows:

$$W\_i(t+1) = \begin{cases} X^{\binom{p}{i}}, & \text{( $\boldsymbol{\theta}\_1^{larm} \le H(\overline{d\_{ri}})$ )} \\ W\_i(t) + H(\overline{d\_{ri}}) (X^{\binom{p}{i}} - W\_i(t)), & \text{( $\boldsymbol{\theta}\_2^{larm} \le H(\overline{d\_{ri}}) < \boldsymbol{\theta}\_1^{larm}$ )} \\ & \text{and} \\ W\_i(t), & \text{(otherwise)} \end{cases} \tag{10}$$

where *θ*1 *learn* are thresholds. *H* (*dri* ¯ ) and *H* (*di* ∗*i* ¯ ) are given by Eq.(11) and these are semifixed function. Especially, *H* (*dri* ¯ ) behaves as the neighborhood function. Here, *i \** shows the nearest weight-fixed neuron from the neuron *i*.

$$H\left(\overline{d\_{ij}}\right) = \frac{1}{1 + \exp\left(\frac{\overline{d\_{ij}} - D}{\varepsilon}\right)}\tag{11}$$

**Figure 2.** Structure of proposed IKFMPAM-WD.

KFMPAM-WD described in 2.3.

**4. Computer experiment results**

The recall process of the proposed IKFMPAM-WD is same as that of the conventional

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

9

Here, we show the computer experiment results to demonstrate the effectiveness of the pro‐

In this experiment, the binary patterns including one-to-many relations shown in Fig. 3 were memorized in the network composed of 800 neurons in the Input/Output Layer and 400 neurons in the Map Layer. Figure 4 shows a part of the association result when "crow" was given to the Input/Output Layer. As shown in Fig. 4, when "crow" was given to the net‐

Table 1 shows the experimental conditions used in the experiments of 4.2 *∼* 4.6.

**3.3. Recall process**

posed IKFMPAM-WD.

**4.2. Association results**

*4.2.1. Binary patterns*

**4.1. Experimental conditions**

where *dij* ¯ shows the normalized radius of the ellipse area whose center is the neuron *i* for the direction to the neuron *j*, and is given by

$$
\overline{d\_{ij}} = \frac{d\_{ij}}{\mathbf{D}\_{ij}}.\tag{12}
$$

In Eq.(11), *D (1 D)* is the constant to decide the neighborhood area size and is the steep‐ ness parameter. If there is no weight-fixed neuron,

$$
\mathfrak{g}^{\text{un-max}} \mathfrak{a} = 0
$$

$$
H(\overline{d\_{i\_{\!<\!n\!=\!}}}) = 0
$$

is used.

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution http://dx.doi.org/10.5772/51581 9

**Figure 2.** Structure of proposed IKFMPAM-WD.

#### **3.3. Recall process**

corresponding to the patterns which are already trained. In Eq.(1), *ai*

, (*θ*1

−*Wi*

(*t*), (otherwise)

¯

¯ −*D*

In Eq.(11), *D (1 D)* is the constant to decide the neighborhood area size and is the steep‐

shows the normalized radius of the ellipse area whose center is the neuron *i*

) and *H* (*di* ∗*i*

)= 1 1 + exp( *dij*

> *dij* ¯ = *dij Dij*

*H* (*di* ∗*i* ¯

is satisfied.

(*t*)), (*θ*2

)(*X* ( *p*)

¯

¯

*H* (*dij* ¯

is satisfied, the connection weights of the neurons in the ellipse whose

*learn* ≤*H* (*dri* ¯ ))

*learn* ≤*H* (*dri* ¯ )<*θ*1 *learn*

) behaves as the neighborhood function. Here, *i*

¯

)<*θ*1 *learn*)

) are given by Eq.(11) and these are semi-

*ε* ) (11)

. (12)

)=0 (13)

and*H* (*di* ∗*i*

the size of the area for the learning vector.

8 Artificial Neural Networks – Architectures and Applications

center is the neuron *r* are updated as follows:

(*t*) + *H* (*dri* ¯

the nearest weight-fixed neuron from the neuron *i*.

for the direction to the neuron *j*, and is given by

ness parameter. If there is no weight-fixed neuron,

**7.** The connection weights of the neuron *r Wr* are fixed.

**8.** (2)*∼* (7) are iterated when a new pattern set is given.

*X* ( *p*)

*Wi*

*Wi*

*learn* are thresholds. *H* (*dri*

**5.** If *d(X(p), Wr) θ t*

*Wi*

where *θ*1

where *dij* ¯

is used.

**6.** (5) is iterated until *d(X(p), Wr)*≤ *θ t*

(*t* + 1)={

fixed function. Especially, *H* (*dri*

and *bi*

are used as

(10)

*\** shows

The recall process of the proposed IKFMPAM-WD is same as that of the conventional KFMPAM-WD described in 2.3.

#### **4. Computer experiment results**

Here, we show the computer experiment results to demonstrate the effectiveness of the pro‐ posed IKFMPAM-WD.

#### **4.1. Experimental conditions**

Table 1 shows the experimental conditions used in the experiments of 4.2 *∼* 4.6.

#### **4.2. Association results**

#### *4.2.1. Binary patterns*

In this experiment, the binary patterns including one-to-many relations shown in Fig. 3 were memorized in the network composed of 800 neurons in the Input/Output Layer and 400 neurons in the Map Layer. Figure 4 shows a part of the association result when "crow" was given to the Input/Output Layer. As shown in Fig. 4, when "crow" was given to the net‐ work, "mouse" (*t*=1), "monkey" (*t*=2) and "lion" (*t*=4) were recalled. Figure 5 shows a part of the association result when "duck" was given to the Input/Output Layer. In this case, "dog" (*t*=251), "cat" (*t*=252) and "penguin" (*t*=255) were recalled. From these results, we can con‐ firmed that the proposed model can recall binary patterns including one-to-many relations.

**Figure 4.** One-to-Many Associations for Binary Patterns (When "crow" was Given).

**Figure 5.** One-to-Many Associations for Binary Patterns (When "duck" was Given).

Figure 6 shows the Map Layer after the pattern pairs shown in Fig. 3 were memorized. In Fig. 6, red neurons show the center neuron in each area, blue neurons show the neurons in areas for the patterns including "crow", green neurons show the neurons in areas for the patterns including "duck". As shown in Fig. 6, the proposed model can learn each learning pattern with various size area. Moreover, since the connection weights are updated not only in the area but also in the neighborhood area in the proposed model, areas corresponding to

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

11

the pattern pairs including "crow"/"duck" are arranged in near area each other.

Threshold of Neurons in Input/Output Layer θ *b in* 0.5

**Table 1.** Experimental Conditions.

**Figure 3.** Training Patterns including One-to-Many Relations (Binary Pattern).

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution http://dx.doi.org/10.5772/51581 11

**Figure 4.** One-to-Many Associations for Binary Patterns (When "crow" was Given).

work, "mouse" (*t*=1), "monkey" (*t*=2) and "lion" (*t*=4) were recalled. Figure 5 shows a part of the association result when "duck" was given to the Input/Output Layer. In this case, "dog" (*t*=251), "cat" (*t*=252) and "penguin" (*t*=255) were recalled. From these results, we can con‐ firmed that the proposed model can recall binary patterns including one-to-many relations.

**Parameters for Learning**

*learn 10-4*

*learn*0.9

*learn*0.1

θ *d* 0.004

*in* 0.5

Threshold for Learning θ *t*

Threshold of Neighborhood Function (1) θ *1*

Threshold of Neighborhood Function (2) θ *2*

Threshold of Difference between Weight Vector

Threshold of Neurons in Input/Output Layer θ *b*

and Input Vector

10 Artificial Neural Networks – Architectures and Applications

**Figure 3.** Training Patterns including One-to-Many Relations (Binary Pattern).

**Table 1.** Experimental Conditions.

**Parameters for Recall (Common)** Threshold of Neurons in Map Layer θ *map* 0.75

**Parameter for Recall (Binary)**

Neighborhood Area Size *D* 3 Steepness Parameter in Neighborhood Functionε 0.91

**Figure 5.** One-to-Many Associations for Binary Patterns (When "duck" was Given).

Figure 6 shows the Map Layer after the pattern pairs shown in Fig. 3 were memorized. In Fig. 6, red neurons show the center neuron in each area, blue neurons show the neurons in areas for the patterns including "crow", green neurons show the neurons in areas for the patterns including "duck". As shown in Fig. 6, the proposed model can learn each learning pattern with various size area. Moreover, since the connection weights are updated not only in the area but also in the neighborhood area in the proposed model, areas corresponding to the pattern pairs including "crow"/"duck" are arranged in near area each other.

Table 3 shows the recall times of each pattern in the trial of Fig. 4 (*t=1∼*250) and Fig. 5 (*t=*251*∼* 500). In this table, normalized values are also shown in ( ). From these results, we can confirmed that the proposed model can realize probabilistic associations based on the

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

13

In this experiment, the analog patterns including one-to-many relations shown in Fig. 7 were memorized in the network composed of 800 neurons in the Input/Output Layer and 400 neurons in the Map Layer. Figure 8 shows a part of the association result when "bear" was given to the Input/Output Layer. As shown in Fig. 8, when "bear" was given to the network, "lion" (*t*=1), "raccoon dog" (*t*=2) and "penguin" (*t*=3) were recalled. Figure 9 shows a part of the association result when "mouse" was given to the Input/Output Layer. In this case, "monkey" (*t*=251), "hen" (*t*=252) and "chick" (*t*=253) were recalled. From these re‐ sults, we can confirmed that the proposed model can recall analog patterns including one-

**Figure 7.** Training Patterns including One-to-Many Relations (Analog Pattern).

**Figure 8:** One-to-Many Associations for Analog Patterns (When "bear" was Given).

weight distributions.

*4.2.2. Analog patterns*

to-many relations.

**Table 2.** Area Size corresponding to Patterns in Fig. 3.

**Figure 6.** Area Representation for Learning Pattern in Fig. 3.

**Table 3.** Recall Times for Binary Pattern corresponding to "crow" and "duck".

Table 3 shows the recall times of each pattern in the trial of Fig. 4 (*t=1∼*250) and Fig. 5 (*t=*251*∼* 500). In this table, normalized values are also shown in ( ). From these results, we can confirmed that the proposed model can realize probabilistic associations based on the weight distributions.

#### *4.2.2. Analog patterns*

**Learning PatternLong Radius** *ai*

**Table 2.** Area Size corresponding to Patterns in Fig. 3.

12 Artificial Neural Networks – Architectures and Applications

**Figure 6.** Area Representation for Learning Pattern in Fig. 3.

**Table 3.** Recall Times for Binary Pattern corresponding to "crow" and "duck".

"crow"–"lion" 2.5 1.5 "crow"–"monkey" 3.5 2.0 "crow"–"mouse" 4.0 2.5 "duck"–"penguin" 2.5 1.5 "duck"–"dog" 3.5 2.0 "duck"–"cat" 4.0 2.5

**Input PatternOutput PatternArea SizeRecall Times** crow lion 11 (1.0) 43 (1.0)

duck penguin 11 (1.0) 39 (1.0)

monkey 23 (2.1) 87 (2.0) mouse 33 (3.0) 120 (2.8)

dog 23 (2.1) 79 (2.0) cat 33 (3.0) 132 (3.4)

**Short Radius** *bi*

In this experiment, the analog patterns including one-to-many relations shown in Fig. 7 were memorized in the network composed of 800 neurons in the Input/Output Layer and 400 neurons in the Map Layer. Figure 8 shows a part of the association result when "bear" was given to the Input/Output Layer. As shown in Fig. 8, when "bear" was given to the network, "lion" (*t*=1), "raccoon dog" (*t*=2) and "penguin" (*t*=3) were recalled. Figure 9 shows a part of the association result when "mouse" was given to the Input/Output Layer. In this case, "monkey" (*t*=251), "hen" (*t*=252) and "chick" (*t*=253) were recalled. From these re‐ sults, we can confirmed that the proposed model can recall analog patterns including oneto-many relations.

**Figure 7.** Training Patterns including One-to-Many Relations (Analog Pattern).

**Figure 8:** One-to-Many Associations for Analog Patterns (When "bear" was Given).

**Input PatternOutput PatternArea SizeRecall Times** bear lion 11 (1.0) 40 (1.0)

mouse chick 11 (1.0) 38 (1.0)

Figure 10 shows the Map Layer after the pattern pairs shown in Fig. 7 were memorized. In Fig. 10, red neurons show the center neuron in each area, blue neurons show the neurons in the areas for the patterns including "bear", green neurons show the neurons in the areas for the patterns including "mouse". As shown in Fig. 10, the proposed model can learn each

Table 5 shows the recall times of each pattern in the trial of Fig. 8 (*t=*1*∼* 250) and Fig. 9 (*t=*251*∼* 500). In this table, normalized values are also shown in ( ). From these results, we can confirmed that the proposed model can realize probabilistic associations based on the

**Table 5.** Recall Times for Analog Pattern corresponding to "bear" and "mouse".

learning pattern with various size area.

**Figure 11.** Storage Capacity of Proposed Model (Binary Patterns).

weight distributions.

raccoon dog 23 (2.1) 90 (2.3) penguin 33 (3.0) 120 (3.0)

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

15

hen 23 (2.1) 94 (2.5) monkey 33 (3.0) 118 (3.1)

**Figure 9.** One-to-Many Associations for Analog Patterns (When "mouse" was Given).

**Table 4.** Area Size corresponding to Patterns in Fig. 7.

**Figure 10.** Area Representation for Learning Pattern in Fig. 7.

**Table 5.** Recall Times for Analog Pattern corresponding to "bear" and "mouse".

**Figure 9.** One-to-Many Associations for Analog Patterns (When "mouse" was Given).

**Table 4.** Area Size corresponding to Patterns in Fig. 7.

14 Artificial Neural Networks – Architectures and Applications

**Figure 10.** Area Representation for Learning Pattern in Fig. 7.

**Learning Pattern Long Radius** *ai*

"bear"–"lion" 2.5 1.5 "bear"–"raccoon dog" 3.5 2.0 "bear"–"penguin" 4.0 2.5 "mouse"–"chick" 2.5 1.5 "mouse"–"hen" 3.5 2.0 "mouse"–"monkey" 4.0 2.5

**Short Radius** *bi*

Figure 10 shows the Map Layer after the pattern pairs shown in Fig. 7 were memorized. In Fig. 10, red neurons show the center neuron in each area, blue neurons show the neurons in the areas for the patterns including "bear", green neurons show the neurons in the areas for the patterns including "mouse". As shown in Fig. 10, the proposed model can learn each learning pattern with various size area.

Table 5 shows the recall times of each pattern in the trial of Fig. 8 (*t=*1*∼* 250) and Fig. 9 (*t=*251*∼* 500). In this table, normalized values are also shown in ( ). From these results, we can confirmed that the proposed model can realize probabilistic associations based on the weight distributions.

**Figure 11.** Storage Capacity of Proposed Model (Binary Patterns).

**Figure 13.** Storage Capacity of Conventional Model [16] (Binary Pattern

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

17

**Figure 14.** Storage Capacity of Conventional Model [16] (Analog Patterns).

**Figure 15.** Association Result for Noisy Input (When "crow" was Given.).

**Figure 12.** Storage Capacity of Proposed Model (Analog Patterns).

#### **4.3. Storage capacity**

Here, we examined the storage capacity of the proposed model. Figures 11 and 12 show the storage capacity of the proposed model. In this experiment, we used the network composed of 800 neurons in the Input/Output Layer and 400/900 neurons in the Map Layer, and 1-to-*P (P=2,3,4)* random pattern pairs were memorized as the area (*ai* =2.5 and *bi* =1.5). Figures 11 and 12 show the average of 100 trials, and the storage capacities of the conventional mod‐ el(16) are also shown for reference in Figs. 13 and 14. From these results, we can confirm that the storage capacity of the proposed model is almost same as that of the conventional mod‐ el(16). As shown in Figs. 11 and 12, the storage capacity of the proposed model does not de‐ pend on binary or analog pattern. And it does not depend on *P* in one-to-*P* relations. It depends on the number of neurons in the Map Layer.

#### **4.4. Robustness for noisy input**

#### *4.4.1. Association result for noisy input*

Figure 15 shows a part of the association result of the proposed model when the pattern "cat" with 20% noise was given during *t=*1*∼* 500. Figure 16 shows a part of the association result of the propsoed model when the pattern "crow" with 20% noise was given *t*=501*∼* 1000. As shown in these figures, the proposed model can recall correct patterns even when the noisy input was given.

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution http://dx.doi.org/10.5772/51581 17

**Figure 13.** Storage Capacity of Conventional Model [16] (Binary Pattern

**Figure 12.** Storage Capacity of Proposed Model (Analog Patterns).

16 Artificial Neural Networks – Architectures and Applications

depends on the number of neurons in the Map Layer.

**4.4. Robustness for noisy input**

the noisy input was given.

*4.4.1. Association result for noisy input*

*(P=2,3,4)* random pattern pairs were memorized as the area (*ai*

Here, we examined the storage capacity of the proposed model. Figures 11 and 12 show the storage capacity of the proposed model. In this experiment, we used the network composed of 800 neurons in the Input/Output Layer and 400/900 neurons in the Map Layer, and 1-to-*P*

and 12 show the average of 100 trials, and the storage capacities of the conventional mod‐ el(16) are also shown for reference in Figs. 13 and 14. From these results, we can confirm that the storage capacity of the proposed model is almost same as that of the conventional mod‐ el(16). As shown in Figs. 11 and 12, the storage capacity of the proposed model does not de‐ pend on binary or analog pattern. And it does not depend on *P* in one-to-*P* relations. It

Figure 15 shows a part of the association result of the proposed model when the pattern "cat" with 20% noise was given during *t=*1*∼* 500. Figure 16 shows a part of the association result of the propsoed model when the pattern "crow" with 20% noise was given *t*=501*∼* 1000. As shown in these figures, the proposed model can recall correct patterns even when

=2.5 and *bi*

=1.5). Figures 11

**4.3. Storage capacity**

**Figure 14.** Storage Capacity of Conventional Model [16] (Analog Patterns).

**Figure 15.** Association Result for Noisy Input (When "crow" was Given.).

*4.4.2. Robustness for noisy input*

**4.5. Robustness for damaged neurons**

the Map Layer are damaged.

conventional model [16].

*4.5.2. Robustness for damaged neurons*

Figures 17 and 18 show the robustness for noisy input of the proposed model. In this experi‐ ment, 10 randam patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer. Figures 17 and 18 are the average of 100 trials. As shown in these figures, the proposed model has robust‐

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

19

Figure 19 shows a part of the association result of the proposed model when the pattern "bear" was given during *t=*1*∼* 500. Figure 20 shows a part of the association result of the proposed model when the pattern "mouse" was given *t*=501*∼* 1000. In these experiments, the network whose 20% of neurons in the Map Layer are damaged were used. As shown in these figures, the proposed model can recall correct patterns even when the some neurons in

Figures 21 and 22 show the robustness when the winner neurons are damaged in the pro‐ posed model. In this experiment, 1*∼* 10 random patterns in one-to-one relations were mem‐ orized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer. Figures 21 and 22 are the average of 100 trials. As shown in these figures, the proposed model has robustness when the winner neurons are damaged as similar as the

ness for noisy input as similar as the conventional model(16).

*4.5.1. Association result when some neurons in map layer are damaged*

**Figure 19.** Association Result for Damaged Neurons (When "bear" was Given.).

**Figure 16.** Association Result for Noisy Input (When "duck" was Given.).

**Figure 17.** Robustness for Noisy Input (Binary Patterns).

**Figure 18.** Robustness for Noisy Input (Analog Patterns).

#### *4.4.2. Robustness for noisy input*

Figures 17 and 18 show the robustness for noisy input of the proposed model. In this experi‐ ment, 10 randam patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer. Figures 17 and 18 are the average of 100 trials. As shown in these figures, the proposed model has robust‐ ness for noisy input as similar as the conventional model(16).

#### **4.5. Robustness for damaged neurons**

**Figure 16.** Association Result for Noisy Input (When "duck" was Given.).

18 Artificial Neural Networks – Architectures and Applications

**Figure 17.** Robustness for Noisy Input (Binary Patterns).

**Figure 18.** Robustness for Noisy Input (Analog Patterns).

#### *4.5.1. Association result when some neurons in map layer are damaged*

Figure 19 shows a part of the association result of the proposed model when the pattern "bear" was given during *t=*1*∼* 500. Figure 20 shows a part of the association result of the proposed model when the pattern "mouse" was given *t*=501*∼* 1000. In these experiments, the network whose 20% of neurons in the Map Layer are damaged were used. As shown in these figures, the proposed model can recall correct patterns even when the some neurons in the Map Layer are damaged.

#### *4.5.2. Robustness for damaged neurons*

Figures 21 and 22 show the robustness when the winner neurons are damaged in the pro‐ posed model. In this experiment, 1*∼* 10 random patterns in one-to-one relations were mem‐ orized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer. Figures 21 and 22 are the average of 100 trials. As shown in these figures, the proposed model has robustness when the winner neurons are damaged as similar as the conventional model [16].

**Figure 19.** Association Result for Damaged Neurons (When "bear" was Given.).

**Figure 23:** Robustness for Damaged Neurons (Binary Patterns).

**Figure 24.** Robustness for Damaged Neurons (Analog Patterns).

**4.6. Learning speed**

Figures 23 and 24 show the robustness for damaged neurons in the proposed model. In this experiment, 10 random patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer. Fig‐ ures 23 and 24 are the average of 100 trials. As shown in these figures, the proposed model

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

21

Here, we examined the learning speed of the proposed model. In this experiment, 10 ran‐ dom patterns were memorized in the network composed of 800 neurons in the Input/ Output Layer and 900 neurons in the Map Layer. Table 6 shows the learning time of the pro‐ posed model and the conventional model(16). These results are average of 100 trials on the Personal Computer (Intel Pentium 4 (3.2GHz), FreeBSD 4.11, gcc 2.95.3). As shown in Table 6, the learning time of the proposed model is shorter than that of the conventional model.

has robustness for damaged neurons as similar as the conventional model [16].

**Figure 20.** Association Result for Damaged Neurons (When "mouse" was Given.).

**Figure 21.** Robustness of Damaged Winner Neurons (Binary Patterns).

**Figure 22.** Robustness of Damaged Winner Neurons (Analog Patterns).

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution http://dx.doi.org/10.5772/51581 21

**Figure 23:** Robustness for Damaged Neurons (Binary Patterns).

**Figure 24.** Robustness for Damaged Neurons (Analog Patterns).

Figures 23 and 24 show the robustness for damaged neurons in the proposed model. In this experiment, 10 random patterns in one-to-one relations were memorized in the network composed of 800 neurons in the Input/Output Layer and 900 neurons in the Map Layer. Fig‐ ures 23 and 24 are the average of 100 trials. As shown in these figures, the proposed model has robustness for damaged neurons as similar as the conventional model [16].

#### **4.6. Learning speed**

**Figure 20.** Association Result for Damaged Neurons (When "mouse" was Given.).

20 Artificial Neural Networks – Architectures and Applications

**Figure 21.** Robustness of Damaged Winner Neurons (Binary Patterns).

**Figure 22.** Robustness of Damaged Winner Neurons (Analog Patterns).

Here, we examined the learning speed of the proposed model. In this experiment, 10 ran‐ dom patterns were memorized in the network composed of 800 neurons in the Input/ Output Layer and 900 neurons in the Map Layer. Table 6 shows the learning time of the pro‐ posed model and the conventional model(16). These results are average of 100 trials on the Personal Computer (Intel Pentium 4 (3.2GHz), FreeBSD 4.11, gcc 2.95.3). As shown in Table 6, the learning time of the proposed model is shorter than that of the conventional model.

#### **5. Conclusions**

In this paper, we have proposed the Improved Kohonen Feature Map Probabilistic Associa‐ tive Memory based on Weights Distribution. This model is based on the conventional Koho‐ nen Feature Map Probabilistic Associative Memory based on Weights Distribution. The proposed model can realize probabilistic association for the training set including one-tomany relations. Moreover, this model has enough robustness for noisy input and damaged neurons. We carried out a series of computer experiments and confirmed the effectiveness of the proposed model.

[6] Watanabe, M., Aihara, K., & Kondo, S. (1995). Automatic learning in chaotic neural

Improved Kohonen Feature Map Probabilistic Associative Memory Based on Weights Distribution

http://dx.doi.org/10.5772/51581

23

[7] Arai, T., & Osana, Y. (2006). Hetero chaotic associative memory for successive learn‐ ing with give up function -- One-to-many associations --, Proceedings of IASTED Ar‐

[8] Ando, M., Okuno, Y., & Osana, Y. (2006). Hetero chaotic associative memory for suc‐ cessive learning with multi-winners competition. *Proceedings of IEEE and INNS Inter‐*

[9] Ichiki, H., Hagiwara, M., & Nakagawa, M. (1993). Kohonen feature maps as a super‐ vised learning machine. *Proceedings of IEEE International Conference on Neural Net‐*

[10] Yamada, T., Hattori, M., Morisawa, M., & Ito, H. (1999). Sequential learning for asso‐ ciative memory using Kohonen feature map. *Proceedings of IEEE and INNS Interna‐*

[11] Hattori, M., Arisumi, H., & Ito, H. (2001). Sequential learning for SOM associative memory with map reconstruction. *Proceedings of International Conference on Artificial*

[12] Sakurai, N., Hattori, M., & Ito, H. (2002). SOM associative memory for temporal se‐ quences. *Proceedings of IEEE and INNS International Joint Conference on Neural Net‐*

[13] Abe, H., & Osana, Y. (2006). Kohonen feature map associative memory with area rep‐ resentation. *Proceedings of IASTED Artificial Intelligence and Applications*, Innsbruck.

[14] Ikeda, N., & Hagiwara, M. (1997). A proposal of novel knowledge representation (Area representation) and the implementation by neural network. *International Con‐*

[15] Imabayashi, T., & Osana, Y. (2008). Implementation of association of one-to-many as‐ sociations and the analog pattern in Kohonen feature map associative memory with area representation. *Proceedings of IASTED Artificial Intelligence and Applications*, Inns‐

[16] Koike, M., & Osana, Y. (2010). Kohonen feature map probabilistic associative memo‐ ry based on weights distribution. *Proceedings of IASTED Artificial Intelligence and Ap‐*

networks. *IEICE-A*, J78-A(6), 686-691, (in Japanese).

tificial Intelligence and Applications. Innsbruck.

*works*, 1944-1948.

*Neural Networks*, Vienna.

*works*, 950-955, Honolulu.

bruck.

*plications*, Innsbruck.

*national Joint Conference on Neural Networks*, Vancouver.

*tional Joint Conference on Neural Networks*, 555, Washington D.C.

*ference on Computational Intelligence and Neuroscience*, III, 430-433.

**Table 6.** Learning Speed.

#### **Author details**

Shingo Noguchi and Osana Yuko\*

\*Address all correspondence to<EMAIL_ADDRESS>
Tokyo University of Technology, Japan

#### **References**

[6] Watanabe, M., Aihara, K., & Kondo, S. (1995). Automatic learning in chaotic neural networks. *IEICE-A*, J78-A(6), 686-691, (in Japanese).

**5. Conclusions**

22 Artificial Neural Networks – Architectures and Applications

the proposed model.

**Table 6.** Learning Speed.

**Author details**

**References**

Shingo Noguchi and Osana Yuko\*

Tokyo University of Technology, Japan

tions, The MIT Press.

*works*, 18(1), 49-60.

ral Networks. The MIT Press.

\*Address all correspondence to<EMAIL_ADDRESS>
[2] Kohonen, T. (1994). Self-Organizing Maps. Springer.

In this paper, we have proposed the Improved Kohonen Feature Map Probabilistic Associa‐ tive Memory based on Weights Distribution. This model is based on the conventional Koho‐ nen Feature Map Probabilistic Associative Memory based on Weights Distribution. The proposed model can realize probabilistic association for the training set including one-tomany relations. Moreover, this model has enough robustness for noisy input and damaged neurons. We carried out a series of computer experiments and confirmed the effectiveness of

> Proposed Model (Binary Patterns) 0.87 Proposed Model (Analog Patterns) 0.92 Conventional Model(16) (Binary Patterns) 1.01 Conventional Model(16) (Analog Patterns) 1.34

[1] Rumelhart, D. E., McClelland, J. L., & the PDP Research Group. (1986). Parallel Dis‐ tributed Processing, Exploitations in the Microstructure of Cognition. 11, Founda‐

[3] Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. *Proceedings of National Academy Sciences USA*, 79, 2554-2558.

[4] Kosko, B. (1988). Bidirectional associative memories. *IEEE Transactions on Neural Net‐*

[5] Carpenter, G. A., & Grossberg, S. (1995). Pattern Recognition by Self-organizing Neu‐

**Learning Time (seconds)**

**Chapter 2**

**Provisional chapter**

**Biologically Plausible Artificial Neural Networks**

**Biologically Plausible Artificial Neural Networks**

Artificial Neural Networks (ANNs) are based on an abstract and simplified view of the neuron. Artificial neurons are connected and arranged in layers to form large networks, where learning and connections determine the network function. Connections can be formed through learning and do not need to be 'programmed.' Recent ANN models lack many physiological properties of the neuron, because they are more oriented to computational

According to the fifth edition of Gordon Shepherd book, *The Synaptic Organization of the Brain*, "information processing depends not only on anatomical substrates of synaptic circuits, but also on the electrophysiological properties of neurons" [51]. In the literature of dynamical systems, it is widely believed that knowing the electrical currents of nerve cells is sufficient to determine what the cell is doing and why. Indeed, this somewhat contradicts the observation that cells that have similar currents may exhibit different behaviors. But in the neuroscience community, this fact was ignored until recently when the difference in behavior was showed to be due to different mechanisms of excitability bifurcation [35]. A bifurcation of a dynamical system is a qualitative change in its dynamics produced by

The type of bifurcation determines the most fundamental computational properties of neurons, such as the class of excitability, the existence or nonexistence of the activation threshold, all-or-none action potentials (spikes), sub-threshold oscillations, bi-stability of rest

A biologically inspired connectionist approach should present a neurophysiologically motivated training algorithm, a bi-directional connectionist architecture, and several other

> ©2012 Rosa, licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2013 Rosa; licensee InTech. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

© 2013 Rosa; licensee InTech. This is a paper distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

distribution, and reproduction in any medium, provided the original work is properly cited.

and spiking states, whether the neuron is an integrator or resonator etc. [25].

João Luís Garcia Rosa

João Luís Garcia Rosa

http://dx.doi.org/10.5772/54177

**1. Introduction**

varying parameters [19].

Additional information is available at the end of the chapter

performance than to biological credibility [41].

features, e. g., distributed representations.

Additional information is available at the end of the chapter

**Provisional chapter**

### **Biologically Plausible Artificial Neural Networks**

**Biologically Plausible Artificial Neural Networks**

João Luís Garcia Rosa Additional information is available at the end of the chapter

João Luís Garcia Rosa

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/54177

#### **1. Introduction**

Artificial Neural Networks (ANNs) are based on an abstract and simplified view of the neuron. Artificial neurons are connected and arranged in layers to form large networks, where learning and connections determine the network function. Connections can be formed through learning and do not need to be 'programmed.' Recent ANN models lack many physiological properties of the neuron, because they are more oriented to computational performance than to biological credibility [41].

According to the fifth edition of Gordon Shepherd book, *The Synaptic Organization of the Brain*, "information processing depends not only on anatomical substrates of synaptic circuits, but also on the electrophysiological properties of neurons" [51]. In the literature of dynamical systems, it is widely believed that knowing the electrical currents of nerve cells is sufficient to determine what the cell is doing and why. Indeed, this somewhat contradicts the observation that cells that have similar currents may exhibit different behaviors. But in the neuroscience community, this fact was ignored until recently when the difference in behavior was showed to be due to different mechanisms of excitability bifurcation [35]. A bifurcation of a dynamical system is a qualitative change in its dynamics produced by varying parameters [19].

The type of bifurcation determines the most fundamental computational properties of neurons, such as the class of excitability, the existence or nonexistence of the activation threshold, all-or-none action potentials (spikes), sub-threshold oscillations, bi-stability of rest and spiking states, whether the neuron is an integrator or resonator etc. [25].

A biologically inspired connectionist approach should present a neurophysiologically motivated training algorithm, a bi-directional connectionist architecture, and several other features, e. g., distributed representations.

©2012 Rosa, licensee InTech. This is an open access chapter distributed under the terms of the Creative

#### **1.1. McCulloch-Pitts neuron**

McCulloch-Pitts neuron (1943) was the first mathematical model [32]. Its properties:

The McCulloch-Pitts neuron represents a simplified mathematical model for the neuron, where *xi* is the *i*-th binary input and *wi* is the synaptic (connection) weight associated with the input *xi*. The computation occurs in soma (cell body). For a neuron with *p* inputs:

$$a = \sum\_{i=1}^{p} x\_i w\_i \tag{1}$$

**1.2. The perceptron**

**1.3. Neural network topology**

Rosenblatt's perceptron [47] takes a weighted sum of neuron inputs, and sends output 1 (spike) if this sum is greater than the activation threshold. It is a linear discriminator: given 2 points, a straight line is able to discriminate them. For some configurations of *m* points, a

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

27

**Figure 3.** Set of linearly separable points. **Figure 4.** Set of non-linearly separable points.

learning algorithm (delta rule) does not work with networks of more than one layer.

there is communication between neurons in different layers (see figure 6).

The limitations of the perceptron is that it is an one-layer feed-forward network (non-recurrent); it is only capable of learning solution of linearly separable problems; and its

In cerebral cortex, neurons are disposed in columns, and most synapses occur between different columns. See the famous drawing by Ramón y Cajal (figure 5). In the extremely simplified mathematical model, neurons are disposed in layers (representing columns), and

**Figure 5.** Drawing by Santiago Ramón y Cajal of neurons in the pigeon cerebellum. (A) denotes Purkinje cells, an example of a

multipolar neuron, while (B) denotes granule cells, which are also multipolar [57].

straight line is able to separate them in two classes (figures 3 and 4).

with *x*0 = 1 and *w*0 = *β* = −*θ*. *β* is the bias and *θ* is the activation threshold. See figures 1 and 2. The are *p* binary inputs in the schema of figure 2. *Xi* is the *i*-th input, *Wi* is the connection (synaptic) weight associated with input *i*. The synaptic weights are real numbers, because the synapses can inhibit (negative signal) or excite (positive signal) and have different intensities. The weighted inputs (*Xi* × *Wi*) are summed in the cell body, providing a signal *a*. After that, the signal *a* is input to an activation function (*f*), giving the neuron output.

**Figure 1.** The typical neuron. **Figure 2.** The neuron model.

The activation function can be: (1) hard limiter, (2) threshold logic, and (3) sigmoid, which is considered the biologically more plausible activation function.

#### **1.2. The perceptron**

2 Artificial Neural Networks

**1.1. McCulloch-Pitts neuron**

• neuron activity is an "all-or-none" process;

• network structure does not change along time.

McCulloch-Pitts neuron (1943) was the first mathematical model [32]. Its properties:

excite a neuron: independent of previous activity and of neuron position;

• synaptic delay is the only significant delay in nervous system; • activity of any inhibitory synapse prevents neuron from firing;

**Figure 1.** The typical neuron. **Figure 2.** The neuron model.

considered the biologically more plausible activation function.

• a certain fixed number of synapses are excited within a latent addition period in order to

The McCulloch-Pitts neuron represents a simplified mathematical model for the neuron, where *xi* is the *i*-th binary input and *wi* is the synaptic (connection) weight associated with the input *xi*. The computation occurs in soma (cell body). For a neuron with *p* inputs:

with *x*0 = 1 and *w*0 = *β* = −*θ*. *β* is the bias and *θ* is the activation threshold. See figures 1 and 2. The are *p* binary inputs in the schema of figure 2. *Xi* is the *i*-th input, *Wi* is the connection (synaptic) weight associated with input *i*. The synaptic weights are real numbers, because the synapses can inhibit (negative signal) or excite (positive signal) and have different intensities. The weighted inputs (*Xi* × *Wi*) are summed in the cell body, providing a signal *a*. After that, the signal *a* is input to an activation function (*f*), giving the neuron output.

The activation function can be: (1) hard limiter, (2) threshold logic, and (3) sigmoid, which is

*xiwi* (1)

*a* = *p* ∑ *i*=1 Rosenblatt's perceptron [47] takes a weighted sum of neuron inputs, and sends output 1 (spike) if this sum is greater than the activation threshold. It is a linear discriminator: given 2 points, a straight line is able to discriminate them. For some configurations of *m* points, a straight line is able to separate them in two classes (figures 3 and 4).

**Figure 3.** Set of linearly separable points. **Figure 4.** Set of non-linearly separable points.

The limitations of the perceptron is that it is an one-layer feed-forward network (non-recurrent); it is only capable of learning solution of linearly separable problems; and its learning algorithm (delta rule) does not work with networks of more than one layer.

#### **1.3. Neural network topology**

In cerebral cortex, neurons are disposed in columns, and most synapses occur between different columns. See the famous drawing by Ramón y Cajal (figure 5). In the extremely simplified mathematical model, neurons are disposed in layers (representing columns), and there is communication between neurons in different layers (see figure 6).

**Figure 5.** Drawing by Santiago Ramón y Cajal of neurons in the pigeon cerebellum. (A) denotes Purkinje cells, an example of a multipolar neuron, while (B) denotes granule cells, which are also multipolar [57].

**Von Neumann computer Biological neural system**

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

29

Non-content addressable Integrated with processor Distributed Content addressable

One or few Simple

Low speed A large number

Stored programs Distributed Parallel Self-learning

*Processor* Complex High speed

*Memory* Separated from processor Localized

*Computing* Centralized Sequential

*Reliability* Very vulnerable Robust

**Table 1.** Von Neumann's computer versus biological neural system [26].

then are the basis of further learning" [21].

auto-associative; if they are different, hetero-associative.

**1.5. Learning**

among facts.

*Expertise* Numeric and symbolic manipulations Perceptual problems

*Operational environment* Well-defined, well-constrained Poorly defined, unconstrained

The Canadian psychologist Donald Hebb established the bases for current connectionist learning algorithms: "When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased" [21]. Also, the word "connectionism" appeared for the first time: "The theory is evidently a form of connectionism, one of the switchboard variety, though it does not deal in direct connections between afferent and efferent pathways: not an 'S-R' psychology, if R means a muscular response. The connections server rather to establish autonomous central activities, which

According to Hebb, knowledge is revealed by associations, that is, the plasticity in Central Nervous System (CNS) allows synapses to be created and destroyed. Synaptic weights change values, therefore allow learning, which can be through internal self-organizing: encoding of new knowledge and reinforcement of existent knowledge. How to supply a neural substrate to association learning among world facts? Hebb proposed a hypothesis: connections between two nodes highly activated at the same time are reinforced. This kind of rule is a formalization of the associationist psychology, in which associations are accumulated among things that happen together. This hypothesis permits to model the CNS plasticity, adapting it to environmental changes, through excitatory and inhibitory strength of existing synapses, and its topology. This way, it allows that a connectionist network learns correlation

Connectionist networks learn through synaptic weight change, in most cases: it reveals statistical correlations from the environment. Learning may happen also through network topology change (in a few models). This is a case of probabilistic reasoning without a statistical model of the problem. Basically, two learning methods are possible with Hebbian learning: unsupervised learning and supervised learning. In unsupervised learning there is no teacher, so the network tries to find out regularities in the input patterns. In supervised learning, the input is associated with the output. If they are equal, learning is called

**Figure 6.** A 3-layer neural network. Notice that there are *A* + 1 input units, *B* + 1 hidden units, and *C* output units. *w*1 and *w*2 are the synaptic weight matrices between input and hidden layers and between hidden and output layers, respectively. The "extra" neurons in input and hidden layers, labeled 1, represent the presence of bias: the ability of the network to fire even in the absence of input signal.

#### **1.4. Classical ANN models**

Classical artificial neural networks models are based upon a simple description of the neuron, taking into account the presence of presynaptic cells and their synaptic potentials, the activation threshold, and the propagation of an action potential.

As advantages, we may say that ANNs are naturally parallel solution, robust, fault tolerant, they allow integration of information from different sources or kinds, are adaptive systems, that is, capable of learning, they show a certain autonomy degree in learning, and display a very fast recognizing performance.

And there are many limitations of ANNs. Among them, it is still very hard to explain its behavior, because of lacking of transparency, their solutions do not scale well, and they are computationally expensive for big problems, and yet very far from biological reality.

ANNs do not focus on real neuron details. The conductivity delays are neglected. The output signal is either discrete (e.g., 0 or 1) or a real number (e.g., between 0 and 1). The network input is calculated as the weighted sum of input signals, and it is transformed in an output signal via a simple function (e.g., a threshold function). See the main differences between the biological neural system and the conventional computer on table 1.

Andy Clark proposes three types of connectionism [2]: (1) the *first-generation* consisting of perceptron and cybernetics of the 1950s. They are simple neural structures of limited applications [30]; (2) the *second generation* deals with complex dynamics with recurrent networks in order to deal with spatio-temporal events; (3) the *third generation* takes into account more complex dynamic and time properties. For the first time, these systems use biological inspired modular architectures and algorithms. We may add a fourth type: a network which considers populations of neurons instead of individual ones and the existence of chaotic oscillations, perceived by electroencephalogram (EEG) analysis. The K-models are examples of this category [30].

**Table 1.** Von Neumann's computer versus biological neural system [26].

#### **1.5. Learning**

4 Artificial Neural Networks

the absence of input signal.

**1.4. Classical ANN models**

very fast recognizing performance.

examples of this category [30].

impoverished explanation of human brain characteristics.

**Figure 6.** A 3-layer neural network. Notice that there are *A* + 1 input units, *B* + 1 hidden units, and *C* output units. *w*1 and *w*2 are the synaptic weight matrices between input and hidden layers and between hidden and output layers, respectively. The "extra" neurons in input and hidden layers, labeled 1, represent the presence of bias: the ability of the network to fire even in

Classical artificial neural networks models are based upon a simple description of the neuron, taking into account the presence of presynaptic cells and their synaptic potentials, the activation threshold, and the propagation of an action potential. So, they represent

As advantages, we may say that ANNs are naturally parallel solution, robust, fault tolerant, they allow integration of information from different sources or kinds, are adaptive systems, that is, capable of learning, they show a certain autonomy degree in learning, and display a

And there are many limitations of ANNs. Among them, it is still very hard to explain its behavior, because of lacking of transparency, their solutions do not scale well, and they are

ANNs do not focus on real neuron details. The conductivity delays are neglected. The output signal is either discrete (e.g., 0 or 1) or a real number (e.g., between 0 and 1). The network input is calculated as the weighted sum of input signals, and it is transformed in an output signal via a simple function (e.g., a threshold function). See the main differences between the

Andy Clark proposes three types of connectionism [2]: (1) the *first-generation* consisting of perceptron and cybernetics of the 1950s. They are simple neural structures of limited applications [30]; (2) the *second generation* deals with complex dynamics with recurrent networks in order to deal with spatio-temporal events; (3) the *third generation* takes into account more complex dynamic and time properties. For the first time, these systems use biological inspired modular architectures and algorithms. We may add a fourth type: a network which considers populations of neurons instead of individual ones and the existence of chaotic oscillations, perceived by electroencephalogram (EEG) analysis. The K-models are

computationally expensive for big problems, and yet very far from biological reality.

biological neural system and the conventional computer on table 1.

The Canadian psychologist Donald Hebb established the bases for current connectionist learning algorithms: "When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased" [21]. Also, the word "connectionism" appeared for the first time: "The theory is evidently a form of connectionism, one of the switchboard variety, though it does not deal in direct connections between afferent and efferent pathways: not an 'S-R' psychology, if R means a muscular response. The connections server rather to establish autonomous central activities, which then are the basis of further learning" [21].

According to Hebb, knowledge is revealed by associations, that is, the plasticity in Central Nervous System (CNS) allows synapses to be created and destroyed. Synaptic weights change values, therefore allow learning, which can be through internal self-organizing: encoding of new knowledge and reinforcement of existent knowledge. How to supply a neural substrate to association learning among world facts? Hebb proposed a hypothesis: connections between two nodes highly activated at the same time are reinforced. This kind of rule is a formalization of the associationist psychology, in which associations are accumulated among things that happen together. This hypothesis permits to model the CNS plasticity, adapting it to environmental changes, through excitatory and inhibitory strength of existing synapses, and its topology. This way, it allows that a connectionist network learns correlation among facts.

Connectionist networks learn through synaptic weight change, in most cases: it reveals statistical correlations from the environment. Learning may happen also through network topology change (in a few models). This is a case of probabilistic reasoning without a statistical model of the problem. Basically, two learning methods are possible with Hebbian learning: unsupervised learning and supervised learning. In unsupervised learning there is no teacher, so the network tries to find out regularities in the input patterns. In supervised learning, the input is associated with the output. If they are equal, learning is called auto-associative; if they are different, hetero-associative.

#### **1.6. Back-propagation**

Back-propagation (BP) is a supervised algorithm for multilayer networks. It applies the generalized delta rule, requiring two passes of computation: (1) activation propagation (forward pass), and (2) error back propagation (backward pass). Back-propagation works in the following way: it propagates the activation from input to hidden layer, and from hidden to output layer; calculates the error for output units, then back propagates the error to hidden units and then to input units.

4. *Adaptation variables*, such as the activation of low voltage or current dependent on *Ca*2+.

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

31

The currents define the type of neuronal dynamical system [20]. There are millions of different electrophysiological spike generation mechanisms. Axons are filaments (there are 72 km of fiber in the brain) that can reach from 100 microns (typical granule cell), up to 4.5 meters (giraffe primary afferent). And communication via spikes may be stereotypical (common pyramidal cells), or no communication at all (horizontal cells of the retina). The speed of the action potential (spike) ranges from 2 to 400 km/h. The input connections ranges from 500 (retinal ganglion cells) to 200,000 (purkinje cells). In about 100 billion neurons in the human brain, there are hundreds of thousands of different types of neurons and at least one hundred neurotransmitters. Each neuron makes on average 1,000 synapses on other

The power of dynamical systems approach to neuroscience is that we can say many things

Consider a quiescent neuron whose membrane potential is at rest. Since there are no changes in their state variables, it is an equilibrium point. All incoming currents to depolarize the neuron are balanced or equilibrated by hyper-polarization output currents: stable equilibrium (figure 7(a) - top). Depending on the starting point, the system may have many trajectories, as those shown at the bottom of the figure 7. One can imagine the time along each trajectory. All of them are attracted to the equilibrium state denoted by the black dot, called *attractor* [25]. It is possible to predict the itinerant behavior of neurons through

Regarding Freeman's neurodynamics (see section 2.5) the most useful state variables are derived from electrical potentials generated by a neuron. Their recordings allow the definition of one state variable for axons and another one for dendrites, which are very different. The axon expresses its state in frequency of action potentials (pulse rate), and

The description of the dynamics can be obtained from a study of system phase portraits, which shows certain special trajectories (equilibria, separatrices, limit cycles) that determine

The excitability is illustrated in figure 7(b). When the neuron is at rest (phase portrait = stable equilibrium), small perturbations, such as *A*, result in small excursions from equilibrium, denoted by PSP (post-synaptic potential). Major disturbances, such as *B*, are amplified by

If a current strong enough is injected into the neuron, it will be brought to a pacemaker mode, which displays periodic spiking activity (figure 7(c)): this state is called the cycle stable limit, or stable periodic orbit. The details of the electrophysiological neuron only determine the

the intrinsic dynamics of neuron and result in the response of the action potential.

about a system without knowing all the details that govern its evolution.

dendrite expresses in intensity of its synaptic current (wave amplitude) [10].

the behavior of all other topological trajectory through the phase space.

position, shape and period of limit cycle.

They build prolonged action potentials and can affect the excitability over time.

**2.1. The neurons are different**

neurons [8].

**2.2. Phase portraits**

observation [10].

BP has a universal approximation power, that is, given a continuous function, there is a two-layer network (one hidden layer) that can be trained by Back-propagation in order to approximate as much as desired this function. Besides, it is the most used algorithm.

Although Back-propagation is a very known and most used connectionist training algorithm, it is computationally expensive (slow), it does not solve satisfactorily big size problems, and sometimes, the solution found is a local minimum - a locally minimum value for the error function.

BP is based on the error back propagation: while stimulus propagates forwardly, the error (difference between the actual and the desired outputs) propagates backwardly. In the cerebral cortex, the stimulus generated when a neuron fires crosses the axon towards its end in order to make a synapse onto another neuron input. Suppose that BP occurs in the brain; in this case, the error must have to propagate back from the dendrite of the postsynaptic neuron to the axon and then to the dendrite of the presynaptic neuron. It sounds unrealistic and improbable. Synaptic "weights" have to be modified in order to make learning possible, but certainly not in the way BP does. Weight change must use only local information in the synapse where it occurs. That's why BP seems to be so biologically implausible.

#### **2. Dynamical systems**

Neurons may be treated as *dynamical systems*, as the main result of Hodgkin-Huxley model [23]. A dynamical system consists of a set of variables that describe its state and a law that describes the evolution of state variables with time [25]. The Hodgkin-Huxley model is a dynamical system of four dimensions, because their status is determined solely by the membrane potential *V* and the variable opening (activation) and closing (deactivation) of ion channels *n*, *m* and *h* for persistent *K*+ and transient *Na*+ currents [1, 27, 28]. The law of evolution is given by a four-dimensional system of *ordinary differential equations* (ODE). Principles of neurodynamics describe the basis for the development of biologically plausible models of cognition [30].

All variables that describe the neuronal dynamics can be classified into four classes according to their function and time scale [25]:

4. *Adaptation variables*, such as the activation of low voltage or current dependent on *Ca*2+. They build prolonged action potentials and can affect the excitability over time.

#### **2.1. The neurons are different**

6 Artificial Neural Networks

function.

**2. Dynamical systems**

models of cognition [30].

1. *Membrane potential*.

the action potential.

to their function and time scale [25]:

**1.6. Back-propagation**

to hidden units and then to input units.

Back-propagation (BP) is a supervised algorithm for multilayer networks. It applies the generalized delta rule, requiring two passes of computation: (1) activation propagation (forward pass), and (2) error back propagation (backward pass). Back-propagation works in the following way: it propagates the activation from input to hidden layer, and from hidden to output layer; calculates the error for output units, then back propagates the error

BP has a universal approximation power, that is, given a continuous function, there is a two-layer network (one hidden layer) that can be trained by Back-propagation in order to approximate as much as desired this function. Besides, it is the most used algorithm.

Although Back-propagation is a very known and most used connectionist training algorithm, it is computationally expensive (slow), it does not solve satisfactorily big size problems, and sometimes, the solution found is a local minimum - a locally minimum value for the error

BP is based on the error back propagation: while stimulus propagates forwardly, the error (difference between the actual and the desired outputs) propagates backwardly. In the cerebral cortex, the stimulus generated when a neuron fires crosses the axon towards its end in order to make a synapse onto another neuron input. Suppose that BP occurs in the brain; in this case, the error must have to propagate back from the dendrite of the postsynaptic neuron to the axon and then to the dendrite of the presynaptic neuron. It sounds unrealistic and improbable. Synaptic "weights" have to be modified in order to make learning possible, but certainly not in the way BP does. Weight change must use only local information in the

Neurons may be treated as *dynamical systems*, as the main result of Hodgkin-Huxley model [23]. A dynamical system consists of a set of variables that describe its state and a law that describes the evolution of state variables with time [25]. The Hodgkin-Huxley model is a dynamical system of four dimensions, because their status is determined solely by the membrane potential *V* and the variable opening (activation) and closing (deactivation) of ion channels *n*, *m* and *h* for persistent *K*+ and transient *Na*+ currents [1, 27, 28]. The law of evolution is given by a four-dimensional system of *ordinary differential equations* (ODE). Principles of neurodynamics describe the basis for the development of biologically plausible

All variables that describe the neuronal dynamics can be classified into four classes according

2. *Excitation variables*, such as activation of a *Na*+ current. They are responsible for lifting

3. *Recovery variables*, such as the inactivation of a current *Na*+ and activation of a rapid current *K*+. They are responsible for re-polarization (lowering) of the action potential.

synapse where it occurs. That's why BP seems to be so biologically implausible.

The currents define the type of neuronal dynamical system [20]. There are millions of different electrophysiological spike generation mechanisms. Axons are filaments (there are 72 km of fiber in the brain) that can reach from 100 microns (typical granule cell), up to 4.5 meters (giraffe primary afferent). And communication via spikes may be stereotypical (common pyramidal cells), or no communication at all (horizontal cells of the retina). The speed of the action potential (spike) ranges from 2 to 400 km/h. The input connections ranges from 500 (retinal ganglion cells) to 200,000 (purkinje cells). In about 100 billion neurons in the human brain, there are hundreds of thousands of different types of neurons and at least one hundred neurotransmitters. Each neuron makes on average 1,000 synapses on other neurons [8].

#### **2.2. Phase portraits**

The power of dynamical systems approach to neuroscience is that we can say many things about a system without knowing all the details that govern its evolution.

Consider a quiescent neuron whose membrane potential is at rest. Since there are no changes in their state variables, it is an equilibrium point. All incoming currents to depolarize the neuron are balanced or equilibrated by hyper-polarization output currents: stable equilibrium (figure 7(a) - top). Depending on the starting point, the system may have many trajectories, as those shown at the bottom of the figure 7. One can imagine the time along each trajectory. All of them are attracted to the equilibrium state denoted by the black dot, called *attractor* [25]. It is possible to predict the itinerant behavior of neurons through observation [10].

Regarding Freeman's neurodynamics (see section 2.5) the most useful state variables are derived from electrical potentials generated by a neuron. Their recordings allow the definition of one state variable for axons and another one for dendrites, which are very different. The axon expresses its state in frequency of action potentials (pulse rate), and dendrite expresses in intensity of its synaptic current (wave amplitude) [10].

The description of the dynamics can be obtained from a study of system phase portraits, which shows certain special trajectories (equilibria, separatrices, limit cycles) that determine the behavior of all other topological trajectory through the phase space.

The excitability is illustrated in figure 7(b). When the neuron is at rest (phase portrait = stable equilibrium), small perturbations, such as *A*, result in small excursions from equilibrium, denoted by PSP (post-synaptic potential). Major disturbances, such as *B*, are amplified by the intrinsic dynamics of neuron and result in the response of the action potential.

If a current strong enough is injected into the neuron, it will be brought to a pacemaker mode, which displays periodic spiking activity (figure 7(c)): this state is called the cycle stable limit, or stable periodic orbit. The details of the electrophysiological neuron only determine the position, shape and period of limit cycle.

When the magnitude of the injected current increases, the limit cycle amplitude increases

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

33

**Figure 8.** Geometry of phase portraits of excitable systems near the four bifurcations can exemplify many neurocomputational

Systems with Andronov-Hopf bifurcations, either sub-critical or supercritical, exhibit low amplitude membrane potential oscillations, while systems with saddle bifurcations, both without and with invariant circle, do not. The existence of small amplitude oscillations

*Resonators* are neurons with reduced amplitude sub-threshold oscillations, and those which do not have this property are *integrators*. Neurons that exhibit co-existence of rest and spiking states, are called *bistable* and those which do not exhibit this feature are *monostable*. See

Inhibition prevents spiking in integrators, but promotes it in resonators. The excitatory inputs push the state of the system towards the shaded region of figure 8, while the inhibitory inputs push it out. In resonators, both excitation and inhibition push the state toward the

creates the possibility of resonance to the frequency of the incoming pulses [25].

properties. Figure taken from [25], available at http://www.izhikevich.org/publications/dsn.pdf.

**2.4. Integrators and resonators**

*2.4.1. Neurocomputational properties*

table 2.

shaded region [25].

and becomes a complete spiking limit cycle [25].

**Figure 7.** The neuron states: rest (a), excitable (b), and activity of periodic spiking (c). At the bottom, we see the trajectories of the system, depending on the starting point. Figure taken from [25], available at http://www.izhikevich.org/publications/dsn. pdf.

#### **2.3. Bifurcations**

Apparently, there is an injected current that corresponds to the transition from rest to continuous spiking, i.e. from the portrait phase of figure 7(b) to 7(c). From the point of view of dynamical systems, the transition corresponds to a *bifurcation* of the dynamical neuron, or a qualitative representation of the phase of the system.

In general, neurons are excitable because they are close to bifurcations from rest to spiking activity. The type of bifurcation depends on the electrophysiology of the neuron and determines its excitable properties. Interestingly, although there are millions of different electrophysiological mechanisms of excitability and spiking, there are only four different types of bifurcation of equilibrium that a system can provide. One can understand the properties of excitable neurons, whose currents were not measured and whose models are not known, since one can identify experimentally in which of the four bifurcations undergoes the rest state of the neuron [25].

The four bifurcations are shown in figure 8: saddle-node bifurcation, saddle-node on invariant circle, sub-critical Andronov-Hopf and supercritical Andronov-Hopf. In *saddle-node bifurcation*, when the magnitude of the injected current or other parameter of the bifurcation changes, a stable equilibrium correspondent to the rest state (black circle) is approximated by an unstable equilibrium (white circle). In *saddle-node bifurcation on invariant circle*, there is an invariant circle at the time of bifurcation, which becomes a limit cycle attractor. In *sub-critical Andronov-Hopf bifurcation*, a small unstable limit cycle shrinks to a equilibrium state and loses stability. Thus the trajectory deviates from equilibrium and approaches a limit cycle of high amplitude spiking or some other attractor. In the *supercritical Andronov-Hopf bifurcation*, the equilibrium state loses stability and gives rise to a small amplitude limit cycle attractor. When the magnitude of the injected current increases, the limit cycle amplitude increases and becomes a complete spiking limit cycle [25].

**Figure 8.** Geometry of phase portraits of excitable systems near the four bifurcations can exemplify many neurocomputational properties. Figure taken from [25], available at http://www.izhikevich.org/publications/dsn.pdf.

Systems with Andronov-Hopf bifurcations, either sub-critical or supercritical, exhibit low amplitude membrane potential oscillations, while systems with saddle bifurcations, both without and with invariant circle, do not. The existence of small amplitude oscillations creates the possibility of resonance to the frequency of the incoming pulses [25].

#### **2.4. Integrators and resonators**

8 Artificial Neural Networks

pdf.

**2.3. Bifurcations**

the rest state of the neuron [25].

**Figure 7.** The neuron states: rest (a), excitable (b), and activity of periodic spiking (c). At the bottom, we see the trajectories of the system, depending on the starting point. Figure taken from [25], available at http://www.izhikevich.org/publications/dsn.

Apparently, there is an injected current that corresponds to the transition from rest to continuous spiking, i.e. from the portrait phase of figure 7(b) to 7(c). From the point of view of dynamical systems, the transition corresponds to a *bifurcation* of the dynamical neuron, or

In general, neurons are excitable because they are close to bifurcations from rest to spiking activity. The type of bifurcation depends on the electrophysiology of the neuron and determines its excitable properties. Interestingly, although there are millions of different electrophysiological mechanisms of excitability and spiking, there are only four different types of bifurcation of equilibrium that a system can provide. One can understand the properties of excitable neurons, whose currents were not measured and whose models are not known, since one can identify experimentally in which of the four bifurcations undergoes

The four bifurcations are shown in figure 8: saddle-node bifurcation, saddle-node on invariant circle, sub-critical Andronov-Hopf and supercritical Andronov-Hopf. In *saddle-node bifurcation*, when the magnitude of the injected current or other parameter of the bifurcation changes, a stable equilibrium correspondent to the rest state (black circle) is approximated by an unstable equilibrium (white circle). In *saddle-node bifurcation on invariant circle*, there is an invariant circle at the time of bifurcation, which becomes a limit cycle attractor. In *sub-critical Andronov-Hopf bifurcation*, a small unstable limit cycle shrinks to a equilibrium state and loses stability. Thus the trajectory deviates from equilibrium and approaches a limit cycle of high amplitude spiking or some other attractor. In the *supercritical Andronov-Hopf bifurcation*, the equilibrium state loses stability and gives rise to a small amplitude limit cycle attractor.

a qualitative representation of the phase of the system.

*Resonators* are neurons with reduced amplitude sub-threshold oscillations, and those which do not have this property are *integrators*. Neurons that exhibit co-existence of rest and spiking states, are called *bistable* and those which do not exhibit this feature are *monostable*. See table 2.

#### *2.4.1. Neurocomputational properties*

Inhibition prevents spiking in integrators, but promotes it in resonators. The excitatory inputs push the state of the system towards the shaded region of figure 8, while the inhibitory inputs push it out. In resonators, both excitation and inhibition push the state toward the shaded region [25].

memory or cognitive maps, for materialists and cognitivists. In the pragmatism view there

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

35

The neurons in the brain form dense networks. The balance of excitation and inhibition allow them to have intrinsic oscillatory activity and overall amplitude modulation (AM) [10, 55]. These AM patterns are expressions of non-linear chaos, not merely a summation of linear dendritic and action potentials. AM patterns create attractor basins and landscapes. In the neurodynamical model every neuron participates, to some extent, in every experience and

The concepts of non-linear chaotic neurodynamics are of fundamental importance to nervous system research. They are relevant to our understanding of the workings of the normal

Typical neuron have many dendrites (input) and one axon (output). The axon transmits information using *microscopic* pulse trains. Dendrites integrate information using continuous waves of ionic current. Neurons are connected by synapses. Each synapse drives electric current. The microscopic current from each neuron sums with currents from other neurons, which causes a *macroscopic* potential difference, measured with a pair of extracellular electrodes (E) as the electroencephalogram (EEG) [10, 18]. EEG records the activity patterns of *mesoscopic* neuron populations. The sum of currents that a neuron generates in response to electrical stimulus produces the post-synaptic potential. The strength of the post-synaptic potential decreases with distance between the synapse and the cell body. The attenuation is compensated by greater surface area and more synapses on the distal dendrites. Dendrites make waves and axons make pulses. Synapses convert pulses to waves. Trigger zones convert waves to pulses. See figure 9. Researchers who base their studies on single neurons think that population events such as EEG are irrelevant noise, because they do not have understanding

**Figure 9.** Typical neuron showing the dendrites (input), the soma (cell body), the axon (output), the trigger zone, and the direction of the action potential. Notice that letters "E" represent the pair of extracellular electrodes. Adapted from [45]

In single neurons, microscopic pulse frequencies and wave amplitudes are measured, while in populations, macroscopic pulse and wave densities are measured. The neuron is microscopic and ensemble is mesoscopic. The flow of the current inside the neuron is revealed by a change in the membrane potential, measured with an electrode inside the cell body, evaluating the dendritic wave state variable of the single neuron. Recall that

is no temporary storage of images and no representational map.

every behavior, via non-linear chaotic mechanisms [10].

brain [55].

*2.5.1. Neuron populations*

of a mesoscopic state [10].

and [10].

**Table 2.** Neuron classification in integrators-resonators/monostable-bistable, according to the rest state bifurcation. Adapted from [25].

#### **2.5. Freeman neurodynamics**

Nowadays, two very different concepts co-exist in neuroscience, regarding the way how the brain operates as a whole [55]: (1) classical model, where the brain is described as consisting of a series of causal chains composed of nerve nets that operate in parallel (the conventional artificial neural networks [20]); (2) neurodynamical model, where the brain operates by non-linear dynamical chaos, which looks like noise but presents a kind of hidden order [10].

 Brain imaging is performed during normal behavior activity, and non-linear dynamics models these data.

In a dynamicist view, actions and choices made are responsible for creation of meanings in brains, and they are different from representations. Representations exist only in the world and have no meanings. The relation of neurons to meaning is not still well understood. In Freeman's opinion, although representations can be transferred between machines, meaning cannot be transferred between brains [10]. Brain activity is directed toward external objects, leading to creation of meaning through learning. Neuron populations are the key to understand the biology of intentionality.

Freeman argues that there are two basic units in brain organization: the *neuron* and the *neuron population*. Although neuron has been the base for neurobiology, masses of interacting neurons forming neuron populations are considered for a macroscopic view of the brain. Like neurons, neuron populations also have states and activity patterns, but they do (different) macroscopic things. Between the microscopic neuron and these macroscopic things, there are *mesoscopic* populations [10].

Neurobiologists usually claim that brains process information in a cause-and-effect manner: stimuli carry information that is conveyed in transformed information. What if stimuli are selected before appearance? This view fails in this case. This traditional view allowed the development of information processing machines. This simplified, or even mistaken, view of neuronal workings, led to the development of digital computers. Artificial Intelligence artifacts pose a challenge: how to attach meaning to the symbolic representations in machines?

Pragmatists conceive minds as dynamical systems, resulted from actions into the world. How are these actions generated? According to a cognitivist view, an action is determined by the form of a stimulus. Intentional action is composed by space-time processes, called short-term memory or cognitive maps, for materialists and cognitivists. In the pragmatism view there is no temporary storage of images and no representational map.

The neurons in the brain form dense networks. The balance of excitation and inhibition allow them to have intrinsic oscillatory activity and overall amplitude modulation (AM) [10, 55].

These AM patterns are expressions of non-linear chaos, not merely a summation of linear dendritic and action potentials. AM patterns create attractor basins and landscapes. In the neurodynamical model every neuron participates, to some extent, in every experience and every behavior, via non-linear chaotic mechanisms [10].

The concepts of non-linear chaotic neurodynamics are of fundamental importance to nervous system research.  Neuron populations*

10 Artificial Neural Networks

from [25].

machines?

**2.5. Freeman neurodynamics**

dynamics models these data.

understand the biology of intentionality.

things, there are *mesoscopic* populations [10].

**sub-threshold oscillations co-existence of rest and spiking states**

*no* (integrator) saddle-node saddle-node

*yes* (resonator) sub-critical supercritical

**Table 2.** Neuron classification in integrators-resonators/monostable-bistable, according to the rest state bifurcation. Adapted

Nowadays, two very different concepts co-exist in neuroscience, regarding the way how the brain operates as a whole [55]: (1) classical model, where the brain is described as consisting of a series of causal chains composed of nerve nets that operate in parallel (the conventional artificial neural networks [20]); (2) neurodynamical model, where the brain operates by non-linear dynamical chaos, which looks like noise but presents a kind of hidden order [10].  Brain imaging is performed during normal behavior activity, and non-linear

In a dynamicist view, actions and choices made are responsible for creation of meanings in brains, and they are different from representations. Representations exist only in the world and have no meanings. The relation of neurons to meaning is not still well understood. In Freeman's opinion, although representations can be transferred between machines, meaning cannot be transferred between brains [10]. Brain activity is directed toward external objects, leading to creation of meaning through learning. Neuron populations are the key to

Freeman argues that there are two basic units in brain organization: the *neuron* and the *neuron population*. Although neuron has been the base for neurobiology, masses of interacting neurons forming neuron populations are considered for a macroscopic view of the brain. Like neurons, neuron populations also have states and activity patterns, but they do (different) macroscopic things. Between the microscopic neuron and these macroscopic

Neurobiologists usually claim that brains process information in a cause-and-effect manner: stimuli carry information that is conveyed in transformed information. What if stimuli are selected before appearance? This view fails in this case. This traditional view allowed the development of information processing machines. This simplified, or even mistaken, view of neuronal workings, led to the development of digital computers. Artificial Intelligence artifacts pose a challenge: how to attach meaning to the symbolic representations in

Pragmatists conceive minds as dynamical systems, resulted from actions into the world. How are these actions generated? According to a cognitivist view, an action is determined by the form of a stimulus. Intentional action is composed by space-time processes, called short-term

*yes no* (bistable) (monostable)

Andronov-Hopf Andronov-Hopf

on invariant circle

Typical neuron have many dendrites (input) and one axon (output). The axon transmits information using *microscopic* pulse trains. Dendrites integrate information using continuous waves of ionic current. Neurons are connected by synapses. Each synapse drives electric current. The microscopic current from each neuron sums with currents from other neurons, which causes a *macroscopic* potential difference, measured with a pair of extracellular electrodes (E) as the electroencephalogram (EEG) [10, 18]. EEG records the activity patterns of *mesoscopic* neuron populations. The sum of currents that a neuron generates in response to electrical stimulus produces the post-synaptic potential. The strength of the post-synaptic potential decreases with distance between the synapse and the cell body. The attenuation is compensated by greater surface area and more synapses on the distal dendrites. Dendrites make waves and axons make pulses. Synapses convert pulses to waves. Trigger zones convert waves to pulses. See figure 9. Researchers who base their studies on single neurons think that population events such as EEG are irrelevant noise, because they do not have understanding of a mesoscopic state [10].

**Figure 9.** Typical neuron showing the dendrites (input), the soma (cell body), the axon (output), the trigger zone, and the direction of the action potential. Notice that letters "E" represent the pair of extracellular electrodes. Adapted from [45] and [10].

In single neurons, microscopic pulse frequencies and wave amplitudes are measured, while in populations, macroscopic pulse and wave densities are measured. The neuron is microscopic and ensemble is mesoscopic. The flow of the current inside the neuron is revealed by a change in the membrane potential, measured with an electrode inside the cell body, evaluating the dendritic wave state variable of the single neuron. Recall that extracellular electrodes are placed outside the neuron (see the Es in figure 9), so cortical potential provided by sum of dendritic currents in the neighborhood is measured. The same currents produce the membrane (intracellular) and cortical (extracellular) potentials, given two views of neural activity: the former, microscopic and the latter, mesoscopic [10].

Cortical neurons, because of their synaptic interactions, form neuron populations. Microscopic pulse and wave state variables are used to describe the activity of the single neurons that contribute to the population, and mesoscopic state variables (also pulse and wave) are used to describe the collective activities neurons give rise. Mass activity in the brain is described by a pulse density, instead of pulse frequency. This is done by recording from outside the cell the firing of pulses of many neurons simultaneously. The same current that controls the firings of neurons is measured by EEG, which does not allow to distinguish individual contributions. Fortunately, this is not necessary.

A population is a collection of neurons in a neighborhood, corresponding to a cortical column, which represents dynamical patterns of activity. The average pulse density in a population can never approach the peak pulse frequencies of single neurons. The activity of neighborhoods in the center of the dendritic sigmoid curve is very near linear. This simplifies the description of populations. Neuron populations are similar to mesoscopic ensembles in many complex systems [10]. The behavior of the microscopic elements is constrained by the embedding ensemble, and it cannot be understood outside a mesoscopic and macroscopic view.

**Figure 10.** Representation of (b) KI and (c) KII sets by networks of (a) KO sets. Available at [9].

equations numerically.

cognitive behaviors [13].

*2.5.3. Freeman's mass action*

rules of evidence [17].

**2.6. Neuropercolation**

resembles the frames in a movie [12, 13].

The advantages of KIII pattern classifiers on artificial neural networks are the small number of training examples needed, convergence to an attractor in a single step and geometric increase (rather than linear) in the number of classes with the number of nodes. The disadvantage is the increasing of the computational time needed to solve ordinary differential

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

37

The Katchalsky K-models use a set of ordinary differential equations with distributed parameters to describe the hierarchy of neuron populations beginning from micro-columns to hemispheres [31]. In relation to the standard KV, K-sets provide a platform for conducting analyzes of unified actions of the neocortex in the creation and control of intentional and

*Freeman's mass action* (FMA) [9] refers to collective synaptic actions neurons in the cortex exert on other neurons, synchronizing their firing of action potentials [17]. FMA expresses and conveys the meaning of sensory information in spatial patterns of cortical activity that

The prevailing concepts in neurodynamics are based on neural networks, which are Newtonian models, since they treated neural microscopic pulses as point processes in trigger zones and synapses. The FMA theory is Maxwellian because it treats the mesoscopic neural activity as a continuous distribution. The neurodynamics of the FMA includes microscopic neural operations that bring sensory information to sensory cortices and load the first percepts of the sensory cortex to other parts of the brain. The Newtonian dynamics can model cortical input and output functions but not the formation of percepts. The FMA needs a *paradigm shift*, because the theory is based on new experiments and techniques and new

*Neuropercolation* is a family of stochastic models based on the mathematical theory of probabilistic cellular automata on lattices and random graphs, motivated by the structural

The collective action of neurons forms activity patterns that go beyond the cellular level and approach the organism level. The formation of mesoscopic states is the first step for that. This way, the activity level is decided by the population, not by individuals [10]. The population is semi-autonomous. It has a point attractor, returning to the same level after its releasing. The state space of the neuron population is defined by the range of amplitudes that its pulse and wave densities can take.

#### *2.5.2. Freeman K-sets*

Regarding neuroscience at the mesoscopic level [10, 11], theoretical connection between the neuron activity at the microscopic level in small neural networks and the activity of cell assemblies in the mesoscopic scale is not well understood [16]. Katzir-Katchalsky suggests treating cell assemblies using thermodynamics forming a hierarchy of models of the dynamics of neuron populations [29] (*Freeman K-sets*): KO, KI, KII, KIII, KIV and KV. Katzir-Katchalsky is the reason for the K in Freeman K-sets.

The KO set represents a noninteracting collection of neurons. KI sets represent a collection of KO sets, which can be excitatory (KI*e*) or inhibitory (KI*i*). A KII set represents a collection of KI*e* and KI*i*. The KIII model consists of many interconnected KII sets, describing a given sensory system in brains. A KIV set is formed by the interaction of three KIII sets [30]. KV sets are proposed to model the scale-free dynamics of neocortex operating on KIV sets [16]. See the representation of KI and KII sets by networks of KO sets in figure 10 [9].

The K-sets mediate between the microscopic activity of small neural networks and the macroscopic activity of the brain. The topology includes excitatory and inhibitory populations of neurons and the dynamics is represented by ordinary differential equations (ODE) [16].

**Figure 10.** Representation of (b) KI and (c) KII sets by networks of (a) KO sets. Available at [9].

The advantages of KIII pattern classifiers on artificial neural networks are the small number of training examples needed, convergence to an attractor in a single step and geometric increase (rather than linear) in the number of classes with the number of nodes. The disadvantage is the increasing of the computational time needed to solve ordinary differential equations numerically.

The Katchalsky K-models use a set of ordinary differential equations with distributed parameters to describe the hierarchy of neuron populations beginning from micro-columns to hemispheres [31]. In relation to the standard KV, K-sets provide a platform for conducting analyzes of unified actions of the neocortex in the creation and control of intentional and cognitive behaviors [13].

#### *2.5.3. Freeman's mass action*

12 Artificial Neural Networks

view.

and wave densities can take.

*2.5.2. Freeman K-sets*

(ODE) [16].

extracellular electrodes are placed outside the neuron (see the Es in figure 9), so cortical potential provided by sum of dendritic currents in the neighborhood is measured. The same currents produce the membrane (intracellular) and cortical (extracellular) potentials, given

Cortical neurons, because of their synaptic interactions, form neuron populations. Microscopic pulse and wave state variables are used to describe the activity of the single neurons that contribute to the population, and mesoscopic state variables (also pulse and wave) are used to describe the collective activities neurons give rise. Mass activity in the brain is described by a pulse density, instead of pulse frequency. This is done by recording from outside the cell the firing of pulses of many neurons simultaneously. The same current that controls the firings of neurons is measured by EEG, which does not allow to distinguish

A population is a collection of neurons in a neighborhood, corresponding to a cortical column, which represents dynamical patterns of activity. The average pulse density in a population can never approach the peak pulse frequencies of single neurons. The activity of neighborhoods in the center of the dendritic sigmoid curve is very near linear. This simplifies the description of populations. Neuron populations are similar to mesoscopic ensembles in many complex systems [10]. The behavior of the microscopic elements is constrained by the embedding ensemble, and it cannot be understood outside a mesoscopic and macroscopic

The collective action of neurons forms activity patterns that go beyond the cellular level and approach the organism level. The formation of mesoscopic states is the first step for that. This way, the activity level is decided by the population, not by individuals [10]. The population is semi-autonomous. It has a point attractor, returning to the same level after its releasing. The state space of the neuron population is defined by the range of amplitudes that its pulse

Regarding neuroscience at the mesoscopic level [10, 11], theoretical connection between the neuron activity at the microscopic level in small neural networks and the activity of cell assemblies in the mesoscopic scale is not well understood [16]. Katzir-Katchalsky suggests treating cell assemblies using thermodynamics forming a hierarchy of models of the dynamics of neuron populations [29] (*Freeman K-sets*): KO, KI, KII, KIII, KIV and KV.

The KO set represents a noninteracting collection of neurons. KI sets represent a collection of KO sets, which can be excitatory (KI*e*) or inhibitory (KI*i*). A KII set represents a collection of KI*e* and KI*i*. The KIII model consists of many interconnected KII sets, describing a given sensory system in brains. A KIV set is formed by the interaction of three KIII sets [30]. KV sets are proposed to model the scale-free dynamics of neocortex operating on KIV sets [16].

The K-sets mediate between the microscopic activity of small neural networks and the macroscopic activity of the brain. The topology includes excitatory and inhibitory populations of neurons and the dynamics is represented by ordinary differential equations

See the representation of KI and KII sets by networks of KO sets in figure 10 [9].

two views of neural activity: the former, microscopic and the latter, mesoscopic [10].

individual contributions. Fortunately, this is not necessary.

Katzir-Katchalsky is the reason for the K in Freeman K-sets.

*Freeman's mass action* (FMA) [9] refers to collective synaptic actions neurons in the cortex exert on other neurons, synchronizing their firing of action potentials [17]. FMA expresses and conveys the meaning of sensory information in spatial patterns of cortical activity that resembles the frames in a movie [12, 13].

The prevailing concepts in neurodynamics are based on neural networks, which are Newtonian models, since they treated neural microscopic pulses as point processes in trigger zones and synapses. The FMA theory is Maxwellian because it treats the mesoscopic neural activity as a continuous distribution. The neurodynamics of the FMA includes microscopic neural operations that bring sensory information to sensory cortices and load the first percepts of the sensory cortex to other parts of the brain. The Newtonian dynamics can model cortical input and output functions but not the formation of percepts. The FMA needs a *paradigm shift*, because the theory is based on new experiments and techniques and new rules of evidence [17].

#### **2.6. Neuropercolation**

*Neuropercolation* is a family of stochastic models based on the mathematical theory of probabilistic cellular automata on lattices and random graphs, motivated by the structural and dynamical properties of neuron populations. The existence of phase transitions has been demonstrated both in discrete and continuous state space models, i.e., in specific probabilistic cellular automata and percolation models. Neuropercolation extends the concept of phase transitions for large interactive populations of nerve cells [31].

they are [31]. It is said that connectivity and dynamics are *scale-free* [13, 14], which states that the dynamics of the cortex is size independent, such that the brains of mice, men, elephants

Biologically Plausible Artificial Neural Networks

http://dx.doi.org/10.5772/54177

39

Scale-free dynamics of the neocortex are characterized by self-similarity of patterns of synaptic connectivity and spatio-temporal neural activity, seen in power law distributions of structural and functional parameters and in rapid state transitions between levels of the

A non-intrusive technique to allow direct *brain-computer interface* (BCI) can be a scalp EEG - an array of electrodes put on the head like a hat, which allows monitoring the cognitive behavior of animals and humans, by using brain waves to interact with the computer. It is a kind of a keyboard-less computer that eliminates the need for hand or voice interaction.

The Neurodynamics of Brain & Behavior group in the Computational Neurodynamics (CND) Lab at the University of Memphis's FedEx Institute of Technology is dedicated to research cognitive behavior of animals and humans including the use of molecular genetic or behavioral genetic approaches, to studies that involve the use of brain imaging techniques, to apply dynamical mathematical and computational models, to neuroethological studies. The research has three prongs of use for BCI: video/computer gaming; to support people with disabilities or physical constraints, such as the elderly; and to improve control of complex machinery, such as an aircraft and other military and civilian uses [24]. The direct brain-computer interface would give those with physical constraints or those operating

Similar to how they found seizure prediction markers, the plan is to use the data to analyze pre-motor movements, the changes in the brain that occur before there's actually movement, and apply that to someone who has a prosthetic device to allow them to better manipulate it. Since the brain is usually multitasking, the researchers will have to pick up the signal for

Instead of the computationally successful, but considered to be biologically implausible supervised Back-propagation [5, 48], the learning algorithm Bio*Rec* employed in Bio*θ*Pred [44, 46] is inspired by the Recirculation [22] and GeneRec [33] (GR) algorithms,