[84] Stanford, T.R., et al., *Perceptual decision making in less than 30 milliseconds.* Nat Neurosci,

[85] Roelfsema, P.R., V.A.F. Lamme, and H. Spekreijse, *Object-based attention in the primary*

[86] Moro, S.I., et al., *Neuronal Activity in the Visual Cortex Reveals the Temporal Order of*

[87] Lui, L.L. and T. Pasternak, *Representation of comparison signals in cortical area MT during a delayed direction discrimination task.* Journal of Neurophysiology, 2011. 106(3): p. 1260-

[88] Mitchell, J.F., K.A. Sundberg, and J.H. Reynolds, *Spatial Attention Decorrelates Intrinsic*

[89] Kara, P., P. Reinagel, and R.C. Reid, *Low response variability in simultaneously recorded*

[90] Swets, J.A., *Measuring the accuracy of diagnostic systems.* Science, 1988. 240(4857): p. 1285-

[91] Buschman, T.J. and E.K. Miller, *Serial, Covert Shifts of Attention during Visual Search Are Reflected by the Frontal Eye Fields and Correlated with Population Oscillations.* Neuron, 2009.

[92] Pillow, J.W., et al., *Spatio-temporal correlations and visual signalling in a complete neuronal*

[93] Williams, Z.M., et al., *Parietal activity and the perceived direction of ambiguous apparent*

*prefrontal and posterior parietal cortices.* Science, 2007. 315(5820): p. 1860-1862.

*behavioral correlations.* Journal of Neuroscience, 2006. 26(30): p. 7779-7790.

*visual cortex of the macaque monkey.* Nature, 1998. 395(6700): p. 376-381.

*Cognitive Operations.* Journal of Neuroscience, 2010. 30(48): p. 16293-16303.

*Activity Fluctuations in Macaque Area V4.* Neuron, 2009. 63(6): p. 879-888.

*retinal, thalamic, and cortical neurons.* Neuron, 2000. 27(3): p. 635-646.

*population.* Nature, 2008. 454(7207): p. 995-U37.

*motion.* Nature Neuroscience, 2003. 6(6): p. 616-623.

Over the past years there has been an increasing surge of interest in automated facial expression analysis but there are several situations where the analysis of the expressions cannot be successfully done with the different classical approaches due to the variations in the environmental conditions (like illumination changes).

A way to face these variations in the environmental conditions is by looking at the existing systems that efficiently locate, extract and analyse the face information. An example of these systems is the brain of the human being.

According to several researchers [2, 27, 36], emotion processing is done in the amygdala, an area located within medial temporal lobes in the brain. Experiments carried out by [35] demonstrated that face recognition process can be achieved with the simple facial features (eyebrows and eyes vs eyebrows) of threatening and friendly faces. These experiments confirm that emotion processing involves other areas like the visual cortex, e.g. V1 [32]. Later, [17] demonstrated that fast, automatic, and parallel processing of unattended emotional faces provide important insights into the specific and dissociated neural pathways in emotion and face perception.

Although, there are different areas involved in the analysis of facial expressions in the brain of the human being, there is no consensus about how this process is done.

Clues from the psychological point of view, like that proposed by Ekman and Friesen [9], point out the importance of the symmetry in this process and the facial asymmetry (with left part of the face stronger than that of right part) is apparent only with deliberate and non-spontaneous (non-natural) expressions. According to this work, the symmetrical expressions show a true or natural emotion. Nevertheless, the asymmetrical expressions show a false or non-natural emotion.

In this research, we take advantage of psychological point of view to interpret our results issued by symmetrical and asymmetrical face measures when we model the visual areas of

©2012 Castellanos Sánchez et al., licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

#### 2 Will-be-set-by-IN-TECH 186 Visual Cortex – Current Status and Perspectives Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 3

brain of the human being. Furthermore, we are inspired by early visual areas for processing the visual stream. Then, the interaction between neurons of visual cortex lead us to introduce our proposed architecture based on the detection of asymmetrical facial expressions and their clustering into natural/non-natural expressions.

**2.1. Psychological and physiological perspective**

other areas like the visual cortex, e.g. V1 [32].

**2.2. Neurophysiological perspective**

**2.3. Computational perspective**

face perception.

From the psychological and physiological perspective, the emotions can be analysed considering the symmetry of the expressions. According to Ekman and Friesen [9] through the analysis of symmetry it can be evaluated the veracity (naturalness) of an emotion. In the brain, the information about emotions is processed in the amygdala [2, 27, 36], an area located within medial temporal lobes in the brain. By using fMRI some researchers [1, 13, 32] have discovered that in this area the processing is modulated by the significance of faces, particularly with fearful expressions, also with others like gaze direction. Visual search paradigms have provided evidence the enhanced capture of attention by threatening faces. Experiments carried out by [35] demonstrated that the recognition process can be achieved with simple facial features (expressions of eyebrows and eyes together vs only eyebrows) of threatening and friendly faces. These experiments confirm that emotion processing involves

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 187

According to neurophysiological studies [15, 16], the task of facial expression recognition also occurs in some regions in the visual cortex (ventral and dorsal pathways). The neurons in these areas respond mainly to gesticulations and identity. In the case of gesticulations, the neurons in the superior temporal sulcus involve the dynamics of the face. In the case of identity, the neurons in the inferior temporal gyrus imply the analysis of invariant facial features. In both cases the processing is modulated by attentional mechanisms [37]. Later, [17] demonstrated that fast, automatic, and parallel processing of unattended emotional faces, provides important insights into the specific and dissociated neural pathways in emotion and

In the figure 1 we show the main pathways of processing visual stream in the cortex of the human being. The dorsal pathways analyse the motion and localisation of information, while ventral pathways analyse information about form and colour. Both pathways have the same source, V1. The visual cortex is the last specialised layer of evolution in mammals; the limbic system (not shown in the figure) is the first and primitive layer in the mammals where the

From the computational point of view, facial expression recognition implies the classification of facial motion and facial structure deformation into abstract representations completely based on visual information. To address this analysis it is necessary to perform two main tasks: face detection and facial feature extraction. In the first one, techniques have been developed to deal with the extraction and normalization of the face considering variations in the pose [10, 28] and illumination [3, 12]. However, according to [11] the main effort has been focused to feature facial extraction where techniques like appearance-based models [6, 18, 19]

Basically, the facial feature extraction has been approached by two main streams [11, 23]: facial deformation extraction models and facial motion extraction models. Motion extraction approaches focus on facial changes produced by facial expressions, whereas deformation-based methods contrast the visual information against face models to extract features produced by expressions and not by age deformation like wrinkles. The main

amygdala is situated in the temporal lobes. Then, it is covered by visual cortex.

work with significant variations in the acquired face images without normalization.

In this chapter, we will present an architecture composed of five stages in order to cluster into natural/non-natural facial expressions in a sequence of images.

The first stage process a sequence of images for eyes and mouth corners detection inspired by the sensibility of cone-cells in the eyes. With these corners we build an anthropometrical grid of six vertical bands.

The second stage is a bio-inspired treatment through three steps:

The third stage computes the ratio between the quantity of active neurons and the maximum quantity of neurons in each of six bands built in the first stage. A process inspired in the visual stream integration in IT (infero-temporal area).

The fourth stage compares each symmetrical band with an empirical threshold *κ* obtained in our experiments.

The end stage clusters in natural/non-natural expressions. This stage uses the results of the fourth stage with a bio-inspired neural network model.

In the following sections, we will present an overview of the state of the art in research of facial expressions. In addition, we present our proposed architecture to analyse the asymmetrical responses of the faces from a sequence of images. We also mention the experiments, discuss the results and finally comment on the future work.

## **2. State of the art**

The analysis of facial expressions constitute a critical and complex part of our non-verbal social interactions. Therefore, over the past years there has been an increasing surge of interest to create tools that allow to analyse the facial expression automatically. This analysis is a complex task; on one hand facial expression structure of human beings are different from one another. On the other hand, faces are analysed from sequences of images captured in different environments with variations of lighting, pose and scale changes that difficult the extraction of their structure. Traditionally, the analysis of facial expressions has been focused on identify emotions; however, both deformation and face information can be originated by other factors (like verbal communication or fatigue) rather than emotions. The emotions can be evaluated from two main perspectives: psychological, physiological and neurophysiological points of view versus computational point of view.

## **2.1. Psychological and physiological perspective**

2 Will-be-set-by-IN-TECH

brain of the human being. Furthermore, we are inspired by early visual areas for processing the visual stream. Then, the interaction between neurons of visual cortex lead us to introduce our proposed architecture based on the detection of asymmetrical facial expressions and their

In this chapter, we will present an architecture composed of five stages in order to cluster into

The first stage process a sequence of images for eyes and mouth corners detection inspired by the sensibility of cone-cells in the eyes. With these corners we build an anthropometrical grid

• Contrast and orientation detection inspired in the simple neurons in V1 (primary visual

The third stage computes the ratio between the quantity of active neurons and the maximum quantity of neurons in each of six bands built in the first stage. A process inspired in the visual

The fourth stage compares each symmetrical band with an empirical threshold *κ* obtained in

The end stage clusters in natural/non-natural expressions. This stage uses the results of the

In the following sections, we will present an overview of the state of the art in research of facial expressions. In addition, we present our proposed architecture to analyse the asymmetrical responses of the faces from a sequence of images. We also mention the experiments, discuss

The analysis of facial expressions constitute a critical and complex part of our non-verbal social interactions. Therefore, over the past years there has been an increasing surge of interest to create tools that allow to analyse the facial expression automatically. This analysis is a complex task; on one hand facial expression structure of human beings are different from one another. On the other hand, faces are analysed from sequences of images captured in different environments with variations of lighting, pose and scale changes that difficult the extraction of their structure. Traditionally, the analysis of facial expressions has been focused on identify emotions; however, both deformation and face information can be originated by other factors (like verbal communication or fatigue) rather than emotions. The emotions can be evaluated from two main perspectives: psychological, physiological and neurophysiological points of

• Integration and smoothing contours due to the performance of complex neurons in V1. • The coherence and integration activity of neurons in MT (middle temporal area) inspire us to detect the motion through a temporal interaction of our consecutive complex neurons.

clustering into natural/non-natural expressions.

of six vertical bands.

our experiments.

**2. State of the art**

natural/non-natural facial expressions in a sequence of images.

The second stage is a bio-inspired treatment through three steps:

cortex) that are modelled by Gabor-like filters.

stream integration in IT (infero-temporal area).

fourth stage with a bio-inspired neural network model.

the results and finally comment on the future work.

view versus computational point of view.

From the psychological and physiological perspective, the emotions can be analysed considering the symmetry of the expressions. According to Ekman and Friesen [9] through the analysis of symmetry it can be evaluated the veracity (naturalness) of an emotion. In the brain, the information about emotions is processed in the amygdala [2, 27, 36], an area located within medial temporal lobes in the brain. By using fMRI some researchers [1, 13, 32] have discovered that in this area the processing is modulated by the significance of faces, particularly with fearful expressions, also with others like gaze direction. Visual search paradigms have provided evidence the enhanced capture of attention by threatening faces. Experiments carried out by [35] demonstrated that the recognition process can be achieved with simple facial features (expressions of eyebrows and eyes together vs only eyebrows) of threatening and friendly faces. These experiments confirm that emotion processing involves other areas like the visual cortex, e.g. V1 [32].

## **2.2. Neurophysiological perspective**

According to neurophysiological studies [15, 16], the task of facial expression recognition also occurs in some regions in the visual cortex (ventral and dorsal pathways). The neurons in these areas respond mainly to gesticulations and identity. In the case of gesticulations, the neurons in the superior temporal sulcus involve the dynamics of the face. In the case of identity, the neurons in the inferior temporal gyrus imply the analysis of invariant facial features. In both cases the processing is modulated by attentional mechanisms [37]. Later, [17] demonstrated that fast, automatic, and parallel processing of unattended emotional faces, provides important insights into the specific and dissociated neural pathways in emotion and face perception.

In the figure 1 we show the main pathways of processing visual stream in the cortex of the human being. The dorsal pathways analyse the motion and localisation of information, while ventral pathways analyse information about form and colour. Both pathways have the same source, V1. The visual cortex is the last specialised layer of evolution in mammals; the limbic system (not shown in the figure) is the first and primitive layer in the mammals where the amygdala is situated in the temporal lobes. Then, it is covered by visual cortex.

## **2.3. Computational perspective**

From the computational point of view, facial expression recognition implies the classification of facial motion and facial structure deformation into abstract representations completely based on visual information. To address this analysis it is necessary to perform two main tasks: face detection and facial feature extraction. In the first one, techniques have been developed to deal with the extraction and normalization of the face considering variations in the pose [10, 28] and illumination [3, 12]. However, according to [11] the main effort has been focused to feature facial extraction where techniques like appearance-based models [6, 18, 19] work with significant variations in the acquired face images without normalization.

Basically, the facial feature extraction has been approached by two main streams [11, 23]: facial deformation extraction models and facial motion extraction models. Motion extraction approaches focus on facial changes produced by facial expressions, whereas deformation-based methods contrast the visual information against face models to extract features produced by expressions and not by age deformation like wrinkles. The main

Then, we extract the active neurons for each vertical band and compute their ratio of active

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 189

Finally, we detect the asymmetries for each pair of vertical bands and cluster it in three classes

In the next subsections we give a brief explanation of the steps of our approach where we

In our proposed approach, the images of each sequence are in RGB colour, because we use the colour during the pre-processing to detect eyes and mouth corners. The process is described below. We suggest sampling capture to 25 frames per second and 1024 × 768 pixels. High

In this stage we realize three steps which are (1) the eyes and mouth detection, (2) we build the grid and finally we utilize only six regions, the six vertical bands (6-RVB) to analyse it (with left side of the stronger than right one). These steps will be briefly explained in the following

In the first step, let *I* = {*I*1, *I*2, *I*3,...  We work on the rectangle that contains the face; next, we define the point corresponding to the center of the face (*xc* and *yc*) and two parameters (width and height) for the size of the face. We use these anthropometric measures in the order to define three regions of interest that probably contain the mouth and eyes (left and right). For each region we find the corresponding coordinates of detected corners in both eyes and mouth and these

In the second step, we use the coordinates of eyes and mouth corners to build an antropomorphical grid on the face. With the processing we can obtain twenty-five small regions, we also split the central part in two regions such as can see in the figure 2 obtaining

Finally in this stage, we use only six symmetrical columns (*R*1, *R*2, *R*3, *R*4, *R*5, *R*6) of proposed

The facial expressions carry much more information that cannot be extracted from traditional facial expression analysis. So we propose, from the bio-inspired point of view, a methodology to analyse the symmetry of facial expressions. The objective with this is to achieve both a perspective, according to brain mechanisms, of the interpretation of facial motion relevance and a methodology that takes advantage of the tolerance to illumination change of the brain

thirty small regions (the step anthropometrical segmentation using black line doted).

mechanisms. This can be divided into three main processing steps:

quantity neurons. Next, each vertical band is compared to their symmetrical band.

applying the self-organizing maps (SOM).

sampling and high resolution is desirable.

coordinates are used in the next step.

grid to analyse expressions in the face.

**3.3. Bio-inspired treatment**

**3.2. Pre-processing**

paragraphs.

assume that the face has been located correctly.

**3.1. Description of input image sequences**

**Figure 1.** The visual pathway. The information comes in from the retina, next it is integrated in LGN, treated in V1, and finally processed in two different pathways : dorsal (MT, MTS, etc.) and ventral (V4, IT, etc.). The amygdala are almond-shaped groups of nuclei located deep within the medial temporal lobes and perform a primary role in the processing and memory of emotional reactions.

difference between these two methods is that deformation-based methods can be applied to single images as well as image sequences, and motion methods need a sequence of images. In both cases the facial features may be processed holistically (the faces are analysed as a whole) or locally (focusing on features from specific areas). In the case of deformation methods, the extraction of features relies on shape and texture changes through a period of time. On one hand the Holistic approaches either use the whole face [7, 25] or partial information about regions in the face like mouth or eyes [14, 21, 31]. For the motion extraction methods, the features are extracted by analysing motion vectors: holistically [8, 29], and locally [26, 34, 39] obtained.

## **3. Proposed bio-inspired architecture**

A general diagram of the proposed methodology is shown in the figure 2. In the beginning, a sequence of images or video is captured in RGB format. Next, a processing based on green and red colours is realized to detect eyes and mouth corners. In this stage, we generate a grid of 6 × 5 based on coordinates of eyes and mouth corners. In this chapter, we only chose the six vertical bands for our experiments, the other combinations for analysis of the antropomorphical grid are subjects our future analysis.

Our bio-inspired treatment obtain the simple and complex neuron responses following the function of basic neurons of visual primary cortex (V1). Next, a temporal integration in complex neuron responses simulates the neurons in middle temporal cortex (MT).

Then, we extract the active neurons for each vertical band and compute their ratio of active quantity neurons. Next, each vertical band is compared to their symmetrical band.

Finally, we detect the asymmetries for each pair of vertical bands and cluster it in three classes applying the self-organizing maps (SOM).

In the next subsections we give a brief explanation of the steps of our approach where we assume that the face has been located correctly.

### **3.1. Description of input image sequences**

In our proposed approach, the images of each sequence are in RGB colour, because we use the colour during the pre-processing to detect eyes and mouth corners. The process is described below. We suggest sampling capture to 25 frames per second and 1024 × 768 pixels. High sampling and high resolution is desirable.

## **3.2. Pre-processing**

4 Will-be-set-by-IN-TECH

**Figure 1.** The visual pathway. The information comes in from the retina, next it is integrated in LGN, treated in V1, and finally processed in two different pathways : dorsal (MT, MTS, etc.) and ventral (V4, IT, etc.). The amygdala are almond-shaped groups of nuclei located deep within the medial temporal

difference between these two methods is that deformation-based methods can be applied to single images as well as image sequences, and motion methods need a sequence of images. In both cases the facial features may be processed holistically (the faces are analysed as a whole) or locally (focusing on features from specific areas). In the case of deformation methods, the extraction of features relies on shape and texture changes through a period of time. On one hand the Holistic approaches either use the whole face [7, 25] or partial information about regions in the face like mouth or eyes [14, 21, 31]. For the motion extraction methods, the features are extracted by analysing motion vectors: holistically [8, 29], and locally [26, 34, 39]

A general diagram of the proposed methodology is shown in the figure 2. In the beginning, a sequence of images or video is captured in RGB format. Next, a processing based on green and red colours is realized to detect eyes and mouth corners. In this stage, we generate a grid of 6 × 5 based on coordinates of eyes and mouth corners. In this chapter, we only chose the six vertical bands for our experiments, the other combinations for analysis of the

Our bio-inspired treatment obtain the simple and complex neuron responses following the function of basic neurons of visual primary cortex (V1). Next, a temporal integration in

complex neuron responses simulates the neurons in middle temporal cortex (MT).

lobes and perform a primary role in the processing and memory of emotional reactions.

obtained.

**3. Proposed bio-inspired architecture**

antropomorphical grid are subjects our future analysis.

In this stage we realize three steps which are (1) the eyes and mouth detection, (2) we build the grid and finally we utilize only six regions, the six vertical bands (6-RVB) to analyse it (with left side of the stronger than right one). These steps will be briefly explained in the following paragraphs.

In the first step, let *I* = {*I*1, *I*2, *I*3,...  We work on the rectangle that contains the face; next, we define the point corresponding to the center of the face (*xc* and *yc*) and two parameters (width and height) for the size of the face. We use these anthropometric measures in the order to define three regions of interest that probably contain the mouth and eyes (left and right). For each region we find the corresponding coordinates of detected corners in both eyes and mouth and these coordinates are used in the next step.

In the second step, we use the coordinates of eyes and mouth corners to build an antropomorphical grid on the face. With the processing we can obtain twenty-five small regions, we also split the central part in two regions such as can see in the figure 2 obtaining thirty small regions (the step anthropometrical segmentation using black line doted).

Finally in this stage, we use only six symmetrical columns (*R*1, *R*2, *R*3, *R*4, *R*5, *R*6) of proposed grid to analyse expressions in the face.

#### **3.3. Bio-inspired treatment**

The facial expressions carry much more information that cannot be extracted from traditional facial expression analysis. So we propose, from the bio-inspired point of view, a methodology to analyse the symmetry of facial expressions. The objective with this is to achieve both a perspective, according to brain mechanisms, of the interpretation of facial motion relevance and a methodology that takes advantage of the tolerance to illumination change of the brain mechanisms. This can be divided into three main processing steps:

#### 6 Will-be-set-by-IN-TECH 190 Visual Cortex – Current Status and Perspectives Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 7

phases.

as active neurons.

function *Gθ* (*x*ˆ, *y*ˆ) is defined as:

for the image *t*.

computationally by the oriented Gabor-like filters [4, 22]:

*Sθ*,*ϕ*(*x*, *y*, *t*) =

*Gθ* (*x*ˆ, *y*ˆ, *ϕ*) = *exp*

*Cθ* (*x*, *y*, *t*) =

where *Cθ* (*x*, *y*, *t*) represent complex cells responses, *π*

*C*(*x*, *y*, *t* − 1) obtaining the map of active complex neurons by

*D*(*x*, *y*, *t*) to compute the number of active neurons.

Then, the complex cells responses are estimated by

• Two simple neurons with different phase are merged to generate a complex neuron.

*t* ∑ *t*=0

where *Sθ* (*x*, *y*, *t*) are simple cells which is the result of the convolution between a pool of Gabor functions (*Gθ* (*x*ˆ, *y*ˆ)), in our case with 8 orientation and 2 phases, and the image *t* of the image sequences, (*x*ˆ, *y*ˆ) are the rotational components and (*x*, *y*) a position of the image. The Gabor

> − *x*ˆ2 + *γ*2*y*ˆ2 2*σ*2

where *x*ˆ = *x* cos *θ* + *y* sin *θ*, *y*ˆ = −*x* sin *θ* + *y* cos *θ*, *λ* is the length-width, *θ* represent the orientation, *ϕ* the phase, *σ* the standard deviation and *γ* represent the relation of aspect.

In the second step, the responses of two diffent phases of simple cells *Sθ* (*x*, *y*, *t*) are then integrated in the complex cells by using a non-linear model that allows to merge the responses.

anti-symmetric phases, respectively. Then, the eight orientations are integrated of the complex cells responses *Cθ* (*x*, *y*, *t*) for obtain one matrix of the map active neurons complex *C*(*x*, *y*, *t*)

Finally the third step, the neurons in the human brain are connected to MT neurons allowing a temporal processing that is defined as the temporal integration between *C*(*x*, *y*, *t*) and

where *D*(*x*, *y*, *t*) is the result of active complex neurons in the image *t*, *C*(*x*, *y*, *t*) and *C*(*x*, *y*, *t* − 1) are the current image and the previous image, respectively. We use the information

2 (*x*, *y*, *t*)2 + *Sθ*,− *π*

2 and −*π*

*D*(*x*, *y*, *t*) = *C*(*x*, *y*, *t*) − *C*(*x*, *y*, *t* − 1) (4)

*Sθ*, *π*

 *cos* 2*π x*ˆ *λ* + *ϕ*

*I*(*x*, *y*, *t*) ∗ *Gθ* (*x*ˆ, *y*ˆ, *ϕ*) (1)

2 (*x*, *y*, *t*)2 (3)

2 are the symmetric and

(2)

• A temporal integration between two consecutive complex neurons models MT neurons (Middle temporal area). The neurons with a major response to a threshold are considered

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 191

In the first proposed step, considering the processing done by the visual cortex, the first neurons that receive the stimuli from the eyes are the simple cells. Physiological evidence [5, 38] affirms that neuronal populations in the primary visual cortex (V1) of mammals exhibit contrast normalization. Neurons that respond strongly to simple visual stimuli, such as sinusoidal gratings, respond less well to the same stimuli when they are presented as part of a more complex stimulus which also excites other, neighbouring neurons. The behaviour of these neurons show a preference to specific orientations, which can be modelled

**Figure 2.** General architecture of bio-inspired proposed model. After video acquisition, a pre-processing generates an anthropometrical grid. Next, a treatment inspired in V1 determines the active neurons. Then, we compare two symmetrical regions: hears, eyes and nose to extract the vector composed of ratio of quantity active neurons. Finally, the asymmetries detection are evaluated with SOM to cluster into natural or non-natural facial expressions.

• Inspiring in simple neurons of V1 (primary visual cortex) and modelled by Gabor-like filters, we extract the simple active neurons in eight different orientations and two different phases.

6 Will-be-set-by-IN-TECH

**Figure 2.** General architecture of bio-inspired proposed model. After video acquisition, a pre-processing generates an anthropometrical grid. Next, a treatment inspired in V1 determines the active neurons. Then, we compare two symmetrical regions: hears, eyes and nose to extract the vector composed of ratio of quantity active neurons. Finally, the asymmetries detection are evaluated with SOM to cluster into

• Inspiring in simple neurons of V1 (primary visual cortex) and modelled by Gabor-like filters, we extract the simple active neurons in eight different orientations and two different

natural or non-natural facial expressions.

In the first proposed step, considering the processing done by the visual cortex, the first neurons that receive the stimuli from the eyes are the simple cells. Physiological evidence [5, 38] affirms that neuronal populations in the primary visual cortex (V1) of mammals exhibit contrast normalization. Neurons that respond strongly to simple visual stimuli, such as sinusoidal gratings, respond less well to the same stimuli when they are presented as part of a more complex stimulus which also excites other, neighbouring neurons. The behaviour of these neurons show a preference to specific orientations, which can be modelled computationally by the oriented Gabor-like filters [4, 22]:

$$\mathcal{S}\_{\theta,\varphi}(\mathbf{x},\mathbf{y},t) = \sum\_{t=0}^{t} I(\mathbf{x},\mathbf{y},t) \ast \mathcal{G}\_{\theta}(\mathbf{\hat{x}},\mathbf{\hat{y}},\boldsymbol{\varrho}) \tag{1}$$

where *Sθ* (*x*, *y*, *t*) are simple cells which is the result of the convolution between a pool of Gabor functions (*Gθ* (*x*ˆ, *y*ˆ)), in our case with 8 orientation and 2 phases, and the image *t* of the image sequences, (*x*ˆ, *y*ˆ) are the rotational components and (*x*, *y*) a position of the image. The Gabor function *Gθ* (*x*ˆ, *y*ˆ) is defined as:

$$\mathcal{G}\_{\theta}(\pounds, \mathfrak{H}, \mathfrak{p}) = \exp\left(-\frac{\mathfrak{k}^2 + \gamma^2 \mathfrak{p}^2}{2\sigma^2}\right) \cos\left(2\pi \frac{\mathfrak{k}}{\lambda} + \mathfrak{p}\right) \tag{2}$$

where *x*ˆ = *x* cos *θ* + *y* sin *θ*, *y*ˆ = −*x* sin *θ* + *y* cos *θ*, *λ* is the length-width, *θ* represent the orientation, *ϕ* the phase, *σ* the standard deviation and *γ* represent the relation of aspect.

In the second step, the responses of two diffent phases of simple cells *Sθ* (*x*, *y*, *t*) are then integrated in the complex cells by using a non-linear model that allows to merge the responses. Then, the complex cells responses are estimated by

$$\mathcal{C}\_{\theta}(\mathbf{x}, \mathbf{y}, t) = \sqrt{\mathcal{S}\_{\theta, \frac{\pi}{2}}(\mathbf{x}, \mathbf{y}, t)^2 + \mathcal{S}\_{\theta, -\frac{\pi}{2}}(\mathbf{x}, \mathbf{y}, t)^2} \tag{3}$$

where *Cθ* (*x*, *y*, *t*) represent complex cells responses, *π* 2 and −*π* 2 are the symmetric and anti-symmetric phases, respectively. Then, the eight orientations are integrated of the complex cells responses *Cθ* (*x*, *y*, *t*) for obtain one matrix of the map active neurons complex *C*(*x*, *y*, *t*) for the image *t*.

Finally the third step, the neurons in the human brain are connected to MT neurons allowing a temporal processing that is defined as the temporal integration between *C*(*x*, *y*, *t*) and *C*(*x*, *y*, *t* − 1) obtaining the map of active complex neurons by

$$D(\mathbf{x}, y, t) = \mathbb{C}(\mathbf{x}, y, t) - \mathbb{C}(\mathbf{x}, y, t - 1) \tag{4}$$

where *D*(*x*, *y*, *t*) is the result of active complex neurons in the image *t*, *C*(*x*, *y*, *t*) and *C*(*x*, *y*, *t* − 1) are the current image and the previous image, respectively. We use the information *D*(*x*, *y*, *t*) to compute the number of active neurons.

## **3.4. Extraction of active neurons**

In this stage, we realize the extraction of active complex neurons of the active map complex neurons (*D*(*x*, *y*, *t*)). For this, we use the six vertical bands and the map of active neurons. The 6-RVB overlapping in the map of active neurons. So, we compute the quantity of active neurons (QAN) for each region (*R*1, *R*2, *R*3, *R*4, *R*5, *R*6) and find the region with the maximum quantity of active neurons (QAN). The QAN is the number of active complex neurons in a region of the image.

sequences in this database and the *κ* index. With *Cκ* values we cluster in natural expression

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 193

These values are sending to SOM 1D that performs 5000 epochs, with a *η* = 1.000 to *η* = 0.0001

We take six columns of proposed face grid. The symmetrical regions are: R1 with R6 (ears),

The tests for our approach took only the colour sequence of images for the CK+ database while for FG-Net and LTI-HIT we take all sequences of images. The last database was split

Figure 3 shows a sequence of image of CK+ database for a non-natural expression. There are 11 asymmetries in both ears and nose regions, and 9 asymmetries in eyes region, the weighted sum is 10 = 11 ∗ 0.3 + 11 ∗ 0.2 + 9 ∗ 0.5. But this ratio is higher than fixed threshold *κ* = 7, then this sequence is non natural. We tested all the experiments for two seconds independently of sampling frequency (see table 1). We fix *κ* according table 1 for test our all available databases. In our experiments this threshold tolerates the generated asymmetries by

We test the databases Cohn-Kanade (CK+) (only colour images) [20], FG-Net [33], and LTI-HIT as shown in the table 2. We confirm that FG-Net is the most natural database. Our

In 2000 and 2010, The Institute of Electrical & Electronics Engineers at Grenoble, France, created the CK and CK+, respectively (the same CK database but 107 new sequences of which 33 are in color), consists of 123 persons with 7 different expressions per person: anger, contempt, disgust, fear, happy, sadness and surprise. The image sequences vary in duration (i.e. 10 to 60 frames) and incorporate the onset of a neutral frame at the moment of peak formation of the facial expressions. This database was captured to 30 frames per second with a resolution 640 × 480. The final frame of each image sequence was coded using FACS (Facial Action Coding System) which describes person's expression in terms of action units (AUs) [20]. Participants were instructed by an experimenter to perform a series of 23 facial displays including single action units and specified emotion like expressions of joy, surprise,

In 2006, Facial Expressions and Emotions from the Munich Technical University created the FG-Net database. This is an image database containing face images showing a number of persons performing six different basic emotions defined by Ekman & Friesen: anger, fear,

We used three databases to test our proposed approach: FG-Net, CK+, and LTI-HIT.

anger, disgust, fear, and sadness. Each display began and ended in a neutral face.

and non-natural expression.

**4. Experimental results**

and they were analysed statistically.

R2 with R5 (eyes) and R3 with R4 (nose).

per question in each interview (sequence of images).

illumination, environment, and personal conditions.

classification error was of 12% in average.

**4.1. Description of databases**

*4.1.1. Cohn-Kanade (CK+) database*

*4.1.2. FG-Net database*

So, we obtain six vectors of the quantity of active neurons (QAN): one for the QAN obtained and one for QAN expected for a region. The last one is the total possible QAN in a region. The first one computes only the active present neurons such that the ratio is always between 0 and 1. This information is used to compare two symmetric regions to detect temporal asymmetries.

## **3.5. Asymmetry detection**

Next stage, we detect the asymmetries for each region (6-RVB) of the map of active complex neurons. To obtain or detect asymmetries in each image during two seconds using 6-RVB we follow the next steps:

The detected asymmetries (*Out*) in the facial expression during the image sequences are weighed for each pair of symmetrical regions (*R*1 with *R*6, *R*2 with *R*5 and *R*3 with *R*4). For each database, we proposed an empirical threshold *κ* that we apply to all image sequence in the same database. This *κ* considers various aspects as the sampling frequency (we used two seconds in our experiments), the personal conditions, head motion and luminance-capture conditions.

#### **3.6. Clustering in natural/non-natural expressions**

For each different database we obtain a *κ* adapted to number of images in each sequence. So, we used the equation 5 for cluster the facial expressions in the natural/non-natural expressions:

$$\mathbf{C}\_{\mathbf{x}} = \frac{\mathbf{C}\_{A}}{\mathbf{C}\_{E}} \frac{2F}{T\kappa} \tag{5}$$

where *CA CE* is the ratio of quantity of active neurons (QAN) in a region, *T* is the total number of images in a typical sequence in the database to compute *κ*, *F* is the capture frequency for the sequences in this database and the *κ* index. With *Cκ* values we cluster in natural expression and non-natural expression.

These values are sending to SOM 1D that performs 5000 epochs, with a *η* = 1.000 to *η* = 0.0001 and they were analysed statistically.

## **4. Experimental results**

8 Will-be-set-by-IN-TECH

In this stage, we realize the extraction of active complex neurons of the active map complex neurons (*D*(*x*, *y*, *t*)). For this, we use the six vertical bands and the map of active neurons. The 6-RVB overlapping in the map of active neurons. So, we compute the quantity of active neurons (QAN) for each region (*R*1, *R*2, *R*3, *R*4, *R*5, *R*6) and find the region with the maximum quantity of active neurons (QAN). The QAN is the number of active complex neurons in a

So, we obtain six vectors of the quantity of active neurons (QAN): one for the QAN obtained and one for QAN expected for a region. The last one is the total possible QAN in a region. The first one computes only the active present neurons such that the ratio is always between 0 and 1. This information is used to compare two symmetric regions to detect temporal

Next stage, we detect the asymmetries for each region (6-RVB) of the map of active complex neurons. To obtain or detect asymmetries in each image during two seconds using 6-RVB we

• First step, we compute the difference between the symmetric regions. For example, the

• So, we calculate a threshold (a minimum and a maximum) using the average and standard

• Then, we detect the asymmetries for each difference of regions in the image sequences, using the "rule": for each difference of regions, we verify that the difference is outside the threshold (i.e., minor to minimum and major to maximum). We count each outsider in the

The detected asymmetries (*Out*) in the facial expression during the image sequences are weighed for each pair of symmetrical regions (*R*1 with *R*6, *R*2 with *R*5 and *R*3 with *R*4). For each database, we proposed an empirical threshold *κ* that we apply to all image sequence in the same database. This *κ* considers various aspects as the sampling frequency (we used two seconds in our experiments), the personal conditions, head motion and luminance-capture

For each different database we obtain a *κ* adapted to number of images in each sequence. So, we used the equation 5 for cluster the facial expressions in the natural/non-natural

2*F*

*CE* is the ratio of quantity of active neurons (QAN) in a region, *T* is the total number of images in a typical sequence in the database to compute *κ*, *F* is the capture frequency for the

*Tκ* (5)

*Cκ* = *CA CE*

• Then, compute average and standard deviation for each difference of the regions.

• Finally, we compute the *Out* for each pair of regions in the images sequence.

**3.6. Clustering in natural/non-natural expressions**

difference between region *R*1 and *R*6, that are the extreme regions.

**3.4. Extraction of active neurons**

region of the image.

**3.5. Asymmetry detection**

follow the next steps:

deviation.

sequence, *Out*.

conditions.

expressions:

where *CA*

asymmetries.

We take six columns of proposed face grid. The symmetrical regions are: R1 with R6 (ears), R2 with R5 (eyes) and R3 with R4 (nose).

The tests for our approach took only the colour sequence of images for the CK+ database while for FG-Net and LTI-HIT we take all sequences of images. The last database was split per question in each interview (sequence of images).

Figure 3 shows a sequence of image of CK+ database for a non-natural expression. There are 11 asymmetries in both ears and nose regions, and 9 asymmetries in eyes region, the weighted sum is 10 = 11 ∗ 0.3 + 11 ∗ 0.2 + 9 ∗ 0.5. But this ratio is higher than fixed threshold *κ* = 7, then this sequence is non natural. We tested all the experiments for two seconds independently of sampling frequency (see table 1). We fix *κ* according table 1 for test our all available databases. In our experiments this threshold tolerates the generated asymmetries by illumination, environment, and personal conditions.

We test the databases Cohn-Kanade (CK+) (only colour images) [20], FG-Net [33], and LTI-HIT as shown in the table 2. We confirm that FG-Net is the most natural database. Our classification error was of 12% in average.

## **4.1. Description of databases**

We used three databases to test our proposed approach: FG-Net, CK+, and LTI-HIT.

### *4.1.1. Cohn-Kanade (CK+) database*

In 2000 and 2010, The Institute of Electrical & Electronics Engineers at Grenoble, France, created the CK and CK+, respectively (the same CK database but 107 new sequences of which 33 are in color), consists of 123 persons with 7 different expressions per person: anger, contempt, disgust, fear, happy, sadness and surprise. The image sequences vary in duration (i.e. 10 to 60 frames) and incorporate the onset of a neutral frame at the moment of peak formation of the facial expressions. This database was captured to 30 frames per second with a resolution 640 × 480. The final frame of each image sequence was coded using FACS (Facial Action Coding System) which describes person's expression in terms of action units (AUs) [20]. Participants were instructed by an experimenter to perform a series of 23 facial displays including single action units and specified emotion like expressions of joy, surprise, anger, disgust, fear, and sadness. Each display began and ended in a neutral face.

#### *4.1.2. FG-Net database*

In 2006, Facial Expressions and Emotions from the Munich Technical University created the FG-Net database. This is an image database containing face images showing a number of persons performing six different basic emotions defined by Ekman & Friesen: anger, fear,

The tests for our approach took only the colour sequence of images for the CK+ database while for FG-Net and LTI-HIT we take all sequences images. The last database was split per

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 195

**Figure 4.** Results for face asymmetry detection in a sequence of images for FGNet. In this case, the boy is

The three databases are very different in its acquisition, illumination and manipulation of capture conditions. For weight these databases, we propose the *κ* index according to table 1. In natural conditions, facial changes on the left side are only about 2% greater than overall right-side changes [30]. We establish a level of 1 to 5 for the personal conditions (hair-style,

question.

suspect to deceitful clues.

**4.2. Remarks**

**Figure 3.** Asymmetry in vertical regions for a sequence of images in CK+. Horizontal red (black) line represents the data; horizontal green (gray) line, the difference between symmetrical regions; the separation between horizontal dashed lines is the symmetrical tolerance, and vertical dotted lines show the asymmetry (we show it only in the left graphics).

surprise, disgust, sadness and happiness. They also add the neutral expression for each person. This database consists of 19 persons with 21 sequences for each person, then it contains 399 different sequences. The sequences of images were captured to 25 frames per second with a resolution of 640 × 480 pixels [33]. This database allows to observe people react as natural as possible. As a consequence, it was tried to wake real emotions by playing video clips or still images after a short introductory phase instead of telling the person to play a role. This is in contrast to the asymmetry of luminance of the face, which includes not only asymmetry introduced by lighting, but also asymmetry introduced by the face itself.

#### *4.1.3. LTI-HIT database*

Finally, in 2010, the Children's Hospital of Tamaulipas created the LTI-HIT database. This database is taken from interviews that consists of 52 persons (11 men and 41 women) out of them 5 persons were put into one type of environment and 47 in the other. Each interview has between 27 and 33 different questions. The participants were between 18 and 66 years old. Each sequence of images were captured to 30 frames per second with resolution of 720 × 480 pixels and duration of 1.6 to 2.5 minutes per interviews. The interviews were made in natural conditions.

The tests for our approach took only the colour sequence of images for the CK+ database while for FG-Net and LTI-HIT we take all sequences images. The last database was split per question.

**Figure 4.** Results for face asymmetry detection in a sequence of images for FGNet. In this case, the boy is suspect to deceitful clues.

#### **4.2. Remarks**

10 Will-be-set-by-IN-TECH

**Figure 3.** Asymmetry in vertical regions for a sequence of images in CK+. Horizontal red (black) line represents the data; horizontal green (gray) line, the difference between symmetrical regions; the separation between horizontal dashed lines is the symmetrical tolerance, and vertical dotted lines show

surprise, disgust, sadness and happiness. They also add the neutral expression for each person. This database consists of 19 persons with 21 sequences for each person, then it contains 399 different sequences. The sequences of images were captured to 25 frames per second with a resolution of 640 × 480 pixels [33]. This database allows to observe people react as natural as possible. As a consequence, it was tried to wake real emotions by playing video clips or still images after a short introductory phase instead of telling the person to play a role. This is in contrast to the asymmetry of luminance of the face, which includes not only

Finally, in 2010, the Children's Hospital of Tamaulipas created the LTI-HIT database. This database is taken from interviews that consists of 52 persons (11 men and 41 women) out of them 5 persons were put into one type of environment and 47 in the other. Each interview has between 27 and 33 different questions. The participants were between 18 and 66 years old. Each sequence of images were captured to 30 frames per second with resolution of 720 × 480 pixels and duration of 1.6 to 2.5 minutes per interviews. The interviews were made in natural

asymmetry introduced by lighting, but also asymmetry introduced by the face itself.

the asymmetry (we show it only in the left graphics).

*4.1.3. LTI-HIT database*

conditions.

The three databases are very different in its acquisition, illumination and manipulation of capture conditions. For weight these databases, we propose the *κ* index according to table 1.

In natural conditions, facial changes on the left side are only about 2% greater than overall right-side changes [30]. We establish a level of 1 to 5 for the personal conditions (hair-style,

**Table 1.** *κ* index is the value for each of the three databases. PC, personal conditions (facial expression); HM, head movement, and LC, luminance conditions. Each feature is between 0 to 5.

skin colour and tolerable static face asymmetry), and, in average, we choose a level 3 to model all personal conditions (PC) for our available databases.

The head motion (HM) is very different too between these three databases. In CK+ all persons are very controlled and the original sequences were cut to show only the evident expression. Again, the FG-Net sequences show activities before and after an evident expression. Finally, the LTI-HIT sequences were not prepared for the major expression. It shows the natural reactions of the interviewed persons. Then, we assign to HM variable the values 1, 3 and 4 to each database, respectively.

The luminance and capture conditions (LC) is too different. CK+ was created in a studio, FG-Net in a controlled environment and LTI-HIT in an uncontrolled environment. Then, we assign to LC variable 2, 3 and 4 to each database, respectively.

**Figure 5.** Asymmetry based on the quantity index. The first DA and SOM blocks correspond to Natural

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 197

on average and standard deviation, while the SOM application shows the centroids guided

We have shown a bio-inspired approach to processing facial gesture for clustering into natural and non-natural facial expressions. It takes advantage of early primary visual areas in the brains of human beings for feature-extraction. Futhermore, it simulates the integration and discrimination of the superior visual areas considering the symmetrical and asymmetrical measures within the respective time span. This process ends in a bio-inspired neural network

The asymmetry between symmetrical regions of the face analysed during two seconds allows us to classify in: natural or non-natural facial expression. The quantity of asymmetries and its relations with different symmetrical regions is applied as a new biometric measure [24] (the precision of measurement is necessary). But we apply it for classification of natural or non

In our bio-inspired proposed architecture, after face detection, an anthropomorphical grid is proposed. For each region, we obtain the ratio of quantity of active neurons (QAN) obtained

This proposed model can work for any database. Furthermore, the database FG-Net and Cohn-Kanade were tested. More than 63% of CK+ video sequences contain asymmetries. This database was built with simulation expressions (non natural expressions) and we confirm

from the modelled simple and complex neurons by our Gabor-like filters.

facial expressions. The last two DA and SOM correspond to Non-Natural facial expression.

for clustering into natural and non-natural facial expressions.

natural expressions (the precision is not necessary).

by the density and dispersion.

**5. Conclusion**

**Table 2.** Natural/non-natural percentages according to *κ* index. The quantity of active neurons is manipulated into two manipulations: statistical analysis of active neurons (DA) and self-organizing maps (SOM). The percentages correspond to the obtained results with 12% ± 8% error with two expert responses.

The figure 4 shows another example of test for a sequence of images in FGNet database. The vertical dotted lines show the asymmetries (only shown the left side). The horizontal red lines are the ratios of quantity, while the horizontal green lines are the difference between symmetrical regions. Finally, the horizontal blue lines are tolerable threshold obtained from this sequence of images. In this case, the boy is suspected to deceitful clues because all the graphics show a higher value (22, 33 and 28 for each pair of vertical band) and the weighed sum is 29.3 = 22 ∗ 0.2 + 33 ∗ 0.5 + 28 ∗ 0.3 that is higher than *κ* = 18.0 index.

The table 2 resume the obtained ratios for natural/non-natural facial expressions with statistical analysis and SOM. If we see SOM percentages, these percentages show FG-Net as greater in natural facial expressions than the other two databases, CK+ also has greater percentage followed by LTI-HIT with non-natural facial expressions.

For statistical analysis we have the opposed results. This difference is due to concentration of values and the dispersion. With a simple statistical analysis, all responses are separated based

**Figure 5.** Asymmetry based on the quantity index. The first DA and SOM blocks correspond to Natural facial expressions. The last two DA and SOM correspond to Non-Natural facial expression.

on average and standard deviation, while the SOM application shows the centroids guided by the density and dispersion.

## **5. Conclusion**

12 Will-be-set-by-IN-TECH

Database (DB) PC HM LC *κ* index

**Table 1.** *κ* index is the value for each of the three databases. PC, personal conditions (facial expression);

skin colour and tolerable static face asymmetry), and, in average, we choose a level 3 to model

The head motion (HM) is very different too between these three databases. In CK+ all persons are very controlled and the original sequences were cut to show only the evident expression. Again, the FG-Net sequences show activities before and after an evident expression. Finally, the LTI-HIT sequences were not prepared for the major expression. It shows the natural reactions of the interviewed persons. Then, we assign to HM variable the values 1, 3 and

The luminance and capture conditions (LC) is too different. CK+ was created in a studio, FG-Net in a controlled environment and LTI-HIT in an uncontrolled environment.  Natural/non-natural percentages according to *κ* index. The quantity of active neurons is manipulated into two manipulations: statistical analysis of active neurons (DA) and self-organizing maps (SOM). The percentages correspond to the obtained results with 12% ± 8% error with two expert

sum is 29.3 = 22 ∗ 0.2 + 33 ∗ 0.5 + 28 ∗ 0.3 that is higher than *κ* = 18.0 index.

percentage followed by LTI-HIT with non-natural facial expressions.

The figure 4 shows another example of test for a sequence of images in FGNet database. The vertical dotted lines show the asymmetries (only shown the left side). The horizontal red lines are the ratios of quantity, while the horizontal green lines are the difference between symmetrical regions. Finally, the horizontal blue lines are tolerable threshold obtained from this sequence of images. In this case, the boy is suspected to deceitful clues because all the graphics show a higher value (22, 33 and 28 for each pair of vertical band) and the weighed

The table 2 resume the obtained ratios for natural/non-natural facial expressions with statistical analysis and SOM. If we see SOM percentages, these percentages show FG-Net as greater in natural facial expressions than the other two databases, CK+ also has greater

For statistical analysis we have the opposed results. This difference is due to concentration of values and the dispersion. With a simple statistical analysis, all responses are separated based

HM, head movement, and LC, luminance conditions. Each feature is between 0 to 5.

all personal conditions (PC) for our available databases.

assign to LC variable 2, 3 and 4 to each database, respectively.

4 to each database, respectively.

responses.

CK+ 3 1 2 6 FG-Net 3 3 3 9 LTI-HIT 3 4 4 11

> We have shown a bio-inspired approach to processing facial gesture for clustering into natural and non-natural facial expressions. It takes advantage of early primary visual areas in the brains of human beings for feature-extraction. Futhermore, it simulates the integration and discrimination of the superior visual areas considering the symmetrical and asymmetrical measures within the respective time span. This process ends in a bio-inspired neural network for clustering into natural and non-natural facial expressions.

> The asymmetry between symmetrical regions of the face analysed during two seconds allows us to classify in: natural or non-natural facial expression. The quantity of asymmetries and its relations with different symmetrical regions is applied as a new biometric measure [24] (the precision of measurement is necessary). But we apply it for classification of natural or non natural expressions (the precision is not necessary).

> In our bio-inspired proposed architecture, after face detection, an anthropomorphical grid is proposed. For each region, we obtain the ratio of quantity of active neurons (QAN) obtained from the modelled simple and complex neurons by our Gabor-like filters.

> This proposed model can work for any database. Furthermore, the database FG-Net and Cohn-Kanade were tested. More than 63% of CK+ video sequences contain asymmetries. This database was built with simulation expressions (non natural expressions) and we confirm

that our results are modelled correctly regarding the capture, illumination and personal conditions.

[8] Dornaika, F., Lazkano, E. & Sierra, B. [2011]. Improving dynamic facial expression recognition with feature subset selection, *Pattern Recognition Letters* 32(5): 740 – 748.

Bio-Inspired Architecture for Clustering into Natural and Non-Natural Facial Expressions 199

[10] Essa, I. A. & Pentland, A. P. [1997]. Coding, analysis, interpretation, and recognition of

[11] Fasel, B. & Luettin, J. [2003]. Automatic facial expression analysis: a survey, *Pattern*

[12] Fellenz, W., Taylor, J., Tsapatsoulis, N. & Kollias, S. [1999]. Comparing template-based, feature-based and supervised classification of facial expressions from static images,

[14] Gu, W., Xiang, C., Venkatesh, Y., Huang, D. & Lin, H. [2012]. Facial expression recognition using radial encoding of local gabor features and classifier synthesis, *Pattern*

[15] Hasselmo, M. E., Rolls, E. T. & Baylis, G. C. [1989]. The role of expression and identity in the face-selective responses of neurons in the temporal visual cortex of the monkey,

[16] Haxby, J. V., Hoffman, E. A. & Gobbini, M. I. [2000]. The distributed human neural

[17] Hung, Y., Smith, M. L., Bayle, D. J., Mills, T., Cheyne, D. & Taylor, M. J. [2010]. Unattended emotional faces elicit early lateralized amygdala-frontal and fusiform

[18] Kim, H.-C., Kim, D., Bang, S.-Y. & Lee, S.-Y. [2004]. Face recognition using the second-order mixture-of-eigenfaces method, *Pattern Recognition* 37(2): 337 – 349. [19] Lanitis, A., Taylor, C. J. & Cootes, T. F. [1997]. Automatic interpretation and coding of face images using flexible models, *IEEE Trans. Pattern Anal. Mach. Intell.* 19: 743–756. [20] Lucey, P., Cohn, J., Kanade, T., Saragih, J., Ambadar, Z. & Matthews, I. [2010]. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression, *Computer Vision and Pattern Recognition Workshops*

[21] Maalej, A., Amor, B. B., Daoudi, M., Srivastava, A. & Berretti, S. [2011]. Shape analysis of local facial patches for 3d facial expression recognition, *Pattern Recognition* 44(8): 1581

[22] Marcelja, S. [1980]. Mathematical description of the reponses of simple cortical cells,

[23] Mitra, S. & Acharya, T. [2007]. Gesture recognition: A survey, *IEEE Transactions on*

[24] Mitra, S. & Savvides, M. [2005]. Analyzing asymmetry biometric in the frequency

[25] Nair, B. M., Foytik, J., Tompkins, R., Diskin, Y., Aspiras, T. & Asari, V. [2011]. Multi-pose face recognition and tracking system, *Procedia Computer Science* 6(0): 381 – 386. [26] Park, S. & Kim, D. [2009]. Subtle facial expression recognition using motion

[27] Phillips, M. L., Young, A. W., Senior, C., Brammer, M., Andrews, C., Calder, A. J., Bullmore, E. T., Perrett, D. I., Rowland, D., Williams, S. C. R., Gray, J. A. & David, A. S.

system for face perception., *Trends in cognitive sciences* 4(6): 223–233.

*(CVPRW), 2010 IEEE Computer Society Conference on*, pp. 94–101.

*Journal of the Optical Society of America A* 70(11): 1297–1300.

*Systems, Man and Cybernetics - Part C* 37(3): 311–324.

domain for face recognition, *IEEE ICASSP* II: 953–956.

magnification, *Pattern Recognition Letters* 30(7): 708 – 716.

*Proceedings of Circuits, Systems, Communications and Computers* pp. 5331–5336. [13] Gosselin, F., Spezio, M. L., Tranel, D. & Adolphs, R. [2011]. Asymmetrical use of eye information from faces following unilateral amygdala damage, *Social Cognitive and*

[9] Ekman, P. [1980]. Asymmetry in facial expression, *Science* 209(4458): 833–834.

facial expressions, *IEEE Trans. Pattern Anal. Mach. Intell.* 19: 757–763.

*Recognition* 36(1): 259 – 275.

*Recognition* 45(1): 80 – 91.

– 1589.

*Affective Neuroscience* 6(3): 330–337.

*Behavioural Brain Research* 32(3): 203 – 218.

activations, *NeuroImage* 50(2): 727 – 733.

The different conditions of each database do not allow a simple characterization. We proposed an experimental index to model these conditions (e.g. personal conditions, head motion and luminance-capture), *κ*. In future, we will determine this index automatically for each database and, more precisely, for each sequence of images into database.

Using the index *κ* we have shown that our approach is independent of each database. This independence was tested in our experiments.

The preliminary steps of our approach show the feasibility to detect suspected persons in nervous or altered situations. Here is the beginning of our methodology to detect deceitful clues in facial expressions.

## **Author details**

#### Claudio Castellanos Sánchez

*Laboratory of Information Technology, Cinvestav-Tamaulipas and División de Estudios de Posgrado e Investigación, Instituto Tecnológico de Ciudad Victoria, Mexico*

#### Manuel Hernández Hernández

*Departamento de Ingeniería en Sistemas Computacionales, Instituto Tecnológico Superior de Tantoyuca, Mexico*

Pedro Luis Sánchez Orellana

*División de Estudios de Posgrado e Investigación, Instituto Tecnológico de Ciudad Victoria, Mexico*

## **6. References**

14 Will-be-set-by-IN-TECH

that our results are modelled correctly regarding the capture, illumination and personal

The different conditions of each database do not allow a simple characterization. We proposed an experimental index to model these conditions (e.g. personal conditions, head motion and luminance-capture), *κ*. In future, we will determine this index automatically for each database

Using the index *κ* we have shown that our approach is independent of each database. This

The preliminary steps of our approach show the feasibility to detect suspected persons in nervous or altered situations. Here is the beginning of our methodology to detect deceitful

*Laboratory of Information Technology, Cinvestav-Tamaulipas and División de Estudios de Posgrado e*

*Departamento de Ingeniería en Sistemas Computacionales, Instituto Tecnológico Superior de*

*División de Estudios de Posgrado e Investigación, Instituto Tecnológico de Ciudad Victoria, Mexico*

[1] Adolphs, R., Tranel, D. & Damasio, A. R. [Sep-1998]. The human amygdala in social

[2] Adolphs, R., Tranel, D., Damasio, H. & Damasio, A. R. [1995]. Fear and the human

[3] Belhumeur, P. N., Hespanha, J. & Kriegman, D. J. [1997]. Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, *IEEE Transactions on Pattern Analysis*

[4] Castellanos-Sánchez, C. [2005]. *Neuromimetic connectionist model for embedded visual perception of motion*, PhD thesis, Université Henri Poincaré (Nancy I), Nancy, France.

[5] Clatworthy, P., Chirimuuta, M., Lauritzen, J. & Tolhurst, D. [2003]. Coding of the contrasts in natural images by populations of neurons in primary visual cortex (v1),

[6] del Solar, J. R. & Quinteros, J. [2008]. Illumination compensation and normalization in eigenspace-based face recognition: A comparative study of different pre-processing

[7] Devi, B. J., Veeranjaneyulu, N. & Kishore, K. [2010]. A novel face recognition system based on combining eigenfaces with fisher faces using wavelets, *Procedia Computer*

and, more precisely, for each sequence of images into database.

*Investigación, Instituto Tecnológico de Ciudad Victoria, Mexico*

judgment., *Nature* 393(6684): 470–474.

*and Machine Intelligence* 19(7): 711–720.

Bibliothèque des Sciences et Techniques.

*Vision Research* 43(18): 1983 – 2001.

*Science* 2(0): 44 – 51.

amygdala, *The Journal of Neuroscience* 15(9): 5879–5891.

approaches, *Pattern Recognition Letters* 29(14): 1966 – 1979.

independence was tested in our experiments.

clues in facial expressions.

Claudio Castellanos Sánchez

Manuel Hernández Hernández

Pedro Luis Sánchez Orellana

**Author details**

*Tantoyuca, Mexico*

**6. References**

conditions.

[1997]. Aspecific neural substrate for perceiving facial expressions of disgust, *Nature* 389(6650): 174 – 194.

© 2012 Yang, licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use,

© 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution,

distribution, and reproduction in any medium, provided the original work is properly cited.

and reproduction in any medium, provided the original work is properly cited.

**Vision as a Fundamentally Statistical Machine**

As a vital driving force of systems neuroscience, visual neuroscience had its conceptual framework established more than 40 years ago based on Hubel and Wiesel's groundbreaking work on the receptive-field properties of visual neurons (Hubel & Wiesel, 1977). This framework was subsequently strengthened by David Marr's influential book (Marr, 2010). In this paradigm, visual neurons are conceived to perform bottom-up, imagebased processing to build a series of symbolic representations of visual stimuli. This paradigm, however, is deeply misleading since the generative sources in the threedimensional (3**D**) physical world of any stimulus, to which visual animals must respond successfully, cannot be determined by image-based processing (due to the inverse optics problem). This is perhaps the reason why "Now, thirty years later, the main problems that occupied Marr remain fundamental open problems in the study of perception" (Marr, 2010),

During the last 30 years, dramatic progress in computing hardware, digital imaging, statistical modeling, and visual neuroscience has promoted researchers to re-examine the computations and representations (see above) for natural vision examined in Marr's book. A range of new ideas have been proposed, many of which are summarized in books (Knill & Richards, 1996; Rao et al., 2002; Purves & Lotto, 2003; Doya et al., 2007; Trommershauser et al., 2011) and reviews (Simoncelli & Olshausen, 2001; Yuille & Kersten, 2006; Geisler, 2008; Friston, 2010). The unified theme is that vision and visual system structure and function must be understood in statistical terms. How this feat can be achieved, however, is not clear at all.

Since humans and other visual animals must respond successfully to visual stimuli whose generative sources cannot be determined in any direct way, the visual system can only generate percepts according to the probability distributions (**PDs**) of visual variables underlying the stimuli. The information pertinent to the generation of these PDs, namely, the statistics of natural visual environments, must have been incorporated into the visual circuitry by successful behavior in the world over evolutionary and developmental time.

as assessed by two prominent vision scientists and Marr's close associates.

Zhiyong Yang

**1. Introduction**

http://dx.doi.org/10.5772/50165

Additional information is available at the end of the chapter

## **Vision as a Fundamentally Statistical Machine**

Zhiyong Yang

16 Will-be-set-by-IN-TECH

[28] Rowley, H. A., Baluja, S. & Kanade, T. [1998]. Neural network-based face detection, *IEEE*

[29] Sánchez, A., Ruiz, J., Moreno, A., Montemayor, A., Hernández, J. & Pantrigo, J. [2011]. Differential optical flow applied to automatic facial expression recognition, *Neurocomputing* 74(8): 1272 – 1282. Selected Papers from the 3rd International Work-Conference on the Interplay between Natural and Artificial Computation

[30] Schmidt, K. L., Liu, Y. & Coh, J. F. [2006]. The role of structural facial asymmetry in in

[31] Venkatesh, Y., Kassim, A. A. & Murthy, O. R. [2009]. A novel approach to classification of facial expressions from 3d-mesh datasets using modified pca, *Pattern Recognition Letters*

[32] Vuilleumier, P. & Pourtois, G. [2007]. Distributed and interactive brain mechanisms during emotion face perception: Evidence from functional neuroimaging,

[34] Wang, T.-H. & Lien, J.-J. J. [2009]. Facial expression recognition system based on rigid and non-rigid motion separation and 3d pose estimation, *Pattern Recognition* 42(5): 962 –

[35] Weymar, M., Law, A., Ohman, A. & Hamm, A. O. [2011]. The face is more than its parts: Brain dynamics of enhanced spatial attention to schematic threat, *NeuroImage* 58(3): 946

[36] Whalen, P. J., Rauch, S. L., Etcoff, N. L., McInerney, S. C., Lee, M. B. & Jenike, M. A. [January-1998]. Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge., *The Journal of Neuroscience* 18(1): 411–418. [37] Williams, M. A., McGlone, F., Abbott, D. F. & Mattingley, J. B. [2005]. Differential amygdala responses to happy and fearful facial expressions depend on selective

[38] Willmore, B. D., Bulstrode, H. & Tolhurst, D. J. [2012]. Contrast normalization contributes to a biologically-plausible model of receptive-field development in primary visual cortex

[39] Xiang, T., Leung, M. & Cho, S. [2008]. Expression recognition using fuzzy

spatio-temporal modeling, *Pattern Recognition* 41(1): 204 – 216.

asymmetry peak facial expressions, *Literality* 11(6): 540–561.

[33] Wallhoff, F. [2006]. Facial expressions and emotion database. URL: *http://www.mmk.ei.tum.de/˜waf/fgnet/feedtum.html*

389(6650): 174 – 194.

(IWINAC 2009).

30(12): 1128 – 1137.

977.

– 954.

*Neuropsychologia* 45(1): 174 – 194.

attention, *NeuroImage* 24(2): 417 – 425.

(v1), *Vision Research* 54(0): 49 – 60.

*Trans. Pattern Anal. Mach. Intell.* 20: 23–38.

[1997]. Aspecific neural substrate for perceiving facial expressions of disgust, *Nature*

Additional information is available at the end of the chapter

http://dx.doi.org/10.5772/50165

## **1. Introduction**

As a vital driving force of systems neuroscience, visual neuroscience had its conceptual framework established more than 40 years ago based on Hubel and Wiesel's groundbreaking work on the receptive-field properties of visual neurons (Hubel & Wiesel, 1977). This framework was subsequently strengthened by David Marr's influential book (Marr, 2010). In this paradigm, visual neurons are conceived to perform bottom-up, imagebased processing to build a series of symbolic representations of visual stimuli. This paradigm, however, is deeply misleading since the generative sources in the threedimensional (3**D**) physical world of any stimulus, to which visual animals must respond successfully, cannot be determined by image-based processing (due to the inverse optics problem). This is perhaps the reason why "Now, thirty years later, the main problems that occupied Marr remain fundamental open problems in the study of perception" (Marr, 2010), as assessed by two prominent vision scientists and Marr's close associates.

During the last 30 years, dramatic progress in computing hardware, digital imaging, statistical modeling, and visual neuroscience has promoted researchers to re-examine the computations and representations (see above) for natural vision examined in Marr's book. A range of new ideas have been proposed, many of which are summarized in books (Knill & Richards, 1996; Rao et al., 2002; Purves & Lotto, 2003; Doya et al., 2007; Trommershauser et al., 2011) and reviews (Simoncelli & Olshausen, 2001; Yuille & Kersten, 2006; Geisler, 2008; Friston, 2010). The unified theme is that vision and visual system structure and function must be understood in statistical terms. How this feat can be achieved, however, is not clear at all.

Since humans and other visual animals must respond successfully to visual stimuli whose generative sources cannot be determined in any direct way, the visual system can only generate percepts according to the probability distributions (**PDs**) of visual variables underlying the stimuli. The information pertinent to the generation of these PDs, namely, the statistics of natural visual environments, must have been incorporated into the visual circuitry by successful behavior in the world over evolutionary and developmental time.

© 2012 Yang, licensee InTech. This is an open access chapter distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. © 2012 The Author(s). Licensee InTech. This chapter is distributed under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

During the last two decades, this statistical concept of vision has been successful in explaining aspects of vision that would be difficult to understand otherwise (see references cited above). In this chapter, I will describe several recent studies that relate the statistics of 2D and 3D natural visual scenes to visual percepts of brightness, saliency, and 3D space.

Vision as a Fundamentally Statistical Machine 203

(a), Standard simultaneous brightness contrast effect. The central square in the dark surround (left panel) appears brighter than the equiluminant square in the light surround (right panel). (b), White's illusion. Although the gray rectangles in the left panel are all equiluminant, the ones surrounded by the generally lighter context on the left appear brighter than those surrounded by the generally darker context on the right. When, however, the luminance of the target rectangles is the lowest (middle panel) or highest (right panel) value in the stimulus, the targets in the generally lighter context (on the left in the middle and right panels) appear less brighter than ones in the generally darker context (called the "inverted White's effect"). (c), Wertheimer-Benary illusion. The triangle embedded in the arm of the black cross appears brighter than the one that abuts the corner of the cross. The slightly different brightness of the equiluminant triangles is maintained whether the presentation is upside down (middle panel), or reflected along the diagonal (right panel). (d), The intertwined cross illusion. The target on the left appears substantially brighter than the equiluminant target on the right. (e), The inverted T-illusion. The inverted T-shape on the left appears somewhat

**Figure 1.** The influence of spatial patterns of luminance on the apparent brightness of a target (the

To examine whether the statistics of natural light patterns predict the perceptual phenomena shown in Fig. 1, we obtained the relevant PDs of luminance in natural scenes by sampling a database of natural scenes (van Hateren & van der Schaaf, 1998) with targetsurround configurations that had the same local geometry as the stimuli in Fig. 1. As a first step, these configurations were superimposed on the images to find light patterns in which the luminance values of both the surround and target regions were approximately homogeneous; for those configurations in which the surround comprised more than one region of the same luminance (see Fig. 1), we also required that the relevant sampled regions meet this criterion. The sampling configurations were moved in steps of one pixel to screen the full image. The mean luminance values of the target and the surrounding regions in the

**2.3. Brightness signifies context-mediated PDs of luminance in natural scenes**

Natural environments comprise objects of different sizes at various distances that are related to each other and the observer in a variety of ways (Yang & Purves, 2003a,b). When the light arising from objects is projected onto an image plane, these complex relationships are transformed into 2D patterns of light intensity with highly structured statistics. Thus, the PD

targets [T] in each stimulus are equiluminant, and are indicated in the insets on the right).

brighter than the equiluminant target on the right (modified from Yang & Purves, 2004).

**2.2. Context-mediated PDs of luminance in natural scenes**

samples were then calculated, and their occurrences tallied.

In the second section of this chapter, I will discuss how the PDs of luminance in specific contexts in natural scenes, referred to as the context-mediated PDs in natural scenes, predict brightness, the perception elicited by the luminance of a visual target. Our results show that brightness generated on this statistical basis accounts for a range of observations, whose causes have been debated for a long time without consensus. In the third section, I will present a simple, elegant model of the context- mediated PDs in natural scenes and a measure of visual saliency derived from these PDs. Our results show that this measure of visual saliency is a good predictor of human gaze in free-viewing both static and dynamic natural scenes. In the fourth section, I will present the statistics of 3D natural scenes and their relationship to human visual space. Our results show that human visual space is not a direct mapping of the 3D physical space but rather generated probabilistically. Finally, I will discuss the implications of these and other results for our understanding of the response properties of visual neurons, the intricate visual circuitries, the large-scale cortical organizations, the operational dynamics of the visual system, and natural vision.

## **2. The statistical structure of natural light patterns determines perceived light intensity**

## **2.1. Introduction**

In this section, I present evidence that the context-mediated PDs of luminance in natural scenes predict brightness, the perception elicited by the luminance of a visual target. A central puzzle in understanding how such percepts are generated by the visual system is that brightness does not correspond in any simple way to luminance. Thus, the same amount of light arising from a given region in a scene can elicit dramatically different brightness percepts when presented in different contexts (Fig. 1) (Kingdom, 2011). For example, in Fig.1 (a), the central square (T) in the left panel appears brighter than the same target in the right panel. This is the standard simultaneous brightness contrast effect.

A variety of explanations have been suggested since the basis for such phenomena was first debated by Helmholtz, Hering, Mach, and others (Gichrist et al., 1999; Purves et al., 2004; Kingdom, 2011). Although lateral inhibition in the early visual processing has often been proposed to account for these "illusions", this mechanism cannot explain instances in which similar overall contexts produce different brightness effects (compare Fig. 1 (a) with Figs. 1 (b) and (e); see also Fig. 1 (c)). This failure has led to several more recent suggestions, including complex filtering (Blakeslee & McCourt, 2004), the idea that brightness depends on detecting edges and junctions that promote the grouping of various luminances into interpretable spatial arrangements (Adelson, 2000; Anderson & Winawer, 2005), and the proposal that brightness is "re-synthesized" from 3D scene properties "inferred" from the stimulus (Wishart et al., 1997).

(a), Standard simultaneous brightness contrast effect. The central square in the dark surround (left panel) appears brighter than the equiluminant square in the light surround (right panel). (b), White's illusion. Although the gray rectangles in the left panel are all equiluminant, the ones surrounded by the generally lighter context on the left appear brighter than those surrounded by the generally darker context on the right. When, however, the luminance of the target rectangles is the lowest (middle panel) or highest (right panel) value in the stimulus, the targets in the generally lighter context (on the left in the middle and right panels) appear less brighter than ones in the generally darker context (called the "inverted White's effect"). (c), Wertheimer-Benary illusion. The triangle embedded in the arm of the black cross appears brighter than the one that abuts the corner of the cross. The slightly different brightness of the equiluminant triangles is maintained whether the presentation is upside down (middle panel), or reflected along the diagonal (right panel). (d), The intertwined cross illusion. The target on the left appears substantially brighter than the equiluminant target on the right. (e), The inverted T-illusion. The inverted T-shape on the left appears somewhat brighter than the equiluminant target on the right (modified from Yang & Purves, 2004).

**Figure 1.** The influence of spatial patterns of luminance on the apparent brightness of a target (the targets [T] in each stimulus are equiluminant, and are indicated in the insets on the right).

#### **2.2. Context-mediated PDs of luminance in natural scenes**

202 Visual Cortex – Current Status and Perspectives

**light intensity**

**2.1. Introduction**

stimulus (Wishart et al., 1997).

During the last two decades, this statistical concept of vision has been successful in explaining aspects of vision that would be difficult to understand otherwise (see references cited above). In this chapter, I will describe several recent studies that relate the statistics of 2D and 3D natural visual scenes to visual percepts of brightness, saliency, and 3D space.

In the second section of this chapter, I will discuss how the PDs of luminance in specific contexts in natural scenes, referred to as the context-mediated PDs in natural scenes, predict brightness, the perception elicited by the luminance of a visual target. Our results show that brightness generated on this statistical basis accounts for a range of observations, whose causes have been debated for a long time without consensus. In the third section, I will present a simple, elegant model of the context- mediated PDs in natural scenes and a measure of visual saliency derived from these PDs. Our results show that this measure of visual saliency is a good predictor of human gaze in free-viewing both static and dynamic natural scenes. In the fourth section, I will present the statistics of 3D natural scenes and their relationship to human visual space. Our results show that human visual space is not a direct mapping of the 3D physical space but rather generated probabilistically. Finally, I will discuss the implications of these and other results for our understanding of the response properties of visual neurons, the intricate visual circuitries, the large-scale cortical

organizations, the operational dynamics of the visual system, and natural vision.

**2. The statistical structure of natural light patterns determines perceived**

In this section, I present evidence that the context-mediated PDs of luminance in natural scenes predict brightness, the perception elicited by the luminance of a visual target. A central puzzle in understanding how such percepts are generated by the visual system is that brightness does not correspond in any simple way to luminance. Thus, the same amount of light arising from a given region in a scene can elicit dramatically different brightness percepts when presented in different contexts (Fig. 1) (Kingdom, 2011). For example, in Fig.1 (a), the central square (T) in the left panel appears brighter than the same

target in the right panel. This is the standard simultaneous brightness contrast effect.

A variety of explanations have been suggested since the basis for such phenomena was first debated by Helmholtz, Hering, Mach, and others (Gichrist et al., 1999; Purves et al., 2004; Kingdom, 2011). Although lateral inhibition in the early visual processing has often been proposed to account for these "illusions", this mechanism cannot explain instances in which similar overall contexts produce different brightness effects (compare Fig. 1 (a) with Figs. 1 (b) and (e); see also Fig. 1 (c)). This failure has led to several more recent suggestions, including complex filtering (Blakeslee & McCourt, 2004), the idea that brightness depends on detecting edges and junctions that promote the grouping of various luminances into interpretable spatial arrangements (Adelson, 2000; Anderson & Winawer, 2005), and the proposal that brightness is "re-synthesized" from 3D scene properties "inferred" from the To examine whether the statistics of natural light patterns predict the perceptual phenomena shown in Fig. 1, we obtained the relevant PDs of luminance in natural scenes by sampling a database of natural scenes (van Hateren & van der Schaaf, 1998) with targetsurround configurations that had the same local geometry as the stimuli in Fig. 1. As a first step, these configurations were superimposed on the images to find light patterns in which the luminance values of both the surround and target regions were approximately homogeneous; for those configurations in which the surround comprised more than one region of the same luminance (see Fig. 1), we also required that the relevant sampled regions meet this criterion. The sampling configurations were moved in steps of one pixel to screen the full image. The mean luminance values of the target and the surrounding regions in the samples were then calculated, and their occurrences tallied.

#### **2.3. Brightness signifies context-mediated PDs of luminance in natural scenes**

Natural environments comprise objects of different sizes at various distances that are related to each other and the observer in a variety of ways (Yang & Purves, 2003a,b). When the light arising from objects is projected onto an image plane, these complex relationships are transformed into 2D patterns of light intensity with highly structured statistics. Thus, the PD of the luminance of, say, the central target in a standard simultaneous brightness contrast stimulus (Fig. 2 (a)) depends on the surrounding luminance values (Fig. 2 (b)).

Vision as a Fundamentally Statistical Machine 205

luminance will always have a higher percentile, and will always elicit a perception of greater brightness compared to any luminance that has a lower percentile. Since the relation Brightness=A(P)+A is not based on a particular luminance within the context in question, but rather on the entire PD of possible luminance values experienced in that context, the context-dependent relationship between brightness and luminance is highly nonlinear (see Fig. 2 (c)). In consequence, the same physical difference between two luminance values will often signify different percentile differences, and thus perceived differences in brightness. Furthermore, because the percentiles change more rapidly as the target luminance approaches the luminance of the surround one would expect greater changes of brightness,

an expectation that corresponds to the well known "crispening" effect in perception.

equiluminant target in the right panel.

**2.4. White's illusion**

Finally, because the same value of target luminance will often correspond to different percentiles in the PDFs of target luminance in different contexts, two targets having the same luminance can elicit different brightness percepts, the higher percentile always corresponding to a brighter percept. Thus, in the standard simultaneous brightness contrast stimulus in Fig. 1 (a), the target (T) in the left panel in Fig. 2 (a) appears brighter than the

White's illusion (Fig. 1 (b)) presents a particular challenge for any explanation of brightness (White, 1979). The equiluminant rectangular areas surrounded by predominantly more luminant regions in the stimulus (on the left in the left panel of Fig. 1 (b); see also the area in the red frame in Fig. 3 (a) appear brighter than areas of identical luminance surrounded by less luminant regions (on the right in the left panel of Fig. 1 (b); see also the area in the blue frame in Fig. 3 (a). The especially perplexing characteristic of this percept is that the effect is opposite that elicited by standard simultaneous brightness contrast stimuli (Fig. 1 (a)). Even more puzzling, the effect reverses when the luminance of the rectangular targets is either the

The explanation for White's illusion provided by the statistical framework outlined above is shown in Fig. 3. When presented separately, as in Fig. 3 (a), the components of White's stimulus elicit much the same effect as in the usual presentation. By sampling the images of natural visual environments using configurations based on these components (Fig. 3 (b)), we obtained the PDFs of the luminance of a rectangular target (T) embedded in the two different configurations of surrounding luminance in White's stimulus. As shown in Fig. 3 (c), when the target in the intermediate range of luminance values (i.e., in between the luminance values at the two crossover points) abuts two dark rectangles laterally (left panel in Fig. 3 (b)), the percentile of the target luminance (red line) is higher than the percentile when the target abuts the two light rectangles (right panel in Fig. 3(b); blue line in Fig. 3 (c)). If, as we suppose, the percentile in the PDF of target luminance within any specific context determines the brightness perceived, the target with an intermediate luminance on the left in Fig. 3 (b) should appear brighter than the equiluminant target on the right. Finally, when all the luminance values in the stimulus are limited to a very narrow range (e.g., from 0 to 100 cd/m2 or from 1000 to 1100 cd/m2), when the sampling configurations are orientated vertically, or when the aspect ratio of

lowest or highest value in the stimulus (middle and right panels in Fig. 1 (b).

Fig. 2 (c) illustrates the supposition that, for any context, the visual system generates the brightness of a target according to the value of its luminance in the probability distribution function (PDF, the integral of PD) of the possible target luminance experienced in that context (Yang & Purves, 2004). This value is referred to subsequently as the percentile of the target luminance among all possible luminance values that co-occur with the contextual luminance pattern in the natural environment. In formal terms, this supposition means that the visual system generates brightness percepts according to the relationship Brightness=A(P)+A, where A and A are constants, and (P) is a monotonically increasing function of the PDF, P.

(a), The brightness elicited by a given target luminance in any context depends on the frequency of occurrence of that luminance relative to all the possible target luminance values experienced in that context in natural environments. This concept is illustrated here using the standard simultaneous brightness contrast stimulus in Fig. 1 (a*)*. The series of squares with different luminance values indicate all the possible occurrences of luminance in the target (T) in the two different contexts; the symbol indicates the relationship of a particular occurrence of luminance to the all possible occurrences of target luminance values experienced in the two contexts in natural environments. (b), This statistical relationship can be derived from the PD of target luminance values co-occurring with the luminance values and pattern of the two contexts of interest. The red and blue curves indicate the PDs of the luminance of the targets in (a), obtained by sampling the natural image database. The size of the sampling configuration was 1°x1°. (c), The brightness elicited by the luminance of the targets in (a) is based on the percentile of that luminance in the PDFs for the two different contexts, which are indicated by the icons (modified from Yang & Purves, 2004).

**Figure 2.** Brightness percepts signify context-mediated PDs of luminance in natural scenes.

By definition, then, the percentile of target luminance for the lowest luminance value within any contextual light pattern is 0% and corresponds to the perception of maximum darkness; the percentile for the highest luminance within any contextual pattern is 100% and corresponds to the maximum perceivable brightness. In any given context, a higher luminance will always have a higher percentile, and will always elicit a perception of greater brightness compared to any luminance that has a lower percentile. Since the relation Brightness=A(P)+A is not based on a particular luminance within the context in question, but rather on the entire PD of possible luminance values experienced in that context, the context-dependent relationship between brightness and luminance is highly nonlinear (see Fig. 2 (c)). In consequence, the same physical difference between two luminance values will often signify different percentile differences, and thus perceived differences in brightness. Furthermore, because the percentiles change more rapidly as the target luminance approaches the luminance of the surround one would expect greater changes of brightness, an expectation that corresponds to the well known "crispening" effect in perception.

Finally, because the same value of target luminance will often correspond to different percentiles in the PDFs of target luminance in different contexts, two targets having the same luminance can elicit different brightness percepts, the higher percentile always corresponding to a brighter percept. Thus, in the standard simultaneous brightness contrast stimulus in Fig. 1 (a), the target (T) in the left panel in Fig. 2 (a) appears brighter than the equiluminant target in the right panel.

## **2.4. White's illusion**

204 Visual Cortex – Current Status and Perspectives

function of the PDF, P.

of the luminance of, say, the central target in a standard simultaneous brightness contrast

Fig. 2 (c) illustrates the supposition that, for any context, the visual system generates the brightness of a target according to the value of its luminance in the probability distribution function (PDF, the integral of PD) of the possible target luminance experienced in that context (Yang & Purves, 2004). This value is referred to subsequently as the percentile of the target luminance among all possible luminance values that co-occur with the contextual luminance pattern in the natural environment. In formal terms, this supposition means that the visual system generates brightness percepts according to the relationship Brightness=A(P)+A, where A and A are constants, and (P) is a monotonically increasing

(a), The brightness elicited by a given target luminance in any context depends on the frequency of occurrence of that luminance relative to all the possible target luminance values experienced in that context in natural environments. This concept is illustrated here using the standard simultaneous brightness contrast stimulus in Fig. 1 (a*)*. The series of squares with different luminance values indicate all the possible occurrences of luminance in the target (T) in the two different contexts; the symbol indicates the relationship of a particular occurrence of luminance to the all possible occurrences of target luminance values experienced in the two contexts in natural environments. (b), This statistical relationship can be derived from the PD of target luminance values co-occurring with the luminance values and pattern of the two contexts of interest. The red and blue curves indicate the PDs of the luminance of the targets in (a), obtained by sampling the natural image database. The size of the sampling configuration was 1°x1°. (c), The brightness elicited by the luminance of the targets in (a) is based on the percentile of that luminance in the PDFs for the two

different contexts, which are indicated by the icons (modified from Yang & Purves, 2004).

**Figure 2.** Brightness percepts signify context-mediated PDs of luminance in natural scenes.

By definition, then, the percentile of target luminance for the lowest luminance value within any contextual light pattern is 0% and corresponds to the perception of maximum darkness; the percentile for the highest luminance within any contextual pattern is 100% and corresponds to the maximum perceivable brightness. In any given context, a higher

stimulus (Fig. 2 (a)) depends on the surrounding luminance values (Fig. 2 (b)).

White's illusion (Fig. 1 (b)) presents a particular challenge for any explanation of brightness (White, 1979). The equiluminant rectangular areas surrounded by predominantly more luminant regions in the stimulus (on the left in the left panel of Fig. 1 (b); see also the area in the red frame in Fig. 3 (a) appear brighter than areas of identical luminance surrounded by less luminant regions (on the right in the left panel of Fig. 1 (b); see also the area in the blue frame in Fig. 3 (a). The especially perplexing characteristic of this percept is that the effect is opposite that elicited by standard simultaneous brightness contrast stimuli (Fig. 1 (a)). Even more puzzling, the effect reverses when the luminance of the rectangular targets is either the lowest or highest value in the stimulus (middle and right panels in Fig. 1 (b).

The explanation for White's illusion provided by the statistical framework outlined above is shown in Fig. 3. When presented separately, as in Fig. 3 (a), the components of White's stimulus elicit much the same effect as in the usual presentation. By sampling the images of natural visual environments using configurations based on these components (Fig. 3 (b)), we obtained the PDFs of the luminance of a rectangular target (T) embedded in the two different configurations of surrounding luminance in White's stimulus. As shown in Fig. 3 (c), when the target in the intermediate range of luminance values (i.e., in between the luminance values at the two crossover points) abuts two dark rectangles laterally (left panel in Fig. 3 (b)), the percentile of the target luminance (red line) is higher than the percentile when the target abuts the two light rectangles (right panel in Fig. 3(b); blue line in Fig. 3 (c)). If, as we suppose, the percentile in the PDF of target luminance within any specific context determines the brightness perceived, the target with an intermediate luminance on the left in Fig. 3 (b) should appear brighter than the equiluminant target on the right. Finally, when all the luminance values in the stimulus are limited to a very narrow range (e.g., from 0 to 100 cd/m2 or from 1000 to 1100 cd/m2), when the sampling configurations are orientated vertically, or when the aspect ratio of the sampling configurations is changed (e.g., from 1:2 to 1:5), the PDFs derived from the database are not much different. These further results are consistent with the observations that White's stimulus elicits the similar effect when presented at a wide range of overall luminance levels, in a vertical orientation, or with different aspect ratios.

Vision as a Fundamentally Statistical Machine 207

The explanation of the Wertheimer-Benary illusion provided by the statistical framework outlined above is shown in Fig. 4. By sampling the images of natural environments using configurations based on the components of the stimulus (Fig. 4 (b)), we obtained the PDFs of target luminance in these contexts. As shown in Fig. 4 (c), when the triangular patch is embedded in a dark bar with its base facing a lighter area, the percentile of the luminance of the triangular patch (red line) is higher than the percentile when the triangular patch abuts a dark corner with its base facing a similar light background (blue line). Accordingly, the same gray patch should appear brighter in the former context than in the latter, as is the case. The PDFs obtained after changing the triangles to rectangles, rotating the configurations in Fig. 4 (b) by 180, or reflecting the configurations along the diagonal of the cross (cf. middle and right panels in Fig. 1 (c)) were much the same as those shown in Fig. 4 (c). These several observations accord with the fact that the Wertheimer-Benary effect is little

(a), The usual presentation of the Wertheimer-Benary stimulus. The separated components of the stimulus (boxed areas) elicit about the same effect as the usual presentation. (b), Configurations used to sample the database (size=0.40.4). (c), The PDFs of target luminance, given the surrounding luminances in (b) (The unit of luminance was cd/m2). The red curve corresponds to the condition shown in the left panel in (b) (Lu=205, Lv=45), and the blue curve to the condition shown in the right panel in (b) (Lu=45, Lv=205) (modified from Yang & Purves, 2004).

The context-mediated PDs of luminance in natural scenes predict equally well other

I showed that brightness percepts do not encode luminance as such, but rather the statistical relationship between the luminance in an area within a particular contextual light pattern and all possible occurrences of luminance in the context that have been experienced by humans in natural environments during evolution. The statistical basis for this aspect of visual perception is quite different from traditional approaches to rationalizing brightness. In the "relational approach" (Gichrist et al., 1999), an idea that evolved from the late 19th C.

**Figure 4.** Statistical explanation of the Wertheimer-Benary illusion.

brightness phenomena shown in Fig. 1 (Yang & Purves, 2004).

*2.6.1. The statistical nature of perception*

changed by such manipulations.

**2.6. Discussion**

(a), The usual presentation of White's illusion; boxed areas indicate the basic components of the stimulus, which when separated elicit about the same effect as the usual presentation. (b), The sampling configurations used to obtain the PDFs of target luminance (the red and blue rectangles), given a pattern of surrounds with luminance values Lu and Lv (size of the sampling configuration in this example was 0.6[H]0.3[V] and the unit of luminance was cd/m2). (c), The PDFs of the luminance of the targets in these contexts (red curve: Lu=145, Lv=105; blue curve: Lu=105, Lv=145). For the middle luminance values lying within the two crossover points (at ~105 and 145), the red curve is above the blue curve; as a result, the luminance configurations in (b) generate White's illusion (as indicated by the insets). For other luminance values of the target, the blue curve is above the red curve; as a result, the luminance configurations in (b*)* generate the inverted White's effect (modified from Yang & Purves, 2004).

**Figure 3.** Statistical explanation of White's illusion.

An aspect of White's illusion that has been particularly difficult to explain is the so-called "inverted White's effect": when the target luminance is either the lowest or the highest value in the stimulus, the effect is actually opposite the usual percept (see the middle and right panels in Fig. 1 (b)). The explanation for this further anomaly is also evident in Fig. 3 (c). When the target luminance is the lowest value in the presentation (see insets), the blue curve is above the red curve. As a result, a relatively dark target surrounded by more light area should now appear darker, as it does (see also the middle panel of Fig. 1 (b)). By the same token, when the target luminance is the highest value in the stimulus (see insets), the blue curve is also above the red curve. Accordingly, the relatively light target surrounded by more dark area should appear brighter, as it does (see also the right panel of Fig. 1 (b)). Thus the statistical structure of natural light patterns predicts not only White's illusion, but the inverted White's effect as well. Notice further that the two crossover points of the blue and red curves shift to the right when the contextual luminances increase, and to the left when they decrease; thus the inverted effect will be apparent, although altered in magnitude, for any luminance values of the surrounding areas.

#### **2.5. The Wertheimer-Benary illusion**

In the Wertheimer-Benary illusion (Fig. 1(c)), the equiluminant gray triangles appear differently bright, the triangle embedded in the arm of the cross looking slightly brighter than the triangle in the corner of the cross.

The explanation of the Wertheimer-Benary illusion provided by the statistical framework outlined above is shown in Fig. 4. By sampling the images of natural environments using configurations based on the components of the stimulus (Fig. 4 (b)), we obtained the PDFs of target luminance in these contexts. As shown in Fig. 4 (c), when the triangular patch is embedded in a dark bar with its base facing a lighter area, the percentile of the luminance of the triangular patch (red line) is higher than the percentile when the triangular patch abuts a dark corner with its base facing a similar light background (blue line). Accordingly, the same gray patch should appear brighter in the former context than in the latter, as is the case. The PDFs obtained after changing the triangles to rectangles, rotating the configurations in Fig. 4 (b) by 180, or reflecting the configurations along the diagonal of the cross (cf. middle and right panels in Fig. 1 (c)) were much the same as those shown in Fig. 4 (c). These several observations accord with the fact that the Wertheimer-Benary effect is little changed by such manipulations.

(a), The usual presentation of the Wertheimer-Benary stimulus. The separated components of the stimulus (boxed areas) elicit about the same effect as the usual presentation. (b), Configurations used to sample the database (size=0.40.4). (c), The PDFs of target luminance, given the surrounding luminances in (b) (The unit of luminance was cd/m2). The red curve corresponds to the condition shown in the left panel in (b) (Lu=205, Lv=45), and the blue curve to the condition shown in the right panel in (b) (Lu=45, Lv=205) (modified from Yang & Purves, 2004).

**Figure 4.** Statistical explanation of the Wertheimer-Benary illusion.

The context-mediated PDs of luminance in natural scenes predict equally well other brightness phenomena shown in Fig. 1 (Yang & Purves, 2004).

#### **2.6. Discussion**

206 Visual Cortex – Current Status and Perspectives

levels, in a vertical orientation, or with different aspect ratios.

generate the inverted White's effect (modified from Yang & Purves, 2004).

**Figure 3.** Statistical explanation of White's illusion.

any luminance values of the surrounding areas.

**2.5. The Wertheimer-Benary illusion**

than the triangle in the corner of the cross.

the sampling configurations is changed (e.g., from 1:2 to 1:5), the PDFs derived from the database are not much different. These further results are consistent with the observations that White's stimulus elicits the similar effect when presented at a wide range of overall luminance

(a), The usual presentation of White's illusion; boxed areas indicate the basic components of the stimulus, which when separated elicit about the same effect as the usual presentation. (b), The sampling configurations used to obtain the PDFs of target luminance (the red and blue rectangles), given a pattern of surrounds with luminance values Lu and Lv (size of the sampling configuration in this example was 0.6[H]0.3[V] and the unit of luminance was cd/m2). (c), The PDFs of the luminance of the targets in these contexts (red curve: Lu=145, Lv=105; blue curve: Lu=105, Lv=145). For the middle luminance values lying within the two crossover points (at ~105 and 145), the red curve is above the blue curve; as a result, the luminance configurations in (b) generate White's illusion (as indicated by the insets). For other luminance values of the target, the blue curve is above the red curve; as a result, the luminance configurations in (b*)*

An aspect of White's illusion that has been particularly difficult to explain is the so-called "inverted White's effect": when the target luminance is either the lowest or the highest value in the stimulus, the effect is actually opposite the usual percept (see the middle and right panels in Fig. 1 (b)). The explanation for this further anomaly is also evident in Fig. 3 (c). When the target luminance is the lowest value in the presentation (see insets), the blue curve is above the red curve. As a result, a relatively dark target surrounded by more light area should now appear darker, as it does (see also the middle panel of Fig. 1 (b)). By the same token, when the target luminance is the highest value in the stimulus (see insets), the blue curve is also above the red curve. Accordingly, the relatively light target surrounded by more dark area should appear brighter, as it does (see also the right panel of Fig. 1 (b)). Thus the statistical structure of natural light patterns predicts not only White's illusion, but the inverted White's effect as well. Notice further that the two crossover points of the blue and red curves shift to the right when the contextual luminances increase, and to the left when they decrease; thus the inverted effect will be apparent, although altered in magnitude, for

In the Wertheimer-Benary illusion (Fig. 1(c)), the equiluminant gray triangles appear differently bright, the triangle embedded in the arm of the cross looking slightly brighter

#### *2.6.1. The statistical nature of perception*

I showed that brightness percepts do not encode luminance as such, but rather the statistical relationship between the luminance in an area within a particular contextual light pattern and all possible occurrences of luminance in the context that have been experienced by humans in natural environments during evolution. The statistical basis for this aspect of visual perception is quite different from traditional approaches to rationalizing brightness. In the "relational approach" (Gichrist et al., 1999), an idea that evolved from the late 19th C. debate between Helmholtz, Hering, and others, brightness percepts are "recovered" by the visual system from explicitly coded luminance contrasts and gradients. Another idea is that brightness depends on intermediate-level visual processes that detect edges, gradients and junctions, which are then grouped into specific spatial layouts (Adelson, 2000; Anderson & Winawer, 2005). Finally, the brightness elicited by a given luminance has been also considered as being "re-synthesized" by processing at several levels of the visual system that is based on inferences about the possible arrangements of surfaces in 3D, their material properties and their illumination (Wishart et al., 1997).

Vision as a Fundamentally Statistical Machine 209

(1)

(2)

**3.2. Context-mediated PDs in natural scenes and visual saliency**

the ICs for the context first and then the other ICs of natural scenes.

ICA filters (i.e., , , *WW W s sc c* ) can be obtained as follows

van der Schaaf, 1998).

alone.

A visual feature is a random variable and co-occurs at certain probabilities with other visual features in natural scenes. We call these the context-mediated PDs in natural scenes. Here, a context refers to the natural scene patch that co-occurs with a visual target in question in space and/or time domains. We proposed to represent the context-mediated PDs in natural scenes using independent components (**ICs**) of natural scenes. There are two reasons for this. First, it has been argued extensively that the early visual cortex represents incoming stimuli in an efficient manner (Simoncelli & Olshausen, 2001). Second, the filters of the ICs of natural scenes are very much like the receptive fields of simple cells in V1 (van Hateren &

To model the context-mediated PDs in static natural scenes, we used a center-surround configuration in which the scene patch within the circular center serves as the target and the scene patch in the annular surround as the context (Xu et al., 2010). We sampled a large number of scene patches from the McGill calibrated color image database of natural scenes (Olmos & Kingdom, 2004). Thus, each sample is a pair of a patch in center ( *Xc* ) and a patch in the surrounding area ( *Xs* ) (Fig. 5 (a)). We developed a model of natural scenes in this configuration (Eq. (1)). In Eq. (1), *As* , *Ac* , and *Asc* are ICs. This model allows us to calculate

> 0 *ss s c sc c sc XA U X A AU*

0 *ss s sc sc c c UW X U W WX*

Therefore, we obtained three sets of ICs. First, the columns of *As* are the ICs for *Xs* . Second, the columns of *Asc* are the ICs for *Xc* that are paired with the ICs for *Xs* . Finally, the

Fig. 5 (b) shows paired chromatic ICs for *Xc* and *Xs* . Fig. 5 (c) shows paired achromatic ICs for *Xc* and *Xs* . The chromatic ICs for the surround have red-green (L-M) or blue-yellow [S- (LM)] opponency. The chromatic paired ICs for the center are extensions of the ICs for the surround. Fig. 5 (d) shows the ICs for *Xc* , including chromatic and achromatic ICs, that are not paired with any ICs for *Xs* . Fig. 5 (e) shows examples of the ICs for the center computed

To obtain the context-mediated PDs in dynamic natural scenes, we used sequences of image patches in which the current frame severed as the target and the three preceding frames as the context. We sampled a large number of sequences of image patches (~ 490,000) from a video database (Itti & Baldi, 2009) and performed the ICA according to Eq. (1). Fig. 6 (a) shows the paired chromatic spatiotemporal ICs. Fig. 6 (b) shows the paired achromatic

columns of *Ac* are the ICs for *Xc* that are not paired with any ICs for *Xs* .

The common deficiency of these several ways of thinking about brightness is their failure to relate the statistics of light patterns experienced in the course of evolution to what the corresponding brightness percepts need to signify (namely, the relationship of a particular occurrence of luminance to all possible occurrences of luminance in a given context). Since light patterns on the retina are the only information the visual system receives, basing brightness percepts on the statistics of natural light patterns allows visual animals to deal optimally with all possible natural occurrences of luminance, employing the full range of perceivable brightness to represent the physical world.

## *2.6.2. Neural instantiation of context-mediated PDs of luminance in natural scenes*

What sort of neural mechanisms, then, could incorporate these statistics of natural light patterns and relate them to brightness percepts? Although the answer is not known, the present results suggest that the circuitry at all levels of the visual system instantiates the statistical structures of light patterns in natural environments. In this conception, the centersurround organization of the receptive fields of retinal ganglion cells provides the initial basis for representing the necessary statistics. A further speculation would be that neural circuitry at the level of visual cortex is organized to instantiate the statistics of luminance patterns with arbitrary target and context shapes and sizes. As a result, the neuronal response at each location would signify the percentile of the target luminance in the PDF pertinent to a given context.

## **3. Visual saliency emerging from context-mediated PDs in natural scenes**

## **3.1. Introduction**

In this section, I present a simple model of the context-mediated PDs in natural scenes and derive a measure of visual saliency from these PDs. Visual saliency is the perceptual quality that makes some items in visual scenes stand out from their immediate contexts (Itti & Koch, 2001). Visual saliency plays important roles in natural vision in that saliency can direct eye movements and facilitate object detection and scene understanding. We developed a model of the context-mediated PDs in natural scenes using a modified algorithm for independent component analysis (**ICA**) (Hyvarinen, 1999) and demonstrated that visual saliency based on the context-mediated PDs in natural scenes is a good predictor of human gaze in freeviewing both static and dynamic natural scenes (Xu et al., 2010).

#### **3.2. Context-mediated PDs in natural scenes and visual saliency**

208 Visual Cortex – Current Status and Perspectives

properties and their illumination (Wishart et al., 1997).

perceivable brightness to represent the physical world.

pertinent to a given context.

**3.1. Introduction**

debate between Helmholtz, Hering, and others, brightness percepts are "recovered" by the visual system from explicitly coded luminance contrasts and gradients. Another idea is that brightness depends on intermediate-level visual processes that detect edges, gradients and junctions, which are then grouped into specific spatial layouts (Adelson, 2000; Anderson & Winawer, 2005). Finally, the brightness elicited by a given luminance has been also considered as being "re-synthesized" by processing at several levels of the visual system that is based on inferences about the possible arrangements of surfaces in 3D, their material

The common deficiency of these several ways of thinking about brightness is their failure to relate the statistics of light patterns experienced in the course of evolution to what the corresponding brightness percepts need to signify (namely, the relationship of a particular occurrence of luminance to all possible occurrences of luminance in a given context). Since light patterns on the retina are the only information the visual system receives, basing brightness percepts on the statistics of natural light patterns allows visual animals to deal optimally with all possible natural occurrences of luminance, employing the full range of

*2.6.2. Neural instantiation of context-mediated PDs of luminance in natural scenes*

What sort of neural mechanisms, then, could incorporate these statistics of natural light patterns and relate them to brightness percepts? Although the answer is not known, the present results suggest that the circuitry at all levels of the visual system instantiates the statistical structures of light patterns in natural environments. In this conception, the centersurround organization of the receptive fields of retinal ganglion cells provides the initial basis for representing the necessary statistics. A further speculation would be that neural circuitry at the level of visual cortex is organized to instantiate the statistics of luminance patterns with arbitrary target and context shapes and sizes. As a result, the neuronal response at each location would signify the percentile of the target luminance in the PDF

**3. Visual saliency emerging from context-mediated PDs in natural scenes**

In this section, I present a simple model of the context-mediated PDs in natural scenes and derive a measure of visual saliency from these PDs. Visual saliency is the perceptual quality that makes some items in visual scenes stand out from their immediate contexts (Itti & Koch, 2001). Visual saliency plays important roles in natural vision in that saliency can direct eye movements and facilitate object detection and scene understanding. We developed a model of the context-mediated PDs in natural scenes using a modified algorithm for independent component analysis (**ICA**) (Hyvarinen, 1999) and demonstrated that visual saliency based on the context-mediated PDs in natural scenes is a good predictor of human gaze in free-

viewing both static and dynamic natural scenes (Xu et al., 2010).

A visual feature is a random variable and co-occurs at certain probabilities with other visual features in natural scenes. We call these the context-mediated PDs in natural scenes. Here, a context refers to the natural scene patch that co-occurs with a visual target in question in space and/or time domains. We proposed to represent the context-mediated PDs in natural scenes using independent components (**ICs**) of natural scenes. There are two reasons for this. First, it has been argued extensively that the early visual cortex represents incoming stimuli in an efficient manner (Simoncelli & Olshausen, 2001). Second, the filters of the ICs of natural scenes are very much like the receptive fields of simple cells in V1 (van Hateren & van der Schaaf, 1998).

To model the context-mediated PDs in static natural scenes, we used a center-surround configuration in which the scene patch within the circular center serves as the target and the scene patch in the annular surround as the context (Xu et al., 2010). We sampled a large number of scene patches from the McGill calibrated color image database of natural scenes (Olmos & Kingdom, 2004). Thus, each sample is a pair of a patch in center ( *Xc* ) and a patch in the surrounding area ( *Xs* ) (Fig. 5 (a)). We developed a model of natural scenes in this configuration (Eq. (1)). In Eq. (1), *As* , *Ac* , and *Asc* are ICs. This model allows us to calculate the ICs for the context first and then the other ICs of natural scenes.

$$
\begin{bmatrix} X\_s \\ X\_c \end{bmatrix} = \begin{bmatrix} A\_s & 0 \\ A\_{sc} & A\_c \end{bmatrix} \begin{bmatrix} \mathcal{U}\_s \\ \mathcal{U}\_{sc} \end{bmatrix} \tag{1}
$$

ICA filters (i.e., , , *WW W s sc c* ) can be obtained as follows

$$
\begin{bmatrix} \boldsymbol{\mathcal{U}}\_{s} \\ \boldsymbol{\mathcal{U}}\_{sc} \end{bmatrix} = \begin{bmatrix} \boldsymbol{\mathcal{W}}\_{s} & \boldsymbol{0} \\ \boldsymbol{\mathcal{W}}\_{sc} & \boldsymbol{\mathcal{W}}\_{c} \end{bmatrix} \begin{bmatrix} \boldsymbol{X}\_{s} \\ \boldsymbol{X}\_{c} \end{bmatrix} \tag{2}
$$

Therefore, we obtained three sets of ICs. First, the columns of *As* are the ICs for *Xs* . Second, the columns of *Asc* are the ICs for *Xc* that are paired with the ICs for *Xs* . Finally, the columns of *Ac* are the ICs for *Xc* that are not paired with any ICs for *Xs* .

Fig. 5 (b) shows paired chromatic ICs for *Xc* and *Xs* . Fig. 5 (c) shows paired achromatic ICs for *Xc* and *Xs* . The chromatic ICs for the surround have red-green (L-M) or blue-yellow [S- (LM)] opponency. The chromatic paired ICs for the center are extensions of the ICs for the surround. Fig. 5 (d) shows the ICs for *Xc* , including chromatic and achromatic ICs, that are not paired with any ICs for *Xs* . Fig. 5 (e) shows examples of the ICs for the center computed alone.

To obtain the context-mediated PDs in dynamic natural scenes, we used sequences of image patches in which the current frame severed as the target and the three preceding frames as the context. We sampled a large number of sequences of image patches (~ 490,000) from a video database (Itti & Baldi, 2009) and performed the ICA according to Eq. (1). Fig. 6 (a) shows the paired chromatic spatiotemporal ICs. Fig. 6 (b) shows the paired achromatic spatiotemporal ICs. Fig. 6 (c) shows the unpaired ICs for the current frame, which are oriented bars and have red-green or blue-yellow opponency.

Vision as a Fundamentally Statistical Machine 211

(a), Selected 28 red/green or blue/yellow paired ICs. (b), Selected 78 paired bright/dark ICs. (c), Examples of unpaired

max ln ( ) ln ( ) *i i sc sc*

*i i*

where max(|) *c s P XX* is the maximum probability of a target, *Xc* , that co-occurs with a context, *Xs* , in natural scenes. Thus, if the probability of the occurrence of a target is low relative to that of the most likely occurrence in the context in natural scenes, the target is

max ln ( | ) ln ( | ) *cs cs S P X X PX X* (4)

*S P u Pu* (5)