∎

11institutetext: I. Tyukin 22institutetext: University of Leicester,
Department of Mathematics, University Road, LE1 7RH, United Kingdom
Tel.: +44-116-2525106
22email<EMAIL_ADDRESS>
Saint-Petersburg State Electrotechnical University, Saint-Petersburg, Prof.
Popova str. 5, Russia 33institutetext: Alexander N. Gorban 44institutetext:
University of Leicester, Department of Mathematics, University Road, LE1 7RH,
United Kingdom 55institutetext: Carlos Calvo 66institutetext: Instituto de
Matemática Interdisciplinar, Faculty of Mathematics, Universidad Complutense
de Madrid, Avda Complutense s/n, 28040 Madrid, Spain 77institutetext: Julia
Makarova 88institutetext: Department of Translational Neuroscience, Cajal
Institute – CSIC, Madrid, Spain;
Lobachevsky State University of Nizhny Novgorod, Gagarin Ave. 23, 603950
Nizhny Novgorod, Russia 99institutetext: Valeri A. Makarov 1010institutetext:
Instituto de Matemática Interdisciplinar, Faculty of Mathematics, Universidad
Complutense de Madrid, Avda Complutense s/n, 28040 Madrid, Spain;
Lobachevsky State University of Nizhny Novgorod, Gagarin Ave. 23, 603950
Nizhny Novgorod, Russia
1010email<EMAIL_ADDRESS>
# High-dimensional brain

A tool for encoding and rapid learning of memories by single neurons

Ivan Tyukin  Alexander N. Gorban  Carlos Calvo  Julia Makarova  Valeri A.
Makarov

(Received: date / Accepted: date)

###### Abstract

Codifying memories is one of the fundamental problems of modern Neuroscience.
The functional mechanisms behind this phenomenon remain largely unknown.
Experimental evidence suggests that some of the memory functions are performed
by stratified brain structures such as, e.g., the hippocampus. In this
particular case, single neurons in the CA1 region receive a highly
multidimensional input from the CA3 area, which is a hub for information
processing. We thus assess the implication of the abundance of neuronal
signalling routes converging onto single cells on the information processing.
We show that single neurons can selectively detect and learn arbitrary
information items, given that they operate in high dimensions. The argument is
based on Stochastic Separation Theorems and the concentration of measure
phenomena. We demonstrate that a simple enough functional neuronal model is
capable of explaining: i) the extreme selectivity of single neurons to the
information content, ii) simultaneous separation of several uncorrelated
stimuli or informational items from a large set, and iii) dynamic learning of
new items by associating them with already “known” ones. These results
constitute a basis for organization of complex memories in ensembles of single
neurons. Moreover, they show that no a priori assumptions on the structural
organization of neuronal ensembles are necessary for explaining basic concepts
of static and dynamic memories.

###### Keywords:

Neural memories Single-neuron learning Perceptron Stochastic Separation
Theorems

††journal: Bull Math Biol

##  1 Introduction

The human brain is arguably amongst the most sophisticated and enigmatic
nature creations. Over millions of years it has evolved to amass billions of
neurons, featuring on average $86\times 10^{9}$ cells Herculano2012 . This
remarkable figure is several orders of magnitude higher than that of the most
mammals and several times larger than in primates Herculano2011 . Whilst
measuring roughly $2\%$ of the body mass, the human brain consumes about
$20\%$ of the total energy Clark1999 .

The significant metabolic cost associated with a larger brain in humans, as
opposed to mere body size - a path that great apes might have evolved
Herculano2011 , must be justified by evolutionary advantages. Some of the
benefits may be related to the development of a remarkably important social
life in humans. This, in particular, requires extensive abilities in formation
of complex memories. Indirectly this hypothesis is supported by the
significant difference among species in the number of neurons in the cortex
Herculano2009  and the hippocampus Andersen2007 . For example, in the CA1 area
of the hippocampus there are $0.39\times 10^{6}$ pyramidal neurons in rats,
$1.3\times 10^{6}$ in monkeys, and $14\times 10^{6}$ in humans.

Evolutionary implications in relation to cognitive functions have been widely
discussed in the literature (see, e.g., Platek2007 ; Sherwood2012 ; Sousa2017
). Recently, it has been shown that in humans new memories can be learnt very
rapidly by supposedly individual neurons from a limited number of experiences
Quiroga:2015 . Moreover, some neurons can exhibit remarkable selectivity to
complex stimuli, the evidence that has led to debates around the existence of
the so-called “grand mother” and “concept” cells Quiroga:2005 ; Quiroga:2009 ;
Quiroga:2012 , and their role as elements of a declarative memory. These
findings suggest that not only the brain can learn rapidly but also it can
respond selectively to “rare” individual stimuli. Moreover, experimental
evidence indicates that such a cognitive functionality can be delivered by
single neurons Quiroga:2015 ; Quiroga:2005 ; Quiroga:2009 . The fundamental
questions, hence, are: How is this possible? and What could be the underlying
functional mechanisms?

Recent theoretical advances achieved within the Blue Brain Project show that
the brain can operate in many dimensions Markram2017 . It is claimed that the
brain has structures operating in up to eleven dimensions. Groups of neurons
can form the so called cliques, i.e., networks of specially interconnected
neurons that generate precise representations of geometric objects. Then the
dimension grows with the number of neurons in the clique. Multidimensional
representation of spatiotemporal information in the brain is also implied in
the concept of generalized cognitive maps (see, e.g., BC2015 ; ACS2016 ;
LimbMov ). Within this theory, spatiotemporal relations between objects in the
environment are encoded as static (cognitive) maps and represented as elements
of an $n$-dimensional space ($n\gg 1$). The cognitive maps as information
items can be learnt, classified, and retrieved on demand IEEE2013 . However,
the questions concerning how the brain or individual neurons can distinguish
among a huge number of different maps and select an appropriate one remain
unknown.

In this work we propose that brain areas with a predominant laminar topology
and abundant signalling routes simultaneously converging on individual cells
(e.g., the hippocampus) are propitious for a high-dimensional processing and
learning of complex information items. We show that a canonical neuronal
model, the perceptron Rosenblatt1962 , in combination with a Hebbian-type of
learning may provide answers to the above mentioned fundamental questions. In
particular, starting from stochastic separation theorems GorbanTyukin:NN:2017
; GorbanTyukin:RSTA:2017  we demonstrate that individual neurons gathering
multidimensional stimuli through a sufficiently large number of synaptic
inputs can exhibit extreme selectivity either to individual information items
or to groups of items. Moreover, neurons are capable of associating and
learning uncorrelated information items. Thus, a large number of signalling
routes simultaneously converging on a large number of single cells, as it is
widely observed in laminar brain structures, translates into a natural
environment for rapid formation and maintenance of extensive memories. This is
vital for social life, and hence may constitute a significant evolutionary
advantage, albeit, at the cost of high metabolic expenditure.

##  2 Fundamental problems of encoding memories

Different brain structures, such as, e.g., the hippocampus, have a pronounced
laminar organization. For example the CA1 region of the hippocampus is
constituted by a palisade of morphologically similar pyramidal cells oriented
with their main axis in parallel and forming a monolayer (Fig. 1A). The major
excitatory input to these neurons comes through Schaffer collaterals from the
CA3 region Amaral1989 ; Ishizuka1990 ; Wittner2007 , which is a hub routing
information among many brain structures. Each CA3 pyramidal neuron sends an
axon that bifurcates and leaves multiple collaterals in the CA1 with dominant
parallel orientation (Fig. 1B). This topology allows multiple parallel axons
conveying multidimensional “spatial” information from one area (CA3)
simultaneously leave synaptic contacts on multiple neurons in another area
(CA1). Thus, we have simultaneous convergence and divergence of the
information content (Fig. 1B, right).

Figure 1: General principles of encoding memories by single neurons in laminar
structures. A) Laminar organization of the CA3 and CA1 areas in the
hippocampus facilitates multiple parallel synaptic contacts between neurons in
these areas by means of Schaffer collaterals. B) Axons from CA3 pyramidal
neurons bifurcate and pass through the CA1 area in parallel (left panel)
giving rise to the convergence-divergence of the information content (right
panel). Multiple CA1 neurons receive multiple synaptic contacts from CA3
neurons. C) Schematic representation of three memory encoding schemes. 1)
Selectivity. A neuron (shown in yellow) receives inputs from multiple
presynaptic cells that code different information items. It detects (responds
to) only one stimulus (purple trace), whereas rejecting the others. 2)
Clustering. Similar to 1, but now a neuron (shown in pink) detects a group of
stimuli (purple and blue traces) and ignores the others. 3) Acquiring
memories. A neuron (shown in green) learns dynamically a new memory item (blue
trace) by associating it with a know one (purple trace).

Experimental findings show that multiple CA1 pyramidal cells distributed in
the rostro-caudal direction are activated near-synchronously by assemblies of
simultaneously firing CA3 pyramidal cells Ishizuka1990 ; Li1994 ; Benito2014 .
Thus, an ensemble of single neurons in the CA1 can receive simultaneously the
same synaptic input (Fig. 1B, left). Since these neurons have different
topology and functional connectivity Rnnerty1983 , their response to the same
input can be different. Moreover, experimental in-vivo results show that long
term potentiation can significantly increase the spike transfer rate in the
CA3-CA1 pathway Fernandez2012 . This suggests that the efficiency of
individual synaptic contacts can be increased selectively.

In this work we will follow conventional and rather general functional
representation of signalling in the neuronal pathways. We assume that upon
receiving an input, a neuron can either generate a response or remain silent.
Forms of the neuronal responses as well as the definitions of synaptic inputs
vary from one model to another. Therefore, here we adopt a rather general
functional approach. Under a stimulus we understand a number of excitations
simultaneously (or within a short time window) arriving to a neuron through
several axones and thus transmitting some “spatially coded” information items
Benito2016 . If a neuron responds to a stimulus (e.g., generates output spikes
or increases its firing rate), we then say that the neuron detects the
informational content of the given stimulus.

We follow the standard machine learning assumptions Vapnik:2000 ,
cucker2002mathematical . The stimuli are generated in accordance with some
distribution or a set of distributions (“Outer World Models”). All stimuli
that a neuron may receive are samples from this distribution. The sampling
itself may be a complicated process, and for simplicity we assume that all
samples are identically and independently distributed (i.i.d.). Once a sample
is generated, a stimuli sub-sample is indepently selected for testing
purposes. If more than one neuron is considered, we will assume that a rule
(or a set of rules) is in place that determines how a neuron is selected from
the set. The rules can be both deterministic and randomized. In the latter
case we will specify this process.

Let us now pose the following fundamental questions related to the information
encoding and formation of memories by single neurons and their ensembles in
laminated brain structures:

  1. 1.

Selectivity: Detection of one stimulus from a set (Fig. 1C.1). Pick an
arbitrary stimulus from a reasonably large set such that a single neuron from
a neuronal ensemble detects this stimulus. Then what is the probability that
this neuron is stimulus-specific, i.e., it rejects all the other stimuli from
the set?

  2. 2.

Clustering: Detection of a group of stimuli from a set (Fig. 1C.2). Within a
set of stimuli we select a smaller subset, i.e., a group of stimuli. Then what
is the probability that a neuron detecting all stimuli from this subset stays
silent for all remaining stimuli in the set?

  3. 3.

Acquiring memories: Learning new stimulus by associating it with one already
known (Fig. 1C.3). Let us consider two different stimuli $\boldsymbol{s}_{1}$
and $\boldsymbol{s}_{2}$ such that for $t\leq t_{0}$ they do not overlap in
time and a neuron detects $\boldsymbol{s}_{1}$, but not $\boldsymbol{s}_{2}$.
In the next interval $(t_{0},t_{1}]$, $t_{1}>t_{0}$ the stimuli start to
overlap in time (i.e., they stimulate the neuron together). For $t>t_{1}$ the
neuron receives only stimulus $\boldsymbol{s}_{2}$. Then what is the
probability that for some $t_{2}\geq t_{1}$ the neuron detects
$\boldsymbol{s}_{2}$?

These questions are in the core of a broad range of puzzling phenomena
reported in Quiroga:2015 ; Quiroga:2005 ; Quiroga:2009 . In what follows we
will show that, remarkably, these three non-trivial fundamental questions can
be answered within a simple classical modeling framework, whereby a neuron is
represented by a mere perceptron equipped with a Hebbian-type of learning.

##  3 Formal statement of the problem

In this section we specify the information content to be processed by neurons
and define a mathematical model of a generic neuron equipped with synaptic
plasticity. Before going any further’ let us first introduce notational
agreements used throughout the text. Given two vectors
$\boldsymbol{x},\boldsymbol{y}\in\mathbb{R}^{n}$, their inner product
$\langle\boldsymbol{x},\boldsymbol{y}\rangle$ is:
$\langle\boldsymbol{x},\boldsymbol{y}\rangle=\sum_{i=1}^{n}x_{i}y_{i}$. If
$\boldsymbol{x}\in\mathbb{R}^{n}$ then $\|\boldsymbol{x}\|$ stands for the
usual Euclidean norm of $\boldsymbol{x}$:
$\|\boldsymbol{x}\|=\langle\boldsymbol{x},\boldsymbol{x}\rangle^{1/2}$. By
$B_{n}(1)=\\{\boldsymbol{x}\in\mathbb{R}^{n}|\ \ \|\boldsymbol{x}\|\leq 1\\}$
we denote a unit $n$-ball centered at the origin; $\mathcal{V}(\Xi)$ is the
Lebesgue volume of $\Xi\subset\mathbb{R}^{n}$, and $|\mathcal{M}|$ is the
cardinality of a finite set $\mathcal{M}$. Symbol $\mathcal{C}(\mathcal{D})$,
$\mathcal{D}\subseteq\mathbb{R}^{m}$ stands for the space of continuous real-
valued functions on $\mathcal{D}$.

###  3.1 Information content and classes of stimuli

We assume that a neuron receives and processes a large but finite set of
different stimuli codifying different information items:

$\mathcal{S}=\\{\boldsymbol{s}_{i}\\}.$ (1)

Figure 2: Codification of high-dimensional information by a neuron. Each of
$M$ stimuli comprises of the “spatial” information,
$\boldsymbol{x}_{i}\in\mathbb{R}^{n}$, (e.g., $M$ images) conducted through
$n$ axons (in yellow) and the temporal part, $c(t-{\tau_{i,j}})$, reflecting
the times of stimuli presentation. A neuron (in blue) receives the stimuli and
generates responses determined by some transfer function $f$.

Figure 2 illustrates schematically the information flow. Each individual
stimulus $i$ is modeled by a function
$\boldsymbol{s}:\mathbb{R}\times\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$:

$\boldsymbol{s}(t,\boldsymbol{x}_{i})=\boldsymbol{x}_{i}\sum_{j}c(t-{\tau_{i,j}}),$
(2)

where $\boldsymbol{x}_{i}\in\mathbb{R}^{n}\setminus\\{0\\}$ is the stimulus
content codifying the information to be transmitted over $n$ individual
“axons”. An example of an information item could be an $l\times k$ image (see
Fig. 2). In this case the dimension of each information item is $n=l\times k$.

In Eq. (2) the function $c(\cdot)$ defines the stimulus context, i.e., the
time window when the stimulus arrives to the neuron. For the sake of
simplicity we use a rectangular window:

$c(t)=\left\\{\begin{array}[]{ll}1,&\mbox{ if }t\in[0,\Delta T]\\\ 0,&\mbox{
otherwise},\end{array}\right.$ (3)

where $\Delta T>0$ is the window length. The time instants of the stimulus
presentations, $\tau_{i,j}$, are ordered and satisfy:

$\tau_{i,j+1}>\tau_{i,j}+\Delta T,\ \ \forall j.$ (4)

Different stimuli arriving to the neuron are added linearly on the neuronal
membrane. Thus, the overall neuronal input $\boldsymbol{S}$ can be written as:

$\boldsymbol{S}(t)=\sum_{i,j}\boldsymbol{x}_{i}c(t-\tau_{i,j}).$ (5)

We assume that the information content of stimuli (5) and (2), i.e., vectors
$\boldsymbol{x}_{i}$ are drawn i.i.d. from some distribution. For convenience,
we partition all information items into two sets:

$\mathcal{M}=\\{\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{M}\\},\ \
\mathcal{Y}=\\{\boldsymbol{x}_{M+1},\dots,\boldsymbol{x}_{M+m}\\},$ (6)

where $M$ is large but finite and $m\geq 1$ is in general smaller than $M$.
The set $\mathcal{M}$ contains a background content for a given neuron,
whereas the set $\mathcal{Y}$ models the informational content relevant to the
task at hand. In other words, to accomplish a static memory task the neuron
should be able to detect all elements from $\mathcal{Y}$ and to reject all
elements from $\mathcal{M}$.

The sets $\mathcal{M}$ and $\mathcal{Y}$ give rise to the corresponding
subsets of stimuli:

$\begin{split}\mathcal{S}(\mathcal{M})=\\{\boldsymbol{s}_{i}\in\mathcal{S}\ |\
\boldsymbol{s}_{i}(\cdot)=\boldsymbol{s}(\cdot,\boldsymbol{x}_{i}),\
\boldsymbol{x}_{i}\in\mathcal{M}\\},\\\
\mathcal{S}(\mathcal{Y})=\\{\boldsymbol{s}_{i}\in\mathcal{S}\ |\
\boldsymbol{s}_{i}(\cdot)=\boldsymbol{s}(\cdot,\boldsymbol{x}_{i}),\
\boldsymbol{x}_{i}\in\mathcal{Y}\\}.\end{split}$ (7)

###  3.2 Neuronal model

To stay within functional description of the information processing let us
consider the most basic class of model neurons, a perceptron Rosenblatt1962 .
A single neuron receives a stimulus $\boldsymbol{s}(t,\boldsymbol{x})$ through
$n$ synaptic inputs (Fig. 2) and its membrane potential, $y\in\mathbb{R}$, is
given by

$y(\boldsymbol{s},\boldsymbol{w})=\langle\boldsymbol{w},\boldsymbol{s}\rangle,$
(8)

where $\boldsymbol{w}\in\mathbb{R}^{n}$ is a vector of the synaptic weights.
The neuron generates a response, $v\in\mathbb{R}$, according to:

$v(\boldsymbol{s},\boldsymbol{w},\theta)=f(y(\boldsymbol{s},\boldsymbol{w})-\theta),$
(9)

where $\theta\in\mathbb{R}$ is the “firing” threshold and
$f:\mathbb{R}\rightarrow\mathbb{R}$ is the transfer function (Fig. 2):
$f\in\mathcal{C}(\mathbb{R})$, $f$ is locally Lipschitz, $f(u)=0$ for
$u\in(-\infty,0]$, and $f(u)>0$ for $u\in(0,\infty)$.

Model (8), (9) captures the summation of postsynaptic potentials and the
threshold nature of the neuronal activation but disregards the specific
dynamics accounted for in other more advanced models. Nevertheless, as we will
show in Sect. 4, this phenomenological model is already sufficient to explain
the fundamental properties of information processing discussed in Sect. 2.

###  3.3 Synaptic plasticity

In addition to the basic neuronal response mechanism (Sect. 3.2), we also
model the synaptic plasticity. The description adopted here relies on the
neuronal firing rate and Hebbian learning. Such a learning rule implies that
the dynamics of $\boldsymbol{w}$ should depend on the product of the input
signal, $\boldsymbol{s}$, and the neuronal output, $v$. We thus arrive to a
modified classical Oja rule Oja1982 :

$\begin{split}&\dot{\boldsymbol{w}}=\alpha
v(\boldsymbol{s},\boldsymbol{w},\theta)y(\boldsymbol{s},\boldsymbol{w})\left(\boldsymbol{s}-\boldsymbol{w}y(\boldsymbol{s},\boldsymbol{w})\right),\\\
&\boldsymbol{w}(t_{0})=\boldsymbol{w}_{0}\in\mathbb{R}^{n},\
\boldsymbol{w}_{0}\neq 0,\end{split}$ (10)

where $\alpha>0$ defines the relaxation time. The multiplicative term $v$ in
(10) ensures that plastic changes of $\boldsymbol{w}$ occur only when an input
stimulus evokes a non-zero neuronal response. The fact that
$\boldsymbol{w}_{0}\neq 0$ reflects the assumption that synaptic connections
have already been established, albeit their efficacy could be subjected to
plastic changes. In addition to capturing general principle of the classical
Hebbian rule, model (10) guarantees that synaptic weights $\boldsymbol{w}$ are
bounded in forward time (see Appendix A) and hence conforms with physiological
plausibility.

##  4 Formation of memories in high-dimensions

In Sect. 2 we formulated three fundamental problems of organization of
memories in laminar brain structures. Let us now show how they can be treated
given that pyramidal neurons operate in high dimensions.

To formalize the analysis let $\mathcal{U}$ be a subset of the stimulus set
$\mathcal{S}$. A neuron (8), (9) parameterized by $(\boldsymbol{w},\theta)$
partitions the set $\mathcal{U}$ into the following subsets:

$\begin{split}\mathrm{Activated}(\mathcal{U},(\boldsymbol{w},\theta))=&\\{\boldsymbol{s}_{i}\in\mathcal{U}\
|\ \ \exists\,t{\geq}t_{0}:\
v(\boldsymbol{s}_{i}(t),\boldsymbol{w},\theta)>0\\},\\\
\mathrm{Silent}(\mathcal{U},(\boldsymbol{w},\theta))=&\\{\boldsymbol{s}_{i}\in\mathcal{U}\
|\ \ v(\boldsymbol{s}_{i}(t),\boldsymbol{w},\theta)=0\ \ \forall\ t\geq
t_{0}\\}.\end{split}$ (11)

The first set corresponds to the stimuli detected by the neuron, while the
second one collects background stimuli.

###  4.1 Extreme selectivity of a single neuron to single stimuli

Consider the case when the set $\mathcal{Y}$ in (6) contains only one element,
i.e. $|\mathcal{Y}|=1$, $\mathcal{Y}=\\{\boldsymbol{x}_{M+1}\\}$, whereas the
set $\mathcal{M}$ is allowed to be sufficiently large ($|\mathcal{M}|=M\gg
1$). Let us also assume that the stimuli with different information content,
$\boldsymbol{s}(\cdot,\boldsymbol{x}_{i})$, do not overlap in time, i.e., we
present them to a neuron one by one.

For a given non-zero $\boldsymbol{x}_{M+1}\in\mathcal{Y}$ and stimulus
$\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})$ such that it is not identically
zero for $t\geq t_{0}$ we can always construct a neuron which would generate a
non-zero response to the stimulus $\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})$
at some $t\geq t_{0}$. In other words,
$\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})\in\mathrm{Activated}(\mathcal{S}(\mathcal{Y}),(\boldsymbol{w},\theta))$.
Mathematically such a neuron can be defined as follows. Let

$\boldsymbol{w}^{\ast}=\frac{\boldsymbol{x}_{M+1}}{\|\boldsymbol{x}_{M+1}\|}.$
(12)

Then the space from which the synaptic weights are chosen can be represented
as a direct sum of the one-dimensional linear subspace
$L^{\|}(\boldsymbol{w}^{\ast})$ spanned by $\boldsymbol{w}^{\ast}$ and an
$(n-1)$-dimensional subspace $L^{\bot}(\boldsymbol{w}^{\ast})$ of
$\mathbb{R}^{n}$ that is orthogonal to $\boldsymbol{w}^{\ast}$. In this
representation, if a neuron with the synaptic weight $\boldsymbol{w}$
generates a non-zero response to $\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})$,
then the coupling weight
$w^{\ast}=\langle\boldsymbol{w},\boldsymbol{w}^{\ast}\rangle$ should satisfy
the following condition (Fig. 3, green area):

${w^{\ast}}>\frac{\theta}{\|\boldsymbol{x}_{M+1}\|}.$

Indeed, such a choice is equivalent to

$v(\boldsymbol{x}_{M+1},\boldsymbol{w},\theta)=f({w^{\ast}}\|\boldsymbol{x}_{M+1}\|-\theta)>0,$

which in turn implies that
$v(\boldsymbol{s}(t,\boldsymbol{x}_{M+1}),\boldsymbol{w}^{\ast},\theta)>0$ at
some $t$ and vice-versa.

Figure 3: Selection of neuronal parameters
$\bar{\theta}=\theta/\|\boldsymbol{x}_{M+1}\|$ and ${w^{\ast}}$, such that the
neuron responds to the relevant information $\boldsymbol{x}_{M+1}$. Neurons
corresponding to points within the green area detect the stimulus
$\boldsymbol{x}_{M+1}$. Brown areas show projections of hypercylinders defined
in Theorem 4.1 for $D_{1}=0.3$, $D_{2}$ = 0.1, $D_{3}=0.03$ and
$\|\boldsymbol{x}_{M+1}\|=0.6$.

Once a neuron that detects relevant information item, i.e.
$\boldsymbol{x}_{M+1}$, is specified we can proceed with assessing its
selectivity properties.

######  Definition 1 (Neuronal Selectivity)

We say that a neuron is selective to the information content $\mathcal{Y}$ iff
it detects the relevant stimuli from the set $\mathcal{S}(\mathcal{Y})$ and
ignores all the others from the set $\mathcal{S}(\mathcal{M})$.

The notion of selectivity, as stated in Definition 1, could be relaxed to
account for partial detection and rejection of information content from
$\mathcal{Y}$ and $\mathcal{M}$, respectively. This naturally gives rise to
various levels of neuronal selectivity determined, for instance, by the
proportion of elements from $\mathcal{M}$ that correspond to stimuli that have
been rejected. As we will see below, different admissible pairs
$(\boldsymbol{w},\theta)$ (Fig. 3) produce different selectivity levels. The
closer to the bisector, the higher the selectivity. One can pick an arbitrary
firing threshold $\theta\geq 0$ and select the synaptic efficiency at
$t=t_{0}$ as:

$\boldsymbol{w}(t_{0})=\frac{\theta+\epsilon}{\|\boldsymbol{x}_{M+1}\|}\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot},\
\ \ \epsilon>0,\ \boldsymbol{w}^{\bot}\in L^{\bot}.$ (13)

It can be shown (see Appendix A) that if the stimulus
$\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})$ is persistent over time and
$\boldsymbol{w}(t_{0})$ satisfies (13) then synaptic efficiency
$\boldsymbol{w}(t,\boldsymbol{w}_{0})$ converges asymptotically (as
$t\rightarrow\infty$) to:

$\boldsymbol{w}_{\infty}=\left\\{\begin{array}[]{ll}\boldsymbol{w}^{\ast},&\
\mbox{ if }\theta<\|\boldsymbol{x}_{M+1}\|\\\
\frac{\displaystyle\theta}{\displaystyle\|\boldsymbol{x}_{M+1}\|}\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot}_{\infty},&\
\mbox{ if }\theta\geq\|\boldsymbol{x}_{M+1}\|,\end{array}\right.$ (14)

where $\boldsymbol{w}^{\bot}_{\infty}$ is an element of $L^{\bot}$.

Figure 4: Example of selective neuronal responses to stimulation with
different $(30\times 38)$-pixels images (only first few stimulus are shown in
the time line). Each neuron responds to its own (relevant) stimulus only and
rejects the other (background) stimuli.

Figure 4 shows typical responses of neurons parameterized by different pairs
$(\boldsymbol{w},\theta)$ and subjected to stimulation by different
information items $\boldsymbol{x}_{i}$. Here $\boldsymbol{x}_{i}$ correspond
to $(30\times 38)$-pixels color images (i.e.,
$\boldsymbol{x}_{i}\in\mathbb{R}^{3420}$). Firing thresholds $\theta$ have
been chosen at random, and weights $\boldsymbol{w}$ have been set in
accordance with (13) with the first three images serving as the relevant
information items for the three corresponding neurons. No plastic changes in
$\boldsymbol{w}$ were allowed. The neurons detect their own (relevant)
stimuli, as expected. Moreover, they do not respond to the stimulation by
other background information items (4 out of $10^{3}$ images are shown in Fig.
4). Thus, the neurons indeed exhibit high stimulus selectivity.

The following theorem provides theoretical justification for these
observations.

###### Theorem 4.1

Let elements of the sets $\mathcal{M}$ and $\mathcal{Y}$ be i.i.d. random
vectors drawn from the equidistribution in $B_{n}(1)$. Consider the sets of
stimuli $\mathcal{S}(\mathcal{M})$ and $\mathcal{S}(\mathcal{Y})$ specified by
(7). Let $(\boldsymbol{w},\theta)$ be the neuron parameters such that

$\boldsymbol{s}_{M+1}\in\mathrm{Activated}(\mathcal{S}({\mathcal{Y})},(\boldsymbol{w},\theta))\
\mbox{ and }\ 0<\theta<\|\boldsymbol{w}\|.$

Then:

1\. The probability that the neuron is silent for all background stimuli
$\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})$ is bounded from below by:

$\begin{split}&P(\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),(\boldsymbol{w},\theta))\
\forall\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})\big{|}\
\boldsymbol{w},\theta)\geq\\\ &\ \ \ \ \ \ \ \
\geq\left[1-\frac{1}{2}\left(1-\frac{\theta^{2}}{\|\boldsymbol{w}\|^{2}}\right)^{\frac{n}{2}}\right]^{M}.\end{split}$
(15)

2\. There is a family of sets parametrized by $D$
($0<D<\min\\{\frac{1}{2},\|\boldsymbol{x}_{M+1}\|\\}$):

$\Omega_{D}=\Big{\\{}(\boldsymbol{w},\theta)\big{|}\ \
\|\boldsymbol{w}-\boldsymbol{w}^{\ast}\|<D,\
D\leq\|\boldsymbol{x}_{M+1}\|-\theta\leq 2D\Big{\\}},$ (16)

where $\boldsymbol{w}^{\ast}=\boldsymbol{x}_{M+1}/\|\boldsymbol{x}_{M+1}\|$,
such that
$\boldsymbol{s}_{M+1}\in\mathrm{Activated}(\mathcal{S}(\mathcal{Y}),(\boldsymbol{w},\theta))$,
for $(\boldsymbol{w},\theta)\in\Omega_{D}$ and

$\begin{split}&P\big{(}\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),(\boldsymbol{w},\theta))\
\forall\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})\big{|}\
\forall(\boldsymbol{w},\theta)\in\Omega_{D}\big{)}\geq\\\ &\ \ \ \ \ \ \ \ \ \
\geq\max_{\varepsilon\in(0,1-2D)}(1-(1-\varepsilon)^{n})\left[1-\frac{1}{2}\rho(\varepsilon,D)^{\frac{n}{2}}\right]^{M}\end{split}$
(17)

where

$\rho(\varepsilon,D)=1-\left(\frac{1-\varepsilon-2D}{1+D}\right)^{2}.$

The proof is provided in Appendix B.

###### Remark 1

For an admissible fixed $D>0$, the volume $\mathcal{V}(\Omega_{D})>0$.
Therefore, the estimate provided by Theorem 4.1 is robust to small
perturbations of $(\boldsymbol{w},\theta)$, and slight fluctuations of
neuronal characteristics are not expected to affect neuronal functionality.

###### Remark 2

Theorem 4.1 (part 2) specifies a non-iterative procedure for constructing sets
of selective neurons. Such neurons detect given stimuli and reject the others,
with high probability. Figure 3 (in brown) shows examples of three projections
of the hypercylinders (16) ensuring robust selective stimulus detection. The
smaller is the cylinder, the higher is the selectivity.

To illustrate Theorem 4.1 numerically we fixed the neuronal dimensionality
parameter $n$ and generated two random sets of information items comprising of
$10^{3}$ elements each, i.e. $\\{\boldsymbol{x}_{i}\\}_{i=1}^{10^{3}}$. One
set was sampled from the equidistribution in a unit ball $B_{n}(1)$ centered
at the origin (i.e. $\|\boldsymbol{x}_{i}\|_{2}\leq 1$), and the other from
the equidistribution in the hypercube $\|\boldsymbol{x}_{i}\|_{\infty}\leq 1$
(a product distribution). For each set of informational items, a neuronal
ensemble of $10^{3}$ single neurons parameterized by
$(\boldsymbol{w}_{i},\theta_{i})$ was created. Each neuron was assigned а
fixed firing threshold $\theta_{i}=0.5$, $i=1,\dots,10^{3}$, whereas the
synaptic efficiencies were set as
$\boldsymbol{w}_{i}=(\theta_{i}+\epsilon)\boldsymbol{x}_{i}/\|\boldsymbol{x}_{i}\|$,
$\epsilon=0.05$. For these neuronal ensembles and their corresponding stimuli
sets we evaluated output of each neuron and assessed the neuronal selectivity
(see Def. 1). The procedure was repeated 10 times. This was followed by
evaluation of the frequencies of selective neurons in the pool for each $n$.

Figure 5: Extreme selectivity to stimuli and memory capacity of single
neurons. A) Stimulus selectivity vs the neuron dimension. The selectivity
index steeply increases for $n\in[10,20]$. For $n>20$ practically all neurons
become selective to a set of $10^{3}$ random stimuli. B) Memory capacity with
reliability $0.95$ of a neuronal ensemble vs the neuron dimension. For both
types of stimuli the memory capacity grows exponentially (straight lines show
regressions).

Figure 5A shows frequencies of selective neurons in an ensemble, for $10^{3}$
stimuli taken from: i) a unit ball (red), ii) a hypercube (blue), and iii) the
estimate provided by Theorem 4.1 (dashed). For $n$ small ($n<6$) neurons
exhibit no selectivity, i.e., they confuse different stimuli and generate
nonspecific responses. As expected, when neuronal dimensionality, $n$,
increases, the neuronal selectivity increases rapidly; and at around $n=20$ it
approaches $100\%$.

###  4.2 Extreme selectivity of a single neuron and ensemble memory capacity

The property of a neuron to respond selectively to a single element from a
large set of stimuli can be related to the notion of memory capacity of a
neuronal ensemble comprising of a set of selective neurons.

Recall that in the framework of associative memory hopfield1987neural , for
each informational item (pattern) $\boldsymbol{x}_{i}$ from the set
$\mathcal{M}$ there is a vicinity $\mathcal{V}_{i}$ associated with
$\boldsymbol{x}_{i}$ and corresponding to all admissible perturbations of
$\boldsymbol{x}_{i}$. Suppose that for each $\boldsymbol{x}_{i}$ there is a
neuron in the ensemble that is activated for all stimuli with informational
content $\boldsymbol{x}$ in $\mathcal{V}_{i}$ and is silent for all other
stimuli, i.e. for stimuli with $\boldsymbol{x}$ in $\cup_{j\neq
i}\mathcal{V}_{j}$. The maximal size of the set $\mathcal{M}$ for which this
property holds will be referred to as the (absolute) memory capacity of the
ensemble (cf. hopfield1987neural , barrett2004individual , leung1995stability
).

This conventional mechanistic definition of memory capacity, however, is too
restrictive to account for variability and uncertainty that biological
neuronal ensembles and systems are to deal with. Indeed, informational items
themselves may bear a degree of uncertainty resulting in that
$\mathcal{V}_{i}\cap\mathcal{V}_{j}\neq\varnothing$ for some $j,i$, $i\neq j$.
Furthermore, errors in memory retrievals are known to occur in classical
artificial associative memory models too (see e.g., hopfield1987neural ,
amit1987storing , leung1995stability ). To be able to formally quantify such
errors in relation to the number of informational items an ensemble is to
store, we extend the classical notion as follows.

Suppose that for each $\boldsymbol{x}_{i}$ there is a neuron in the ensemble
that is activated for all stimuli with informational content
$\boldsymbol{x}\in\mathcal{V}_{i}$ and, with probability $\phi$, is silent for
all stimuli with $\boldsymbol{x}\in\mathcal{V}_{j}$, $j\neq i$. The maximal
size of the set $\mathcal{M}$ for which this property holds will be referred
to as the memory capacity with reliability $\phi$ of the ensemble.

Assuming that $\mathcal{V}_{i}$ are sufficiently small an estimate of the
memory capacity with reliability $\phi$ of a neuronal ensemble follows from
Theorem 4.1.

###### Corollary 1

Let elements of the sets $\mathcal{M}$ and $\mathcal{Y}$ be i.i.d. random
vectors drawn from the equidistribution in $B_{n}(1)$. Consider the set of
stimuli $\mathcal{S}(\mathcal{M})$ as defined in (7). Then for a given fixed
$\phi\in(0,1)$ the maximal size $\overline{M}$ of the stimuli set
$\mathcal{S}(\mathcal{M})$ for which the following holds

$P(\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),(\boldsymbol{w},\theta))\
\forall\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})\big{|}\
\boldsymbol{w},\theta)\geq\phi$

grows at least exponentially with the neuronal dimension $n$:

$\overline{M}>-\ln\left(\phi\right)\left(2e^{\alpha n}-1\right),\ \mbox{ where
}\
\alpha=\ln\left[\frac{\|\boldsymbol{w}\|}{\sqrt{\|\boldsymbol{w}\|^{2}-\theta^{2}}}\right]>0.$
(18)

The proof is given in Appendix C.

Figure 5B illustrates how the memory capacity with reliability $\phi$ grows
with neuronal dimension $n$. For each neuronal dimension $n$ we generated
i.i.d. samples $\mathcal{M}$ with $|\mathcal{M}|=M$ from the equidistribution
in $B_{n}(1)$ and the $n$-cube $[-1,1]^{n}$. For each sample, we defined
neuronal ensembles comprising of $M$ neurons with synaptic weights
$\boldsymbol{w}_{i}=\boldsymbol{x}_{i}/\|\boldsymbol{x}_{i}\|$ and thresholds
$\theta_{i}=0.5$, and calculated the proportion of neurons in the ensemble
that are activated by each stimulus. If the proportion was smaller than $0.05$
of the total number of neurons, we incremented the value of $M$, generated a
new sample $\mathcal{M}$ with increased cardinality $M$, and repeated the
experiment. The values of $M$ corresponding to samples at which the process
stopped have been recorded and retained. These constituted empirical estimates
of the maximal number of stimuli for which the proportion of neurons
responding to a single stimulus is at most $0.05=1-\phi$. Figure 5B shows
empirical means of such numbers for the unit ball and in the hypercube. As
follows from these observations, memory capacity grows exponentially with the
neuron dimension in both cases. Such a fast growth can easily cover quite
exigent memory necessities.

###  4.3 Selectivity of a single neuron to multiple stimuli

To organize memories, the ability to associate different information items is
essential (Fig. 1C2). To determine if such associations are feasible at the
level of single neurons we assess neuronal selectivity to multiple stimuli. In
particular, we consider the set $\mathcal{Y}$ [Eq. (6)] containing $m>1$
random vectors:
$\mathcal{Y}=\\{\boldsymbol{x}_{M+1},\dots,\boldsymbol{x}_{M+m}\\}$. As in
Sect. 4.1, here we assume that all stimuli do not overlap in time and arrive
to the neuron separately. The question of interest is: Can we find a neuron
[i.e., parameters $(\boldsymbol{w},\theta)$], such that it would generate a
non-zero response to all $\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{Y})$ and,
with high enough probability, would be silent to all
$\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})$?

Below we will show that this is indeed possible, provided that the neuronal
dimensionality, $n$, is large enough. Moreover, the separation can be achieved
by a neuron with the vector of synaptic weights,
$\boldsymbol{w}=\boldsymbol{w}^{\ast}$, closely aligned with the mean vector
of the stimulus set $\mathcal{Y}$:

$\bar{\boldsymbol{x}}=\frac{1}{m}\sum_{i=1}^{m}\boldsymbol{x}_{M+i},\
{\boldsymbol{w}^{\ast}=\frac{\bar{\boldsymbol{x}}}{\|\bar{\boldsymbol{x}}\|}}.$
(19)

This vector points to the center of the group to be separated from the set
$\mathcal{M}$. In low dimensions, e.g. when $n=2$, such functionality appears
to be extremely unlikely. However, high dimensional neurons can accomplish
this task with probability close to one. Formal statement of this property is
provided in Theorem 4.2.

###### Theorem 4.2

Let elements of the sets $\mathcal{M}$ and $\mathcal{Y}$ be i.i.d. random
vectors drawn from the equidistribution in $B_{n}(1)$. Consider the sets of
stimuli $\mathcal{S}(\mathcal{M})$ and $\mathcal{S}(\mathcal{Y})$ specified by
(7) and let $D,\,\varepsilon,\,\delta\in(0,1)$ be chosen such that

$\theta^{\ast}=\frac{(1-\varepsilon)^{3}-\delta(m-1)}{\sqrt{m(1-\varepsilon)[1-\varepsilon+\delta(m-1)]}}\in(D,1).$
(20)

Let $\boldsymbol{w}^{\ast}=\bar{\boldsymbol{x}}/\|\bar{\boldsymbol{x}}\|$ and
consider the set:

$\Omega_{D}=\Big{\\{}(\boldsymbol{w},\theta)\big{|}\
\|\boldsymbol{w}-\boldsymbol{w}^{\ast}\|<D,\
\theta\in(0,\theta^{\ast}-D]\Big{\\}}.$

Then

$\begin{split}&P\Big{(}[\boldsymbol{s}_{i}\in\mathrm{Activated}(\mathcal{S}(\mathcal{Y}),\boldsymbol{w},\theta)\
\forall\ \boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{Y})\mathcal{]}\ \&\ \\\ &\
\
[\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),\boldsymbol{w},\theta)\
\forall\ \boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})]\Big{|}\
(\boldsymbol{w},\theta)\in\Omega_{D}\Big{)}\geq
p(\varepsilon,\delta,D,m),\end{split}$ (21)

where

$\begin{split}p(\varepsilon,\delta,D,m)=&(1-(1-\varepsilon)^{n})^{m}\prod_{d=1}^{m-1}\left(1-d\left(1-\delta^{2}\right)^{\frac{n}{2}}\right)\left[1-\frac{1}{2}\Delta^{\frac{n}{2}}\right]^{M},\\\
&\Delta=1-\frac{\theta^{2}}{(1+D)^{2}}.\end{split}$

The proof is provided in Appendix D. The theorem admits the following
corollary.

###### Corollary 2

Suppose that the conditions of Theorem 4.2 hold. Let $\theta^{\ast}>2D$ and
consider the set:

$\Omega_{D}^{\ast}=\Big{\\{}(\boldsymbol{w},\theta)\big{|}\
\|\boldsymbol{w}-\boldsymbol{w}^{\ast}\|<D,\
\theta\in[\theta^{\ast}-2D,\theta^{\ast}-D]\Big{\\}}.$

Then

$\begin{split}&P\Big{(}[\boldsymbol{s}_{i}\in\mathrm{Activated}(\mathcal{S}(\mathcal{Y}),\boldsymbol{w},\theta)\
\forall\ \boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{Y})\mathcal{]}\ \&\ \\\ &\
[\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),\boldsymbol{w},\theta)\
\forall\
\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})]\Big{|}(\boldsymbol{w},\theta)\in\Omega_{D}^{\ast}\Big{)}\geq\\\
&(1-(1-\varepsilon)^{n})^{m}\prod_{d=1}^{m-1}\left(1-d\left(1-\delta^{2}\right)^{\frac{n}{2}}\right)\left[1-\frac{1}{2}\Delta^{\frac{n}{2}}\right]^{M},\\\
&\Delta=1-\left(\frac{\theta^{\ast}-2D}{1+D}\right)^{2}.\end{split}$ (22)

###### Remark 3

Estimates (21), (22) hold for all feasible values of $\varepsilon$ and
$\delta$. Maximizing the r.h.s of (21), (22) over feasible domain of
$\varepsilon$, $\delta$ provides lower-bound “optimistic” estimates of the
neuron performance.

###### Remark 4

The term $\theta^{\ast}$ in Theorem 4.2 and Corollary 2 is an upper bound for
the firing threshold $\theta$. The larger is the value of $\theta$, the higher
is the neuronal selectivity to multiple stimuli. The value of $\theta^{\ast}$,
however, decays with the number of stimuli $m$.

The extent to which the decay mentioned in Remark 4 affects neuronal
selectivity to a group of stimuli depends largely on the neuronal dimension,
$n$. Note also that the probability of neuronal selective response to multiple
stimuli, as provided by Theorem 4.2, can be much larger if elements of the set
$\mathcal{Y}$ are spatially close to each other or positively correlated
TyuGor2017knowlege  (see also Lemma 4 in Appendix F).

###### Remark 5

Similarly to the case considered in Corollary 1, the maximal size of the
stimuli set $\mathcal{S}(\mathcal{M})$ for which selective response is
ensured, with some fixed probability, grows exponentially with dimension $n$.
Indeed, denoting $\phi=(1-z)^{\overline{M}}$, letting $z=1/2\Delta^{n/2}$
(with $\Delta$ defined in Theorem 4.2) and invoking (34), (35) from the proof
of Corollary 1, we observe that

$\overline{M}>-\ln(\phi)(z^{-1}-1)=-\ln(\phi)(2e^{\beta n}-1),\
\beta=\ln\frac{1+D}{\sqrt{(1+D)^{2}-\theta^{2}}}.$

Thus, for $M=|\mathcal{S}(\mathcal{M})|\leq\overline{M}$, the r.h.s. of (21)
is bounded from below by

$(1-(1-\varepsilon)^{n})^{m}\prod_{d=1}^{m-1}\left(1-d\left(1-\delta^{2}\right)^{\frac{n}{2}}\right)\phi.$

Similar estimate can be provided for the case considered in Corollary 2.

Figure 6: Selectivity of a single neuron to multiple stimuli. Panel (A)
corresponds to the case when the informational content vectors,
$\boldsymbol{x}_{i}$, are sampled from the equidistribution in the unit ball
$B_{n}(1)$, and panel (B) corresponds to the equidistribution in the $n$-cube
centered in the origin. In both cases the neuronal selectivity approaches
$100\%$ when the dimension $n$ grows. In (A) dashed curves show the estimates
provided by Theorem 4.2. Parameter values: $\varepsilon=0.01$, $D=0.001$,
$\delta=(1-\varepsilon)/2(m-1),\theta=\theta^{*}-D$.

To illustrate Theorem 4.2 we conducted several numerical experiments. For each
$n$ we generated $M=10^{3}$ of background information items
$\boldsymbol{x}_{i}$ (the set $\mathcal{M}$) and $m=2,5,8$ relevant vectors
(the sets $\mathcal{Y}$). In the first group of experiments all $M+m$ i.i.d.
random vectors were chosen from the equidistribution in $B_{n}(1)$. Neuronal
parameters were set in accordance with Theorem 4.2 (i.e., Eqs. (19) – (21)).
Figure 6A illustrates the results.

Similarly to the case of neuronal selectivity to a single item (Fig. 5A), we
observe a steep growth of the selectivity index with the neuronal dimension.
The sharp increase occurs, however, at significantly higher dimensions. The
number of random and uncorrelated stimuli, $m$, to which a neuron should be
able to respond selectively is fundamentally linked to the neuron
dimensionality. For example, the probability that a neuron is selective to
$m=5$ random stimuli becomes sufficiently high only at $n>400$. This contrasts
sharply with $n=120$ for $m=2$.

Our numerical experiments also show that the firing threshold specified in
Theorem 4.2 for arbitrarily chosen fixed values of $\delta$ and $\varepsilon$
is not optimal in the sense of providing the best possible probability
estimates. Playing with $\theta$ one can observe that the values of $n$ at
which neuronal selectivity to multiple stimuli starts to emerge are in fact
significantly lower than those predicted by Eq. (22). This is not surprising.
First, since estimate (22) holds for all admissible values of $\delta$ and
$\varepsilon$, it should also hold for the maximizer of
$p(\varepsilon,\delta,D,m)$. Second, the estimate is conservative in the sense
that it is based on conservative estimates of the volume of spherical cups
$\mathcal{C}_{n}$ (see, e.g., proof of Theorem 4.1). Deriving more accurate
numerical expressions for the latter is possible, although at the expense of
simplicity.

To demonstrate that dependence of the selectivity index on the firing
threshold is likely to hold qualitatively for broader classes of distributions
from which the sets $\mathcal{M}$ and $\mathcal{Y}$ are drawn, we repeated the
simulation for the equidistribution in an $n$-cube centered at the origin. In
this case, Theorem 4.2 does not formally apply. Yet, an equivalent statement
can still be produced (cf. GorbanTyukin:NN:2017 ). In these experiments
synaptic weights were set to
$\boldsymbol{w}=\bar{\boldsymbol{x}}/\|\bar{\boldsymbol{x}}\|$ and
$\theta=0.5\|\bar{\boldsymbol{x}}\|$. The results are shown in Fig. 6B. The
neuron’s performance in the cube is markedly better than that of in
$B_{n}(1)$. Interestingly, this is somewhat contrary to expectations that
might have been induced by our earlier experiments (shown in Fig. 5) in which
neuronal selectivity to a single stimulus was more pronounced for $B_{n}(1)$.

Overall, these results suggest that single neurons can indeed separate random
uncorrelated information items from a large set of background items with
probability close to one. This gives rise to a possibility for a neuron to
respond selectively to various arbitrary uncorrelated information items
simultaneously. The latter property provides a natural mechanism for accurate
and precise grouping of stimuli in single neurons.

###  4.4 Dynamic memory: Learning new information items by association

In the previous sections we dealt with a static model of neuronal functions,
i.e. when the synaptic efficiency $\boldsymbol{w}$ either did not change at
all or the changes were negligibly small over large intervals of stimuli
presentation. In the presence of synaptic plasticity (10), the latter case
corresponds to $0\leq\alpha\ll 1$ in (10). In this section we explicitly
account for the time evolution of the synaptic efficiency,
$\boldsymbol{w}(t,\boldsymbol{w}_{0})$ [Eq. (10)]. As we will see below, this
may give rise to dynamic memories in single neurons.

As before, we will deal with two sets of stimuli, the relevant one,
$\mathcal{S}(\mathcal{Y})$, and the background one,
$\mathcal{S}(\mathcal{M})$. We will consider two time epochs: i) Learning
phase and ii) Retrieval phase. Within the learning phase we assume that all
stimuli from the set $\mathcal{S}(\mathcal{Y})$ arrive to a neuron completely
synchronized, i.e.:

$\tau_{M+1,j}=\tau_{M+2,j}=\cdots=\tau_{M+m,j},\ \ \forall\ j.$ (23)

Such a synchronization could be interpreted as a mechanism for associating or
grouping different uncorrelated information items for the purposes of
memorizing them at a later stage.

The dynamics of the synaptic weights for $t\geq t_{0}$ is given Eq. (10) with
the input signal $\boldsymbol{s}$ replaced with:

$\bar{\boldsymbol{s}}(t)=\sum_{i=1}^{m}\boldsymbol{s}_{M+i}(t).$ (24)

Let $\boldsymbol{w}_{0}={\boldsymbol{w}(t_{0})}$ and $\theta$ satisfy the
following condition:

$\begin{split}&\exists\ \boldsymbol{s}_{k}\in\mathcal{S}(\mathcal{Y})\ \mbox{
such that }\
\boldsymbol{s}_{k}\in\mathrm{Activated}(\mathcal{S}(\mathcal{Y}),\boldsymbol{w}_{0},\theta)\\\
&\boldsymbol{s}_{i}\in\mathrm{Silent}({\mathcal{S}},\boldsymbol{w}_{0},\theta)\
\mbox{for all}\
\boldsymbol{s}_{i}\in{\mathcal{S}}\setminus\\{\boldsymbol{s}_{k}\\}.\end{split}$
(25)

Thus, at $t=t_{0}$ only one information item is “known” to the neuron. All
other relevant items from the set $\mathcal{Y}$ are “new” in the sense that
the neuron rejects them at $t=t_{0}$. Theorem 4.1 specifies the sets of
neuronal parameters $\boldsymbol{w}_{0},\theta$ for which condition (25) holds
with probability close to one if $n$ is large enough.

The question is: What is the probability that, during the learning phase the
synaptic weights $\boldsymbol{w}(t,\boldsymbol{w}_{0})$ evolve in time so that
the neuron becomes responsive to all
$\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{Y})$ whilst remaining silent to all
$\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})$ (Fig. 1C.3)? In other words,
the neuron learns new items and recognizes them in the retrieval phase. The
following theorem provides an answer to this question.

###### Theorem 4.3

Let elements of the sets $\mathcal{M}$ and $\mathcal{Y}$ be i.i.d. random
vectors drawn from the equidistribution in $B_{n}(1)$. Consider the sets of
stimuli $\mathcal{S}(\mathcal{M})$ and $\mathcal{S}(\mathcal{Y})$ specified by
(7). Let (23) hold, the dynamics of neuronal synaptic weights satisfy (10),
(24), and $(\boldsymbol{w}_{0},\theta)$ be chosen such that condition (25) is
satisfied. Pick $\varepsilon,\delta\in(0,1)$ such that

$(1-\varepsilon)^{3}>\delta(m-1).$

Moreover, suppose that

  1. 1.

There exist $L,\kappa>0$ such that

$\int_{t}^{t+L}v(\bar{\boldsymbol{s}}(\tau),\boldsymbol{w}(\tau,{\boldsymbol{w}_{0}}),\theta)\langle\bar{\boldsymbol{s}}(\tau),\boldsymbol{w}(\tau,{\boldsymbol{w}_{0}})\rangle^{2}{d\tau}>\kappa,\
\ \forall\ t\geq t_{0}.$

  2. 2.

The firing threshold, $\theta$, satisfies

$0<\theta<\frac{(1-\varepsilon)^{3}-\delta(m-1)}{\sqrt{m(1-\varepsilon)[(1-\varepsilon)+\delta(m-1)]}}={\theta^{\ast}}.$

Then for, any $0<D\leq\theta^{\ast}-\theta$, there is $t_{1}(D)>t_{0}$ such
that

$\begin{split}&P([\mathcal{S}(\mathcal{Y})\in\mathrm{Activated}(\mathcal{S},\boldsymbol{w}(t,{\boldsymbol{w}_{0}}),\theta)]\
\&\
[\mathcal{S}(\mathcal{M})\in\mathrm{Silent}(\mathcal{S},\boldsymbol{w}(t,{\boldsymbol{w}_{0}}),\theta)])\geq\\\
&(1-(1-\varepsilon)^{n})^{m}\prod_{d=1}^{m-1}\left(1-d\left(1-\delta^{2}\right)^{\frac{n}{2}}\right)\left[1-\frac{1}{2}\left(1-\frac{\theta^{2}}{(1+D)^{2}}\right)^{\frac{n}{2}}\right]^{M}\end{split}$

for all $t\geq t_{1}(D)$.

The proof is provided in Appendix E.

Figure 7: Dynamic memory: Learning new information items by association. A)
Example of the dynamic association of a known stimulus (neuron’s response to
the known stimulus is shown by green curve) and a new one (neuron’s response
shown by orange curve). Two relevant stimuli out of 502 are learnt by the
neuron. At $t\approx 2$ (red circle) the orange curve crosses the threshold
(red dashed line) and stays above it for $t>2$. Thus the neuron detects the
corresponding stimulus for $t>2$. B) Same as in A but for $m=4$ and $m=12$.
Parameter values: $\varepsilon=0.01$, $D=0.001$,
$\delta=(1-\varepsilon)^{3}/2(m-1)$, $\alpha=1$, $M=500$,
$\theta=\theta^{*}-D$, $n=400$.

Figure 7 illustrates the theorem numerically. First we assumed that the
relevant set $\mathcal{Y}$ consists of $m=2$ items. One of them is considered
as “known” to the neuron (Fig. 7A, green). Its informational content,
$\boldsymbol{x}_{M+1}$, satisfies the condition
$\langle\boldsymbol{w}_{0},\boldsymbol{x}_{M+1}\rangle>\theta$, i.e., this
stimulus evokes membrane potential above the threshold at $t=t_{0}$.
Consequently, the neuron detects this stimulus selectively as described in
Sect. 4.1. For the second relevant stimulus (Fig. 7A, orange), however, we
have $\langle\boldsymbol{w}_{0},\boldsymbol{x}_{M+2}\rangle<\theta$.
Therefore, the neuron cannot detect such a stimulus alone. The background
stimuli from the set $\mathcal{S}(\mathcal{M})$ are also sub-threshold (Fig.
7A, back curves).

During the learning phase, the neuron receives $M=500$ background and $m=2$
relevant stimuli. The relevant stimuli from the set $\mathcal{S}(\mathcal{Y})$
appear simultaneously, i.e., they are temporarily associated. The synaptic
efficiency changes during the learning phase by action of the relevant
stimuli. Therefore, the membrane potential,
$y(t)=\langle\boldsymbol{w}(t,{\boldsymbol{w}_{0}}),\bar{\boldsymbol{s}}(t)\rangle$,
progressively increases when the relevant stimuli arrive (Fig. 7A, green
area). These neuronal adjustments give rise to a new functionality.

At some time instant (marked by red circle in Fig. 7A) the neuron becomes
responsive to the new relevant stimulus (Fig. 7A, orange), which is
synchronized with the “known” one. Note that all other background stimuli,
that show no temporal associativity, remain below the threshold (Fig. 7A,
black traces). Thus, after a transient period, the neuron learns new stimulus.
Once the learning is over, the neuron detects selectively either of the two
relevant stimuli.

The procedure just described can be used to associate together more than two
relevant stimuli. Figure 7B shows examples for $m=4$ and $m=12$. In both cases
the neuron was able to learn all relevant stimuli, whilst rejecting all
background ones. We observed, however, that increasing the number of
uncorrelated information items to be learnt, i.e. the value of $m$, reduces
the gap between firing thresholds and the membrane potentials evoked by
background stimuli. In other words, the neuron does detect the assigned group
of new stimuli, but with lower accuracy. This behavior is consistent with the
theoretical bound on $\theta$ prescribed in the statement of Theorem 4.3.

##  5 Discussion

Theorems 4.1–4.3 and our numerical simulations demonstrate that the extreme
neuronal selectivity to single and multiple stimuli, and the capability to
learn uncorrelated stimuli observed in a range of empirical studies
Quiroga:2005 , Quiroga:2009 , Quiroga:2015  can be explained by simple
functional mechanisms implemented in single neurons. The following basic
phenomenological properties have been used to arrive to this conclusion: i)
the dimensionality $n$ of the information content and neurons is sufficiently
large, ii) a perceptron neuronal model, Eq. (9), is an adequate representation
of the neuronal response to stimuli, and iii) plasticity of the synaptic
efficiency is governed by Hebbian rule (10). A crucial consequence of our
study is that no a priori assumptions on the structural organization of
neuronal ensembles are necessary for explaining basic concepts of static and
dynamic memories.

Our approach does not take into account more advanced neuronal behaviors
reproduced by, e.g., models of spike-timing dependent plasticity Markram1997
and firing threshold adaptation fontaine2014spike . Nevertheless, our model
captures essential properties of neuronal dynamics and as such is generic
enough for the purpose of functional description of memories.

Firing threshold adaption, as reported in fontaine2014spike , steers firing
activity of a stimulated neuron to a homeostatic state. In this state, the
value of the threshold is just large/small enough to maintain reasonable
firing rate without over/under-excitation. In our model, such a mechanism
could be achieved by setting the value of $\theta$ sufficiently close to the
highest feasible values specified in Theorems 4.1 and 4.2.

In addition to rather general model of neuronal behavior, another major
theoretical assumption of our work was the presumption that stimuli
informational content is drawn from an equidistribution in a unit ball
$B_{n}(1)$. This assumption, however, can be relaxed, and results of Theorems
4.1–4.3 generalized to product measures. Key ingredients of such
generalizations are provided in GorbanTyukin:NN:2017 , and their practical
feasibility is illustrated by numerical simulations with information items
randomly drawn from a hypercube (Figs. 5–7).

Our theoretical and numerical analysis revealed an interesting hierarchy of
cognitive functionality implementable at the level of single neurons. We have
shown that cognitive functionality develops with the dimensionality or
connectivity parameter $n$ of single neurons. This reveals explicit
relationships between levels of the neural connectivity in living organisms
and different cognitive behaviors such organisms can exhibit (cf. Lobov2017 ).
As we can see from Theorems 4.1, 4.2 and Figs. 5, 6, the ability to form
static memories increases monotonically with $n$. The increase of cognitive
functionality, however, occurs in steps.

For $n$ small ($n\in[1,10]$), neuronal selectivity to a single stimulus does
not form. It emerges rapidly when the dimension parameter $n$ exceeds some
critical value, around $n=10\div 20$ (see Fig. 5A). This constitutes the first
critical transition. Single neurons become selective to single information
items. The second critical transition occurs at significantly larger
dimensions, around $n=100-400$ (see Fig. 6). At this second stage the neuronal
selectivity to multiple uncorrelated stimuli develops. The ability to respond
selectively to a given set of multiple uncorrelated information items is
apparently crucial for rapid learning “by temporal association” in such
neuronal systems. This learning ability as well as formation of dynamic
memories are justified by Theorem 4.3 and illustrated in Fig. 7.

In the core of our mathematical arguments are the concentration of measure
phenomena exemplified in GorTyu:2016 ; GorbanTyukin:RSTA:2017  and stochastic
separation theorems GorbanTyukin:NN:2017 ; GorTyuRom2016 . Some of these
results, which have been central in the proofs of Theorem 4.2 and 4.3, namely,
the statements that random i.i.d. vectors from equidistributions in $B_{n}(1)$
and product measures are almost orthogonal with probability close to one, are
tightly related to the notion of effective dimensionality of spaces based on
$\epsilon$-quasiorthogonality introduced in Hecht ; Kurkova . In these works
the authors demonstrated that in high dimensions there exist exponentially
large sets of quasiorthogonal vectors. In GorTyu:2016 , however, as well as in
our current work (see Lemma 3) we demonstrated that not only such sets exist,
but also that they are typical.

Finally, we note that the number of multiple stimuli that can be selectively
detected by single neurons is not extraordinarily large. In fact, as we have
shown in Figs. 6 and 7, memorizing $8$ information items at the level of
single neurons requires more than $400$ connections. This suggests, that not
only new memories are naturally packed in quanta, but also that there is a
limit on this number that is associated with the cost of implementation of
such a functionality. This cost is the number of individual functional
synapses. Balancing the costs in living beings is of course a subject of
selection and evolution. Nevertheless, as our study have shown, there is a
clear functional gain that these costs may be paid for.

##  6 Conclusion

In this work we analyzed the striking consequences of the abundance of
signalling routes for functionality of neural systems. We demonstrated that
complex cognitive functionality derived from extreme selectivity to external
stimuli and rapid learning of new memories at the level of single neurons can
be explained by the presence of multiple signalling routes and simple
physiological mechanisms. At the basic level, these mechanisms can be reduced
to a mere perceptron-like behavior of neurons in response to stimulation and a
Hebbian-type learning governing changes of the synaptic efficiency.

The observed phenomenon is robust. Remarkably, a simple generic model offers a
clearcut mathematical explanation of a wealth of empirical evidence related to
in-vivo recordings of “Grandmother” cells, “concept” cells, and rapid learning
at the level of individual neurons Quiroga:2005 ; Quiroga:2009 ; Quiroga:2015
. The results can also shed light on the question why Hebbian learning may
give rise to neuronal selectivity in prefrontal cortex lindsay2017hebbian  and
explain why adding single neurons to deep layers of artificial neural networks
is an efficient way to acquire novel information while preserving previously
trained data representations draelos2016neurogenesis .

Finding simple laws explaining complex behaviours has always been the driver
of progress in Mathematical Biology and Neuroscience. Numerous examples of
such simple laws can be found in the literature (see e.g. roberts2014can ;
jurica2013sensory ; gorban2016evolution ; perlovsky2006toward ). Our results
not only provide a simple explanation of the reported empirical evidence but
also suggest that such a behavior might be inherent to neuronal systems and
hence organisms that operate with high-dimensional informational content. In
such systems, complex cognitive functionality at the level of elementary
units, i.e., single neurons, occurs naturally. The higher the dimensionality,
the stronger the effect. In particular, we have shown that the memory capacity
in ensembles of single neurons grows exponentially with the neuronal
dimension. Therefore, from the evolutionary point of view, accommodating large
number of signalling routes converging onto single neurons is advantageous
despite the increased metabolic costs.

The considered class of neuronal models, being generic, is of course a
simplification. It does not capture spontaneous firing, signal propagation in
dendritic trees, and many other physiologically relevant features of real
neurons. Moreover, in our theoretical assessments we assumed that the
informational content processed by neurons is sampled from an equidistribution
in a unit ball. The results, however, can already be generalized to product
measure distributions (see, e.g., GorbanTyukin:NN:2017 ). Generalizing the
findings to models offering better physiological realism is the focus of our
future works.

###### Acknowledgements.

This work has been supported by Innovate UK grants KTP009890 and KTP010522, by
the Spanish Ministry of Economy and Competitiveness under grant
FIS2014-57090-P, the Russian Federation Ministry of Education state assignment
(No. 8.2080.2017/4.6), “Initiative scientific project” of the main part of the
state plan of the Ministry of Education and Science of Russian Federation
(task No. 2.6553.2017/BCH Basic Part), and by the Russian Science Foundation
project 15-12-10018 (numerical assessment and results). Alexander N. Gorban
was supported by the Ministry of Education and Science of Russian Federation
(Project No. 14.Y26.31.0022).

##  Appendix A Dynamics of coupling weights

The following results demonstrate that the neuronal model provided in Section
3 is well-posed.

###### Lemma 1

Consider (9), (10) with the function $\boldsymbol{s}(\cdot,\boldsymbol{x})$,
$\boldsymbol{x}\in\mathbb{R}^{n}$ defined as in (2). Then

  1. 1)

solutions $\boldsymbol{w}(\cdot,\boldsymbol{w}_{0})$ of (10) are defined for
all $t\geq t_{0}$, and are unique and bounded in forward time.

If, in addition, $\theta\geq 0$ and there exist numbers $L,\delta>0$ such
that:

$\int_{t}^{t+L}{v(\boldsymbol{s}(\tau,\boldsymbol{x}),\boldsymbol{w}(\tau,\boldsymbol{w}_{0}),\theta)\langle\boldsymbol{s}(\tau,\boldsymbol{x}),\boldsymbol{w}(\tau,\boldsymbol{w}_{0})\rangle^{2}}\,\mathrm{d}\tau>\delta,\
\ \ \forall t\geq t_{0},$ (26)

then

  1. 2)

$\boldsymbol{x}/\|\boldsymbol{x}\|$ is an attractor, that is:

$\lim_{t\rightarrow\infty}\boldsymbol{w}(t,\boldsymbol{w}_{0})=\frac{\boldsymbol{x}}{\|\boldsymbol{x}\|}.$
(27)

Proof of Lemma 1.

1\. The right-hand side of (10) is continuous in $\boldsymbol{w}$ and piece-
wise continuous in $t$ with finite number of discontinuities of the first kind
in any finite interval containing $t_{0}$, independently on the values of
$\boldsymbol{w}$. Hence, in accordance with Peano Theorem, solutions of (10)
are defined on some non-empty interval containing $t_{0}$. Let $\mathcal{T}$
be the maximal interval of this solution’s definition (to the right of
$t_{0}$). Since the right-hand side of (10) is locally Lipschitz in
$\boldsymbol{w}$ the solution $\boldsymbol{w}(\cdot,\boldsymbol{w}_{0})$ is
uniquely defined on $\mathcal{T}$.

To show that $\mathcal{T}=[t_{0},\infty)$ consider

$V(\boldsymbol{w})=1-\|\boldsymbol{w}\|^{2}.$

In the interval $\mathcal{T}$ we have:

$\dot{V}=-2\alpha vy^{2}V.$

Given that $vy^{2}\geq 0$, the above expression implies that

$|1-\|\boldsymbol{w}_{0}\|^{2}|\geq|1-\|\boldsymbol{w}(t,\boldsymbol{w}_{0})\|^{2}|\geq\|\boldsymbol{w}(t,\boldsymbol{w}_{0})\|^{2}-1.$

Consequently,

$\|\boldsymbol{w}(t,\boldsymbol{w}_{0})\|\leq\left(1+|1-\|\boldsymbol{w}_{0}\|^{2}|\right)^{\frac{1}{2}}$
(28)

for all $t\geq t_{0}$, $t\in\mathcal{T}$. Let $t_{1}$ be an arbitrary point in
the interval $\mathcal{T}$. Recall that the right-hand side of (10) is
continuous and locally Lipschitz with respect to $\boldsymbol{w}$ (uniformly
in $t$). Thus (28) implies existence of some
$\Delta(\boldsymbol{w}_{0},\boldsymbol{x})>0$, independent on $t_{1}$, such
that the solution $\boldsymbol{w}(\cdot,\boldsymbol{w}_{0})$ is defined on the
interval $[t_{0},t_{1}+\Delta(\boldsymbol{w}_{0},\boldsymbol{x})]$. Given that
$t_{1}$ was chosen arbitrarily in $\mathcal{T}$, we can conclude that
$\mathcal{T}=[t_{0},\infty)$ (cf. Theorem 3.3 Khalil_2002 ).

2\. For the sake of convenience, we denote

$p(t)=v(\boldsymbol{s}(t,\boldsymbol{x}),\boldsymbol{w}(t,\boldsymbol{w}_{0}),\theta)\langle\boldsymbol{s}(t,\boldsymbol{x}),\boldsymbol{w}(t,\boldsymbol{w}_{0})\rangle^{2}.$

Condition (26) assures that both $\boldsymbol{x}\neq 0$,
$\boldsymbol{w}_{0}\neq 0$. Moreover, since
$V(\boldsymbol{w}(t,\boldsymbol{w}_{0}))$ is defined for all $t\geq t_{0}$, we
can conclude that

${|V(t)|=\left|V_{0}e^{-{2}\alpha\int_{t_{0}}^{t}p(\tau)\mathrm{d}\tau}\right|\leq|V_{0}|e^{-{2}\alpha\delta\left\lfloor\frac{t-t_{0}}{L}\right\rfloor}.}$

Hence

$\lim_{t\rightarrow\infty}\|\boldsymbol{w}(t,\boldsymbol{w}_{0})\|=1.$ (29)

Consider:

$\begin{array}[]{ll}\boldsymbol{w}(t,\boldsymbol{w}_{0})=&e^{-\alpha\int_{t_{0}}^{t}p(\tau)\mathrm{d}\tau}\boldsymbol{w}_{0}+\\\
&{\alpha\left[\int_{t_{0}}^{t}e^{-\alpha\int_{\tau}^{t}p(s)\mathrm{d}s}v(\boldsymbol{s}(\tau,\boldsymbol{x}),\boldsymbol{w}(\tau,\boldsymbol{w}_{0}),\theta)\langle\boldsymbol{s}(\tau,\boldsymbol{x}),\boldsymbol{w}(\tau,\boldsymbol{w}_{0})\rangle\sum_{j}c(\tau-\tau_{j})\,\mathrm{d}\tau\right]}\boldsymbol{x}.\end{array}$

Observe that the first term decays exponentially to $0$, whereas the second
term is proportional to $\boldsymbol{x}$. Moreover, since $\theta\geq 0$, the
term
$v(\boldsymbol{s}(\tau,\boldsymbol{x}),\boldsymbol{w}(\tau,\boldsymbol{w}_{0}),\theta)\langle\boldsymbol{s}(\tau,\boldsymbol{x}),\boldsymbol{w}(\tau,\boldsymbol{w}_{0})\rangle\geq
0$ for all $\tau\geq t_{0}$. Hence the coefficient in front of
$\boldsymbol{x}$ is non-negative. This, combined with (29), implies that (27)
holds. $\square$

Note that Lemma 1 apply to stimuli classes that are broader than the one
defined by (2), (3). The results hold e.g. for the functions $c(\cdot)$ in (2)
that are non-negative, piece-wise continuous, and bounded. On the other hand,
to determine convergence and asymptotic properties of
$\boldsymbol{w}(\cdot,\boldsymbol{w}_{0})$ for $t\geq t_{0}$ (part 2 of the
lemma) one needs to check that condition (26) holds. A drawback of this
condition is that it requires availability of signals
$v(\boldsymbol{s}(t,\boldsymbol{x}),\boldsymbol{w}(t,\boldsymbol{w}_{0}),\theta)$,
$\langle\boldsymbol{s}(t,\boldsymbol{x}),\boldsymbol{w}(t,\boldsymbol{w}_{0})\rangle$
for all $t\geq t_{0}$.

For $c(\cdot)$ specified by (2) this latter condition can be drastically
simplified. To see this, let us get a somewhat deeper geometrical insight into
the dynamics of $\boldsymbol{w}$ governed by (10). In order to bring the
discussion in line with the question of neuronal selectivity, consider the
stimuli sets (6), (7) with $\mathcal{Y}=\\{\boldsymbol{x}_{M+1}\\}$, and
suppose that stimuli $\boldsymbol{s}(\cdot,\boldsymbol{x}_{i})$,
$i=1,\ldots,M$ do not evoke any neuronal responses, i.e.,
$v(\boldsymbol{s}(\cdot,\boldsymbol{x}_{i}),\boldsymbol{w}(\cdot,\boldsymbol{w}_{0}),\theta)=0$
for all $i=1,\ldots,M$. Hence no changes in $\boldsymbol{w}$ occur if the
stimulus $\boldsymbol{s}$ in (10) is any of
$\boldsymbol{s}(\cdot,\boldsymbol{x}_{i})$, $i=1,\ldots,M$.

Consider system (10) with $\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})$. The
variable $\boldsymbol{w}$ may change only over those intervals of $t$ when
$\boldsymbol{s}(\cdot,\boldsymbol{x}_{M+1})\neq 0$. Between these intervals
$\boldsymbol{w}(t,\boldsymbol{w}_{0})$ is constant. Let the stimulus be
persistent in the sense that for any $t^{\prime}\geq t_{0}$ there is a
$t^{\prime\prime}$ such that
$\boldsymbol{s}(t^{\prime\prime},\boldsymbol{x}_{M+1})\neq 0$. Thus, without
loss of generality and for the purposes of assessing asymptotic behavior of
$\boldsymbol{w}(t,\boldsymbol{w}_{0})$ at $t\rightarrow\infty$ variable
$\boldsymbol{s}(t,\boldsymbol{x}_{M+1})$ in (8) – (10) may be replaced with
$\boldsymbol{x}_{M+1}$.

Recall that $\boldsymbol{w}(t,\boldsymbol{w}_{0})$ can be represented as a sum

$\boldsymbol{w}(t,\boldsymbol{w}_{0})={w^{\ast}}(t,\boldsymbol{w}_{0})\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot}(t,\boldsymbol{w}_{0}),\
{w^{\ast}}(t,\boldsymbol{w}_{0})=\langle\boldsymbol{w}(t,\boldsymbol{w}_{0}),\boldsymbol{w}^{\ast}\rangle,$

where $\boldsymbol{w}^{\ast}$ is defined in (12) and $\boldsymbol{w}^{\bot}\in
L^{\bot}$. In this representation,

$\begin{split}&\dot{\boldsymbol{w}}=\dot{w}^{\ast}\boldsymbol{w}^{\ast}+\dot{\boldsymbol{w}}^{\bot}=\alpha
f(\langle\boldsymbol{x}_{M+1},{w^{\ast}}\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot}\rangle-\theta)\langle\boldsymbol{x}_{M+1},{w^{\ast}}\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot}\rangle(\boldsymbol{x}_{M+1}-\\\
&\langle\boldsymbol{x}_{M+1},{w^{\ast}}\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot}\rangle[{w^{\ast}}\boldsymbol{w}^{\ast}+\boldsymbol{w}^{\bot}])=\left[\alpha
f({w^{\ast}}\|\boldsymbol{x}_{M+1}\|-\theta)\|\boldsymbol{x}_{M+1}\|^{2}(1-{w^{\ast}}^{2})\right]{w^{\ast}}\boldsymbol{w}^{\ast}-\\\
&\left[\alpha
f({w^{\ast}}\|\boldsymbol{x}_{M+1}\|-\theta)\|\boldsymbol{x}_{M+1}\|^{2}{w^{\ast}}^{2}\right]\boldsymbol{w}^{\bot}\end{split}$

or, equivalently,

$\displaystyle\dot{w}^{\ast}$ $\displaystyle=$
$\displaystyle\alpha\|\boldsymbol{x}_{M+1}\|^{2}f({w^{\ast}}\|\boldsymbol{x}_{M+1}\|-\theta)(1-{w^{\ast}}^{2}){w^{\ast}}$
(30) $\displaystyle\dot{\boldsymbol{w}}^{\bot}$ $\displaystyle=$
$\displaystyle-\left[\alpha
f({w^{\ast}}\|\boldsymbol{x}_{M+1}\|-\theta)\|\boldsymbol{x}_{M+1}\|^{2}{w^{\ast}}^{2}\right]\boldsymbol{w}^{\bot}.$
(31)

Obviously, $L^{\|}$, $L^{\bot}$, and the set

$\mathcal{W}(\boldsymbol{x}_{M+1},\theta)=\\{({w^{\ast}},\boldsymbol{w}^{\bot}),\
{w^{\ast}}\in\mathbb{R},\boldsymbol{w}^{\bot}\in L^{\bot}\
|{w^{\ast}}\|\boldsymbol{x}_{M+1}\|-\theta\leq 0\\}$

are invariant with respect to (10). Let $\boldsymbol{x}_{M+1}\neq 0$,
$\theta\geq 0$, and
$\boldsymbol{w}_{0}\notin\mathcal{W}(\boldsymbol{x}_{M+1},\theta)$. Then two
non-trivial alternatives (Fig. 8) are possible:

Figure 8: Sketch of the dynamics of ${w^{\ast}}$. Thick black curve shows the
r.h.s. of (30) as a function of ${w^{\ast}}$ for two cases:
$\theta<\|x_{M+1}\|$ (left) and $\theta>\|x_{M+1}\|$ (right). Blue (red) dots
correspond to stable (unstable) equilibria. Green arrows mark trajectories. In
the first case (left) ${w^{\ast}}$ tends to 1, whereas in the second (right)
it goes asymptotically to $\theta/\|x_{M+1}\|$.

  * A:

If $\theta<\|\boldsymbol{x}_{M+1}\|$ then ${w^{\ast}}(t,\boldsymbol{w}_{0})\to
1$ and, according to (31), $\boldsymbol{w}^{\bot}(t,\boldsymbol{w}_{0})\to 0$
as $t\to\infty$. Thus,

$\lim_{t\to\infty}\boldsymbol{w}(t)=\frac{\boldsymbol{x}_{M+1}}{\|\boldsymbol{x}_{M+1}\|}=\boldsymbol{w}^{\ast}.$

  * B:

If $\theta\geq\|\boldsymbol{x}_{M+1}\|$ then
${w^{\ast}}(t,\boldsymbol{w}_{0})\to\theta/\|\boldsymbol{x}_{M+1}\|$ as
$t\to\infty$. There is no guarantee, however that
$\boldsymbol{w}^{\bot}(t,\boldsymbol{w}_{0})$ converges to the origin
asymptotically. Thus, there is a $\boldsymbol{w}^{\bot}_{\infty}\in L^{\bot}$:

$\lim_{t\to\infty}\boldsymbol{w}(t)=\frac{\theta}{\|\boldsymbol{x}_{M+1}\|}\boldsymbol{w}^{\ast}+{\boldsymbol{w}^{\bot}_{\infty}}.$

The above result can now be formalized as

###### Lemma 2

Consider (9), (10) with the function $\boldsymbol{s}(\cdot,\boldsymbol{x})$,
$\boldsymbol{x}\in\mathbb{R}^{n}$ defined as in (2). Let $\theta\geq 0$ and
$\langle\boldsymbol{w}_{0},\boldsymbol{x}\rangle>\theta$. Furthermore, let the
stimulus $\boldsymbol{s}(\cdot,\boldsymbol{x})$ be persistent in the sense
that for any $t^{\prime}\geq t_{0}$ there is a $t^{\prime\prime}>t^{\prime}$
such that $\boldsymbol{s}(t^{\prime\prime},\boldsymbol{x})\neq 0$. Then the
following alternatives hold:

  * 1)

If $\theta<\|\boldsymbol{x}\|$ then
$\lim_{t\rightarrow\infty}\boldsymbol{w}(t,\boldsymbol{w}_{0})=\boldsymbol{x}/\|\boldsymbol{x}\|$.

  * 2)

If $\theta\geq\|\boldsymbol{x}\|$ then
$\lim_{t\rightarrow\infty}\langle\boldsymbol{w}(t,\boldsymbol{w}_{0}),\boldsymbol{x}/\|\boldsymbol{x}\|\rangle=\theta/\|\boldsymbol{x}\|$.

Note that alternative 1) in Lemma 2 is equivalent to the second statement of
Lemma 1. Alternative 2) corresponds to the case when condition (26) of Lemma 1
is not satisfied.

##  Appendix B Proof of Theorem 4.1

1\. Let us first assume that $\|\boldsymbol{w}\|=1$. Notice that the condition

$\langle\boldsymbol{w},\boldsymbol{x}_{i}\rangle\leq\theta\ \ \
\forall\boldsymbol{x}_{i}\in\mathcal{M},$ (32)

assures that $v=0$ and hence
$\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),(\boldsymbol{w},\theta))\
\ \forall\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})$.

In this case the neuron is silent for all stimuli except
$\boldsymbol{s}_{M+1}$ that does evoke a response by construction. Therefore,
it is sufficient to estimate the probability that (32) holds.

Let $\mathcal{C}_{n}(\boldsymbol{w},\theta)$ be the spherical cap:

$\mathcal{C}_{n}(\boldsymbol{w},\theta)=\\{\boldsymbol{x}\in B_{n}(1)\ |\
\langle\boldsymbol{w},\boldsymbol{x}\rangle>\theta\\}.$

Then the ratio of volumes
$\mathcal{V}(\mathcal{C}_{n}(\boldsymbol{w},\theta))/\mathcal{V}(B_{n}(1))$ is
the probability that a random vector
$\boldsymbol{x}_{i}\in\mathcal{C}_{n}(\boldsymbol{w},\theta)$. Observe that

$\frac{\mathcal{V}(\mathcal{C}_{n}(\boldsymbol{w},\theta))}{\mathcal{V}(B_{n}(1))}\leq\frac{1}{2}(1-\theta^{2})^{\frac{n}{2}}.$

Thus, the probability that all $\boldsymbol{x}_{i}\in\mathcal{M}$ are outside
the cap $\mathcal{C}_{n}(\boldsymbol{w},\theta)$ is bounded from below:

$P=\left[1-\frac{\mathcal{V}(\mathcal{C}_{n}(\boldsymbol{w},\theta))}{\mathcal{V}(B_{n}(1))}\right]^{M}\geq\left[1-\frac{1}{2}(1-\theta^{2})^{\frac{n}{2}}\right]^{M},$
(33)

which is equivalent to (15), given that $\|\boldsymbol{w}\|=1$.

Let $\|\boldsymbol{w}\|\neq 1$. Noticing that, for $\|\boldsymbol{w}\|>0$

$\langle\boldsymbol{w},\boldsymbol{x}_{i}\rangle\leq\theta\ \ \
\forall\boldsymbol{x}_{i}\in\mathcal{M}\ \Leftrightarrow\
\langle\boldsymbol{w}/\|\boldsymbol{w}\|,\boldsymbol{x}_{i}\rangle\leq\theta/\|\boldsymbol{w}\|\
\ \ \forall\boldsymbol{x}_{i}\in\mathcal{M},$

and substituting $\theta/\|\boldsymbol{w}\|$ in place of $\theta$ in (33)
results in (15).

2\. Let us show that for $(\boldsymbol{w},\theta)\in\Omega_{D}$ the neuron
detects the relevant stimulus $\boldsymbol{s}_{M+1}$, i.e., $v>0$. Using (16)
we observe that

$\begin{split}\langle\boldsymbol{w},\boldsymbol{x}_{M+1}\rangle-\theta&=\langle\boldsymbol{w}-\boldsymbol{w}^{\ast},\boldsymbol{x}_{M+1}\rangle+\|x_{M+1}\|-\theta\geq\langle\boldsymbol{w}-\boldsymbol{w}^{\ast},\boldsymbol{x}_{M+1}\rangle+D\geq\\\
&\geq-\|\boldsymbol{w}-\boldsymbol{w}^{\ast}\|\|\boldsymbol{x}_{M+1}\|+D>D(1-\|\boldsymbol{x}_{M+1}\|)\geq
0,\end{split}$

implying that
$\boldsymbol{s}_{M+1}\in\mathrm{Activated}(\mathcal{Y},(\boldsymbol{w},\theta))$.

Let us evaluate the probability that the neuron rejects all background stimuli
for all $(\boldsymbol{w},\theta)\in\Omega_{D}$. According to (16) the
following holds:

$\frac{\theta}{\|\boldsymbol{w}\|}\geq\frac{\|\boldsymbol{x}_{M+1}\|-2D}{1+D},\
\ \ \forall(\boldsymbol{w},\theta)\in\Omega_{D}.$

Moreover, $\|\boldsymbol{x}_{M+1}\|\geq 1-\varepsilon$ with probability
$p=1-(1-\varepsilon)^{n}$. Therefore, with probability larger or equal to $p$,
the ratio $\frac{\theta}{\|\boldsymbol{w}\|}$ us bounded from below as:

$\frac{\theta}{\|\boldsymbol{w}\|}\geq\frac{1-\varepsilon-2D}{1+D}.$

Finally, since the value of $\varepsilon$ can be chosen arbitrarily in the
interval $(0,1-2D)$ and taking into account that the right-hand side of (33)
is a monotone and increasing function with respect to $\theta$ in the interval
$[0,1]$, estimate (17) immediately follows from (33) and (15). $\square$

##  Appendix C Proof of Corollary 1

Consider (15) and denote

$z=\frac{1}{2}\left[1-\frac{\theta^{2}}{\|\boldsymbol{w}\|^{2}}\right]^{\frac{n}{2}},\
\ \ \phi=(1-z)^{\overline{M}}.$ (34)

According to (34), $(1-z)^{M}\geq\phi$ for all $0<M\leq\overline{M}$. Given
that $z\in(0,1)$, from Eq. (34) we get $\ln(\phi)=\overline{M}\ln(1-z)$.
Recall that $\ln(1-z)>-z/(1-z)$, $\forall z\in(0,1)$. Thus, we can conclude
that

$\overline{M}>-\ln(\phi)\frac{1-z}{z}=-\ln(\phi)(z^{-1}-1)=-\ln(\phi)\left(2e^{\alpha
n}-1\right),$ (35)

where $\alpha$ is given by (18). Thus, according to (35), for
$0<M\leq-\ln(\phi)\left(2e^{an}-1\right)<\overline{M}$ the following holds

$P(\boldsymbol{s}_{i}\in\mathrm{Silent}(\mathcal{S}(\mathcal{M}),(\boldsymbol{w},\theta))\
\forall\boldsymbol{s}_{i}\in\mathcal{S}(\mathcal{M})\big{|}\
\boldsymbol{w},\theta)\geq\phi.$

$\square$

##  Appendix D Proof of Theorem 4.2

The proof of the Theorem is essentially contained in Lemmas 3 and 4 (Sect. F).
Consider the set $\mathcal{Y}$. With probability
$(1-(1-\varepsilon)^{n})^{m}$, all elements $\boldsymbol{x}_{i}\in\mathcal{Y}$
satisfy the condition $\|\boldsymbol{x}_{i}\|\geq 1-\varepsilon$. Hence, using
Lemma 3 we have that the following inequality

$|\langle\boldsymbol{x}_{i},\boldsymbol{x}_{j}\rangle|\leq\frac{\delta}{1-\varepsilon},\
\ \forall\boldsymbol{x}_{i},\boldsymbol{x}_{j}\in\mathcal{Y},\ i\neq j$

holds with probability

$p_{0}\geq(1-(1-\varepsilon)^{n})^{m}\prod_{d=1}^{m-1}\left(1-d(1-\delta^{2})^{\frac{n}{2}}\right).$

This implies that, with probability $p_{0}$, the following conditions are met

$\|\boldsymbol{x}_{i}\|\geq 1-\varepsilon,\ \ \
-\frac{(m-1)\delta}{1-\varepsilon}\leq\sum_{j=1,\ j\neq
i}^{m}\langle\boldsymbol{x}_{i},\boldsymbol{x}_{j}\rangle\leq\frac{(m-1)\delta}{1-\varepsilon},\
\ \forall\ \boldsymbol{x}_{i}\in\mathcal{Y}.$

Consider
$\ell(\boldsymbol{x})=\langle\boldsymbol{w}^{\ast},\boldsymbol{x}\rangle-\theta^{\ast}+D$.
Invoking Lemma 4 and setting $\beta_{1}=\delta/(1-\varepsilon)$,
$\beta_{2}=-\delta/(1-\varepsilon)$, we can conclude that, with probability
$p_{0}$,

$\ell(\boldsymbol{x})\geq D,\ \ \forall\boldsymbol{x}\in\mathcal{Y}.$

In fact, we can conclude that with probability $p_{0}$

$\ell_{0}(\boldsymbol{x})=\langle\boldsymbol{w},\boldsymbol{x}\rangle-\theta=\ell(\boldsymbol{x})+\langle\boldsymbol{w}-\boldsymbol{w}^{\ast},\boldsymbol{x}\rangle-\theta+(\theta^{\ast}-D)>0,\
\ \forall\ (\boldsymbol{w},\theta)\in\Omega_{D},\
\boldsymbol{x}\in\mathcal{Y}.$

Thus, the probability that $\ell_{0}(\boldsymbol{x})>0$ for all
$\boldsymbol{x}\in\mathcal{Y}$ and that $\ell_{0}(\boldsymbol{x})\leq 0$ for
all $\boldsymbol{x}\in\mathcal{M}$ is bounded from below by

$(1-(1-\varepsilon)^{n})^{m}\prod_{d=1}^{m-1}\left(1-d(1-\delta^{2})^{\frac{n}{2}}\right)\left[1-\frac{1}{2}\left(1-\frac{\theta^{2}}{\|\boldsymbol{w}\|^{2}}\right)^{\frac{n}{2}}\right]^{M}.$

Noticing that $\|\boldsymbol{w}\|\leq 1+D$, we can conclude that (21) holds.
$\square$

##  Appendix E Proof of Theorem 4.3

According to Lemma 1, solutions $\boldsymbol{w}(t,\boldsymbol{w}_{0})$ are
defined for all $t\geq t_{0}$. Moreover, condition 1 of the theorem and Lemma
1 imply that

$\lim_{t\rightarrow\infty}\boldsymbol{w}(t,{\boldsymbol{w}_{0}})=\frac{\sum_{i=1}^{m}\boldsymbol{x}_{M+i}}{\|\sum_{i=1}^{m}\boldsymbol{x}_{M+i}\|}=\bar{\boldsymbol{x}}/\|\bar{\boldsymbol{x}}\|=\boldsymbol{w}^{\ast}.$
(36)

Let $D>0$ be chosen so that

$0<\theta+D\leq\theta^{\ast}.$

Given that $0<\theta<\theta^{\ast}$, such $D$s always exist. Equation (36)
implies that there is a $t_{1}(D)>t_{0}$ such that

${\|\boldsymbol{w}(t,{\boldsymbol{w}_{0}})-\boldsymbol{w}^{\ast}\|<D,\
\theta\in(0,\theta^{\ast}-D]\ \forall\ t\geq t_{1}(D).}$ (37)

The theorem now follows immediately from Theorem 4.2. $\square$

##  Appendix F Auxiliary results

######  Lemma 3 (cf. Gorban et. al. 2016, GorTyu:2016 )

Let
$\mathcal{Y}=\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},\dots,\boldsymbol{x}_{k}\\}$
be a set of $k$ i.i.d. random vectors from the equidistribution in the unit
ball $B_{n}(1)$. Let $\delta,r\in(0,1)$, and suppose that
$\|\boldsymbol{x}_{i}\|\geq r$, for all $i\in\\{1,\dots,k\\}$.

Then the probability that the elements of $\mathcal{Y}$ are pair-wise
$\delta/r$-orthogonal, that is

$|\cos(\angle(\boldsymbol{x}_{i},\boldsymbol{x}_{j}))|\leq\frac{\delta}{r}\
\mbox{for all}\ i\neq j\ \ i,j\in\\{1,\dots,k\\},$

is bonded from below as

$\begin{split}&\mathcal{P}\left(|\cos(\angle(\boldsymbol{x}_{i},\boldsymbol{x}_{j}))|\leq\frac{\delta}{r}\
\forall\ i,j\in\\{1,\dots,k\\},\ i\neq j\ \left|\ \|\boldsymbol{x}_{i}\|\geq
r,\ 1\leq i\leq k\right.\right)\\\ &\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
\geq\prod_{d=1}^{{k}-1}\left(1-d\left(1-\delta^{2}\right)^{\frac{n}{2}}\right).\end{split}$

Proof of Lemma 3. Let $\boldsymbol{x}_{i}$, $i=1,\dots,k$ be random vectors
satisfying conditions of the lemma. Let $E_{\delta}(\boldsymbol{x}_{i})$ be
the delta-thickening of the largest equator of $B_{n}(1)$ that is orthogonal
to $\boldsymbol{x}_{i}$. There is only one such equator, and it is uniquely
determined by $\boldsymbol{x}_{i}$. Consider the following probabilities:

$\begin{split}&P(\boldsymbol{x}_{2}\in\ E_{\delta}(\boldsymbol{x}_{1}))\\\
&P([\boldsymbol{x}_{3}\in\
E_{\delta}(\boldsymbol{x}_{2})]\&[\boldsymbol{x}_{3}\in\
E_{\delta}(\boldsymbol{x}_{1})])\\\ &P([\boldsymbol{x}_{4}\in\
E_{\delta}(\boldsymbol{x}_{3})]\&[\boldsymbol{x}_{4}\in\
E_{\delta}(\boldsymbol{x}_{2})]\&[\boldsymbol{x}_{4}\in\
E_{\delta}(\boldsymbol{x}_{1})])\\\ &\cdots\\\ &P([\boldsymbol{x}_{k}\in\
E_{\delta}(\boldsymbol{x}_{k-1})]\&\cdots\&[\boldsymbol{x}_{k}\in\
E_{\delta}(\boldsymbol{x}_{1})]).\end{split}$

Pick $\boldsymbol{x}_{i},\boldsymbol{x}_{j}\in\mathcal{Y}$, $i\neq j$. Recall
that, for any random events $A_{1},\dots,A_{k}$, the probability

$P(A_{1}\&A_{2}\&\cdots\&A_{k})\geq 1-\sum_{i=1}^{k}(1-P(A_{i})).$ (38)

According to (38), the probability that $\boldsymbol{x}_{i}\in
E_{\delta}(\boldsymbol{x}_{j})$ is bounded from below by
$1-\left(1-\delta^{2}\right)^{\frac{n}{2}}$ (cf. GorTyu:2016 , Proposition 3;
see also Fig. 1 in GorTyu:2016  for illustration). Then

$\begin{split}&P(\boldsymbol{x}_{2}\in\ E_{\delta}(\boldsymbol{x}_{1}))\geq
1-\left(1-\delta^{2}\right)^{\frac{n}{2}}\\\ &P([\boldsymbol{x}_{3}\in\
E_{\delta}(\boldsymbol{x}_{2})]\&[\boldsymbol{x}_{3}\in\
E_{\delta}(\boldsymbol{x}_{1})])\geq
1-2\left(1-\delta^{2}\right)^{\frac{n}{2}}\\\ &P([\boldsymbol{x}_{4}\in\
E_{\delta}(\boldsymbol{x}_{3})]\&[\boldsymbol{x}_{4}\in\
E_{\delta}(\boldsymbol{x}_{2})]\&[\boldsymbol{x}_{4}\in\
E_{\delta}(\boldsymbol{x}_{1})])\geq
1-3\left(1-\delta^{2}\right)^{\frac{n}{2}}\\\ &\cdots\\\
&P([\boldsymbol{x}_{k}\in\
E_{\delta}(\boldsymbol{x}_{k-1})]\&\cdots\&[\boldsymbol{x}_{k}\in\
E_{\delta}(\boldsymbol{x}_{1})])\geq
1-(k-1)\left(1-\delta^{2}\right)^{\frac{n}{2}}.\end{split}$ (39)

The fact that $\boldsymbol{x}_{i}\in E_{\delta}(\boldsymbol{x}_{j})$ combined
with the condition that $\|\boldsymbol{x}_{i}\|\geq r$,
$\|\boldsymbol{x}_{j}\|\geq r$ imply:

$|\cos(\angle(\boldsymbol{x}_{i},\boldsymbol{x}_{j}))|\leq\frac{\delta}{r}.$

Finally, given that $\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{k}$ are drawn
independently and that the distribution is rotationally invariant, the
probability that all vectors in $\mathcal{Y}$ are pair-wise orthogonal is the
product of all probabilities in the left-hand side of (39). Thus the statement
follows. $\square$

###### Lemma 4

Let $\mathcal{Y}=\\{\boldsymbol{x}_{1},\dots,\boldsymbol{x}_{m}\\}$ be a
finite set from $B_{n}(1)$. Let $\|\boldsymbol{x}_{i}\|\geq 1-\varepsilon$,
$\varepsilon\in(0,1)$ for all $\boldsymbol{x}_{i}\in\mathcal{Y}$, and
$\beta_{1},\beta_{2}\in\mathbb{R}$ be such that the following condition holds:

$\beta_{2}(m-1)\leq\sum_{j\in\\{1,\dots,m\\},\ j\neq
i}\langle\boldsymbol{x}_{i},\boldsymbol{x}_{j}\rangle\leq\beta_{1}(m-1)\
\mbox{for all}\ i=1,\dots,m.$ (40)

Consider

$\begin{split}\ell(\boldsymbol{x})=\left\langle\frac{\bar{\boldsymbol{y}}}{\|\bar{\boldsymbol{y}}\|},\boldsymbol{x}\right\rangle-\frac{1}{\sqrt{m}}\left(\frac{(1-\varepsilon)^{2}+\beta_{2}(m-1)}{\sqrt{1+(m-1)\beta_{1}}}\right),\
\bar{\boldsymbol{y}}=&\frac{1}{m}\sum_{i=1}^{m}\boldsymbol{x}_{i},\end{split}$

and suppose that parameters $\beta_{1},\beta_{2}$ satisfy:

$(1-\varepsilon)^{2}+\beta_{2}(m-1)>0,\ 1+(m-1)\beta_{1}>0.$

Then

$\ell(\boldsymbol{x}_{i})\geq 0\ \mbox{for all}\
\boldsymbol{x}_{i}\in\mathcal{Y}.$ (41)

Proof of Lemma 4. Consider the set $\mathcal{Y}$. According to the lemma
assumptions, $\|\boldsymbol{x}_{i}\|\geq 1-\varepsilon$ for some given
$\varepsilon\in(0,1)$ and all $i=1,\dots,m$. Consider now the mean vector
$\bar{\boldsymbol{y}}$

$\bar{\boldsymbol{y}}=\frac{1}{m}\sum_{i=1}^{m}\boldsymbol{x}_{i},$

and evaluate the following inner products

$\left\langle\frac{\bar{\boldsymbol{y}}}{\|\bar{\boldsymbol{y}}\|},\boldsymbol{x}_{i}\right\rangle=\frac{1}{m\|\bar{\boldsymbol{y}}\|}\left(\|\boldsymbol{x}_{i}\|^{2}+\sum_{j\in\\{1,\dots,m\\},\
j\neq i}\langle\boldsymbol{x}_{i},\boldsymbol{x}_{j}\rangle\right),\
i=1,\dots,m.$

According to assumption (40), the following holds

$\left\langle\frac{\bar{\boldsymbol{y}}}{\|\bar{\boldsymbol{y}}\|},\boldsymbol{x}_{i}\right\rangle\geq\frac{1}{m\|\bar{\boldsymbol{y}}\|}\left((1-\varepsilon)^{2}+\beta_{2}(m-1)\right),$

and, respectively,

$\frac{1}{m}\left(1+(m-1)\beta_{1}\right)\geq\langle\bar{\boldsymbol{y}},\bar{\boldsymbol{y}}\rangle=\|\bar{\boldsymbol{y}}\|^{2}\geq\frac{1}{m}\left((1-\varepsilon)^{2}+\beta_{2}(m-1)\right)$

Let $(1-\varepsilon)^{2}+\beta_{2}(m-1)>0$ and $1+\beta_{1}(m-1)>0$. It is
clear that for $\ell$, as defined by (41), the following holds for all
$i=1,\dots,m$: $\ell(\boldsymbol{x}_{i})\geq 0$. $\square$

## References

  * [1] D.G. Amaral and M.P. Witter.  The three-dimensional organization of the hippocampal formation: a review of anatomical data.  Neuroscience, 31:571–591, 1989.
  * [2] Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky.  Storing infinite numbers of patterns in a spin-glass model of neural networks.  In Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications, pages 428–431. World Scientific, 1987.
  * [3] P. Andersen, R. Morris, D. Amaral, T. Bliss, and J. O’Keefe, editors.  The hippocampus book.  2007\.
  * [4] Lisa Feldman Barrett, Michele M Tugade, and Randall W Engle.  Individual differences in working memory capacity and dual-process theories of the mind.  Psychological bulletin, 130(4):553, 2004.
  * [5] N. Benito, A. Fernandez-Ruiz, V.A. Makarov, J. Makarova, A. Korovaichuk, and O. Herreras.  Spatial modules of coherent activity in pathway-specific lfps in the hippocampus reflect topology and different modes of presynaptic synchronization.  Cerebral Cortex, 11(7):1738–1752, 2014.
  * [6] N. Benito, G. Martin-Vazquez, J. Makarova, V.A. Makarov, and O. Herreras.  The right hippocampus leads the bilateral integration of gamma-parsed lateralized information.  eLife, 5:e16658, doi: 10.7554/eLife.16658, 2016.
  * [7] C. Calvo, J.A. Villacorta-Atienza, V.I. Mironov, V. Gallego, and V.A. Makarov.  Waves in isotropic totalistic cellular automata: Application to real-time robot navigation.  Advances in Complex Systems, 19(4):1650012–18, 2016.
  * [8] D.D. Clark and L. Sokoloff.  Circulation and energy metabolism of the brain.  In G.J. Siegel, B.W. Agranoff, R.W. Albers, S.K. Fisher, and M.D. Uhler, editors, Basic Neurochemistry: Molecular, Cellular and Medical Aspects, pages 637–670. Lippincott, Philadelphia, 1999.
  * [9] F. Cucker and S. Smale.  On the mathematical foundations of learning.  Bulletin of the American mathematical society, 39(1):1–49, 2002\.
  * [10] T. J. Draelos, N. E. Miner, C. C. Lamb, C. M. Vineyard, K. D. Carlson, C. D. James, and J. B. Aimone.  Neurogenesis deep learning.  arXiv preprint arXiv:1612.03770, 2016.
  * [11] A. Fernandez-Ruiz, V.A. Makarov, and O. Herreras.  Sustained increase of spontaneous input and spike transfer in the ca3-ca1 pathway following long term potentiation in vivo.  Frontiers in Neural Circuits, 6:71, 2012.
  * [12] C.T. Finnerty and J.G.R. Jefferys.  Functional connectivity from ca3 to the ipsilateral and contralateral ca1 in the rat dorsal hippocampus.  Neuroscience, 56(1), 1993.
  * [13] B. Fontaine, J.L. Peña, and R. Brette.  Spike-threshold adaptation predicted by membrane potential dynamics in vivo.  PLoS computational biology, 10(4):e1003560, 2014.
  * [14] A.N. Gorban and I.Y. Tyukin.  Stochastic separation theorems.  Neural Networks, 94:255–259, 2017.
  * [15] A.N. Gorban and I.Y. Tyukin.  Blessing of dimensionality: mathematical foundations of the statistical physics of data.  Phiolosphical Transactions of the Royal Society A. doi:10.1098/rsta.2017.0237, 2018.
  * [16] A.N. Gorban, I.Y. Tyukin, and I. Romanenko.  The blessing of dimensionality: Separation theorems in the thermodynamic limit.  IFAC-PapersOnLine, 49(24):64–69, 2016.  2th IFAC Workshop on Thermodynamic Foundations for a Mathematical Systems Theory TFMST 2016.
  * [17] A.N. Gorban, I.Yu. Tyukin, D.V. Prokhorov, and K.I. Sofeikov.  Approximation with random bases: Pro et contra.  Information Sciences, 364–365:129–145, 2016.
  * [18] A.N. Gorban, T.A. Tyukina, E.V. Smirnova, and L.I. Pokidysheva.  Evolution of adaptation mechanisms: Adaptation energy, stress, and oscillating death.  Journal of theoretical biology, 405:127–139, 2016.
  * [19] R. Hecht-Nielsen.  Context vectors: General-purpose approximate meaning representations self-organized from raw data.  In J. Zurada, R. Marks, and C. Robinson, editors, Computational Intelligence: Imitating Life, pages 43 – 56. IEEE Press, 1994.
  * [20] S. Herculano-Houzel.  The human brain in numbers: a linearly scaled-up primate brain.  Front Hum Neurosci, 2009.
  * [21] S. Herculano-Houzel.  Gorilla and orangutan brains conform to the primate cellular scaling rules: Implications for human evolution.  Brain, Behavior and Evolution, 77:33–44, 2011.
  * [22] S. Herculano-Houzel.  The remarkable, yet not extraordinary, human brain as a scaled-up primate brain and its associated cost.  Proceedings of National Academy of Science, 109:10661–10668, 2012\.
  * [23] John J Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  In Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications, pages 411–415. World Scientific, 1987.
  * [24] N. Ishizuka, J. Weber, and D.G. Amaral.  Organization of intrahippocampal projections riginating from ca3 pyramidal cells in the rat.  J Comp Neurol, 295(580-623), 1990.
  * [25] M.J. Ison, R. Quian Quiroga, and I. Fried.  Rapid encoding of new memories by individual neurons in the human brain.  Neuron, 87(1):220–230, 2015.
  * [26] P. Jurica, S. Gepshtein, I. Tyukin, and C. van Leeuwen.  Sensory optimization by stochastic tuning.  Psychological Review, 120(4):798–816, 2013.
  * [27] P.C. Kainen and V. Kurkova.  Quasiorthogonal dimension of euclidian spaces.  Appl. Math. Lett., 6(3):7–10, 1993.
  * [28] H. Khalil.  Nonlinear Systems (3d edition).  Prentice Hall, 2002.
  * [29] Chi-Sing Leung, Lai-Wan Chan, and Edmund Lai.  Stability, capacity, and statistical dynamics of second-order bidirectional associative memory.  IEEE Transactions on Systems, Man, and Cybernetics, 25(10):1414–1424, 1995.
  * [30] X.G. Li, P. Somogyi, A. Ylinen, and G. Buzsaki.  The hippocampal ca3 network: an in vivo intracellular labeling study.  J Comp Neurol, 339:181–208, 1994.
  * [31] G. W. Lindsay, M. Rigotti, M. R. Warden, E. K. Miller, and S. Fusi.  Hebbian learning in a random network captures selectivity properties of prefrontal cortex.  bioRxiv, page 133025, 2017.
  * [32] S.A. Lobov, M.O. Zhuravlev, V.A. Makarov, and V.B. Kazantsev.  Noise enhanced signaling in stdp driven spiking-neuron network.  Mathematical Modelling of Natural Phenomena, 12(4):109–124, 2017\.
  * [33] H. Markram, J. Lubke, M. Frotscher, and B. Sakmann.  Regulation of synaptic efficacy by coincidence of postsynaptic aps and epsps.  Science, 275(5297):213–215, 1997.
  * [34] E. Oja.  A simplified neuron model as a principal component analyzer.  Journal of Mathematical Biology, 15:267–273, 1982.
  * [35] L. I. Perlovsky.  Toward physics of the mind: Concepts, emotions, consciousness, and symbols.  Physics of Life Reviews, 3(1):23–55, 2006.
  * [36] M. Platek, J.P. Keenan, and T. K. Shackelford.  Evolutionary Cognitive Neuroscience.  MIT Press, 2007.
  * [37] R. Quian Quiroga.  Concept cells: the building blocks of declarative memory functions.  Nature Reviews Neuroscience, 13(8):587–597, 2012.
  * [38] R. Quian Quiroga, L. Reddy, G. Kreiman, C. Koch, and I. Fried.  Invariant visual representation by single neurons in the human brain.  Nature, 435(7045):1102–1107, 2005.
  * [39] M.W. Reimann, M. Nolte, M. Scolamiero, K. Turner, R. Perin, G. Chindemi, P. Dlotko, R. Levi, K. Hess, and H. Markram.  Cliques of neurons bound into cavities provide a missing link between structure and function.  Front. Comput. Neurosci., 11:48, 2017.
  * [40] A. Roberts, D. Conte, M. Hull, R. Merrison-Hort, A. K. al Azad, E. Buhl, R. Borisyuk, and S.R. Soffe.  Can simple rules control development of a pioneer vertebrate neuronal network generating behavior?  Journal of Neuroscience, 34(2):608–621, 2014.
  * [41] F. Rosenblatt.  Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms.  Spartan Books, 1962.
  * [42] C.C. Sherwood, A.L. Bauernfeind, S. Bianchi, M.A. Raghanti, and P.R. Hof.  Human brain evolution writ large and small.  Prog Brain Res, 195:237–254, 2012.
  * [43] A.M. Sousa, K.A. Meyer, G. Santpere, F.O. Gulden, and N. Sestan.  Evolution of the human nervous system function, structure, and development.  Cell, 170(2):226–247, 2017.
  * [44] I.Y. Tyukin, A. N. Gorban, K. Sofeikov, and I. Romanenko.  Knowledge transfer between artificial intelligence systems.  arXiv preprint arXiv:1709.01547, 2017.
  * [45] V. Vapnik and O. Chapelle.  Bounds on error expectation for support vector machines.  Neural Computation, 12(9):2013–2036, 2000.
  * [46] J.A. Villacorta-Atienza, C. Calvo, S. Lobov, and V.A. Makarov.  Limb movement in dynamic situations based on generalized cognitive maps.  Mathematical Modelling of Natural Phenomena, 12(4):15–29, 2017\.
  * [47] J.A. Villacorta-Atienza, C. Calvo, and V.A. Makarov.  Prediction-for-compaction: Navigation in social environments using generalized cognitive maps.  Biological Cybernetics, 109(3):307–320, 2015.
  * [48] J.A. Villacorta-Atienza and V.A. Makarov.  Neural network architecture for cognitive navigation in dynamic environments.  IEEE Transactions on Neural Networks and Learning Systems, 24(12):2075–2087, 2013.
  * [49] I.V. Viskontas, R. Quian Quiroga, and I. Fried.  Human medial temporal lobe neurons respond preferentially to personally relevant images.  Proc. Nat. Acad. Sci., 106(50):21329–21334, 2009.
  * [50] L. Wittner, D.A. Henze, L. Zaborszky, and G. Buzsaki.  Three-dimensional reconstruction of the axon arbor of a ca3 pyramidal cell recorded and filled in vivo.  Brain Struct Funct, 212(1):75–83, 2007.