# On Organizational Principles of Neural Systems

Xin Li1
University at Albany
<EMAIL_ADDRESS>This work was partially supported by NSF IIS-2401748 and
BCS-2401398.

###### Abstract

How do we understand natural intelligence? Inspired by classical embodied
cognition and the emerging multimodal interaction, we study the organizational
principles of neural systems at three levels (device/implementation,
circuit/algorithm, and system/computational) in this survey paper. Our main
contributions consist of 1) Device/Implementation level: reproducibility of
macroscopic states in polychronization neural group (PNG) serves as the
physical basis of associative memory; 2) Circuit/Algorithm level: canonical
microcircuits implement a universal predictive coding algorithm underlying all
high-level cognitive functions; 3) System/Computational level: modeling
sensorimotor interaction for embodied cognition plays a fundamental role in
understanding natural intelligence. At each level, we use mathematical models
as our abstractions and study their organizational principles (e.g., entropy
reduction, predictive coding, and coordinate transformation). The unifying
theme is that natural intelligence evolves by recycling a simple navigation
trick in different coordinates (reference frames) hundreds of thousands of
times. By representing space as a latent sequence, the neocortex solves the
embodied cognition problem by building parallel cognitive maps at a large
scale. We hope this survey article can inspire new research at the
intersection of neuroscience and learning systems, helping bridge the gap
between natural and artificial intelligence.

Keywords: neural systems, polychronization neural group (PNG), predictive
coding, sensorimotor interaction, embodied cognition, Tolman-Eichenbaum
Machine (TEM)

##  I Introduction

How do we understand natural intelligence? If we use evolutionary theory as
guidance, reaching the pinnacle of natural intelligence takes several phases -
the human brain. The human brain contains approximately 100 billion neurons,
each forming connections with thousands of other neurons, resulting in
trillions of synapses. With only power of 20 watts, the human brain can
achieve artificial general intelligence, including spatial navigation, object
recognition, and various high-level cognitive functions such as language and
abstract thoughts. The gap between natural and artificial intelligence is
still significant. As Carver Mead positioned in [1] (pp. 8), “Our capability
to realize simple neural functions is strictly limited by our understanding of
their organizational principles, and not by difficulties in implementation.”
What do we know about the organizational principles of neural systems?

In 1949, Donald Hebb [2] formulated his famous “fire together, wire together”
postulate. Hebbian learning has since become a dogma in neuroscience, though
it only covers the organizational principle at the most basic level. The rich
interaction between excitatory and inhibitory neurons can lead to limit cycle
oscillation [3] \- a periodic behavior closely related to the phenomenon of
hysteresis. The implementation of Hebbian learning (e.g., via spike time-
dependent plasticity (STDP) [4, 5]) was discovered much later in the late
1990s. Polychronization neural group (PNG) [6] is a more recently developed
theory at the neuronal level. It emphasizes the role of STDP in the time-
locking mechanism and the vast diversity of firing patterns, contributing to
the neocortex’s unprecedented memory capacity.

In 1957, VB Mountcastle [7] proposed an astonishing hypothesis about the
principle of column organization of the neocortex. Despite apparent functional
differences among cortical areas, Mountcastle has discovered that all cortical
columns anatomically share a similar six-layered structure, which can be
modeled by a canonical microcircuit for predictive coding [8]. Modeling
cortical columns and their interactions has revealed the uniformity and
concrete layered structure of the neocortex of mammals[9], in support of
Mountcastle’s hypothesis. However, connecting the structure of column
organization with specific functions is not fruitful [10].

From the late 1960s to the early 1970s, David Marr published a theory of
cerebellar cortex [11], a theory of cerebral neocortex [12], and a theory of
Archicortex [13]. His inquiry into how the brain works through the human
vision system ended abruptly with his death in 1980. The Tri-Level Hypothesis
was published in his postmortem masterpiece, which laid the foundation for
modern computational vision [14]. In 1989, Damasio’s consciousness theory [15]
suggests that consciousness arises from emotions and feelings rooted in the
body’s physiological states. He introduced the concept of convergence zones to
characterize the time-locking mechanism underlying neural populations. In
predictive coding [16] and the closely related free-energy principle [17], the
critical principle is that the brain is constantly generating and updating a
model of the world to predict sensory input. This theory is based on the idea
that the brain is proactive rather than reactive, constantly trying to infer
the causes of the sensory information it receives. Although these theories
have offered competing interpretations of how the brain works at the system
level, their connections with lower levels remain elusive.

In 2004, Jeff Hawkins published his book on intelligence [18], which re-
advocated Vernon Moutcastle’s pursuit of a universal cortical processing
algorithm. His ideas were more recently developed into a more comprehensive
thousand brain theory [19], emphasizing the importance of reference frames and
voting mechanisms among cortical columns. It was conjectured that each
cortical column generated a reference frame or a map that is essentially a
coordinate system used by the brain to represent the location of objects.
Despite the intellectual appeal of the thousand brain theory, it lacks a
mathematical formulation or scientific rigor. Given the lack of a complete
theory, NeuroAI [20] has emerged as a new paradigm to catalyze the revolution
of using AI as a proxy model for neuroscience research. Nevertheless, the
neuroscience-based approach toward achieving artificial general intelligence
(AGI) has not caught as much attention as the CS-based approach [21] (e.g.,
ChatGPT by OpenAI).

The motivation behind this paper is mainly two-fold. On the one hand,
neuroscience needs $evolution$ [22]. This is because the neural system is a
product of evolution due to the irreversibility of time. Evolutionary design
[23] is the only plausible logic we can use to understand the organization and
functions of the brain. For example, the real mysteries of visual perception
[24], such as the presence of various defects (e.g., blind spot and inverted
design), can best be solved by tracking the evolutionary path of mammalian
vision systems. By treating the world as an outside memory [25], we can gain a
new perspective toward revealing the brain’s mechanism. On the other hand,
neuroscience needs $behavior$ [26]. If evolutionary pressure is the original
drive, the flexibility of animal behaviors is the direct evidence for the
evolution of intelligence [20]. Goal-oriented behaviors supersede other
cognitive functions (e.g., vision and language) because intelligence must have
goals and motivations [19]. Instead of pursuing a representation of the
external world, we argue that an action-oriented representation is more
appropriate for modeling natural intelligence.

Based on the above observations, we study the organizational principles of
neural systems at three levels (device/implementation, circuit/algorithm, and
system/computational) following Marr’s tri-level framework in this paper. The
unifying theme is that intelligent agents evolve by adapting their behavior to
the physical environment and the ever-increasingly complex behavior in social
groups. Our pursuit of action-oriented representation is based on the marriage
of dynamical system and predictive coding frameworks, which arguably
implements Vernon Moutcastle’s universal processing algorithm. However, these
two frameworks operate at different scales (“more is different” [27]) -
dynamical system theory is needed for the characterization of single neurons,
leading to the physical basis of associative memory [28]. At the cortical
column, Kalman filtering-based prediction (a.k.a. simultaneous location and
mapping (SLAM) in robotics [29]) offers a computational abstraction of
canonical microcircuits [30, 8]. Combining these two lines of ideas leads to a
powerful memory-based prediction framework, which has already been
successfully applied to the construction of cognitive maps [31] (e.g., Tolman-
Eichenbaum Machine (TEM) [32]). By considering space as a latent sequence
[33], we show that action-oriented representation can lead to a unified
interpretation of embodied cognition beyond spatial navigation. For example,
we have picked the vision and language models to demonstrate how they can be
understood as spatial navigation in the latent space defined by coordinate
transformations [34, 35, 36]. Here is a summary of key organizational
principles at different levels (refer to Fig. 1):

  * •

Device/Implementation level: reproducibility and scalability of macroscopic
states in polychronization neural groups (PNG) serve as the physical basis of
associative memory. STDP-based time-locking promotes the reproducibility of
macroscopic states by reducing Kolmogorov-Sinai (KS)-entropy. Polychronization
is scalable temporally in that the time-locking of multiple PNGs can occur
similarly to that of individual neurons.

  * •

Circuit/Algorithm level: predictive coding is the universal processing
cortical algorithm for modeling elementary sensorimotor interaction. Canonical
microcircuits of predictive processing can be interpreted as the
generalization of associative memory to cortical columns. Sensorimotor
interaction at the cortical column level admits an abstraction by the Kalman
filtering-based simultaneous location and mapping (SLAM) model.

  * •

System/Computational level: The flexibility of behavior is the foundation of
intelligent systems. By attaching to sensory and motor systems, the neocortex
generates adaptive behaviors by minimizing prediction errors in a parsimonious
and self-consistent manner. Using coordinate transformations (e.g., egocentric
to allocentric), action-oriented representations can solve various high-level
cognition problems by navigating and associative memory recall in the latent
space.

Figure 1: Organizational principles of neural systems at three levels
(computational, algorithm, and implementation) and the logic behind the
evolution of natural intelligence (embodied cognition: memory-based prediction
coding is combined with various coordinate transformations).

The rest of this paper is organized as follows. In Sec. II, we first lay out
the theoretical framework to discuss the high-level ideas about organizational
principles across different levels (i.e., “more is different”) and the
relationship between computation and representation (i.e., does intelligence
require representation?). In Sec. III, we study the organizational principles
at the lowest (device) level. Dynamical system theory is adopted to study the
polychronization of neural groups, laying the physical basis for associative
memory. In Sec. IV, we focus on the mid-level of cortical columns (analogous
to circuits). The canonical microcircuits for predictive processing are
interpreted from a spatial navigation perspective. We generalize the
simultaneous localization and mapping (SLAM) model for column organization and
briefly discuss the non-column organizational principle underlying the
hippocampus and entorhinal cortex. In Sec. V, we move to the highest (system)
level and present a unified framework for understanding cognitive functions
based on modeling sensorimotor interactions. We also highlight the connections
with the existing embodied cognition theory. In Sec. VI, we apply our
organizing principles to three cognitive tasks: spatial navigation, object
recognition, and language comprehension. A toy model named the Tolman-
Eichenbaum Machine (TEM) was adopted to facilitate the unification across
different tasks.

##  II Theoretical Framework

###  II-A Emergent Property: More Is Different

The reductionist hypothesis has been well accepted by many scientists (e.g.,
understanding matter’s behavior in terms of atomic and subatomic particles in
condensed matter physics). However, its limitation has also been well
documented in “more is different” [27] where an alternative constructionist
hypothesis is developed. As Anderson advocated, the hierarchy of science
ranges from particle physics at the most fundamental level through many
layers, including atomic, molecular, and cell biology, up to psychology and
social sciences at the highest levels. An important observation with such a
hierarchy is the concept of $emergence$ [37] (a.k.a. collective behavior
[28]), where new properties and behaviors arise at each level of complexity
that are not predictable from the laws governing the lower levels. For
example, the collective behavior of electrons in solids gives rise to
phenomena like superconductivity and ferromagnetism, which the physics of
individual electrons cannot fully explain.

A similar observation with emergence can be made in neural systems. The
fundamental laws guiding the behavior of single neurons (e.g., Hodgkin–Huxley
model [38]) have been well known. However, a group of excitatory and
inhibitory neurons can demonstrate much more sophisticated behavior such as
multiple hysteresis and limit cycle activities [3]. From neural groups to
cortical columns, a new emergent property arises from the layered organization
and rich interaction between sensory and motor neurons. Modeling cortical
columns by canonical microcircuits gives rise to new emergent
characterizations, such as predictive processing capabilities [30]. One level
higher, the fruitful interaction between the sensory and motor cortex
consisting of hundreds or thousands of cortical columns serves as the physical
basis of flexibility or adaptive behavior (the foundation of intelligence)
[19]. From single neuron and neural groups to cortical columns and areas,
these emergent properties require new principles and explanations [37].

The complexity and uniqueness of phenomena at different scales of organization
become even more profound for neural systems due to their interaction with the
physical environment and social groups. Mobility and intelligence are closely
related. Actions taken by an animal are inherently constrained by motion in
the physical world. During the evolution of intelligent agents, the
flexibility of behavior is directly related to the physical motion of limbs,
head, and eyes. From insect-level [25] to reptile-level intelligence, the
vertebral column (spine) makes a big difference in adaptive behavior; from
mammals to primates and humans, freeing hands from walking and for object
manipulation is another major milestone with profound implications on the
evolution of human vision systems [39]. Not to mention the origin of natural
languages [40] \- in addition to design features and uniqueness of human
speech, social and cultural implications of natural languages are another
coercive evidence for the emergent property of hierarchical organization.

###  II-B Computation and Representation

In David Marr’s tri-level framework for computational vision [14], he laid out
three levels: computational theory, representation/algorithm, and hardware
implementation. The first and last levels are less controversial: the goal of
computation is determined by the task such as the functional specification of
different cortical areas in the human brain or software applications in
computer systems; the hardware implementation can be analog (e.g., based on
neural computation [41]) or digital (e.g., based on Von Neumann architecture).
The primary controversy concerns the mid-level of representation: there is a
long open debate between anti-representationalism [42] and representationalism
[43]. In conventional cognitive science, cognition is often thought of as
involving internal representations [44] — mental symbols or constructs that
represent the external world inside the mind. Anti-representationalism, as
pioneered by Brooks [25], questions or rejects the necessity or usefulness of
this concept in explaining cognitive processes. The influential embodied
cognition paradigm argues for using the world as its own model [25] or an
outside memory [24].

Our stance is to pursue the middle ground between these two extreme views.
Inspired by the ecological approach to vision by James J. Gibson [45], we
advocate an embodied, dynamic, and action-oriented representation with
ecological rationality. The new insight brought about by our approach can be
summarized into the following three theses:

Thesis 1 (Localized embodied representation): Internal representations,
regardless of their existence, are deeply influenced and shaped by localized
bodily interactions with the environment. The locality constraint is due to
local receptive fields of the sensory organs of an agent.

Thesis 2 (Dynamic stabilized representation): Internal representations,
instead of being static, are continuously updated through interaction with the
external world. Dynamic representation maintains its stability by positive and
negative feedback loops.

Thesis 3 (Action-oriented representation): Internal representations serve the
purpose of guiding action, rather than merely modeling the world. In other
words, internal representations are seen as tools for interaction with the
environment, integrating the idea of embodiment with the concept of dynamic
representation.

We note that the above three theses are consistent with an emerging paradigm
of NeuroAI [20] at the interaction of AI and neuroscience. Moravec has argued
it [46] that modeling sensorimotor interaction is at a more fundamental level
than modeling abstract thoughts, including language. From the evolution
perspective, nature has discovered a clever trick of compressing the
allocortex (the old brain) into a cortical column. Then, this trick was
recycled by evolution to achieve higher intelligence from reptile brains to
mammalian brains and from primate brains to human brains. As advocated by a
thousand brain theory [19], each cortical column is characterized by a
reference frame with minimum sensory input and motor output requirement.
Therefore, modeling sensorimotor interaction in a minimal form should focus on
the canonical microcircuits responsible for motor control [47] and cognitive
maps [31] for spatial navigation. Through coordinate transformations among
cortical areas [34, 35, 36], other cognitive functions such as object
recognition and abstract thoughts can also be implemented as spatial
navigation in parallel generalized cognitive maps.

Our envisioned approach to artificial general intelligence (AGI) is also fully
aligned with the developing paradigm in robotics research called Agent AI
[48]. The key idea behind Agent AI consists of two components: 1) an
intelligent agent collects the state-action pairs from the physical world as
expert demonstration training data; 2) an agent generator/simulation imitates
the expert’s behavior in the virtual environment. Unlike conventional
reinforcement or imitation learning, a fundamentally new insight by Agent AI
is the decoupling strategy (separation of the learning process from task-
specific reward function), which allows the policy to generalize across
different tasks without explicit dependency on a task-specific reward
function. Such decoupling-based generalization can explain how emergent
properties or sophisticated behaviors can arise from simpler components or
rules (e.g., the synchronized firing of fireflies and the swimming pattern
generated by a school of fish [49]). From this perspective, human language can
also be interpreted as emergent behavior because language emerges from
individual interactions within a community over time. Similarly, object
recognition in vision also exemplifies emergent behavior, as complex
capabilities for recognizing and interpreting visual stimuli emerge from the
interactions of simpler processing units within biological or artificial
systems. Multi-stable visual perception [50] and human behavior [51] represent
a concrete example of broken symmetry [37] when emergence occurs. Building
upon this new insight, we advocate implementing decoupling by coordinating
transformations among cortical areas.

##  III Device Level: Polychornization Neural Groups

In this section, we formulate the phenomenon of polychronization among neural
groups under an ergodic theoretic framework and introduce the concept of
motion-induced transformation as an implicit characterization of neurodynamic
systems at the lowest level. The reproducibility and scalability of
macroscopic states in polychronization neural groups (PNG) serve as the
physical basis of associative memory.

###  III-A Ergodic Theory of Polychronization

Toy Model of Neurodynamic System. Consider a bipartite graph with two sets of
nodes (refer to Fig. 2a): sensory/motor neurons
$\textbf{X}=\\{X_{1},...,X_{n}\\}$ and memory neurons
$\textbf{Y}=\\{Y_{1},...,Y_{m}\\}$ and the edges connecting them have the
matrix of propagation delay $\textbf{D}=[d_{ij}],1\leq i\leq n,1\leq j\leq m$
representing the propagation delay from $X_{i}$ to $Y_{j}$. Suppose the firing
time for sensory/motor neurons is denoted by $\\{t_{1},t_{2},...,t_{n}\\}$,
then the corresponding time for the $i$-th spike to reach the memory neuron
$Y_{j}$ can be expressed by $T_{ij}=t_{i}+d_{ij}$ where $1\leq i\leq n,1\leq
j\leq m$. If all $n$ spikes arriving at $Y_{j}$ within an internal of $\delta$
\- i.e., $\Delta=max_{i\neq k}|T_{ij}-T_{kj}|<\delta$, then $Y_{j}$ is time-
locked. For better biological plausibility, we also consider extending the
above model by replacing a single layer memory neurons with a complete graph
(refer to Fig. 2b). That is, one can introduce intragroup delays $D_{ij},1\leq
i,j\leq m$ among $\textbf{Y}=\\{Y_{1},...,Y_{m}\\}$ to enrich the interaction
among the group of memory neurons.

a) b)

Figure 2: a) A bipartite graph of sensorimotor-memory neurons ($d_{ij}$ is the
latency between $X_{i}$ and $Y_{j}$); b) a complete graph abstracting the set
of extended memory neurons ($D_{ij}$ is the latency between $Y_{i}$ and
$Y_{j}$).

An important intuition behind our combinatorics formulation is based on Ramsey
theory [52] that states that a certain degree of order must occur as the scale
of the object becomes large. In neural systems, the degrees of order is
characterized by the synchronicity among neurons; the object’s scale
translates to the astronomical number of neurons and synaptic connections. If
we use different constellations in the night sky as a metaphor for the pattern
associated with PNG, Ramsey theory implies that any pattern of
polychronization can be found if the structure of PNG is rich enough. Along
this line of reasoning, we formally have the following result for the
bipartite model of sensorimotor-memory interaction.

Lemma 1 (Arithmetic progression lemma)

Suppose both inter-group and intra-group delay matrix consists of positive
constant $d_{ij}=D_{ij}=\alpha>0$. For periodic input patterns with a period
of $T$ and offsets of an arithmetic progression
$\textbf{X}=\\{t_{1},t_{2},...,t_{n}\\}$ with the interval of
$\Delta=t_{k+1}-t_{k}$, we can always find a time-locked solution in the
extended memory neuron group as $m\rightarrow\infty$.

$Proof$. Using timing-to-phase mapping (e.g., threshold phasor associative
memory (TPAM) model [53]), we map the period of $T$ in neuronal oscillation
[54] to a unit circle (refer to Fig. 3). Then Weil’s Equidistribution Theorem
[55] guarantees the uniform distribution of arithmetic progression sequence on
the unit circle, implying the richness of structure. For an arbitrary
$\Delta$, we can find an integer value $k$ such that
$k\alpha=-\Delta~{}mod~{}T$. ∎

Similar to the constellation theorem in ergodic theory [56], we want to relax
the arithmetic progression constraint and observe the time-locking result for
an arbitrary set of offsets $\textbf{X}=\\{t_{1},t_{2},...,t_{n}\\}$. It turns
out that this is not difficult thanks to the richness of structure assured by
Weil’s Equidistribution Theorem. Formally, we have

Theorem 1 (Time-Locking for Arbitrary Offsets)

Suppose both inter-group and intra-group delay matrix consists of positive
constant $d_{ij}=D_{ij}=\alpha>0$. For periodic input patterns with a period
of $T$ and arbitrary set of offsets $\textbf{X}=\\{t_{1},t_{2},...,t_{n}\\}$,
we can always find a time-locked solution in the extended memory neuron group
as $m\rightarrow\infty$.

$Proof$. Since the arithmetic progression sequence forms the uniform
distribution on the unit circle, we can find an integer value $l$ such that
$l\alpha=t_{k}-t_{k+1}~{}mod~{}T$ for each offset $t_{k},1\leq k\leq n$.
Repeating this strategy for $1\leq k\leq n$ would identify the desirable PNG.
∎

Phase Space Characterization. Next, we consider the generalization of the
time-locking result from the periodic input of a single spike (only one offset
considered) to a spike train without periodicity constraint (multiple offsets
considered). We assume that the spike train fired by a neuron can be
represented by $x(t)=\sum_{i}\delta(t-t_{i})$ where $0\leq t_{i}\leq T$. Note
that the bounded value $T\approx 100ms$ reflects the physical constraint of
latency [57]. To make our analysis more tractable, a timing-to-phase mapping
(refer to Fig. 3) can be obtained by applying Fourier transform to the spike
train- i.e., $X(w)=\mathcal{F}[x(t)]=\sum_{i}exp(-jwt_{i}),w=\frac{2\pi}{T}$.

Figure 3: Timing-to-phase mapping for phase space reconstruction of spike
trains (cited from [53]).

Here is another important intuition behind the ergodic formulation [56].
Instead of working with the spike sequences generated by individual neurons,
the focus shifts to a global view of spatio-temporal patterns associated with
a group of neurons. The phase space of a perceptual system will be
characterized by an ordered set of spike sequences $\\{x_{i}(t)\\}_{i=1}^{n}$,
where $i$ denotes the index of sensory neurons (with some assumed spatial
ordering). Similarly, we can define the phase space for a memory system by an
unordered set of spike sequences $\\{y_{j}(t)\\}_{j=1}^{m}$, where $j$ denotes
the index of memory neurons (no order is needed). Using timing-to-phase
mapping, the encoding transformation from the phase space of sensory neurons
to that of memory neurons can be represented by
$\textbf{E}:\textbf{S}^{n}\rightarrow\textbf{S}^{m}$ where S denotes the unit
circle and $\textbf{S}^{k}=\textbf{S}^{1}\times...\times\textbf{S}^{1}$ is the
fundamental group of torus [58].

Motion-Induced Transformation. In neurodynamic systems, trajectories in the
phase space are activated by the interaction with the external environment.
For example, finger swiping of a mahjong tile can inform the brain of the
character and symbols through tactile sense; visual perception also relies on
the saccadic motion of pupils in understanding the physical scene. These
motion patterns render the firing activities on sensory/motor neuron group X,
which are then mapped to the space of memory neurons Y by E. When the motion
patterns change from $t$ to $t+\delta t$, the induced changes on X can be
characterized by a group of transformation
$\textbf{T}:\textbf{S}^{n}\rightarrow\textbf{S}^{n}$. To facilitate our
analysis, we assume that the physical motion is translational which can be
satisfied by making the time interval $\delta t$ infinitesimally small.
Consequently, the trajectory or orbit of motion-induced transformation in the
phase space of sensory input can be represented by a sequence
$\\{\textbf{T}^{1},...,\textbf{T}^{k}\\}$; the corresponding orbit in the
phase space of memory output is denoted by
$\\{\textbf{E}\circ\textbf{T}^{1},...,\textbf{E}\circ\textbf{T}^{k}\\}$.

In ergodic theory, rotation on a unit circle is a measure-preserving
transformation. It follows that the above-defined motion-induced
transformation also satisfies the measure-preserving property. Therefore, we
can have the following recurrence result.

Lemma 2 (Corollary of Poincare’s Recurrence Theorem)

For any $\bm{x}\in\textbf{X}$ and a motion-induced transformation T, there
exists a positive integer $n$ such that $\textbf{T}^{n}(\bm{x})$ is
arbitrarily close to $\bm{x}$.

The physical implication of the above result is that if one keeps finger-
swiping the same mahjong tile, the rendered firing activities by tactile
receptors might fluctuate and wander in the phase space, but will eventually
return to the vicinity of the starting position. However, a biologically more
important question is how to detect the change on the tile surface. In other
words, how are the encoded firing activities in memory neurons $\ tY$’s phase
space organized into different orbits? To answer this question, we need to
borrow a property with a random matrix - i.e., the matrix entries are assumed
to be statistically independent. With this assumption, we show that the
probability of two orbits intersecting in memory neurons’ phase space is
infinitesimally small.

Lemma 3 (Non-overlapping orbits with random delay matrix)

For a delay matrix whose entries are perturbed by i.i.d. uniform noise, i.e.,
$d_{ij}=D_{ij}=\alpha+\epsilon$, where $\epsilon\sim U[0,T]$, the probability
of inducing an intercepting orbit is negligible.

The proof is straightforward at least for the special case when the threshold
for time-locking $\delta=0$. The physical intuition here is that different
sensory stimuli will be encoded in different orbits in the phase space of
memory neurons. The invariance of sensory stimuli is largely determined by the
group of motion-induced transformation
$\\{\textbf{T}^{1},...,\textbf{T}^{k}\\}$ thanks to the extension of
Poincare’s Recurrence Theorem to Multiple Birkhoif Recurrence Theorem as
follows.

Lemma 4 (Corollary of Multiple Recurrence Theorem)

If X is compact metric and the group of motion-induced transformations
$\\{\textbf{T}^{1},...,\textbf{T}^{k}\\}$ Of X into X commute, then there
exists a multiply recurrent point.

In ergodic theory, the above commuting Multiple Recurrence Theorem is the
constellation theorem which states that inside certain rich structures,
certain patterns will be inevitably found. Its biological implication is that
when $m\rightarrow\infty$ (e.g., from single-layer to multi-layer), the
structure of PNG in Y will be rich enough to accommodate any input from the
sensory neuron X. Infinitely many input patterns in X will be self-organized
into non-intersecting orbits in Y due to polychronization.

###  III-B Entropy and Reproducibility of Microscopic States in PNG

Entropy and Free Energy. In parallel to the second law of thermodynamics
(entropy of isolated systems left to spontaneous evolution cannot decrease),
we observe that the phenomenon of polychronization tends to decrease the
entropy in neurodynamic systems. This observation is consistent with the
predictive coding theory [16] and free-energy principle [17] in neuroscience.
To formalize our intuition, we need to have a rigorous definition of entropy.

Definition of Kolmogorov-Sinai (KS) Entropy [59]

Kolmogorov-Sinai (KS) Entropy (a.k.a. metric entropy) measures the complexity
of a dynamical system, quantifying the rate at which information about the
system’s state is generated over time. If the phase space X is partitioned
into $\textbf{X}_{1}\cup...\cup\textbf{X}_{K}$, then the KS-entropy is defined
by $H(\textbf{X})=-\sum_{k=1}^{K}P(\textbf{X}_{k})log(P(\textbf{X}_{k}))$.

Unlike entropy production in dissipative systems [60], entropy tends to
decrease over time in collective neurodynamic systems, leading to better
predicting future states based on past states. This is the essence of short-
term and long-term memory in neural systems due to forming PNGs. Formally, we
have

Theorem 2 (Polychronization Reduces KS Entropy)

The formation of polychronization neural groups monotonically decreases the
KS-entropy of spike trains.

Sketch of Proof. For a group of asynchronous neurons, their spike sequences
will be uniformly mapped to the cells (tiled orbits) in the phase space,
corresponding to the case of maximum entropy. After polychronization, Lemma 3
implies that time-locked spike sequences will be mapped to the same cell in
the phase space, leading to a nonuniform distribution with lower entropy.

Figure 4: Energy Landscape of a Hopfield Network, highlighting the current
state of the network (up the hill), an attractor state to which it will
eventually converge, a minimum energy level and a basin of attraction shaded
in green. Note how the update of the Hopfield Network is always going down in
energy.

The above theorem is closely related to energy-based models (e.g., Hopfield
network [28] for associative memory and energy-based learning [61]). In
thermodynamics, the evolution from a non-equilibrium state to a steady state
(local minimum of energy landscape) is the basis of reproducible dynamical
structures [60]. A similar observation has been made for the associative
recall of neural systems - e.g., in Hopfield network [28], each steady state
corresponding to a local minimum of a Lyapunov function is associated with a
stored memory pattern (refer to Fig. 4). Our principle of entropy reduction is
consistent with the principle of free-energy minimization because both reflect
the idea that minimizing surprise (arising from sensory exchanges with the
physical world) enables a neural system to resist a natural tendency to
disorder. The reproducibility of these microscopic states distinguishes
physical systems with memory from those without memory.

Reproducibility of Macroscopic States. Following ET Jaynes’s intuition [62],
entropy is a measure of experimental reproducibility. In non-equilibrium
thermodynamics [60], energy dissipation and entropy production extremal
principles claim that many microscopic states are compatible with a single
macroscopic state, but only the latter can be reproduced. Nature has
discovered that the fluctuation arising from the uncertainty in the external
environment only requires less than a practically infinite number of neurons
to characterize. Depending on the complexity of the external environment, an
organism can learn an embodied internal representation [44] to reproduce
macroscopic states. The reproducibility of macroscopic states is at the
foundation of learning and memory. As Heraclitus says, “no man ever steps in
the same river twice”; but how does a man know that he steps in the river in
the first place?

The above question can be modeled as the perturbation of the sensory input by
motion-induced transformation. Given the trajectory in the phase space of
sensory/motor neurons as induced by $\\{\textbf{T}^{1},...,\textbf{T}^{k}\\}$,
how can we guarantee the invariance of the encoded outcome in the phase space
of memory neurons? Our reasoning consists of two steps. First, we show how to
achieve group invariance (GI) to order
$\\{\textbf{T}^{1},...,\textbf{T}^{k}\\}$, confirming the commuting property
of motion-induced transformations. Then we consider the perturbation
introduced to the polychronization process, leading to the bifurcation
phenomenon related to multistable perception [50].

Theorem 3 (GI of Motion-Induced Transformations)

Suppose $\\{i(1),...,i(k)\\}$ is a permutation of $\\{1,...,k\\}$, the encoded
outcome in the phase space of memory neurons induced by
$\\{\textbf{T}^{i(1)},...,\textbf{T}^{i(k)}\\}$ is the same as that by
$\\{\textbf{T}^{1},...,\textbf{T}^{k}\\}$.

$Proof$. Based on the toy model shown in Fig. 2, we observe that if we rewire
the delay matrix $D_{jk}$ into $D_{i(j)i(k)}$, the same process of
polychronization can be associated with the permuted neuron group.

Now, we consider the solution space of the polychronization process and ask:
do the orbits of different PNGs intersect? can we reverse the orbit (i.e.,
does $\textbf{T}^{i}\rightarrow\textbf{T}^{j}$ imply
$\textbf{T}^{j}\rightarrow\textbf{T}^{i}$)? is the polychronization result
unique for any given X? The answers are negative. We summarize our findings
into the following theorem in the phase space.

Theorem 4 (Irreversibility and Bifurcation Theorem)

Motion-induced transformations generate orbits in the phase space:
$\textbf{T}^{1}\rightarrow...\rightarrow\textbf{T}^{k}$. These orbits cannot
be reversed and generally do not interact with each other. Under special
conditions, the same initial state can end up with two different steady
states, a phenomenon called bifurcation.

Note that the time irreversibility of polychronization is based on the
fundamental law of physics (e.g., transmission delays are always positive).
The question of intersection between different orbits is partially answered by
Lemma 3 for a random delay matrix. If the delay matrix is not random, we might
observe that some small perturbation of the initial trajectory can lead to
dramatically different outcomes (conceptually similar to the well-known
butterfly effect [63]). For the celebrated Kuramoto model [64], a bimodal
steady state was studied in [65], where two counter-rotating groups of phase-
locked oscillators emerge. This bifurcation phenomenon contributes to the
neural basis of multistable perception [66].

Temporal Scalability of PNGs. The hierarchical organization of neural systems
in terms of timescale has been documented in the literature [67, 68]. In [69],
hierarchical organization of cell assemblies were regarded as a neural syntax
(analogous to the syntax in natural languages); and constituents of the neural
syntax are organized by dynamically changing constellations of synaptic
weights (called “synapsembles”). Temporally scalable extension of PNGs
(constellations) formation includes two key ideas [70]: 1) sequential recall -
upon future presentation of any stimulus in the sequence, the corresponding
PNGs and its subsequent ones will be activated sequentially until the end of
the sequence; 2) scaffolded recall - if the stimulus sequence is presented to
two brain areas simultaneously, a scaffolded representation (e.g., Helen
Keller’s connecting perceptual sensation of water with the tactile abstraction
of the word [71]) resulting in more efficient memorization and recall.

This section summarizes the analogy between thermodynamics and neurodynamics
in Table I and the following proposition. In parallel to time and ensemble
average in thermodynamics, we have the group of transformations induced by
motion and the corresponding PNGs generated by the same stimulus class.
Macroscopic properties of neurodynamic systems, including sensory stimulus
classes and motion-induced transformations, are reproducible from the observed
PNGs. We conjecture that the reproducibility of macroscopic states achieved by
PNGs serves as the physical basis of neural memory. Following the Hebbian
learning postulate, we can see how the time-locking mechanism [15] shapes the
connections among neurons (e.g., feedforward, feedback, and lateral) into
functionally distinct processing streams, constituting the anatomical
structures of cortical columns. This circuitry of the neocortex brings us to
the next level of understanding the organizational principles of neural
systems.

Proposition 1 (Principle of Polychronization):

Polychronous neural groups have higher reproducibility than asynchronous
groups. Self-organization of neural systems is to monotonically increase
reproducibility or predictability (equivalent to entropy reduction). The
number of reproducible microscopic states of neural groups determines the
memory capacity of neural systems.

description | thermodynamics | neurodynamics
---|---|---
interacting unit | particles | neurons
phase space | positions and momentum | firing patterns
order parameter | pressure and temperature | stimulus and motion
system dynamics | N-body problem | motion-induced transformation
steady states | attractors | PNGs
order quantification | Gibbs entropy | metric/KS-entropy
evolutionary law | entropy non-decreasing | entropy non-increasing

TABLE I: Analogy between thermodynamic and neurodynamic systems.

##  IV Circuit Level: Modeling Cortical Columns

Polychronization neural group-based memory provides the framework from which
the brain generates predictions about the world, according to the principles
of predictive coding [18]. Predictive coding [16] theorizes that the brain
continuously makes predictions about incoming sensory information based on
experiences (i.e., memory). Polychronization in microcircuits can represent
and update these predictions over time [30]. By having neurons fire in
specific temporal patterns, the brain can encode and update predictions about
future sensory events [72].

An important new insight this work brings is a $minimalist$ view toward
predictive coding - i.e., from an evolutionary perspective, what will be the
most primitive function for cortical columns to perform? We argue that it
solves the simultaneous localization and mapping (SLAM) problem for spatial
navigation [29]. Nature has discovered an elegant solution to compress the
neural machinery for cognitive maps to a minimal form (i.e., canonical
microcircuits [8]) and recycle it to achieve higher intelligence by
generalizing the concept map into the latent space through evolution. In this
section, we first briefly review the biological findings with canonical
microcircuits for predictive coding and then derive a computational
abstraction of canonical microcircuits to implement SLAM and its latent space
generalization.

a) b)

Figure 5: Computational abstraction of the sensorimotor interaction within a
cortical column. a) a cortical column is characterized as an individual
sensorimotor machine [19] (explaining why it is a structure without a function
[10]); b) the abstraction of cortical columns by a SLAM model [29], which
implements predictive coding for associative memory [73].

###  IV-A Canonical Microcircuits for Predictive Coding

Predictive coding is an influential theory in computational neuroscience with
rich connections with the Bayesian brain [74] and variational inference [17].
The fundamental idea behind predictive coding is that the brain uses an
internal model/representation to predict sensory input based on movements and
past sensory experience [30]. This idea can be implemented by treating the
neocortex as an architecture trained in an unsupervised setting to perform
inference and learning simultaneously to incoming sensory signals [75]. A key
advantage of predictive coding in terms of computational efficiency is that
only bipolar (positive and negative) prediction errors are transmitted up the
hierarchy. It follows that the embodied internal representation can be
modified without bottom-up input, which allows for simulating and predicting
the environment (the offline mode of human thoughts).

Several canonical microcircuits such as Douglas-Martin model [76] and
Haeusler-Maass model [77] have been studied for predictive coding in the
literature. These models have revealed a remarkable correspondence between the
microcircuitry of the cortical column and the connectivity required by
predictive coding. In Douglas-Martin model [76], information flow through the
cortical column is characterized by a stereotypical pattern of fast excitation
followed by slower and longer-lasting inhibition in the cat visual system. The
three neuronal populations receive thalamic drive and amplify transit thalamic
inputs to generate sustained activities while maintaining a balance between
excitation and inhibition. This model was further extended by Haeusler and
Maass in [77], which closely resembles the canonical circuits required by
predictive coding. The feedforward prediction errors from a lower cortical
level arrive at granular layers and are passed to excitatory and inhibitory
interneurons in supragranular layers, encoding expectations. Meanwhile, the
connections between excitatory and inhibitory neurons in supragranular layers
enable deep pyramidal cells and excitatory interneurons to generate feedback
predictions, which descend to a lower hierarchical level.

The predictive coding framework has found successful applications in both
visual perception [16] and sensorimotor interaction [78]. In visual
perception, the inverse (top-down) model passes the prediction of neural
activities about a lower level from the higher level of the visual cortex; the
forward (bottom-up) model carries the prediction residues between the
prediction and actual low-level activities at the lower level of the visual
cortex. In sensorimotor control, the inverse model characterizes the
transformation from a sensory coordinate system to a motor coordinate system;
and the forward model abstracts the transformation from a motor coordinate
system to a sensory coordinate system. Here, we take a minimalist approach to
modeling sensorimotor interaction for spatial navigation. As shown in Fig. 5,
we hypothesize that the interaction between sensory input and motor output of
cortical columns can be abstracted by a simultaneous localization and map
(SLAM) model treating observation features (landmarks) and cognitive maps
(positions) as peer hidden variables. Note that associative memory can also be
implemented by predictive coding [73]. Therefore, the SLAM model can be
interpreted as associative memory at the level of cortical columns (one level
above neural groups).

###  IV-B Toy Model: SLAM Model and Its Generalization

In spatial navigation, predictive coding aims to resolve location uncertainty.
Estimating uncertain spatial relationships is the holy grail in autonomous
vehicle research. Originated from stochastic map [79], many successful
solutions to the so-called simultaneous localization and map (SLAM) problem
[29] can be interpreted as Kalman filtering [80], which is closely related to
predictive coding. Mathematically, the SLAM problem can be solved by Kalman
filtering as follows. Assume
$\bm{x}(k)=[x_{v}^{T}(k)~{}p_{1}^{T}~{}\cdots~{}p_{N}^{T}]$ where $x_{v}$
denotes the state of the robot/vehicle and $[p_{1}^{T}~{}\cdots~{}p_{N}^{T}]$
is the state of the landmark locations. The state transition and observation
models can be written as
$\bm{x}(k+1)=\textbf{F}(k)\bm{x}(k)+\bm{u}(k+1)+\bm{v}(k+1),\bm{z}_{i}=\textbf{H}_{i}\bm{x}(k)+\textbf{w}_{i}(k)$,
where $\textbf{F}(k)$ is the state transition matrix, $\bm{u},\bm{v}$ is the
vector of control inputs and uncorrelated noise errors, respectively; and
$\textbf{H}_{i}$ is the observation matrix for the $i$-th landmark,
$\bm{z}_{i}$ is the sensor output when observing this landmark. With such an
estimation-theoretic formulation of the SLAM problem, classical Kalman
filtering can estimate vehicle and landmark locations alternatively. In
particular, one can recursively estimate the covariance
$\textbf{P}(p|q)=E[\bm{x}(p|q)\bm{x}(p|q)^{T}|\textbf{Z}^{q}]$m where
$\textbf{Z}^{q}$ represents the sequence of observations taken up to time $q$.
The algorithm alternates between the prediction of state estimates and the
update of state estimate covariances until it reaches convergence (i.e.,
establishing an internal representation). Any change in the external
environment (e.g., a novel landmark) will cause the production of large
prediction errors.

Theorem 5 (Convergence of the Covariance Matrix in the SLAM Solution) [29].

The determinant of any submatrix of the map covariance matrix P decreases
monotonically as successive observations are made.

If each cortical column can implement a simplified SLAM model, we conjecture
that a group of cortical columns can realize a more powerful computation by
generalizing the SLAM model in the following two aspects. First, the state
variable in the SLAM model can contain more information about the vehicle than
its position. For example, the pose and scale information can be also included
in the state observation (e.g., [81]). Second, instead of working with a
single map, an SLAM model can simultaneously estimate multiple maps and
resolve the location uncertainty within each. For example, the position in a
map about the camera angle and distance implicitly encodes the pose and scale
information (called bearing and sketch maps in parallel map theory [82]). In
summary, the generalized SLAM model allows us to estimate the latent variables
(i.e., the locations in multiple maps) and multiple maps simultaneously.
Thanks to the linear formulation using Kalman filtering, nice properties of
the original SLAM model (e.g., convergence of covariance matrix in Theorem 4)
are preserved by the generalized model (e.g., semantic SLAM [83]).

Topographic Maps. A latent space generalization of maps is from the
conventional Euclidean space to the topological space. For example,
$R^{3}\rightarrow\textbf{S}^{1}\times\textbf{S}^{1}$ (torus topology [58]) has
been found useful to model the multi-scale population activities of grid cells
[84]. Such topological map can easily transform allocentric grid cell
representations to egocentric goal coordinates for goal-directed navigation.
Another example of a topological map is the spatial
representation/organization of musical keys [85] \- the 24 major and minor
keys can be mapped to a torus structure $\textbf{S}^{1}\times\textbf{S}^{1}$
embedded in $R^{4}$. Similarly, for visual perception, the space of natural
images has been embedded into a the topology of a Klein bottle when inspected
locally (e.g., $3\times 3$ patches) [86].

It has also been shown that simple and complex cells [87], which can be
interpreted as latent space generalization of a map for simplified object
recognition, are indeed implemented at the level of cortical columns,
specifically in the primary visual cortex (V1) of the cerebral cortex. Within
a cortical column in V1, simple cells are organized to respond preferentially
to similar orientations. As one moves perpendicularly through the cortical
surface, the preferred orientation of these cells changes systematically. If
the input received by simple cells does not match the brain’s predictions
(based on previous experiences and context), these cells might generate an
error signal. This signal represents the discrepancy between what is predicted
and what is perceived. Like simple cells, complex cells are primarily located
in the primary visual cortex and are organized within cortical columns.
Complex cells respond to specific orientations and movements of edges and
lines, but unlike simple cells, their response is less dependent on the exact
position of the stimulus within the visual field. They integrate information
over a larger area and are less sensitive to the precise details of the
stimulus.

###  IV-C Non-Column Organization of Other Cortical Areas

The concept of a ”cortical column” primarily pertains to the cerebral cortex,
particularly the neocortex, where neurons are organized into columns that
extend from the brain’s surface down to the white matter. Each column tends to
process similar types of information, a principle most clearly demonstrated in
sensory cortices. In contrast, the hippocampus and entorhinal cortex, while
they do have their own distinct types of organization, do not follow this
columnar architecture. Their organization is more related to the specific
types of processing they are involved in, like memory and spatial navigation,
and less about the columnar structure seen in other cortical regions.
Therefore, we opt to briefly review the organizational principles underlying
hippocampus and entorhinal cortex at the end of this section.

Figure 6: Nonperiodic place cells are generated by linear summation of output
from periodic grid cells. [88]. Note that different combinations can give rise
to remapping in the hippocampus.

Grid and Place Cells. Grid cells are found in the entorhinal cortex,
particularly in the medial entorhinal cortex (MEC). These cells fire in a
hexagonal grid pattern across the environment and are crucial for providing a
spatial metric or coordinate system for navigation. It has been hypothesized
that the self-organization of neuronal networks based on Turing’s reaction-
diffusion principle [89] can produce grid-like patterns. Computer-based
simulation has confirmed the plausible connection between Turing symmetry-
breaking and grid-like networks using a spin glass model [90]. Place cells are
primarily located in the hippocampus, a region not organized into the classic
cortical columnar structure in the cerebral cortex. Each place cell (refer to
Fig. 6) represents a spatial location in the environment, known as a “place
field”. Based on this observation, we can think of Euclidean space as a latent
sequence [33] that can be embedded into the firing of place cells. The
hippocampus, where these cells reside, is more commonly associated with its
role in memory formation and spatial navigation, rather than columnar
organization.

In summary, we conclude this section by highlighting the role of cortical
columns in connecting the principle of device-level and system-level
connections. 1) Leveling down: canonical microcircuits at cortical columns
expand the reproducibility of macroscopic states by the principle of entropy
reduction. Unlike polychronization for neural groups, predictive coding
reflects a more sophisticated entropy reduction strategy at the circuit level
to capture regularity in the external environment. 2) Leveling up: canonical
microcircuits for predictive coding serve as a bridge to support more advanced
cognitive functions beyond single cortical columns. As advocated in a thousand
brain theory [19], nature has discovered a clever trick of compressing the old
brain into the minimum form of cortical columns. Then such a trick has been
recycled repeatedly by evolution to facilitate the reach of higher
intelligence, from spatial navigation to object recognition and from adaptive
behavior to verbal communication.

##  V System Level: High-Level Cognitive Functions

Imagine that you are looking at 150,000 cortical columns, each of which
represents a building block like a Lego piece. How will you build an
intelligent machine that can learn under memory-based predictive coding?
Inspired by the concept of a universal reference frame in the recent
development of thousand brain theory [19], we hypothesize that map-like
representations serve as a unified framework for implementing high-level
cognitive functions. We will start with the proposal of viewing space as a
latent sequence at the system level. Then we will use coordinate
transformations across different cortical areas for knowledge representation
across varying reference frames. Two principles of embodied cognition, namely
manifold untangling and the world as an outside memory, will be constructed to
derive bipolar action-oriented representations as the bridge to connect with
predictive coding at the column level.

###  V-A Modeling Sensorimotor Interaction

As argued in neuroAI [20], one of the greater discoveries by evolution is that
the embodied sensorimotor interaction serves as a universal solution to
achieving flexibility or natural intelligence. By compressing the “old brain”
into cortical columns, the “new brain” (neocortex) connects the sensory cortex
with the motor cortex to generate adaptive and flexible behaviors. This
compression trick has been iterated for a hundred thousand times to produce
higher-level cognitive functions, including object recognition and
manipulation, as well as language and abstract thoughts during the evolution
of natural intelligence. As argued in [19], the fundamental question about
understanding intelligence is knowledge representation (any world model is the
embodiment of knowledge) - i.e., how can we universally represent all
knowledge about the physical world by reference frames? Note that the
Euclidean space, the embodied model of cognitive maps learned by SLAM, is only
a special case of reference frame. It does not solve the problem of space
representation at the system level. Our first intuition is that all reference
frames, including Euclidean space, can be represented by a sequence in the
latent space [33].

Knowledge as a Latent Sequence. It was first recognized in [33] that the
mental representation of space is an emergent property of latent higher-order
sequence learning. However, the study of a clone structure cognitive graph
(CSCG) model [91] for latent representations is limited to the hippocampus
only. Following our intuition above, we conjecture that the higher-order graph
scaffolding endowed by the CSCG model can map sensory inputs to unique
contexts in a much broader abstraction of spatial and conceptual relationships
in the external environment. If all knowledge representations can be mapped to
a latent sequence, this sequence consisting of PNGs becomes a natural bridge
for connecting various cognitive functions across different cortical areas.

As recently demonstrated by [92], powerful transformer architectures [93],
when equipped with recurrent position encodings, can precisely reproduce the
spatial representation of the hippocampal formation (i.e., place and grid
cells). Similarly to our generalization of SLAM in the previous section, we
can envision that by considering a more general position encoding strategy in
the latent space, the transformer architecture, when equipped with CSCG-based
graph scaffolding, can manage the learning of not only Euclidean space but
also many more abstract representations [94]. For example, the grammar in
natural language can be interpreted as a high-order and hierarchical extension
of position encoding in transformers (we will elaborate on this idea in the
next section).

Can knowledge representations be embedded into a sequence in the latent space?
We conjecture the answer is affirmative based on the following arguments. As
argued in [95], a knowledge representation plays five essential roles. We can
verify that a latent sequence abstraction satisfies the following
requirements. First, the latent sequence is a surrogate model to support
ontological reasoning about the physical world. This can be seen by the
canonical circuits of predictive processing as demonstrated by Fig. 5. Second,
latent sequence is a fragmentary theory of intelligent reasoning and human
expression. For humans, intelligent reasoning is mostly based on natural
language, which itself is sequential and admits a latent space representation
[96]. Third, latent sequence supports efficient computation as manifested by
the ubiquitous transformer architecture [93]. The unjustified position
encoding strategy (i.e., using sine and cosine functions to encode distinct
positions) is another indirect evidence for the sufficiency of latent
sequence.

Knowledge representation through coordinate transformations. Cognitive maps
are a special example of organizing knowledge in low-dimensional space (e.g.,
Euclidean) [97]. Space as a latent sequence facilitates our pursuit of a
universal knowledge representation by more general concepts of reference
frames [98] because any reference frame is associated with an abstraction of
the space concept. An important new insight brought about by the influential
paradigm of embodied cognition [99] is: embodied cognition depends on the
kinds of experience that come from having a body with various sensorimotor
capacities that are themselves embedded in a more encompassing psychological
context. As shown in Fig. 7a, the closed-loop formulation of sensorimotor
interaction by efference copy and corollary discharge [100] offers a canonical
model at the system level for learning latent sequences as embodied internal
representations.

a) b)

Figure 7: Modeling sensorimotor interactions at the system level [30]. a)
forward and inverse models; b) positive and negative prediction errors are
computed by type-1 and type-2 neurons.

By combining the generalized SLAM model with the CSCG-based latent sequence,
we argue that any experience related to goal-oriented behaviors can be learned
by the canonical forward and inverse models in sensorimotor interaction. To
see this, we note that 1) action is the generalized form of navigation
(inverse model). The ultimate objective of motor commands is to achieve a
specific goal, regardless of navigation or manipulation; 2) sensing is a
generalized observation (forward model) that provides feedback signals for
motor adjustment. Two clever tricks achieve the flexibility of adaptive
behaviors: 1) parallel maps [82] \- that is, a group of cortical columns can
simultaneously learn multiple latent sequences by connecting different sensory
modalities with various motor controllers; 2) build a hierarchy [9] \- that
is, temporal binding of PNGs enables the learning of more complex latent
sequences from simple ones (e.g., splitter cell [101] instead of place cells).

We can model the forward and inverse processes of sensorimotor interaction by
coordinate transformations [34, 35, 36]. Coordinate transformations are
essential to the flexibility of behavior because sensory information encoded
into the coordinates of sensory epithelial tissues (e.g., skin and retina)
must be transformed to the coordinates of muscles for movement and vice versa.
The implementation of sensorimotor learning via cortical transformations has
been an under-researched topic. In a perspective paper [102], the modeling of
sensorimotor learning is cast under a state space framework (refer to Fig. 8).
The key idea is to abstract the sensory and motor cortex, as shown in Fig. 8a,
into latent sensory and motor variables. The closed-loop learning involves two
state spaces: weight space (a space of synaptic weight) and activity space (a
space of motor activity). Note that a point in the weight space defines the
entire connectivity structure of the neural network; a point in the activity
space jointly represents the activity of all neurons involved in the network.
The synaptic states in the weight space constrain the neural trajectories in
the activity space; conversely, the neural states occupied in the activity
space constrain how the synaptic state evolves in the weight space during
learning.

Coordinate transformations in the latent space. From the state-space
perspective, learning can be formulated as a process of exploring and
navigating in the latent weight/activity space in search of a solution for a
desired behavioral outcome [102]. Such a latent space navigation perspective
is well aligned with the generalized SLAM model at the column level and
general enough for linking behavior to activity patterns (geometry) of neural
population [103] at various scales - e.g., from object recognition and motor
control to decision-making and memory consolidation. The most challenging open
problem is the learning and updating of the internal model [44] \- e.g., from
supervised learning [104, 44] to self-supervised learning [105, 106]? Two
important new insights brought about by our approach are 1) to take a naive
Bayesian approach [107] to bipolar action-oriented representation [108]
instead of pursuing the construction of an internal model in conventional
cognitive science; 2) to $offload$ the world as an outside memory which can be
instantaneously accessed locally (by the sensory organ’s receptive fields)
instead of storing it internally as a global model.

Geometrically, our first intuition is to pursue a shortcut of
mapping/transformation between perception (sensory cortex) and action (motor
cortex) by untangling the neural manifold in the latent space of sensory and
motor variables. In state space, a neural manifold can be defined by the
trajectory of synaptic states that can be traversed in the activity space
under current connectivity. Manifold untangling via neural population geometry
has been studied for sensory perception in [103, 57]. The key idea is that the
tangled manifold at a lower level, such as V1, must be untangled to facilitate
object recognition at a higher level, such as IT. Similarly, neural activities
in motor command are shown to avoid tangling too [109]; computation via neural
population dynamics has been extended from motor control to decision-making
and working memory in [110]. To our knowledge, the idea of connecting these
two lines of research has not been considered in the open literature. We make
the following proposition as a shortcut to adaptively maintain an equilibrium
between the agent and the environment [108] while bypassing the full
reconstruction of the internal model (Fig. 8a).

Proposition 2 (Principle of Manifold Untangling):

The task of sensorimotor learning via coordinate transformations can be
facilitated by simultaneously untangling the neural manifold of sensory and
motor variables in the latent space.

The initial evidence supporting the above proposition can be found in recent
studies of abstract/disentangled representations [94, 111]. Using feedforward
neural networks as a proxy model, the authors show that an abstract
representation that is disentangled in the latent space can be obtained from
multi-task learning. This research shows that disentangled representations
[112] of sensory and motor variables may arise from adaptive behaviors and
serve as a universal solution to generate higher-level cognitive functions.
Note that the generalized SLAM model is still based on linear Kalman
filtering, which dictates the necessity of manifold untangling to overcome the
barrier of nonlinearity. Referring to Fig. 8b, we observe that the potent
equilibrium between the internal model and the external environment can be
easily maintained by correcting the prediction error in the latent space
(after untangling) because disentanglement enables few-shot learning and
improves generalization in novel tasks [113].

a) b)

Figure 8: Modeling sensorimotor interactions in the state space of neuronal
networks [102]. a) An embodied internal model of the environment generates
corrective signals to drive the neural system in the direction that minimizes
the prediction error; b) the error-reducing mechanism in the activity space
pushes the action toward the direction of minimizing prediction error.

###  V-B Embodied Cognition Meets Predictive Coding

Why is a cortical column a structure without a function? Our second intuition
is related to the debate between representationalism and anti-
representationalism. Given the column organization of the neocortex, one
cannot help wondering if 150,000 cortical columns convey the internal
representation of the environment. Surprisingly, the functional role of
cortical columns was debatable and unclear [10]. Since Mountcastle’s discovery
in the 1950s [7], it has not been successful in determining the function of
cortical columns by searching for species differences in sensory performance
that correlate with their presence or absence. Search for species differences
in visual performance that correlate with presence or absence. The ocular
dominance columns are not required for any visual faculty. Furthermore, it has
been shown that the expression of ocular dominance columns can vary greatly
even within the same species or even within the same individual’s visual
cortex. So how can we explain the principle of column organization if cortical
columns cannot be associated with a specific function?

The answer is that it is cortical areas rather than cortical columns as the
proper scale for studying functional specificity. As our modeling of
sensorimotor interaction shows, the column organization of the neocortex is to
support adaptive and flexible behaviors. A moment of thought reveals that the
subtle implication underlying the flexibility (or universality) is that
individual building blocks cannot be function-specific. Using Lego pieces as
an analogy - one can build a car or tank with Legos, but a single Lego piece
does not have a specific association with the holistic moving function
provided by the entire system. Instead, we argue that a more fruitful question
is: how can various high-level cognitive functions be implemented by a
universal processing algorithm based on sensorimotor interaction at the column
level?

To answer this question, we resort to the theory of embodied cognition [114,
99] where human cognition is believed to be rooted in sensorimotor processing.
From an evolutionary perspective [22], a proper understanding of neural
systems has to take both the agent (the physical embodiment) and the world
(the physical environment) into account. The heuristic principles of embodied
cognition include: 1) cognition is situated and time-pressured; 2) we offload
cognitive work onto the environment; 3) cognition is for action. We formalize
these principles more rigorously in this paper as follows. The generalized
SLAM reconstruction is a mathematical abstraction of the first principle
across different levels (i.e., cortical maps [115] serve as embodied internal
representations). More importantly, the last two principles can be unified by
a paradigm shift from static internal representation to dynamic action-
oriented representation [22]. By coupling the agent with the world, we can
simplify goal-oriented behaviors into the following principle.

Proposition 3 (Principle of Local Off-loading)

The neocortex constantly interacts with the physical environment (instead of
mentally constructing an internal representation) by minimizing the potential
discrepancy between the target and actual goals in action or behavior. But
this interaction is constrained by the locality of sensory organs and,
therefore, only offloads a localized version of the world as the outside
memory.

This principle can be interpreted as a localized refinement of the theory of
embodied cognition [25]. The agent constantly interacts with the physical
environment, but instead of mentally constructing a global internal
representation, it “uses the world as its own model” [25] and offloads
cognition onto the localized environment constrained by the receptive field of
a sensory organ. Predictive coding is essentially the simple (but not simpler)
solution to learning a parsimonious and self-consistent representation [116]
with the locality constraint. Moreover, by recognizing that cognition is for
action, we can simplify the objective of action-oriented representation -
regardless of the type of cognitive tasks (i.e., the functional specificity of
the neocortex), a universal guiding principle is what Ashby calls
“homeostasis” [23]. The predictive coding framework offers a computationally
principled solution to maintain stability in the face of changing but still
locally constrained external environments by feedback loops. In other words,
no matter what kind of cognitive tasks an agent deals with, they eventually
boil down to positive or negative feedback - a simple yet elegant control
mechanism discovered by nature.

Bipolar action-oriented representation also helps the understanding of
Mountcastle’s discovery. The seeming lack of a function with cortical columns
is likely due to its role as a universal building block that can be
iteratively recycled to support new functions through evolutionary design.
Moreover, we note that the great flexibility of cortical areas can also be
interpreted as “lacking a specific function” because they can adapt to a great
variety of functionalities when tapped to different pairing of sensory inputs
and motor outputs (e.g., Helen Keller unlocked the secret of language despite
being blind and deaf [71]). The organizational principle at the system level
reflects a clever recycling trick of canonical microcircuits at the level of
cortical columns. But more is different [27] \- the new emergent property
arises from the interaction among the cortical areas. Visu-motor interaction
inspired the invention of tools; auditory-motor interaction contributed to the
origin of speech [40]; Intraparietal Sulcus (IPS), angular gyrus, and
supramarginal gyrus enable human mathematical reasoning.

Despite the apparent functional difference, almost all high-level cognitive
functions are rooted in goal-oriented behaviors [117]. The new insight brought
about by embodied cognition is that all goal-oriented behaviors boil down to
two basic types (positive vs. negative) of predictive coding errors in bipolar
action-oriented representation. This claim can be justified as follows. As
shown in Fig. 7a, the brain is hypothesized to have a mechanism of
transforming the efference copy of the motor command into the sensory
coordinate system. This transformed version of the efference copy has been
called corollary discharge [100] to distinguish the cause of sensory input
between internal and external changes. This line of reasoning has led to
internal representations (a.k.a. internal models) responsible for predicting
sensory input. During the prediction step, bottom-up (green arrows) represents
the sensory input and top-down (red arrows) denotes the prediction outcome.
During the update step, we note that action-oriented internal-representation
neurons in the layers of the canonical microcircuit could be hypothesized to
perform the prediction step of the Kalman filter, generating expectations
about sensory inputs or the state of the environment. As shown in Fig. 7b,
positive type-I neurons (sensory-prediction) and negative type-II neurons
(prediction-sensory) could be responsible for comparing these predictions to
actual sensory inputs, thus identifying prediction errors.

a) b)

Figure 9: The Tolman-Eichenbaum Machine (TEM) model [92]: a) spatial
navigation task encoded into sequence prediction problem; b) TEM consists of
path integration and memory network.

##  VI Applications to Computational Neuroscience and Learning Systems

In this section, we attempt to construct proxy models for three cognitive
tasks: spatial navigation, object recognition, and language acquisition. In
addition to the organizational principles identified above, our construction
requires the following three additional principles beyond the system level.
They jointly characterize the abstraction of evolving natural intelligence by
evolutionary design.

Proposition 4 (Principle of Generalization/Abstraction):

All high-level cognitive functions can be abstracted into spatial navigation
in different latent spaces. Latent space is a generalization of Euclidean
space into topographic maps which can be represented by learned by a sequence
of PNGs.

This principle is inspired by recent work on structured sequence learning [33]
for modeling the hippocampus. Space as a latent sequence has greatly
facilitated the representation of Euclidean space by neural codes. If it is a
simple yet elegant idea discovered by nature, we conjecture that the neocortex
has adopted a similar latent space abstraction during the evolution of
intelligence. In other words, nature has recycled the same trick for achieving
higher intelligence through generalization and abstraction.

Proposition 5 (Principle of Integration/Factorization):

Conceptually similar to the design of integrated circuits, cortical columns
serve as the basic building block for integration. Sophisticated cognitive
functions can be factorized into smaller pieces to facilitate an integrated
implementation.

This principle was consistent with Ashby’s design of intelligence systems
[23]. It is argued that the modular design of cortical columns has been
repeated for hundred thousand times to reach higher intelligence [10]. The
neocortex’s hierarchical organization [118] reflects the elegance of design by
nature to mirror not only the nested structure in the physical world but also
the embodied and localized solution to use the world as its own model.

Proposition 6 (Principle of Attention/Voting Mechanism):

When the same physical object is perceived by more than one sensory modality,
some consensus among cortical columns needs to be reached by voting.
“attention is all you need” [93].

This principle was discussed in a thousand brain theory [19]. It was
originally conceived as a solution to the ill-defined binding problem [119].
Our position is that the attention mechanism is necessary for embodied
cognition because of the locality constraint with the sensory organ of any
intelligent agent. The biological plausibility of this principle lies in that
the cells with long-distance connections offer a neural mechanism of
implementing voting among cortical columns. The stability of sensory
perception is the result of voting.

Based on the above three additional principles, we will construct three models
using a navigation and memory modules as in the generic SLAM models. The
unifying theme for these constructions is to show that high-level cognition
functions such as object recognition and language comprehension can be
factorized similarly to the task of spatial navigation with a longer
evolutionary history. The latent space generalization of the SLAM model offers
a powerful and unified framework for implementing embodied cognition based on
dynamic and action-oriented representations.

Navigation Module. This module is an abstraction of physical motion by the
agent, including moving around (e.g., walking, running, flying, and so on) and
looking/touching around (i.e., saccadic and tactile movement) in the physical
world. For spatial navigation, It has been long hypothesized that place cell
firing without cue results from path integration [98]. Among various models of
path integration [120, 121], continuous attractor neural networks (CANN) [122]
represents a biologically plausible implementation of the loop including the
hippocampus, the subiculum, the parasubiculum, and the superior layers of the
entorhinal cortex. The hippocampus and entorhinal cortex work together to
perform path integration. The entorhinal cortex provides the hippocampus with
metric (grid cells) and directional (head direction cells) information. In
turn, the hippocampus integrates this data with its own place cell information
and other contextual information to update the agent’s position in space. In
SLAM, small sensor readings or movement estimation errors can accumulate over
time, leading to drift. Similarly, in path integration, slight inaccuracies in
estimating distance or direction can lead to errors in the internal
representation of location. Both systems require mechanisms for error
correction, often using external reference points (landmarks) in various
coordinates.

Memory Module. This module is an abstraction of “landmark” recall by the agent
across different scales (both local and global). Associative memory (AM) is a
higher-order cognitive function that integrates information across various
brain regions (above the level of cortical columns). Classical models such as
Hopfield network [28] only represent a crude approximation of associative
memory. Modern perspective [123] employs a continuous energy landscape,
allowing for a more dynamic and flexible representation of states. Since
neurons in modern Hopfield networks can have continuous values, the network’s
capacity and flexibility in representing information have been greatly
improved. However, these models do not directly support the classification of
memory (e.g., declarative vs. procedural memory [124]) as high-level cognitive
functions. Their rich connections with language imply that a proper level of
studying declarative/procedural model [125] is even higher than that of
language (we will come back to this issue later).

###  VI-A Cognitive Maps and the Navigation Model

Following James J. Gibson’s ecological approach to vision [45], we consider
that spatial navigation is among the first problems elegantly solved by
nature’s evolutionary design. If cortical columns represent a compressed form
of the old brain, how does the neocortex invent a global positioning system
(GPS) by scaling up the SLAM model? For spatial navigation, goal-oriented
behavior becomes as simple as: are we getting closer to the target? The
agent’s action can be deemed intelligent if it helps achieve this simplified
objective. Using the principles of abstraction and composition, the GPS
problem can be factorized into two independent tasks: where and what. We will
first present the system-level solutions to these subproblems and then
integrate them into the build of cognitive maps [31].

Cognitive Maps. In the pioneering work of Tolman [126], cognitive maps were
proposed as a systematic organization of knowledge that spanned all behavioral
domains including spatial and non-spatial. To facilitate our discussion, we
start with the task of spatial navigation because it has been widely studied
in the literature. Following the setup in [127, 31], we adopt a reinforcement
learning (RL) and graph representation framework for cognitive maps. Note that
RL and Kalman filtering (KF) can be unified under a predictive coding
framework in that both involve making predictions and then adjusting these
predictions based on error feedback. Therefore, canonical microcircuits of
cortical columns serve as the building blocks for RL and KF. A toy model for
studying cognitive maps is the so-called Tolman-Eichenbaum Machine (TEM) [32].
We introduce the definition of the Tolman-Eichenbaum Machine (TEM) first.

Definition of Tolman-Eichenbaum Machine (TEM)

A Tolman-Eichenbaum Machine (TEM) integrates the concept of a cognitive map
with modern understanding of memory systems in the brain. It generalized the
concept of cognitive map from spatial information to various non-spatial types
of information.

Fig. 9a shows a toy example of $(action,sensory)$ pairs and the navigation
problem based on the TEM model. The problem of spatial navigation can be
formulated as how to learn and infer the external environment from a sequence
in the latent space S. For a sequence of actions, environment graph G encodes
a distribution over the latent sequence, establishing the mapping between an
arbitrary topology and PNG sequences (i.e., the mirrored internal
representation). Such latent space encoding is a graph learning problem (e.g.,
clone structure cognitive graph [91]). Using clone structure cognitive graph
(CSCG) [91], TEM can learn allocentric spatial representation from a high-
order sequence of egocentric sensory inputs. Such principle of “space as a
latent sequence” also turns out general enough to accommodate non-spatial
cognitive tasks. We opt to present the CSCG model as a topological extension
of the SLAM model where the environment need not be Euclidean. Therefore, CSCG
learning can also be implemented by variations of graph-based SLAM such as the
expectation-maximization (EM) algorithm that has been applied to learning
cloned hidden Markov Models (HMMs) [128].

a) b)

Figure 10: Change of reference frames and its implementation by path
integration. a) egocentric to allocentric transformation [129]; b) end-to-
effector transformation [130].

Fig. 9b illustrates the factorization of the TEM model into building blocks of
path integration and memory network (based on associative memory). A moment of
thought will reveal that this TEM model can be interpreted as another SLAM
problem formulation but at a different scale: with grid and place cells, path
integration can redefine the map at a larger spatial scale; with simple and
complex cells (and visual cortex built upon them which we will elaborate
next), memory network can recognize more sophisticated landmarks. However, the
principles of parsimony and self-consistency are still applicable. As shown in
the figure, the computational solution to this problem decomposes the learning
into two factors (conceptually analogous to location and map in the SLAM
model): graph topology and sensory observation. Note that the Kalman-
filtering-based solution to the SLAM problem at the column level also admits
an EM interpretation. Therefore, The standard Baum-Welch algorithm that
alternates between sensory prediction (E-step) and map updating (M-step) can
implement EM-based learning of cognitive maps.

Echolocation and the Bat Model. Echolocating bats, comprising an estimated
$25\%$ of living mammals, represent a concrete example for studying spatial
navigation without seeing (i.e., “bats are blind mammals”) [131]. The
biological sonar system helps echolocating bats orient and forage in dark
environments. The tight coupling of acoustic signal processing with vocal-
motor control is based on the coordination between auditory and motor systems.
Recent experiments with bats [132, 133] have shown that wild bats seldom
exhibit random search but repeatedly forage in goal-directed, long, and
straight flights, suggesting map-based navigation. Based on our analysis
above, we offer the following explanation for constructing cognitive maps for
bats. Beaconing represents a parsimonious solution to reach a goal and route
following is a generalized strategy when the shortest path is not directly
feasible. The path integration problem is solved by somatosensation
(contributing to the integration of self-motion information); associative
memory in bats is limited to simple categories of food and obstacles.
Therefore, cognitive maps in bats yield self-triangulation [134], which can be
interpreted as a special case of the SLAM solution. As experimental evidence,
a biomimetic sonar implementing the SLAM model for bats’ navigation has been
reported in BatSLAM [135] with a simplified navigation task of obstacle
avoidance (the lowest-level layer in the three-layer design, called “avoid
machine” in [25]). From avoid machine to wander machine and legged robots that
can balance [136], we have witnessed the problem of spatial navigation has
been largely solved in robotics research (e.g., the commercial products such
as sweeping robots developed by iRobot and quadrupedal robots developed by
Boston dynamics).

###  VI-B Object Recognition and the Vision Model

How do we define objects? From an evolutionary perspective, “objects” are
artificial because they are made by mankind. Stone tools were made by our
ancestors around 2.6 million to 1.4 million years ago. As visually
hypothesized in the movie “2001: A Space Odyssey”, the mysterious appearance
of an alien monolith caught the attention of a tribe of hominines. A brave
human touched the monolith, and like being struck by thunder, the idea of
using a bone as a weapon was born. The rest is the history of tool-making by
humans. The moral of this story is that vision [39] “has its evolutionary
ratio rooted in improved motor control.” Therefore, a more fruitful approach
than pure vision is understanding vision from an embodied cognition
perspective [24, 137].

Embodied cognition indeed has its footprint in understanding how vision works
- e.g., Gibson’s ecological approach to vision [45] and Ballard’s animate
vision [138]. However, the open literature has not well-developed the subtle
relationship between vision and motor control. A fundamental question seems to
be: how can spatial navigation skills be recycled by evolutionary design to
support the task of motor control (for object manipulation)? To appreciate the
analogy between navigation and manipulation, we resort to the recently
developed thousand brain theory [19, 139] in which the role of reference frame
is emphasized for understanding how the neocortex works.

What egocentric coordinates to spatial navigation are allocentric coordinates
to object manipulation (refer to Fig 10a). The transformation problem between
allocentric and egocentric spatial representations has been studied in [129].
Based on such knowledge of conversion between different reference frames, we
envision a counterpart of a generalized SLAM problem for object manipulation
to its original navigation formulation. As shown in Table II, we note that the
mathematical formulation of motor control (e.g., neural optimal control
hierarchy (NOCH) [140]) is also based on Bellman equation [141], which is
equivalent to Kalman filtering in the SLAM formulation. Both motor control and
spatial navigation heavily rely on the error feedback of linear predictive
coding for closed-loop optimization.

| spatial navigation | object manipulation
---|---|---
motor control | lower limbs | upper limbs
spatial representation | egocentric | allocentric
sensory observation | landmarks | keypoints
action unit | moving direction/speed | end effector commands
error control | visual feedback | perturbations
hierarchy | multiple spatial scales | three-level decomposition
visual pathway | dorsal | ventral

TABLE II: Analogy between spatial navigation and object manipulation.

Vision for motor control facilitates an embodied cognition approach to
understanding object recognition. From spatial navigation to object
manipulation, the attention of our mental power shifts from egocentric to
allocentric coordinates. Following this logic, we can define “object” as an
analogy to “environment” [142] but unlike the natural world, objects from
stone tools to autonomous vehicles represent an artificial world created by
humans. Such shift of attention is responsible for the separation of the
visual pathway into dorsal (where) and ventral (what) streams [143] by
evolutionary design. Both streams are hierarchically organized but
functionally specialized. Attention to a particular attribute of a visual
stimulus can be mediated partially by the selective increase of the
corresponding neural system (i.e., the ventral stream). We summarize our
reasoning into the following conjecture.

Conjecture 1 (Vision for Motor Control):

Attention is all you need to recycle the trick of spatial navigation for
object manipulation.

From Navigation to Object Manipulation and Recognition. Inspired by the
analogy between spatial navigation and object manipulation (Table II), we
suggest that the TEM model invented for spatial navigation can be transformed
into a model for object manipulation, a stepping stone to the dual task of
object recognition [144]. As shown in Fig. 11, the modified TEM model consists
of two modules: 1) instead of path integration, the traverse in space is
achieved by saccadic movement concerning an allocentric coordinate; 2) instead
of memory network, the task of making a sensory prediction is based on
different keypoint-position pairings. Using the generalized SLAM solution, we
can see how a cognitive map originally used for spatial navigation in
egocentric coordinates can be recycled for object manipulation in allocentric
coordinates.

Figure 11: Modified TEM model for object recognition: path integration and
memory network are replaced by saccadic motion and keypoint recall,
respectively.

Surprisingly, the relationship between object manipulation and object
recognition has not been widely investigated in neuroscience literature.
According to the concept of affordances introduced by James J. Gibson, when we
see an object, we recognize its physical attributes and its potential use
based on its shape, size, and other properties, which directly influence how
we manipulate the object. Such an ecological perspective is consistent with
the view that vision facilitates motor control. The shared visual pathways
between visual perception and motor control also support the hypothesis that
object manipulation and object recognition can be formulated as dual problems
with chicken-and-egg flavor. Formally, we have the following conjecture about
the potential connection.

Conjecture 2 (Object Manipulation for Recognition):

Object manipulation and recognition are two sides of the same coin. If we
consider the hierarchical structure for facilitating information flow: object
manipulation represents a divergent flowing direction (namely the
decomposition of a complex motion into multiple simpler building blocks);
object recognition represents the opposite/convergent flowing direction (i.e.,
the combination of multiple local features into a single identity).

The above conjecture is supported by the analogy between vision and motor
control [142] (refer to Fig. 12). In the motor areas of the frontal lobe, it
was discovered that neurons with similar preferred directions are interleaved
with mini-columns with nearly orthogonal preferred directions [145]. Such
organizational principle is similar to the directional preference of simple
cells in primary visual cortex (V1) [87]. From an embodied cognition
perspective, manipulation and recognition are interconnected by the forward
and inverse processes of interacting with the object. More broadly, the
complementary nature between manipulation and recognition makes them a natural
fit for the hierarchical model for a close-loop sensorimotor control [146].
From manipulation to recognition, we can formulate it as a generalization
problem [142] \- i.e., from visuomotor learning (manipulation) to visual alone
(recognition). A hierarchical extension of associative memory (AM) module
(e.g., [147, 148] can solve this generalization problem involving vision and
motor control, as we will elaborate next.

Figure 12: Analogy between visual perception and motor control [142]. a)
visual system is characterized by many-to-one mapping; b) motor system is
characterized by one-to-many mapping.

a) b)

Figure 13: Invariance computation via identity-preserving transformations: a)
two-stream hypothesis in visual information processing [143] (we conjecture
that the interaction between spatial and object processing contributes to
identity-preserving transformations required by invariant object recognition);
b) IT population explicitly encodes the identity information along with pose,
scale, position, context [57] (they define the latent space of invariant
object recognition).

Manifold Untangling. If vision’s objective is to support motor control, goal-
oriented behaviors naturally dictate the requirement of object recognition
such as invariance and selectivity [149]. Invariance is needed for robust
motor control so the goal can be achieved regardless of the interference from
the external environment. Selectivity is necessary for accurate motor control
to avoid undesirable confusion (e.g., mistaking a wife for a hat [150]). The
question of how the vision system solves the problem of invariant object
recognition has not been completely understood; a contemporary view of single
neurons in the inferior temporal (IT) cortex [57] shows that it simultaneously
conveys explicit information about object identity and its position, size,
pose, and context (refer to Fig. 13b). However, the mechanism of such
population representation (a.k.a., neural population geometry [103]) remains
elusive. Toward an improved understanding, we come up with the following
conjecture.

Conjecture 3 (Active Exploration for Invariant Object Recognition):

Motor control contributes to invariant object recognition by allowing active
exploration of the environment. All latent codes (including the identity of an
object) are hierarchically embedded into the visual cortex to facilitate
associative recall.

The critique of pure vision supports the above conjecture [39] and the theory
of animate vision [151]. From the perspective of active exploration, an
important new insight is the temporal consistency of object identity and its
implications into identity-preserving transformations. During active
exploration, physical motion induced by the object or observer naturally
defines the constraint space of invariant identity. In dynamical systems,
these order parameters characterize the volume of the constraint space. After
parameterization, the corresponding latent codes and the object’s identity
code are embedded into the stimulus data (sensory observation). They become
tangled due to nonlinear projection (photoreceptors serve as the information
bottleneck [152] between the environment and the visual cortex). The object
recognition problem can be mathematically formulated as manifold untangling,
which apparently has been solved by the visual cortex [149].

It has been hypothesized that the visual cortex solves the manifold untangling
problem by gradually unfolding it along the visual hierarchy [57]. Based on
conjecture 3, we believe that a hierarchical extension of the TEM model (e.g.,
AM via predictive coding [73] or latent energy-based model [153]) can
implement the idea of gradual unfolding as follows. At each level, sensory
stimuli are encoded into a code in the latent space (this latent code drives
the attractor in the AM-based recall); the latent codes generated locally are
fed forward along the hierarchy to produce a more global representation of the
sensory stimuli by higher-dimensional latent codes. These latent codes are
represented by neural populations such as PNGs and support associative recall
across different scales. A working memory model in latent representation [154]
shares some similarities to ours except the use of variational autoencoder
(from V1 and V2 to V4 and IT, the dimensionality of latent codes increases
instead of decreases). A computational implementation of the hierarchical
extension of the TEM model is via a recently developed universal Hopfield
network (UHN) [155]. In summary, the problem of manifold untangling can be
solved by AM assuming the feasibility of factorization into latent codes.
There is no need to re-bind those latent codes, implying that the binding
problem is moot [156].

Factorization and Attention. As mentioned above, the factorization principle
is already reflected by the functional specification of different cortical
areas such as visual vs. motor systems. The factorization of the visual
pathway into the ventral and dorsal streams [143] can also be understood from
the perspective of visual-motor factorization. Similarly, both visual and
motor systems are further factorized along the hierarchy: for motor, it
consists of the musculoskeletal plant, spinal cord, and the cerebral cortex
[146]; for vision, it consists of V1, V2, V4, and IT [57]. An important new
insight this work brings is recognizing the interaction between the ventral
and dorsal streams for animate vision.

The dorsal visual pathway also starts from V1 and V2, but includes the middle
temporal (MT) and medial superior temporal (MST) areas. The dorsal pathway is
responsible for spatial processing, such as motion perception and the object’s
spatial location relative to the viewer. Equally important is the encoding of
spatial transformations by dorsal pathways (refer to Fig. 13a) because the end
objective of object recognition is to support motor control (e.g., to grab a
glass of water). This is particularly true for the invariance of object
recognition to ego-motion (the source of identity-preserving transformations).
Dorsal and ventral streams converge at higher levels of processing, such as
the parietal and prefrontal cortex. In these regions, spatial information from
the dorsal stream and object representations from the ventral stream are
integrated. This integration allows for a more holistic understanding of the
visual scene, where spatial cues are combined with object information to
achieve invariant object recognition.

Using coordinate transformations, we suggest that the encoding of spatial
transformations by dorsal pathways can be implemented similarly to that of
object selectivity by the ventral pathway. At first sight, one might believe
that the space of identity-preserving transformations is much smaller than
that of object identities. A moment of thought reveals this might be a
misunderstanding of the interaction between the ventral and dorsal pathways.
What and where streams play an equally important role in the human vision
system. Due to the hierarchical organization of the visual cortex, the
identity-preserving transformations for V1 and V2 are different from those for
V4 and IT. For example, the enumeration of identity-preserving transformations
for a simple object such as “$\top$” consisting of a vertical edge and a
horizontal edge might end up with a few options; the total number of identity-
preserving transformations for a complex object such as a human face can
experience exponential growth. Fig. 13b) only shows the tiling of identity and
identity-preserving transformation in 2D space [57] as a toy example of
visualization.

Another byproduct of dorsal-ventral factorization is the mechanism of
selective attention in visual search [157, 158] as a goal-directed behavior
strategy. Visual attention [159, 160] facilitates the task of visual
perception and search by prioritizing some aspects in the vast amount of
information while ignoring others. Such a simple idea can be elegantly
implemented by the linear weighting strategy such as the transformer
architecture [93]. However, the prioritization or weight assignment procedure
is intrinsically related to the strategy of factorization (e.g., sparse
factorization for neural attention modeling [161]). Among five factors that
guide attention in visual search [162], we note that they all boil down to the
fundamental task of motor control (i.e., the control of saccadic movement),
which essentially represents the direction (the bearing map in parallel map
theory [82]) toward the target. To achieve any specific goal, attention is all
you need to approach the goal from an arbitrary position. Regardless of
spatial navigation or visual search, directional cues play a fundamental role
in all goal-directed behavior.

Figure 14: The analogy between transformer architecture (left) and TEM machine
(right) [92]: both use the self-attention mechanism to generate a weighted
prediction [92].

###  VI-C Language with Perception and Action: the Embodied Model

As eloquently argued by Moravec in [46], language is a relatively young trick
discovered by nature compared to sensorimotor knowledge. Probing into the
origin of language [40] was considered a debatable topic, yet there is a wide
consensus on the importance of natural languages to the evolution of
intelligence, especially from the perspective of its relationship to Darwinian
natural selection [163]. Can language evolve without the attachment to a
physical body? The answer is likely to be negative due to the fundamental role
played by the motor system in language comprehension [164]. The close tie
between language and action has been formulated by “bi-directional hypothesis”
[165]. More specifically, embodied language processing emphasizes a rich
interaction between language and action known as “experiential trace
hypothesis” [166]. It says that each time an individual interacts with the
world, traces of that particular experience are left in our brain. These
traces can be accessed again when people think of words or sentences that
remind them of that experience. Additionally, these traces in our brains are
related to the actions to which they are related. Based on this hypothesis, we
can connect several existing theories of embodied language with the organizing
principles discussed above.

Theory of Event Coding (TEC). TEM model constructed for the study of spatial
navigation is also useful to a study of the relationship between perception
and action as in the theory of event coding (TEC) [167]. The key insight
underlying TEC is that perceptual and action features share a common code such
as the treatment of sensory observation and spatial action by peer variables
in the EM-based computation. By viewing actions as the result of anticipatory
sensory consequences, we observe that the TEM model is well aligned with the
ideo-motor theory of cognition [168]. As shown in a recent work [92], TEM
models admit a transformer-based interpretation (refer to Fig. 14). Given the
significant influence of transformer architecture on language modeling [93],
TEM model might serve as a proxy for TEC to study its implications in the
modeling of natural languages.

The discovery of mirror neurons supports the bi-directionality of the
association between sensory and motor representation in TEC [169] by single-
neuron recordings in neurophysiology. Mirror neurons are special because they
form a cortical system matching observation and execution of goal-oriented
motor actions [169]. Mirror neurons provide direct evidence for the common
coding shared by perception and action. The human mirror system includes
Broca’s area, a cortical region critical to speech production, which makes it
a plausible candidate for language-related action simulation. In action
simulation theory of motor cognition [170], humans understand and predict the
actions of other people by mentally $simulating$ those actions within their
own minds. This theory is closely related to the mechanisms of language
acquisition because babies’ brains internally simulate their parents’ actions
as if they were performing those actions themselves. In other words, action
simulation is the theoretical foundation for language acquisition and
comprehension.

Embodied language. Action-oriented representation facilitates understanding
the constitution between language and action in the theory of embodied
language [164]. By restoring the central role played by the motor system in
language processing, we can gain deeper insight into the essence of language
from a motor resonance or action simulation perspective [171]. The literature
shows two kinds of motor resonance related to language comprehension:
communicative and referential [164]. The former occurs when the motor system
responds to the communicative act; the latter occurs when it responds to the
communication content. Both motor resonance types facilitate language
comprehension by grounding abstract linguistic representations in sensorimotor
experiences. For example, individuals may activate motor representations
associated with the action of grasp when seeing others grasp a bottle of water
(communicative motor resonance) or reading the word “grasp” (referential motor
resonance).

The class of referential motor resonance is the source of abstract thoughts or
covert movements in our mental world which are hierarchically organized. The
relationship between hierarchical motor control [172] and the nested structure
of natural languages [173] can be summarized as follows. Both language and
motor control demonstrate compositionality, whereby complex structures are
built from simpler elements according to combinatorial rules. In natural
language, sentences are composed of phrases composed of smaller units such as
words and morphemes. This syntactic structure manifests as the ability to
generate an infinite variety of meaningful sentences from a finite set of
words and grammatical rules. Similarly, in motor control, movements can be
decomposed into smaller units, such as motor primitives or submovements, which
are organized hierarchically to generate complex behaviors. Moreover,
movements can comprise sequences of simpler actions or motor primitives,
combined to produce more complex behaviors.

Language with Sensory Deprivation. Based on the theory of motor cognition, all
language behaviors have semantic and pragmatic aspects [164]. However, speech
and sign language represent overt motor behaviors; by contrast, inner speech
and abstract thought are examples of covert movements - the results of action
simulations. Such a perspective can be supported by the ability of humans to
acquire language regardless of sensory deprivation (e.g., the story of Helen
Keller). As demonstrated by comparing proficient Braille readers with
untrained subjects [174], an expansion of the sensorimotor cortical area can
be identified with the former, justifying the plasticity of tactile
representation. For Helen Keller [71], the secret of language was revealed to
her by Ms. Sullivan through a playful experiment with water - i.e., how the
sensory experience with water translated to a symbolic representation
abstracted by finger dancing. Such an observation suggests a fundamental
understanding of language as motor resonance - a cortical transformation in
the latent sequence space [33].

Using the CSCG model [33], we can map sensory inputs (the feeling of water and
its symbolic/tactile abstraction by Braille) to unique sequences in the latent
space. The association between these two sequences plays a surrogate role of
mirror neurons - e.g., when infants observe others speaking or producing
linguistic gestures, mirror neurons may fire, allowing them to mimic those
actions. This imitation serves as an important mechanism for language
acquisition, as infants learn to produce sounds, words, and eventually
sentences by mimicking the speech patterns of those around them. In the
situation of sensory deprivation, the task of mimicking or imitation learning
[175] becomes more challenging for Helen Keller. What Ms. Sullivan achieved is
to use temporal synchronization as an additional clue to help Helen solve the
association or binding problem using tactile sense only. Normal children can
more easily solve a similar problem by associating visual writing with speech
pronunciation when learning their native language. Regardless of different
sensory modalities, language is another eloquent evidence for the emergent
property of sensorimotor interactions.

##  VII Conclusions

Unlike generative pre-trained transformers (GPT), the human brain only
requires a power consumption of 20 Watts to achieve natural general
intelligence. Nature achieves such a marvelous invention by a long period of
evolution - from invertebrate to vertebrate, from old brain (allocortex) to
new brain (neocortex). To unlock the secret of the neocortex, we study
organizational principles at three levels: neural groups, cortical columns,
and cortical areas. The polychronization neural groups form the physical basis
of associative memory. Sensorimotor interaction at the cortical column leads
to a canonical microcircuit for predictive coding, which can be linked with
Kalman filtering-based solution to SLAM. Using the Tolman-Eichenbaum Machine
(TEM) as the toy model built on two modules (navigation and memory), we
presented a unified understanding of high-level cognitive functions as the
generalization of spatial navigation in latent space. More specifically, we
take object recognition and language comprehension as two concrete examples to
demonstrate the universality of the TEM model. Observing how emergent
properties (i.e., broken symmetry) arise from the polychronization of neurons
at different scales (groups, columns, and areas) is enlightening. When
empowered by motion-induced transformations (again across different scales),
the class of associative memory models serves as a universal solution to the
problem of generalization in natural intelligence.

## References

  * [1] C. Mead, _Analog VLSI and neural systems_. Addison-Wesley, 1989, iSBN 9780201059922.
  * [2] D. Hebb, “The organization of behavior; a neuropsychological theory.” 1949.
  * [3] H. R. Wilson and J. D. Cowan, “Excitatory and inhibitory interactions in localized populations of model neurons,” _Biophysical journal_ , vol. 12, no. 1, pp. 1–24, 1972.
  * [4] G.-q. Bi and M.-m. Poo, “Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type,” _Journal of neuroscience_ , vol. 18, no. 24, pp. 10 464–10 472, 1998.
  * [5] N. Caporale and Y. Dan, “Spike timing–dependent plasticity: a hebbian learning rule,” _Annu. Rev. Neurosci._ , vol. 31, pp. 25–46, 2008.
  * [6] E. M. Izhikevich, “Polychronization: computation with spikes,” _Neural computation_ , vol. 18, no. 2, pp. 245–282, 2006.
  * [7] V. B. Mountcastle, “Modality and topographic properties of single neurons of cat’s somatic sensory cortex,” _Journal of neurophysiology_ , vol. 20, no. 4, pp. 408–434, 1957.
  * [8] A. M. Bastos, W. M. Usrey, R. A. Adams, G. R. Mangun, P. Fries, and K. J. Friston, “Canonical microcircuits for predictive coding,” _Neuron_ , vol. 76, no. 4, pp. 695–711, 2012.
  * [9] “On the computational architecture of the neocortex: Ii the role of cortico-cortical loops,” _Biological cybernetics_ , vol. 66, no. 3, pp. 241–251, 1992.
  * [10] J. C. Horton and D. L. Adams, “The cortical column: a structure without a function,” _Philosophical Transactions of the Royal Society B: Biological Sciences_ , vol. 360, no. 1456, pp. 837–862, 2005.
  * [11] D. Marr, “A theory of cerebellar cortex,” _The Journal of Physiology_ , vol. 202, no. 2, pp. 437–470, 1969.
  * [12] ——, “A theory for cerebral neocortex,” _Proceedings of the Royal society of London. Series B. Biological sciences_ , vol. 176, no. 1043, pp. 161–234, 1970.
  * [13] ——, “Simple memory: A theory for archicortex,” _Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences_ , pp. 23–81, 1971.
  * [14] ——, _Vision: A computational investigation into the human representation and processing of visual information_. MIT press, 1982.
  * [15] A. R. Damasio, “Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition,” _Cognition_ , vol. 33, no. 1-2, pp. 25–62, 1989.
  * [16] R. P. Rao and D. H. Ballard, “Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects,” _Nature neuroscience_ , vol. 2, no. 1, pp. 79–87, 1999.
  * [17] K. Friston, “The free-energy principle: a unified brain theory?” _Nature reviews neuroscience_ , vol. 11, no. 2, pp. 127–138, 2010.
  * [18] J. Hawkins and S. Blakeslee, _On intelligence_. Macmillan, 2004.
  * [19] J. Hawkins, _A thousand brains: A new theory of intelligence_. Basic Books, 2021.
  * [20] A. Zador, S. Escola, B. Richards, B. Ölveczky, Y. Bengio, K. Boahen, M. Botvinick, D. Chklovskii, A. Churchland, C. Clopath _et al._ , “Catalyzing next-generation artificial intelligence through neuroai,” _Nature communications_ , vol. 14, no. 1, p. 1597, 2023.
  * [21] J. Pei, L. Deng, S. Song, M. Zhao, Y. Zhang, S. Wu, G. Wang, Z. Zou, Z. Wu, W. He _et al._ , “Towards artificial general intelligence with hybrid tianjic chip architecture,” _Nature_ , vol. 572, no. 7767, pp. 106–111, 2019\.
  * [22] P. Cisek and B. Y. Hayden, “Neuroscience needs evolution,” p. 20200518, 2022.
  * [23] W. R. Ashby, “An introduction to cybernetics,” 1956.
  * [24] J. K. O’regan, “Solving the” real” mysteries of visual perception: the world as an outside memory.” _Canadian Journal of Psychology/Revue canadienne de psychologie_ , vol. 46, no. 3, p. 461, 1992.
  * [25] R. A. Brooks, “Intelligence without representation,” _Artificial intelligence_ , vol. 47, no. 1-3, pp. 139–159, 1991.
  * [26] J. W. Krakauer, A. A. Ghazanfar, A. Gomez-Marin, M. A. MacIver, and D. Poeppel, “Neuroscience needs behavior: correcting a reductionist bias,” _Neuron_ , vol. 93, no. 3, pp. 480–490, 2017.
  * [27] P. W. Anderson, “More is different: Broken symmetry and the nature of the hierarchical structure of science.” _Science_ , vol. 177, no. 4047, pp. 393–396, 1972.
  * [28] J. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities.” _Proceedings of the national academy of sciences_ , vol. 79, no. 8, pp. 2554–2558, 1982.
  * [29] M. G. Dissanayake, P. Newman, S. Clark, H. F. Durrant-Whyte, and M. Csorba, “A solution to the simultaneous localization and map building (slam) problem,” _IEEE Transactions on robotics and automation_ , vol. 17, no. 3, pp. 229–241, 2001.
  * [30] G. B. Keller and T. D. Mrsic-Flogel, “Predictive processing: a canonical cortical computation,” _Neuron_ , vol. 100, no. 2, pp. 424–435, 2018.
  * [31] J. C. Whittington, D. McCaffary, J. J. Bakermans, and T. E. Behrens, “How to build a cognitive map,” _Nature neuroscience_ , vol. 25, no. 10, pp. 1257–1272, 2022.
  * [32] J. C. Whittington, T. H. Muller, S. Mark, G. Chen, C. Barry, N. Burgess, and T. E. Behrens, “The tolman-eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation,” _Cell_ , vol. 183, no. 5, pp. 1249–1263, 2020.
  * [33] R. V. Raju, J. S. Guntupalli, G. Zhou, M. Lázaro-Gredilla, and D. George, “Space is a latent sequence: Structured sequence learning as a unified theory of representation in the hippocampus,” _arXiv preprint arXiv:2212.01508_ , 2022.
  * [34] R. A. Andersen, L. H. Snyder, C.-S. Li, and B. Stricanne, “Coordinate transformations in the representation of spatial information,” _Current opinion in neurobiology_ , vol. 3, no. 2, pp. 171–176, 1993.
  * [35] A. Pouget and T. J. Sejnowski, “Spatial transformations in the parietal cortex using basis functions,” _Journal of cognitive neuroscience_ , vol. 9, no. 2, pp. 222–237, 1997.
  * [36] M. Graf, “Coordinate transformations in object recognition.” _Psychological Bulletin_ , vol. 132, no. 6, p. 920, 2006.
  * [37] P. W. Anderson and D. L. Stein, _Broken symmetry, emergent properties, dissipative structures, life: are they related?_ Springer, 1987.
  * [38] A. L. Hodgkin and A. F. Huxley, “A quantitative description of membrane current and its application to conduction and excitation in nerve,” _The Journal of physiology_ , vol. 117, no. 4, p. 500, 1952.
  * [39] P. S. Churchland, V. S. Ramachandran, and T. J. Sejnowski, “A critique of pure vision,” 1993.
  * [40] C. F. Hockett and C. D. Hockett, “The origin of speech,” _Scientific American_ , vol. 203, no. 3, pp. 88–97, 1960.
  * [41] J. A. Hertz, _Introduction to the theory of neural computation_. Crc Press, 2018.
  * [42] A. Chemero, “Anti-representationalism and the dynamical stance,” _Philosophy of Science_ , vol. 67, no. 4, pp. 625–647, 2000.
  * [43] P. Haselager, A. De Groot, and H. Van Rappard, “Representationalism vs. anti-representationalism: a debate for the sake of appearance,” _Philosophical psychology_ , vol. 16, no. 1, pp. 5–24, 2003.
  * [44] D. E. Rumelhart, G. E. Hinton, R. J. Williams _et al._ , “Learning internal representations by error propagation,” 1985.
  * [45] J. J. Gibson, “The ecological approach to visual perception: Classic edition,” 1979.
  * [46] H. Moravec, _Mind children: The future of robot and human intelligence_. Harvard University Press, 1988.
  * [47] R. A. Schmidt, T. D. Lee, C. Winstein, G. Wulf, and H. N. Zelaznik, _Motor control and learning: A behavioral emphasis_. Human kinetics, 2018.
  * [48] Z. Durante, Q. Huang, N. Wake, R. Gong, J. S. Park, B. Sarkar, R. Taori, Y. Noda, D. Terzopoulos, Y. Choi _et al._ , “Agent ai: Surveying the horizons of multimodal interaction,” _arXiv preprint arXiv:2401.03568_ , 2024\.
  * [49] D. J. Sumpter, _Collective animal behavior_. Princeton University Press, 2010.
  * [50] D. A. Leopold and N. K. Logothetis, “Multistable phenomena: changing views in perception,” _Trends in cognitive sciences_ , vol. 3, no. 7, pp. 254–264, 1999.
  * [51] A. S. Pillai and V. K. Jirsa, “Symmetry breaking in space-time hierarchies shapes brain dynamics and behavior,” _Neuron_ , vol. 94, no. 5, pp. 1010–1026, 2017.
  * [52] R. L. Graham, B. L. Rothschild, and J. H. Spencer, _Ramsey theory_. John Wiley & Sons, 1991, vol. 20.
  * [53] E. P. Frady and F. T. Sommer, “Robust computation with rhythmic spike patterns,” _Proceedings of the National Academy of Sciences_ , vol. 116, no. 36, pp. 18 050–18 059, 2019.
  * [54] G. Buzsaki, _Rhythms of the Brain_. Oxford university press, 2006.
  * [55] E. M. Stein and R. Shakarchi, _Fourier analysis: an introduction_. Princeton University Press, 2011, vol. 1.
  * [56] H. Furstenberg, _Recurrence in ergodic theory and combinatorial number theory_. Princeton University Press, 2014, vol. 10.
  * [57] J. J. DiCarlo, D. Zoccolan, and N. C. Rust, “How does the brain solve visual object recognition?” _Neuron_ , vol. 73, no. 3, pp. 415–434, 2012.
  * [58] A. Hatcher, _Algebraic topology_ , 2005.
  * [59] P. Walters, _An introduction to ergodic theory_. Springer Science & Business Media, 2000, vol. 79.
  * [60] D. Kondepudi and I. Prigogine, _Modern thermodynamics: from heat engines to dissipative structures_. John Wiley & Sons, 2014.
  * [61] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang, “A tutorial on energy-based learning,” _Predicting structured data_ , vol. 1, no. 0, 2006\.
  * [62] E. T. Jaynes, “Information theory and statistical mechanics,” _Physical review_ , vol. 106, no. 4, p. 620, 1957.
  * [63] E. N. Lorenz, “Deterministic nonperiodic flow,” _Journal of atmospheric sciences_ , vol. 20, no. 2, pp. 130–141, 1963.
  * [64] J. A. Acebrón, L. L. Bonilla, C. J. P. Vicente, F. Ritort, and R. Spigler, “The kuramoto model: A simple paradigm for synchronization phenomena,” _Reviews of modern physics_ , vol. 77, no. 1, p. 137, 2005.
  * [65] E. A. Martens, E. Barreto, S. H. Strogatz, E. Ott, P. So, and T. M. Antonsen, “Exact results for the kuramoto model with a bimodal frequency distribution,” _Physical Review E_ , vol. 79, no. 2, p. 026204, 2009.
  * [66] P. Sterzer, A. Kleinschmidt, and G. Rees, “The neural bases of multistable perception,” _Trends in cognitive sciences_ , vol. 13, no. 7, pp. 310–318, 2009.
  * [67] R. Perin, T. K. Berger, and H. Markram, “A synaptic organizing principle for cortical neuronal groups,” _Proceedings of the National Academy of Sciences_ , vol. 108, no. 13, pp. 5419–5424, 2011.
  * [68] G. Buzsaki and A. Draguhn, “Neuronal oscillations in cortical networks,” _science_ , vol. 304, no. 5679, pp. 1926–1929, 2004.
  * [69] G. Buzsáki, “Neural syntax: cell assemblies, synapsembles, and readers,” _Neuron_ , vol. 68, no. 3, pp. 362–385, 2010.
  * [70] M. Dabagia, C. H. Papadimitriou, and S. S. Vempala, “Computation with sequences in the brain,” _arXiv preprint arXiv:2306.03812_ , 2023.
  * [71] H. Keller, _The story of my life_. WW Norton & Company, 2004.
  * [72] H. C. Barron, R. Auksztulewicz, and K. Friston, “Prediction and memory: A predictive coding account,” _Progress in neurobiology_ , vol. 192, p. 101821, 2020.
  * [73] T. e. a. Salvatori, “Associative memories via predictive coding,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 3874–3886, 2021.
  * [74] K. Friston, “A theory of cortical responses,” _Philosophical transactions of the Royal Society B: Biological sciences_ , vol. 360, no. 1456, pp. 815–836, 2005.
  * [75] B. Millidge, A. Seth, and C. L. Buckley, “Predictive coding: a theoretical and experimental review,” _arXiv preprint arXiv:2107.12979_ , 2021.
  * [76] R. J. Douglas and K. Martin, “A functional microcircuit for cat visual cortex.” _The Journal of physiology_ , vol. 440, no. 1, pp. 735–769, 1991\.
  * [77] S. Haeusler and W. Maass, “A statistical analysis of information-processing properties of lamina-specific cortical microcircuit models,” _Cerebral cortex_ , vol. 17, no. 1, pp. 149–162, 2007.
  * [78] M. Larkum, “A cellular mechanism for cortical associations: an organizing principle for the cerebral cortex,” _Trends in neurosciences_ , vol. 36, no. 3, pp. 141–151, 2013.
  * [79] R. Smith, M. Self, and P. Cheeseman, “Estimating uncertain spatial relationships in robotics,” in _Autonomous robot vehicles_. Springer, 1990, pp. 167–193.
  * [80] H. V. Poor, _An introduction to signal detection and estimation_. Springer Science & Business Media, 2013.
  * [81] V. Fragoso, J. DeGol, and G. Hua, “gdls*: Generalized pose-and-scale estimation given scale and gravity priors,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2210–2219.
  * [82] L. F. Jacobs and F. Schenk, “Unpacking the cognitive map: the parallel map theory of hippocampal function.” _Psychological review_ , vol. 110, no. 2, p. 285, 2003.
  * [83] S. L. Bowman, N. Atanasov, K. Daniilidis, and G. J. Pappas, “Probabilistic data association for semantic slam,” in _International conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 1722–1729.
  * [84] M. Stemmler, A. Mathis, and A. V. Herz, “Connecting multiple spatial scales to decode the population activity of grid cells,” _Science Advances_ , vol. 1, no. 11, p. e1500816, 2015.
  * [85] C. L. Krumhansl and E. J. Kessler, “Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys.” _Psychological review_ , vol. 89, no. 4, p. 334, 1982.
  * [86] G. Carlsson, T. Ishkhanov, V. De Silva, and A. Zomorodian, “On the local behavior of spaces of natural images,” _International journal of computer vision_ , vol. 76, pp. 1–12, 2008.
  * [87] D. H. Hubel, _Eye, brain, and vision._ Scientific American Library/Scientific American Books, 1995.
  * [88] E. I. Moser, E. Kropff, and M.-B. Moser, “Place cells, grid cells, and the brain’s spatial representation system,” _Annu. Rev. Neurosci._ , vol. 31, pp. 69–89, 2008.
  * [89] A. TURING, “The chemical basis of mokphogenesis,” _Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences_ , vol. 237, no. 641, pp. 37–72, 1952.
  * [90] M. C. Fuhs and D. S. Touretzky, “A spin glass model of path integration in rat medial entorhinal cortex,” _Journal of Neuroscience_ , vol. 26, no. 16, pp. 4266–4276, 2006.
  * [91] D. George, R. V. Rikhye, N. Gothoskar, J. S. Guntupalli, A. Dedieu, and M. Lázaro-Gredilla, “Clone-structured graph representations enable flexible learning and vicarious evaluation of cognitive maps,” _Nature communications_ , vol. 12, no. 1, p. 2392, 2021.
  * [92] J. C. Whittington, J. Warren, and T. E. Behrens, “Relating transformers to models and neural representations of the hippocampal formation,” _arXiv preprint arXiv:2112.04035_ , 2021.
  * [93] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [94] W. J. Johnston and S. Fusi, “Abstract representations emerge naturally in neural networks trained to perform multiple tasks,” _Nature Communications_ , vol. 14, no. 1, p. 1040, 2023.
  * [95] R. Davis, H. Shrobe, and P. Szolovits, “What is a knowledge representation?” _AI magazine_ , vol. 14, no. 1, pp. 17–17, 1993.
  * [96] J. Yu, A. I. Cristea, A. Harit, Z. Sun, O. T. Aduragba, L. Shi, and N. Al Moubayed, “Language as a latent sequence: Deep latent variable models for semi-supervised paraphrase generation,” _AI Open_ , vol. 4, pp. 19–32, 2023.
  * [97] R. Bottini and C. F. Doeller, “Knowledge across reference frames: Cognitive maps and image spaces,” _Trends in Cognitive Sciences_ , vol. 24, no. 8, pp. 606–619, 2020.
  * [98] A. D. Redish and D. S. Touretzky, “Cognitive maps beyond the hippocampus,” _Hippocampus_ , vol. 7, no. 1, pp. 15–35, 1997.
  * [99] L. Shapiro, _Embodied cognition_. Routledge, 2019.
  * [100] T. B. Crapse and M. A. Sommer, “Corollary discharge across the animal kingdom,” _Nature Reviews Neuroscience_ , vol. 9, no. 8, pp. 587–600, 2008\.
  * [101] I. Cone and C. Clopath, “Latent representations in hippocampal network model co-evolve with behavioral exploration of task structure,” _Nature Communications_ , vol. 15, no. 1, p. 687, 2024.
  * [102] H. Sohn, N. Meirhaeghe, R. Rajalingham, and M. Jazayeri, “A network perspective on sensorimotor learning,” _Trends in Neurosciences_ , vol. 44, no. 3, pp. 170–181, 2021.