#  Interpretable Semiotics Networks Representing Awareness
11affiliationtext: Independent Researcher, Israel<EMAIL_ADDRESS>22affiliationtext: Independent Researcher, Israel<EMAIL_ADDRESS>
David Kupeev Corresponding Author Eyal Nitcany

###### Abstract

Humans perceive objects daily and communicate their perceptions using various
channels. Here, we describe a computational model that track and simulate
objects’ perception, and their representations as they pass in communication.

We describe two key components of our internal representation (’observed’ and
’seen’) and relate them to familiar computer vision terms (encoding and
decoding). These elements joined together to form semiotic networks, which
simulate awareness in object perception and human communication.

Nowadays, most neural networks are uninterpretable. On the other hand, our
model is free from this disadvantages. We performed several experiments and
demonstrated the visibility of our model.

We describe how our network may be used as preprocessing unit to any
classification network. In our experiments the compound network overperforms
in average the classification network at datasets with small training data.

Future work would leverage our model to gain better understanding of human
communications and personal representations.

Keywords: network awareness, network interpretability, semiotics, dialog,
perceptual classifier, limited training data

###### Contents

  1. 1 Introduction
  2. 2 Related Work
  3. 3 Modeling Person-Person and Person-Object Communication using Semiotic Networks
  4. 4 Implementation of Semiotics Networks Using Convolutional Autoencoder
    1. 4.1 Perception of an object by one person: attractors
    2. 4.2 Perception of an object by two persons: bipartite orbits
      1. 4.2.1 Bipartite orbits of the first type
      2. 4.2.2 Bipartite orbits of the second type
  5. 5 Attractors-based classifier
    1. 5.1 Vanilla classifier
    2. 5.2 Stochastic classifier
  6. 6 Semiotics Interpretation of the model
    1. 6.1 Perception of a visual object by a person
    2. 6.2 Person to person communication
  7. 7 Experimental Results
    1. 7.1 Implementation of Algorithm 1 for person-to-person communication
      1. 7.1.1 Attractors
      2. 7.1.2 Bipatite Orbits
    2. 7.2 Classifiers
  8. 8 Discussion and Further Work
  9. A Definition of the sequence $U$ of the images transmuted between the persons in person-to-person CONN implemented using encoder/decoder operations
  10. B Partitioning a subsequence of $U$
  11. C Proof of Statement 4.3
  12. D Derivation of Eq. 29 from analysis of the meanings a statement.
  13. E Proof of Statement 4.6 (See page 4.6.)

##  1 Introduction

Perception of objects by persons can be thought of as an internal
representation of the outer world, which can be communicated via various
modalities (e.g., text, sound, vision, etc.). Furthermore, the same object can
be described in different channels. For example, an image of a dog, or barking
sound would set us to believe that a dog is around.

Perception had a few properties, which are in general agnostic to the modality
of the perceived input channel. First, perception is mostly subjective. This
means that a specific object that is perceived in on manner, may be perceived
differently by another person. In other words, two persons may have different
internal representation of the same object. For example, two persons that see
a dog might think that this is a nice dog (the first person) or a frighten dog
(the second). Although, they both ’observe’ the same object (dog), they attend
to different properties and thus may ’see’ other aspects of it. These
’observe’ and ’seen’ representations are the building block in our model and
are used to mimic human perception. They enable one to observe an object and
transform (’see’) it. Furthermore, this process can be identified also when
only a single person is involved. We call it an ’internal cycle’, and it
occurs during the process of perception of an object. During this process, an
object is being perceived (observed), projected onto the ’internal space’
(’seen’) and then this representation is being used as an observed input to
generate another internal representation in a cycle, till a termination of the
perception act. It all happens internally and thus can reflect changes that
are seen from the outside with regards to the internal representation of an
object perceived by a person.

We note that similar process occurs for different modalities such as text,
sound, visual (image), etc. and when various modalities are combined. For
example, when sound and image are involved together to generate a perception.
This may occur for example, when a person hears barks and then after a while
sees a dog and conclude that this dog was barking.

Note that the internal personal representation is not revealed. Yet, another
abstract level is used and serve as a common basis for communication between
persons/systems.

People are very good in object perception under various conditions. For
example, you can probably identify your car under different light conditions,
and some would be able to recognize their car just by hearing its “unique”
soundings. Computer vision algorithms that are specialize in classification
would like to possess such a property and indeed many neural networks are
trained with various inputs and are agnostic to many differences such as
light, rotation, etc.

The mathematical model of human communication presented in this article was
developed to represent the existance of the objects seen by a person, as well
as the existence of objects the person is aware of them as the seen. Further
the mathematical relations have been obtained describing other semiotic
phenomena of human-object and human-human communication (Sec. 6). It worth to
note that initially, these new aspects were not the focus of our attention.
The ability to describe supplementary semiotic phenomena testifies to
effectiveness of the model.

The contributions of this paper are as follows:

  * •

Up to our understanding our research is the first attempt to model image
visual perception jointly with the the derived inter-person
communication[Communication_textbook].

  * •

We model human perception using a sequence of ’observed’ and ’seen’
personalized images. This provide actual interpretability of the states of the
modeling network.

  * •

Through the paper we consider communication either between person or
internally ’in the person’. However, ’person’ should be interpreted in general
sense, meaning to be any sort of system including computer system. On the same
note, the processes described in this paper support both internal and external
communication through a unified equation. The details for implementing in
different systems (e.g. internal representation of object in a person, or
communication between two persons) can differ.

  * •

We model the ’observed to seen’ operation as composition of encoder and
decoder operations of convolution autoencoders. This allows to represent an
act of he object perception as a sequence of iterations converging to
attractor [Belkin].

  * •

Our model describes several semiotics phenomena of person-object and person to
person communication.

  * •

Up to our understanding we introduce the notion of bipartite orbits in
dynamics systems.

  * •

We develop an attractor based stochastic classifier for classical computer
vision classification tasks. The classifier is interpretable and shows
promising performance compared to standard methods, when dealing with limited
training datasets.

##  2 Related Work

Surprisingly, there is limited research available on simulating person-to-
person communication. Osgood-Schramm[Osgood_Schramm_model] introduces a cyclic
model of communication between humans. In their encoder-decoder fashion model,
the encoder outputs serve as the transmitted images. In contrast, our model
takes a different approach by utilizing encoder outputs as an internal
representation of an individual’s input perception.

A few years ago, Google introduced the DeepDream network [DeepDream], which
bears some resemblance to our work in terms of the concept of observed and
seen images and the cycle between them. In their work, the images representing
what a person sees in the input image are treated as input to the network. In
our work, on the other hand, we simulate the seen images as the network’s
output. This fundamental difference accounts for the fact that while delving
deep into DeepDream often produces unrealistic ’dream’ images, our approach
tends to generate more realistic ’normal’ images. Additionally, DeepDream can
be viewed as representing one person within our context, where its observe-
seen cycle is internal. Our model extends and generalizes this methodology to
simulate communication.

Our network is implemented using the encoding-decoding operations of an
autoencoder, which ultimately converge to attractors in its final version
[Belkin]. The key difference between our classifier and approaches that
utilize denoising autoencoders [adv_autoenc, adv_autoenc_2] lies in the
iterative nature of the encoding and decoding operations, which leads to
convergence to attractors.

Our model is employed to model the semiotic properties of person-object
perception and person-to-person communication. Furthermore, it can be utilized
in computer vision classification tasks. By leveraging our observed-to-seen
model, we have created an image classifier that exhibits high interpretability
and performs well with small training datasets.

In [paper_17], attractors were utilized for classification in the field of
speech recognition. In [B2_paper], the attractor-based classifier is employed
to estimate the normalized entropy [34] of the probability vector. This
approach is used to detect novel sceneries, such as out-of-distribution or
anomaly samples, rather than specifically focusing on the classification task.

Our classifier is stochastic and builds upon the vanilla version, which is
non-stochastic. The vanilla version represents an input sample as a single
convergent point in the image space, resulting from encoder and decoder
operations. In contrast, in [paper_17], a sample is represented as a series of
convergence points in the latent space, obtained through MC dropout.

Modern networks, including those employed for classification tasks, commonly
offer confidence values alongside the predicted output class. These confidence
values can be obtained through various means, such as utilizing historical
information or running the network multiple times, such as in ensemble
approaches. In our approach, our focus lies in enriching the attractor-based
representation by introducing minor perturbations during the convergence
process, specifically in the image domain. Our main objective is to enhance
the expressiveness of the attractor-based representation rather than
estimating confidence.

Specifically, in our stochastic classifier, an input sample is represented by
a set of attractors that are in close proximity to the sample, thereby
enhancing the informativeness of the attractor-based representation. The
construction of the attractor sets involves randomized iterative alternations
of the samples in the image domain. Referring to [paper_17], the alternation
of samples during recursive autoencoder operations is achieved using Monte
Carlo (MC) dropout. Notably, a meaningful distinction arises between our
attractor representation and dropout [paper_17]. The dropout mechanism
generates outputs that represent known samples with similar representations
and unknown ones with dissimilar representations. However, our representation
typically assigns different attractors to all input samples, including those
used in training, thereby enhancing the informativeness of the attractor-based
representation.

Technically, our representation may resemble the SIFT approach [SIFT]. There,
instead of considering a specific pixel, sift considers the neighboring area
of the pixel, known as the ’vicinity’, where the histogram representations for
gradient directions are calculated. In our approach the ’histogram bins’ are
comprised of the results of convergence of the algorithm and are not
predefined.

Our work has some common ground with RNN networks. In both, an internal state
is preserved and is used and updated when new inputs are being processed. In
this light, our model can be examined as a few RNN networks, each representing
one person, that communicate with each other. In classical RNN networks, the
internal state can receive any value (with some implementation detail
limitations). On the other hand, our model attempt to preserve its internal
model within a certain ’pre-defined’ set.

Our model introduces innovative techniques for analyzing and interpreting
visual information in a social context. By integrating person-to-person
communication cues and object perception capabilities, our approach aims to
model social perception of visual objects. In the subsequent sections, we will
provide a detailed description of our model and discuss the experimental
results that demonstrate its effectiveness.

##  3 Modeling Person-Person and Person-Object Communication using Semiotic
Networks

In this section we introduce the Concious Neural Network (CONN) for modeling
communication between persons perceiving visual images. Consider two persons
$P_{1}$ and $P_{2}$ (Fig. 1 (A)). The first person tends to see in all input
images the cats, the second the dogs. Assume the persons perceive the images
iteratively: the $P_{1}$ converts an observed input image $Im$ to an internal
image with some features of the cat, refereed as ’seen’ image for the
observed, and then continues iteratively recognize ’catness’ in the seen
image. Similarly, the person $P_{2}$ tends to see ’dogness’ in the perceived
images. This is expressed as

$Im=obs_{1}\xrightarrow{O2S_{P}}seen_{1}=obs_{2}\xrightarrow{O2S_{P}}seen_{2}=obs_{3}\xrightarrow{O2S_{P}}\dots,$
(1)

where the seen images obtained at each iteration became the observed for the
next iteration, and $P\in\\{P_{1},P_{2}\\}$.

The persons communicate, representing to each other their seen images,
obtained from the input images after a certain number of iterations. In
general, the transmitted representations do not have to be of image modality
(they may be, for example, textual descriptions). Meanwhile, we confine
ourselves to the case where these representations are themselves the internal
seen images.

The introduced scheme is a particular case of a ’DoWheel’ scheme, where the
images of Eq. 1 form the person aligned stream (PAS) associated with the
persons [AE_paper].

Figure 1:  (A) Person-to-person CONN. (B) Person-to-person CONN with observed-
to-seen transformation implemented as composition of encoder and decoder
operations shown in blue and green respectively.

Our scheme allows to model perception of a visual object by a person as a
private case of person to person communication. The Fig. 2 illustrates
perception of an object (an apple) by a person. The $O2S$ transformation
associated with an apple does not depend on the input and always returns ’a
raw’ (non-personalized) image of the apple. The image becomes the observed by
the person and is further personalized in $obs\rightarrow seen$
transformations.

To reflect uncertainty involved in human perception [paper_prob_perception,
Rubin_vase2] we add some randomness to Eq. 2 as described in the following
section.

Figure 2:  Perception of an object by the person may be treated as a private
case of the person to person communication.

##  4 Implementation of Semiotics Networks Using Convolutional Autoencoder

In general, the $obs\rightarrow seen$ person-dependent transformation of Eq. 1
may be modeled in different ways. We model this transition using $enc$ and
$dec$ operations of convolution autoencoders performing transformations from
the image space to a latent space and back:

$obs\rightarrow
seen:\,\,obs\xrightarrow{enc}enc(obs)\xrightarrow{dec}seen=dec(enc(obs)).$ (2)

Note that both ’observed’ and ’seen’ representation are of the object modality
and not in latent space. Transition from ’observed’ to ’seen’ is through
latent based representations.

Using these operations we construct a concious neural network for person-to-
person communication. The network is illustrated at Fig.1 (B), its work is
described by Algorithm 1.

Algorithm 1 A concious neural network for person-to-person communication of
$k$ transactions.
Input: an image $Im_{1}$ which is related to person $P_{1}$ (marked by ’init’
in Fig. 1 (B))
Output: A sequence of transaction images $Im_{1},Im_{2},\dots Im_{k},\dots$.

  1. 1.

Set $iter=1$;

  2. 2.

Set $person\\_id=1$;

  3. 3.

While $iter<=n\\_iters$ do:

    1. 3.1.

Use $person\\_id$ parameters ($nsteps_{person\\_id}$);

    2. 3.2.

Perform $nsteps_{person\\_id}$ encoding/decoding iterations of Eq. 2 on
$Im_{iter}$ to receive the image representation (current $Im$).

    3. 3.3.

Decode the previous encoding result $lat$ ($\text{current }Im=dec(lat)$) to
receive $Im_{iter+1}$ ($Im_{iter+1}=\text{current }Im$ after this operation).

    4. 3.4.

Increase $iter$ by 1 and change $person\\_id$ to other $person\\_id$

    5. 3.5.

Send $Im_{iter}$ to the updated person and return to step 2

###  4.1 Perception of an object by one person: attractors

In this subsection, following in Sect.,3, the perception of an object by a
person will be modeld s a specific case of person-to-person communication.
Furthermore, we model the transformation from ’observed’ to ’seen’ as a
sequence of encoder/decoder operations of an autoencoder. This will lead us to
the fixed point property [Belkin], which will be essential in our modeling of
the semiotics of human perception in Sect.,6.

As shown in Sect. 3, the perception of an object by a person can be considered
as a specific case of person-to-person communication. When modeling the
perception of an object, each internal communication cycle of images
associated with a person begins with the same observed image $Im$ (Fig. 2).
This cycle is described as follows:

$Im=obs_{1}\xrightarrow{O2S_{P}}seen_{1}=obs_{2}\xrightarrow{O2S_{P}}seen_{2}=obs_{3}\xrightarrow{O2S_{P}}\dots,.\xrightarrow{O2S_{P}}seen_{n},$
(3)

where $n=nsteps$ represents the length of the internal communication loop of
the person.

Using the notation of Eq. 2 we may write this sequence as:

$Im\rightarrow dec(enc(Im))\rightarrow,\dots\rightarrow[dec(enc)]^{n}(Im),$
(4)

where $[dec(enc)]^{n}$ denotes $n$ compositions of $dec(enc)$ function.

The process takes an input image $Im$, encodes it into the latent space using
$enc(Im)$, and then decodes it back to the image space. This encoding/decoding
procedure is repeated $n$ times, resulting in an image representation in the
original modality. As $n$ approaches infinity, this sequence has been studied
in [Belkin], where the phenomenon of attractors has been identified. It has
been empirically shown that for overparameterized autoencoders, as $n$
approaches infinity, the sequence described in Eq. 4 converges to an
attractor. We have also observed similar phenomena for autoencoders that are
not overparameterized, where the CONNs map input examples to attractors (see
Sect.7).

For an input image $Im$ be an input image we call the final representation of
$Im$ in the image space

$\widehat{x}=\lim_{n\to\infty}[dec(enc)]^{n}(Im),$ (5)

if such limit exist, the percept image of $Im$.

From [Belkin] above follows that there is empirical evidence that the percept
image for overparameterized autoencoder is attractor.

One can observe that for the percept image $\widehat{x}$, there holds the
fixed point property:

$dec(enc)(\widehat{x})=\widehat{x}.$ (6)

This equation indicates that applying the encoding and decoding operations to
the percept image results in the same image.

We will use this fixed point property of the percept image in Sect. 6.1 where
semiotic properties of the person-object relationship are studied.

###  4.2 Perception of an object by two persons: bipartite orbits

In this subsection, we will delve into inter-person communication. We will
study the asymptotic properties of the image sequence exchanged between
individuals in our CONN model (Sect. 3). These properties will play an
important role in our subsequent exploration of the semiotic properties of
interpersonal communication in Sect.6.2.

In contrast to the previous subsection, our exposition will not limited to
sequences implemented through encoder/decoder operations. Instead, it is
applicable to sequences defined in terms of general metric spaces. The
representation of the image sequence transmitted between individuals in the
person-to-person CONN, which is implemented using encoder/decoder operations,
can be found in Appendix A.

What periodicity is being referred to? One may assume that the sequence of the
images ’perceived’ by the person as reflected in Eq. 1 of our CONN model
converged to ’attractors’. For example, for a a ’dog-like’ person, the
sequence is converged to a dog image. However, when more than one person is
involved, this assumption may not hold anymore for the whole sequence of
intertrasmitted images, because there is no guarantee that both persons share
the same ’attractors’. For example, if one is a ’dog-like’ person (i.e., the
attractors are comprised of dogs images only) and the other is a ’cat-like’
person, then a joint attractor is of a low choice. A ’dog-like’ person is
unllikely ’to see’ a cat image and vise versa for the ’cat-like’ person.

Although the sequence of intertransmitted images, generally speaking, does not
converge as a sequence of elements, our empirical findings, together with
theoretical endorsement, indicate that this is an asymptotically periodic
sequence [asymptotically_periodic], as we shall see later.

Specifically, we will identify two types of periodicity in the sequence of
transmitted images between the persons. Both types are observed when the
external communication parameter of the algorithm (the number of information
exchanges between the persons) tends to infinity. The difference lies in
whether the internal communication parameters (the number of observed/seen
transformations in Eq. 1) also tend to infinity. These two types of
periodicity are studied in Sects. 4.2.1 and 4.2.2, respectively.

Figure 3:  Illustration of bipartite orbit for (A) The bipartite orbit of the
first type, for $niter$ tending to infinity (Definition 4.1), (B) The
bipartite orbit of the second type, for $niter$, $nsteps_{1}$, and
$nsteps_{2}$ tending to infinity (Definition 4.4). Elements of the bipartite
orbit are shown in red. .

From a mathematical perspective, we will examine sequences consisting of
compositions of functions $F_{1}$ and $F_{2}$, applied alternately to the
elements of the sequence. The only requirement for $F_{1}$ and $F_{2}$ is that
they are functions operating in metric spaces.

####  4.2.1 Bipartite orbits of the first type

In the previous section (4.1), attractors were considered, as modeling
perception of an object by one person. Interestingly, when human communication
is simulated, an assymptotically periodic sequence of inter-person transmitted
images had been identified. We denote this property as convergence to
bipartite orbits.

Formally, let $F_{P_{1}}$ and $F_{P_{2}}$ be two continuous functions
$X\rightarrow X$ where $X$ is a metric space and $Im\in X$ to be a initial
point (’an image’). Consider a sequence $W(Im)$ starting with $Im$ and
consisting of subsequent application of $nsteps_{1}$ times of $F_{P_{1}}$,
then $nsteps_{2}$ times of $F_{P_{2}}$, then $nsteps_{1}$ times of $F_{P_{1}}$
etc., where $nsteps_{1}$ and $nsteps_{2}$ are given numbers (representing the
internal number of steps for convergence of Persons 1 and 2). If we denote

$\begin{array}[]{l}S_{1,P_{1}}=Im,\,F_{P_{1}}(Im),\dots,[F_{P_{1}}]^{nsteps_{1}}(Im);\\\\[3.00003pt]
\;\;T_{1,P_{1}}=[F_{P_{1}}]^{nsteps_{1}}(Im);\\\\[4.49997pt]
S_{2,P_{2}}=T_{1,P_{1}}\,F_{P_{2}}(T_{1,P_{1}}),\dots,[F_{P_{2}}]^{nsteps_{2}}(T_{1,P_{1}});\\\\[3.00003pt]
\;\;T_{2,P_{2}}=[F_{P_{2}}]^{nsteps_{2}}(T_{1,P_{1}});\\\\[4.49997pt]
S_{3,P_{1}}=T_{2,P_{2}}\,F_{P_{1}}(T_{2,P_{2}}),\dots,[F_{P_{1}}]^{nsteps_{1}}(T_{2,P_{2}});\\\\[3.00003pt]
\;\;T_{3,P_{1}}=[F_{P_{1}}]^{nsteps_{1}}(T_{2,P_{2}});\;\dots\;\;\;\;\;.\\\
\end{array}$ (7)

then one may write

$W(Im)=S_{1,P_{1}},\,S_{2,P_{2}},\,S_{3,P_{1}},\dots
S_{iter,P_{i(iter)}},\dots$ (8)

.111This representation is identical to Eq. 33, but there, $W$ was defined in
terms of encoding and decoding operations.

Now focus on the ’final’ internal elements $T_{1,P_{1}}$, $T_{2,P_{2}}$,
$T_{3,P_{1}}\ldots$ in this representation (see Eq. 7). These elements
represent the final image of each person, that is later sent to the other
person. They are marked in green in Fig. 4. These elements comprise a sub-
sequence $U$ of $W$:

$\begin{multlined}U(Im,nsteps_{1},nsteps_{2})=Im\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}T_{1,P_{1}}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}\\\
T_{2,P_{2}}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}T_{3,P_{1}}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}T_{4,P_{2}}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}\dots\;.\end{multlined}U(Im,nsteps_{1},nsteps_{2})=Im\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}T_{1,P_{1}}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}\\\
T_{2,P_{2}}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}T_{3,P_{1}}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}T_{4,P_{2}}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}\dots\;.$
(9)

Figure 4:  Illustration of bipartite convergence of the first type (Definition
4.1). A bipartite convergent sequence $U$ (Eq. 35) is shown in green ’within’
sequence $W$ (Eq.8). Starting from the $N(\epsilon)$-th iteration sequence $U$
may be partitioned by subsequences $U=L_{0},L_{1},\dots$ such that for every
$m_{1},m_{2}>=0$ $d_{SAD}(L_{m_{1}}-L_{m_{2}})<\epsilon$.

Denote

$\begin{multlined}F_{1}=[F_{P_{1}}]^{nsteps_{1}},F_{2}=[F_{P_{2}}]^{nsteps_{2}}.\end{multlined}F_{1}=[F_{P_{1}}]^{nsteps_{1}},F_{2}=[F_{P_{2}}]^{nsteps_{2}}.$
(10)

Figure 5:  Partitioning $U$ by segments of $K=3$ and $K=4$.

For any $K>0$ one may partition $1+(n+1)*K$ members of the sequence by $Im$
following the matrix of $(n+1)$ rows and $K$ columns (see Appendix B). Like
this, for any $K>0$ we may partition the whole sequence by $Im$ following
subsequent segments of length $K$, see Fig. 5. Our convergence will be defined
by way of columns of the matrix whose lines are the $K$-length segments of
such partitioning.

Specifically, for any $K>0$ the sequence may be written as follows:

$\begin{array}[]{@{}cccccccc@{}}\lx@intercol
<EMAIL_ADDRESS>T_{1+0\cdot K,P_{1}}&\xrightarrow{F_{2}}&T_{1+0\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}&\cdots&\xrightarrow{F_{2}}&T_{1+0\cdot
K+K-1,P_{2}}&\xrightarrow{F_{1}}\\\ T_{1+1\cdot
K,P_{1}}&\xrightarrow{F_{2}}&T_{1+1\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}&\cdots&\xrightarrow{F_{2}}&T_{1+1\cdot
K+K-1),P_{2}}&\xrightarrow{F_{1}}\\\ \dots&&&&&&&\\\ T_{1+n\cdot
K,P_{1}}&\xrightarrow{F_{2}}&T_{1+n\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}&\cdots&\xrightarrow{F_{2}}&T_{1+n\cdot
K+K-1,P_{2}}&\xrightarrow{F_{1}}\\\ \dots\;\;.&&&&&&&\\\ \end{array}$ (11)

For $j=0,\dots,K-1$ the $j$’s column of this representation is written as

$C_{j}(nsteps_{1},nsteps_{2})=\begin{pmatrix}T_{1+0\cdot
K+j,P_{i(j+1)}}\\\\[4.49997pt] T_{1+1\cdot K+j,P_{i(j+1)}}\\\ \dots\\\
T_{1+n\cdot K+j,P_{i}{(j+1)}}\\\ \dots\\\ \end{pmatrix},$ (12)

where $i()$ is defined in Eq. 31 and the column comprises the sequence indexed
by $n$.

######  Definition 4.1 (Bipartite convergence of the first type to orbit).

A sequence $U(Im,nsteps_{1},nsteps_{2})$ is a bipartite convergent sequence of
the first type converging to orbit $(b_{j}\;|\;j=0,\dots,K-1)$ if for a
certain $K>0$ and for every $j=0,\dots,K-1$ column
$C_{j}(nsteps_{1},nsteps_{2})$ as a sequence converges to $b_{j}$ at $n$
tending to infinity.

The definition is illustrated at Fig 3. At Fig 4 a bipartite convergent
sequence $U$ is shown in green ’within’ the sequence $W$ (Eq.8).

The following statement is obvious.

######  Statement 4.2.

A sequence $U=U(Im,nsteps_{1},nsteps_{2})$ is a bipartite sequence of the
first type if and only if it is an asymptotically $K$-periodic
sequenceLABEL:asymptotically_periodic w.r.t. index $iter$ (Eq. 8). In this
case the sequence comprising the period of $U$ is the bibartite orbit of $U$.
∎

For a bipartite sequence starting with an image $Im$ alternation of $nsteps$
parameters typically leads to alternation of the period, as illustrated in SI
Text A.

Under what conditions a sequence $U(Im,nsteps_{1},nsteps_{2})$ is a bipartite
convergent sequence of the first type? Although the existence of such orbits
was not formally proven, in our experiments with the autoencoder generated
images (we did not specify that the autoencoder is overparameterized), we
empirically observed convergence to the bipartite orbits for every initial
$Im$, $nsteps_{1}$, and $nsteps_{2}$ (Sect. 7). In addition, we developed a
simplified computational model for simulating inter-person communication (see
SI Text B). The running of the model consistently demonstrates convergence to
what can be referred to as the first type orbit of the simplified model.

For a bipartite sequence starting with an image $Im$ alternation of $nsteps$
parameters typically leads to alternation of the period length, as illustrated
in SI Text A.

For the first type bipartite orbits there hold the following statement.

######  Statement 4.3.

The elements of bipartite orbits $(b_{0},b_{1},\dots,b_{K-1})$ of the first
type satisfy the following properties:

  1. 1.

They form a loop with respect to alternating $F_{1}$, $F_{2}$ operations:

$b_{0}\xrightarrow{F_{2}}b_{1}\xrightarrow{F_{1}}\dots\xrightarrow{F_{2}}b_{K-1}\xrightarrow{F_{1}}b_{0},$
(13)

where $F_{1}$, $F_{2}$ are defined as in Eq. 10:

$\displaystyle F_{1}=[F_{P_{1}}]^{nsteps_{1}},$ $\displaystyle
F_{2}=[F_{P_{2}}]^{nsteps_{2}}.$

  2. 2.

These elements are also alternating fixed points of functions $G_{1}$,
$G_{2}$:

$\displaystyle G_{1}(b_{h})=b_{h},\text{ for even }h,$ (14) $\displaystyle
G_{2}(b_{h})=b_{h},\text{ for odd }h.$

where

$\begin{gathered}G_{1}=[\,F_{1}\cdot F_{2}\,]^{K/2},\\\ G_{2}=[\,F_{2}\cdot
F_{1}\,]^{K/2}.\end{gathered}$ (15)

###### Proof.

See C. ∎

It should be noted that the elements comprising the bipartite orbits of the
first type are not necessarily the attractors of $F_{1}$ or $F_{2}$. This is
due to ’non-deep’ character of the intra-person communication (the
$nsteps_{1},nsteps_{2}$ are not tending to infinity).

We call $G_{1}$, and $G_{2}$ the personalized operators of the seen in the
dialog and consider their interpretation in Sect.6.2 where the semiotics
aspects of the person to person communication will be studied.

####  4.2.2 Bipartite orbits of the second type

In this subsection, similar to the previous one, we will explore the periodic
properties of the converging sequences, which will be further interpreted from
a semiotic perspective in Sect. 6.2.

Specifically, we will consider the behavior of the ’internal’ person
parameters, $nsteps_{1}$ and $nsteps_{2}$, as they tend towards infinity in
$U(Im,nsteps_{1},nsteps_{2})$, as defined in Eq. 9. These parameters represent
the steps involved in converging towards the ’innate representations’. These
representations, implemented in our model using autoencoders as described in
Sect.,4, are characterized as fixed points.)

Additionally, as mentioned earlier, our assumption regarding $F_{1}$ and
$F_{2}$ is that they are functions operating within metric spaces.

The convergence of the sequence $U(Im,nsteps_{1},nsteps_{2})$, as discussed in
Sect. 4.2.1, describes its behavior as $iter$ of Algorithm, 1 (or
equivalently, Eq. 8) tends to infinity. Now, let us consider the behavior of
$U$ when both $nsteps_{1}$ and $nsteps_{2}$ also tend to infinity.

######  Definition 4.4 (Bipartite convergence of the second type to orbit).

A sequence $U(Im,nsteps_{1},nsteps_{2})$ is a bipartite convergent sequence of
the second type converging to orbit $(b_{j}\;|\;j=0,\dots,K-1)$ if for a
certain $K>0$ and for every $j=0,\dots,K-1$ column
$C_{j}(nsteps_{1},nsteps_{2})$ (Eq.12) as a sequence converges to $b_{j}$ at
$n$, $nsteps_{1}$, and $nsteps_{2}$ tending to infinity.

Likewise the convergence of the first type, the definition stipulates that the
behavior of $U$ is like the shown at Fig. 3.

Like Statement 4.2 the following statement is obvious.

######  Statement 4.5.

A sequence $U=U(Im,nsteps_{1},nsteps_{2})$ is a bipartite sequence of the
second type if and only if it is an asymptotically $K$-periodic sequence
w.r.t. $nsteps_{1}$, $nsteps_{2}$, and $iter$. In these cases the sequence
comprising the period of $U$ is the bibartite orbit of $U$. ∎

In our experiments we empirically observed convergence to the bipartite orbits
of the second type for every initial $Im$. (Sect. 7). The same phenomena was
observed for a simplified model of the inter-person communication (see SI Text
B).

The following questions arise.

  1. 1.

Under what conditions $U$ is as a bipartite sequence of the second type?

  2. 2.

What are the properties of the bipartite sequence of the second type?

The following statement answers to these questions under certain natural
conditions.

######  Statement 4.6.

If both $F_{P_{1}}$, $F_{P_{2}}$ satisfy certain natural conditions (See E.1),
then $U(Im,nsteps_{1},nsteps_{2})$ a bipartite convergent sequence of the
second type converging to orbit orbit $B=(b_{0},b_{1},\dots b_{K-1})$ of the
second type. There are hold the following properties.

  1. 1.

$B$ is comprised of alternating attractors of $F_{P_{1}}$, $F_{P_{2}}$ (see
Eq. 34).

In particular there hold the fixed point relations:

$\displaystyle F_{P_{1}}(b_{h})=b_{h},\text{ for odd }h,$ (16) $\displaystyle
F_{P_{2}}(b_{h})=b_{h},\text{ for even }h,$

  2. 2.

They form a loop with respect to alternating $\widehat{F_{1}}$,
$\widehat{F_{2}}$ operations:

${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}b_{0}}\xrightarrow{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\widehat{F_{2}}}}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}b_{1}}\xrightarrow{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\widehat{F_{1}}}}\dots\xrightarrow{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\widehat{F_{2}}}}\,{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}b_{K-1}}\,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\xrightarrow{\widehat{F_{1}}}}\,{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}b_{0}},$
(17)

where

$\widehat{F_{i}}(Im)=\lim_{n\to\infty}[F_{P_{i}}]^{n}(Im));$ (18)

  3. 3.

These elements are also alternating fixed points of functions
$\widehat{G_{1}}$, $\widehat{G_{2}}$:

$\displaystyle\widehat{G_{1}}(b_{h})=b_{h},\text{ for even }h,$ (19)
$\displaystyle\widehat{G_{2}}(b_{h})=b_{h},\text{ for odd }h,$

where

$\displaystyle\widehat{G_{1}}=[\,\widehat{F_{1}}\cdot\widehat{F_{2}}\,]^{K/2},$
(20)
$\displaystyle\widehat{G_{2}}=[\,\widehat{F_{2}}\cdot\widehat{F_{1}}\,]^{K/2}.$

###### Proof.

See E.1. ∎ ∎

One may that Eqs. 17 and 19 are the counterparts of Eqs. 13 and 14 from Sect.
4.2.1 respectively. The essential difference between the two types of the
orbits is that the orbits of the first type are comprised of attractors. The
semiotics aspects of the both types of the orbits are considered in Sect. 6.
In particular, we demonstrate that the $b_{h}$ values with even (resp. odd)
values of $h$ correspond to what person $P_{1}$ (resp. $P_{2}$) is aware of as
seen in the dialogue. These elements are represented by the red (resp. blue)
portions in Eq. 17.

In SI Text C we study monocular algebras associated with the bipartite orbits
of the both types.

Finally, Table 1 summarizes the properties of bipartite orbits.

Table 1: Properties of Bipartite Orbits # |  Property | Type of bipartite orbit |  References
---|---|---|---
The First |  The Second
1 |  Infinity limit parameters |  $nsteps_{1}$, $nsteps_{2}$ |  $niter$, $nsteps_{1}$, $nsteps_{2}$ |
2 |  Semiotics interpretation |  |  Sect. 6
3 |  Attractors of $F_{P_{1}},$ $F_{P_{2}}$ |  Typically not |  Yes |  Eq. 34, Eq. 16
4 |  Alternating cyclic transition functions |  $F_{1},F_{2}$ |  $\widehat{F_{1}},\widehat{F_{2}}$ |  Eq. 10, Eq. 13, Eq. 17
5 |  Fixed points identities functions |  $G_{1},G_{2}$ |  $\widehat{G_{1}},\widehat{G_{2}}$ |  Eq. 15, Eq. 14, Eq. 19
6 |  Algebraic operation for theme loop formation |  $FM$ |  $\widehat{FM}$ |  SI Text C
7 |  Validation of existence |  Observed experimentally |  Proven under certain natural conditions. Also, observed experimentally. |  Statement 4.6

##  5 Attractors-based classifier

In the previous sections, we introduced a model for person-to-person
communication that relies on observed-to-seen transformations that converge to
attractors. This model allows us to describe several semiotic phenomena
related to Person-Object and Person-Person communication (see Section 6). In
this section, we demonstrate that the model can also be applied to classical
computer vision tasks, such as classification.

Our objective is to investigate whether our person-object communication model,
which is based on attractors [Belkin], can be applied to classification tasks.
We introduce an attractor-based classifier network and demonstrate its
effectiveness for tasks with small training datasets. Our classifier provides
interpretability, enabling us to understand how the network perceives the
input image as a whole. However, the effectiveness and interpretability come
at a cost of longer inference time, as input samples take longer to converge
to attractors.

In the following subsections, we describe two types of attractor-based
classifiers: vanilla and stochastic. While the vanilla classifier is more
suitable for representing the semiotic properties of communication (see Sect.
6), the stochastic variant demonstrates superior performance properties.
Specifically, the stochastic attractor-based classifier shows improved
classification performance, making it a more suitable choice for
classification tasks with small training datasets.

###  5.1 Vanilla classifier

In this subsection, we introduce the conversion of an image classifier $M$
into a vanilla CONN classifier. This conversion can be viewed as the addition
of a new ’perceptual’ layer that precedes the input of $M$.

Overall, our approach involves applying the same transformation to the labeled
sets of training and test examples, $TR$ and $TE$, respectively. This
transformation converts these sets into sets $ATR$ and $ATE$. These
transformed sets consist of attractors and are used as the new training and
test sets for the classifier.

First consider a mapping of image samples to attractors using an autoencoder.
In [Belkin], it was empirically demonstrated that, for an overparameterized
autoencoder, for an arbitrary image $Im$, the sequence composed of encoder and
decoder operations of the autoencoder typically converges to the input image
stored as an attractor: The attractor for an overparameterized autoencoder is
defined as:

$\widehat{F}(Im)=\lim\limits_{n\to\infty}[dec(enc)]^{n}(Im).$ (21)

Given a training set $TR$, we first train an overparameterized autoencoder $A$
to memorize $TR$ (without using the labels of $TR$). We then construct a new
training set comprised of attractors:

$ATR=\\{\widehat{F}(Im)\,|\,Im\in TR\\}$

We assign the same labels to the images $\widehat{F}(Im)$ as to $Im$. Due to
memorization, $ATR$ is identical to $TR$. (In the following subsection, we
define the stochastic classifier, which follows the general CONN classifiers
framework. Notably, in the stochastic classifier, $ATR$ differs from $TR$.)
Set $ATR$ is used to train the benchmark classifier $M$.

During inference, test data samples $Im$ from $TE$ are converged to attractors
(using Eq. 21), resulting in a set $ATE$ consisting of attractors from $ATR$,
including potentially spurious attractors [Belkin]. Finally, the elements of
$ATE$ are passed to the classifier $M$ for prediction.

In Fig. 6 (A), we can see a vanilla classifier $C$ comprising of an
overparameterized autoencoder $A$ and a benchmark classifier $M$. For the
vanilla classifier, transformation $F$ refers to the conversion to attractors,
as defined in Eq. 21.

Figure 6:  (A): The attractor-based classifier $C$ is constructed by first
converting the training dataset $TR$ and test dataset $TE$ samples into new
training and test datasets $ATR$ and $ATE$, respectively. This conversion
takes place prior to the training of the benchmark classifier $M$, which is
subsequently trained on the $ATR$ dataset and tested on the $ATE$ dataset. The
conversion is achieved through the application of transformation $F$, shown in
(C). (B): The benchmark classifier $M$ is used for comparison with the
attractor-based classifier (refer to Sect. 7). (C): For the vanilla classifier
(refer to Sect. 5.1), the transformation $F$ denotes the transformation
$\widehat{F}$ to attractor (Eq. 21). For the stochastic classifier (refer to
Sect. 5.2), $F$ denotes the transformation $F^{*}$ to the averaged randomized
assemble of attractors (Eq. 22).

The ability of autoencoder $A$ to accurately preserve the original information
in the input samples has an essential impact on the classifier’s performance.
In particular, our goal is for images from the training data to converge to an
attractor $a(Im)$, such that $a(Im)$ is similar to $Im$. Therefore, for higher
performance of the classifier $C$, we require a lower reconstruction error of
autoencoder $A$ on the training data. Currently, the memorization of training
data is demonstrated for autoencoders trained on data sets consisting of up to
several hundred examples [Belkin]. Due to this factor, the current usage of
the CONN-based classifier is restricted to situations where the available
training data is limited in size.

Experimental results for the vanilla attractor-based classifier are presented
in Sect. 7.

In our approach, the mapping of input samples to attractors can be categorized
as prototype-based methods [proto]. These methods classify new data points by
comparing them to a set of existing prototypes or examples that represent the
classes in the data.

The suggested approaches demonstrate that the performance of prototype-based
methods can be improved by exploring similarities between prototypes. This
approach has been used, for example, in [shufle] for the person re-
identification problem. In the case of the attractor-based classifier, we
explore this approach in a stochastic classifier considered in the following
subsection.

###  5.2 Stochastic classifier

The vanilla classifier considered in the previous section is based on the
representation of an input sample $Im$ by a single attractor (Eq. 21).
However, this representation may not be sufficiently informative.

This is illustrated in Fig. 7, which shows a two-class classification problem.
An image space $X$ is divided into two classes by the decision boundary
$(A,B)$. For a vanilla attractor-based classifier $C$ a test sample $Im$, is
represented by attractor $\widehat{F}(x)=d$ (Eq. 21). A vanilla classifier $C$
will map an input $Im$ to $d$ following application of $M$ to $d$. Thus,
assuming the benchmark classifier $M$ correctly classifies $d$, vanilla
classifier $C$ will mistakenly classify $Im$.

Figure 7: Comparison of image representation by attractors in vanilla and
stochastic attractor-based classifiers in an image space $X$. The $a,b,c$
(respectively $d$) are memorized training examples for the first (respectively
second) class, shown in the basins [Belkin] of the attractors. An image $Im$
is represented in the vanilla classifier by attractor $d$. An attractor
assembly $SA(x)$ obtained by randomized transitions to attractors starting
with $Im$ consists of multiple instances of $a,b,c$, and $d$, providing a more
informative representation of $x$ by attractors of different classes. The
image space $X$ is divided into two classes by the decision boundary $(A,B)$.

Our objective is to enhance the informativeness of the attractor-based
representation. Instead of representing $Im$ solely by a sequence of elements
converging to an attractor, as done in the vanilla classifier, we introduce
$J>0$ sequences that start with $Im$ and converge to attractors. These
sequences are constructed in a similar manner as described in Eq. 21, but they
also incorporate random augmentations. As a result, we obtain an ensemble of
$J$ attractors, which may contain repetitions, that are similar to $Im$. This
ensemble offers a more comprehensive description of $Im$ through the
utilization of attractors, as demonstrated in Fig. 7. Additionally, we derive
the final attractor representation $F^{*}(Im)$ by averaging the ensemble in
the image domain (see below).

Likewise the vanilla classifier, the stochastic classifier is depicted in Fig.
6 (A). The difference between the two classifiers is that the transformation
$F$ in the present case converts to the averaged randomized collection of
attractors, as defined further in Eq. 22.

Accordingly, for overparameterized autoencoder and an input image $Im$ define
an Averaged Randomized Assemble of Attractors

$F^{*}(Im)=\frac{1}{J}\sum_{j=1}^{J}a_{j},$ (22)

where assemble

$\\{a_{j}\mid j=1,\ldots,J\\}$

is comprised of $J$ attractors

$a_{j}=\lim_{i\to\inf}x_{i,j},$ (23)

where

$x_{0,j}=Im\,,$

and

$x_{i+1,j}=dec(enc(\tau_{j}(x_{i,j},\gamma_{i})))$ (24)

for $i>=0$.

The term $\tau_{j}(x_{i,j},\gamma_{i})$ represents a sampling of random
augmentation of the image $x_{i,j}$, where the magnitude of augmentation is
denoted by $\gamma_{i}$. The specific augmentation applied is dependent on the
index $j$.

When $\gamma=0$, it signifies that no augmentation is applied to the image. On
the other hand, when $\gamma=1$, it indicates the maximum possible level of
augmentation. We assign the value of $\gamma_{i}$ using the formula:

$\gamma_{i}=\beta^{\frac{1}{i+1}}\,,$ (25)

where the parameter $\beta>1$ controls the relaxation of the augmentation
amplitude as $i$ increases.

As discussed above, the stochastic classifier was designed to explore more
informative image representations compared to the vanilla classifier. Let’s
revisit Fig. 7 and explore the operational differences between these two types
of classifiers. For a given image $Im$ from the figure, the vanilla classifier
behaves similarly to a one-nearest neighbor approach. It converts $Im$ to the
’nearest’ memorized training example $d$, which is then typically assigned the
ground truth label by the application of $M$. The stochastic classifier, on
the other hand, generalizes the vanilla classifier by representing $Im$
through a combination of several ’nearest’ training examples following the
application of the benchmark classifier, resulting in more accurate
classification compared to the vanilla classifier.

Intuitively, when adding new training examples, the ’density’ of the
attractors around the image $Im$ increases, and the representation of $Im$
improves. However, as we continue to add more and more attractors, the
additional representations become less and less significant in improving the
overall representation of $Im$. The improvements start to relax because the
additional attractors start capturing finer and finer details that might not
have a significant impact on the overall representation.

Our experimental results in Sect. 7.2 demonstrate that the stochastic
classifier significantly outperforms the vanilla one. These also demonstrate
that as we add more training examples, the improvement in performance of the
stochastic classifier compared to the vanilla one starts to level off,
indicating that the stochastic classifier is already efficiently utilizing the
available training data.

The key takeaway from our experimental results in Sect. 7.2 is that the
stochastic classifier consistently outperforms the benchmark classifier on
small training databases used in the experiments. Specifically, it exhibits
significant superiority in general over the benchmark when the MNIST training
database contains less than 10 samples. However, as the number of training
samples increases, the benchmark classifier starts to slightly outperform the
stochastic one. This behavior can be attributed to the saturation of
improvement in the stochastic classifier’s performance with increasing
training examples. These results are further illustrated in Fig. 19.

It is worth noting that although the stochastic CONN classifier explores
augmentations, the approach itself is not an augmentation of the training
examples. In fact, the number of training examples in the stochastic
classifier remains the same as in the vanilla version.

Technically, our representation may resemble the SIFT approach [SIFT] where
the histogram representations for gradient directions are calculated, but in
our approach the histogram bins are comprised of the results of convergence of
the algorithm and are not predefined.

The transformation of $\widehat{F}(Im)$ into an attractor (Eq. 21) represents
the final form of the observed transformations (Eq. 1). Therefore, it is
natural to refer to it as the seen transformation of an image $Im$ that is
observed by a vanilla classifier $C_{V}$:

$seen_{C_{V}}(Im)=\widehat{F}(Im).$ (26)

Similarly, we will refer to $F^{*}(Im)$ (Eq. 22) as the seen transformation by
a stochastic classifier $C_{S}$:

$seen_{C_{S}}(Im)=F^{*}(Im).$ (27)

.

In the stochastic CONN classifier, we perform a series of converging
sequences, where each sequence is terminated by an attractor. The attractors
in the series may vary, but they demonstrate consistency throughout the
series. For example, the set of attractors obtained during descents [1:50] in
the series is typically equal to the attractor set obtained during descents
[50:100]. Additionally, the terminating elements (attractors) are predefined,
meaning they are determined solely by the training examples.

This allows us to view the stochastic CONN classifier from the perspective of
visual perception, particularly in relation to multistable perception.
Multistable perception, as demonstrated by the Rubin’s face-vase illusion and
similar phenomena [Ittelson], involves the perception of different patterns.
These patterns are typically consistent and predefined for individuals over
time, although different individuals may perceive different patterns. For
instance, in the Rubin’s vase/face illusion, the perceived patterns typically
consist of either a vase or a face.

In this regard, the stochastic CONN classifier mimics the properties of
consistency and predefinency observed in human multistable perception. In
Section 8, we will consider how these properties are related to the awareness
of CONN networks.

##  6 Semiotics Interpretation of the model

In this section, we will explore how our model describes the semiotic
phenomena of human perception. We begin by discussing the perception of a
visual object by a single person, followed by an exploration of two-person
communication.

###  6.1 Perception of a visual object by a person

The goal of this section is to specify the semiotic relations that describe
human perception of visual objects and demonstrate how the communication model
introduced in Sect. 3 incorporates these relations. We proceed as follows:
first, we will formalize properties of human visual object perception, to
receive relevant mathematical relationships. Then we consider how these
relations are represented in our model.

We focus on the ’atomic’ perception, which involves the process of identifying
a specific object within a specified period of time. It is important to note
that the perception of objects in different times and spaces, i.e., object
perception in a general sense, is beyond the scope of the current work.

Persons see and ’see’ objects. In other words, they are doing two separate
actions. First, they see, namely perceive objects using their designated
devices – usually their eyes. Then, they become aware of that object. Further
actions may be taken based on the perception to accomplish specific tasks. For
example, imagine a situaltion in which a car is coming fast towards you.
First, you see the car (’see’), then you identify the car approaching you
(’aware’), and finally you step onto the sidewalk (’action’). Here we
formalize the first two steps - ’see’ and ’aware’.

The process of seeing the physical image $x$ is complex and involves various
stages of image processing, feature extraction, and visual perception
mechanisms in the human visual system. It encompasses the physiological and
cognitive processes through which the visual information from the image $x$ is
interpreted and translated into the perceived image $seen(x)$. This includes
the extraction of relevant visual features, the integration of contextual
information, and the interpretation of the visual scene based on the
individual’s cognitive processes and prior knowledge.

It’s important to note that the process of seeing the image is subjective and
may vary among individuals due to differences in their visual perception
abilities, cognitive processes, and prior experiences. Factors such as
lighting conditions, viewing distance, visual acuity, and cognitive biases can
also influence the perceived image.

In our model, we conceptualize the process of ’seening’ the image as a series
of successive image processing steps that generate a new image of similar
modality. Note that in our considerations we assume that the internal
perceived image is represented as mathematical object.222 Thanks to Andres
Luure for this remark.

Let $x$ represent a specific image that is observed by a person. For example,
$x$ could be a digital image. The seeing process involves the conversion of
$x$ into a perceived image denoted as $seen(x)$. We can treat $seen(x)$ as a
new image, of similar modality, which represents the image that the person
perceives. This conversion can be represented mathematically as:

$x\rightarrow seen(x).$

For example, $x$ is a given image of a dog, and $seen(x)$ be an initial
internal visual representation of the dog. Note that internal visual
representation may differ from the initial one, but it is still an image.

The $seen$ function might be slightly distinct for different persons. For
example, people may see dissimilar details in an observed image. Note that
attention is only a part of the internal representation. This reflects the
phenomena that people perceive objects differently.

Further, we formalize the process of seeing as sequential application of
$seen$ function:

$x\rightarrow seen(x)\rightarrow seen(seen(x))\rightarrow\dots\;.$ (28)

For example, during perception process of an image, its details may become
clearer in a gradual fashion, this is illustrated at Fig. 8.

Figure 8: From left to right: the result of subsequent application of $seen$
operator observed in the experiments. Simulated is the visual perception of a
person with the seen functionality biased to perception of even numbers.

What are the equations that reflect the image perception awareness? Direct
access to awarness metrics is hard (may involve operations procedures, or
require dedicated equipment, that is expensive), therefore we use a mediated
method of semantic analysis.

Consider the statement ’I see this image’. In this sentence, ’this image’ has
two meanings. First, it refers to the object itself (in the relevant
modality). For example, an image of a dog. Second, it refers to an internal
perceived image which is a translation of the original image. An inherent
property of a consistent communication system is to make these two meanings
close to each other, namely, to make $\widehat{x}$ equal to
$seen(\widehat{x})$:

$seen(\widehat{x})=\widehat{x}\ ,$ (29)

where $\widehat{x}$ is the final image representation referred as ’this
image’. Detailed analysis of the meaning of the above sentence w.r.t. Eq. 29
is given in Appendix D.

Final representation of image perception is formalized as convergence

$\lim\limits_{n\to\infty}seen^{[n]}(x)=\widehat{x}\ .$ (30)

Summing up the above, seeing of visual objects may be formalized by:

  * •

An operator $seen()$ acting in the image domain;

  * •

Sequential application of $seen()$ to the initial image (Eq. 28).

In addition, awareness in perceiving of visual objects is formalized as:

  * •

Convergence equation (Eq. 30);

  * •

The fixed pointy property (Eq. 29).

How these properties are described by our model of Sect. 3? Equation 2
represents $seen$ operator:

$obs\rightarrow
seen:\,\,obs\xrightarrow{enc}enc(obs)\xrightarrow{dec}seen=dec(enc(obs)).$

Step 3 of Algorithm 1 represents Eq. 28 at $iter$ going to infinity. Equations
6 and 5 are related to convergence to attractors [Belkin] and represent
respectively Eqs. 29 and 30.

In such a way our model describes the following semiotic phenomena: the
existence of the objects seen by a person, as well as the existence of objects
the person is aware of them as such.

It is worth noting that although these aspects were not initially the primary
focus of our attention, our model allows for the description of other semiotic
aspects as well. We demonstrate this below

###  6.2 Person to person communication

In this section, we will explore how the semiotic properties of person-to-
person communication are manifested in the communication model introduced in
Sect. 3.

We analyze a sequence of ’transmitting’ images that constitute the
communication process and investigate how various semiotic properties are
reflected in our model. In particular, we examine the sequence $U(Im)$ as
defined in Eq. 35, focusing on the properties of $U$ that were discussed in
Sect. 4.2.

Interpersonal dialogues can, after a certain point, become repetitive and
circular. In our model, the process of interpersonal communication is always
convergent, resulting in an orbit - a repetitive loop of images (Sect. 4.2.1).
The phenomenon of converging dialogues to a cycle, starting at a certain
point, is the first semiotic phenomenon of interpersonal communication modeled
by our representation.

One of the semiotic properties of communication is that, during a dialogue,
individuals perceive the objects of the interaction in a way that is unique to
them, but this perception is also influenced by the other participant. In our
model, this property is reflected in the structure of the personalized
operators $G_{1}$ abd $G_{2}$ of the seen in the dialog, defined as shown in
Eqs. 10, 15:

$\displaystyle G_{1}=[\,F_{1}\cdot F_{2}\,]^{K/2},\;\;G_{2}=[\,F_{2}\cdot
F_{1}\,]^{K/2},$ where $\displaystyle
F_{1}=[F_{P_{1}}]^{nsteps_{1}},\;\;F_{2}=[F_{P_{2}}]^{nsteps_{2}}.$

It should be noted that the values of $F_{P_{1}}$ and $nsteps_{1}$ depend on
person ${P_{1}}$ only but not on $P_{2}$, and the same applies when swapping
the indices 1 and 2. This highlights the uniqueness of each person in the
dialog. However, the structure of the operators $G_{1}$ and $G_{2}$ also
demonstrates that the way a person perceives the interaction is influenced by
the other person, which reflects the aforementioned semiotic property. It is
worth noting that our model reveals a non-obvious aspect of dialogues: not
only does the content of a dialogue depend on the other participant (’what’),
but also the way in which an individual perceives it can be influenced by the
other participant (’how’).

We may feel the entities of the dialog, yet they possess an elusive quality
that may evade our conscious recognition. Similarly to person-object
communication (Sect. 6.1), our awareness may be limited to the ultimate form
of these entities. Let us show how this phenomenon is represented by our
model.

The current discussion is focused on $P_{1}$, but the same considerations are
applicable to $P_{2}$ as well. Consider an inter-person communication dialog
that begins with an image $Im$ and a sequence $U(Im)$ of images transmitted
during the dialog, as described in Eq. 35.

$\begin{multlined}U(Im)=Im_{1}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{2}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{3}\\\
\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{4}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{5}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}\dots\;\;,\end{multlined}U(Im)=Im_{1}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{2}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{3}\\\
\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{4}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{5}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}\dots\;\;,$

Assuming that $U$ is a bipartite convergent sequence of the first type (as
defined in Definition 4.1), we consider an orbit of $U$ as described in
Statement 4.3. For every fixed point $b_{h}$ in the orbit, where $h$ is even,
there exists a sub-sequence of $U$ (as defined in to Eq. 11) that converges to
$b_{h}$:

$\begin{split}U_{b_{h}}=Im\xrightarrow{F_{1}}T_{1+0\cdot
K,P_{1}}\xrightarrow{F_{2}}\cdots\xrightarrow{F_{1}}T_{1+0\cdot
K+h,P_{1}}\xrightarrow{G_{1}}\\\ T_{1+1\cdot
K+h,P_{1}}\xrightarrow{G_{1}}\cdots T_{1+2\cdot
K+h,P_{1}}\xrightarrow{G_{1}}\cdots\;b_{h}\end{split}$

The outputs of operator $G_{1}$ in this sequence represent the entities that
are seen in the dialog by person $P_{1}$. This is analogous to the person-
object communication discussed in Sect. 6.1. Specifically, the fixed point
property of $b_{h}$ expressed in Eq. 14:

$G_{1}(b_{h})=b_{h}$

represents the person’s awareness that the entity $b_{h}$ is present in the
dialogue.

The existence of both types of objects - those that are seen in the dialog and
those that the person is aware of as the seen - represents the third semiotic
phenomenon of inter-person communication described by our model.

At times, the images that we see in the dialogue may be twofold. On one hand,
we experience them as reflecting the view of the second person, as discussed
earlier. In this sense, they are ”imposed” on us. On the other hand, upon
closer inspection, we may begin to feel that these images are actually our
own, pre-existing before the start of the communication, with no connection to
the other person. In this sense, our dialogue merely served as a pretext for
their manifestation. Our model provides a representation of this phenomenon,
shedding light on the complexities of inter-person communication, as supported
by the following.

Consider what happens with the sequence $U(Im)$ when $nsteps_{1}$ and
$nsteps_{2}$ increase. This may be seen as deeper processing by the
individuals of the information received in interpersonal communication. As
discussed in Sect. 4.2 (Statement 4.6), the orbits of the first type become
closer to the second type. The former are comprised of the elements the person
is aware of as seen in the dialog (as discussed above). The latter consist of
attractors $b_{h}$ that satisfy Eq. 16:

$\displaystyle F_{P_{1}}(b_{h})=b_{h},\text{ for odd }h,$ $\displaystyle
F_{P_{2}}(b_{h})=b_{h},\text{ for even }h,$

where (Eq. 34):

$F_{P_{i}}\text{ is composition }dec(enc)\text{ of }P_{i},i=1,2.$

As shown in Sect. 6.1 (Eq. 2), the attractors represent the objects that
persons are aware of as seen. Attractors of a person do not depend on another
person (in the autoencoder model, they depend on training data of the
respective person) and do not depend on external input or other individuals.
These attractors describe internal image representations that are inherently
embedded within the person. However, the specific attractor to which the
sequence converges depends on the input and the other person involved in the
communication.

In such a way, our model represents the above phenomena: sometimes the
communication dialogue serves just as a signal to ’wake up’ one of the
predefined internal representations. And this is the fourth phenomenon of
inter-person communication described by our model.

##  7 Experimental Results

###  7.1 Implementation of Algorithm 1 for person-to-person communication

Our application implements Algorithm 1 for two-person communication. The
autoencoders $A_{P_{1}}$ and $A_{P_{2}}$ were implemented following
[autoenc_impl, vae], and trained at odd and even digits (30508 and 29492
images) from MNIST database[mnist] respectively. The autoencoders are Multi-
Layer Perceptrons consisting of 6 hidden layers, with a depth of 512 units,
and the latent space 2, trained at 20 epochs.

####  7.1.1 Attractors

In our experiments with attractors, we visualized the autoencoder latent space
similarly to [vae]: a rectangle covering the field of encoder values in the
2-dimensional latent space is partitioned by small squares. Each square is
visualized by plotting the image obtained through the reconstruction decoder
operation at the square’s center.

To encounter the attractors, we randomly sampled 2000 points in the latent
space of every autoencoder and iteratively applied the encoder-decoder
iterations to each point until the consecutive vectors became sufficiently
close. Each terminated vector was assigned to a predefined small bin in the
latent space, where the bin corresponds to the location where the vector
falls. The multiple elements falling into the same bin were considered as
representing the same attractor. The number of counted elements in each bin
indicates the cardinality of the respective attractor’s basin. The resulting
distribution of counter values for the autoencoders is visualized in Fig. 9,
where the bins are represented by circles with radii proportional to the bin
counter.

For 247 samples, the sequences for autoencoder $A_{P_{1}}$ did not converge to
a fixed point. Instead, they exhibited convergence to a cyclic sequence. In
Fig. 10, we visualize these cycles as follows: For every element in the cycle,
we display its subsequent number within the cycle at the element’s location in
the latent space.

These results demonstrate that memorization of training examples is not
necessary for the convergence of encoding-decoding operations to attractors.
Typically, the sequences started from random samples are converged to
attractors, while in rare cases, they exhibit convergence to cycles.

(A)

(B)

Figure 9:  (Best viewed in color) Attractors yielded by autoencoders without
memorization of the training examples. Red circles denote locations of the
attractors, and the radii are proportional to the number of random samples
falling into the attractor basins. (A) and (B): for autoencoders trained at
odd and even RMNIST digits respectively. In (A), some samples result in
sequences that converge to cycles, as shown in yellow. Refer to Fig. 10.

(A)

(B)

Figure 10: (Best viewed in color) The elements comprising the cycles of
decoder/encoder operations for autoencoder $A_{P_{1}}$ used in the
experiments. (A) Yellow field (’ring’) from Fig. 9,(A) in zoom. (B) A part of
the ’ring’ from (A) in zoom. See the text for further details.

####  7.1.2 Bipatite Orbits

In our past experiments with the implementation of Algorithm,1, we varied the
number of parameters $nsteps$ and initialization images. For each setting of
$nsteps$, the observed orbits comprised a small set of orbits for a large set
of initializations. The specific set of observed orbits depended on the
starting image of the algorithm.

For example, Fig. 11 demonstrates a second type orbit of the person-to-person
dialogue in Algorithm 1, implemented with autoencoders $A_{P_{1}}$ and
$A_{P_{2}}$. The values of $nsteps$ for $P_{1}$ and $P_{2}$ were set to 50.
For such sufficiently large values, the final images of the sequence produced
by the encoding/decoding operations in Step 33.2. of the algorithm became
indistinguishable. The first type orbits become indistinguishable from the
second type orbits.

Figure 11:  A bipartite orbit of the second type of the period length $K=6$.
(A)-(G): Subsequent temporal snapshots of the application implementing
Algorithm 1. The arrows mark the $Im_{iter}$ images transmitting between the
persons (step 33.5. of the algorithm): blue and green arrows denote the images
transferred from $P_{2}$ to $P_{1}$ and from $P_{1}$ to $P_{2}$ respectively.
(A), (C), and (E): attractors for $A_{P_{2}}$ forming the orbit. Note the
difference between ’8’-s depicted in (C) and (E). (B), (D), and (F):
attractors for $A_{P_{1}}$ forming the orbit.

###  7.2 Classifiers

$obs$ $seen_{C_{V}}(obs)$ $seen_{C_{S}}(obs)$ g.t. label: ’4’ g.t. label of
$seen$ for $C_{V}$: ’4’ g.t. label of $seen$ for $C_{S}$: ’4’ $obs$
$seen_{C_{V}}(obs)$ $seen_{C_{S}}(obs)$ g.t. label: ’7’ g.t. label of $seen$
for $C_{V}$: ’7’ g.t. label of $seen$ for $C_{S}$: ’7’ $obs$
$seen_{C_{V}}(obs)$ $seen_{C_{S}}(obs)$ g.t. label: ’0’ g.t. label of $seen$
for $C_{V}$: ’0’ g.t. label of $seen$ for $C_{S}$: ’0’ Figure 12:  Examples of
the images ’seen’ by our vanilla classifier (second column) and stochastic
classifier (third column) in our experiments for an ’observed’ image from the
train dataset (first column). The classifiers were trained for 200 epochs on
the R8 restricted MNIST dataset[mnielsen, mnielsen2]. The $M(seen)$ denotes
the label assigned by the benchmark classifier $M$ to the ’seen’ image, as
shown at Fig. 6 (A)).  $obs$ $seen_{C_{V}}(obs)$ $seen_{C_{S}}(obs)$ g.t.
label: ’4’ $M(seen)=\text{'9'}$ $M(seen)=\text{'4'}$ $obs$ $seen_{C_{V}}(obs)$
$seen_{C_{S}}(obs)$ g.t. label: ’1’ $M(seen)=\text{'1'}$ $M(seen)=\text{'1'}$
$obs$ $seen_{C_{V}}(obs)$ $seen_{C_{S}}(obs)$ g.t. label: ’0’
$M(seen)=\text{'5'}$ $M(seen)=\text{'5'}$ Figure 13:  Examples of the images
’seen’ by our vanilla classifier (second column) and stochastic classifier
(third column) in our experiments for an observed image from the test dataset
(first column). The vanilla classifier ’sees’ in the observed image from the
first row the memorized training example from the first row of Fig. 12.
Construction of the seen images for the observed images from the first and
second row is shown in Figs. 14 and 15 respectively. See capture of Fig. 12
for further details.

(A)

(B)

(C)

(D)

Figure 14: Construction of $seen_{C_{S}}(obs)$ for the observed image by the
stochastic classifier, shown in the upper row of Fig. 13. For each iteration,
20 images generated by Eq. 24 are displayed. (A) exhibits random augmentations
of the input image before the $dec(enc())$ operation at the first iteration
$i=0$ of Eq. 24. Subsequently, (B), (C), and (D) depict the sub-ensemble after
the 2nd, 5th, and 14th iterations, respectively.

(A)

(B)

(C)

(D)

Figure 15:  The same as Fig. 14, but for the ’observed’ image by the
stochastic classifier, shown in the middle row of Fig. 13.

We compared the performance of a standard MLP classifier $M$ with our CONN
vanilla and benchmark classifiers while training on datasets consisting of a
small number of training examples.

The classifier $M$ was trained at 40 training settings, produced by
combinations of 10 training datasets and 4 numbers of training epochs. The
training datasets $TR$ were constructed by randomly selecting 5, 6, 7, 8, 9,
10, 20, 30, 40, and 50 examples for every digit from the MNIST training
dataset, following [mnielsen, mnielsen2]. The number of training epochs was
chosen as 25, 50, 100, and 200. The test set $TE$ was constructed by randomly
selecting 1000 examples from the MNIST test set.

For every training setting, we obtained the accuracies of $M$ trained on the
respective training set with the respective number of epochs, and tested it on
the test set $TE$. See Fig. 6 (B).

Our vanilla and benchmark CONN classifiers incorporate fully convolutional
autoencoders, similar to the autoencoder with the Cosid nonlinearity trained
on MNIST images [BelkinSI]. We trained the autoencoders with the same
architecture on 10 training datasets $TR$, carefully tuning the
hyperparameters to minimize the training error. For more detailed information,
refer to Appendix, LABEL:appendix:training_details.

Furthermore, we mapped the training and test sets $TR$ and $TE$ to the
training and test sets $ATR$ and $ATE$ ’seen’ by the classifiers. See Fig. 6.
The mapping was performed following Eqs. 26 and 27 for the vanilla and
stochastic classifiers, respectively. Examples of the images ’seen’ by the
CONN classifiers $C$ are shown in Fig. 13.

Next, we obtained the performance results for both CONN classifiers by
training $M$ on sets $ATR$ and testing it on $ATE$. The results are compared
with the performance of $M$ trained on datasets $TR$ and tested on $TE$. The
comparison demonstrates that the stochastic CONN classifier outperforms both
the benchmark classifier and the vanilla CONN classifier. See Figs. 16 and 17.

Our benchmark classifier $M$, which is embedded in the vanilla and stochastic
classifiers (Fig. 6), is a 3-layer MLP. It takes an input size of 28x28
pixels, has two hidden layers with widths of 500 and 100 neurons respectively,
and has an output size of 10 classes.

The augmentations of the stochastice classifier (Eq. 24) were applied using
[imgaug]. The ensemble length $J$ of Eq. 22 was set to 500, and the number of
iterations (Eq. 23) to 30. The relaxator $\beta$ of Eq. 25 was set to 2.6.

Furthermore, our experiments for stochastic classifiers were extended to
reflect a certain dependence of the accuracies of the classifiers on the
initialization of the networks. We performed 200 initializations for each
considered training set $TR$, followed by testing on $TE$. We measured the
mean and standard deviation of the respective accuracies of $M$ (Fig. 6,(B)).
Then we performed the same 200 initializations of $M$ before training on $ATR$
sets and measured the mean and standard deviation of the respective accuracies
of $M$ on the test set $ATE$.

The obtained results, shown in Figs. 16– 19, confirm the outperformance of the
stochastic classifier over the benchmark.

(A)

(B)

(C)

(D)

(E)

(F)

(G)

(H)

Figure 16: (Best viewed in color) (A), (B), (E), and (F): Comparison of
accuracy between the stochastic CONN classifier (in blue), the vanilla CONN
classifier (in green), and the benchmark classifier (in red) for different
numbers of training epochs: 25, 50, 100, and 200, respectively. (C), (D), (G),
and (H): Pointwise accuracy differences between the stochastic CONN classifier
and the benchmark classifier for graphs (A), (B), (E), and (F), respectively.
The x-axis tick values represent the size of the restricted MNIST training
databases. For example, R10 corresponds to a training database comprising 10
examples per digit selected randomly from the training MNIST dataset
[mnielsen, mnielsen2].

plain

(A)

(B)

Figure 17: (Best viewed in color) Cumulative representation of the accuracy
functions shown in Fig. 16. (A) Pointwise maxima of the accuracy functions for
the stochastic CONN classifier (in blue), the vanilla CONN classifier (in
green), and the benchmark classifier (in red) across the number of training
epochs: 25, 50, 100, and 200. (B) Pointwise difference between the maximum
functions of the stochastic CONN classifier and the benchmark classifier, as
depicted in (A) for both classifiers. The x-axis tick values correspond to the
size of the training databases, as in Fig. 16 .

(A)

(B)

(C)

(D)

Figure 18: (Best viewed in color) (A), (B), (C), and (D): Average accuracy
values for the stochastic CONN classifier (in blue), and the benchmark
classifier (in red), over series of 200 random initializations before
training, shown for different numbers of training epochs: 25, 50, 100, and
200, respectively. Dashed lines denote upper and lower bounds determined by
half of the standard deviation. The x-axis tick values correspond to the size
of the training databases, as in Fig. 16.

Figure 19: (Best viewed in color) Average accuracy values for the stochastic
CONN classifier (in blue), and the benchmark classifier (in red), over series
of 200 random initializations before training, and different numbers of
training epochs: 25, 50, 100, and 200. Dashed lines denote upper and lower
bounds determined by half of the standard deviation. The x-axis tick values
correspond to the size of the training databases, as in Fig. 16.

##  8 Discussion and Further Work

Our CONN model describes communication between individuals, where participants
receive information in an external modality and convert it to an internal
modality. Additionally, the participants are partially or fully conscious of
the received information and exchange this perceived information with each
other in the external modality. The model is structured as a sequence of
observed and seen operations and employs subject-associated autoencoders for
implementation. The input to the autoencoders consists of images in the
external modality, while the latent space represents the internal modality.

Furthermore, the model also encompasses the perception of objects by
individuals as a specific case.

The CONNs simulate perceptual awareness in two aspects. Firstly, they model
the observed/seen functionality of the visual perceptual awareness (Sect.
6.1). Additionally, they emulate the phenomenon of multistable human
perception, which is elicited by ambiguous images such as the Rubin face-vase
[RubinFaceVase]. As discussed in Sect. 5.2, stochastic CONN classifier
specifically emulates the properties of consistency and predefinency observed
in human multistable perception. On the other hand, the importance of
multistable perception for perceptual awareness has long been recognized
[Leopold_multistable, Lumer]. Recent neuroscience research establishes a
connection between multistable phenomena and perceptual awareness, suggesting
that multistability can play a crucial role in understanding the constructive
process of perceptual inference [Saracini]. Thus, CONN mimics multistable
perception, which is recognized as essential for awareness. This represents
the second aspect of CONN’s functionality in simulating awareness.

Under our model, the flow of information involved in perceiving an object by a
person converges to a fixed point, which can be treated as a single-element
cycle. This convergence characterizes the awareness of perceiving an object.
Similarly, in the two-person communication model, we have experimentally
observed and proven, under certain natural conditions, that the modeled flow
of information between the participants exhibits the property of converging to
a bipartite cycle (Statement [statement_1]). In this sense, the bipartite
orbits, when considered as a whole, can be seen as an ’attractor of
interpersonal communication’, representing what can be referred to as the
collective consciousnessLABEL:collective_consciousness within this
communication.

We believe that our model can cope both full and partial awareness of
perception. The model is mainly composed of two phases: internal phase and
external one. The length of the internal phase directly affects awareness of
the result it provided. We showed (Statement 4.6) that if $nsteps$ is large
enough, convergence of the inter-person flow to bi-separation is guaranteed.
Interestingly, even when $nsteps$ is as small as $1$, where guarantees are not
provided, bi-separations (Definitions 4.1) still occurs (see Sect. 4.2.1).

In a wide sense we may consider our model as decision-make. The model is
composed of internal and external phases and can cope both short and long
decision-making processes. The internal process is iterative and an inaccurate
decision (but still valid) may result if the number of iterations is too
small. Yet a valid decision can be returned at any time (iteration). This
process can thus incorporate both fast and long decision-making procedures and
can explain both reflexes and regular decisions, under the same procedure.

Our work explains that under certain conditions communication may not be
effective. Further research is required to explore which conditions need to be
changed to better our communication channels. Furthermore, advertisers,
sellers and any system that attempts to convince others to pursue any action,
may use this work to design and test their messages. We offer a simulative
test-ground and theoretical/experimental tool to study expected behaviors in
communication situations.

Specifically, when designing a new item, such as furniture, clothing items, or
a new car form, can we estimate how well its appearance will be received by
the target audience? Ultimately, the acceptance of fashion-dependent objects
relies on individuals’ internal preferences, which are influenced by various
factors and can change over time. It seems plausible to acquire knowledge
about these preferences through observed/seen modeling. This can be achieved
by training on streams of media data without processing input/output
specifically associated with individual persons. Such an approach can be
applied in the creation of fashion-dependent objects, including advertising
imaging. Exploring this avenue is one of the directions of our research.

Table 2: Types of CONN networks Network Type |  Definition and Properties |  Models/Simulates |  Performance of the CONN classifier at classification task with small training data w.r.t. a SOTA network: Sect.7
---|---|---|---
Vanilla |  Person-Object: following[Belkin];

Person-Person: Sects. 3 and 4.2 |  Modeling semiotic phenomena: Sect. 6 |  Underperforms
Stochastic |  Person-Object: Sect. 5.2 |  Mimicking multistable perception: Sect. 5.2 |  Overperforms

Our model may be applied to classical computer vision tasks, such as
classification. Our CONN classifier, based on the one-person communication
model, demonstraed superior performance compared to a SOTA neural network
classifier at classification tasks with small training data (Sect. 5).

For a given classifier $M$ and training and test sets, our model offers an
alternative approach to the conventional training and testing of the
classifier. Instead, we convert the sets into their perceptual counterparts,
which consist of images that the CONN network ”perceives” in the original sets
(as discussed in Sect. 5). Subsequently, classifier $M$ is trained and
evaluated on these transformed sets. The transformed sets can be considered as
perceptual versions of the sets, framing the overall process as the
”perceptualization of a classifier.” Importantly, our experimental results
demonstrate that this perceptualized version of the classifier consistently
outperforms the original version across a range of small training datasets.

It may be noted that from an engineering perspective, the transition to the
perceptual versions of the training and test sets can be interpreted as a
preprocessing layer that precedes the input of data into the classifier

The adaptability and flexibility of human perception in interpreting ambiguous
visual stimuli considered above reflects the robustness and generalization
abilities of the human visual system. Another manifestation of these abilities
is resilience to adversarial attacks. It is widely acknowledged that human
perception exhibits greater resilience against adversarial attacks compared to
neural networks (see, for example,
[Adversarial_examples_in_the_physical_world, limitations_deep_learning]).
Whether our classifier which mimics the properties of human perception is
resilient to adversarial attacks?

A classical adversarial attack is one in which an input image with
indistinguishable noise is given to a neural network and the network returns a
wrong answer, while returning a correct answer to the original input image
(without the additional noise). Humans are mostly immune to these kinds of
attacks and therefore return the correct answer to both original and noised
images. Many papers explored these [adv_1, adv_2, adv_3, adv_4] attacks and
attempted to offer a few approaches how to deal with them. Various approaches
to achieving resilience in deep learning models have been proposed, including
the use of autoencoders for denoising [adv_autoenc, adv_autoenc_2] and
ensembles [adv_autoenc, adv_Lin]. Another approach involves adding noise to
the training examples [adv_noise_train_only], test examples [adv_Lin], or both
[adv_noise_train_test]. However, such approaches in general do not guarantee
ideal defense, as demonstrated by the lack of a method demonstrating perfect
resilience[failed_defences].

A classical adversarial attack occurs when a neural network receives an input
image with indistinguishable noise and provides an incorrect answer, despite
correctly classifying the original image without the additional noise. In
contrast, humans are largely immune to these types of attacks and consistently
provide correct answers for both the original and noised images. Numerous
papers have explored these attacks and have attempted to propose approaches to
mitigate them (see [adv_1, adv_2, adv_3, adv_4] for examples).

Various strategies have been proposed to enhance resilience in deep learning
models. These include the use of autoencoders for denoising [adv_autoenc,
adv_autoenc_2], ensembles [adv_autoenc, adv_Lin], and the introduction of
noise to training examples [adv_noise_train_only], test examples [adv_Lin], or
both [adv_noise_train_test]. However, it should be noted that these approaches
do not guarantee ideal defense, as demonstrated by the absence of a method
that provides perfect resilience [failed_defences].

In Section 2, it was noted that the primary difference between the vanilla
CONN classifier and autoencoders [adv_autoenc, adv_autoenc_2] is the iterative
process of combining decoder and encoder operations (Eq. 2), leading to
convergence towards attractors. This unique characteristic of the vanilla CONN
classifier has the potential to hinder perturbation attacks on the classifier,
as discussed in Sect. LABEL:resil_vanilla.

The stochastic CONN classifier offer additional defensive measures. These
mechanisms include employing ensemble methods and introducing noise during
both the training and testing phases. Overall, we anticipate that designing a
successful attack against stochastic CONNs would be exceedingly challenging
due to the considered defense mechanisms.

Investigating and analyzing the resilience properties of our classifier in the
face of various adversarial attacks forms another direction of our ongoing
research.

Our work deals with one person perception (internal perception) and
communication between two persons, but this model can be elaborated to more
than two persons. In addition, it is not limited to persons and any system
that has internal representation and a limited number of options (state
machine) may use this work to further investigate its behavior. Note that the
current system only supports the following conditions: 1) input and output
modalities are the same, 2) it works in iterative (trainable) fashion. Under
these conditions we show convergence of input to certain output (attractors,
orbits). The further research is needed to relax these conditions.

Thanks to Prof. Andres Luure and Prof. Michael Bialy for their valuable
contributions and insightful discussions. Special thanks to Nadav Kadvil for
the fruitful discussions.

We acknowledge the use of Michael Nielsen’s GitHub repository [mnielsen2] for
the implementation of the restricted MNIST datasets in our work, and the
Alexander Jung’s repository [imgaug] used for image augmentations. We would
like to acknowledge the following references for providing code examples that
inspired and influenced the development of our CONN implementation:
[autoenc_impl, vae, vae_shashankdhar]. Additionally, we would like to credit
the Pixabay image collection [pixabay] for providing the images used in this
paper.

##  Appendix A Definition of the sequence $U$ of the images transmuted between
the persons in person-to-person CONN implemented using encoder/decoder
operations

Human communication is simulating in our approach by transmitting images from
one person to the other. The first step a person takes when receiving an image
is transforming it to the latent space (Algorithm 1). Following the person
yields the result of step 33.2. . Let $S_{iter,P_{i(iter)}}$ be the sequence
of images generated in this operation (see Eq. 32), and $W$ be all sequence of
images (see Eq. 33), where $iter=1,\dots,n\\_iters$ and

$i(iter)=\begin{cases}1,&\text{if}\ iter\text{ is odd}\\\ 2,&\text{if}\
iter\text{ is even}\end{cases}.$ (31)

The $iter$ refers to ’external’ iteration and $i(iter)$ to the person’s index.
Let

$\begin{multlined}S_{iter,P_{i(iter)}}=Im_{iter},\,[dec(enc)]^{1}(Im_{iter}),\,\dots,\\\
Im_{iter+1}=[dec(enc)]^{nsteps_{i(iter)}}(Im_{iter})\;,\end{multlined}S_{iter,P_{i(iter)}}=Im_{iter},\,[dec(enc)]^{1}(Im_{iter}),\,\dots,\\\
Im_{iter+1}=[dec(enc)]^{nsteps_{i(iter)}}(Im_{iter})\;,$ (32)

where $dec$ and $enc$ are encoding and decoding operations of $P_{i(iter)}$.
Let

$W(Im)=S_{1,P_{1}},\,S_{2,P_{2}},\,S_{3,P_{1}},\,S_{4,P_{2}},\dots\;,$ (33)

where $Im=Im_{1}$. For example, $S_{5,P_{1}}$ refers to the sequence of images
in the fifth transaction of Person one.

Let

$F_{P_{i}}\text{ be composition }dec(enc)\text{ of }P_{i},i=1,2.$ (34)

Consider the sequence of transmitting elements $Im_{iter}$ in $W(Im)$:

$\begin{multlined}U(Im)=Im_{1}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{2}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{3}\\\
\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{4}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{5}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}\dots\;\;,\end{multlined}U(Im)=Im_{1}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{2}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{3}\\\
\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}Im_{4}\xrightarrow{[F_{P_{2}}]^{nsteps_{2}}}Im_{5}\xrightarrow{[F_{P_{1}}]^{nsteps_{1}}}\dots\;\;,$
(35)

where $Im_{1}=Im$.

##  Appendix B Partitioning a subsequence of $U$

We show below an example of partitioning the first $1+(n+1)*K$ members of
sequence $U$ (Eq. 11) by the first element of $U$ ($Im$) following the matrix
of $(n+1)$ rows and $K$ columns. For simplicity the arguments of $U$ are not
shown. The partitioning proceeds similarly to function reshape of Python. In
our example $K=2$ and $n=2$. Using the Python notation for indexing of $U$:

$\begin{array}[]{@{}cccc@{}}\lx@intercol
<EMAIL_ADDRESS>T_{1+0\cdot K,P_{1}}&\xrightarrow{F_{2}}&T_{1+0\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}\\\ T_{1+1\cdot
K,P_{1}}&\xrightarrow{F_{2}}&T_{1+1\cdot K+1,P_{2}}&\xrightarrow{F_{1}}\\\
\dots&&&\\\ T_{1+n\cdot K,P_{1}}&\xrightarrow{F_{2}}&T_{1+n\cdot
K+1,P_{2}}&.\\\ \end{array}$ (36)

##  Appendix C Proof of Statement 4.3

If we denote

$\begin{multlined}G_{1}=[\,F_{1}\cdot F_{2}\,]^{K/2},G_{2}=[\,F_{2}\cdot
F_{1}\,]^{K/2}.\end{multlined}G_{1}=[\,F_{1}\cdot
F_{2}\,]^{K/2},G_{2}=[\,F_{2}\cdot F_{1}\,]^{K/2}.$

(Eq. 15 ) then one may define the sequences $L_{n}$ with $G_{1}$, $G_{2}$
acting on their elements as follows:

$\begin{array}[]{@{}cccccccc@{}}L_{0}=&(T_{1+0\cdot
K,P_{1}}&\xrightarrow{F_{2}}&T_{1+0\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}&\dots&\xrightarrow{F_{2}}&T_{1+0\cdot
K+K-1,P_{2}})\\\ &G_{1}\downarrow&&G_{2}\downarrow&&&G_{2}\downarrow&\\\
L_{1}=&(T_{1+1\cdot K,P_{1}}&\xrightarrow{F_{2}}&T_{1+1\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}&\dots&\xrightarrow{F_{2}}&T_{1+1\cdot
K+K-1,P_{2}})\\\ &G_{1}\downarrow&&G_{2}\downarrow&&&G_{2}\downarrow&\\\
\dots&&&&&&&\\\ L_{n}=&(T_{1+n\cdot K,P_{1}}&\xrightarrow{F_{2}}&T_{1+n\cdot
K+1,P_{2}}&\xrightarrow{F_{1}}&\dots&\xrightarrow{F_{2}}&T_{1+n\cdot
K+K-1,P_{2}})\\\ &G_{1}\downarrow&&G_{2}\downarrow&&&G_{2}\downarrow&\\\
\dots&&&&&&&\\\ &b_{0}&&b_{1}&&&b_{K-1}&\\\ \end{array}$ (37)

Since $G_{1}$, $G_{2}$ are continuous, elements $b_{h}$ are the fixed
points[Belkin] for $G_{1}$, $G_{2}$, see Eq. 14. From the convergence of the
columns of Eq.37 to $b_{h}$ and continuity of $F_{1},F_{2}$ follows Eq. 13.
This proves Statement 4.3.

##  Appendix D Derivation of Eq. 29 from analysis of the meanings a statement.

Consider the meanings of the statement ’I see this image’. One meaning is ’the
result of my seeing is this image’. This implies equivalence between two
images. The first one is what the person refers as ’I see’. This may be
represented as the result of application of $seen$ operator to some
$\widehat{x}$. The second image is ’this image’. In such a way the meaning may
be expressed as

$seen(\widehat{x})=\text{ 'this image'}.$

What is $\widehat{x}$ to whom the ’see’ is applied? Note, that another meaning
of ’I see this image’ is ’see this image’, i.e. ’see’ is applied to ’this
image’. This specifies $\widehat{x}$:

$\widehat{x}\text{ is 'this image'}.$

In such a way there hold the fixed point property of Eq. 29:

$seen(\widehat{x})=\widehat{x}\ ,$

##  Appendix E Proof of Statement 4.6 (See page 4.6.)

######  Statement E.1.

If the following assumptions hold:

  1. (A1)

both $F_{P_{1}}$, $F_{P_{2}}$ determine finite numbers of attractors at $X$;

  2. (A2)

every attractor $a$ of $F_{P_{1}}$ (resp. $F_{P_{2}}$) with a certain vicinity
$\delta(a)$ belongs to the basin of a certain attractor $a^{\prime}$ of
$F_{P_{2}}$ (resp. $F_{P_{1}}$);

  3. (A3)

$X$ is limited set in a finite-dimensional normed space;

  4. (A4)

convergence of $F_{P_{r}}$, $r\in\\{1,2\\}$ to attractors is uniform: for any
$\epsilon>0$ there exist $N$ such that for any $x\in X$ there hold

$dist(F_{P_{r}}^{[n]},\lim_{n\to\infty}F_{P_{r}}^{[n]})<\epsilon$

for any $n>=N$,

then the sequence $U(Im)$ (Eq. 35) is a sequence of the bipartite convergence
of the second type, converging to a bipartite orbit consisting of attractors
of $F_{P_{i}}$ and $F_{P_{2}}$.

###### Proof.

Clearly the set $X=\\{Im\\}$ of all images satisfies (A3).