In the brain, the mapping from the retinal position and eye direction to the external target location that controls reaching movements is believed to take place in the posterior parietal cortex (PPC) (Jeannerod, 1997). Zipser and Andersen (1988) looked at the responses of the neurons in area 7a of PPC and trained an artificial neural network on the mapping from eye direction and retinal position to head-centered coordinates. The network consisted of three layers where units in the first layer code for retinal position and eye direction. The activity of the output layer indicated the head-centered location of the target. The model produced similar response properties as the real neurons of PPC. The neurons in the hidden layer became tuned to retinal position but they are also modulated by eye position. Like the saccade control described above, this learned mapping fulfills the criteria of monotonicity, continuity and convexity.

The type of population coding that is found in area 7a, and that also emerges in the hidden layer of the model, is often called a gain field (Zipser and Andersen, 1988; Buneo and Andersen, 2006). Like other types of population coding, different neurons take care of different parts of the mapping and the final result is obtained by weighing together the contributions of each neuron (**Figures 3**, **4**). However, a gain field is characterized by the fact that the neurons are primarily organized along only some of the input dimensions. For example, Zipser and Andersen (1988) found that neurons coding for target position were retinotopically tuned, but responded differently depending on the eye positions.

Salinas and Abbott (1995) also addressed the question of how the brain can transfer information from sensory to motor system using population codes. They investigated the coordinate transformations in visually guided reaching and proposed a model that uses a Hebbian learning mechanism to learn the sensory-motor mapping. Unlike Zipser and Andersen, they assume that the input space is already covered by a large set of prototypes coded by a set of basis units. They view the problem of eye–hand coordination as a form of function approximation where the problem is to find the appropriate weights for the outputs of each basis unit to obtain the desired mapping (**Figure 4**). They show how these weights can be learned using Hebbian learning and the general model they present can be used to describe arbitrary mappings between spaces.

We now turn to the slightly more complicated situation where reaching is followed by grasping an object. Here, we not only need to locate the target, we also need to shape the hand in the correct way both before reaching the object and subsequently to grasp it. Despite the added complexity, this too can be seen as a mapping between two spaces. In the case that only visual information is used, the input space codes for the location and the shape of the object while the output space minimally contains the movement direction for reaching, the parameters to preshape the hand and finally the force vectors to perform the grasping movement.

Although little is known about how shapes are represented in the brain, work in mathematics (Kendall, 1984) and computer graphics (Blanz and Vetter, 1999; Kilian et al., 2007) show that it is possible to design shape spaces where different shapes can be synthesized from combinations of basic shapes in a way that is similar to how basis units work together to represent a point in a space using a population code. For a known rigid object however, it is sufficient to code the orientation of the object. This can be done in a three dimensional space of the rotation angles that describes the orientation of the object2 . The orientation can be represented in a way similar to position by a set of basis units that together code for all possible orientations of the object. Here the final mapping is between a six-dimensional space representing position and orientation to a space that represents the critical parameters of the reaching movement.

There are a number of spaces that could potentially be involved in eye–hand coordination. Depending on task constraints, the brain is thought to use both egocentric and allocentric representations of space (Crawford et al., 2004) and there is evidence that neurons in PPC code for targets in relation to both gaze (Batista et al., 1999) and hand (Buneo and Andersen, 2006). Investigating spatial representations for reaching in the superior parietal lobule (SPL) of PPC, Buneo and Andersen (2006) found evidence for representations of targets in both eyecentered coordinates and of the difference in position between the hand and target.

**Figure 5** summarizes some of the coordinate systems involved in eye–hand coordination. The target object can be represented in either allocentric or one of the egocentric spaces. For reaching, an egocentric frame of reference is more suitable but as we have seen, there are several egocentric spaces corresponding at least to the eye and the hand, but it is likely that many more exist and presumably the brain is able to map freely between them. To grasp an object, its representation must be mapped on the space that contains the possible motor actions. The dimensionality of this space is high enough to contain all possible grasp movements, but still of limited dimensionality. For the brain to learn these mapping in an efficient way, it is necessary that where possible, these mappings fulfill the three conditions of monotonicity, continuity and convexity. We submit that population codes are used to make this possible.

#### 6. MECHANISMS

Population codes of spaces as described above are instances of a coding scheme where each input is coded by the distance to a number of prototypes. The optimal stimulus for each neuron, or basis unit, in the population can be considered the prototype for that unit. One such form of population coding is given by the chorus transform proposed by Edelman (1999) who calls it a

FIGURE 5 | Coordinate systems in eye–hand coordination. The position of an object can be coded in in relation to allocentric space, the eye or the hand and potentially also other parts of the body. To reach and grasp the object, the brain must map representations of the object from a sensory space to a motor space in order to control the arm and hand in an appropriate way.

2Euler angles are generally problematic as orientation representations, in particular because they do not fulfill the three criteria of monotonicity, continuity and convexity. However, there exist other representations, such as quarternions, that do.

"chorus of prototypes." In the simplest case, the response of each unit is the similarity to the prototype measured by some suitable metric. **Figure 6** shows an example of a chorus transform. The input to the transform is an image of a face. This face is compared to each of five face prototypes and the resulting transform is the set of similarity measurements. The chorus transform thus describes a type of population coding of the input.

An important property of the chorus transform is that it preserves Voronoi tessellations of the input space (Edelman, 1999, p. 268). It also approximately preserves the inter-point distances in the original space. This means that category boundaries in the input space are mostly preserved in the output space (Edelman and Shahbazi, 2012). This has several critical consequences for both neural coding and mappings of spatial representations:

Stimulus generalization. The spatial representation naturally supports generalization since novel stimuli will be coded by the similarity to known stimuli and the coding will gradually change if the stimulus gradually changes. There is thus a continuous mapping from stimulus properties to the representation of the stimulus.

Discrimination. Since discrimination borders are typically Voronoi borders and these are mostly preserved by the coding, it means that discrimination borders in the input space are preserved in the coding space.

Categorization. For the same reason, categories induced by the Voronoi tessellation are preserved in the population coding.

When looking at mapping between two spaces coded by a population of units we note that these properties of the chorus transform imply that a linear mapping from such a representation also have these properties. This has consequences for sensory-motor mapping between spaces:

Interpolation. Since similar stimuli are coded by similar population codes, similar stimuli will be mapped to similar

FIGURE 6 | In the chorus transform (and in RBF networks) an input is mapped to its similarity to a number or prototypes. Here, an image x is compared to five face prototypes with different orientation. Each comparison produces one component of the chorus transform that will here consist of five similarity measures.

motor outputs. This is equivalent to the continuity criteria introduced above.

Sensory-motor categories. When object categories are represented as Voronoi borders in the stimulus space, different stimulus categories can be mapped to different motor actions by a single mapping. When the input crosses the Voronoi border between two categories, so will the output, and in the same way a small part of the input space represents a particular category, a corresponding part of the output space can represent actions suitable for that category. Furthermore, since Voronoi borders are preserved in the mapping it follows that the convexity criteria is also fulfilled by these types of mappings.

The most commonly used computational architecture that uses the chorus transform is the radial basis function (RBF) network (Moody and Darken, 1989). This artificial neural network consists of three layers (**Figure 4**). The middle layer consists of units that are tuned to different stimulus prototypes and their response is maximal when the input is identical to the prototype. The prototypes can be selected in different ways. One possibility is to use one prototype for each exemplar that has been encountered. Alternatively, the prototypes can be selected by trying to cover the input space by suitably spaced prototypes. Finally, the prototypes can be found by learning. Once selected, each unit in the middle layer contributes to the output depending on how well the input matches its prototype. RBF-networks have been used in numerous applications for both categorization and function approximation tasks and can easily learn complex relations between their input and output.

A type of RBF-network that is of special interest is normalizing radial basis function networks (Bugmann, 1998). This model differs from the standard model in that the output is normalized. This is an operation that has been suggested to be implemented by lateral inhibition and it is ubiquitous in the brain (Carandini and Heeger, 2012). The importance of the normalization stage is that it makes the output of the RBF-network consist of a convex combination of the outputs of the individual units, where each output is weighed by how close its prototype is to the input. This property guarantees that the output will be a continuous function of the input that quickly converges on the correct mapping during learning. Like other multilayer feed-forward networks, RBF-networks are universal approximators, which means that they can learn any mapping between finite dimensional spaces with any desired accuracy as long as there are a sufficient number of hidden units.

Salinas and Abbott (1995) investigated how the number of units influenced the accuracy of the coding and decoding of different magnitudes. More recently, Eliasmith and Anderson (2003) presented general mathematical recipes for how low dimensional quantities can be coded and decoded in the brain using population codes as well as suggestions for how such quantities can be combined in different ways to implement different arithmetic operations and mappings between spaces. The same type of reasoning can be applied to many different tasks including other sensorimotor transformations, learning and short-term memory (Pouget and Snyder, 2000). This shows that population codes are a general way to code quantities in one- or multi-dimensional spaces and to perform arbitrary operations on them. This further supports that the machinery available to the brain is ideally suited for processing spatial representations.

It is interesting to note that the basis unit coding has many similarities to approaches in control theory, such as locally weighted learning (Atkeson et al., 1997). In fact, Stulp and Sigaud (2015) have shown that many models and algorithms working according to these principles use exactly the same underlying model as the three-layer network described above. This lends support to the idea that there is something fundamental about these types of mechanisms where functions are computed using units that each react to different parts of the input space and the output is subsequently calculated as a combination of the outputs from those individual units.

To summarize, we have proposed that mappings between spaces consist of two steps. The first is a comparison between the input and a number of prototypes and the second is the weighting of the output from each prototype unit by its similarity to the input. The chorus transform provides a good model for the usefulness of population codes, both as a way to represent points in psychological spaces and as a mechanism for mapping between such spaces. RBF-networks constitute the canonical way to model learning of such mappings, but many other models are possible.

#### 7. CONCLUSION

This article has treated two levels of spaces in the brain psychological and neurocognitive. The psychological spaces, for example the color space, can be studied in psychophysical experiments, in particular with the aid of discrimination or similarity judgments. These spaces can often be represented in a small number of dimensions and we have shown how conceptual spaces can be used to model categorization processes. Neurocognitive representations are implemented implicitly using populations coding where different neurons process different regions of the spaces and allow for efficient

#### REFERENCES

mappings between spaces. Furthermore, spatial coding naturally supports generalization from learned examples by interpolation and extrapolation. We have also argued that the psychological spaces naturally emerge from the neural codings.

Although there exist examples of topographic representations in the brain, the spatial representations are typically not topographically organized. This is not even the case for the representations of physical space in the hippocampus. Instead, a population code is used to implicitly represent the spaces. The main advantage of this is that it allows the brain to potentially learn any functional mapping and not only those that can be represented by mappings between two-dimensional spaces.

The main function of spatial representations is to make the mapping from perception to action more efficient. Many models of computations with population codes use explicit representations of perceptual and motor variables. This is useful when investigating what the model is doing, but does not mean that we should expect to find such neurons in the brain, where there is no need to decode the population codes until the final stage when they are used to produce movements. To reveal the low-dimensional spatial representations and to make them match the psychological results, it is necessary to decode the population codes in a low-dimensional space but such a decoding is never explicitly required by the brain itself.

By analyzing the neural representations and reducing them to low-dimensional representations, we have argued that they to a large extent can explain the structure of the psychological spaces. Moreover, we have shown how spatial representations are useful as a basis for categorization and sensory-motor mappings and how they can be implicitly coded by populations of neurons. This suggests that spatial representations can be found everywhere in the brain.

#### AUTHOR CONTRIBUTIONS

CB and PG contributed equally to the reported research and the writing of the article.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Balkenius and Gärdenfors. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

## Information Compression, Multiple Alignment, and the Representation and Processing of Knowledge in the Brain

J. Gerard Wolff\*

*CognitionResearch.org, Menai Bridge, UK*

The *SP theory of intelligence*, with its realization in the *SP computer model*, aims to simplify and integrate observations and concepts across artificial intelligence, mainstream computing, mathematics, and human perception and cognition, with information compression as a unifying theme. This paper describes how abstract structures and processes in the theory may be realized in terms of neurons, their interconnections, and the transmission of signals between neurons. This part of the SP theory—*SP-neural*—is a tentative and partial model for the representation and processing of knowledge in the brain. Empirical support for the SP theory—outlined in the paper—provides indirect support for SP-neural. In the abstract part of the SP theory (SP-abstract), all kinds of knowledge are represented with *patterns*, where a pattern is an array of atomic symbols in one or two dimensions. In SP-neural, the concept of a "pattern" is realized as an array of neurons called a *pattern assembly*, similar to Hebb's concept of a "cell assembly" but with important differences. Central to the processing of information in SP-abstract is information compression via the matching and unification of patterns (ICMUP) and, more specifically, information compression via the powerful concept of *multiple alignment*, borrowed and adapted from bioinformatics. Processes such as pattern recognition, reasoning and problem solving are achieved via the building of multiple alignments, while unsupervised learning is achieved by creating patterns from sensory information and also by creating patterns from multiple alignments in which there is a partial match between one pattern and another. It is envisaged that, in SP-neural, short-lived neural structures equivalent to multiple alignments will be created via an inter-play of excitatory and inhibitory neural signals. It is also envisaged that unsupervised learning will be achieved by the creation of pattern assemblies from sensory information and from the neural equivalents of multiple alignments, much as in the non-neural SP theory—and significantly different from the "Hebbian" kinds of learning which are widely used in the kinds of artificial neural network that are popular in computer science. The paper discusses several associated issues, with relevant empirical evidence.

Keywords: multiple alignment, cell assembly, information compression, unsupervised learning, artificial intelligence

#### Edited by:

*Asim Roy, Arizona State University, USA*

#### Reviewed by:

*Jonathan C. W. Edwards, University College London, UK Luis C. Lamb, Federal University of Rio Grande do Sul, Brazil*

> \*Correspondence: *J. Gerard Wolff<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Cognition, a section of the journal Frontiers in Psychology*

Received: *18 May 2016* Accepted: *29 September 2016* Published: *03 November 2016*

#### Citation:

*Wolff JG (2016) Information Compression, Multiple Alignment, and the Representation and Processing of Knowledge in the Brain. Front. Psychol. 7:1584. doi: 10.3389/fpsyg.2016.01584*

#### 1. INTRODUCTION

The SP theory of intelligence, and its realization in the SP computer model, is a unique attempt to simplify and integrate observations and concepts across artificial intelligence, mainstream computing, mathematics, and human perception and cognition. The name "SP" derives from the central importance in the theory of information compression, something that may be seen as a process of maximizing the Simplicity of a body of information, by removing information that is repeated, whilst retaining as much as possible of its nonrepeated expressive Power. Also, the theory itself may be seen to compress empirical information by combining simplicity in the theory with wide-ranging explanatory and descriptive power.

This paper, which draws on Wolff (2006, chapter 11) with revisions and updates, describes how abstract structures and processes in the SP theory may be realized in terms of neurons, their interconnections, and the transmission of impulses between neurons. This part of the SP theory—called SP-neural—may be seen as a tentative and partial theory of the representation and processing of knowledge in the brain. As such, it may prove useful as a source of ideas for theoretical and empirical investigations in the future. For the sake of clarity, the abstract parts of the theory, excluding SP-neural, will be referred to as SP-abstract.

It is envisaged that SP-neural will be further developed in the form of a computer model. As with the existing computer model of SP-abstract (which, unless otherwise stated, will be referred to as "the SP computer model"), the development of the new computer model of SP-neural will help to guard against vagueness in the theory, it will serve as a means of testing ideas to see whether or not they work as anticipated, and it will be a means of demonstrating what the model can do, and validating it against empirical data.

The next section says something about the theoretical orientation of this research. Then SP-abstract will be described briefly as a foundation for the several sections that follow, which describe aspects of SP-neural and associated issues.

#### 2. THEORETICAL ORIENTATION

Cosmologist John Barrow has written that "Science is, at root, just the search for compression in the world" (Barrow, 1992, p. 247), an idea which may be seen to be equivalent to Occam's Razor because, in accordance with the remarks above about the name "SP" and the theory itself, a good theory should combine conceptual Simplicity with descriptive or explanatory Power.

This works best when the range of phenomena to be described or explained is large. But this has not always been observed in practice: Newell (1973, p. 303) urged researchers in psychology to address "a genuine slab of human behavior"; and McCorduck (2004, pp. 417, 424) has described how research in artificial intelligence became fragmented into many narrow sub-fields.

In the light of these observations, and in the spirit of research on "unified theories of cognition" (Newell, 1990) and "artificial general intelligence1 ," the SP programme of research has attempted to simplify and integrate observations and concepts across a broad canvass, resisting the temptation to concentrate only on one or two narrow areas.

#### 3. SP-ABSTRACT IN BRIEF

As a basis for the description of SP-neural, this section provides a brief informal account of SP-abstract. The theory is described most fully in Wolff (2006) and quite fully but more briefly in Wolff (2013). Details of other publications in the SP programme, most of them with download links, are shown on (http://www. cognitionresearch.org/sp.htm).

#### 3.1. Origins and Foundations of the SP Theory

The origins of SP theory are mainly in a body of research by Attneave (1954) and Barlow (1959, 1969) and others suggesting that much of the workings of brains and nervous systems may be understood as compression of information, and my own research on language learning (summarized in Wolff, 1988) suggesting that, to a large extent, the learning of language may be understood in the same terms. There is more about the foundations of the theory in Wolff (2014d).

#### 3.2. Elements of SP-Abstract

In SP-abstract, all kinds of knowledge are represented with patterns, where a pattern is an array of atomic symbols in one or two dimensions. At present, the SP computer model2 works only with 1D patterns but it is envisaged that the model will be generalized to work with 2D patterns. In this connection, a "symbol" is simply a "mark" that can make a yes/no match with any other symbol—no other result is permitted.

In most of the examples shown in this paper, symbols are shown as alphanumeric characters or short strings of characters but, when the SP system is used to model biological structures and processes, such representations may be interpreted as lowlevel elements of perception such as formants or formant ratios in the case of speech or lines and junctions between lines in the case of vision (see also Section 4.2).

To help cut through mathematical complexities associated with information compression, the SP system—SP-abstract and its realization in the SP computer model—is founded on a simple, "primitive" idea: that information may be compressed by finding full or partial matches between patterns and merging or "unifying" the parts that are the same. This principle— "Information Compression via the Matching and Unification of Patterns" (ICMUP)—provides the foundation for a powerful concept of multiple alignment, borrowed and adapted from bioinformatics. The multiple alignment concept, outlined in Section 3.5, below, is itself central in the workings of SP-abstract

1 See, for example, "Artificial General Intelligence", Wikipedia, http://bit.ly/ 1ZxCQPo, retrieved 2016-01-19.

2The current version of the SP computer model is SP71, the source code for which may be downloaded via a link near the bottom of www.cognitionresearch.org/sp.htm. This version of the computer model is very similar to SP70, described in Wolff (2006, Sections 3.9.2, 9.2).

and is the key to versatility and adaptability in the SP system. It has the potential to be as significant for the understanding of "intelligence" in a broad sense as is DNA for biological sciences.

#### 3.3. SP Patterns, Multiple Alignment, and the Representation and Processing of Knowledge

In themselves, SP patterns are not very expressive. But in the multiple alignment framework (Section 3.5) they become a very versatile medium for the representation of diverse forms of knowledge. And the building of multiple alignments, together with processes for unsupervised learning (Sections 3.4, 3.7), has proved to be a powerful means of modeling diverse aspects of intelligence.

The two things together—SP patterns and multiple alignment—have the potential to be a "Universal Framework for the Representation and Processing of Diverse Kinds of Knowledge" (UFK), as discussed in Wolff (2014c, Section III).

An implication of these ideas is that there would not, for example, be any difference between the representation and processing of non-syntactic cognitive knowledge and the representation and processing of the syntactic forms of natural language. A framework that can accommodate both kinds of knowledge is likely to facilitate their seamless integration, as discussed in Section 3.8.2.

#### 3.4. Early Stages of Learning

The SP theory is conceived as a brain-like system that receives New patterns via its "senses" and stores some or all of them, in compressed form, as Old patterns. In broad terms, this is how the system learns.

In the SP system, all learning is "unsupervised3 ," meaning that it does not depend on assistance by a "teacher," the grading of learning materials from simple to complex, or the provision of "negative" examples of concepts to be learned meaning examples that are marked as "wrong" (cf. Gold, 1967). Notwithstanding the importance of schools and colleges, it appears that most human learning is unsupervised. Other kinds of learning, such as "supervised" learning (learning from labeled examples)4 , or "reinforcement" learning (learning with carrots and sticks)5 , may be seen as special cases of unsupervised learning (Wolff, 2014b, Section V).

At the beginning of processing by the system, when the repository of Old patterns is empty6 , New patterns are stored as they are received but with the addition of system-generated "ID" symbols at the beginning and end.  Here, the lower-case letters are atomic symbols that may represent actual letters but could represent basic elements of speech (such as formant ratios or formant transitions), or basic elements of vision (such as lines or corners), and likewise with other sensory data.

Later, when some Old patterns have been stored, the system may start to recognize full or partial matches between New and Old patterns. If a New pattern is exactly the same as an Old pattern (excluding the IDsymbols), then frequency measures for that pattern and its constituent symbols are incremented. These measures, which are continually updated at all stages of processing, have an important role to play in calculating probabilities of structures and inferences and in guiding the processes of building multiple alignments (Section 3.5) and unsupervised learning.

With partial matches, the system will form multiple alignments like the one shown in **Figure 1**, with a New pattern in row 0 and an Old pattern in row 1.

From a partial match like this, the system creates Old patterns from the parts that match each other and from the parts that don't. Each newly-created Old pattern will be given systemgenerated ID-symbols.  In addition, the system forms an abstract pattern like this: "E 1 B #B D #D C #C #E" which records the sequence ["B 1 t h e #B," ("D 1 s m a l l #D" or "D 2 b i g #D"), "C 1 h o u s e #C"] in terms the ID-symbols of the constituent patterns.

Notice how "s m a l l" and "b i g" have both been given the ID-symbol "D" at their beginnings and the ID-symbol "#D" at their ends. These additions, coupled with the use of the same two ID-symbols in the abstract pattern "E 1 B #B D #D C #C #E" has the effect of assigning "s m a l l" and "b i g" to the same syntactic category, which looks like the beginnings of the "adjective" part of speech.

The overall result in this example is a collection of SP patterns that functions as a simple grammar to describe the phrases the small house and the big house.

In practice, the SP computer model may form many other multiple alignments, patterns and grammars which are much less tidy than the ones shown. But, as outlined in Sections 3.5, 3.7, the system is able to home in on structures that are "good" in terms of information compression.

As we shall (see Sections 3.5, 3.8.1, and 6), SP patterns, within the SP system, are remarkably versatile and expressive, with at least the power of context-sensitive grammars (Wolff, 2006, Chapter 5).

**0 t h e s m a l l h o u s e 0 ||| || ||| 1 A 1 t h e b i g h o u s e #A 1** FIGURE 1 | A multiple alignment produced by the SP computer model showing a partial match between a New pattern (in row 0) and an Old pattern (in row 1).

3 See "Unsupervised learning," Wikipedia, bit.ly/22nEPL2, retrieved 2016-03-17. 4 See "Supervised learning," Wikipedia, bit.ly/1nR4ybK, retrieved 2016-03-17.

5 See "Reinforcement learning," Wikipedia, bit.ly/1R0RoDv, retrieved 2016-03-17. 6Although it is likely that, contrary to what Noam Chomsky and others have suggested, a newborn child does not have any kind of detailed knowledge of the structure of natural language, it is likely he or she does have inborn knowledge such as how to suck milk from a breast. In this respect (and others), the SP theory, insofar it is seen as a model of human cognition, is not entirely accurate.

#### 3.5. The Multiple Alignment Concept

The multiple alignment shown in **Figure 1** is unusually simple because it contains only two patterns. More commonly, the system forms "good" multiple alignments like the one shown in **Figure 2**, with one New pattern (in row 0) and several Old patterns (one in each of rows 1–9)7 . As a matter of convention, the New pattern is always shown in row 0, but the order of the Old patterns across the other rows is not significant.

A multiple alignment like the one shown in **Figure 2** is built in stages, using heuristic search at each stage to weed out structures that are "bad" in terms of information compression and retaining those that are "good." Problems of computational complexity are reduced or eliminated by a scaling back of ambition: instead of searching for theoretically-ideal solutions, one merely searches for solutions that are "good enough."

In this example, multiple alignment achieves the effect of parsing the sentence into parts and sub-parts, such as a sentence ("S") defined by the pattern in row 6, one kind of noun phrase ("NP") defined by the pattern that appears in row 5, and another kind of noun phrase shown in row 8, a verb phrase ("VP") defined by the pattern in row 3, nouns ("N") defined by the patterns in rows 4 and 7, and so on. But there is much more than this to the multiple alignment concept as it has been developed in the SP programme. It turns out to be a remarkably versatile framework for the representation and processing of diverse kinds of knowledge—non-verbal patterns and pattern recognition, logical and probabilistic kinds of "rules" and several kinds of reasoning, and more (Sections 3.8.1, 6).

A point worth mentioning here is that, although the multiple concept is entirely non-hierarchical, it can model several kinds of hierarchy and heterarchy (Section 3.8.1), as illustrated by the parsing example in **Figure 2**. And such hierarchies or heterarchies may not always be "strict" because any pattern may be aligned with any other pattern and, within one multiple alignment, any pattern may be aligned with two or more other patterns.

#### 3.6. Deriving a Code Pattern from a Multiple Alignment

From a multiple alignment like the one shown in **Figure 2**, the SP system may derive a code pattern—a compressed encoding of the sentence—as follows: scan the multiple alignment from left to right, identifying the ID-symbols that are not matched with any other symbol and create an SP pattern from the sequence of such symbols. In this case, the result is the pattern "S 0 2 4 3 7 6 1 5 #S." This code pattern has several existing or potential uses including:

#### 3.7. Later Stages of Learning

As we saw in Section 3.4, the earliest stage of learning in SP-neural—when the repository of Old patterns is empty or nearly so—is largely a matter of absorbing New information

7 In this case, the SP computer model was supplied with an appropriate set of Old patterns. It did not learn them for itself.

directly with little modification except for the addition of systemgenerated ID-symbols. Later, when there are more Old patterns in store, the system begins to create Old patterns from partial matches between New and Old patterns. Part of this process is the creation of abstract patterns that describe sequences of lower-level patterns.

As the system begins to create abstract patterns, it will also begin to form multiple alignments like the one shown in **Figure 2**. And, as it begins to form multiple alignments like that, it will also begin to form code patterns, as described in Section 3.6.

At all stages of learning, but most prominent in the later stages, is a process of inferring one or more grammars that are "good" in terms of their ability to encode economically all the New patterns that have been presented to the system. Here, a "grammar" is simply a collection of SP patterns8 .

Inferring grammars that are good in terms of information compression is, like the building multiple alignments, a stage-bystage process of heuristic search through the vast abstract space of alternatives, discarding "bad" alternatives at each stage, and retaining a few that are "good." As with the building of multiple alignments, the search aims to find solutions that are "good enough," and not necessarily perfect. These kinds of heuristic search may be performed by means of genetic algorithms, simulated annealing, and other heuristic techniques.

It is envisaged that the SP computer model will be developed so that, in this later phase of learning, learning processes will be applied to code patterns as well as to New patterns. It is anticipated that this may overcome two weaknesses in the SP computer model as it is now: that, while it forms abstract patterns at the highest level, it does not form abstract patterns at intermediate levels; and that it does not recognize discontinuous dependencies in knowledge (Wolff, 2013, Section 3.3).

In Wolff (2006, Chapter 9), there is a much fuller account of unsupervised learning in the SP computer model.

#### 3.8. Evaluation of SP-Abstract

The SP theory in its abstract form may be evaluated in terms of "simplicity" and "power" of the theory itself (discussed in Section 3.8.1 next), in terms of its potential to promote simplification and integration of structures and functions in natural or artificial systems that conform to the theory (Section 3.8.2 below), and in comparison with other AI-related systems.

#### 3.8.1. Simplicity and Power

In terms of the principles outlined in Section 2, the SP system, with multiple alignment center stage, scores well. One relatively simple framework has strengths and potential in the representation of several different kinds of knowledge, in several different aspects of AI, and it has several potential benefits and applications:

• Representation and processing of diverse kinds of knowledge. The SP system (SP-abstract) has strengths and potential in the representation and processing of: class hierarchies and heterarchies, part-whole hierarchies and heterarchies, networks and trees, relational knowledge, rules used in several kinds of reasoning, patterns with pattern recognition, images with the processing of images (Wolff, 2014a), structures in planning and problem solving, structures in three dimensions (Wolff, 2014a, Section 6), knowledge of sequential and parallel procedures (Wolff, 2014b, Section IV-H). It may also provide an interpretive framework for structures and processes in mathematics (Wolff, 2014d, Section 10).

There is a fuller summary in Wolff (2014c, Section III-B) and much more detail in Wolff (2006, 2013).

In short, the SP theory, in accordance with Occam's Razor, demonstrates a favorable combination of simplicity and power across a broad canvass. As in other areas of science, this should increase our confidence in the validity and generality of the theory.

#### 3.8.2. Simplification and Integration

Closely related to simplicity and power in the SP theory are two potential benefits arising from the use of one simple format (SP patterns) for all kinds of knowledge and one relatively simple framework (chiefly multiple alignment) for the processing of all kinds of knowledge:

8The term "grammar" has been adopted partly because of the origins of the SP system in research on the learning of natural language (Wolff, 1988) and partly because the term has come to be used in areas outside computational linguistics, such as pattern recognition.

With regard to the seamless integration of diverse kinds of knowledge, this is clearly needed in the understanding and production of natural language. To understand what someone is saying or writing, we obviously need to be able to connect words and syntactic structures with their non-syntactic meanings, and likewise, in reverse, when we write or speak to convey some meaning.

This has not yet been explored in any depth with the SPabstract conceptual framework but preliminary trials with the SP computer model suggest that it is indeed possible to define syntactic-semantic structures in a set of SP patterns and then, with those patterns playing the role of Old patterns, to analyse a sample sentence and to derive its meanings (Wolff, 2006, Section 5.7, Figure 5.18), and, in a separate exercise with the same set of Old patterns, to derive the same sentence from a representation of its meanings (Wolff, 2006, Figure 5.19).

#### 3.8.3. Distinctive Features and Advantages of the SP System Compared with Other AI-Related Systems

In several publications, such as Wolff (2006, 2007, 2014e), potential benefits and applications of the SP system have been described.

More recently, it has seemed appropriate to say what distinguishes the SP system from other AI-related systems and, more importantly, to describe advantages of the SP system compared AI-related alternatives. Those points have now been set out in some detail in The SP theory of intelligence: its distinctive features and advantages (Wolff, 2016). Of particular relevance to this paper are the several advantages of the SP system compared with systems for deep learning in artificial neural networks (Wolff, 2016, Section V).

Since many AI-related systems may also be seen as models of cognitive structures and processes in brains, Wolff (2016) may also be seen to demonstrate the relative strength of the SP system in modeling aspects of human perception and cognition.

In this connection, the SP system appears to have some advantages compared with concepts developed in research in "neural-symbolic computation," described in d'Avila Garcez et al. (2015), de Penning et al. (2011), d'Avila Garcez et al. (2009), Komendantskaya et al. (2007), and d'Avila Garcez (2007) amongst other publications. The main apparent advantages are:

#### 4. INTRODUCTION TO SP-NEURAL

As we have seen in Section 3, SP-abstract is a relatively simple system with descriptive and explanatory power across a wide range of observation and phenomena in artificial intelligence and related areas. How can such a system have anything useful to say about the extraordinary complexity of brains and nervous systems, both in their structure and in their workings?

An answer in brief is that SP-neural—a realization of SPabstract in terms of neurons, their interconnections, and the transmission of impulses between neurons—may help us to interpret neural structures and processes in terms of the relatively simple concepts in SP-abstract. To the extent that this is successful, it may—like any good theory in any field—help us to understand empirical phenomena in our area of interest, it may help us to make predictions, and it may suggest lines of investigation.

It is anticipated that SP-neural will work in broadly the same way as SP-abstract, but the characteristics of neurons and their interconnections raise some issues that do not arise in SP-abstract and its realization in the SP computer model. These issues will be discussed at appropriate points in this and subsequent sections.

This section introduces SP-neural in outline, and sections that follow describe aspects of the theory in more detail, drawing where necessary on aspects of SP-abstract that have been omitted from or only sketched in Section 3.

#### 4.1. Sensory Data and the Receptor Array

**Figure 3**, to be discussed in this and the following subsections, shows in outline how a portion of the multiple alignment shown in **Figure 2**, may be realized in SP-neural. with associated patterns and symbols.

In the figure, "sensory data" at the bottom means the visual, auditory or tactile data entering the system which, in this case, corresponds with the phrase "t h e b r a v e." In a more realistic illustration, the sensory data would be some kind of analog signal. Here, the letters are intended to suggest the kinds of low-level perceptual primitives outlined below.

It is envisaged that, with most sensory modalities, the receptor array would be located in the primary sensory cortex. Of course,

a lot of processing goes on in the sense organs and elsewhere between the sense organs and the primary sensory cortices. But it seems that most of this early processing is concerned with the identification of the perceptual primitives just mentioned.

As with SP-abstract, it is anticipated that SP-neural will, at some stage, be generalized to accommodate patterns in two dimensions, such as visual images, and then the sensory data may be received in two dimensions, as in the human eye.

Between the sensory data and the receptor array (above it in the figure), there would be, first, cells that are specialized to receive particular kinds of input (auditory, visual, tactile etc.). These send signals to neurons that encode the sensory data as neural symbols, the neural equivalents of "symbols" in SP-abstract. In the receptor array, each letter enclosed in a solid ellipse represents a neural symbol, expressed as a single neuron or, more likely, a small cluster of neurons. As we shall see Section 5.1, the reality is more complex, at least in some cases.

In vision, neural symbols in the receptor array would represent such low-level features as lines, corners, colors, and the like, while in speech perception, they would represent such things as formants, formant ratios and transitions, plosive and fricative sounds, and so on. Whether or how the SP concepts can be applied in the discovery or identification of features like these is an open question (Wolff, 2013, Section 3.3). For now, we shall assume that they can be identified and can be used in the creation and use of higher-level structures.

#### 4.2. Pattern Assemblies

In the rest of **Figure 3**, each broken-line rectangle with rounded corners represents a pattern assembly—corresponding to a "pattern" in SP-abstract. The word "assembly" has been adopted within the expression "pattern assembly" because the concept is quite similar to Hebb's concept of a "cell assembly"—a cluster of neurons representing a concept or other coherent mental entity. Differences between Hebb's concept of a cell assembly and the SP concept of a pattern assembly are described in the Appendix.

Within each pattern assembly, as represented in the figure, each character or group of characters enclosed in a solid-line ellipse represents a neural symbol which, as already mentioned, corresponds to a "symbol" in SP-abstract. As with neural symbols in the receptor array, it is envisaged that each neural symbol would comprise a single neuron or, more likely, a small cluster of neurons.

It is supposed that, within each pattern assembly, there are lateral connections between neural symbols—but these are not shown in the figure.

It is envisaged that most pattern assemblies would represent knowledge that is learned and not inborn, and would be located mainly outside the primary sensory areas of the cortex, in other parts of the sensory cortices. Pattern assemblies that integrate two or more sensory modalities may be located in "association" areas of the cortex.

Research with fMRI recordings from volunteers (Huth et al., 2016) has revealed "semantic maps" that "show that semantic information is represented in rich patterns that are distributed across several broad regions of cortex. Furthermore, each of these regions contains many distinct areas that are selective for particular types of semantic information, such as people, numbers, visual properties, or places. We also found that these cortical maps are quite similar across people, even down to relatively small details"9 . Of course, this research says nothing about whether or not the knowledge is represented with pattern assemblies and their interconnections. But it does apparently confirm that knowledge is stored in several regions of the cortex and throws light on how it is organized.

Although most parts of the mammalian cerebral cortex has six layers and many convolutions, it may be seen, topologically, as a sheet which is very much broader and wider than it is thick. Correspondingly, it is envisaged that 1D and 2D pattern assemblies will be largely "flat" structures, rather like writing or pictures on a sheet of paper. That said, it is quite possible, indeed likely, that pattern assemblies would take advantage of two or more layers of the cortex, not just one.

Incidentally, since 2D SP patterns may provide a basis for 3D models, as described in Wolff (2014a, Sections 6.1, 6.2), flat neural structures in the cortex may serve to represent 3D concepts.

#### 4.3. Connections between Pattern Assemblies

In **Figure 3**, the solid or broken lines that connect with neural symbols represent axons, with arrows representing the direction of travel of neural impulses. Where two or more connections converge on a neural symbol, we may suppose that, contrary to the simplified way in which the convergence is shown in the figure, there would be a separate dendrite for each connection.

Axons represented with solid lines are ones that would be active when the multiple alignment in **Figure 2** is in the process of being identified. Broken-line connections show a few of the many other possible connections.

As mentioned in Section 4.2, it is envisaged that there would be one or more neural connections between neighboring neural symbols within each pattern assembly but these are not marked in the figure.

Compared with what is shown in the figure, it likely that, in reality, there would be more "levels" between basic neural symbols in the receptor array and ID-neural-symbols representing pattern assemblies for relatively complex entities like the words "one," "brave," "the," and "table," as shown in the figure.

In this connection, it is perhaps worth emphasizing that, as with the modeling of hierarchical structures in multiple alignments (Section 3.5), while pattern assemblies may form "strict" hierarchies, this is not an essential feature of the concept, and it is likely that many neural structures formed from pattern assemblies may be only loosely hierarchical or not hierarchical at all.

#### 4.4. SP-Neural, Quantities of Knowledge, and the Size of the Brain

Given the foregoing account of how knowledge may be represented in the brain, a question that arises is "Are there enough neurons in the brain to store what a typical person knows?" This is a difficult question to answer with any precision but an attempt at an answer, described in Wolff (2006, Section 11.4.9), reaches the tentative conclusion that there are. In brief:

9From the website of the Gallant Lab at UC Berkely, retrieved 2016-05-02, http://bit.ly/1WvvLhX. See also "Brain 'atlas' of words revealed," BBC News, 2016- 04-27, bbc.in/1SGESLz.

10This is consistent with another estimate, not quoted in Wolff (2006, Section 11.4.9), that there may be as many as 86 billion neurons in the human brain (Herculano-Houzel, 2012).

#### 4.5. Neural Processing

In broad terms, it is envisaged that, for a task like the parsing of natural language or pattern recognition:

	- Excitatory signals would be sent from those excited neural symbols to pattern assemblies that can receive signals from them directly. In **Figure 3**, these would be all the pattern assemblies except the topmost pattern assembly.
	- Within each pattern assembly, excitatory signals will spread laterally via the connections between neighboring neural symbols.
	- Pattern assemblies would become excited, roughly in proportion to the number of excitatory signals they receive.

As in stage 1, the level of excitation of any pattern assembly would depend on the number of excitatory signals it receives, but building up from stage to stage so that the highest-level pattern assemblies are likely to be most excited.

4. Repeat stages 2 and 3 until there are no more pattern assemblies that can be sent excitatory signals.

The "winning" pattern assembly or pattern assemblies, together with the structures below them that have, directly or indirectly, sent excitatory signals to them, may be seen as neural analogs of multiple alignments (NAMAs), and we may guess that they provide the best interpretations of a given portion of the sensory data.

If the whole sentence, "f o r t u n e f a v o u r s t h e b r a v e," is processed by SP-neural with pattern assemblies that are analogs of the SP patterns provided for the example shown in **Figure 2**, we may anticipate that the overall result would be a pattern of neural excitation that is an analog of the multiple alignment shown in that figure.

When a neural symbol or pattern assembly has been "recognized" by participating in a winning (neural) multiple alignment, we may suppose that some biochemical or physiological aspect of that structure is increased as an at least approximate measure of the frequency of occurrence of the structure, in accordance with the way in which SP-abstract keeps track of the frequency of occurrence of symbols and patterns (Section 3.4).

Some further possibilities are discussed in Sections 5, 9.

#### 5. SOME MORE DETAIL

The bare-bones description of SP-neural in Section 4 is probably inaccurate in some respects and is certainly too simple to work effectively. This section and the ones that follow describe some other features which are likely to figure in a mature version of SP-neural, drawing on relevant empirical evidence where it is available.

#### 5.1. Encoding of Information in the Receptor Array

With regard to the encoding of information in the receptor array, it seems that the main possibilities are these:

	- a. Horizontal distribution of alternatives. The several alternatives may be distributed "horizontally," in a plane that is parallel to the surface of the cortex.
	- b. Vertical distribution of alternatives. The several alternatives may be distributed "vertically" between the outer and inner surfaces of the cortex, and perpendicular to those surfaces.

4. Temporal codes. A stimulus that varies with time may be encoded via "the time-varying pattern of activity in small groups of receptors and central neurons." (Squire et al., 2013).

In support of option 1.a, there is evidence that neurons in the visual cortex (of cats) are arranged in columns perpendicular to the surface of the cortex, where, for example, all the neurons in a given column respond most strongly to a line at one particular angle in the field of view, that—within a "hypercolumn" containing several columns—the preferred angle increases progressively from column to column, and that there are many hypercolumns across the primary visual cortex (Barlow, 1982). "Hubel and Wiesel point out that the organization their results reveal means that each small region, about 1 mm2 at the surface, contains a complete sequence of ocular dominance and a complete sequence of orientation preference." (Barlow, 1982, pp. 148–149).

Leaving out the results for ocular dominance, these observations are summarized schematically in **Figure 4**. In terms of this scheme, the way in which the receptor array is shown in **Figure 3**, is a considerable simplification—each neural symbol in the receptor array in that figure should really be replaced by a hypercolumn.

With something like the intensity of a stimulus, it seems that, at least in some cases: "... activity in one particular population of somatosensory neurons ... leads the CNS to interpret it as painful stimulus ...." (Squire et al., 2013, p. 503), while "An entirely separate population of neurons ... would signal light pressure." (Squire et al., 2013). Since it is likely that relevant receptors appear repeatedly across one's skin, this appears to be another example of option 1.a.

There seems to be little evidence of encoding via option 1.b. Indeed, since the concept of a cortical column is, in effect, defined by the fact that all the neurons in any one column have the same kind of receptive field, this seems to rule out the 1.b option (see also Section 5.2).

FIGURE 4 | Schematic representation of one hypercolumn in the receptor array in the cortex. Each letter represents a neural symbol that responds to a particular small pattern in the sensory data. The ellipsis, "...," in each row and each column represents other neural symbols that would be shown in a more comprehensive representation of the given hypercolumn. Each vertical sequence of letters, all of one kind such as "a" or "b," represents a simple column in the cortex.

But, with respect to option 2, it appears that in some cases, as noted above, the intensity of a stimulus may be encoded via the rates of firing of neurons, together with the numbers of neurons that are activated (option 3). And, since we can perceive and remember time-varying stimuli such as the stroking of a finger across one's skin, or the rising or falling pitch of a note, some kind of temporal encoding must be available (option 4).

Here, it must be acknowledged that options 3 and 4 appear superficially to be outside the scope of the SP theory, in view of the emphasis in many examples on discrete atomic symbols. But, as we know from the success of digital recording, or indeed digital computing, any continuum may be encoded digitally, in keeping with the digital nature of the SP theory. How the SP theory may be applied to the digital encoding and processing of continua has been discussed elsewhere in relation to vision (Wolff, 2014a) and the development of autonomous robots (Wolff, 2014b).

#### 5.2. Why Are There Multiple Neurons with the Same Receptive Fields in Columns in the Cortex?

As we have seen (Section 5.1), some aspects of vision are mediated via columns of neurons in the primary visual cortex in which each column contains many neurons with receptive fields that are all the same, all of them responding, for example, to a line in the visual field with a particular orientation.

Why, at each of several locations across the visual cortex, should there be many neurons with the same receptive field, not just one? There seem to be two possible answers to this question (and they are not necessarily mutually exclusive):

With regard to the first point, the receptor array may have a useful role to play, inter alia, as a short-term memory for many sensory patterns pending their longer-term storage (Section 11). In vision, for example, the receptor array may store many short glimpses of a scene, as outlined in Section 5.6, until such time as further processing may be applied to weld the many glimpses into a coherent structure (Wolff, 2014b) and to transfer that structure to longer-term memory.

### 5.3. The Labeled Line Principle

Section 4.5 suggests that normally, at some early stage in sensory processing, raw sensory data is encoded in terms of the excitation of neuronal symbols in a receptor array, then excited neural symbols send excitatory signals to appropriate neural symbols within pattern assemblies, and pattern assemblies that are sufficiently excited send excitatory signals on to other pattern assemblies, and so on. As we shall see (Section 9), it is likely that, in this processing, there will also be a role for inhibitory processes.

At first sight, it may be thought that, in the same way that each location in the receptor array should provide an alphabet of alternative encodings (Section 5.1), the same should be true for the location of each neural symbol within each pattern assembly. But if a neural symbol in a pattern assembly (let's call it "NS1") receives signals only from neural symbols in the receptor array that represent a given feature, let us say, "a," then, in accordance with the "labeled line" principle (Squire et al., 2013, p. 503), NS1 also represents "a."

For most sensory modalities, this principle applies all the way from each sense organ, through the thalamus, to the corresponding part of the primary sensory cortex11. It seems reasonable to suppose that the same principle will apply onwards from each primary sensory cortex into non-primary sensory cortices and non-sensory association areas.

#### 5.4. How the Ordering or 2D Arrangement of Neural Symbols May Be Respected

In SP-neural, as in SP-abstract and the SP computer model, the process of matching one pattern with another should respect the orderings of symbols. For example, "A B C D" matched with "A B C D" should be rated more highly in terms of information compression than, for example, "A B C D" matched with "C A D B12."

It appears that this problem may be solved by the adoption, within SP-neural, of the following feature of natural sensory systems:

"Receptors within [the retina and skin surface] communicate with ganglion cells and those ganglion cells with central neurons in a strictly ordered fashion, such that relationships with neighbors are maintained throughout. This type of pattern, in which neurons positioned side by side in one region communicate with neurons positioned side-by-side in the next region, is called a typographic pattern." (Squire et al., 2013, p. 504) (emphasis in the original).

#### 5.5. How to Accommodate the Variable Sizes of Sensory Patterns

A prominent feature of human visual perception is that we can recognize any given entity over a wide range of viewing distances, with correspondingly wide variations in the size, on the retina, of the image of that entity.

For any model of human visual perception that is based on a simplistic or naive process for the matching of patterns, this aspect of visual perception would be hard to reproduce or to explain. But the SP system is different: (1) Knowledge of entities that we may recognize are always stored in a compressed form; (2) The process of recognition is a process of compressing the incoming data; (3) The overall effect is that an image of a thing to be recognized can be matched with stored knowledge of that entity, regardless of the original size of the image.

As an example, consider how the concept of an equilateral triangle (as white space bounded by three black lines all of the same length) may be stored and how an image of such a triangle may be recognized. Regarding storage, there are three main redundancies in any image of that kind of triangle: (1) The white space in the middle may be seen as repeated instances of a symbol representing a white pixel; (2) Each of the three sides of the triangle may be seen as repeated instances of a symbol representing a black pixel; and (3) There is redundancy in that the three sides of the triangle are the same.

All three sources of redundancy may be encoded recursively as suggested in **Figure 5**13, which shows a multiple alignment modeling the recognition of a one-dimensional analog of a triangle.

Column 0 shows information about the triangle to be recognized, comprising three "corners" and three sides of the triangle, each one represented by just two "points."

The pattern "LN ln1 point LN #LN #LN" in columns 1 and 2 is a self-referential and thus recursive definition of a line as a sequence of "points." It is self-referential because, within the body of the pattern, it contains a reference to itself via the symbols at the beginning and end of the pattern: "LN #LN." Because there is no limit to this recursion, it may represent a line containing any number of points. In a similar way, a second side is encoded via the same pattern in columns 6 and 7, and, again with the same pattern, the third line is encoded in columns 12 and 12.

In columns 4, 9 and 15 in the figure, the pattern "SG sg1 CR #CR LN #LN #SG" shows one of the three elements of a triangle as a corner ("CR #CR") followed by a line ("LN #LN"). And the recursion to encode multiple instances of that structure is in self-referential occurrences of the pattern "TR tr1 SG #SG TR #TR #TR" in columns 5, 10, and 22. Strictly speaking, the encoding is for a polygon, not a triangle, because there is nothing to stop the recursive repetition of "SG sg1 CR #CR LN #LN #SG." And, in terms of the problem, as described above, the representation is incomplete because there is nothing to show that the three sides of the triangle are the same.

These encodings account for the redundancy in the repetition of points along a line and also the redundancy in the repetition of three sides of a triangle. In a 2D version, they would also account for the redundancy in the white space within the body of the triangle, because they would allow most of the white space to be eliminated via shrinkage of the representation to the minimum needed to express the concept of a triangle.

11Thus, for example, "Even within one function, mappings of neurons [within the thalamus] are preserved so that there is separation of neurons providing touch information from the arm vs. from the leg and of neurons responding to low vs. high sound frequencies ...." (Squire et al., 2013, p. 507). Also, "Nuclei in the central pathways often contain multiple maps." but "The functional significance of multiple maps in general, however, remains to be clarified." (Squire et al., 2013).

12A possible exception is when one pattern is a mirror image or inversion of another, since Leonardo da Vinci, by repute, could read mirror writing as easily as ordinary writing, and it is now well established that people wearing inverting spectacles can learn quite quickly to see the world as if it was the right way up (Stratton, 1897).

13Compared with the multiple alignments shown in **Figures 1**, **2**, this multiple alignment has been rotated by 90◦ . The choice between these alternative presentations of multiple alignments depends entirely on what fits best on the page.

#### 5.6. We See Much Less than We Think We See

we are looking at with great detail from left to right and from top to bottom. But research shows otherwise:

Most people with normal vision have a powerful sense that their eyes are a window on to a kind of cinema screen that shows what

• In the phenomenon of inattentional blindness, people may fail to notice salient things in their visual fields when they are looking for something else, even if they are trained observers. In a recent demonstration (Drew et al., 2013), radiologists were asked to search for lung-nodules in chest x-rays but many of them (83%) failed to notice the image of a gorilla, 48 times the size of the average nodule, that was inserted into one of the radiographs.

It seems that part of the reason for this failure to see things is that photoreceptors are concentrated at the fovea (Squire et al., 2013, p. 502), and cones are only found in that region (Squire et al., 2013), so that, with two eyes, we are, to a large extent, looking at the world through a keyhole composed of two circumscribed and largely overlapping views, one from each eye.

It seems that our sense that the world is displayed to us on a wide and deep cinema screen is partly because our perception of any given scene draws heavily on our memories of similar scenes and partly because we can piece together what will normally be a partial view of what we are looking at from many short glimpses through the "keyhole" as we move our gaze around the scene.

The SP theory provides an interpretation for these things as follows:

With regard to the second point, it should perhaps be said that partial overlap between "keyhole" views is not an essential part of building up a big picture from smaller views. But if two or more views do overlap, it is useful if they can be stitched together,

14See "Blind spot (vision)," Wikipedia, bit.ly/1oI0vyI, retrieved 2016-04-08.

thus removing the overlap. And partial overlap may be helpful in establishing the relative positions of two or more views.

#### 5.7. A Resolution Problem and Its Possible Resolution

As we have seen (Section 5.1), each hypercolumn in the primary visual cortex of cats occupies about 1 mm2 at the surface of the cortex, and it seems likely that each such hypercolumn provides a means of encoding one out of an alphabet of perceptual primitives, such as a line at a particular angle.

Assuming that this interpretation is correct, and if we view the primary visual cortex as if it was film in an old-style camera or the image sensor in a digital camera, it may seem that the encoding of perceptual primitives, with 1 mm2 for each one, is remarkably crude. How could such a system—with the area of the primary visual cortex corresponding to the area of our field of view—create that powerful sense that, through our eyes, we see a detailed "cinema screen" view of the world (Section 5.6).

Part of the answer is probably that we see much less than we think we see (Section 5.6). But it seems likely that another part of the answer is to reject the assumption that the whole of the primary visual cortex corresponds to the area of our field of view. In the light of the remarks in Section 5.6, it seems more likely that, normally, in each of the previously-mentioned glimpses of a scene, all of the primary visual cortex or most of it is applied in the assimilation and processing of information capture by the fovea and, perhaps, parts of the retina that are near to the fovea.

In support of this idea: "Cortical magnification describes how many neurons in an area of the visual cortex are 'responsible' for processing a stimulus of a given size, as a function of visual field location. In the center of the visual field, corresponding to the fovea of the retina, a very large number of neurons process information from a small region of the visual field. If the same stimulus is seen in the periphery of the visual field (i.e., away from the center), it would be processed by a much smaller number of neurons. The reduction of the number of neurons per visual field area from foveal to peripheral representations is achieved in several steps along the visual pathway, starting already in the retina (Barghout-Stein, 1999)"15 .

With this view of visual processing, what appears superficially to be a rather course-grained recording and analysis of visual data, may actually be very much more detailed. As described in Section 5.6, it seems likely that our view of any scene is built up partly from memories and partly from many small snapshots or glimpses of the scene. And it seems like that each such snapshot or glimpse is processed using a relatively large neural resource.

#### 5.8. Grandmother Cells, Localist and Distributed Representations

In terms of concepts that have been debated about how knowledge may be represented in the brain, the ID-neuralsymbols for any pattern assembly are very much like the concept of a grandmother cell—a cell or small cluster of cells in one's brain that represents one's grandmother so that, if the cell or

15See "Cortical magnification," Wikipedia, bit.ly/1qJsQX1, emphasis in the original, retrieved 2016-04-14.

cells were to be lost, one would lose the ability to recognize one's grandmother16 .

It seems that the weight of observational and experimental evidence favors the belief that such cells do exist (Gross, 2002; Roy, 2013). This is consistent with the observation that people who have suffered a stroke or are suffering from dementia may lose the ability to recognize members of their close family.

Since SP-neural, like Hebb's (1949) theory of cell assemblies, proposes that concepts are represented by coherent groups of neurons in the brain, it is very much a "localist" type of theory. As such, it is quite distinct from "distributed" types of theory that propose that concepts are encoded in widely-distributed configurations of neurons, without any identifiable location or center.

However, just to confuse matters, SP-neural does not propose that all one's knowledge about one's grandmother would reside in a pattern assembly for that lady. Probably, any such pattern assembly would, in the manner of object-oriented design as discussed in Section 6 and illustrated in **Figure 6**, be connected to and inherit features from a pattern assembly representing grandmothers in general, and from more general pattern assemblies such as pattern assemblies for such concepts as "person" and "woman." And again, a pattern assembly for "person" would not be the sole repository of all one's knowledge about people. That pattern assembly would, in effect, contain "references" to pattern assemblies describing the parts of a person, their physiology, their social and political life, and so on.

Thus, while SP-neural is unambiguously localist, it proposes that knowledge of any entity or concept is likely to be encoded not merely in one pattern assembly for that entity or concept but also in many other pattern assemblies in many parts of the cortex, and perhaps elsewhere.

#### 5.9. Positional Invariance

With something simple like a touch on the skin, or a pin prick, it is not too difficult to see how the sensation may be transmitted to the brain via any one of many relevant receptors located in many different areas of the skin. But with something more complex, like an image on the retina of a table, a house, or a tree, and so on, it is less straightforward to understand how we might recognize such a thing in any part of our visual field.

For each entity to be recognized, it seems necessary at first sight to provide connections, directly or indirectly, from every part of the receptor array to the relevant pattern assembly. In terms of the schematic representation shown in **Figure 3**, it would mean repeating the connections for "t h e" and "b r a v e" in each of many parts of the receptor array. Bearing in mind the very large number of different things we may recognize, the number of necessary connections would become very large, perhaps prohibitively so.

However, things may be considerably simplified via either or both of two provisions:

1. For reasons outlined in Section 5.6, it seems likely that, with vision, we build up our perception of a scene, partly from memories of similar scenes and partly via many relatively narrow "keyhole" views of what is in front of us. If that is correct, and if, as suggested in Section 5.7, most of the primary visual cortex is devoted to analysing information received via the fovea and, perhaps, via parts of the retina that are very close to the fovea, then the need to provide for any given pattern in many parts of the receptor array may be greatly reduced. Since, by moving our eyes, we may view any part of a scene, it is possible that any given entity would need only one or two sets of connections between the receptor array and the pattern assembly for that entity.

2. As noted in Section 4.3, it seems likely that, with regard to **Figure 3**, there would, in a more realistic example, be several levels of structure between neural symbols in the receptor array and relatively complex structures like words. At the first level above the receptor array there would be pattern assemblies for relatively small recurrent structures, and the variety of such structures would be relatively small. This should ease any possible problems in connecting the receptor array to pattern assemblies.

If it turns out that the number of necessary connections is indeed too large to be practical, or if there is empirical evidence against such numbers, then a possible alternative to what has been sketched in this paper is some kind of dynamic system for the making and breaking of connections between the receptor array and pattern assemblies. It seems likely that permanent or semipermanent connections would be very much more efficient and the balance of probabilities seems to favor such a scheme.

In connection with positional invariance, it is relevant to note that "... lack of localization is quite common in higherlevel neurons: receptive fields become larger as the features they represent become increasingly complex. Thus, for instance, neurons that respond to faces typically have receptive fields that cover most of the visual space. For these cells, large receptive fields have a distinct advantage: the preferred stimulus can be identified no matter where it is located on the retina." (Squire et al., 2013, p. 579). A tentative and partial explanation of this observation is that repetition of neurons that are sensitive to each of several categories of low-level feature—in the receptor array and as ID-neural-symbols for "low-level" pattern assemblies—is what allows positional invariance to develop at higher levels.

#### 6. NON-SYNTACTIC KNOWLEDGE IN SP-NEURAL

As was emphasized in Section 3, the SP system (SP-abstract) has strengths and potential in the representation and processing of several different kinds of knowledge, not just the syntax of natural language. That versatility has been achieved using the mechanisms in SP-abstract that were outlined in that section. If those mechanisms can be modeled in SP-neural, it seems likely that the several kinds of knowledge that may be represented and processed in SP-abstract may also be represented and processed in SP-neural.

As an illustration, **Figure 6** shows a simple example of how, via multiple alignment, the SP computer model may recognize an unknown creature at several different levels of abstraction,

16See "Grandmother cell," Wikipedia, bit.ly/1UDulyV, retrieved 2016-08-26.

and **Figure 7** suggests how part of the multiple alignment, with associated patterns, may be realized in terms of pattern assemblies and their inter-connections.

**Figure 6** shows the best multiple alignment found by the SP computer model with four symbols representing attributes of an unknown creature (shown in column 0) and a collection of Old patterns representing different creatures and classes of creature, some of which are shown in columns 1–4, one pattern per column. In a more detailed and realistic example, symbols like "eats," "retractile-claws," and "breathes," would be represented as patterns, each with its own structure.

From this multiple alignment, we can see that the unknown creature has been identified as an animal (column 4), as a mammal (column 3), as a cat (column 2) and as a specific cat, "Tibs" (column 1). It is just an accident of how the SP computer model has worked in this case that the order of the patterns across columns 1–4 of the multiple alignment corresponds with the level of abstraction of the classifications. In general, the order of patterns in columns above 0 is entirely arbitrary, with no significance.

**Figure 7** shows how part of the multiple alignment from **Figure 6** may be realized in SP-neural. The figure contains pattern assemblies for "animal" and "mammal," corresponding to patterns from columns 4 and 3 of the multiple alignment. Notice that the left-right order of the pattern assemblies is different from the order of the patterns in the multiple alignment, in accordance with the remarks, above, about the workings of the SP computer model, and also because there is no reason to believe that pattern assemblies are represented in any particular order.

Neural connections amongst the things that have been mentioned so far are very much the same as alignments between neural symbols in **Figure 6**: "eats" on the left connects with "eats" in the "animal" pattern assembly; "furry" connects with "furry" in the "mammal" pattern assembly, and the "A" and "#A" connections for those two pattern assemblies correspond with the alignments of symbols in the multiple alignment. As in **Figure 3**, some neural connections are shown with broken lines to suggest that they would be relatively inactive during the neural processing which identifies one or more "good" NAMAs. And as before, it is envisaged that there would be one or more neural connections between each neural symbol and its immediate neighbors within each pattern assembly, but these are not marked in the figure.

The inclusion of a pattern assembly for "reptile" in **Figure 7**, with some of its neural connections, is intended to suggest some of the processing involved in identifying one or more winning NAMAs. In the same way that the pattern for "mammal" is

creature and a collection of Old patterns representing different creatures and classes of creature. Adapted from Figure 6.7 in Wolff (2006), with permission.

receiving excitatory signals from the pattern for "animal," one would expect excitatory signals to flow to pattern assemblies for the other main groups of animals, including reptiles. Ultimately, "reptile" would fail to feature in any winning NAMA because of evidence from the neural symbols "furry," "purrs," and "whitebib."

#### 7. REPETITION AND RECURSION

Like any good database or dictionary, the repository of Old patterns in SP-abstract should only contain one copy of any given SP pattern. But in something like Jack Sprat could eat no fat, His wife could eat no lean, the words could, eat, and no each occur twice. With an example like this, it seems reasonable to suppose that there is only one stored pattern for each of the repeated words, and likewise for the many other examples of entities that are repeated within something larger, witness the many legs of a centipede.

In SP-abstract, this apparent difficulty has been overcome by saying that each SP pattern in a multiple alignment is an appearance of the pattern, not the pattern itself—which allows us to have multiple instances of a pattern in a multiple alignment without breaking the rule that the repository of Old patterns should contain only one copy of each pattern. But in SPneural, it is not obvious how to create an "appearance" of a pattern assembly that is not also a physical structure of neurons and their interconnections—but the speed with which we can understand natural language seems to rule out what appears to be the relatively slow growth of new neurons and their interconnections.

How we can create new mental structures quickly arises again in other connections, as discussed in Section 11. If we duck these questions for the time being and return to parsing, it may be argued that with something like Jack Sprat could eat no fat, His wife could eat no lean, the first instance of could is represented only for the duration of the word by the stored pattern for could, so that the same pattern can be used again to represent the second instance of could—and likewise for eat and no. But it appears that this line of reasoning does not work with a recursive structure like the very very very fast car.

Native speakers of English know that with a phrase like the very very very fast car, the word very may in principle be repeated any number of times. This observation, coupled with the observation that recursive structures are widespread in English and other natural languages, suggests strongly that the most appropriate parsing of the phrase is something like the multiple alignment shown in **Figure 8**. Here, the repetition of very is represented via three appearances of the pattern "ri ri1 ri #ri i #i #ri," a pattern which is self-referential because the inner pair of symbols "ri #ri" can be matched with the same two symbols, one at the beginning of the pattern and one at the end. Because the recursion depends on at least two instances of "ri ri1 ri #ri i #i #ri" being "live" at the same time, it seems necessary for SP-neural to be able to model multiple appearances of any pattern.

That conclusion, coupled with the above-mentioned arguments from the speed at which we can speak, and the speed

with which we can imagine new things, argues strongly that SP-neural—and any other neural theory of cognition—must have a means of creating new mental structures quickly. It seems unlikely that these things could be done via the growth of new neurons and their interconnections.

The tentative answer suggested here is that, in processes like parsing or pattern recognition, including examples with recursion like that shown in **Figure 8**, virtual copies of pattern assemblies may be created and destroyed very quickly via the switching on and switching off of synapses (Section 11). Clearly, more detail is needed for a fully satisfactory answer.

Pending that better answer, **Figure 9** shows tentatively how recursion may be modeled in SP-neural, with neural symbols and pattern assemblies corresponding to selected symbols and patterns in **Figure 8**. On the left of that figure, we can see how the neural symbol "very" connects with a matching neural symbol in the pattern assembly "i i1 very #i." Further right, we can see how the first and last neural symbols in "i i1 very #i" connect with matching neural symbols in the pattern assembly "ri ri1 ri #ri i #i #ri."

In the figure, the self-referential nature of the pattern assembly "ri ri1 ri #ri i #i #ri" can be seen in the neural connection between "ri" at the beginning of that pattern assembly and the matching neural symbol in the body of the same pattern assembly, and likewise for "#ri" at the end of the pattern assembly. Although it is unclear how this recursion may achieve the effect of repeated appearances of the pattern assembly at the speed with which we understand or produce speech, the analysis appears to be more reliable than what is described

in Wolff (2006, Section 11.4.2), especially Figure 11.10 in that section.

#### 8. SP-NEURAL: AN OUTPUT PERSPECTIVE

An inspection of **Figure 3**—showing how, in SP-neural, a small portion of natural language may be analyzed by pattern assemblies and their interconnections—may suggest that if we wish to reverse the process—to create language instead of analysing it—then the innervation would need to be reversed: we may guess that two-way neural connections would be needed to support the production of speech or writing as well as their interpretation.

But a neat feature of SP-abstract is that one set of Old patterns, together with the processes for building multiple alignments, will support both the analysis and the production of language. So it is reasonable to suppose that if SP-neural works at all, a similar duality will apply to pattern assemblies and their interconnections, without the need for two-way connections amongst pattern assemblies and neural symbols (but see Section 8.3).

Of course, speaking or writing would need peripheral motor processes that are different from the peripheral sensory processes required for listening or reading, but, more centrally, the processes for analysing language or producing it may use the same mechanisms17 .

The reason that SP-abstract, as expressed in the SP computer model, can work in "reverse" so to speak, is that, from a multiple alignment like the one shown in **Figure 2**, a code pattern like "S 0 2 4 3 7 6 1 8 5 #S" may be derived, as outlined in Section 3.6.  An Answer to the Apparent Paradox of "Decompression by Compression"

That the SP system should be able to reconstruct a sentence that was originally compressed by means of the same system (Section 8) may seem paradoxical. How is it that a system that is dedicated to information compression should be able, so to speak, to drive compression in reverse?

A resolution of this apparent paradox is described in Wolff (2006, Section 3.8). In brief, the key to the conjuring trick is to ensure that, after the sentence has been compressed, there is enough residual redundancy in the code pattern to allow further compression, and to ensure that this further compression will achieve the effect of reconstructing the sentence.

#### 8.2. Meanings in the Analysis and Production of Language

Of course, parsing a sentence (as shown in Section 3.5) or constructing a sentence from a code pattern (as shown in Section 8) are very artificial applications with natural language. Normally, when we read some text or listen to someone speaking, we aim to derive meaning from the writing or the speech. And when we write or speak, it seems, intuitively, that the patterns of words that we are creating are derived from some kind of underlying meaning that we are trying to express.

It is envisaged that, in future development of SP-abstract and the SP computer model, the ID-symbols in code patterns will provide some kind of bridge between syntactic forms and representations of meanings, thus facilitating the processes of understanding the meanings of written or spoken sentences and of creating sentences to express particular meanings.

As noted at the end of Section 3.8.2, there are preliminary examples of how, with the SP computer model, a sentence may be analyzed for its meaning (Wolff, 2006, Section 5.7, Figure 5.18), and how the same sentence may be derived from a representation of its meaning (Wolff, 2006, Figure 5.19).

#### 8.3. But There Are Projections from the Sensory Cortex to Subcortical Nuclei

Although as we have seen earlier in Section 8, SP-neural, via principles established in SP-abstract, provides for the creation of language, and other kinds of knowledge, without the need for efferent connections from the cortex back along the path of afferent nerves, there is evidence that such connections do exist:

"Neurons of the cerebral cortex send axons to subcortical regions .... Subcortical projections are to those nuclei in the thalamus and brainstem that provide ascending sensory information. By far the most prominent of these is to the thalamus: the neurons of a primary sensory cortex project back to the same thalamic nucleus that provides input to the cortex. This system of descending connections is truly impressive because the number of descending corticothalamic axons greatly exceeds the number of ascending thalamocortical axons. These connections permit a particular sensory cortex to control the activity of the very neurons that relay information to it." (Squire et al., 2013, p. 509).

17Of course, things are a little more complicated with output processes because sensory feedback is normally an important part of speaking or writing.

But the descending nerves described in this quotation may have a function that is quite different from the creation of sentences or other patterns of activity. One possible role for such nerves may be "the focussing of activity so that relay neurons most activated by a sensory stimulus are more strongly driven and those in surrounding less well activated regions are further suppressed." (Squire et al., 2013, p. 509).

#### 9. THE POSSIBLE ROLES FOR INHIBITION IN SP-NEURAL

A familiar observation is that, if something like a fan is switched on near us, we notice the noise for a while and then come to ignore it. And if, later, the fan is switched off, we notice the relative quiet for a while and then cease to be aware of it. In general, it seems that we are relatively sensitive to changes in our environment and relatively insensitive to things that remain constant.

It has been accepted for some time that the way we adapt to constant stimuli is due to inhibitory neural structures and processes in our brains and nervous systems, that inhibitory structures and processes are widespread in the animal kingdom, and that they have a role in reducing the amount of information that we need to process (von Békésy, 1967).

Regarding the last point, it is clearly inefficient for anyone to be constantly registering, second-by-second, the noise of a nearby fan: "noise, noise, noise, noise, noise, ..." and likewise for the state of relative quietness when the fan is switched off. In terms of information theory, there is redundancy in the second-by-second recurrence of the noise (or quietness), and we can eliminate most or all of the redundancy and thus compress the information—by simply recording that the noise is "on" and that it is continuing (and likewise, mutatis mutandis, for quiet). This is the "run-length encoding" technique for compression of information,18 it is essentially what adaptation does, and, in neural tissue, it appears to be mediated largely by "lateral" inhibition.

With lateral inhibition in sensory neurons, there are inhibitory connections between neighboring neurons so that, when they are both stimulated, they tend to inhibit each other, and thus reduce their rates of firing where there is strong uniform stimulation. But inhibition is reduced where strong stimulation gives way to weaker stimulation, leading to a local swing in the rate of firing (Ratliff et al., 1963; see also Wolff, 2006, Section 2.3.1; there is more about lateral inhibition in Squire et al., 2013, p. 505). There are similar effects in the time dimension. Again, Barlow (1982) says, in connection with neurons in the mammalian cortex that receive inputs from both eyes, "... it is now clear that input from one eye can, and frequently does, inhibit the effects of input from the other eye, ..." (p. 147).

Taking these observations together, we may abstract a general rule: When, in neural processing, two or more signals are the same, they tend to inhibit each other, and when they are different, they don't. The overall effect should be to detect redundancy in information and to reduce it, whilst retaining non-redundant information, in accordance with the central principle in the SP theory—that much of computing and cognition may, to a large extent, be understood as information compression.

In a similar vein: "Lateral inhibition represents the classic example of a general principle: most neurons in sensory systems are best adapted for detecting changes in the external environment. ... As a rule, it is change which has the most significance for an animal ... This principle can also be explained in terms of information processing. Given a world that is filled with constants—with uniform objects, with objects that move only rarely—it is most efficient to respond only to changes." (Squire et al., 2013, p. 578).

In view of the widespread occurrence of inhibitory mechanisms in the brain19, and in view of their apparent

18See "Run-length encoding," Wikipedia, bit.ly/21JlB1T, retrieved 2016-03-04.

19"These [aspiny or sparsely spiny nonpyramida] interneurons constitute approximately 15–30% of the total population of cortical neurons, and they appear to be mostly GABAergic, representing the main components of inhibitory cortical circuits ...." (Squire et al., 2013, p. 45); "Synaptic inhibition in the mammalian brain

importance for the compression of information, and thus for selective advantage (Wolff, 2014d, Section 4), it is pertinent to ask what role or roles they may play in SP-neural. Here are some possibilities:

	- As we have seen, lateral inhibition can have the effect of inhibiting signals from neighboring sensory neurons when they are receiving stimulation which is the same of nearly so. This may be seen as an example of ICMUP.
	- In accordance with the abstract general rule suggested above, inhibitory processes may serve as a means of detecting redundancy between a New pattern assembly like "t a b l e" and an Old pattern assembly like "N 9 t a b l e #N":
		- We may suppose that there are inhibitory links between neighboring neural symbols in the Old pattern assembly so that, if all of the neural symbols in the body of that Old pattern assembly (i.e., "t a b l e") are stimulated, or most of them, then mutual inhibition amongst those neural symbols will suppress their response. And, as with lateral inhibition in sensory neural tissue, inhibition in one area can mean enhanced responses at the boundaries with neighboring areas, which, in this case, would be the ID-symbols "N" and "9" on the left, and "#N" on the right. Then, excitatory signals from "N" and "#N" may go on to higher-level patterns that contain nouns, as suggested by the broken-line links from those two neural symbols in **Figure 3**. Since there is no link to export excitatory signals from "9," no such signals would be sent.
		- Alternatively, we may suppose that a stored pattern assembly like "N 9 t a b l e #N" has a background rate of firing and that, when matching stimulation is received for the neural symbols "t a b l e," the background rate of firing in the corresponding neural symbols in "N 9 t a b l e #N" is reduced, with an associated upswing in the rates of firing of the neural symbols "N" and "9" and "#N," as before.

alignments (NAMAs) means identifying one or two of the most excited pattern assemblies, with structures below them that feed excitation to them. Here, inhibition may play a part by enhancing the status of the most excited pattern assemblies and suppressing the rest. How inhibition may achieve that kind of effect is discussed quite fully by von Békésy (1967, Chapters II and V), and also in Shamma (1985).

More information and discussion about the possible roles of inhibition in the cerebral cortex may be found in Isaacson and Scanziani (2011).

#### 10. UNSUPERVISED LEARNING IN SP-NEURAL

This section considers how the learning processes in SP-abstract, which are outlined in Sections 3.4, 3.7, may be realized in SPneural.

It seems likely that neural structures for the detection of "low level" features like lines and corners in vision, or formant ratios and transitions in hearing, are largely inborn20 , although "It is a curious paradox that, while [Hubel and Wiesel] have consistently argued for a high degree of ontogenetic determination of structure and function in the visual system, they are also the authors of the best example of plasticity in response to changed visual experience." (Barlow, 1982, p. 150), and "It has ... been shown convincingly that the orientation preference of cells can be modified, ..." (Barlow, 1982). Also, "In the somatosensory system, if input from a restricted area of the body surface is removed by severing a nerve or by amputation of a digit, the portion of the cortex that was previously responsive to that region of the body surface becomes responsive to neighboring regions ...." (Squire et al., 2013, p. 508).

But it is clear that most of what we learn in life is at a "higher" level which, in SP-neural, will be acquired via the the creation and destruction of pattern assemblies, as discussed in the following subsections.

#### 10.1. Creating Old Pattern Assemblies

Let us suppose that a young child hears the speech equivalent of "t h e b i g h o u s e" in accordance with the example in Section 3.4. As we have seen, when the repository of Old patterns is empty or nearly so, New patterns are stored directly as Old patterns, somewhat like a recording machine, but with the addition of ID-symbols at their beginnings and ends.

It seems unlikely that a young child would grow new neurons to store a newly-created Old pattern assembly like "A 1 t h e b i g h o u s e #A," as discussed in Section 3.4. It seems much more likely that a pattern assembly like that would be created by some kind of modification of pre-existing neural tissue comprising sequences or areas of unassigned neural symbols with lateral connections between them as suggested in Section 4.2. Pattern assemblies would be created by the switching on and off of synapses at appropriate points, in

is mediated principally by GABA receptors." (Squire et al., 2013, p. 169); "One of the great mysteries of synaptic integration is why there are so many different types of inhibitory interneurons. ... more than 20 different types of inhibitory interneuron have been described in the CA1 region of the hippocampus alone." (Squire et al., 2013, p. 249).

20"For all systems except the olfactory, the receptor neurons you were born with are the ones you will live with." (Squire et al., 2013, p. 503).

a manner that is more like a tailor cutting up pre-woven cloth than someone knitting or crocheting each item from scratch.

In accordance with the labeled line principle (Section 5.3), the meaning of each symbol in a newly-created pattern assembly would be determined by what it is connected to, as described in Section 10.2.

Similar principles would apply when Old patterns are created from partial matches between patterns, as described in Section 3.4.

#### 10.2. Creating Connections between Pattern Assemblies

As with the laying down of newly-created Old patterns (Section 10.1), it seems unlikely that connections between pattern assemblies, like those shown in **Figure 3**, would be created by growing new axons or dendrites. It seems much more likely that such connections would be established by switching on synapses between each of the two neurons to be connected and pre-existing axons or dendrites, somewhat like the making of connections in a telephone exchange (see Section 11).

This idea, together with the suggestions in Section 10.1 about how Old pattern assemblies may be created, is somewhat like the way in which an "uncommitted logic array" (ULA)21 may, via small modifications, be made to function like any one of a wide variety of "application-specific integrated circuits" (ASICs)22 , or how a "field-programmable gate array" (FPGA)23 may be programmed to function like any one of a wide variety of integrated circuits.

#### 10.3. Destruction of Pattern Assemblies and Their Interconnections

In the SP theory, patterns and pattern assemblies are never modified—they are either created or destroyed. The latter process occurs mainly in the process of searching for "good" grammars to describe a given set of New patterns, as outlined in Section 3.7. At each stage, when a few "good" grammars are retained in the system, the rest are discarded. This means that any pattern assembly in one or more of the "bad" grammars that is not also in one or more of the "good" grammars may be destroyed.

It seems likely that, in a process that may be seen as a reversal of the way in which pattern assemblies and their interconnections are created, the destruction of a pattern assembly does not mean the physical destruction of its neurons. It seems more likely that all neural connections from the pattern assembly are broken by switching off relevant synapses (Sections 10.3, 11) and that its constituent neurons are retained for later use in other pattern assemblies.

#### 10.4. Searching for Good Grammars

It must be admitted that, apart from the remarks in forgoing subsections about the creation and destruction of pattern assemblies and their inter-connections, it is unclear how, in SPneural, one may achieve anything equivalent to the process of searching the abstract space of possible grammars that has been implemented in the SP computer model.

One possibility is to simplify things as follows. Instead of evaluating whole grammars, as in the SP computer model, it may be possible to achieve roughly the same effect by evaluating pattern assemblies in terms of their effectiveness or otherwise for the economical encoding of New information and, periodically, to discard those pattern assemblies that do badly.

#### 10.5. What about Hebbian Learning?

Readers familiar with issues in AI or neuroscience may wonder what place, if any, there may be in SP-neural for the concept of "Hebbian" learning. This idea, proposed by Hebb (1949), is that:

"When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased." (p. 62).

Variants of this idea are widely used in versions of "deep learning" in artificial neural networks (Schmidhuber, 2015) and have contributed to success with such systems24 .

But in Wolff (2016, Section V-D) I have argued that:

21See "Gate array," Wikipedia, bit.ly/1UdB46j, retrieved 2016-03-20.

22See "Application-specific integrated circuit," Wikipedia, bit.ly/1pUs2y8, retrieved 2016-03-20.

23See "Field-programmable gate array," Wikipedia, bit.ly/1Hgi9iH, retrieved 2016- 03-20.

24See, for example, "Don't despair if Google's AI beats the world's best Go player," MIT Technology Review, bit.ly/1p7Wzb7, 2016-03-08; and "Google unveils neural network with "superhuman" ability to determine the location of almost any image", MIT Technology Review, bit.ly/1p5qmSe, 2016-02-24.

25It may be argued that Hebbian learning may apply in such cases because a single experience may be mentally rehearsed. But that begs the question of how the one experience is remembered between when it first occurred and the first rehearsal and likewise later on. And, while rehearsal may be helpful in some cases, it seems that there are many things we do remember after a single experience, without rehearsal.

high standard—because of the complexity of the abstract space of possible solutions that needs to be searched.

Does this mean that Hebbian learning is dead? Probably not:

#### 11. THE PROBLEMS OF SPEED AND EXPRESSIVENESS IN THE CREATION AND DESTRUCTION OF NEURAL STRUCTURES

A general issue for any neural theory of the representation and processing of knowledge, is how to account for the speed with which we can create neural structures, and, probably, destroy them, bearing in mind that such structures must be sufficiently versatile to accommodate the representation and processing of a wide range of different kinds of knowledge. This issue arises mainly in the following connections:

One possible solution, which is radically different from SP-neural, is to suppose that our knowledge is stored in some chemical form such as DNA, and that the kinds of mental processes mentioned above might be mediated via the creation and modification of such chemicals. Another possibility is that learning is mediated by epigenetic mechanisms, as outlined in Baars and Gage (2010, Section 7.4). Without wishing to prejudge what the primary mechanism of learning may be, or whether perhaps there are several such mechanisms, this paper focusses on SP-neural and how it may combine speed with expressiveness, as seems to be required for the kinds of functions outlined above.

At first sight, the problem of speed in the creation of neural structures is solved via the long-established idea that we can remember things for a few seconds via a "short-term memory26 " that is distinct from "long-term memory27," and "working memory28." But there is some uncertainty about the extent to which these three kinds of memory may be distinguished, one from another, and there is considerable uncertainty about how they might work, and how information may be transferred from one kind of memory to another.

As a proffered contribution to discussions in this area, the suggestion here is that, in any or all of short-term memory, working memory, and long-term memory, SP-neural may achieve the necessary speed in the creation of new structures, combined with versatility in the representation and processing of diverse kinds of knowledge, by the switching on and off of synapses in pre-established neural structures and their interconnections, as outlined in Sections 10.1, 10.2.

With regard to possible mechanisms for the switching on and off of synapses:

26"Short-term memory," Wikipedia, bit.ly/1RzAVHN, retrieved 2016-04-04.

27"Long-term memory," Wikipedia, bit.ly/1M9uPhh, retrieved 2016-04-04.

28"Working memory," Wikipedia, bit.ly/1PQq0UA, retrieved 2016-04-04.

With all these possible mechanisms, key questions are: do they act fast enough to account for the speed of the phenomena described above; and can they provide the basis for memories that can last for 50 years or more.

#### 12. ERRORS OF OMISSION, COMMISSION, AND SUBSTITUTION

A prominent feature of human perception is that we have a robust ability to recognize things despite disturbances of various kinds. We can, for example, recognize a car when it is partially obscured by the leaves and branches of a tree, or by falling snow or rain.

One of the strengths of SP-abstract and its realization in the SP computer model is that, in a similar way, recognition of a New pattern or patterns is not unduly disturbed by errors of omission, commission, and substitution in those data (Wolff, 2006, Chapter 6, Wolff, 2013, Section 4.2.2). This is because of the way the SP computer model searches for a global optimum in the building of multiple alignments, so that it does not depend on the presence or absence of any particular feature or combination of features in the New information that is being analyzed.

In its overall structure, SP-neural seems to lend itself to that kind of robustness in recognition in the face of errors in data. But the devil is in the detail. In further development of the theory, and in the development of a computer model of SP-neural, it will be necessary to clarify the details of how that kind of robustness may be achieved. In shaping this aspect of SP-neural, the principles that have been developed in SP-abstract are likely to prove useful and, with empirical evidence from brains and nervous systems, they may serve as a touchstone of success.

### 13. CONCLUSION

As was mentioned in the Introduction, SP-neural is a tentative and partial theory. That said, the close relationship between SP-neural and SP-abstract, the incorporation into SP-abstract of many insights from research on human perception and cognition, strengths of SP-abstract in terms of simplicity and power (Section 3.8.1), and advantages of SP-abstract compared with other AI-related systems (Section 3.8.3)—lend support to SP-neural as it is now as a conceptual model of the representation and processing of knowledge in the brain, and a promising basis for further research.

Naturally, we may have more confidence in some parts of the theory than others. Arguably, the parts that inspire most confidence are these:

and cognition may be understood in terms of information compression via the matching and unification of patterns.

As with SP-abstract, areas of uncertainty in SP-neural may be clarified by casting the theory in the form of a computer model and testing it to see whether or not it works as anticipated. It is envisaged that this would be part of a proposed facility for the development of the SP machine (Wolff and Palade, 2016), a means for researchers everywhere to explore what can be done with the SP machine and to create new versions of it.

At all stages in its development, the theory may suggest possible investigations of the workings of brains and nervous systems. And any neurophysiological evidence may have a bearing on the perceived validity of the theory and whether or how it may need to be modified.

#### AUTHOR CONTRIBUTIONS

This is part of a long term research programme by JGW, developing the SP theory of intelligence. SP-neural, which is the subject of this paper, was first outlined in Chapter 11 of "Unifying Computing and Cognition" and, in this paper, has been considerably refined and developed.

#### FUNDING

The research is funded by CognitionResearch.org.

#### ACKNOWLEDGMENTS

I'm grateful to referees for constructive comments on earlier drafts of this paper.

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Wolff. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

#### APPENDIX

#### Cell Assemblies and Pattern Assemblies

The main differences between Hebb's (1949) concept of a "cell assembly" and the SP-neural concept of a "pattern assembly" are:

With regard to the last point, possible models for sharing of structures are illustrated in **Figure A1**.

In literal sharing, structures B and C in the figure both contain structure A. In sharing by copying, structures B and C each contains a copy of structure A. While in sharing by reference, structures B and C each contains a reference to structure A, in much the same way that a paper like this one contains references to other publications.

From Hebb's (1949) descriptions of the cell assembly concept, it is difficult to tell which of these three possibilities are intended.

By contrast with the concept of a pattern assembly in SPneural, sharing is almost always achieved by means of neural "references" between structures. For example, a noun like "table" is likely to have neural connections to the many grammatical contexts in which it may occur, as suggested by the two brokenline connections from each of "N" and "#N" in the pattern assembly for "table" shown in **Figure 3**. Notice that, in this example, the putative direction of travel of nerve impulses is not relevant—it is the neural connection that counts.

In the SP system, it is intended that literal sharing should be impossible and that sharing by copying may only occur on the relatively rare occasions when the system has failed to detect the corresponding redundancy, and not always then.

## Linking Neural and Symbolic Representation and Processing of Conceptual Structures

Frank van der Velde1, 2 \*, Jamie Forth3 , Deniece S. Nazareth1 and Geraint A. Wiggins 4

*1 Department of Cognitive Psychology and Ergonomics, Faculty of Behavioural, Management and Social sciences, University of Twente, Enschede, Netherlands, 2 Institute of Psychology (IOP), Leiden, Netherlands, 3 Department of Computing, Goldsmiths University of London, London, United Kingdom, 4 Computational Creativity Lab, Cognitive Science Group, School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom*

We compare and discuss representations in two cognitive architectures aimed at representing and processing complex conceptual (sentence-like) structures. First is the Neural Blackboard Architecture (NBA), which aims to account for representation and processing of complex and combinatorial conceptual structures in the brain. Second is IDyOT (Information Dynamics of Thinking), which derives sentence-like structures by learning statistical sequential regularities over a suitable corpus. Although IDyOT is designed at a level more abstract than the neural, so it is a model of cognitive function, rather than neural processing, there are strong similarities between the composite structures developed in IDyOT and the NBA. We hypothesize that these similarities form the basis of a combined architecture in which the individual strengths of each architecture are integrated. We outline and discuss the characteristics of this combined architecture, emphasizing the representation and processing of conceptual structures.

#### Edited by:

*Leonid Perlovsky, Harvard University and Air Force Research Laboratory, United States*

#### Reviewed by:

*Antonio Chella, University of Palermo, Italy Dorina Rajanen, University of Oulu, Finland*

#### \*Correspondence: *Frank van der Velde*

<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Cognition, a section of the journal Frontiers in Psychology*

Received: *31 July 2016* Accepted: *17 July 2017* Published: *10 August 2017*

#### Citation:

*van der Velde F, Forth J, Nazareth DS and Wiggins GA (2017) Linking Neural and Symbolic Representation and Processing of Conceptual Structures. Front. Psychol. 8:1297. doi: 10.3389/fpsyg.2017.01297* Keywords: cognitive architecture, memory representation, hebbian learning, compositional learning, incremental learning, In situ representations

### 1. INTRODUCTION

The ability to represent and process conceptual structures, as found in language processing, reasoning, and in generating conceptual representations from visual and auditory perception, are key elements of human cognition. They can be studied with the aim to understand human cognition and its relation to the brain. But they can also be targets for the development of artificial cognitive systems. These aims can be combined to various degrees, because a cognitive architecture that provides an understanding of a (neural) cognitive process can also be used in artificial systems, and, conversely, the way in which an artificial system processes complex information can reveal aspects of human processing as well.

Here, we discuss and relate the different representations used in two cognitive architectures, one neural and one symbolic, in which complex conceptual structures can be represented and processed. That is, we discuss and illustrate the different ways in which complex conceptual structures are represented or learned in the two architectures and how these representations could be related.

In particular, we aim to outline how combined representations could be developed, for use in a combined architecture in which aspects of our neural and symbolic architectures are integrated. We hypothesize that such a combined architecture could serve as a model of human conceptual processing and its relation to the brain. When implemented, it could also serve as a new artificial

**122**

architecture in which forms of neural (parallel) hardware and neural and symbolic forms of learning and processing could be integrated. We are as yet at the beginning of the integration of our architectures, which is also a reason why we focus on issues of representation here.

The neural representation in our integration is that used in the Neural Blackboard Architecture (NBA), which is aimed to represent and process conceptual structures in language (e.g., van der Velde and de Kamps, 2006, 2010), reasoning and other cognitive domains (van der Velde, 2016a). The NBA assumes that conceptual representations in the brain consist of dedicated network structures, or neural assemblies, that develop over time and that can be distributed over wide areas in the brain and cortex. A fundamental characteristic of these network-like conceptual representations is that they are always content addressable, whether they are activated in isolation or whether they are parts of more complex (and even hierarchical) conceptual structures, such as sentences in language.

The NBA provides "neural blackboards" that afford the representation and processing of complex conceptual structures based on neural assembly conceptual representations in specific cognitive domains. Examples are neural blackboards for sentence structures, phonological structures, sequences, and relations as used in reasoning. In each domain, a dedicated neural blackboard will provide a range of specialized structural elements that can bind in a neural manner to the neural assemblies (e.g., representing "words" in language). The neural bindings, implemented with neural circuits, allow the creation and processing of more complex cognitive structures (e.g., "sentences") in a combinatorial manner.

The symbolic representation in our integration is that used in Information Dynamics of Thinking (IDyOT). IDyOT derives (e.g., sentence-like) structures by learning statistical sequential regularities over a linguistic corpus (Wiggins, 2012b; Wiggins and Forth, 2015; Forth et al., 2016). IDyOT is unusual as a machine learning formalism in that it is symbolic in nature, but it generates and gives explicit semantics to its own symbols, in a bottom-up learning process, which is optimized by a general, data-independent principle of information efficiency, conceptualized as predictive accuracy. These symbols correspond with concepts in the semantics of the system. Another unusual aspect of IDyOT's operation is that both representations and sequential models are optimized simultaneously with respect to the prediction accuracy of the models, causing a trade-off between overfitting and accuracy that we propose as a model of the corresponding trade-off in human cognition. The explanation of this process is a novel contribution of the current paper.

The representational links between IDyOT and NBA concern the nature of the dedicated structural elements that allow processing and representation of complex conceptual structures, the way these elements could be activated during processing, and the underlying semantics of the architectures in the form of conceptual spaces that possess a geometrical structure (Gärdenfors, 2000, 2014).

In the NBA, the dedicated structural elements form the neural blackboards. The kinds of elements used and the way they are activated derive from analyses of the cognitive domains at hand, as in the sentence NBA (e.g., van der Velde and de Kamps, 2006, 2010). However, the combination of NBA with IDyOT provides the possibility to derive these structural elements by learning from real corpora. Conversely, the NBA could provide a neural implementation of the more higher-level formal account as provided by IDyOT. Thus, IDyOT potentially supplies a higherlevel formal account and learning abilities to the operations of the NBA. Conversely, the NBA provides a route toward a neural implementation of IDyOT, which could also form the basis of in parallel operating hardware.

#### 2. THEORETICAL POSITION AND NOVELTY

Our theoretical position here is that the representations used in NBA and IDyOT are in fact two different representations of the same thing, at different levels of abstraction, but with focus on similar representational affordances. In the following sections, we describe the representations, and the relations between them but, as always, to understand the representations it is necessary also to understand the processes that work over them.

The novelty in the current paper lies in several places, primarily in the thorough-going comparison between the representations and corresponding processes in the two architectures. The entire description of IDyOT memory construction is also novel, and we present a novel simulation of neural activity based on the NBA, which allows for a detailed comparison with brain activity observed in human (sentence) processing. To the best of our knowledge, such a detailed potential comparison between human brain activity and simulated model activity is not available in the case of high-level cognitive processing, such as sentence comprehension. This also strongly motivates the integration of our architectures, because that would endow the NBA with the learning capabilities of IDyOT, based on real corpora (as outlined below). In turn, the dynamics and structure of the NBA would then allow a comparison between the representations and underlying processing as learned by IDyOT and human brain activity.

The structure of the paper is as follows. In the next two sections we briefly describe the representations used in NBA and IDyOT in turn, also giving detail of processing where appropriate. In the sections that follow, we discuss a number of specific links between NBA and IDyOT and the potential benefits of their integration.

#### 3. NEURAL BLACKBOARD ARCHITECTURE

In our outline of Neural Blackboard Architecture, or NBA for short, we focus on the representation of concepts (e.g., underlying words) in the architecture and the representational structures that are used to integrate concepts in more complex cognitive structures, such as relations and sentences.

The basis of concept representation in the NBA are "neural assemblies," as proposed by Donald Hebb (1949). In the view of Hebb, these neural assemblies develop over time by interconnecting the neurons in the brain that are involved in processing (sensory) information and generating actions related to the concept they represent. However, unlike Hebbian assemblies, conceptual representations in the NBA are not only associative. Instead, they can (and mostly will) contain relational structures as well.

**Figure 1** illustrates a neural assembly representation of cat. It would be distributed over different areas in the cortex and brain, depending on the kind of information involved, including networks processing perceptual information about cats and networks that can produce specific actions (e.g., pronouncing the word "cat"). But also networks representing emotional content or associations related to cats belong to the assembly, and networks that instantiate relations, such as cat is pet.

The combination of perception and action in the assembly structure of a concept entails that both the patterns (and activation) of "incoming" and "outgoing" connections determine the meaning of a concept. For example, neurons observed in the medial temporal cortex responded to a person whether the person or his or her name was presented (Quian Quiroga, 2012). These "perception" networks do in fact belong to the assembly structure of a concept, because without them the concept could not be activated (or was not learned). But this would capture only part of the role and (hence) meaning of the conceptual representation involved. Equally important would be the effect of these neurons on downstream processing (van der Velde, 2015).

The notion that conceptual representations interconnect sensory information processing and action generation underscores their role in producing behavior. The ability to produce behavior is a crucial aspect of cognition (and hence of every neuro-cognitive model) because the evolution of cognition depended on the ability to produce behavior. Without advocating a behavioristic view of cognition (e.g., as the basis of modeling cognition) we do argue that the prime role of cognition is to intervene in the reflex (cf. Shanahan, 2010). In this view, the need for a connection structure that transfers sensory activity to motor activity should always be at the background of a neural-cognitive model.

So, in **Figure 1**, it is not just the gray oval that represents the concept cat, but instead the entire network structure to which it is connected. The gray oval could play the role of a higher-level representation of the concept in the sense that it interconnects the concept to other networks. But it would be wrong to see this as the "genuine" encoding of the concept. Without the networks to which it is connected, the gray oval does not encode anything.

An important feature of conceptual representations given by neural assemblies is that they are "in situ." This entails that they cannot be copied and transported to create more complex structural representations with them (e.g., as found in language or reasoning). Instead, the same assembly (or a part thereof) is always activated when the concept it represents is tokened. One consequence of this kind of representation is that an assembly can develop and grow over time, as originally discussed by Hebb.

Another direct consequence of the in situ nature of a neural assembly is that the concept it represents is content addressable. This entails that the same assembly (or part thereof) will be activated when sufficient information about the concept it

S, sentence; V, verb.

represents is available (e.g., perceived), even when the concept is part of a (complex) sentence structure.