For both real-world and Fribble objects searches, visual inspection of the ordering of stimuli by ROI response, that is, as shown in **Figures 5**, **9**, fails to yield any specific insights. *A priori*, we would expect shape properties to smoothly transition as measured responses decreased. That we did not observe this transition may stem from the fact that our measurements reflect a mix of multiple coding units or noise in our fMRI data (despite averaging). For real-world objects, note that the construction of our four-dimensional search space using MDS

5Scatter plot examples only are given here in Fribble spaces as they are more easily evaluated visually in one two-dimensional plot.

6Stimulus examples only are given here for SIFT searches as similarities of the real-world object stimulus set are easier to see than they are for Fribbles that all look predominantly similar within a given class to the uninitiated reader.

may also limit our ability to detect the fine-grained organization of the stimuli, yet maintain the broader similarity groupings of these same images. At the same time—in light of our finding of evidence for local inhibition—we might alternatively expect that that similar-looking stimuli would appear at opposite ends of the line of sorted stimuli. Interestingly, such a visual disconnect between top-ranked stimuli for single ventral pathway neurons was observed by Woloszyn and Sheinberg (2012).

More broadly, frequently visited stimuli clustered together in SIFT space—evoking both extreme high and low responses, consistent with the observations above—can be united by coarse shape (e.g., width in **Figures 5D**, **7C**), surface properties (e.g., brightness in **Figure 5D** or texture for S1 class 2 in **Table 1**), and fine internal contours (e.g., sharp-edged holes for S5 class 2 in **Table 1**). This observed selectivity for shapes is consistent with the findings of Yamane et al. (2008) and Hung et al. (2012), who successfully identified two- and three-dimensional contour preferences for neurons in V4 and IT using uniform-gray blob stimuli. Unlike these prior studies, our work employs real-world stimuli and thus identifies classes of preferred shapes likely to be encountered in normal life experience. Observed selectivity for surface properties is a more novel finding, though Tanaka et al. (1991) observed such selectivities in primate IT neurons in the context of perception of object drawings. While some searches provided insights about cortically relevant visual properties, many searches performed for real-world objects revealed no clear patterns among stimuli evoking extreme cortical region responses, clustered together in SIFT-based space. This lack of clear patterns likely reflects the difficulty of capturing the diversity of real-world visual properties in a four dimensional space, as discussed in Section 3.5.

We also note that changes in the cortical representation of the stimuli due to repeated exposures across the three study sessions may have made interpretation of our results more difficult across our entire study. However, arguing against this possibility, our observation of stronger search performance for subjects viewing Fribble stimuli—novel images with significant similarity in appearance within each class (thereby increasing the likelihood of overlap in the neural representations of individual stimuli)—suggests that potential adaptation or learning effects did not constitute a significant problem.

#### **4.3. SELECTIVITY TO VISUAL PARTS**

Fribble objects, and corresponding "Fribble spaces," were more controlled in their span of visual properties than were the realworld stimuli. Frequently visited stimuli in each Fribble space can cluster around a three-dimensional coordinate. Each dimension corresponds to variations of a single component shape morphed between two options, such as a star/circle head or flat/curved feet, as in **Figure 4A**. Thus, clustering around a point indicates slight variations on three component shapes, with focus around a fixed holistic shape. However, across subjects, there was no clear pattern of preferred holistic Fribble shapes, nor of preferred shapes for any of the three varying component "appendages." For some searches, frequently visited Fribble stimuli evoking strong cortical responses varied along one axis or two axes while staying constant on the remaining one(s). Depending on the brain region being interrogated, one to three component shapes were able to account for such selectivity.

In interpreting this result, we note that it is possible that the appendage-based construction and variations of Fribble stimuli may have biased subjects to rely on perceptual strategies focused on object parts. Nonetheless, observations on cortical responses in these subjects may be supported by evidence for parts-based neural representations observed in subjects viewing less-structured real-world objects—for example, ROIs selective for rectangular statue bases or for bag handles in **Table 1**. One possibility is that more local selectivity for parts of an object, rather than the whole, may be associated with cortical areas particularly earlier in the ventral pathway—an organization that would be consistent with the focus of early and intermediate stages of vision on spatially-distinct parts of a viewed image, pooled together over increasingly broader parts of the image at higher stages of vision (Riesenhuber and Poggio, 1999).

# **5. CONCLUSIONS**

Our study is one of the first to address head on the challenge of identifying intermediate-level feature representation in human ventral cortex. That is, although there is a great deal known about early visual coding and increasing knowledge regarding high-level visual representation (Huth et al., 2012), the field has been relatively silent (with the exception of Tanaka, 1996, Yamane et al., 2008, Hung et al., 2012) on how simple edge-like features are combined to encode more complex features such as parts, textures, and complex shapes. Here we explored this question in two ways. First, by advancing the application of a novel research methodology—real-time methods for rapidly measuring and processing the BOLD signal on a trial-by-trial basis. Second, by introducing a new research approach as applied to human neuroimaging—search methods for efficiently seeking the image or images that are most effective in driving specific brain regions. Although our overall findings are somewhat mixed regarding what we have learned about intermediate-level neural coding, we observed sufficient consistency—in particular, with respect to apparent highlevel local inhibition—to suggest that as these methods mature, they offer a promising new direction for exploring the finegrained neural representation of visual stimuli within the human brain.

# **ACKNOWLEDGMENTS**

This research was funded by an NIH EUREKA Award #1R01MH084195-01, the Temporal Dynamics of Learning Center at UCSD (NSF Science of Learning Center SMA-1041755), and, in part by a grant from the Pennsylvania Department of Health's Commonwealth Universal Research Enhancement Program. Daniel D. Leeds was supported by an NSF IGERT Fellowship through the Center for the Neural Basis of Cognition, an R. K. Mellon Fellowship through Carnegie Mellon University, and the Program in Neural Computation Train Program (NIH Grant T90 DA022762).

# **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fncom.2014. 00106/abstract

# **REFERENCES**

Wennerberg, M. (2009). *Version 2.9.5*. Tynningo: Norrkross Software.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 01 June 2014; accepted: 19 August 2014; published online: 12 September 2014.*

*Citation: Leeds DD, Pyles JA and Tarr MJ (2014) Exploration of complex visual feature spaces for object perception. Front. Comput. Neurosci. 8:106. doi: 10.3389/fncom. 2014.00106*

*This article was submitted to the journal Frontiers in Computational Neuroscience. Copyright © 2014 Leeds, Pyles and Tarr. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# A neuromorphic system for video object recognition

# *Deepak Khosla\*, Yang Chen and Kyungnam Kim*

*HRL Laboratories, LLC, Malibu, CA, USA*

#### *Edited by:*

*Antonio J. Rodriguez-Sanchez, University of Innsbruck, Austria*

#### *Reviewed by:*

*Daniel L. Yamins, Massachussetts Institute of Technology, USA Rangachar Kasturi, University of South Florida, USA*

#### *\*Correspondence:*

*Deepak Khosla, HRL Laboratories, LLC, 3011 Malibu Canyon Road, Malibu, CA 90265-4797, USA e-mail<EMAIL_ADDRESS>
Automated video object recognition is a topic of emerging importance in both defense and civilian applications. This work describes an accurate and low-power neuromorphic architecture and system for real-time automated video object recognition. Our system, Neuormorphic Visual Understanding of Scenes (NEOVUS), is inspired by computational neuroscience models of feed-forward object detection and classification pipelines for processing visual data. The NEOVUS architecture is inspired by the ventral (*what*) and dorsal (*where*) streams of the mammalian visual pathway and integrates retinal processing, object detection based on form and motion modeling, and object classification based on convolutional neural networks. The object recognition performance and energy use of the NEOVUS was evaluated by the Defense Advanced Research Projects Agency (DARPA) under the Neovision2 program using three urban area video datasets collected from a mix of stationary and moving platforms. These datasets are challenging and include a large number of objects of different types in cluttered scenes, with varying illumination and occlusion conditions. In a systematic evaluation of five different teams by DARPA on these datasets, the NEOVUS demonstrated the best performance with high object recognition accuracy and the lowest energy consumption. Its energy use was three orders of magnitude lower than two independent state of the art baseline computer vision systems. The dynamic power requirement for the complete system mapped to commercial off-the-shelf (COTS) hardware that includes a 5.6 Megapixel color camera processed by object detection and classification algorithms at 30 frames per second was measured at 21.7 Watts (W), for an effective energy consumption of 5.45 nanoJoules (nJ) per bit of incoming video. These unprecedented results show that the NEOVUS has the potential to revolutionize automated video object recognition toward enabling practical low-power and mobile video processing applications.

**Keywords: object detection, object classification, airborne, video image processing, neuromorphic, bio-inspired, low-power, real-time processing**

# **INTRODUCTION**

Unmanned platforms are becoming one of the major sources of data for intelligence and surveillance both on and off the battlefield. High resolution and wide field-of-view sensors are resulting in large volume of images and videos that then need to be processed and analyzed. Two problems arise from these emerging trends: (1) High bandwidth is required to send data from the platform to ground stations even with good compression and (2) High workload is imposed on analysts and end-users to process the data. One solution to these problems is to perform on-board automated image and video analysis (e.g., detect, recognize and track objects of interest) to enable better and timely situational awareness, reduce the amount of data to be streamed, and thus reduce the end-user workload. However, rapid and accurate situational awareness is virtually impossible on-board size- and power-constrained mobile platforms using current state of the art video-processing methods. Video data from these platforms typically includes a large number of objects with few pixels on target and occur under changing illumination, occlusion and clutter conditions. Conventional computer vision methods are generally engineered for specific and limited domains, lack robustness and are computationally expensive, making them unsuitable for onboard processing.

This work presents a real-time video object recognition system that is suitable for onboard processing, for example, on unmanned intelligence, surveillance and reconnaissance (ISR) platforms. This work was partially performed under the DARPA Neovision2 program (DARPA, 2011) whose goal was to enable an unattended visual object-recognition capability on unmanned reconnaissance platforms. Our system NEOVUS departs from traditional, domain-specific engineered video-processing capabilities and is instead inspired by neuromorphic algorithms that model visual information processing (Mishkin et al., 1983; Huang and Grossberg, 2010; LeCun et al., 2010). The goal of NEOVUS is to detect and classify objects of interest (e.g., car, truck, person and boat) in videos in real-time, while consuming significantly lower power than state of the art computer vision.

The NEOVUS combines two key capabilities: (1) Object detection that finds potential objects in the image and outlines a bounding box around them. It combines form-based detection using attention approaches to detect entities based on form (e.g., shape, color, intensity) and motion-based detection mediated by spatial attention. (2) Object classification that then classifies the detected objects based on a feed-forward multi-layered convolutional neural network approach (ConvNet) (LeCun et al., 2010; Farabet et al., 2011). The combination of detection followed by classification provides object recognition results. The NEOVUS is implemented in COTS hardware to achieve realtime performance at low size, weight, and power (SWaP). Several components and capabilities of NEOVUS have been previously described by us Chen et al. (2011), Khosla et al. (2013a), Khosla et al. (2013b), Honda et al. (2013). This paper describes the complete system end-to-end, provides additional details of components and key capabilities, and describes in detail the results of DARPA evaluation on urban datasets.

The rest of the paper is organized as follows. Section Architecture describes the NEOVUS architecture. Section Algorithms describes the underlying algorithms for object detection and classification. Section Hardware Mapping describes mapping of the NEOVUS to COTS hardware. Section Results and Discussion describes object recognition performance and energy consumption results from evaluation of the NEOVUS. Finally, Section Conclusions provides conclusions and discusses some future directions of our work.

#### **ARCHITECTURE**

The NEOVUS is a neuromorphic object-recognition architecture and system that is inspired by the ventral (*what*) and dorsal (*where*) streams of the mammalian visual pathway (Mishkin et al., 1983). It is based on and consistent with neuroscience theories and models of mammalian pathways implicated in visual processing (Mishkin et al., 1983; Huang and Grossberg, 2010). The NEOVUS consists of three primary functions: retinal processing, object detection and object classification, and five components (I– V). **Figure 1** shows the architecture that combines retinal processing (I), object detection (II–IV) and object classification (V) for fast and accurate object recognition. **Table 1** below summarizes these functional components (I–V).

Data flow description and interaction between the components have been previously described (Khosla et al., 2013a). Unlike traditional computer vision approaches (e.g., Felzenszwalb et al., 2010; Kembhavi et al., 2011) that use engineered features and raster-scan processing, this architecture first detects objects and then classifies them based on a set of learned features. The object recognition results are then displayed to the end user for operation and decision-making. In an automated processing application, such results can be used to determine what data are to be transmitted to the end-user, fulfilling the requirements of bandwidth and workload reduction as outlined in the Introduction section. For example, one may wish to transmit sample images of only certain classes of objects (e.g., red car) when the system achieves certain confidence level in its classification results.

#### **ALGORITHMS**

# **OBJECT DETECTION**

The purpose of object detection is to locate potential object regions in the video and pass them on to the object classification stage. This module detects motion and form by modeling interacting dorsal and ventral pathways based on the two stream

models proposed by neuroscientists (Mishkin et al., 1983; Huang and Grossberg, 2010). It needs to operate with a high probability of detection even at reasonably high probability of false positives so as not to miss potential true objects. The false positives are then discarded during the classification stage as background or irrelevant objects. We now describe details of the object detection algorithm.

#### *Static object detection*

Static object detection emulates the attentional pathways that detect objects based on their form saliency with respect to the background. The algorithm is motivated by research on using spectral analysis for visual saliency (Hou and Zhang, 2007). **Figure 2** shows the block diagram of the major steps in our approach to detecting static objects using the RS method (Honda et al., 2013).

*Targeted contrast enhancement (TCE).* The original RS approach works only on gray-scale images. This works well on bright objects, but not on dark objects. TCE enhances gray-scale images for specified colors (e.g., black) which then enables us to easily detect objects with these colors. This idea can be extended to any arbitrary color for a potential object of interest. TCE is performed by using an un-normalized Gaussian function with mean value of the target color and variance incorporated into a single user-specified sensitivity parameter β. The color response for pixel value *V*(*c*) and a select set of *M* target colors *T* = {*Ti*(*c*) : *i* ∈ ... *M*} is:

$$R(V, T\_i) = \exp\left(-\beta\_i \sum\_{\mathcal{c}} \left(V(\mathcal{c}) - T\_i(\mathcal{c})\right)^2\right),$$

where *c* indexes the color channel and β is a sensitivity parameter for the similarity between that color channel and the target color. Usually β is set to 1.0, but can be a varying value for different colors depending on the application. The response *R* is 1 when the image matches the target color and falls off based on deviation from the target color values. Thus, TCE results in multiple gray-scale response maps, one for each target color. **Figure 3** illustrates the effect of TCE for a dark vehicle against dark background.

*Anti-aliasing and down-sampling.* In order to efficiently deal with large images, we down sample the response maps from previous step. Down-sampling can significantly reduce the computational load for subsequent processing. In addition, humans often fixate their attention to a specific scale when looking for objects. Down-sampling can achieve a similar effect by setting the scale for object detection. It effectively creates a pyramid of scales for each image from the previous step.

*Saliency calculation.* Each image from the previous step is now processed by the saliency algorithm. The resulting saliency map can be thought of as an approximate representation of human pre-attentive visual response in a bitmap format. We use Spectral Residual Saliency (RS) approach (Hou and Zhang, 2007) due to its speed and efficiency. The RS method exploits the inverse power law of natural images with the observation that the average of log-spectrums is locally smooth. This enables detecting salient objects based on the log-spectrum of individual images rather than *ensemble* of images. The key steps of this algorithm include:

These processing steps highlight the modes in the frequency domain, the idea being that objects are defined by boundaries constructed from ridges in the Fourier domain. The output is a set of saliency maps that are further processed to produce individual blobs bracketing objects.

*Region detection.* This step converts each saliency map into detections represented by their attributes (e.g., position, size, orientation, and score). A score is associated with each blob that indicates the confidence score based on its peak saliency value.

*Post-processing and fusion.* This final step uses various parameters, such as, object size, aspect ratio, and score to filter out false positives. In more general situation where there are multiple saliency maps or saliency maps of multiple scales, detections from these multiple maps are fused together with the results from motion to produce a final set of object detections, which will be discussed in Section Detection Fusion.

**FIGURE 3 | An illustration of targeted contrast enhancement for color black ([0, 0, 0]) and** *β* **= 55.** The low contrast vehicle in the shade of the tree is enhanced so that it clearly stands out from the shadows.

# *Moving object detection*

*Stationary platform.* This module is used to detect foreground (moving) objects from a stationary platform, e.g., tower-mounted camera. The implementation in this work adopts our previous motion-based saliency technique that detects motion by modeling the background and comparing it to the input image (Kim et al., 2005). Key steps in our implementation are described below:

*Codebook generation.* The background model is constructed and updated by adopting our previous work on a codebook-based method (Kim et al., 2005). A codebook consists of one or more codewords that are formed by using samples from each pixel and clustering them based on a color distortion metric and a brightness ratio. A codeword typically contains an RGB-vector and other features such as the minimum and maximum brightness, occurrence frequency, the maximum negative run-length (MNRL) (Kim et al., 2005), and the first and last access times. These values are effectively used to evaluate feature differences, filter out non-background pixels, and recycle unused codewords.

*Color model and filtering.* The color model separates the color and brightness distortion evaluations to handle the problem of illumination changes, for example, shading and highlights. The color and brightness conditions are satisfied only when both the pure colors and brightness lie within acceptable bounds of the codeword. During training, a fat codebook encodes every incoming pixel in the training set. This fat codebook is then filtered to remove codewords that might contain moving foreground objects. We define MNRL as the maximum interval in time that the codeword has not recurred during the training period (Kim et al., 2005). A codeword with a large MNRL is eliminated from the codebook.

*Foreground detection.* To detect foreground in an incoming image frame, each sample pixel is matched against the corresponding background model. Unlike other Gaussian or kernel based methods, this step does not require probability calculations. The codebook method is fast as it only calculates the distance between the sample and the nearest cluster's mean in the codebook. The pixel is classified as foreground if no matching codeword is found; otherwise it is a background. This is followed by region extraction similar to that described in Section Static object detection.

*Moving platform.* When the sensor platform is moving, we leverage motion information in the video for object detection (Chen et al., 2011; Khosla et al., 2013a). Our approach illustrated in **Figure 4** works well even in the presence of jitter and vibration.

*Feature extraction and matching.* Our approach uses Scale Invariant Feature Transform (SIFT) (Lowe, 1999) features due to its attractive properties of invariance to scale, orientation, and affine distortions. SIFT descriptors represent the gradient orientation histograms and can be used to determine similarity between key points. The matching step matches the key points between a pair of images based on Euclidean distance between their SIFT descriptors and outputs a candidate set of matching points.

*Ground-plane homography estimation.* Some of the candidate matches from the previous step may be incorrect due to noise and inherent limitations of SIFT. To remove these false matches, we apply RANSAC (Fischler and Bolles, 1981) which is an iterative method to estimate parameters of a mathematical model from a set of observed data that contains outliers. We use RANSAC to find a ground-plane homography transform that best fits the candidate matches. This provides accurate matches and transformation (i.e., homography) between a pair of images.

*Frame-to-reference registration.* This step warps the frames into a global coordinate reference. Our approach uses a time window size of N frames, with the middle frame being the reference. Each frame is warped to the reference frame in the window based on the homography transformation. This enables all frames in a time window to be stabilized with respect to the center reference frame.

*Difference accumulation.* This step first computes the difference image between the reference frame and each frame registered to it. The difference image corresponds to independently moving objects against the ground plane. This moving pixel detection process accumulates the differences from several registered frames to increase confidence of detection. For example, with window size *N* = 3, the reference frame *Fi*, its previous frame *Fi* − 1 and next frame *Fi* + 1 are used to compute the frame differences.

*Region extraction.* The previous step effectively produces a motion saliency map, where higher values indicate more prominent motion due to object motion. We then apply the same region

extraction method as described in Section Static Object Detection to obtain a set of candidate detections.

#### *Detection fusion*

The final step in object detection is to combine the form and motion processing detections into a single detection set prior to object classification. Our approach performs fusion within the static object detection followed by its fusion with motion detection. Detections are all represented simply by the detection boxes in terms of their location, size, and score.

For fusion of detections from different saliency channels (e.g., intensity and TCE) and scales, we use several fusion stages. The first stage called "modal" fusion is motivated by observing that different saliency channels carry different importance in a particular application. For example, intensity channel is usually more important than any single channel from TCE, unless the application has a specific goal of finding objects of specific color (e.g., finding all red cars on the road). To account for this we shift the "mode" of detection scores (distribution) by adding an offset to the scores of detections from intensity channel such that detections from intensity channel have higher scores in general than detections from other channels, e.g., a dark channel. After that, fusion of the detections is accomplished by the unions of the detections from different channels, which are prioritized by their scores and will be further filtered in a resource-limited system before they reached the classification stage. The second fusion stage deals with overlapping detections. Detections in different scale can overlap or objects can split split-up in higher resolutions maps. In our implementation, we keep the enclosing and overlapping detections from lower-resolution. In case of detections from different color channels, we keep the detection with the higher score.

The next step is to merge motion detections with the results from fused detections from static object detection as described above. Here we use a variant of the second fusion scheme described above based on detection overlapping. Our experience is that motion detection is much more reliable than saliency based detection, therefore we keep all detections from motion detection. For the fused detections from static object detection, we only keep them if they do not have significant overlap with any detections from motion detection since such overlap indicates they are the same object. A typical example of the detection process from NEOVUS is shown in **Figure 5**.

#### **OBJECT CLASSIFICATION**

Convolutional neural networks (ConvNets) (LeCun et al., 2010) is a supervised deep-learning neural network with multiple layers of similarly structured convolutional feature extraction operations followed by a linear neural network (NN) classifier. ConvNets are an excellent model for image recognition because the structure allows automatic learning of image features as opposed to engineered ones used in traditional computer vision approaches

(Felzenszwalb et al., 2010; Kembhavi et al., 2011). ConvNets typically consist of alternating layers of simple and complex cells as found in mammalian visual cortex. Simple cells perform template matching and complex cells pool these results to achieve invariance. A typical ConvNet has several of 3-layer convolution stages followed by a classifier stage which is a linear NN with one or more hidden layers. Each convolution stage has three layers: (1) a filter bank layer (convolutions) to simulate simple cells (2) nonlinearity activation layer, and (3) feature pooling layer to simulate complex cells. The entire network can be trained using backpropagation with stochastic gradient descent (LeCun et al., 2010). Due to its feedforward nature (non-recursive) and uniform computation within each convolution stage, ConvNets are computationally very efficient. It has been implemented in NeuFlow data-flow processor (Farabet et al., 2011) on COTS field programmable gate arrays (FPGAs) to enable low-power, real-time operation. Our prototype hardware evaluation system described in Section Hardware Mapping used the NeuFlow approach.

Our ConvNet implementation follows the traditional architecture outlined above (LeCun et al., 2010) and has three stages. We first resize input RGB color images of candidate target region to 86 by 86 pixels regardless of their original aspect ratio. Then we convert the image to YUV color space, and further process the Y channel by local subtractive and divisive normalization (LeCun et al., 2010). The convolution layer of first stage has eight convolution filter kernels of 7 by 7 pixels, followed by point-wise sigmoid activation function (tanh()) and max-pooling in 4 by 4 pixels neighborhood and subsampling with a stride of 4 pixels, resulting in eight feature maps of 20 by 20 pixels each at the end of stage 1. The remaining stages are detailed in **Figure 6**. The output of the convolution layer at stage 3 is a 128-D vector, which is then fed to the tanh() non-linear activation layer followed by a fully-connected linear NN classifier. We chose the network parameters based on prior experience with similar datasets and validation.

To train our ConvNet, we use training videos provided by DARPA. Manually annotated video clips were used to train and a separate test set was used to quantify their performance (Section Results and Discussion). These videos contain hand-annotated ground truth for objects of interest in each dataset. The annotated image regions and their class labels are extracted and used to train the ConvNet. Depending on data sets and type of objects, we end up with a few hundreds to a few hundreds of thousands samples for the training step.

# **HARDWARE MAPPING**

The NEOVUS hardware prototype was implemented on commercial COTS hardware (**Figure 7**) and can process both recorded and live video. For energy evaluation purposes, we processed live video from a 5.6 Megapixel RGB color camera at 30 frames per second. The camera connects to a frame grabber (via CameraLink) and a laptop computer (PCI-Express). The computer runs the object detection algorithm and sends image region corresponding to each detected object to a COTS FPGA board (Xilinx ML605 Virtex-6) for object classification. The FPGA board runs NeuFlow implementation of a trained ConvNet and sends the results to the laptop for subsequent display. The dynamic power of the complete system that includes the 5.6 Mpixel camera, object detection, and object classification components running at 30 frames per second was measured by

**FIGURE 6 | ConvNet structure implemented in the NEOVUS.**

DARPA during a formal evaluation at 21.7 Watts (W). This translates to effective energy consumption of 5.45 nanoJoules (nJ) per bit of incoming video.

# **RESULTS AND DISCUSSION**

In this section, we describe results of NEOVUS evaluation by DARPA on three urban area video datasets (Tower, Helicopter and TAILWIND, **Figure 8**) during summative testing conducted at the end of the Neovision2 program. The Tower dataset is filmed from a fixed camera on top of the Stanford University Hoover tower and the Helicopter dataset is filmed from a helicopter flying over Los Angeles. In both cases, the 1080p video imagery is converted to 8 bit PNG frames for analysis. A third dataset from DARPA TAILWIND (Tactical Aircraft to Increase Long Wave Infrared Nighttime Detection) program is captured from an airplane operating at two different altitudes. The Tower and Helicopter datasets are now available for download (iLab Neovision2 annotated video datasets, 2013a1 ). Each video dataset consists of variable number of object classes (**Figure 9**). The video frames typically have multiple objects per frame that may be occluded or even overlapping in some cases. The videos are processed through NEOVUS and its outputs in the form of object locations, bounding boxes, and class labels is logged for every frame. The NEOVUS logged results are evaluated using ground-truth and evaluation software. The ground-truth is created by human annotators at VideoMining Corporation who labeled 10 object classes (Boat, Car, Container, Cyclist, Helicopter, Person, Plane, Tractor-Trailer, Bus, and Truck) in these datasets. Each object is enclosed in an oriented rectangle that best encompasses the object. Ten percent of the datasets are annotated by two independent annotators and their outputs are compared to assess quality and consistency of annotations; significant differences between the two sets trigger a review of the annotation process to assure annotation quality. The groundtruth was created under DARPA supervision and no performer in the program had any control of that process. The evaluation software uses Normalized Multiple Object Thresholded Detection Accuracy (NMOTDA) score (Kasturi et al., 2009; iLab Neovision2 Performance Evaluation Protocol, 2013b2 ) to evaluate performance per sequence:

$$\text{NMOTDA} = 1 - \frac{\sum\_{i=1}^{N \text{frams}} \left( c\_m m^{(t)} + c\_f m f^{(t)} \right)}{\sum\_{i=1}^{N \text{frams}} \left( N\_G^{(t)} \right)}$$

1Available online at: http://ilab.usc.edu/neo2/dataset/

2Available online at: http://ilab.usc.edu/neo2/dataset/neovision2-perfor mance-evaluation-protocol.pdf

where *m*(*t*) = number of missed detections, *f* (*t*) = number of false positives, and *N*(*t*) *G* = number of ground-truth objects in the *t*th frame. The summations are carried out over all evaluated frames. In Neovision2 evaluations, the cost functions *cm* and *cf* for missed detects and false positives, respectively were set to identity. This is a sequence-based measure which penalizes false detections, missed detections and object fragmentation. Note that maximizing NMOTDA for the sequence is the same as finding the optimal assignment of ground truth boxes to system output boxes at each frame image.

**Figure 10** gives an example of NEOVUS results superimposed on a frame of Tower dataset. By aggregating the results, one can plot the ROC curves as Pd (probability of correct recognition) vs. FPPI (False Positives Per Image). **Figure 11** shows the NEOVUS results for four object classes on the Tower dataset. Overall, NEOVUS achieved excellent object recognition at about 90% per-frame accuracy and 0.1 FPPI.

In each data domain, multiple sequences are used for evaluation. The summary of the performance over the entire dataset, i.e., over all the video clips, is calculated using Weighted NMOTDA (WNMOTDA). WNMOTDA is calculated for each class separately and is given by,

$$\text{WNMOTDA}\_{i} = \frac{\sum\_{j=1}^{NVC} \text{NMOTDA}\_{ij} \* \text{NGT}\_{ij}}{\sum\_{j=1}^{NVC} \text{NGT}\_{ij}}$$

where *WNMOTDAi* is the NMOTDA for class *i* calculated over all the video clips, *NMOTDAij* is the NMOTDA measure calculated for class *i* in video clip *j*, *NVC* is the total number of video clips, and *NGTij* is the number of ground truth objects of class *i* in video clip *j*. Finally an average WNMOTDA score is generated for all object classes for each domain by ignoring the class labels. Before scoring, identical boxes are merged into one. Overlapped boxes (if Overlap\_Ratio is over 20%) are merged into one and their union is used instead. All of these above calculations are part of the evaluation software.

The recognition performance and energy consumption results of summative testing were formally released by DARPA and are presented in **Figure 12** (DARPA, 2011). Five teams participated in the program and evaluation. Three neuromorphic teams (C, D, E) developed neuromorphic vision algorithms. Two baseline algorithms, based on the Deformable Part Model algorithm (Felzenszwalb et al., 2010), representing typical computer vision methods were implemented to serve as a benchmark for comparing against the neuromorphic algorithms. HRL's NEOVUS (Team C) was the best performer with high recognition accuracy (WNOMTDA) and the lowest energy use (nJ/bit) amongst all five teams. Particularly noteworthy is that the NEOVUS energy use was at least three orders of magnitude lower than the computer vision systems (Teams A and B, **Figure 12**). These unprecedented results show that neuromorphic

algorithms and hardware have the potential to revolutionize lowpower situational awareness applications, e.g., on-board small unmanned aerial vehicles (UAV).

Our current low-power NEOVUS implementation is mature enough to be deployed on mobile platforms toward onboard processing. **Figure 13** supports this claim by analyzing platform payload available SWaP and extrapolating our measured energy use (5.45 nanoJoules per bit) to other camera resolutions. For example, NEOVUS could run onboard small UAV's, such as the AeroVironment Raven (Raven RQ-11, 20143 ), in real-time for a 1-Megapixel camera. Larger UAVs, such as the Boeing-Insitu ScanEagle

3Available online at: http://www.avinc.com/uas/adc/raven/.

(Boeing Insitu ScanEagle, 20144 ), can hold up to a 6-Megapixel camera and still support NEOVUS processing in real-time.

4Available online at: http://www.insitu.com/systems/scaneagle

performance of all objects regardless of class.

# **CONCLUSIONS**

This work described a neuromorphic object recognition system inspired by neuroscience findings of object recognition pathways in the mammalian visual system. From a practical perspective, the NEOVUS is a proven collection of neuromorphic algorithms, software, and architecture for automated and accurate video object recognition at significantly lower power than state of the art computer vision. It processes video based on human-like search and classification models that are significantly different from computer vision brute-force raster-scan approaches. The NEOVUS was successfully evaluated on real-world urban video datasets and proven to accurately recognize objects at lowpower. The successful hardware design and mapping of NEOVUS to COTS hardware can help enable potential autonomous and mobile applications. This onboard processing can reduce both the requirements for data bandwidth and analyst man power. While the NEOVUS hardware was geared for autonomous on-board processing, it is just as applicable to off-board processing of live or recorded images and videos. For off-board processing, the highly efficient algorithms used in NEOVUS can enable video analysis in faster than real-time even with COTS computer hardware.

The NEOVUS described in this work is a feed-forward, bottom-up object recognition architecture. However, models and algorithms for top-down attention and processing can be added to the current architecture with little modifications. For example, goal-directed search and classification, e.g., find and track all white trucks) can be added to our framework. Future work

will include these top-down aspects and onboard evaluation of this capability. This is expected to yield the greatest level of improvement toward enabling practical systems.

# **ACKNOWLEDGMENTS**

This work was partially supported by the Defense Advanced Research Projects Agency Neovision2 program (contract HR0011-10-C-0033). The views and conclusions contained in this document are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. The authors would like to acknowledge Drs. Yann LeCun and Clement Farabet at the New York University (NYU) for their support on the classification algorithms and NeuFlow hardware mapping.

#### **REFERENCES**

Werblin, F. S., and Dowling, J. (2010). "Evolution of retinal circuitry: from then to now," in *Handbook of Brain Microcircuits*, 1st Edn., eds G. Shepherd and S. Grillner (New York, NY: Oxford University Press), 200–207.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 14 April 2014; accepted: 29 October 2014; published online: 28 November 2014.*

*Citation: Khosla D, Chen Y and Kim K (2014) A neuromorphic system for video object recognition. Front. Comput. Neurosci. 8:147. doi: 10.3389/fncom.2014.00147*

*This article was submitted to the journal Frontiers in Computational Neuroscience. Copyright © 2014 Khosla, Chen and Kim. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Shape representations in the primate dorsal visual stream

Tom Theys 1, 2, Maria C. Romero1 , Johannes van Loon2 and Peter Janssen1 \*

*1 Laboratorium voor Neuro- en Psychofysiologie, Katholieke Universiteit Leuven, Leuven, Belgium, 2 Afdeling Experimentele Neurochirurgie en Neuroanatomie, Katholieke Universiteit Leuven, Leuven, Belgium*

The primate visual system extracts object shape information for object recognition in the ventral visual stream. Recent research has demonstrated that object shape is also processed in the dorsal visual stream, which is specialized for spatial vision and the planning of actions. A number of studies have investigated the coding of 2D shape in the anterior intraparietal area (AIP), one of the end-stage areas of the dorsal stream which has been implicated in the extraction of affordances for the purpose of grasping. These findings challenge the current understanding of area AIP as a critical stage in the dorsal stream for the extraction of object affordances. The representation of three-dimensional (3D) shape has been studied in two interconnected areas known to be critical for object grasping: area AIP and area F5a in the ventral premotor cortex (PMv), to which AIP projects. In both areas neurons respond selectively to 3D shape defined by binocular disparity, but the latency of the neural selectivity is approximately 10 ms longer in F5a compared to AIP, consistent with its higher position in the hierarchy of cortical areas. Furthermore, F5a neurons were more sensitive to small amplitudes of 3D curvature and could detect subtle differences in 3D structure more reliably than AIP neurons. In both areas, 3D-shape selective neurons were co-localized with neurons showing motor-related activity during object grasping in the dark, indicating a close convergence of visual and motor information on the same clusters of neurons.

Keywords: object, shape, visual cortex, macaque, depth, parietal cortex, dorsal stream

# Introduction

Visual object analysis in natural conditions is computationally demanding but critical for survival, hence the primate brain devotes considerable computing power to solve this problem. Lesion studies in monkeys (Ungerleider and Mishkin, 1982) and patients (Goodale et al., 1991) have demonstrated that the visual system beyond primary visual cortex consists of two subdivisions, a ventral stream directed toward the temporal cortex for object recognition and categorization, and a dorsal stream directed to the parietal cortex for spatial vision and the planning of actions (**Figure 1A**). Since primates not only recognize and categorize objects, but also grasp and manipulate those objects, it comes as no surprise that objects are processed in both the ventral and the dorsal visual stream.

The first recording experiments in the ventral stream, which is critical for object recognition, were published more than four decades ago (Gross et al., 1969), and the accumulated knowledge about the properties of individual neurons has spurred the development of a large number of computational models on object and shape analysis for object recognition (Riesenhuber and Poggio, 1999; Poggio and Ullman, 2013). However, neurophysiological evidence for the visual

#### Edited by:

*Antonio J. Rodriguez-Sanchez, University of Innsbruck, Austria*

#### Reviewed by:

*Mazyar Fallah, York University, Canada Charles Edward Connor, Johns Hopkins University School of Medicine, USA*

#### \*Correspondence:

*Peter Janssen, Laboratorium voor Neuro- en Psychofysiologie, Katholieke Universiteit Leuven, Herestraat 49, Bus 1021, B-3000 Leuven, Belgium<EMAIL_ADDRESS>
> Received: *28 March 2014* Accepted: *20 March 2015* Published: *22 April 2015*

#### Citation:

*Theys T, Romero MC, van Loon J and Janssen P (2015) Shape representations in the primate dorsal visual stream. Front. Comput. Neurosci. 9:43. doi: 10.3389/fncom.2015.00043*

FIGURE 1 | Cortical areas processing object shape. (A) Overview of the macaque brain illustrating the locations of the areas involved in processing object shape, and the most important connections between these areas. Unidirectional arrows indicate the presumed flow of visual information along the dorsal stream, the bidirectional arrow between AIP and TEs indicates that the direction of information flow is unclear at present. Note that most connections in extrastriate cortex are bidirectional. CIP, caudal intraparietal area; LIP, lateral intraparietal area; AIP, anterior intraparietal area; F5a, anterior subsector of area F5; TEs, subsector of area TE in the anterior Superior Temporal Sulcus. (B) Schematic flow chart of visual 3D information. Dark boxes indicate areas of the dorsal visual stream, open boxes indicate ventral stream areas, hatched boxes indicate areas selective for higher-order disparity. Boxes with question marks indicate unknown areas.

analysis of objects in the dorsal stream has only recently emerged, and biologically-plausible models for the dorsal stream are scarce (Fagg and Arbib, 1998; Molina-Vilaplana et al., 2007). Robots have to interact with objects in an unpredictable environment, and artificial vision systems that operate based on principles used by the primate dorsal stream areas could undoubtedly advance the fields of computer vision and robotics (Kruger et al., 2013). To stimulate the interaction between neurophysiology and computational modeling, it is important to review recent progress in our understanding of the neural representation of object shape in the primate dorsal visual stream.

Objects contain both two-dimensional (2D: e.g., contour, color, texture) and three-dimensional (3D: e.g., orientation in depth and depth structure) information. Originating in primary visual cortex, at least three different pathways are sensitive to depth information (**Figure 1B**). The MT/V5–MST/FST pathway is primarily involved in the visual analysis of moving stimuli and ego-motion, and FST neurons are selective for three-dimensional shape defined by structure-from-motion (Mysore et al., 2010). The ventral pathway V4–TEO–TEs builds a very detailed representation of the depth structure of objects, and finally the V3A–CIP–AIP–F5a pathway analyses object shape for grasping and manipulation. These pathways should not be regarded as entirely separate entities, since numerous interactions between them exist at different levels in the hierarchy. Rather, each pathway has its own specialization and can function independently of the other pathways.

In this review we will focus on the properties of individual neurons in the parietal and frontal cortex, the hierarchy of cortical areas that links early visual areas to the motor system, and the relation between neuronal firing and behavior. Neurons in other parietal areas, such as area 5 in the medial bank of the IPS (Gardner et al., 2007) and area V6A in the medial parietooccipital cortex (Fattori et al., 2012) also respond selectively to objects of different sizes and orientations. However, the role of these neurons in computing object shape to guide the preshaping of the hand during grasping is less clear at present. We will first discuss the coding of two-dimensional (2D) shape in areas LIP and AIP, the network of areas involved in processing threedimensional (3D) shape investigated with fMRI, and finally the single-cell properties of neurons involved in 3D shape coding in the dorsal stream.

# Two-Dimensional Shape Selectivity in the Dorsal Visual Stream

The first report of shape selectivity in the dorsal stream was a study by Sereno et al. (Sereno and Maunsell, 1998) in the lateral intraparietal area (LIP), an area in posterior parietal cortex (**Figure 1**) traditionally associated with eye movement planning and visual attention (Colby and Goldberg, 1999; Andersen and Buneo, 2002). In this study, many LIP neurons showed clear selectivity for simple two-dimensional (2D) shapes appearing in the receptive field (RF) in the absence of any eye movements. However, size and position invariance—two properties that are believed to be essential for genuine shape selectivity were only tested in a small number of neurons. A more recent study (Janssen et al., 2008) confirmed the presence of shapeselective responses in LIP. However, a more systematic test of size and position invariance revealed that LIP neurons rarely exhibit these properties. In many cases shape-selective responses arose because of accidental interactions between the shape and the RF, such as a partial overlap. The RF structure of these LIP neurons was frequently inhomogeneous with multiple local maxima, and could even depend on the stimulus and the task: for example the RF tested with small shapes could be different from the RF tested with saccades. The lack of tolerance to changes in stimulus position in LIP neurons represented the first evidence that the shape representation in the dorsal stream is fundamentally distinct from the shape representation in the ventral visual stream, which is characterized by shape selectivity and tolerance of the shape preference to changes in stimulus position.

Just anterior to LIP lies area AIP (**Figure 1**), an area known to be critical for object grasping (Gallese et al., 1994; Murata et al., 2000; Baumann et al., 2009). Romero et al. (2012), recorded in area AIP using 2D images of familiar (e.g., fruits) and unfamiliar (tools) objects. Almost all AIP neurons showed significant selectivity to these images of objects, but subsequent testing with silhouettes and outline stimuli revealed that this selectivity was primarily based on the contours of the images. A follow-up study (Romero et al., 2013), demonstrated that for most AIP neurons, the presence of binocular disparity in these images was not necessary, and that a population of AIP neurons represents primarily relatively simple stimulus features present in images of objects, such as aspect ratio and orientation.

The observation of neural selectivity for shape contours in AIP does not allow determining which shape features are being extracted by AIP neurons. For example, is the entire contour necessary or are parts of the shape contour (possibly corresponding to grasping affordances) sufficient to evoke AIP responses? Romero et al. (2014) used a systematic stimulus reduction approach, in which outline stimuli were fragmented into 4, 8, or 16 parts, the latter measuring merely 1–1.5◦ . Following previous studies in the ventral visual stream (Tanaka, 1996), the authors determined the minimal effective shape feature as the smallest fragment to which the neural response was at least 70% of the response to the intact outline. The example AIP neuron illustrated in **Figure 2** responded strongly to the outline of a key but not to the outline of a monkey hand. However, some of the smallest fragments in the test still elicited robust responses in this neuron. Hence although AIP is thought to be involved in extracting grasping affordances, the AIP responses were mainly driven by very simple shape features, but not by object parts that can be grasped (**Figure 2**). Similar to previous observations in neighboring area LIP, the fragment selectivity depended strongly on the spatial position of the stimulus, since even small position shifts (2.5◦ ) evoked radically different responses and therefore a very different shape selectivity. Basic orientation selectivity or differences in eye movements could not explain the fragment responses. These results suggest that AIP neurons may not extract grasp affordances. Future studies should determine how the 2D-shape representation changes in ventral premotor areas.

# A Network of Cortical Areas Sensitive to the Depth Structure of Objects

The depth structure of objects (i.e., flat, convex, or concave) can be specified by a large number of depth cues such as motion parallax, texture gradients, and shading. Many studies investigating the neural basis of 3D object vision have used random dot

each box the stimulus is illustrated, the color of each box represents the normalized firing rate of the neuron to that stimulus (maximum response was 29 spikes/s). Top row: intact object contour, second row: the four fragments derived from subdividing the object contour into four fragments along the

from which it was derived. The original object images from which the contours were derived are illustrated on the left side (arrows pointing to their respective contour stimuli). All stimuli were presented at the center of the RF. Reproduced with permission from (Romero et al., 2014).

stereograms (**Figure 3**), in which depth information is exclusively defined by the gradients of binocular disparity, for obvious reasons. First of all, binocular disparity is the most powerful depth cue (Howard and Rogers, 1995): even when present in isolation, disparity evokes a very vivid percept of depth that is—in contrast to motion parallax or shading—unambiguous with respect to the sign of depth (near or far, convex or concave). Moreover, physiologists are particularly keen on this type of stimuli because one can easily determine whether a neuron (or even a cortical area in the case of fMRI) is responding to the depth from disparity in the stimulus and not to other stimulus features: simply presenting the stimulus to one eye only removes all depth information in the stimulus while preserving shape and texture (Janssen et al., 1999; Durand et al., 2007). In contrast, for other depth cues such as texture gradients, determining which aspect of the stimulus the neuron responds to requires numerous control stimuli. Finally, stereograms also allow precise behavioral control: monkeys and humans can be trained to discriminate depth in stereograms, and varying the percentage of correlation between the dots in the images presented to the left and the right eye (disparity coherence) furnishes a parametric manipulation of the strength of the depth stimulus which can then be related to behaviorial performance.

A series of functional imaging and single-cell studies in monkeys (Janssen et al., 2000a; Durand et al., 2007; Joly et al., 2009) as well as imaging experiments in humans (Georgieva et al., 2009) have suggested that many cortical areas located in both the dorsal and the ventral visual stream are sensitive to the depth structure of objects (**Figure 1A**). This network contains AIP which, as mentioned earlier, is known to be critical for grasping (Gallese et al., 1994; Murata et al., 2000), a region (TEs) in the inferior temporal cortex (ITC), the end-stage of the ventral visual stream and critical for object recognition, and a subsector of the ventral premotor cortex (area F5a, Joly et al., 2009). The observation that not only ventral stream but also dorsal stream areas are processing 3D object information came as no surprise, since many years before these imaging studies, investigations in a patient with a ventral stream lesion had already indicated that object grasping can be intact while object recognition is severely impaired (Goodale et al., 1991; Murata et al., 2000). The object analysis required for object grasping was presumably performed by her intact dorsal stream areas (Murata et al., 2000; James et al., 2003).

# Single-Cell Studies in the Dorsal Visual Stream on the Visual Analysis of 3D Structure

fMRI can identify regions that are activated more by curved surfaces than by flat surfaces, but a detailed understanding of the neuronal selectivity in these areas requires invasive electrophysiological recordings of single neurons. Early in the hierarchy of the dorsal visual stream, the Caudal Intraparietal area (CIP) has been studied using inclined planar surfaces in which depth was defined by binocular disparity and/or texture gradients (Tsutsui et al., 2002). CIP neurons can signal the 3D-orientation

(the tilt) of large planar surfaces when either disparity or texture gradients are used as a depth cue (i.e., cue invariance). A more recent report suggests that CIP neurons can also be selective for disparity-defined concave and convex surfaces (Katsuyama et al., 2010). Rosenberg et al. (2013) showed that individual CIP neurons jointly encode the tilt and slant of large planar surfaces. In view of the anatomical connections of CIP, which run along the lateral bank of the IPS toward area AIP (Nakamura et al., 2001), and more recent preliminary monkey fMRI findings (Van Dromme and Janssen, unpublished observations), CIP could be an important—but not the only (Borra et al., 2008)—input area for AIP. However, since reversible inactivation of area CIP does not cause a grasping deficit (Tsutsui et al., 2001) but sometimes a perceptual deficit in the discrimination of tilt and slant, the role of area CIP in computing 3D object shape for object grasping remains largely unknown.

Previous studies in area AIP had reported object-selective responses in this area (Murata et al., 2000) but it was unclear whether these neurons encoded differences in 3D structure, 2D contour, orientation or any other feature that differed between the objects used in those experiments. Srivastava et al. (2009) recorded single-cell activity in the AIP of awake fixating rhesus monkeys using disparity-defined curved surfaces. A large proportion of AIP neurons responded selectively to concave and convex surfaces that had identical contours, as illustrated by the example neuron in **Figure 4**. This neuron fired vigorously when a convex surface was presented, but not at all when the surface was concave, a selectivity which could not be accounted for by the responses to the monocular presentations. Since this neuron preserved its selectivity across positions in depth (data not shown), the neuron must have responded to a change in binocular disparity along the surface of the stimulus, i.e., higher-order disparity selectivity. The same study observed that the neuronal properties in AIP were markedly different from the ones in TEs: AIP neurons fired much faster to the presentation of curved surfaces (a population latency of 60–70 ms in AIP compared to 90–100 ms in TEs), but appeared less sensitive to small differences in 3D structure, including the sign of curvature: while TEs neurons frequently showed similar responses to curved surfaces with different degrees of curvedness (provided they had the same sign, i.e.,

either convex or concave) and large response differences when the sign of curvature changed (even for very slightly curved surfaces), the response of AIP neurons declined monotonically as the degree of curvedness was decreased. These observations were the first demonstration that a distinct representation of the depth structure of objects exists in the dorsal visual stream. A followup study (Theys et al., 2012b), showed that the large majority of the AIP neurons are primarily sensitive to the disparity gradients on the boundary of the stimulus but largely ignore the depth structure information on the surface, which represents another difference with TEs neurons.

The ventral premotor cortex (PMv) represents the main target of area AIP (Borra et al., 2008). Reversible inactivation of PMv produces a grasping deficit that is highly similar to the one seen after AIP inactivation (Gallese et al., 1994; Fogassi et al., 2001). However, recent studies have provided functional and anatomical evidence suggesting that PMv is not a homogeneous area. A monkey fMRI study (Joly et al., 2009) showed that a subsector of PMv, area F5a, is more activated by curved surfaces than by flat surfaces at different positions in depth, similar to AIP. This activation was located in the depth of the more anterior part of the inferior ramus of the arcuate sulcus. Recent anatomical studies (Belmalih et al., 2009; Gerbella et al., 2011) described differences in the cytoarchitectonics and anatomical connectivity between area F5a and the surrounding subsectors of PMv: F5a does not project directly to primary motor cortex, but does so through its connections with F5p. Moreover, F5a is more strongly connected to the parietal (AIP) and prefrontal (areas 46 and 12) cortex. Based on the anatomical connectivity of F5a, Gerbella et al. (2011) coined the term "pre-premotor cortex" for area F5a, indicating that this subsector of PMv could represent a stage upstream from the more widely studied F5p and F5c sectors of PMv.

Guided by a monkey fMRI study (Joly et al., 2009), Theys et al. (2012a) targeted the F5a subsector with microelectrode recordings to investigate in detail to what extent this region differs functionally from the other subsectors of PMv. Accurately predicted by fMRI, neurons selective for disparity-defined curved surfaces were located in F5a but not in surrounding regions of PMv. The example neuron in **Figure 5** responded to a convex depth profile but not to a concave depth profile irrespective of the position in depth of the stimulus, and monocular responses could not account for the selectivity (data not shown). Remarkably in view of its anatomical location in the premotor cortex, the responses of these F5a neurons appeared very "visual," with robust increases in firing rate when visual stimuli (that the monkey could not grasp) appeared on a display, and with relatively short response latencies (70–80 ms) compared to 50–60 ms for AIP, which is consistent with the higher position in the cortical hierarchy of F5a compared to AIP. These strong visual responses to images presented on a display were unexpected, since previous studies (Graziano et al., 1997; Graziano and Gross, 1998) did not observe responses to images of objects presented on a display in PMv neurons with bimodal visuotactile responses. i.e., responses to tactile stimulation of the face or hand and to visual presentation of objects near the face or hand.

These observations raised the question whether F5a could still be considered part of PMv. To that end, (Theys et al., 2012a) after having established higher-order disparity selectivity in a cluster of F5a neurons—recorded from the same neurons while the animal was grasping objects in the light (i.e., visually-guided grasping) and in the dark (i.e., memory-guided grasping). Surprisingly, almost all F5a neurons selective for disparity-defined depth structure were also active in the light when the monkey was grasping objects that did not resemble the random-dot

stereograms. However, most of these F5a neurons were virtually silent when the animal was grasping the same objects in the dark. Hence, in contrast to earlier reports stating that all PMv neurons remain active during grasping in the dark (Raos et al., 2006), F5a neurons selective for the depth structure of objects are mostly visual-dominant. These results do not imply that all F5a neurons are visual-dominant, since multi-unit recordings of clusters of 3D-structure selective neurons revealed strong graspingrelated activity in the dark. The implication of these findings is that (visual-dominant) 3D-structure selective F5a neurons are co-localized and strongly connected with visuomotor and motordominant neurons that remain active in the absence of visual information. Such functional clusters may form neural modules in which visual object information is mapped onto motor commands. The presence of activity during grasping in the dark implies that F5a is effectively a subsector of PMv, which differs from the other subsectors of PMv by the presence of visualdominant responses during grasping and selectivity to the depth structure of objects.

The previous studies suggest that the hierarchy of dorsal stream cortical areas involved in 3D-shape processing is largely serial. Although most likely an oversimplification (e.g., the role of feedback connections is unknown and ignored here), the CIP– AIP–F5a serial chain of areas provides a unique opportunity to investigate how the 3D-shape representation changes along the dorsal pathway so that the underlying computations might be revealed. In a first attempt to address this question, Theys et al. (2013) recorded in F5a and AIP in the same animals during visual presentation of 3D surfaces, various approximations of these surfaces and during object grasping. The sensitivity for depth structure was measured by plotting the average responses of a population of neurons to curved surfaces with varying degrees of the disparity variation (from very convex, over almost flat to very concave surfaces). Although interindividual differences between the two animals were present, the sensitivity functions were virtually identical in F5a and AIP. Furthermore, testing F5a neurons with planar (i.e., least-square) and discrete approximations of the smoothly curved surfaces showed again strong similarities between AIP and F5a (**Figure 6**): the majority of neurons in AIP and F5a was also selective for discrete approximations of the convex and concave surfaces consisting of three separate planes in depth, but in F5a, the linear approximation evoked significantly less responses compared to the smoothly curved surfaces. Finally, AIP neurons encoding depth structure from disparity were also tested during object grasping, and similar to F5a, most of these AIP neurons were also strongly active during grasping. The only difference between the AIP and the F5a multi-unit activity consisted of stronger and faster responses in AIP during object fixation and higher activity in F5a during the hand movement epoch before object lift, suggesting that AIP neurons are mainly active during the visual analysis of the object whereas F5a neurons remain active throughout the trial. Overall, the representation of depth structure in premotor area F5a was highly similar to that in parietal areas AIP, which makes it difficult to identify the computations that take place between these two different stages in the dorsal stream shape hierarchy. Future studies may be able to document the differences in the object representation between F5a and AIP using different stimuli or tasks.

The strong correspondence between depth structure selectivity and grasping responses in AIP and F5a is consistent with

the hypothesis that the pre-shaping of the hand during visuallyguided object grasping relies on 3D-object information in these two areas. The question still remains why the primate brain would maintain two highly similar areas that communicate by means of (metabolically expensive) long-range connections. At this point the data suggest that the neural object representation in AIP does not have to become much more elaborate to guide the hand during grasping. Furthermore, F5a can directly interact with other premotor areas such as F5p, which AIP can also access directly or indirectly through either parietal area PFG or F5a. The interpretation of "motor" activity, i.e., activity during grasping in the dark, may be crucial in this respect. Both F5a and AIP contain large numbers of neurons that are active in the dark, in the absence of visual information, which are typically not selective for disparity-defined depth structure. Traditionally, this "motor" type of activity has been interpreted as related to action planning, since the activity remains high in the delay period when the animal is waiting for the go-signal to execute the grasping action. However, the activity in the dark in AIP may have a different status: AIP neurons may receive corollary information from premotor areas about the movement planning which can be integrated with visual information during visually-guided grasping (Rizzolatti and Luppino, 2001), an idea that has never been tested. Under this hypothesis, F5a motor activity is genuinely sub-serving action planning, whereas AIP motor activity is simply a corollary discharge reflecting premotor signals.

# Conclusions

More than four decades of research have been devoted to investigations of the object representation in the ITC (for review, see Tanaka, 1996). ITC neurons respond selectively to shapes, and at the same time achieve selectivity invariance, i.e., these neurons exhibit tolerance of shape preference for stimulus transformations such as changes in retinal position, size, the visual cue defining the shape and occlusion. These neuronal properties are believed to be essential to support robust object recognition in an ever changing environment. Single-cell studies have demonstrated that neurons in TEs, a subsector of the ITC located anteriorly and therefore one of the end-stage areas of the ventral visual stream, also encode the 3D structure of surfaces (Janssen et al., 2000a,b) defined by binocular disparity and 3D orientation defined by disparity and texture (Liu et al., 2004), and TEs activity is causally related to the categorization of depth structure (Verhoef et al., 2012). Undoubtedly, a complex hierarchy of visual areas along the ventral visual stream supports the high-level 3D object representation culminating in TEs.

In the dorsal visual stream, the neural representations of depth structure can be traced from mid-level visual area CIP, which presumably receives input from early visual area V3A, until the motor system (F5a). Somewhere along this dorsal pathway, visual object representations are transformed into motor commands (grip type representations) that control the preshaping of the hand during object grasping. As outline above, inactivation studies have demonstrated that at least AIP and F5 are both important for motor control during grasping, but the role of other parietal areas such as PFG (Bonini et al., 2012) and V6A (Fattori et al., 2010) in grasping requires further study. Investigating the neural representation of object shape demands systematic stimulus manipulations (e.g., stimulus reduction), therefore visual object representations can be primarily studied in neurons that respond to images of objects (either 3D or 2D), as in AIP and F5a. Neurons in F5p and F5c, in contrast, respond selectively to real-world objects (Raos et al., 2006) but not to 3D images of objects (Theys et al., 2012a), most likely because these areas represent grip types, which are not activated by images of objects.

Numerous very basic questions remain to be addressed in future studies: for example, how do the RFs change along the dorsal pathway, which computations take place at different stages, what is the role of feedback projections? As outlined earlier, it is also unclear at present how the ventral stream division of this 3D-shape network is organized. Our comprehension of this network will only increase when studies combine electrophysiological recordings, imaging and ultimately also computational modeling.

# References

# Acknowledgments

This work was supported by Geconcerteerde Onderzoeksacties (2010/19), Fonds voor Wetenschappelijk Onderzoek Vlaanderen Grants (G.0495.05, G.0713.09), Programmafinanciering (PFV/10/008), IUAP VII-11, and ERC-2010-StG-260607.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2015 Theys, Romero, van Loon and Janssen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Modeling the shape hierarchy for visually guided grasping

#### *Omid Rezai 1 \*, Ashley Kleinhans 2,3, Eduardo Matallanas 4, Ben Selby1 and Bryan P. Tripp1 \**

*1 Department of Systems Design Engineering, Centre for Theoretical Neuroscience, University of Waterloo, Waterloo, ON, Canada*

*2 Mobile Intelligent Autonomous Systems, Council for Scientific and Industrial Research, Pretoria, South Africa*

*3 School of Mechanical and Industrial Engineering, University of Johannesburg, Johannesburg, South Africa*

*4 ETSI Telecomunicación, Universidad Politécnica de Madrid, Madrid, Spain*

#### *Edited by:*

*Antonio J. Rodriguez-Sanchez, University of Innsbruck, Austria*

#### *Reviewed by:*

*Abdelmalik Moujahid, University of the Basque Country UPV/EHU, Spain Ko Sakai, University of Tsukuba, Japan Peter Janssen, Catholic University of Leuven, Belgium*

#### *\*Correspondence:*

*Omid Rezai and Bryan P. Tripp, Systems Design Engineering, University of Waterloo, 200 University Avenue West, Waterloo, ON N2L 3G1, Canada e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
The monkey anterior intraparietal area (AIP) encodes visual information about three-dimensional object shape that is used to shape the hand for grasping. We modeled shape tuning in visual AIP neurons and its relationship with curvature and gradient information from the caudal intraparietal area (CIP). The main goal was to gain insight into the kinds of shape parameterizations that can account for AIP tuning and that are consistent with both the inputs to AIP and the role of AIP in grasping. We first experimented with superquadric shape parameters. We considered superquadrics because they occupy a role in robotics that is similar to AIP, in that superquadric fits are derived from visual input and used for grasp planning. We also experimented with an alternative shape parameterization that was based on an Isomap dimension reduction of spatial derivatives of depth (i.e., distance from the observer to the object surface). We considered an Isomap-based model because its parameters lacked discontinuities between similar shapes. When we matched the dimension of the Isomap to the number of superquadric parameters, the superquadric model fit the AIP data somewhat more closely. However, higher-dimensional Isomaps provided excellent fits. Also, we found that the Isomap parameters could be approximated much more accurately than superquadric parameters by feedforward neural networks with CIP-like inputs. We conclude that Isomaps, or perhaps alternative dimension reductions of visual inputs to AIP, provide a promising model of AIP electrophysiology data. Further work is needed to test whether such shape parameterizations actually provide an effective basis for grasp control.

**Keywords: AIP, CIP, grasping, 3D shape, cosine tuning, superquadrics, Isomap**

# **1. INTRODUCTION**

The macaque anterior intraparietal area (AIP) receives input from the visual cortex, and is involved in visually guided grasping. A large fraction of neurons in this area encode information about three-dimensional object shapes from visual input (Murata et al., 2000; Sakaguchi et al., 2010). Responses are typically relatively invariant to object position in depth (Srivastava et al., 2009). The responses of some neurons are also invariant to other properties. For example, some are orientation-tuned but not highly sensitive to object shape (Murata et al., 2000). AIP has a strong recurrent connection with premotor area F5, which is involved in hand shaping for grasping (Rizzolatti et al., 1990; Luppino et al., 1999; Borra et al., 2008). Reversible inactivation of AIP leads to grasping impairment, specifically a mismatch between object shape and hand preshape (Gallese et al., 1994; Fogassi et al., 2001). AIP is therefore thought to provide visual information for grasp control (Jeannerod et al., 1995; Fagg and Arbib, 1998).

The focus of this paper is the pathway from V3 and V3A, to the caudal intraparietal area (CIP), to visual-dominant neurons in AIP (Nakamura et al., 2001; Tsutsui et al., 2002). This pathway makes binocular disparity information available for grasp control. Most V3 neurons are selective for binocular disparity (Adams and Zeki, 2001). V3 sends a major projection to V3A (Felleman et al., 1997), which is also strongly activated during binocular disparity processing (Tsao et al., 2003). Both V3 and V3A project to CIP (Katsuyama et al., 2010). CIP neurons are selective for depth gradients (Taira et al., 2000; Tsutsui et al., 2002; Rosenberg et al., 2013) and curvature (Katsuyama et al., 2010). Neurons in AIP receive disynaptic input from V3A via CIP (Nakamura et al., 2001; Borra et al., 2008). Visual-dominant AIP neurons are selective for 3D object shape (Srivastava et al., 2009; Sakaguchi et al., 2010) cued by binocular disparity, consistent with input from this pathway.

AIP also receives many other inputs that we do not model in the present study. The first of these is the premotor area F5, which together with AIP forms a circuit for grasp-related visuomotor transformations. AIP also receives input from the second somatosensory (SII) cortical region (Krubitzer et al., 1995; Fitzgerald et al., 2004; Gregoriou et al., 2006), which may provide tactile feedback and memory-based somatosensory expectations for grasping. Strong connections with other parietal areas are also identified, as well as with prefrontal areas 46 and 12. Area 12 is implicated in high level non-spatial processing including encoding of objects in working memory, suggesting that AIP may be influenced by visual memory of object features (Borra et al., 2008). AIP also contains other neurons that fire in conjunction with motor plans in addition to or instead of visual input (Sakata et al., 1997; Murata et al., 2000; Taira et al., 2000). Interestingly, AIP also receives subcortical input (via the thalamus) from both the cerebellum and basal ganglia (Clower et al., 2005). Finally, AIP receives input from the inferotemporal cortex (IT), which is likely to provide additional visual information about shapes. Our present focus however is the visual input from CIP.

The main goal of this study is to model the neural spike code of object-selective visual-dominant AIP neurons. In particular, we wanted to know whether there are certain sets of shape parameters that are consistent with the responses of visual AIP neurons, and which can furthermore be estimated in a physiologically plausible way from the information available in CIP.

We therefore compared two ways of parameterizing shapes. First we considered the superquadric family of shapes, a continuum that includes cuboids, ellipsoids, spheres, octahedra, and cylinders, and which can also be extended in various ways to model more complex shapes (Solina and Bajcsy, 1990). We considered superquadrics because they play a role in robotic grasp control (Duncan et al., 2013) that seems to be similar to the role of AIP in primate grasp control, i.e., they represent shapes compactly as a basis for grasp planning. We also considered an alternative shape parameterization that is based on non-linear dimension reduction of the depth field. In particular, we used an Isomap (Tenenbaum et al., 2000). We considered Isomap parameters partly because they are continuous, i.e., similar shapes have similar parameters. This is consistent with datasets in which similar 3D stimuli elicit similar spike rate patterns in AIP (Theys et al., 2012, Figure 10; Srivastava et al., 2009, Figure 11C).

This study is one of the first to model the mapping from CIP to AIP. Oztop et al. (2006) modeled AIP as a hidden layer in a multilayer perceptron network that mapped visual depth onto hand configuration. The output layer of this model (corresponding to F5) was a self-organizing map of subnetworks that corresponded to different hand configurations. Prevete et al. (2011) developed a mixed neural and Gaussian-mixture model in which AIP received monocular infero-temporal input. This model did not include stereoscopic input from CIP. The FARS grasping model (Fagg and Arbib, 1998) did not address in detail how AIP activity arises from visual input. While past AIP models have been relatively abstract, here our goal is to fit published tuning curves from AIP recordings, and furthermore to do so using depth-related input from a model of CIP. As far as we are aware, there have not been previous attempts to model AIP tuning in terms of either superquadric parameters or non-linear dimension reduction of depth features.

# **2. MATERIALS AND METHODS**

This study consists of three main parts. The first is a model of tuning for depth features in the caudal intraparietal area (CIP, see Section 2.1.1). The second is a model of tuning for threedimensional shape features in the anterior intraparietal area (AIP, see Section 2.1.2). Finally, the third is an investigation of physiologically plausible feedforward mappings between CIP and AIP (see Section 2.5).

#### **2.1. COSINE-TUNING MODELS OF NEUROPHYSIOLOGICAL DATA**

We tested how well various tuning curves from the CIP and AIP electrophysiology literature could be approximated by cosine-tuned neuron models. In particular, given a vector *x* of stimulus variables, we modeled the net current, *I*, driving spiking activity in each neuron as

$$I = \tilde{\phi}^T \mathbf{x} + b,\tag{1}$$

where *b* is a bias term and φ˜ is parallel to the neuron's preferred direction in the space of stimulus parameters. Longer φ˜ corresponds to higher sensitivity of the neuron to variations along its preferred direction.

We used a normalized version of the leaky-integrate-and-fire (LIF) spiking model. In this model, the membrane potential *V* has subthreshold dynamics τ*RCV*˙ = −*V* + *I*, where τ*RC* is the membrane time constant and *I* is the driving current. The neuron spikes when *V* >= 1, after which *V* is held at 0 for a post-spike refractory time τ*ref* before subthreshold integration begins again. These neurons have spike rate

$$r = \frac{1}{\mathfrak{r}\_{ref} - \mathfrak{r}\_{RC} \cdot \ln\left(1 - \frac{1}{T}\right)}.\tag{2}$$

Except where noted, τ*RC* was included among the optimization parameters and constrained to the range [0.02*s*, 0.2*s*]. In some cases (where noted), when the basic cosine-LIF model (above) produced poor fits, we also added Gaussian background noise to *I*. Such background noise more realistically reflects the input to neurons *in vivo* (Carandini, 2004) and causes the LIF model to emit more realistic, irregular spike trains. It also has the potential to produce better tuning curve fits. The reason is that depending on the amplitude of the noise, the spike-rate function may be compressive [as in Equation (2)], sigmoidal, or nearly linear. In these cases we fixed τ*ref* = 0.005*s* and τ*RC* = 0.02*s*, included the noise variance as an optimization parameter, and interpolated the spike rate from a lookup table based on simulations. Given a tuning curve from the electrophysiology literature and a list of hypothesized tuning variables, we found least-squares optimal parameters φ˜ and *b* mainly, and either τ*RC* or σ*noise* (as noted in the corresponding sections), using Matlab's *lsqcurvefit* function. This function uses Matlab's trust-region-reflective algorithm, which is based partly on Coleman and Li (1994), to solve a non-linear curve-fitting problem in the sense of least-squares. We retried each optimization with at least 1000 random initial points in order to increase the probability of finding a global optimum.

We preferred cosine tuning models over more complex nonlinear models for a number of reasons, including that they are simple and that cosine tuning is widespread in the cortex and elsewhere (Zhang and Sejnowski, 1999). (See more detailed rationale in the Discussion).

#### *2.1.1. CIP Tuning*

We approximated CIP responses in terms of depth and its first and second spatial derivatives. CIP has been proposed to encode these variables (Orban et al., 2006), and they have been the basis for several experimental studies of CIP responses (Sakata et al., 1998; Taira et al., 2000; Tsutsui et al., 2001; Katsuyama et al., 2010; Rosenberg et al., 2013).

We fit cosine-tuned LIF neuron models to tuning curves from Tsutsui et al. (2002) and Rosenberg et al. (2013), and from Katsuyama et al. (2010), in which the stimuli varied in terms of first and second derivatives of depth, respectively. The stimuli in Katsuyama et al. (2010) consisted of curved surfaces with depth

$$z = \frac{1}{2} \left( K\_1 x^2 + K\_2 y^2 \right). \tag{3}$$

*K*1 and *K*2 were varied to produce two levels of "curvedness,"

$$C = \sqrt{\frac{K\_{\text{max}}^2 + K\_{\text{min}}^2}{2}}$$

and a range of "shape indices"

$$SI = \frac{2}{\pi} \arctan \frac{K\_{\text{max}} + K\_{\text{min}}}{K\_{\text{max}} - K\_{\text{min}}},$$

where *Kmax* and *Kmin* are the larger and smaller curvatures along the *x* and *y* axes, respectively.

In terms of the depth *z*, the principal curvature along the *x* axis is

$$K\_{\mathbf{x}} = \frac{\partial^2 \mathbf{z} / \partial \mathbf{x}^2}{\left(1 + (\partial \mathbf{z} / \partial \mathbf{x})^2\right)^{3/2}} \tag{4}$$

(de Vries et al., 1993). For these stimuli ∂*z*/∂*x* = 0 at the center, and so *Kx* = ∂2*z*/∂*x*2.

#### *2.1.2. AIP Tuning*

Following Sakata et al. (1998) and Murata et al. (2000) and consistent with the role of AIP in grasping (Fagg and Arbib, 1998), we took the visual-dominant neurons in AIP to be responsive to three-dimensional shape. Available tuning curves (e.g., Murata et al., 2000) span small numbers of data points relative to the large space of shape variations that are relevant to hand pre-shaping. For this reason we fit models to various "augmented" tuning curves that matched published tuning curves for some shapes, and made assumptions about how these neurons might respond to other shapes (see **Figure 2**). These assumptions were based on additional data for separate AIP neurons (see below). Our augmented tuning curves spanned four of the shapes in Murata et al. (2000), specifically a sphere, cylinder, cube, and plate. Two other shapes (ring and cone) were omitted for simplicity, because they require additional superquadric shape parameters (see Section 2.2). The augmented tuning curves spanned four sizes and four orientations for each of the four shapes. Due to symmetries in the shapes, there were a total of 36 points in these tuning curves (see **Figure 1**). Four of these points corresponded to AIP data, and the rest (the augmented points) were extrapolated from the data.

We based the augmented points on additional data from other AIP neurons, including aggregate data. Murata et al. (2000) provide shape-tuning curves for six different object-type visualdominant AIP neurons. We tested different augmented versions of these curves with various combinations of size and orientation

**FIGURE 1 | The complete set of 36 shapes used in the augmented tuning curves.** Four basic shapes (sphere, cube, plate, and cylinder) were adapted from Murata et al. (2000). In order to constrain the models more fully, and in particular to ensure that tuning curves included more points than there were parameters in our models, we augmented these basic shapes by adding copies with different sizes (shown with 4 different colors) and orientations (i.e., horizontal, vertical, tilted forward 45◦, tilted backward 45◦). Note that due to the symmetry of the basic shapes, some orientations are redundant (e.g., rotating a sphere does not create a distinguishable shape).

tuning (see **Figure 2**). Murata et al. (2000) reported (without plotting shape tuning for these neurons) that most object-type neurons were orientation selective, and that 16/26 were sizeselective. Therefore, we created two augmented tuning curves for each of the six shape-tuning curves. Both were orientationselective; one was size-selective and the other was size-invariant. For the size-selective tuning curves we assumed that spike rate increased monotonically with size (consistent with Murata et al., 2000, Figure 19; note that preference for intermediate sizes was reported only for motor-dominant neurons). We assumed that orientation tuning was roughly Gaussian and fairly narrow (consistent with Murata et al., 2000, Figure 18). Some AIP neurons are orientation selective with only mild selectivity across various elongated shapes (Sakata et al., 1998). Therefore, we created a final augmented tuning curve that was orientation selective but responded equally to cylinders and plates. **Figure 1** shows an example of an augmented tuning curve and its relationship to the data. This procedure made the tuning curve optimization more challenging. This was important because even our simple cosine-tuned neuron models had more parameters than the number of points in the published tuning curves (see Section 3). It also allowed us to make use of additional AIP data.

#### **2.2. SUPERQUADRICS**

We modeled AIP shape tuning both on the parameters of the superquadric family of shapes, and on an Isomap dimension reduction of depth features. The superquadric family is a continuum that includes cuboids, ellipsoids, spheres, octahedra, and cylinders as examples. Superquadrics are often used to approximate observed shapes as an intermediate step in robotic grasp control (Ikeuchi and Hebert, 1996; Biegelbauer and Vincze, 2007; Goldfeder et al., 2007; Huebner et al., 2008; Duncan et al., 2013). In this context, superquadric shape parameters are typically estimated from 3D point-cloud data using iterative non-linear optimization methods (Huebner et al., 2008).

Their role in robotics suggests that superquadrics are a plausible model of AIP shape tuning. Specifically, they can be parameterized from visual information and they contain

**FIGURE 2 | An example of an augmented AIP tuning curve. (A)** Tuning curve adapted from Murata et al. (2000), Selectivity for the shape, size, and orientation of objects for grasping in neurons of monkey parietal area AIP, 2580-2601, with permission. (See their Figure 11.) **(B)** The four points from the same tuning curve that belong to the basic superquadric family (a ring and cone are excluded from the current study). The spike rates are

plotted as 3D bars. **(C)** An augmented tuning curve that includes the points in **(B)**, as well as other rotations and scales. This augmented tuning curve is both size-tuned and orientation-tuned, as were the majority of object-type visual neurons in Murata et al. (2000). Another large minority were orientation-tuned but not size-tuned. As in **Figure 1**, the colors correspond to different sizes.

information about an object that is useful as a basis for grasp planning. One goal of the present study was to examine their physiological plausibility more closely, by fitting superquadrictuned neuron models to AIP tuning curves. The surface of a superquadric shape is defined in *x*-*y*-*z* space as

$$
\left(\frac{x}{A\_1}\right)^{1/\epsilon\_1} + \left(\frac{y}{A\_2}\right)^{1/\epsilon\_2} + \left(\frac{z}{A\_3}\right)^{1/\epsilon\_3} = 0,
$$

where *A* > 0 are scale parameters and > 0 are curvature parameters. Values of close to zero correspond to squared corners, while values close to one correspond to rounded corners. For example a sphere has *A*1 = *A*2 = *A*3 and 1 = 2 = 3 = 1. We also used another parameter, θ, that described the orientation of the superquadric. θ was composed of three angles, one per coordinate. The rotation of the superquadric is done applying the rotation matrix described in Equation 5.

We generated a database of 40,000 shapes that included spheres, cylinders, plates, and cubes as well as variations on these shapes with different scales in each dimension, and rotated versions of them. Our database contained roughly equal numbers of boxlike, sphere-like, and cylinder-like shapes. For round edges we set = 1. For squared edges we drew from an exponential distribution that was shifted slightly away from zero, *p* = 10*H*( − η) exp (−( − η)/0.1) with η = 0.01, where *H* is the Heaviside step function. The shift away from 0 (perfectly sharp corners) helped to avoid numerical problems. The objects had widths between 0.02 m and 0.12 m. We also allowed arbitrary rotations in three dimensions (except where symmetry made rotations redundant), so that each shape had a total of nine parameters.

This study considers only the basic superquadric family, which does not include all the shapes for which AIP responses have been reported. However, the basic family can also be extended in various ways to deal with more complex shapes. For exam-

$$R(\theta\_1, \theta\_2, \theta\_3) = \begin{bmatrix} \cos(\theta\_2) \cdot \cos(\theta\_3) & \cos(\theta\_1) \cdot \sin(\theta\_3) + \sin(\theta\_1) \cdot \sin(\theta\_2) \cdot \cos(\theta\_3) \cdot \sin(\theta\_1) \cdot \sin(\theta\_3) - \cos(\theta\_1) \cdot \sin(\theta\_2) \cdot \cos(\theta\_3) \\\ -\cos(\theta\_2) \cdot \sin(\theta\_3) \cdot \cos(\theta\_1) \cdot \cos(\theta\_3) - \sin(\theta\_1) \cdot \sin(\theta\_2) \cdot \sin(\theta\_3) \cdot \sin(\theta\_1) \cdot \cos(\theta\_3) + \cos(\theta\_1) \cdot \sin(\theta\_2) \cdot \sin(\theta\_3) \\\ \sin(\theta\_2) & -\sin(\theta\_1) \cdot \cos(\theta\_2) \end{bmatrix} \tag{5}$$

ple, hyperquadrics introduce asymmetry (Kumar et al., 1995), and trees of superquadrics can be used to approximate complex shapes with arbitrary precision (Goldfeder et al., 2007).

#### **2.3. CREATION OF DEPTH MAPS**

CIP receives input from V3 and V3A, which encode binocular disparity information (Anzai et al., 2011). Disparity is monotonically related to visual depth, or distance from observer to surface. As a simplified model of this input we created depth maps, i.e., grids of distances from a viewpoint to object surfaces. We created depth maps from the shapes in our superquadric database by finding intersections of the surfaces with rays at various visual angles from the view point. We used a 16 × 16 grid of visual angles. Grid spacing was closer near the center than in the periphery, in order to reflect higher visual acuity near the fovea and also to ensure that a few rays intersected with the smallest shapes (specifically, distances from the center were *a*1.5, where *a* were evenly-spaced points). The grid covered ± 10◦ of visual angle in each direction. The object centers were at a depth of 0.75 m from the viewpoint. Depth at each grid point was found as the intersection of the superquadric surface with a line from the observation point (**Figure 3**).

#### **2.4. ISOMAP SHAPE PARAMETERS**

Within the superquadric family there is typically more than one set of parameters that can describe a given shape. For example, a tall box can either be parameterized as a tall box or a wide box on its end. This is not very problematic in robotics, because an iterative search for matching parameters finds one of these solutions. However, our goal was to model a feedforward mapping from depth (V3A) to shape parameters (AIP). In order to use the superquadric parameters as the basis for an AIP tuning we therefore needed the superquadric-to-depth function to be invertible.

superquadric was centered at (0, 0, 0.75) relative to an observer at (0, 0, 0). Rays were traced between the observation point and a grid of points in the frontoparallel plane at *z* = 0.75, and intersections (red dots) were found with the superquadric surface. The depth map consisted of a grid of distances from (0, 0, 0) to these intersections.

We achieved this by restricting the ranges of angles. For example, for box-like shapes we restricted all angles to within ±π/4. This resulted in a unique set of superquadric parameters for each shape. However, large discontinuities remained, in that some very similar shapes sometimes had very different parameters. For example, a tall box at an angle slightly less than π/4 has a depth map that is very similar to a wide box at angle just greater than −π/4 radians. Similar discontinuities seem to exist regardless of the angle convention. We anticipated that these discontinuities would impair feedforward mapping in a neural network, so we also explored an alternative low-dimensional shape parameterization.

In the alternative model, neurons were tuned to an Isomap (Tenenbaum et al., 2000) derived from depth data. Isomap is a non-linear dimension-reduction method in which samples are embedded in a lower-dimensional space in such a way that geodesic distances (i.e., distances along the shortest paths through edges between neighboring points) are maintained as well as possible. This method ensured that similar depth maps would be close together in the shape-parameter space, minimizing parameter discontinuities like those of the superquadric parameters. We constructed an Isomap of the first and second spatial derivatives of the depth maps in the horizontal and vertical directions.

We tested whether our augmented AIP tuning curves (above) were consistent with cosine tuning for these shape parameters. We also tested how well these shape parameters could be approximated by a neural network with CIP parameters as input.

#### **2.5. NEURAL NETWORK MODELS OF CIP-TO-AIP MAP**

In addition to fitting cosine-LIF models to neural tuning curves in CIP and AIP, we also developed feedforward networks to map from CIP variables to AIP variables. Our general approach was to decode shape parameters from the spike rates of CIP models.

We experimented with several different networks including neural engineering framework networks (Eliasmith and Anderson, 2003; Eliasmith et al., 2012), multilayer perceptrons trained with the back-propagation algorithm (Haykin, 1999) and convolutional networks (LeCun et al., 1998).

In each case the output units were linear. Linear decoding of the tuning parameters was of interest because decoding weights can be multiplied with preferred directions to give synaptic weights for any cosine tuning curve over the decoded variables (Eliasmith and Anderson, 2003). Specifically, suppose we have presynaptic rates **r***pre* and linearly decoded estimates **p**ˆ = **r***pre* of shape parameters **p**, where is a matrix of decoding weights. In this case the family of cosine tuning curves over **p**ˆ is

$$r\_{\text{post}} = G\left(\tilde{\phi}^T \hat{\mathbf{p}} + b\right),\tag{6}$$

where φ˜*T***p**ˆ + *b* is the driving current, φ˜ is the neuron's preferred direction, *G* is a physiological model of the current-spike rate relationship, and *b* is a bias current. Such a tuning curve can then be obtained with synaptic weights (from all *presynaptic* neurons to a single *postsynaptic* neuron)

$$\mathbf{w}^T = \tilde{\phi}^T \boldsymbol{\Phi}.\tag{7}$$

This allows us to draw general conclusions about how well our various models can account for AIP tuning, and how they would relate to future data.

Equations 6 and 7 are important components of the Neural Engineering Framework (Eliasmith and Anderson, 2003; Eliasmith et al., 2012), a method of developing large-scale neural circuit models.

# **3. RESULTS**

#### **3.1. CIP TUNING**

**Figure 4** shows an optimal fit of a cosine-tuned LIF model to a tuning curve from Katsuyama et al. (2010). Following their convention the spike rates are shown as a function of shape index, separately for the two curvedness levels. Inspection of the tuning curve revealed that it contained an expansive non-linearity, so we included Gaussian background noise in the model (as described in Section 2). To improve the fit further, in addition to tuning variables *X* = ∂2*z*/∂*x*2 and *Y* = ∂2*z*/∂*y*2 we introduced new tuning variables 1 2 - 3(*X*)2 − 1 and 1 2 - 3(*Y*)2 − 1 . The rationale for their inclusion was that these are the non-linear functions for which linear reconstruction is (with reasonable assumptions) most accurate from populations of LIF neurons tuned to *X* and *Y* (Eliasmith and Anderson, 2003). However, the fit to the Katsuyama et al. (2010) data remained poor despite these measures.

We considered whether a linear-nonlinear receptive field model with depth inputs might produce a better fit. Such models are essentially cosine tuning models with multiple input variables on a grid. However, the depth stimuli in this case (see Equation 3) consisted of linear combinations of *x*2 and *y*2, so any receptivefield model over the depth field has an equivalent cosine tuning model over *K*1 and *K*2. Therefore, the neuron is not cosine tuned to either depth or the curvature parameters.

**Figure 5** shows an example of a more complex non-linear neuron model that fits the data. This model is based on non-linear interactions between nearby inputs on the same dendrite, which suggest that pyramidal cells may function similarly to multilayer perceptrons (Polsky et al., 2004). The input to this model was a 3 × 3 depth grid. The model contained 50 dendritic branches, each of which was cosine tuned to the depths. The linear kernels (analogous to preferred directions) were random. The output of each branch was a sigmoid function of the point-wise product of the depth stimulus and the linear kernel. The spike rate was a least-squares optimal weighted sum of the branch outputs. This was found using a matrix pseudoinverse that used 14 singular values. We also created another version of this model (not shown) in which the tuning curve was augmented with additional stimuli (completing the outer circle of points in **Figure 5B**) and it was assumed that the neuron would respond to these stimuli at the background spike rate. This version of the model therefore fit 26 points, and we used 20 singular values in the pseudoinverse. The fit was similar in this case.

We also constructed another alternative model of this cell that was based on a more detailed model of V3A activity. Specifically, instead of a 3 × 3 depth grid, this model received input from seven non-linear functions of depth at each point. Five of these were Gaussian functions based on "tuned near," "tuned zero," and "tuned far" neurons (Poggio et al., 1988). Two were sigmoidal functions based on "near" and "far" tuning (Poggio et al., 1988). This model (not shown) reproduced the tuning curve somewhat less accurately than the non-linear cell model above. This was the case regardless of minor variations in the set of input tuning functions and their parameters.

**Figure 6** shows a cosine-tuning fit of data from Tsutsui et al. (2002). This tuning curve is an average over multiple cells that were tuned to depth gradients of visual stimuli. The best fitting cosine-tuning model has a notably different shape than the aggregate data. In particular, the actual spike rates are fairly constant far away from the preferred stimulus, while the model spike rates continue to decrease farther from the preferred stimulus.

depth, and horizontal and vertical second spatial derivatives of depth. The stimuli in Katsuyama et al. (2010) varied only in terms of the second derivatives. We also added non-linear tuning functions to improve the fit (see text). The left and right tuning curves are for two different levels of curvedness.

Rosenberg et al. (2013) provide several additional CIP tuning curves over 49 different plane stimuli. Some of these tuning curves are clearly not consistent with cosine tuning for first derivatives of depth or disparity, e.g., with multimodal responses to surface tilt. We fit the non-linear model of **Figure 5** to seven of these tuning curves (their Figures 4, 5B). Using 20 singular values, the correlations between data and our best model fits were *r* = 0.98 ± 0.01 *SD* for the four tuning curves in their Figure 4, and *r* = 0.78 ± 0.09 *SD* for the three tuning curves in their Figure 5B. (These fits are somewhat closer than fits reported by Rosenberg et al. to Bingham functions, which is unsurprising as our model has more parameters.) Using 40 singular values, our correlations improved to *r* = 0.91 ± 0.02 *SD* for the tuning curves in their Figure 5B.

In summary, the spike rates of these CIP neurons varied with the first and second spatial derivatives of depth, but not in a way that is consistent with cosine tuning to either the depth map, its first and second derivatives, or low-order polynomial functions of these derivatives. Other models, which are physiologically plausible but more complex, fit the data more closely.

#### **3.2. AIP TUNING**

**Figure 7** shows an example cosine-tuning fit of an augmented tuning curve in superquadric space. This fit is based on a noisefree LIF neuron. For this dataset the shapes were rotated only in one dimension, so we avoided angle discontinuities by using a 2D direction vector in place of the angle. The optimized parameters were the 8-dimensional preferred direction vector φ˜, the bias *b*, and the membrane time constant τ*RC*. Across the 36 points in the augmented tuning curve, the spike rate error (difference between augmented and model spike rates) was 0.70 ± 1.57 (mean ± *SD*).

**Figure 8** shows the means and standard deviations of spikerate errors for each of the augmented tuning curves. Good fits were obtained for some of the neurons (#1 and #3 in Murata et al., 2000, Figure 10, and the second in Figure 11, which we label #5). This was true for both size-invariant and size-selective augmented tuning curves. Neuron #1 had low spike rates for the stimuli that we studied. Neurons #3 was highly selective for cylinders, and #5 was more broadly tuned but also preferred cylinders. The worst fits were obtained for neuron #6 which responded strongly to plates and cylinders but not to cubes or spheres.

**Figure 9A** shows the means and standard deviations of spike-rate errors for each of the augmented tuning curves in an 8-dimensional Isomap space. We plot the results for the 8-dimensional Isomap in order to match the number of superquadric parameters. The cosine tuning errors (−0.88 ± 10.68 spikes/s; mean ± *SD*) were larger than those in the superquadric space (−0.53 ± 6.75 spikes/s). The difference between these variances was significant according to Levene's test [*W*(1, 910) = 41.3; *p* < 0.001].

**Figure 9B** shows how the error declined with higherdimensional Isomaps. Error variance with the 16-dimensional

Isomap (−1.77 ± 6.35) was not significantly different from that of the 8-parameter superquadric [Levene's Test; *W*(1, 910) = 1.83; *p* = 0.18]. (Recalculating the variances around 0 instead of −1.77 and −0.89 did not make the difference significant; *p* = 0.058). The cosine-tuning fits were excellent in the 32-dimensional Isomap space, with significantly lower variance [−0.17 ± 1.29 spikes/s; *W*(1, 910) = 316.2; *p* < 0.001]. This higher-dimensional shape representation is therefore consistent with the data and with the augmented tuning curves.

# **3.3. MAPPING FROM CIP TO AIP**

We trained multi-layer perceptrons in order to understand whether the superquadric or Isomap models of AIP were more consistent with mapping from CIP input. Because CIP neurons are sensitive to depth and to first and second spatial derivatives of depth, we used these as inputs to the networks. Specifically the inputs consisted of 16 × 16 depth maps, their 16 × 16 horizontal and vertical derivatives, and their 16 × 16 horizontal and vertical second derivatives. The derivatives were approximated by convolving with 3 × 3 kernels (e.g., [ 111 ] *T*[ 1 0 −1 ] and [ 111 ] *T*[ 0.5 −1 0.5 ]). The total number of inputs was therefore 16 × 16 × 5 = 1280. The hidden layers had logistic activation functions. The weights and biases were trained with the backpropagation algorithm in Matlab's Neural Network Toolbox. The output layer had a linear activation function in order to model the input to cosine-tuned neurons, as described in the Methods. A dataset of 40000 rotated superquadric objects was generated, from which depth and curvature images were derived. This dataset was divided into 28000 objects for training the network and 12000 objects to validate the results obtained in the training.

**Figure 10** shows results from networks with two hidden layers, the first with 600 units and the second with 300 units. The scatter plots show the network's output vs. the actual values of the validation dataset. In **Figure 10A** is the network's result for the superquadric shape parameter 1. The other scatterplots in **Figures 10C,E** illustrate the network's approximation of the scale and orientation parameters *A*1 and θ1. Approximation of the other six parameters was similar (e.g., the scatterplots for 2 and 3 resemble that for 1). The scatterplots **Figures 10B,D,F** illustrate the network's approximation of Isomap parameters. The first, fourth, and seventh dimensions are shown as illustrative examples.

Approximation of the Isomap parameters was much more accurate than approximation of the superquadric parameters. This outcome was very consistent across a variety of networks of different sizes, with one or two hidden layers, with pre-training of hidden layers as autoencoders, etc. We also experimented with networks that contained a hidden layer of LIF neurons with random preferred directions over various local kernels, and optimal linear estimates of the shape parameters from the hidden-layer activity (Eliasmith and Anderson, 2003). The results were also similar in this case, although (as expected) more neurons were required to achieve performance like that of the more fullyoptimized multilayer perceptrons.

**Figure 11** compares the distribution of the network's Isomap approximation errors with the distribution of pairwise distances between shape examples in our database. The errors were much smaller than typical distances between examples.

We also experimented with a wide variety of larger networks, including convolutional networks, using the cuda-convnet package (Krizhevsky et al., 2012). These networks did not substantially outperform the multilayer perceptron of **Figures 10**, **11** (lowest mean Euclidean error 0.066 as opposed to 0.081 in **Figure 11**). We also trained some convolutional networks with only the depth map as input, and with a 3 × 3 kernel in the first convolutional layer. Interestingly, some of the resulting kernels resembled the kernels that we created manually to approximate the first and second derivatives.

# **4. DISCUSSION**

This study examined the neural code for three-dimensional shape in visual-dominant AIP neurons. AIP is critical for hand preshaping in grasping, and these neurons encode properties that are relevant to grasping including object shape, size, and orientation.

Our motivation for testing superquadric parameters as a model of AIP tuning was that superquadrics have been used in robotics, in a role that we take to be similar to the role of AIP in the primate brain. Specifically, they have been used as compact approximate representations of point clouds on which to base grasp planning. Such a representation is useful because it allows generalization from training examples to unseen examples, e.g., by interpolating between known solutions for known sets of parameters. An alternative approach in robotics is to cluster point clouds into discrete shape categories (Detry et al., 2013). We see the Isomap as an intermediate approach with some of the advantages of both superquadric fitting and clustering. The Isomap is data-driven and adapts to the statistics of the environment (like clustering), but its parameters make up a low-dimensional and continuous space (like those of superquadrics). Furthermore, unlike the superquadric representation, the Isomap representation does not have large discontinuities between very similar shapes.

We found that cosine tuning on a 32-dimensional Isomap accounted well for the tuning curves of object-selective AIP neurons. We also found that, in contrast with superquadric parameters, the Isomap parameters could be approximated fairly well by various neural networks with CIP-like input.

### **4.1. AUGMENTED TUNING CURVES**

Available AIP data includes the responses of individual neurons to only a few different shapes, in fact fewer shapes than there are parameters in even the simplest superquadric model. To more vigorously test the different shape parameterizations as a basis for plausible neural tuning, and to incorporate additional aggregate information on shape tuning (e.g., the fact that most visual-dominant AIP neurons are orientation selective), we created "augmented" tuning curves that included both data and extrapolations of the data. It is likely that some of these augmented tuning curves were unrealistic. While the general trends in our AIP fitting results are informative (e.g., that Isomap fits improve and outperform superquadrics as dimensions increase), the details depend on our augmentation assumptions. For example, we found that the Isomap error declined more rapidly when we excluded orientation-selective/shape-invariant tuning curves from the analysis. This limitation does not affect interpretation of our other main result, i.e., that superquadrics were poorly approximated by feedforward neural networks while Isomaps were well approximated.

Future modeling would be facilitated by tuning curves with greater numbers of data points. For example, the dataset in Lehky et al. (2011) includes responses of 674 inferotemporal neurons to a common set of 806 images. A relatively extensive AIP dataset was recently collected (Schaffelhofer and Scherberger, 2014), but no tuning curves from this dataset have yet been published.

# **4.2. COSINE TUNING**

We were primarily interested in cosine-tuning models for several reasons, not least because cosine tuning is widespread in the brain (see many examples in Zhang and Sejnowski, 1999). Linear-nonlinear receptive field models of the early visual system

are another kind of cosine tuning, with multiple tuning variables on a 2D grid. Furthermore, a practical advantage of cosine tuning models is that they require only *n* + 1 tuning parameters for *n* stimulus variables (in contrast a full *n*-dimensional Gaussian tuning curve has *n* + *n*2 parameters). This is important because published tuning curves in CIP and AIP consist of relatively few points, so models with large numbers of parameters may be underconstrained. Cosine tuning is also physiologically realistic in that it can arise from linear synaptic integration. For example, if a matrix *W* of synaptic weights has *n* large singular values, then

the post-synaptic neurons are tuned to a *n*-dimensional space (if *W* = *UVT* then the preferred directions are in the first *n* columns of *U*). Cosine tuning curves are also optimal for linear decoding (Salinas and Abbott, 1994). There are also many neurons that do not appear to be cosine tuned, for example speed-tuned neurons in the middle temporal area (Nover et al., 2005). However, where applicable, cosine tuning models provide rich insight into neural activity. We therefore attempted to fit such models to the data where possible. Many AIP tuning curves over similar stimuli with different curvatures vary smoothly and monotonically (Srivastava et al., 2009), consistent with cosine tuning.

Cosine tuning to modest numbers of Isomap parameters (relative to the 256-element depth maps on which they were based) accounted for the AIP data and for our augmented AIP tuning curves.

In contrast, we concluded that the CIP neurons we modeled were not cosine tuned to the stimulus variables with which they have been examined. CIP has been proposed to encode first and second derivatives of depth (Orban et al., 2006). Various neurons in CIP respond to disparity gradient (Shikata et al., 1996; Sakata et al., 1998), texture gradient (Tsutsui et al., 2001), and/or perspective cues for oriented surfaces (Tsutsui et al., 2001). (Accordingly, visual-dominant AIP neurons also respond to monocular visual cues as well as disparity cues, and respond most strongly when disparity and other depth cues are congruent (Romero et al., 2013). Sakata et al. (1998) describe various neurons in CIP as axis-orientation-selective and surface-orientationselective. The former were sensitive to the orientation of a long cylinder, consistent with two-dimensional tuning for horizontal and vertical curvature. The latter were selective for the orientation of a flat plate, consistent with two-dimensional tuning for depth gradient. Furthermore, Sakata et al. (1998) also recorded a neuron that preferred a cylinder of certain diameter which was tilted back and to the right, but did not respond strongly to a square column of similar dimensions. This suggests selectivity for both first and second derivatives within the same neuron. Katsuyama et al. (2010) recorded CIP responses to curved surfaces that varied in terms of their second derivatives. Tuning to the first and second derivatives of depth is physiologically plausible in that these quantities are linear functions of the depth field, which is available from V3A. We therefore attempted to fit models that were cosine tuned over these variables, but we obtained poor fits.

While CIP neurons are certainly responsive to these variables (and more complex non-linear models of tuning to these variables fit the data closely) it is possible that there are other related variables that provide a more elegant account of these neurons' responses. Notably, some CIP neurons prefer intermediate cylinder diameters (Sakata et al., 1998), whereas cosine tuning for curvature would be constrained to monotonic changes with respect to curvature. Also, some of the neurons in Rosenberg et al. (2013) are clearly non-cosine-tuned for depth slope.

Some CIP tuning curves (see e.g., **Figure 6**) seem to be fairly similar to rectified cosine functions (Salinas and Abbott, 1994) with a negative offset, except that their baseline rates are not zero. In general, spike sorting limitations, which cannot be completely avoided in extracellular recordings (Harris et al., 2000), are a potential source of uncertainty in tuning curves. However, if misclassification rates had been substantial then multi-peaked tuning curves might have been expected, and none were reported in these studies.

### **4.3. RELATIONSHIP TO SHAPE REPRESENTATION IN IT**

Area IT has been shown to represent medial axes and surfaces of objects (Yamane et al., 2008; Hung et al., 2012). AIP has significant connections with IT areas including the lower bank of the superior temporal sulcus (STS), specifically areas TEa and TEm (Borra et al., 2008). These areas partially correspond to functional area TEs, which encodes curvature of depth (Janssen et al., 2000) similarly to CIP. However, AIP responds to depth differences much earlier than TEs (Srivastava et al., 2009). It is possible that a shape representation in IT, with some similarities to that in CIP, provides longer latency reinforcement and/or correction of shape representation in AIP.

# **4.4. FUTURE WORK**

A key direction for future work is to test how well the Isomap shape representation works for robotic grasp planning. This would provide important information about the functional plausibility of this representation. For example, if Isomap-based shape parameters cannot be used to shape a hand for effective grasping, this will strongly suggest that there are critical differences between AIP tuning parameters and Isomap parameters. On the other hand, if the Isomap representation performs well, it may suggest a new biologically-inspired approach for robotic grasping.

An apparent advantage of the Isomap approach is that it is data-driven and makes no prior assumptions about shapes. It would be informative to build Isomaps for less idealized shapes that monkeys might grasp in nature.

Other non-linear dimension-reduction methods (e.g., Yan et al., 2007) could also be compared with the Isomap in terms of fitting AIP data and providing an effective basis for grasp planning. We would expect differences relative to Isomap tuning to be subtle relative to available AIP data, but perhaps distinct advantages would appear in a grasp control system. One interesting possibility would be to emphasize features that are related to reward or performance (Bar-Gad et al., 2003).

Another important direction for future work is to extend the model to include motor-dominant AIP neurons and to F5 neurons as in e.g., Theys et al. (2012, 2013) and Raos et al. (2006).

Finally, our models produced constant spike rates in response to static inputs. A more sophisticated future model would account for response timing and dynamics (Sakaguchi et al., 2010). The Neural Engineering Framework (Eliasmith and Anderson, 2003) provides a principled approach to modeling dynamics in systems of spiking neurons.

# **ACKNOWLEDGMENTS**

Financial support was provided by CrossWing Inc.; NSERC and Mitacs (Canada); DAAD-NRF (South Africa); and the Spanish Ministry of Education and Consejo Social UPM (Spain). We thank Paul Calamai, Renaud Detry, and André Nel for helpful discussions.

### **REFERENCES**

the lateral intraparietal area links the area V3A and the anterior intraparietal area in macaques. *J. Neurosci.* 21, 8174–8187.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 16 May 2014; accepted: 26 September 2014; published online: 27 October 2014.*

*Citation: Rezai O, Kleinhans A, Matallanas E, Selby B and Tripp BP (2014) Modeling the shape hierarchy for visually guided grasping. Front. Comput. Neurosci. 8:132. doi: 10.3389/fncom.2014.00132*

*This article was submitted to the journal Frontiers in Computational Neuroscience. Copyright © 2014 Rezai, Kleinhans, Matallanas, Selby and Tripp. This is an openaccess article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Feature integration and object representations along the dorsal stream visual hierarchy

# **Carolyn Jeane Perry1,2\* and Mazyar Fallah1,2,3,4**

1 Visual Perception and Attention Laboratory, School of Kinesiology and Health Science, York University, Toronto, ON, Canada

2 Centre for Vision Research, York University, Toronto, ON, Canada

3 Departments of Biology and Psychology, York University, Toronto, ON, Canada

4 Canadian Action and Perception Network, York University, Toronto, ON, Canada

#### **Edited by:**

Antonio J. Rodriguez-Sanchez, University of Innsbruck, Austria

#### **Reviewed by:**

Da-Hui Wang, Beijing Normal University, China Christopher Pack, McGill University, Canada

#### **\*Correspondence:**

Carolyn Jeane Perry, Visual Perception and Attention Laboratory, School of Kinesiology and Health Science, York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada e-mail: carolynperry08@ gmail.com