PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized
  Perception with Neural Sensors
Conventional image sensors digitize high-resolution images at fast frame
rates, producing a large amount of data that needs to be transmitted off the
sensor for further processing. This is challenging for perception systems
operating on edge devices, because communication is power inefficient and
induces latency. Fueled by innovations in stacked image sensor fabrication,
emerging sensor-processors offer programmability and minimal processing
capabilities directly on the sensor. We exploit these capabilities by
developing an efficient recurrent neural network architecture, PixelRNN, that
encodes spatio-temporal features on the sensor using purely binary operations.
PixelRNN reduces the amount of data to be transmitted off the sensor by a
factor of 64x compared to conventional systems while offering competitive
accuracy for hand gesture recognition and lip reading tasks. We experimentally
validate PixelRNN using a prototype implementation on the SCAMP-5
sensor-processor platform.
Haley M. So, Laurie Bose, Piotr Dudek, Gordon Wetzstein
PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors

###### Abstract

Conventional image sensors digitize high-resolution images at fast frame rates, producing a large amount of data that needs to be transmitted off the sensor for further processing. This is challenging for perception systems operating on edge devices, because communication is power inefficient and induces latency. Fueled by innovations in stacked image sensor fabrication, emerging sensor-processors offer programmability and minimal processing capabilities directly on the sensor. We exploit these capabilities by developing an efficient recurrent neural network architecture, PixelRNN, that encodes spatio-temporal features on the sensor using purely binary operations. PixelRNN reduces the amount of data to be transmitted off the sensor by a factor of \(64\times\) compared to conventional systems while offering competitive accuracy for hand gesture recognition and lip reading tasks. We experimentally validate PixelRNN using a prototype implementation on the SCAMP-5 sensor-processor platform.

## 1 Introduction

Increasingly, cameras on edge devices are being used for enabling computer vision perception tasks rather than for capturing images that look beautiful to the human eye. Applications include various tasks in virtual and augmented reality displays, wearable computing systems, drones, robotics, and the internet of things, among many others. For such edge devices, low-power operation is crucial, making it challenging to deploy large neural network architectures which traditionally leverage modern graphics processing units for inference.

A plethora of approaches have been developed in the "TinyML" community to address these challenges. Broadly speaking, these efforts focus on developing smaller [25] or more efficient network architectures, often by pruning or quantizing larger models [10]. Platforms like TensorFlow Lite Micro enable application developers to deploy their models directly to power-efficient microcontrollers which process data closer to the sensor. Specialized artificial intelligence (AI) accelerators, such as Movidius' Myriad vision processing unit, further reduce the power consumption. While these approaches can optimize the processing component of a perception system, they do not reduce the large amount of digitized sensor data that needs to be transmitted to the processor in the first place, via power-hungry interfaces such as MIPI-CSI, and stored in the memory. This omission is highly significant as data transmission and memory access are among the biggest power sinks in imaging systems [20]. This raises the question of how to design perception systems where sensing, data communication, and processing components are optimized end to end.

Efficient perception systems could be designed such that important task-specific image and video features are encoded directly on the imaging sensor using in-pixel processing, resulting in the sensor's output being significantly reduced to only these sparse features. This form of in-pixel feature encoding mechanism could significantly reduce the required bandwidth, thus reducing power consumption of data communication, memory management, and downstream processing. Event sensors [19] and emerging focal-plane sensor-processors [53] are promising hardware platforms for such perception systems because they can directly extract either temporal information or spatial features, respectively, on the sensor. These features can be transmitted off the sensor using low-power parallel communication interfaces supporting low bandwidths.

Our work is motivated by the limitations of existing feature extraction methods demonstrated on these emerging sensor platforms. Rather than extracting simple temporal gradients [19] or spatial-only features via convolutional neural networks (CNNs) [7, 6], we propose in-pixel recurrent neural networks (RNNs) that efficiently extract spatio-temporal features on sensor-processors for bandwidth-efficient perception systems. RNNs are state-of-the-art network architectures for processing sequences, such as video in computer vision tasks [31]. Inspired by the emerging paradigm of neural sensors [39], our in-pixel RNN frame
work, dubbed PixelRNN, comprises a light-weight in-pixel spatio-temporal feature encoder. This in-pixel network is jointly optimized with a task-specific downstream network. We demonstrate that our architecture outperforms event sensors and CNN-based sensor-processors on perception tasks, including hand gesture recognition and lip reading, while drastically reducing the required bandwidth compared to any traditional sensor based approaches. Moreover, we demonstrate that PixelRNN offers better performance and lower memory requirements than larger RNN architectures in the low-precision settings of in-pixel processing.

Our work's contributions include

* the design and implementation of in-pixel recurrent neural networks for sensor-processors, enabling bandwidth-efficient perception on edge devices;
* the demonstration that our on-sensor spatio-temporal feature encoding maintains high performance while significantly reducing sensor-to-processor communication bandwidth with several tasks, including hand gesture recognition and lip reading;
* the experimental demonstration of the benefits of in-pixel RNNs using a prototype implementation on the SCAMP-5 sensor-processor.

## 2 Related Work

Performing feature extraction on power and memory constrained computing systems requires the union of multiple fields: machine learning, specialized hardware, and network compression techniques.

Machine Learning on the Edge.Edge computing devices are often subject to severe power and memory constraints, leading to various avenues of research and development. On the hardware side, approaches include custom application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other energy efficient AI accelerators. However, this does not address the issue of data transmission from imaging sensors, which is one of the main sources of power consumption [20]. To circumvent the memory constraints, network compression techniques are introduced. They fall into roughly five categories [10]: 1. parameter reduction by pruning redundancy [4, 46, 26, 57]; 2. low-rank parameter factorization [15, 28, 49]; 3. carefully designing structured convolutional filters [16, 48, 52]; 4. creating smaller models [22, 1, 8]; 5. parameter quantization [13, 42, 27, 50, 35, 58, 30]. In this work, we move compute directly onto the sensor and also apply ideas and techniques mentioned above.

Beyond Frame-based Sensing.Event-based cameras have been gaining popularity [19] as the readout is asynchronous and often sparse, triggered by pixel value changes above a certain threshold. However, these sensors are not programmable and do data compression with a simple fixed function. Another emerging class of sensors include focal plane sensor-processors, also known as pixel processor arrays. Along with supporting traditional sensing capabilities, these sensors have a processing element embedded into each pixel. While conventional vision systems have separate hardware for sensing and computing, sensor-processors perform both tasks "in pixel," enabling efficient, low-latency and low-power computation. Recently, sensor-processors with some programmability have emerged [9, 40, 37, 43, 55, 18]. Further advances in 3D fabrication techniques, including wafer-level hybrid bonding and stacked CMOS image sensors, set the stage for rapid development of increasingly more capable programmable sensors.

In-pixel Perception.In the past few years, there has been a surge of advances in neural networks for vision tasks as well as an increasing desire to perform tasks on constrained mobile and wearable computing systems. Sensor-processors are a natural fit for such systems as they can perform sophisticated visual computational tasks at a significantly lower power than traditional hardware. Some early chips [36, 44] were based on implementing convolution kernels in a recurrent dynamical "Cellular Neural Network" model [12, 45]. In 2019, Bose et al. created "A Camera that CNNs" - one of the first works to implement a deep convolutional neural network on the sensor [6]. Since then, there have been a number of other works in CNNs on programmable sensors [47, 14, 51, 21, 7, 32, 34, 33]. These works extract features in the spatial domain, but miss a huge opportunity in failing exploit temporal information. Purely CNN based approaches do not utilize or capitalize on the temporal redundancy or information of the sequence of frames. Our work introduces light-weight extraction of spatio-temporal features, better utilizing the structure of the visual data, all while maintaining low bandwidth and high accuracy.

## 3 In-pixel Recurrent Neural Networks

Emerging sensor-processors with in-pixel processing enable the development of end-to-end-optimized on-sensor and downstream networks off-sensor. In this section, we describe a new on-sensor recurrent spatio-temporal feature encoder that significantly improves upon existing temporal- or spatial-only feature encoders for video processing, as shown in the next section. The proposed pipeline is illustrated in Figure 1.

### In-Pixel CNN-based Feature Encoder

Convolutional neural networks are among the most common network architectures in computer vision. They are
written as

\[\text{CNN}\;\left(\mathbf{x}\right) =\left(\phi_{n-1}\circ\phi_{n-2}\circ\ldots\circ\phi_{0}\right) \left(\mathbf{x}\right),\] \[\phi_{i} =\mathbf{x}_{i} \mapsto\psi\left(\mathbf{w}_{i}*\mathbf{x}_{i}+\mathbf{b}_{i} \right), \tag{1}\]

where \(\mathbf{w}_{i}*\mathbf{x}_{i}:\mathbb{N}^{N_{i}\times M_{i}\times C_{i}} \mapsto\mathbb{N}^{N_{i+1}\times M_{i+1}\times C_{i+1}}\) describes the multi-channel convolution of CNN layer \(i\) and \(\mathbf{b}_{i}\) is a vector containing bias values. Here, the input image \(\mathbf{x}_{i}\) has \(C_{i}\) channels and a resolution of \(N_{i}\times M_{i}\) pixel and the output of layer \(i\) is further processed by the nonlinear activation function \(\psi\).

The SCAMP-5 system used in this work lacks native multiplication operations at the pixel level. Due to this limitation, works storing network weights \(\mathbf{w}_{i}\) in pixel typically restrict themselves to using binary, \(\{-1,1\}\), or ternary \(\{-1,0,1\}\) values. This reduces all multiplications to sums or differences, which are highly efficient native operations.

### In-pixel Spatio-temporal Feature Encoding

Recurrent neural networks (RNNs) are state-of-the-art network architectures for video processing. Whereas a CNN only considers each image in isolation, an RNN extracts spatio-temporal features to process video sequences more effectively. Network architectures for sensor-processors must satisfy two key criteria. First, they should be small and use low-precision weights. Second, they should comprise largely of local operations as the processors embedded within each pixel can only communicate with their direct neighbors (e.g., [9]).

To satisfy these unique constraints, we devise an RNN architecture that combines ideas from convolutional gated recurrent units (GRUs) [2] and minimal gated units [56]. The resulting simple, yet effective PixelRNN architecture, is written as

\[\mathbf{f}_{t} =\psi_{f}\left(\mathbf{w}_{f}*\text{CNN}\left(\mathbf{x}_{t} \right)+\mathbf{u}_{f}*\mathbf{h}_{t-1}\right), \tag{2}\] \[\mathbf{h}_{t} =\mathbf{f}_{t}\odot\mathbf{h}_{t-1},\] (3) \[\mathbf{o}_{t} =\psi_{o}\left(\mathbf{w}_{o}*\text{CNN}\left(\mathbf{x}_{t} \right)+\mathbf{u}_{o}*\mathbf{h}_{t-1}\right), \tag{4}\]

where \(\mathbf{w}_{f}\), \(\mathbf{u}_{f}\), \(\mathbf{w}_{o}\), \(\mathbf{u}_{o}\) are small convolution kernels and \(\psi_{f}\) is either the sign (when working with binary constraints) or the sigmoid function (when working with full precision). We include an optional nonlinear activation function \(\psi_{o}\) and an output layer \(\mathbf{o}_{t}\) representing the values that are actually transmitted off sensor to the downstream network running on a separate processor. For low-bandwidth operation, the output layer \(o\) is only computed, and values transmitted off the sensor, every \(K\) frames. The output layer can optionally be omitted, in which case the hidden state \(\mathbf{h}_{t}\) is streamed off the sensor every \(K\) frames.

PixelRNN uses what is commonly known as a "forget gate", \(\mathbf{f}_{t}\), and a hidden state \(\mathbf{h}_{t}\), which are updated at each time step \(t\) from the input \(\mathbf{x}_{t}\). RNNs use forget gates to implement a "memory" mechanism that discards redundant spatial features over time. PixelRNN's forget gate is also motivated by this intuition, but our update mechanism in Eq. 3 is tailored to working with binary constraints using values \(\{-1,1\}\). In this case, Eq. 3 flips the sign of \(\mathbf{f}_{t}\) in an element-wise manner rather than decaying over time. This mechanism works very well in practice when \(\mathbf{h}_{t}\) is re-initialized to all ones every 16 time steps.

PixelRNN's architecture includes alternative spatial- or temporal-only feature extractors as special cases. For example, it is intuitive to see that it models a conventional CNN by omitting the recurrent units. We specifically write out the output gate in our definition to make it intuitive how PixelRNN also approximates a difference camera as a special case, which effectively implement a temporal-only fea

Figure 1: The perception pipeline of PixelRNN can be broken down into an on-sensor encoder and a task-specific decoder. On the left is the camera equipped with a sensor–processor, which offers processing and memory at the pixel level. The captured light is directly processed by a CNN that extracts spatial features, which are further processed by a convolutional recurrent neural network with built-in memory and temporal feature extraction. Here we show our PixelRNN variant on the right, \(\star\) being the convolution operator, \(\odot\) element-wise multiplication, and \(\psi\) a nonlinear function. Instead of sending out full \(256\times 256\) values at every time step, our encoder compresses by \(64\times\). While we show this pipeline for a lip reading task, the decoder can be designed for any perception task.
ture encoder. In this case, \(\mathbf{h}_{t}=\mathbf{x}_{t},\mathbf{w}_{o}=1,\mathbf{u}_{o}=-1\) and \(\psi_{o}\left(x\right)=\begin{array}{cc}-1&\text{for x}<-\delta\\ 0&\text{for }-\delta\leq\text{x}\leq\delta\\ 1&\text{for }\delta<\text{x}\end{array},\) for some threshold \(\delta\). The image formation model of event cameras [19] is asynchronous and a difference camera represents only a crude approximation, but it serves as a pedagogically useful temporal-only encoder in the context of this discussion.

### Learning Quantized In-pixel Parameters

PixelRNN uses binary weights to reduce all multiplications to sums. To learn these parameters efficiently, we parameterize each of these values \(w\) using a continuous value \(\tilde{w}\) and a quantization function \(q\) such that

\[w=q\left(\tilde{w}\right),\quad q:\mathbb{R}\rightarrow\mathcal{Q}. \tag{5}\]

Here, \(q\) maps a continuous value to the closest discrete value in the feasible set \(\mathcal{Q}\), i.e., \(\{-1,1\}\).

One can employ surrogate gradient methods [3, 54], continuous relaxation of categorical variables using Gumbel-Softmax [29, 38], or other approaches to approximately differentiate through \(q\). For the binary weights we use, \(w=q(\tilde{w})=\text{sign}(\tilde{w})\), and we found approximating the gradient of the sign function with the derivative of \(\text{tanh}(mx)\) produced very good results:

\[\frac{\partial q}{\partial\tilde{w}}\approx m\cdot(1-\tanh^{2}(m\tilde{w})) \tag{6}\]

where \(m>0\) controls the steepness of the \(\tanh\) function, which is used as a differentiable proxy for \(q\approx\tanh(m\tilde{w})\) in the backward pass. The larger \(m\) is, the more it resembles the sign function and the more the gradient resembles the delta function.

### Implementation Details

We implement our per-frame CNN feature encoder, testing both 1 or 2 layers and process images at a resolution of \(64\times 64\) pixels by downsampling the raw sensor images before feeding them into the CNN. In all experiments, our PixelRNN transmits data off the sensor every 16 frames. Thus, we achieve a reduction in bandwidth by a factor of 64\(\times\) compared to the raw data. In all of our experiments, we set the function \(\psi_{o}\) to the identity function.

Additional implementation details are found in the supplement and source code will be made public for enhanced reproducibility.

## 4 Experiments

Evaluating Feature Encoders.As discussed in the previous section, RNNs require a CNN-based feature encoder as part of their architecture. In-pixel CNNs have been described in prior work [7, 32, 34], albeit not in the context of video processing with RNNs.

Table 1 summarizes the simulation performance of re-implementations of various CNN architectures on image classification using the MNIST, CIFAR-10 dataset, and hand gesture recognition from individual images.

Bose et al. [6, 7] describes a 2-layer CNN with ternary weights. The two works have the same architecture but differ drastically in the sensor-processor implementation. Liu et al. [32, 34] describes 1- and 3-layer CNNs with binary weights using a different architecture than Bose while using similar sensor-processor implementations concepts. Our feature encoder is a binary 2-layer variant of Bose et al.'s CNN. Each layer has 16 kernels of size \(5\times 5\) and are followed with a non-linearity and maxpooling of \(4\times 4\). The 16 \(16\times 16\) channels are then concatenated into a single \(64\times 64\) image size to serve as the input to the next convolutional layer or to the PixelRNN. All of these CNNs are roughly on-par with some performing better at some tasks than others. Ours strikes a good balance between accuracy and model size. We do not claim this CNN to be a contribution of our work, but include this brief comparison for completeness.

Baseline Architectures.We use several baselines for our analysis. The RAW camera mode simply outputs the input frame at every time step and represents the naive imaging approach. The simulated difference camera represents a simple temporal-only feature encoder. We also include several RNN architectures, including long short-term memory (LSTM), gated recurrent unit (GRU), minimal gated unit (MGU), a simple RNN (SRNN), and our PixelRNN. Moreover, we evaluated each of the RNN architectures using 1-layer and 2-layer CNN feature encoders. The output of all RNNs is read from the sensor only once every 16 time steps. All baselines represent options for in-pixel processing and their respective output is streamed off the sensor. In all cases, a single fully-connected network layer processes this output on a downstream processor to compute the final clas

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline Model Name & MNIST & CIFAR-10 & Hand & \# Model & Size \\  & & & Gesture & Params & (MB) \\ \hline Bose [7, 6] & 95.0\% & 39.8\% & 43.4\% & 257 & 0.05 \\ Liu 2020 [32] & 80.0\% & 32.5\% & 57.4\% & 258 & 0.03 \\ Liu 2022 [34] & **95.1\%** & 32.6\% & 60.2\% & \(2,374\) & 0.30 \\ Our CNN & 90.9\% & **43.1\%** & **68.1\%** & 802 & 0.10 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Comparing CNN Feature Encoders. We simulated different in-pixel CNNs using image classification on MNIST, CIFAR-10, and the Cambridge hand gesture recognition based on different implementations. All CNN architectures perform roughly on par with our binary 2-layer CNN encoder striking a good balance between accuracy and model size. The model size is computed by multiplying the number of model parameters by the quantization of the values.**
sification scores. This fully-connected layer is trained end to end for each of the baselines. While this fully-connected layer is simple, it is effective and could be customized for a specific task.

Additional details on these baselines, including formulas and training details, are listed in the supplement. Table 2 shows an overview of these, listing the number of model parameters and the readout bandwidth for each of them.

Datasets.For the hand gesture recognition task, we use the Cambridge Hand Gesture Recognition dataset. This dataset consists of 900 video clips of 9 gesture classes; each class contains 100 videos. For the lip reading task, we use the Tulips1 dataset. This dataset is a small Audiovisual database of 12 subjects saying the first 4 digits in English; it was introduced in [41].

Accuracy vs. Memory.In Figure S1, we evaluate the accuracy of several baseline architectures on two tasks: hand gesture recognition (left) and lip reading (right). We compare the baselines described above, each with 1- and 2-layer CNN encoders and binary or full 32-bit floating point precision. For the full-precision networks, PixelRNN achieves an accuracy comparable to the best models, but it provides one of the lowest memory footprints. Comparing the networks with binary weights, PixelRNN also offers the best accuracy with a memory footprint comparable to the next best method. Surprisingly, larger architectures, such as GRUs and LSTMs, do not perform well when used with binary weights. This can be explained by the increasing difficulty of reliably training increasingly large networks with binary parameter constraints. Leaner networks, such as SRNN and PixelRNN, can be trained more robustly and reliably in these settings. Note that the memory plotted on the x-axis represents all intermediate features, per pixel, that need to be stored during a single forward pass through the RNN. We do not count the network parameters in this plot, because they do not dominate the memory requirements and can be shared among the pixels.

Constraints of the Experimental Platform.Our hardware platform, SCAMP-5, provides an equivalent to 32 bytes of memory per pixel for storing intermediate features. This limit is illustrated as dashed vertical lines in Figure S1, indicating that only low-precision RNN networks are feasible on this platform. The available memory is computed as follows. SCAMP-5 has 6 analog registers per pixel, each we

\begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{\multirow{2}{*}{Model Name}} & \# Model & \multicolumn{1}{c}{Readout} \\  & & Parameters & Bandwidth \\ \hline \multirow{3}{*}{
\begin{tabular}{c} **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S** **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S**lass **S** **S**lass **S**lass **S**lass **S**lass **S** **S**lass **S**lass **S**lass **S**lass **S** **S**lass **S**lass **S**lass **S**lass **S** **S**lass **S**lass **S** **S**lass **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S**lass **S** **S** **S**lass **S** **S** **S** **S** **S**lass **S** **S** **S** **S** **S**S** **S** **S** **S** **S**S** **S**S** **S** **S** **S**S** **S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S** **S**S**S** **S**S**S** **S**S**S** **S**S**S**S** **S**S**S**S**S **S**S**S**S**S**S **
assume is equivalent to 1 byte: 2 registers store the model weights, 2 are reserved for computation, leaving 2 bytes per pixels for the intermediate features. SCAMP-5 consists of an array of \(256\times 256\) pixel processors, however our approach operates on a smaller effective image size of \(64\times 64\). This allows us to consider a single "pixel" to comprise of a block of \(4\times 4\) pixel elements, increasing the effective bytes per pixel to 32 bytes.

Accuracy vs. Bandwidth.We select a readout bandwidth of 4,096 values (every 16 frames) based on the available bandwidth of our hardware platform, the SCAMP-5 sensor. In Figure 3 we evaluate the effect of further reducing this bandwidth on the accuracy for the PixelRNN architecture. Bandwidth is controlled using a max-pooling layer operating at differing sizes from \(1\times 1\) through \(8\times 8\) and then at multiples of 8 up to \(64\times 64\) before inputting the intensity images to PixelRNN. The resulting output bandwidths range between 1 to 4096. We ran each experiment ten times and the best performances of each are plotted for hand gesture recognition and lip reading. We observe that the bandwidth could be further reduced to about 1,000 values every 16 frames without significantly degrading the accuracy on these datasets. However, decreasing the bandwidth beyond this also reduces the accuracy.

## 5 Experimental Prototype

### Pixel-Level Programmable Sensors

SCAMP-5 [9] is one of the emerging programmable sensors representative of the class of focal-plane sensor-processors (FPSP). Unlike conventional image sensors, each of the \(256\times 256\) pixels is equipped with an arithmetic logic unit, 6 local analog and 13 digital memory registers, control and I/O circuitry, and access to certain registers of the four neighboring pixels. SCAMP-5 operates in single-instruction multiple-data (SIMD) mode with capabilities to address individual pixels, patterns of pixels, or the whole array. It features mixed-mode operation execution that allows for low-power compute prior to A/D conversion. Most importantly, SCAMP-5 is programmable.

### Implementation of PixelRNN on SCAMP-5

The pipeline for our prototype feature extractor is shown in Figure 4. Because of the memory architecture on SCAMP-5, performing multiple convolutions and different updates of gates and states require us to split the focal plane into 16 parallel processors with a smaller effective image size. The input image is binarized, downsampled to \(64\times 64\), and duplicated out to a \(4\times 4\) grid of parallel processor elements (PE) of size \(64\times 64\). Each PE performs a convolution with a \(5\times 5\) kernel, yielding 16 feature maps per convolutional layer. The 16 \(64\times 64\) features undergo a ReLU activation, maxpooling and binarization to 16 \(16\times 16\). These are then concatenated to create a single \(64\times 64\) input to the RNN or another convolutional layer if desired. This process makes use of the image transformation methods for SCAMP-5 introduced by [5]. Our RNN uses both the output of the CNN and the hidden state to update the hidden state and compute an output every 16 time steps. The RNN gates are calculated via convolution and element-wise multiplication. To suit the SCAMP-5 architecture, we limited operations to addition, XOR, and negation, and trained a binary version of PixelRNN, binarizing weights and features to -1 and 1. Instead of multiplications, we now just need addition and subtraction.

Memory Allocation.SCAMP-5's analog and digital registers are limited in number and present different challenges. Analog registers cannot hold values for long periods before decaying. The decay is exacerbated if one moves information from pixel to pixel such as in shifting an image. We found using analog registers with a routine to refresh their content to a set of quantized values inspired by [7] helped circumvent some of the challenges. This allowed the storage of binary weights for convolutions and the hidden state for prolonged periods of time. The remaining memory registers were used for performing computations and storing intermediate feature maps.

Convolution Operation.A single pixel cannot hold all weights of a single kernel, so the weights are spread across a single analog register plane as shown in Figure 4. To perform a convolution, SCAMP-5 iterates through all 25

Figure 3: **Bandwidth Analysis. We can control the bandwidth of data read off the sensor using increasingly larger max-pooling layers before inputting the intensity images to PixelRNN at the cost of decreased accuracy.**
weights in the \(5\times 5\) kernel, each time multiplying it with the whole image and adding to a running sum. The image is then shifted, the next weight fills the register plane, and the process continues until the feature is computed. We include a detailed diagram in the supplement and more information can be found in [7].

RNN Operation.Figure 5 shows the layout of the sequence of operation in the RNN. Each pixel contains 6 analog registers, named A,B,C,D,E,and F. We refer to register plane A as all the A registers across the entire image sensor. In Figure 5, the \(256\times 256\) pixels are split into a \(4\times 4\) larger processor elements of size \(64\times 64\). In register plane A, we take the output from the CNN and the previous hidden state and duplicate it to two other PEs in plane A. Register plane B holds the corresponding weights \(\textbf{w}_{f}\), \(\textbf{u}_{f}\), \(\textbf{w}_{o}\), \(\textbf{u}_{o}\) for the convolution operators needed. 4 convolutions are simultaneously run on one register plane. The outputs in plane B are shifted and added. A binarization is applied to get \(\textbf{f}_{t}\). This is then used to update a hidden state via element-wise multiplication every time step. Every 16 time

Figure 4: This pipeline shows the sequence of operations from left to right. The input image is downsampled, duplicated, and binarized. Stored convolutional weights perform 16 convolutions, to produce 16 feature maps in the \(4\times 4\) grid of processor elements. A ReLU activation is applied, followed by max-pooling, downsampling, and binarization. This can either be fed to another CNN layer or to the input of the RNN. The RNN takes in the output of the CNN and the previous hidden state \(\textbf{h}_{t-1}\). The hidden state \(\textbf{h}_{t}\) is updated every timestep. The output \(\textbf{o}_{t}\) is read out every 16 frames, yielding 64\(\times\) decrease in bandwidth.

Figure 5: To implement the PixelRNN on SCAMP-5, the image plane is split into a \(4\times 4\) grid of processor elements shown above. Two analog register planes are used, Register planes A and B. Above, we show the sequence of operations from left to right. The input from the CNN and the previous hidden state are duplicated in A. These 4 PEs are convolved \(*\) with the corresponding gate weights stored in plane B. The resulting convolutions in the second column are then added to compute the output \(\textbf{o}_{t}\) and the forget gate \(\textbf{f}_{t}\). Note that an in-place binarization is applied to \(\textbf{f}_{t}\). The hidden state \(\textbf{h}_{t}\) is updated via an element-wise multiplication \(\odot\) of \(\textbf{h}_{t-1}\) and \(\textbf{f}_{t}\).
steps, SCAMP-5 outputs the \(64\times 64\) image corresponding to the output gate \(\mathbf{o}_{t}\). Our spatio-temporal feature encoder distills the salient information while giving a 64\(\times\) decrease in bandwidth.

Accounting for Analog Uncertainty.As with all analog compute, a certain amount of noise should be expected. As such, we treat each of the SCAMP's analog registers to contain values split across equally spaced discrete intervals. During convolutions, the binary image and binary weights are XNOR-ed. Depending on the result, we either add or subtract an analog value approximately equal to \(10\). As the analog registers have an approximate range of values \(-128\) to \(127\), the interval cannot be increased without risking saturation during convolutions. However, there is uncertainty when it comes to the precision and uniformity of the intervals. Along with decay, this uncertainty, spatial non-uniformity, and noise affect the operations that follow. In the RNN, these effects accumulate for 16 frames, leading to a significant amount noise. To account for these effects, we trained binary models in simulation with varying amounts of added Gaussian noise in the CNN and the RNN prior to quantization of the features. We also fine-tuned the off-sensor layer on the training set outputs from SCAMP-5.

Assessment.To test our prototype, we uploaded the video datasets to SCAMP-5 in sequence and saved out the outputs every 16 frames (see supplement for additional details). In our current implementation, it takes roughly 95 ms to run a single frame through the on-sensor encoder. The \(64\times 64\) output region then goes through the off-sensor linear layer decoder. We evaluate the performance using the models trained with and without noise. The results shown in Table 3 highlight the benefits of training with noise, as well as the difficulty that comes with working with analog registers. We see even running the same train set through SCAMP-5 two separate times does not result in the same performance. Without the noise-trained model, we reached 61.1% on the hand gesture recognition test set. Performance improved to 73.3% when we used the weights from training with noise. Similarly, the performance on lip reading was boosted to 70.0% when using a model trained on noise. While added noise during training helps, the noise characteristics of SCAMP-5 are much more complex. Such issues may be mitigated in future sensor-processors with sufficient digital registers to avoid having to rely upon the use of analog computation. While limited by noise, we demonstrated the first in-pixel spatio-temporal encoder for bandwidth compression.

## 6 Discussion

In the traditional computer vision pipeline, full-frame images are extracted from the camera and are fed into machine learning models for different perception tasks. While completely viable in systems not limited by compute, memory, or power, many edge devices do not offer this luxury. For systems like AR/VR devices, robotics, and wearables, low-power operation is crucial, and even more so if the system requires multiple cameras. The community has already been working on creating smaller, more efficient models, as well as specialized accelerators. However, the communication between camera and processor that consumes nearly 25% of power in these systems [20] has not been optimized. In this work, we demonstrate how running a simple in-pixel spatio-temporal feature extractor can decrease the bandwidth, and hence power associated with readout, by 64\(\times\). Even with highly quantized weights and signals and a very simple decoder, we still maintain good performance on hand gesture recognition and lip reading. We studied different RNN architectures and presented PixelRNN that performs well in highly quantized settings, we studied just how small we could make the bandwidth before affecting performance, shown in Figure 3, and implemented a physical prototype with one of the emerging sensors, SCAMP-5, that is paving the way for future sensors.

Limitations and Future Work.One of the biggest challenges of working with SCAMP5 is accounting for the analog noise, but the platform offers great flexibility to program the data movement between pixels to implement prototypes. While SCAMP-5 offers many exciting capabilities, it is still limited in memory and compute as all circuitry needs to fit in a single pixel. Until recently, adding circuitry or memory to image sensors compromised the fill factor, which worsens the imaging performance and limits achievable image

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & Train Set & Train Set & Test Set \\  & Accuracy run 1 & Accuracy run 2 & Accuracy \\ \hline \hline
**Hand Gesture Recognition** & & & \\ Noise-free Model & 100.0\% & 64.0\% & 61.1\% \\ Model trained with noise & 95.3\% & 70.8\% & 73.3\% \\ \hline
**Lip Reading** & & & \\ Noise-free Model & 100.0\% & 78.9\% & 50.0\% \\ Model trained with noise & 98.5\% & 84.85\% & 70.0\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Experimental Results.** We run the training sets through the SCAMP-5 implementation twice. The first outputs are used to fine-tune the off-sensor linear layer decoder. In theory, the train set accuracy of the second runs should be close. With the noise accumulated through the analog compute, however, the SCAMP-5 implementation is not deterministic. Adding Gaussian noise during training increases the test-set performance.
resolution. With the new developments in stacked CMOS image sensors, future sensors will be able to host much more compute and memory on the sensor plane, allowing us to design more expressive models and to apply tools like architecture search to optimize where compute happens in a system [17]. Until then, we are limited to light-weight in-pixel models. Noise related to analog compute can also be mitigated by switching over to digital compute. Our work is not only applicable to SCAMP-5 but to all future focal-plane processors.

Conclusion.Emerging image sensors offer programability and compute directly in the pixels. Our work is the first to demonstrate how to capitalize on these capabilities using efficient RNN architectures, decreasing the bandwidth of data that needs to be read off the sensor as well as stored and processed by downstream application processors by a factor of 64\(\times\). We believe our work paves the way for other inference tasks of future artificial intelligence-driven sensor-processors.

## Acknowledgements

This project was in part supported by the National Science Foundation and Samsung.
