Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking
  Neural Network
Human vision incorporates non-uniform resolution retina, efficient eye
movement strategy, and spiking neural network (SNN) to balance the requirements
in visual field size, visual resolution, energy cost, and inference latency.
These properties have inspired interest in developing human-like computer
vision. However, existing models haven't fully incorporated the three features
of human vision, and their learned eye movement strategies haven't been
compared with human's strategy, making the models' behavior difficult to
interpret. Here, we carry out experiments to examine human visual search
behaviors and establish the first SNN-based visual search model. The model
combines an artificial retina with spiking feature extraction, memory, and
saccade decision modules, and it employs population coding for fast and
efficient saccade decisions. The model can learn either a human-like or a
near-optimal fixation strategy, outperform humans in search speed and accuracy,
and achieve high energy efficiency through short saccade decision latency and
sparse activation. It also suggests that the human search strategy is
suboptimal in terms of search speed. Our work connects modeling of vision in
neuroscience and machine learning and sheds light on developing more
energy-efficient computer vision algorithms.
Yunhui Zhou, Dongqi Han, Yuguo Yu
# Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking Neural Network

###### Abstract

Human vision incorporates non-uniform resolution retina, efficient eye movement strategy, and spiking neural network (SNN) to balance the requirements in visual field size, visual resolution, energy cost, and inference latency. These properties have inspired interest in developing human-like computer vision. However, existing models haven't fully incorporated the three features of human vision, and their learned eye movement strategies haven't been compared with human's strategy, making the models' behavior difficult to interpret. Here, we carry out experiments to examine human visual search behaviors and establish the first SNN-based visual search model. The model combines an artificial retina with spiking feature extraction, memory, and saccade decision modules, and it employs population coding for fast and efficient saccade decisions. The model can learn either a human-like or a near-optimal fixation strategy, outperform humans in search speed and accuracy, and achieve high energy efficiency through short saccade decision latency and sparse activation. It also suggests that the human search strategy is suboptimal in terms of search speed. Our work connects modeling of vision in neuroscience and machine learning and sheds light on developing more energy-efficient computer vision algorithms.

## 1 Introduction

Biological vision is well adapted for survival in natural environment. In most cases, it needs to fulfill at least four requirements: high resolution, large visual field, low response latency, and low energy cost. However, having both high resolution and large visual field will increase the computational and energy cost, and may also increase response latency. To balance between these requirements, most vertebrates' retina has high resolution at center and low resolution in periphery [1; 2; 3; 4; 5; 6]. This allows animals to cover a large visual field by a small number of cells, while still keeping a subjectively clear vision. For example, the monocular visual field of the human eye is \(160^{\circ}\times 135^{\circ}\)[7], but the number of ganglion cells on retina is only 0.7-1.5 million per eye [8].

To compensate for the low resolution peripheral vision, many animals use the "fixation-saccade" eye movement strategy to view the world [9; 10; 11; 12; 13; 14; 15; 16; 17; 18]. To decrease response latency and energy cost, the eye movement strategy needs to be efficient and focus mostly on task-related regions. Indeed, humans can
dynamically adjust fixation locations according to task demands[19], leading to better performance in visual search tasks than random eye movement strategy[20].

Biological vision also uses spiking neurons to process information. This has inspired the spiking neural network (SNN), which is in theory as computationally powerful as conventional artificial neural networks (ANN)[21]. SNN running on neuromorphic chips can be much more energy-efficient than ANN running on GPU or CPU[22] due to its event-driven computation and collocated processing and memory[23].

In summary, **non-uniform resolution retina**, **efficient eye movement strategy**, and **computation by spiking neurons** may be crucial in supporting energy-efficient human vision. However, most current computer vision models process uniform-resolution images and is based on ANN. Their energy efficiency is unsatisfactory given that the human brain consumes only 20 Watts[24]. Although a few models were trained to process non-uniform resolution images and make eye movements, their eye movement strategies had various limitations (such as requiring a fixed fixation number[25, 26, 27, 28, 29, 30, 31, 32], or can only fixate at discrete set of locations[33]), and the learned strategy haven't been compared with that of human's, so it is unknown whether these models can achieve human-level performance. Besides, as far as we know, no model has fully incorporated the three properties of human vision.

The goal of this work is to establish the first bio-inspired visual search model (BVSM) that utilizes features of human vision to achieve efficient saccade decision and low energy cost (Fig. 1C). To compare BVSM's eye movement strategy to that of human's, we train both human subjects and the model on a classical visual search task that needs to find a single Gabor target embedded in naturalistic noise image (Fig. 1A)[20]. The task is chosen for its following properties:

* **Controllable**: The features of target and background are controlled, so the target visibility across the retina can be conveniently measured by psychophysical experiment. This allows a fair comparison between BVSM's and human's search performance.
* **Explainable**: There exists a Bayesian optimal search strategy for this task based on human target visibility data[20, 34], which allows a comparison between the optimal and BVSM's search strategy.
* **Challenging**: The target is much smaller (1/2943) and has lower contrast than the background. Conventional network that processes full-resolution original images fails to classify whether the image contains a target.
* **Naturalistic**: The image has \(1/f\) amplitude spectrum on spatial frequencies, so it is statistically similar to natural images[35] though the image contents are not the same.

BVSM can learn efficient stochastic eye movement policy by reinforcement learning (RL) to search for the target (Fig. 1D and Supplementary Video). To summarize our contributions:

* We for the first time integrate the non-uniform resolution retina, efficient eye movement strategy, and computation by SNN into BVSM. BVSM autonomously develops sparse coding during training and leverages population coding to achieve ultra-low saccade decision latency.
* We show that BVSM achieves high energy efficiency compared to conventional ANNs that process uniform-resolution images without making eye movements. Our work establishes a starting point for future research on human-like vision model.
* We compare the visual search strategies of humans, BVSM, and the optimal strategy, showing that BVSM can either learn a human-like or a near-optimal strategy that outperforms humans. The comparison contributes to a deeper understanding of human eye movement strategy.

## 2 Related Work

Bayesian optimal visual search modelThe Bayesian optimal visual search model uses experimentally measured target visibility data to optimally compute the probability of target locations and select the next fixation location[20, 34] (Appendix B). However, this model has a drawback as its inputs are random signals generated by the target visibility data instead of the actual search image, and generating the signals requires knowing the actual target location. Therefore, the Bayesian optimal search model is not suitable for real-life applications. In contrast, the BVSM learns an efficient search strategy directly from the search image without prior knowledge of target location or target visibility data, so it is more applicable to real-life scenarios.
Hard attention modelsBVSM is a kind of hard attention model that down-samples the input image and makes eye movements. Currently, the most influential hard attention model architecture is probably the recurrent attention model (RAM) [26, 27]. RAM contains a feature extraction network, a recurrent network (RNN), a saccade decision network, and a classification network. Other model structures usually either lack a short-term memory [28, 36], or use simple manually designed memory [25, 29, 31, 37]. Some models also use Transformer to integrate information from multiple fixations [32, 38]. BVSM has a similar architecture to RAM, but it is implemented as an SNN. We do not use a Transformer architecture because humans may only have a memory capacity of fewer than 20 items in visual search [39, 40, 41, 42]. Conventional RNNs should be able to handle a memory of this size.

In terms of the eye movement behavior, some previous models can only fixate at a pre-defined set of discrete locations [33, 38, 43], or require setting a fixed number of fixations for each trial [25, 26, 27, 28, 29]. In particular, RAM-based model may perform worse when fixating more times than it was trained on [31]. BVSM makes continuous-valued saccade decisions, and it naturally employs a variable number of fixations to do visual search by generating a confidence rating to predicted target location.

Many previous works are tested on visual search for MNIST-style targets, and the tasks are relatively easy because the search target is large (\(>\)4.79%) compared to the background image [25, 26, 27, 28, 33, 36]. We find that the same issue also exists for models that were tested on natural images [29, 29] by inspecting the datasets used for training (PASCAL VOC 2007/2010 [44, 45]). Training hard attention models to search for tiny objects has not been thoroughly studied yet. BVSM is tested on a challenging visual search task that has been studied in neuroscience [20, 34, 46, 47]. The search target is only 0.034% of the size of the search image, and humans can only achieve a mean correct response rate of about 86% [47]. It is interesting to see how hard attention models will perform in this task.

In summary, various limitations exist in previous hard attention models, which we aim to overcome in our model. Table S1 shows a detailed comparison between BVSM and previous works.

Population coding with SNNThe SNN's output variables are often encoded by the firing rate of output neurons. To calculate the firing rate, SNN needs to process the input for a number of time steps, which is called the inference time or latency. Lower latency means faster inference and lower energy cost, but it also limits the information carried by the firing rate of a single neuron. Previous works on using SNN in regression task (like objection detection) usually have high latency (100-3,500 time steps) to precisely calculate the firing rate [48, 49, 50, 51, 52, 53], but this decreases the energy efficiency of SNN. To decrease the latency while maintaining the regression accuracy, we can use the weighted sum of the firing rate of a neuron population to encode a variable. Recently, SNN with population coding has achieved good performance on a variety of continuous-action RL tasks, while lowering the latency to only 5 time steps [54, 55]. In this work, we use population coding to achieve very low saccade decision latency (4 time steps). Table S2 shows a comparison of our SNN to previous models.

Figure 1: **A**. An example search image with target enlarged at lower right. Symbol "0" denotes degrees of visual angle viewing from 70 cm distance. **B**. Human visual search experiment setup. **C**. Structure of BVSM. See Fig. S1 for details. **D**. An example search trial by BVSM. Cross and circle are the current fixation location and target location. Color map shows the probability distribution of the next fixation. The actual next fixation is randomly sampled from this distribution.
Methods

### Visual Search Task

The visual search task is to find a single Gabor target embedded in a circular naturalistic noise image (\(1/f\) amplitude spectrum) (Fig. 1A)[20; 47]. The background noise image has a diameter of \(15^{\circ}\) (651 pixels) with root-mean-squared (standard deviation of pixels divided by the mean) contrast of 0.2. The target is a 6 cycles/degree Gabor with a diameter of 0.3\({}^{\circ}\) (12 pixels), oriented 45\({}^{\circ}\) counterclockwise from the vertical, and windowed by a symmetrical raised cosine. The screen to display the image has a physical size of \(54.3744\times 30.2616\) cm, resolution of \(1920\times 1080\), and was placed 70 cm away from human subjects (Fig. 1B). We used the same target contrast range in human experiment and modeling (detailed below). The background noise is varied in each trial.

### Human Experiment

29 human subjects (14 females, aged 18-30) participated in the experiment. For 15 subjects, we first used a two-interval force-choice (2IFC) task to measure the target visibility \(d^{\prime}\) at fovea as a function of target contrast, and individually estimated the target contrast for \(d^{\prime}=3.0\). These target contrasts (ranged 0.11-0.136) were used for all following experiments of each subject. We then conducted another 2IFC task to measure target visibility at various peripheral locations. Finally, subjects finished 200-400 visual search trials (Fig. 1B). Visual search always started from fixating at image center. The rest 14 subjects only participated in the visual search experiment with a fixed target contrast of 0.15. We call the first 15 subjects "Low TC (target contrast) Group", and the rest 14 subjects "High TC Group". See Appendix A for the details of the detection and visual search experiment.

### Bio-inspired Visual Search Model

Fig. 1C and Fig. S1 show the structure of BVSM. It contains an artificial retina and four networks.

**Artificial retina.** We use Foveal Cartesian Geometry (FCG)[56] to non-uniformly down-sample the original image (Fig. S1A). FCG copies pixels inside the fovea, and samples pixels in periphery from a series of rings with increasingly larger widths (Appendix C.1). The sampled pixels are then assembled to form a square image as the output. We set the size of the output image to be 224x224 with a fovea size of 16x16 pixels. To restrict the entire search image within the visual field, we sample from a region twice the diameter of the search image. The configurations of artificial retina are closely related to visual search performance. In Appendix C.5, we also examined the model's target visibility at peripheral locations using a similar 2IFC task as in human experiment.

**Feature extraction network (FEN).** The FEN processes retinal-transformed images by 7 convolution blocks and 3 linear heads (Fig. S1B), and outputs the current fixation location (2D vector), target location relative to the fixation location (2D vector), and the estimated error between predicted and true target location (scalar). The output locations are all in the original full-resolution image's coordinate space. FEN uses QCFS function[57] as activation function: \(f\left(x\right)=\lambda\cdot\text{clip}\left(\frac{1}{T}\text{floor}(\frac{xT }{\lambda}+0.5),0,1\right)\), where \(\lambda\) is a trainable parameter and \(T=4\). In the last layer of fully connected heads, we linearly transform the output of a population of neurons into the output vectors (population coding). FEN is converted to SNN with latency \(T=4\) after training (Appendix C.2).

**Recurrent network (RNN).** We propose a novel spiking RNN that uses 64 integrate-and-fire (IF) neurons (equation S10) to memorize fixation locations (Fig. S1C). The network's inputs are the predicted fixation location from FEN and the RNN hidden state. Spiking RNN adds another temporal dimension (the SNN inference time in each fixation) to the existing temporal dimension (sequence of fixations). To maintain the real-time processing property of SNN, we set the RNN hidden state as the output spikes from the last time step of the previous fixation, and the hidden state is used in computation at the first time step of the next fixation. See Appendix C.3 and Algorithm S1 for details.

**Actor and critic networks.** The actor network makes saccade decisions (Fig. S1D). If the estimated target location error from FEN is lower than a threshold (0.58\({}^{\circ}\), or 25 pixels), the next fixation location will be sampled near the predicted target location; otherwise the actor will process the RNN output by an SNN which outputs the mean and covariance of the next fixation location's distribution by population coding. The next fixation location can be anywhere within the square region enclosing the circular search image. The critic network measures saccade value (Fig. S1E). If the estimated
target location error is low, it will calculate the Q-value by the next fixation location and the predicted target location; otherwise it will calculate the Q-value by the next fixation location and RNN output. The critic network is not a spiking network, as it is not used in inference after training. See Appendix C.4 for details of the actor and critic networks.

### Reward Functions

The agent receives the sum of two negative rewards after each saccade. The first reward is inhibition of return (IOR) that tracks the location of the most recent 8 fixations based on previous human research [40; 42], and discourages the agent to fixate at previously fixated regions (Fig. S3A,B). The IOR reward after a saccade to the \((f+1)^{\text{th}}\) fixation location is a semi-circle function centered at each previously fixated location:

\[R_{\text{IOR}}=\min_{i\in\{0\cdots 7\}}\left(-\frac{1}{r}\sqrt{\max\left(r^{2} -d_{f+1,f-i}^{2},0\right)}\right), \tag{1}\]

where \(d_{f+1,f-i}\) is the distance between the \((f+1)^{\text{th}}\) and \((f-i)^{\text{th}}\) fixation, and \(r\) is the radius of the semi-circle function.

The second reward is the saccade amplitude reward that discourages making large saccades (Fig. S3C,D). Making large saccade costs more time and energy [58], and induces strong motion blur that disrupts visual processing [59]. We test two different saccade amplitude reward functions:

\[R_{\text{SA}}=-0.5+0.5\times[\exp{(-\text{Saccade amplitude}/2.5^{ \circ})}-1], \tag{2}\] \[R_{\text{SA}}=-\text{Saccade amplitude}/7.5^{\circ}. \tag{3}\]

To learn human-like search strategy, we set \(r=0.5^{\circ}\) for IOR reward, and use equation 2 for saccade amplitude reward. We call this setting "HP (hyperparameter) 1". To learn Bayesian-inference-like search strategy, we set \(r=2.5^{\circ}\) for IOR reward, and use equation 3 for saccade amplitude reward. We call this setting "HP 2". We also test BVSM's performance under some changes to the two reward functions (Table S5). We set the discount factor \(\gamma=0.95\).

### BVSM Training and Evaluation

BVSM is trained in three stages. First, we train the FEN to detect target in random samples of retinal transformed images. Second, we convert the FEN to SNN, and fine-tune the SNN. Third, we fix the FEN, and use soft actor-critic (SAC) [60] to train the RNN, actor, and critic to do visual search. We directly train the SNN in RNN and actor by spatio-temporal backpropagation (STBP) [61]. Table S3 shows all hyperparameters. Fig. S4, S5, and S6 show the loss and task performance during each training stage. BVSM is evaluated for 10,000 trials after training. Details are shown in Appendix C.6.

### Automatic Search Termination

Search trial ends either by reaching the maximum fixation number (50 in training, 200 in testing), or by a "double check and stop" rule. The latter is triggered when the estimated target location error of two consecutive fixations is below 0.58\({}^{\circ}\) (25 pixels) and the distance between the two predicted target locations is less than 0.5\({}^{\circ}\). If any of the final two fixations are within 1\({}^{\circ}\) away from the true target location (same as in human experiment), the trial is labeled as correct; otherwise error. If a trial does not automatically terminate when reaching the maximum fixation number, it is labeled as error.

## 4 Results

### Target Detection Performance Comparison

The spiking FEN accurately regresses the current fixation location (Fig. 2A) and target location when the distance between target and fixation location is small and the target contrast is high (Fig. 2B). FEN outputs a low estimated target location error when the target locates near the fixation, which means a high prediction confidence; otherwise it outputs a high estimated error. We can set the target detection threshold as 25 pixels (Fig. 2C). If the estimated error is below the threshold, the network will fixate at the predicted target location in the next fixation.
In the detection task (Fig. 3A), FEN performs similar to humans in central vision, but worse than humans in peripheral vision (Fig. 3B,C). In addition, conventional ANN that processes uniform-resolution images fails to detect the small target (Appendix D). This shows the importance of using non-uniform resolution retina when needing to discriminate fine details from a large background.

### Visual Search Strategy Comparison

After training with hyperparamter group 1 (HP 1), BVSM's search strategy shows some similarity with humans' strategy. The distribution of fixation location of both BVSM and humans has weak doughnut shape when searching for low contrast target (Fig. 4A,C), and does not show such feature when searching for high contrast target (Fig. 4B,D).

BVSM and humans exhibit similar saccade amplitude (Fig. 4E) and fixation number distributions (Fig. 4F). Both of them favor small saccades. BVSM more frequently completes search with very few fixations, especially when the target is near the initial fixation location (Fig. 4G). BVSM quickly finds the target in such cases, while humans require more fixations. As the target distance increases, the search speeds of BVSM and humans become closer (Fig. 4G).

In addition, BVSM has higher response accuracy and search speed than humans for both low and high contrast targets (Table 1). In particular, it surpasses human response accuracy by about 11% for low contrast target. The model is better at searching than humans despite having lower target visibility in peripheral vision.

Although BVSM searches better, its search strategy is limited to circular scanning (Fig. S8A). While humans may use similar scanpath, they exhibit a more diverse search strategy, such as repeatedly

Figure 3: Detection task performance of BVSM and humans. **A**: In detection task, subjects are required to keep fixating at image center (red solid line), but report whether target is present at a peripheral location (blue dashed line) within a single fixation. **B**, **C**: Relationship between target eccentricity and response accuracy. In B, each gray dot represents a combination of target eccentricity, target angular position, and target contrast. In C, each gray dot represents a combination of subject index, target eccentricity, and target angular position. The black curve represents the fitted Weibull function (with inverted x-axis).

Figure 2: FEN regression results of 1024 random retinal transformed images. **A**: Relationship between predicted and true fixation location. 0\({}^{\circ}\) represents image center. **B**: Relationship between predicted and true target location within 4.0\({}^{\circ}\) away from fixation location. A and B only show the regression results for horizontal location. The results for vertical location are similar. **C**: Relationship between target distance and estimated target location error. The red horizontal line shows the decision threshold (0.58\({}^{\circ}\), 25 pixels) used in visual search.
fixating on suspected target locations (Fig. S8B, blue) and scanning the search image even when the target is close to the initial fixation location (Fig. S8B, yellow). This may explain why humans search more slowly when the target distance is small. Additionally, humans sometimes use irregular scanpaths with short saccades (Fig. S8B, green), but the reason for this is currently unclear.

Previous research has suggested that the human search strategy in this task is suboptimal due to the preference for short saccades [47], which increases the overlapping area of central vision in consecutive fixations. To improve BVSM's search performance, the radius of the IOR reward function was increased to \(2.5^{\circ}\) (similar to the size of FEN's effective visual field), and the saccade amplitude reward was changed to equation 3, which penalizes large saccades relatively weaker than equation 2. The entropy target parameter in SAC was also lowered to make the strategy less stochastic (Table S3). These modifications result in a stronger doughnut-shaped fixation strategy that is similar to the ELM searcher (Fig. S9), and the search speed is closer to the ELM searcher (Table 1, see methods in Appendix B).

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Target contrast & Percent correct & Med(FixNum) & Mean(FixNum) \\ \hline BVSM (HP 1) & 0.11–0.136 & \(97.8\pm 0.2\%\) & \(8.4\pm 0.5\) & \(13.3\pm 0.3\) \\ BVSM (HP 1) & 0.15 & \(99.5\pm 0.1\%\) & \(6.0\pm 0.0\) & \(8.4\pm 0.2\) \\ Human (n=15) & 0.11–0.136 & \(86.5\pm 6.3\%\) & \(10.0\pm 2.7\) & \(16.7\pm 4.7\) \\ Human (n=14) & 0.15 & \(97.5\pm 1.5\%\) & \(8.0\pm 1.9\) & \(13.4\pm 4.5\) \\ \hline BVSM (HP 2) & 0.11–0.136 & \(98.2\pm 0.1\%\) & \(7.0\pm 0.0\) & \(10.8\pm 0.1\) \\ ELM & \(d^{\prime}=3.0\) & \(97.8\pm 0.0\%\) & \(8.0\pm 0.0\) & \(8.8\pm 0.1\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Search performance comparison. BVSM results are averaged from models trained with 5 random seeds. Human results are averaged from all subjects. ELM results are averaged from 5 independent evaluations, each consisting of 10,000 trials. Mean and median fixation number (FixNum) are calculated from correct trials. For the ELM model, target visibility (\(d^{\prime}\)) of 3.0 corresponds to target contrast of 0.11-0.136 (Appendix A.3). HP 1 and HP 2 means hyperparameter gourp 1 and group 2. Performance data are shown in mean \(\pm\) standard deviation.

Figure 4: BVSM and human eye movement metrics. **A-D**: Distribution of fixation location. TC: target contrast. **E**. Distribution of saccade amplitude. **F**: Distribution of fixation number in correct visual search trials. **G**: Median fixation number as a function of target eccentricity in correct trials. In this figure, BVSM is trained under hyperparameter group 1 (HP 1).
### BVSM Has Low Estimated Energy Cost and High Robustness

We estimate the energy consumption of BVSM, BVSM based on ANN, and conventional ANN that process full-resolution images without making eye movements (see methods in Appendix C.8). BVSM's estimated energy consumption is two orders of magnitude lower than that of ANN counterparts, even if BVSM needs on average 10 fixations to find the target (Table S4, Fig. 5C). This is because BVSM automatically develops sparse coding during training (Fig. 5B, see methods in Appendix C.7). Most neurons have a low firing rate (Fig. 5A), and the average firing rate is only 0.089 (maximum possible firing rate is 1.0). The sparse activation in the network is similar to what was found in the animal brain [62; 63; 64; 65; 66]. The low saccade decision latency of only 4 time steps also increases the energy efficiency by preventing repeated processing of input images.

BVSM's search performance is robust to modest changes in the discount factor and the reward function (Table S5). However, removing either of the two reward functions will cause a notable slowdown in search speed, and may also decrease the response accuracy (Table S5). If we double the target's diameter, BVSM will search much faster and more accurately (Table S5).

## 5 Discussion

Benefit of non-uniform resolution retina and eye movementsAlthough non-uniform resolution retina and eye movements are common features of biological vision [1; 2; 3; 4; 5; 6], they are not widely used computer vision due to several possible reasons. Processing a single uniform-resolution image by a single forward computation is simpler, faster, and easier to train compared to selecting multiple fixations on the same image. Moreover, the use of multiple fixations prevents parallelization due to temporal dependency on successive fixations, and may offset the efficiency of non-uniform down-sampling.

However, the bio-inspired visual system has potential benefits that are yet to be explored. In real-life situations, robots do not process many single, unrelated images, but a continuous stream of visual input, so the saccade rate is very likely to be lower than the input frame rate, and making multiple fixations on the same image frame is unlikely to happen. Therefore, the benefit on computational efficiency of non-uniform sampling will be more significant. Besides, many computer vision tasks do not require the algorithm to simultaneously have large visual field, high visual resolution, low response latency, and low energy cost; but for robots that freely move in natural environments, these four features may likely be required, and the bio-inspired visual system may provide a good solution under this scenario.

For the visual search task in this research, conventional network that processes full-resolution images fails to detect the target (Appendix D), but non-uniform down-sampling can enlarge the relative

Figure 5: **A**: Distribution of neuron firing rate in BVSM. Firing rate is 1.0 if a neuron fires at every time step. TC: target contrast. **B**: The change in population sparseness of FEN during training. **C**: Estimated energy cost of BVSM (BVSM-SNN), BVSM based on ANN (BVSM-ANN), and conventional ANN that process full resolution images without making eye movements (FullRes-ANN). We assume BVSM-SNN and BVSM-ANN use on average 10 fixations to find the target.
size of the target, and make the detection easier. Another possible search method is to divide the image into several patches and process each patch until finding the target. The output of the BVSM's artificial retina is 1/8.4 of the size of the original image. If we divide the image into eight patches and examine each sequentially, the median and mean fixation number will always be 4.5. This sliding window strategy is quicker when the target size is very small and the background offers no hints of the target's location, so the peripheral vision is ineffective. However, violating either of these conditions can render the sliding window strategy inefficient.

For example, when the target's size doubles (to \(0.6^{\circ}\)), BVSM achieves 99.9% search accuracy with a median fixation number of only two fixations, outpacing the sliding window strategy. In natural images, object locations are often statistically related, so the background can provide cues to target positions. As a result, we believe BVSM is crucial for efficient visual search, particularly when a large visual field and low computational cost are needed, and when low-resolution peripheral vision can offer information about the target's location.

Benefit of SNNBVSM achieves low estimated energy cost mainly due to its sparse SNN and low saccade decision latency. Sparsity is important for avoiding spike congestion and decreasing hardware latency in real neuromorphic chips[67]. BVSM automatically learns to use sparse coding without explicitly adding sparsity constraints during training. Future studies may investigate whether the model will also develop sparse coding on more complex natural images.

Compared to previous object detection SNNs with latencies of 100-3500 time steps[48, 49, 50, 51, 68, 69], BVSM makes saccade decisions in just 4 time steps and finds the target in 10-16 fixations on average. The low saccade decision latency is achieved by using population coding, which mimics the animal brain[70]. Population coding decreases the SNN latency by only increasing the neurons in the last layer, so it is much more energy efficient than increasing the SNN latency which causes all neurons to fire more. It's a promising choice for future SNNs that encode continuous-valued output variables.

Human visual search strategyBVSM's search speed slows down with a more human-like search strategy. This provides another evidence that the human eye movement strategy is suboptimal in terms of search speed[47]. A possible cause of the suboptimality is the preference for small saccades[47], which causes a large overlap of foveal regions between successive fixations. Similar saccade amplitude distribution has been observed in a wide range of tasks and stimulus images[71, 72, 73, 74], so probably humans emphasis more on minimizing saccade cost (e.g. saccade duration and saccadic suppression) than using the strictly optimal policy. Additionally, the difficulty of the task may prompt individuals to make cautious small saccades to ensure thorough searching and avoid omissions.

Despite having better peripheral target visibility, human search accuracy is worse than the model. In detection task, subjects know the target location, so the visibility may be higher than that in visual search[20, 47]. In detection task without target location cue, the response accuracy at fovea dropped to about 95%, and the detection rate of target at periphery is still higher than the model (Fig. S10, Appendix A.4). However, in visual search which involves multiple fixations, a lower foveal visibility may cause humans to find several suspected target locations and needs to compare between them, which slows down visual search and decreases the accuracy. This was what many subjects reported after finishing the experiment.

Measuring target visibility without a target location cue may seem like a more realistic reflection of the visual search experiment. However, in a sequence of eye movements, target visibility may increase at future fixation locations, even to the level comparable to the visibility measured with target location cue[75, 76]. Therefore, the actual target visibility during visual search may fall between the visibility measured with and without target location cue, but it is difficult to measure since we can't accurately predict where humans will fixate in the future.

Limitations of BVSMBVSMis only tested on artificial images, requires GPU for training, and is not trained end-to-end, which hinders energy-saving during training. BVSM's retina is also simple compared to the human retina, and it only remembers the past fixation locations and lacks the diverse search strategies of humans, who may remember suspected target locations or scan the image for confirmation of an already found target. Future models could address these limitations by incorporating event camera input, maintaining more information in memory, and including fixational eye movement to improve coding efficiency and discrimination of fine details[77, 78]. A world model and top-down attention[79, 80] could also be added to BVSM to improve its generalizability for diverse
vision tasks. For real-life applications, we should also be careful about potential adversarial attacks that draw BVSM's attention to incorrect locations.

ConclusionWe propose BVSM, the first bio-inspired model integrating non-uniform resolution retina, efficient eye movement strategy and SNN. It excels in executing visual searches with ultra-low saccade decision latency and sparse activations, which lay the foundation for integration with neuromorphic chips to achieve high energy efficiency. The comparison with human eye movement behavior provides another piece of evidence that the human search strategy is suboptimal on certain tasks, and it also reveals the limitations of BVSM. Future research should focus on enhancing the scalability and versatility of the model to handle a variety of visual tasks using eye movements in natural images.

## References

* Clarke and Whitteridge [1976] P G Clarke and D Whitteridge. The cortical visual areas of the sheep. _The Journal of Physiology_, 256(3):497-508, 1976. doi: [https://doi.org/10.1113/jphysiol.1976.sp011335](https://doi.org/10.1113/jphysiol.1976.sp011335). URL [https://physoc.onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1976.sp011335](https://physoc.onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1976.sp011335).
* Rowe and Stone [1976] Michael H. Rowe and Jonathan Stone. Properties of ganglion cells in the visual streak of the cat's retina. _Journal of Comparative Neurology_, 169(1):99-125, 1976. ISSN 10969861. doi: 10.1002/cne.901690106. URL [https://onlinelibrary.wiley.com/doi/abs/10.1002/cne.901690106](https://onlinelibrary.wiley.com/doi/abs/10.1002/cne.901690106).
* Dunlop et al. [1987] S. A. Dunlop, W. A. Longley, and L. D. Beazley. Development of the area centralis and visual streak in the grey kangaroo macropus fuliginosus. _Vision Research_, 27(2):151-164, 1987. ISSN 00426989. doi: 10.1016/0042-6989(87)90178-7. URL [https://www.sciencedirect.com/science/article/pii/0042698987901787](https://www.sciencedirect.com/science/article/pii/0042698987901787).
* Gonzalez-Soriano et al. [1995] J. Gonzalez-Soriano, E. Rodriguez-Veiga, P. Martinez-Sainz, S. Mayayo-Vicente, and P. Marin-Garcia. A Quantitative Study of Ganglion Cells in the German Shepherd Dog Retina. _Anatomia, Histologia, Embryologia_, 24(1):61-65, 1995. ISSN 14390264. doi: 10.1111/j.1439-0264.1995.tb00010.x. URL [https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1439-0264.1995.tb00010.x](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1439-0264.1995.tb00010.x).
* Bozzano and Collin [2000] Anna Bozzano and Shaun P. Collin. Retinal ganglion cell topography in elasmobranchs. _Brain, Behavior and Evolution_, 55(4):191-208, 2000. ISSN 00068977. doi: 10.1159/000006652. URL [https://www.karger.com/DOI/10.1159/000006652](https://www.karger.com/DOI/10.1159/000006652).
* Tyrrell et al. [2013] Luke P. Tyrrell, Bret A. Moore, Christopher Loftis, and Esteban Fernandez-Juricic. Looking above the prairie: Localized and upward acute vision in a native grassland bird. _Scientific Reports_, 3, 2013. ISSN 20452322. doi: 10.1038/srep03231.
* Wisl fields [1990] Spector RH. Visual fields. In H Kenneth Walker, W Dallas Hall, and J Willis Hurst, editors, _Clinical Methods: The History, Physical, and Laboratory Examinations_, pages 565-572. Boston: Butterworths, 3rd edition, 1990.
* Curcio and Allen [1990] Christine A. Curcio and Kimberly A. Allen. Topography of ganglion cells in human retina. _Journal of Comparative Neurology_, 300(1):5-25, 1990. doi: [https://doi.org/10.1002/cne.903000103](https://doi.org/10.1002/cne.903000103). URL [https://onlinelibrary.wiley.com/doi/abs/10.1002/cne.903000103](https://onlinelibrary.wiley.com/doi/abs/10.1002/cne.903000103).
* Easter and Johns [1974] Stephen S. Easter and Pamela R. Johns. Horizontal compensatory eye movements in goldfish (carassius auratus). _Journal of comparative physiology_, 92(1):37-57, 1974. ISSN 0340-7594. doi: 10.1007/bf00696525.
* Collewijn [1977] H. Collewijn. Eye- and head movements in freely moving rabbits. _The Journal of Physiology_, 266(2):471-498, 1977. ISSN 14697793. doi: 10.1113/jphysiol.1977.sp011778.
* Paul et al. [1990] H. Paul, H. O. Nalbach, and D. Varju. Eye movements in the rock crab Pachygrapsus marmoratus walking along straight and curved paths. _Journal of Experimental Biology_, 154:81-97, 1990. ISSN 00220949. doi: 10.1242/jeb.154.1.81.
* Schilstra and Van Hateren [1998] C. Schilstra and J. H. Van Hateren. Stabilizing gaze in flying blowflies. _Nature_, 395(6703):654, 1998. ISSN 00280836. doi: 10.1038/27114.
* Ott [2001] M. Ott. Chameleons have independent eye movements but synchronise both eyes during saccadic prey tracking. _Experimental Brain Research_, 139(2):173-179, 2001. ISSN 00144819. doi: 10.1007/s002210100774.
* Land [2011] Michael F. Land. Oculomotor behaviour in vertebrates and invertebrates. In Simon P. Liversedge, Iain D. Gilchrist, and Stefan Everling, editors, _The Oxford Handbook of Eye Movements_, pages 3-15. Oxford University Press, New York, NY, 08 2011. ISBN 9780199539789. doi: 10.1093/oxfordhb/9780199539789.013.0001. URL [https://doi.org/10.1093/oxfordhb/9780199539789.013.0001](https://doi.org/10.1093/oxfordhb/9780199539789.013.0001).
* Tyrrell et al. [2015] Luke P. Tyrrell, Shannon R. Butler, and Esteban Fernandez-Juricic. Oculomotor strategy of an avian ground forager: Tilted and weakly yoked eye saccades. _Journal of Experimental Biology_, 218(16):2651-2657, 2015. ISSN 00220949. doi: 10.1242/jeb.122820.
* Land [2019] Michael Land. Eye movements in man and other animals. _Vision Research_, 162:1-7, 2019. ISSN 0042-6989. doi: [https://doi.org/10.1016/j.visres.2019.06.004](https://doi.org/10.1016/j.visres.2019.06.004). URL [https://www.sciencedirect.com/science/article/pii/S0042698919301245](https://www.sciencedirect.com/science/article/pii/S0042698919301245).
* Michaiel et al. [2020] Angie M Michaiel, Elliott TT Abe, and Cristopher M Niell. Dynamics of gaze control during prey capture in freely moving mice. _eLife_, 9:e57458, jul 2020. ISSN 2050-084X. doi: 10.7554/eLife.57458. URL [https://doi.org/10.7554/eLife.57458](https://doi.org/10.7554/eLife.57458).
* Park et al. [2022] Soon Young Park, Kenneth Holmqvist, Diederick C. Niehorster, Ludwig Huber, and Zsofia Viranyi. How to improve data quality in dog eye tracking. _Behavior Research Methods_, 2022. ISSN 15543528. doi: 10.3758/s13428-022-01788-6.
* Yarbus [1967] Alfred L. Yarbus. Eye Movements During Perception of Complex Objects. In Alfred L. Yarbus, editor, _Eye Movements and Vision_, pages 171-211. Plenum Press, 1967. doi: 10.1007/978-1-4899-5379-7_8.
* Najemnik and Geisler [2005] Jiri Najemnik and Wilson S. Geisler. Optimal eye movement strategies in visual search. _Nature_, 434(7031):387-391, 2005. ISSN 00280836. doi: 10.1038/nature03390. URL [https://www.nature.com/articles/nature03390](https://www.nature.com/articles/nature03390).
* Maass [1997] Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. _Neural Networks_, 10(9):1659-1671, 1997. ISSN 08936080. doi: 10.1016/S0893-6080(97)00011-7.
* Rajendran et al. [2019] Bipin Rajendran, Abu Sebastian, Michael Schmuker, Narayan Srinivasa, and Evangelos Eleftheriou. Low-Power Neuromorphic Hardware for Signal Processing Applications: A review of architectural and system-level design approaches. _IEEE Signal Processing Magazine_, 36(6):97-110, 2019. ISSN 15580792. doi: 10.1109/MSP.2019.2933719.
* Schuman et al. [2022] Catherine D. Schuman, Shruti R. Kulkarni, Maryam Parsa, J. Parker Mitchell, Prasanna Date, and Bill Kay. Opportunities for neuromorphic computing algorithms and applications. _Nature Computational Science_, 2(1):10-19, 2022. ISSN 26628457. doi: 10.1038/s43588-021-00184-y.
* Kety [1957] Seymour S. Kety. The General Metabolism of the Brain in Vivo. In Derek Richter, editor, _Metabolism of the Nervous System_, pages 221-237. Pergamon Press, New York, NY, 1957.
* Ranzato [2014] Marc Aurelio Ranzato. On learning where to look. _arXiv_, pages 1-14, 2014.
* Mnih et al. [2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent models of visual attention. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_, volume 27. Curran Associates, Inc., 2014. URL [https://proceedings.neurips.cc/paper/2014/file/09c6c3783b4a70054da74f2538ed47c6-Paper.pdf](https://proceedings.neurips.cc/paper/2014/file/09c6c3783b4a70054da74f2538ed47c6-Paper.pdf).
* Conference Track Proceedings_, pages 1-10, 2015. URL [http://arxiv.org/abs/1412.7755](http://arxiv.org/abs/1412.7755).
* Dabane et al. [2022] Ghassan Dabane, Laurent U Perrinet, and Emmanuel Dauce. What you see is what you transform: Foveated spatial transformers as a bio-inspired attention mechanism. In _2022 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8, 2022. doi: 10.1109/IJCNN55064.2022.9892313.
* Alexe et al. [2012] Bogdan Alexe, Nicolas Heess, Yee Teh, and Vittorio Ferrari. Searching for objects driven by context. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_, volume 25. Curran Associates, Inc., 2012. URL [https://proceedings.neurips.cc/paper/2012/file/1068c6e4c8051cf44e9ea8072e3189e2-Paper.pdf](https://proceedings.neurips.cc/paper/2012/file/1068c6e4c8051cf44e9ea8072e3189e2-Paper.pdf).
* Lyu et al. [2018] Jie Lyu, Zejian Yuan, Dapeng Chen, Yun Zhao, and Hui Zhang. Learning fixation point strategy for object detection and classification. In _2018 24th International Conference on Pattern Recognition (ICPR)_, pages 2081-2086, 2018. doi: 10.1109/ICPR.2018.8545615.
* Elsayed et al. [2019] Gamaleldin Elsayed, Simon Kornblith, and Quoc V Le. Saccader: Improving accuracy of hard attention models for vision. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc., 2019. URL [https://proceedings.neurips.cc/paper/2019/file/8dd48d6a2e2cad213179a3992c0be53c-Paper.pdf](https://proceedings.neurips.cc/paper/2019/file/8dd48d6a2e2cad213179a3992c0be53c-Paper.pdf).
* Rangrej et al. [2023] Samruddhdi B Rangrej, Kevin J Liang, Tal Hassner, and James J Clark. Glitr: Glimpse transformers with spatiotemporal consistency for online action prediction. In _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_, pages 3402-3412, Los Alamitos, CA, USA, jan 2023. IEEE Computer Society. doi: 10.1109/WACV56688.2023.00341. URL [https://doi.ieeecomputersociety.org/10.1109/WACV56688.2023.00341](https://doi.ieeecomputersociety.org/10.1109/WACV56688.2023.00341).
* Kumari and Chakravarthy [2022] Sweta Kumari and V. Srinivasa Chakravarthy. Biologically inspired image classifier based on saccadic eye movement design for convolutional neural networks. _Neurocomputing_, 513:294-317, 2022. ISSN 18728286. doi: 10.1016/j.neucom.2022.09.027.
* Najemnik and Geisler [2009] Jiri Najemnik and Wilson S. Geisler. Simple summation rule for optimal fixation selection in visual search. _Vision Research_, 49(10):1286-1294, 2009. ISSN 00426989. doi: 10.1016/j.visres.2008.12.005.
* Burton and Moorhead [1987] G. J. Burton and Ian R. Moorhead. Color and spatial structure in natural scenes. _Appl. Opt._, 26(1):157-170, Jan 1987. doi: 10.1364/AO.26.000157. URL [https://opg.optica.org/ao/abstract.cfm?URI=ao-26-1-157](https://opg.optica.org/ao/abstract.cfm?URI=ao-26-1-157).
* Dauce et al. [2020] Emmanuel Dauce, Pierre Albiges, and Laurent U. Perrinet. A dual foveal-peripheral visual processing model implements efficient saccade selection. _Journal of Vision_, 20(8):1-20, 2020. ISSN 15347362. doi: 10.1167/JOV.20.8.22.
* Akbas and Eckstein [2017] Emre Akbas and Miguel P. Eckstein. Object detection through search with a foveated visual system. _PLoS Computational Biology_, 13(10):e1005743, 2017. ISSN 15537358. doi: 10.1371/journal.pcbi.1005743. URL [https://doi.org/10.1371/journal.pcbi.1005743](https://doi.org/10.1371/journal.pcbi.1005743).
* Rangrej et al. [2022] Samruddhdi B. Rangrej, Chetan L. Srinidhi, and James J. Clark. Consistency driven sequential transformers attention model for partially observable scenes. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2508-2517, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: 10.1109/CVPR52688.2022.00255. URL [https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00255](https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00255).
* McCarley et al. [2003] Jason S McCarley, Ranziao F Wang, Arthur F Kramer, David E Irwin, and Matthew S Peterson. How much memory does oculomotor search have? _Psychological science_, 14(5):422-6, 2003. ISSN 0956-7976. doi: 10.1111/1467-9280.01457. URL [http://www.ncbi.nlm.nih.gov/pubmed/12930471](http://www.ncbi.nlm.nih.gov/pubmed/12930471).
* Beck et al. [2006] Melissa R. Beck, Matthew S. Peterson, and Miroslava Vomela. Memory for where, but not what, is used during visual search. _Journal of Experimental Psychology: Human Perception and Performance_, 32(2):235-250, 2006. ISSN 00961523. doi: 10.1037/0096-1523.32.2.2.235.
* Dickinson and Zelinsky [2007] Christopher A. Dickinson and Gregory J. Zelinsky. Memory for the search path: Evidence for a high-capacity representation of search history. _Vision Research_, 47(13):1745-1755, 2007. ISSN 00426989. doi: 10.1016/j.visres.2007.02.010.
* Kaunitz et al. [2016] Lisandro N. Kaunitz, Elise G. Rowe, and Naotsugu Tsuchiya. Large Capacity of Conscious Access for Incidental Memories in Natural Scenes. _Psychological Science_, 27(9):1266-1277, 2016. ISSN 14679280. doi: 10.1177/0956797616658869.
* Rangrej and Clark [2021] Samrudhdih B. Rangrej and James J. Clark. A probabilistic hard attention model for sequentially observed scenes. _arXiv e-prints_, art. arXiv:2111.07534, November 2021. doi: 10.48550/arXiv.2111.07534.
* Everingham et al. [2007] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. [http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html](http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html), 2007.
* Everingham et al. [2010] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. [http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html](http://www.pascal-network.org/challenges/VOC/voc2010/workshop/index.html), 2010.
* Najemnik and Geisler [2008] Jiri Najemnik and Wilson S. Geisler. Eye movement statistics in humans are consistent with an optimal search strategy. _Journal of Vision_, 8(3):1-14, 2008. ISSN 1534-7362. doi: 10.1167/8.3.4. URL [https://jov.arvojournals.org/article.aspx?articleid=2122647](https://jov.arvojournals.org/article.aspx?articleid=2122647).
* Zhou and Yu [2021] Yunhui Zhou and Yuguo Yu. Human visual search follows a suboptimal Bayesian strategy revealed by a spatiotemporal computational model and experiment. _Communications Biology_, 4(1):1-16, 2021. ISSN 23993642. doi: 10.1038/s42003-020-01485-0. URL [http://dx.doi.org/10.1038/s42003-020-01485-0](http://dx.doi.org/10.1038/s42003-020-01485-0).
* Kim et al. [2020] Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: Spiking neural network for energy-efficient object detection. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(07):11270-11277, Apr. 2020. doi: 10.1609/aaai.v3a4i07.6787. URL [https://ojs.aaai.org/index.php/AAAI/article/view/6787](https://ojs.aaai.org/index.php/AAAI/article/view/6787).
* Kim et al. [2021] Seijoon Kim, Seongsik Park, Byunggook Na, Jongwan Kim, and Sungroh Yoon. Towards Fast and Accurate Object Detection in Bio-Inspired Spiking Neural Networks through Bayesian Optimization. _IEEE Access_, 9:2633-2643, 2021. ISSN 21693536. doi: 10.1109/ACCESS.2020.3047071.
* Chakraborty et al. [2021] Biswadeep Chakraborty, Xueyuan She, and Saibal Mukhopadhyay. A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection. _IEEE Transactions on Image Processing_, 30:9014-9029, 2021. ISSN 19410042. doi: 10.1109/TIP.2021.3122092.
* Miquel et al. [2021] Joaquin Royo Miquel, Silvia Tolu, Frederik E.T. Scholler, and Roberto Galeazzi. RetinaNet Object Detector Based on Analog-to-Spiking Neural Network Conversion. _2021 8th International Conference on Soft Computing and Machine Intelligence, ISCMI 2021_, pages 201-205, 2021. doi: 10.1109/ISCMI53840.2021.9654818.
* Shen et al. [2022] Guobin Shen, Dongcheng Zhao, and Yi Zeng. Backpropagation with biologically plausible spatiotemporal adjustment for training deep spiking neural networks. _Patterns_, 3(6):100522, 2022. ISSN 26663899. doi: 10.1016/j.patter.2022.100522. URL [https://doi.org/10.1016/j.patter.2022.100522](https://doi.org/10.1016/j.patter.2022.100522).
* Patel et al. [2019] Devdhar Patel, Hananel Hazan, Daniel J. Saunders, Hava T. Siegelmann, and Robert Kozma. Improved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to Atari Breakout game. _Neural Networks_, 120:108-115, 2019. ISSN 18792782. doi: 10.1016/j.neunet.2019.08.009.
* Tang et al. [2021] Guangzhi Tang, Neelesh Kumar, Raymond Yoo, and Konstantinos P. Michmizos. Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control. _Proceedings of the 2020 Conference on Robot Learning_, 155:2016-2029, 2021. URL [http://arxiv.org/abs/2010.09635](http://arxiv.org/abs/2010.09635).
* Xu et al. [2022] Rui Xu, Yifei Wu, Xiaoling Qin, and Peng Zhao. Population-coded spiking neural network with reinforcement learning for mapless navigation. _Proceedings of the International Conference on Cyber-Physical Social Intelligence, ICCSI 2022_, pages 518-523, 2022. doi: 10.1109/ICCSI55536.2022.9970598.
* Martinez and Altamirano [2006] Jose Martinez and Leopoldo Altamirano. A new foveal cartesian geometry approach used for object tracking. In _Proceedings of the 24th IASTED International Conference on Signal Processing, Pattern Recognition, and Applications_, SPPRA'06, page 133-139, USA, 2006. ACTA Press. ISBN 0889865809.
* Bu et al. [2021] Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, and Tiejun Huang. Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks. In _International Conference on Learning Representations_, 2021.
* Baloh et al. [1975] Robert W. Baloh, Andrew W. Sills, Warren E. Kumley, and Vicente Honrubia. Quantitative measurement of saccade amplitude, duration, and velocity. _Neurology_, 25(11):1065-1070, 1975. ISSN 1526632X. doi: 10.1212/wnl.25.11.1065.
* Bridgeman et al. [1975] Bruce Bridgeman, Derek Hendry, and Lawrence Stark. Failure to detect displacement of the visual world during saccadic eye movements. _Vision Research_, 15(6):719-722, 1975. ISSN 00426989. doi: 10.1016/0042-6989(75)90290-4.
* Haarnoja et al. [2018] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and Applications. _arXiv_, 2018. URL [http://arxiv.org/abs/1812.05905](http://arxiv.org/abs/1812.05905).
* Yujie et al. [2018] Wu Yujie, Deng Lei, Li Guoqi, Zhu Jun, and Shi Luping. Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks. _Frontiers in Neuroscience_, 12:1-12, 2018.
* Vinje and Gallant [2000] William E. Vinje and Jack L. Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. _Science_, 287(5456):1273-1276, 2000. ISSN 00368075. doi: 10.1126/science.287.5456.1273.
* Weliky et al. [2003] Michael Weliky, Jozsef O. Fiser, Ruskin H. Hunt, and David N. Wagner. Coding of natural scenes in primary visual cortex. _Neuron_, 37(4):703-718, 2003. ISSN 08966273. doi: 10.1016/S0896-6273(03)00022-9.
* Franco et al. [2007] Leonardo Franco, Edmund T. Rolls, Nikolaos C. Aggelopoulos, and Jose M. Jerez. Neuronal selectivity, population sparseness, and ergodicity in the inferior temporal visual cortex. _Biological Cybernetics_, 96(6):547-560, 2007. ISSN 03401200. doi: 10.1007/s00422-007-0149-1.
* Froudarakis et al. [2014] Emmanouil Froudarakis, Philipp Berens, Alexander S. Ecker, R. James Cotton, Fabian H. Sinz, Dimitri Yatsenko, Peter Saggau, Matthias Bethge, and Andreas S. Tolias. Population code in mouse V1 facilitates readout of natural scenes through increased sparseness. _Nature Neuroscience_, 17(6):851-857, 2014. ISSN 15461726. doi: 10.1038/nn.3707.
* Yoshida and Ohki [2020] Takashi Yoshida and Kenichi Ohki. Natural images are reliably represented by sparse and variable populations of neurons in visual cortex. _Nature Communications_, 11(1), 2020. ISSN 20411723. doi: 10.1038/s41467-020-14645-x.
* Peres and Rhodes [2022] Luca Peres and Oliver Rhodes. Parallelization of Neural Processing on Neuromorphic Hardware. _Frontiers in Neuroscience_, 16:1-20, 2022. ISSN 1662453X. doi: 10.3389/fnins.2022.867027. URL [https://www.frontiersin.org/articles/10.3389/fnins.2022.867027](https://www.frontiersin.org/articles/10.3389/fnins.2022.867027).
* Kugele et al. [2021] Alexander Kugele, Thomas Pfeil, Michael Pfeiffer, and Elisabetta Chicca. Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision. In Christian Bauckhage, Juergen Gall, and Alexander Schwing, editors, _Pattern Recognition_, volume 13024 LNCS, pages 297-312. Springer International Publishing, 2021. ISBN 9783030926588. doi: 10.1007/978-3-030-92659-5_19.
* Shen et al. [2022] Li Shen, Ma Zhong, and Yang Chaojie. Conversion of CNN to SNN for Aerospace Object Detection Tasks. _2022 5th International Conference on Pattern Recognition and Artificial Intelligence, PRAI 2022_, pages 919-923, 2022. doi: 10.1109/PRAI55851.2022.9904132.
* Averbeck et al. [2006] Bruno B Averbeck, Peter E Latham, and Alexandre Pouget. Neural correlations, population coding and computation. _Nature reviews neuroscience_, 7(5):358-366, 2006.
* Bahill et al. [1975] A. T. Bahill, D. Adler, and L. Stark. Most naturally occurring human saccades have magnitudes of 15 degrees or less. _Investigative Ophthalmology_, 14(6):468-469, 1975. ISSN 00209988.
* Castelhano et al. [2009] Monica S. Castelhano, Michael L. Mack, and John M. Henderson. Viewing task influences eye movement control during active scene perception. _Journal of Vision_, 9(3), 2009. ISSN 15347362. doi: 10.1167/9.3.6.
* Wang et al. [2011] Wei Wang, Cheng Chen, Yizhou Wang, Tingting Jiang, Fang Fang, and Yuan Yao. Simulating human saccadic scanpaths on natural images. _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_, pages 441-448, 2011. ISSN 10636919. doi: 10.1109/CVPR.2011.5995423.
* Devillez et al. [2015] Helene Devillez, Anne Guerin-Dugue, and Nathalie Guyader. How task difficulty influences eye movements when exploring natural scene images. _2015 23rd European Signal Processing Conference, EUSIPCO 2015_, pages 1536-1540, 2015. doi: 10.1109/EUSIPCO.2015.7362641.
* Gersch et al. [2004] Timothy M. Gersch, Eileen Kowler, and Barbara Dosher. Dynamic allocation of visual attention during the execution of sequences of saccades. _Vision Research_, 44(12):1469-1483, 2004. ISSN 00426989. doi: 10.1016/j.visres.2003.12.014.
* Baldauf and Deubel [2008] Daniel Baldauf and Heiner Deubel. Properties of attentional selection during the preparation of sequential saccades. _Experimental Brain Research_, 184(3):411-425, 2008. ISSN 00144819. doi: 10.1007/s00221-007-1114-x.
* Rucci et al. [2007] Michele Rucci, Ramon Iovin, Martina Poletti, and Fabrizio Santini. Miniature eye movements enhance fine spatial detail. _Nature_, 447(7146):852-855, 2007. ISSN 0028-0836. doi: 10.1038/nature05866. URL [http://www.nature.com/articles/nature05866](http://www.nature.com/articles/nature05866).
* Segal et al. [2015] Irina Yonit Segal, Chen Giladi, Michael Gedalin, Michele Rucci, Mor Ben-Tov, Yam Kushinsky, Alik Moketichev, and Ronen Segev. Decorrelation of retinal response to natural scenes by fixational eye movements. _Proceedings of the National Academy of Sciences_, 112(10):3110-3115, 2015. ISSN 0027-8424. doi: 10.1073/pnas.1412059112. URL [http://www.pnas.org/lookup/doi/10.1073/pnas.1412059112](http://www.pnas.org/lookup/doi/10.1073/pnas.1412059112).
* Gazzalezy and Nobre [2012] Adam Gazzalezy and Anna C. Nobre. Top-down modulation: Bridging selective attention and working memory. _Trends in Cognitive Sciences_, 16(2):129-135, 2012. ISSN 13646613. doi: 10.1016/j.tics.2011.11.014. URL [http://dx.doi.org/10.1016/j.tics.2011.11.014](http://dx.doi.org/10.1016/j.tics.2011.11.014).
* Mittal et al. [2020] Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. In Hal Daume III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 6972-6986. PMLR, 13-18 Jul 2020. URL [https://proceedings.mlr.press/v119/mittal20a.html](https://proceedings.mlr.press/v119/mittal20a.html).
Human Experiment Details

### Human Subjects

29 human subjects (14 females, aged 19-30 years) participated in the study for monetary compensation. All subjects were tested to have normal or corrected-to-normal vision, and reported no neurological or psychological disease history. This study was approved by the Ethics Committee of our institution, and all subjects provided written informed consent.

### Apparatus

We used a 24.5-inch BenQ ZOWIE XL2540 LCD monitor (resolution 1920x1080, physical size 54.3744x30.2616 cm, refresh rate 240 Hz) to present the stimuli. Stimuli were generated by custom MATLAB (MathWorks) scripts and presented using MATLAB and the Psychophysics Toolbox [81, 82, 83]. Subjects were seated 70 cm away from the screen, and their heads were fixed by a forehead and chin rest. Binocular eye movements were recorded at 2,000 Hz with a TRACKPixx3 eye-tracker (Vpixx Technologies, Canada). Before each experiment, subjects first performed a 13-point calibration until the average test-retest measurement error of both eyes fell below 0.5\({}^{\circ}\). The eye-tracker was re-calibrated whenever the eye-tracking accuracy failed to meet the experiment requirements (see below).

### Target Contrast Selection Experiment

This experiment used a two-interval-force-choice (2IFC) task to measure the relationship between foveal target visibility (\(d^{\prime}\) in signal detection theory [84]) and target contrast, and selected a target contrast for each individual that has the same foveal visibility. 15 subjects participated in the experiment. The details of the procedure were reported in [85].

In each trial, subjects fixated at image center and viewed two stimulus images for 250 ms each. One image was pure background noise, and the other was background noise with a target embedded at image center. Subjects had to select the image with the target. Within each trial, we used the eye-tracker to continuously monitor subject's fixation location. If the subject fixated more than 1\({}^{\circ}\) away from the image center, the trial would be aborted. We tested 4-5 levels of target contrast (ranging from 0.03 to 0.12), and each contrast level had 200 valid trials. In each trial, the target contrast is randomly selected from these levels. Visibility is calculated as [84]:

\[d^{\prime}=\frac{z(p(\text{Hit}))-z(1-p(\text{CR}))}{\sqrt{2}},\] (S1)

where \(p(\text{Hit})\) is the hit rate, \(p(\text{CR})\) is the correct rejection rate, and \(z(x)\) is the inverse standard normal cumulative distribution function. The hit (or correct rejection) rate was defined as the number of trials that target appeared in the first (or second) interval, and the subject made a correct response, divided by the total number of trials that target appeared in the first (or second) interval.

For each subject, we selected the target contrast that had a foveal visibility of 3.0 and used it for all following experiments. This was done by first fitting a Weibull function between the hit and correct rejection rate and target contrast, and then use equation S1 to find the target contrast that makes \(d^{\prime}=3.0\). The Weibull function is fitted by maximum likelihood estimation implemented in the Palamedes Toolbox [83]. Therefore, each subject had a different target contrast (ranged 0.11-0.136).

### Target Visibility Map Experiment

This experiment measured the target visibility as a function of retinal location and fixation duration. The procedure was the same as the first task, except that the target contrast was fixed at the selected level, and the target was presented at different locations for different durations. The details of the procedure were reported in [85].

The experiment had three versions. Four subjects participated in the first version, four subjects participated in the second version (one participated in both the first and second versions), and the rest eight subjects participated in the third version.
In the first version, the subjects were required to fixate at the image center and identify which of the two stimulus intervals contained a target at a cued location. The experiment contained five sessions. In each session, the target appeared in one of the four cardinal directions relative to the image center, or appeared at the image center. We chose 3-4 different target locations (except when the target was at the image center) and 4-5 levels of stimulus duration in each session. When the target appeared at the image center, the stimulus duration was chosen from 4 to 200 ms, and each level had 100 trials. When the target appeared at peripheral locations, the stimulus duration was chosen from 50 to 700 ms, and each combination of stimulus duration and target location had 50 trials. There were about 5000 trials in total. The tested target locations were determined for each subject by a pilot experiment.

The second version was the same as the first version, except that the target location cue was not shown before each trial, and the target location and stimulus duration were randomly chosen from all possible combinations in each trial. There were about 5000 trials in total.

The third version was the same as the first version, except that there was only one level of stimulus presentation time (250 ms). It was used to measure the visibility map efficiently and give subjects some training to prepare for the visual search experiment.

Subjects should maintain the fixation at the image center in each trial. We used the eye-tracker to continuously monitor subject's fixation location. If the subject fixated more than 1\({}^{\circ}\) away from the image center, the trial would be aborted. Aborted trials were not counted in the number of trials that the subject has finished.

### Target Visibility Data Analysis Method

We use the visual information accumulation model proposed by Zhou and Yu[85] to fit the experimentally measured target visibility data. The model describes a leaky integration of visual signals at each location over time within a fixation, and the relationship between target visibility \(d^{\prime}\) and location \((x,y)\) and fixation duration \(T\) can be written as:

\[a(x,y)=p_{1}\cdot\exp\left(-p_{2}\cdot\sqrt{x^{2}+p_{5}\cdot y^{ 2}}\right),\] (S2) \[k(x,y)=p_{3}\cdot\exp\left(-p_{4}\cdot\sqrt{x^{2}+p_{5}\cdot y^ {2}}\right),\] (S3) \[d^{\prime}(x,y,T)=a(x,y)\sqrt{\frac{1-\exp(-k(x,y)T)}{k(x,y) \left[1+\exp(-k(x,y)T)\right]}},\] (S4)

where \(p_{1}\sim p_{5}\) are free parameters. The values of these parameters were fitted by minimizing the mean-squared error between the calculated and measured visibility data by the GlobalSearch algorithm in MATLAB's Global Optimization Toolbox.

### Visual Search Experiment

In visual search task, subjects were told to search for the target as fast and accurately as possible, but there is no upper limit on search time. For the 15 subjects that participated in the detection task, the target contrast was the individually selected contrast (ranged 0.11-0.136, called low target contrast group). The rest 14 subjects did not participate in the detection task, and the target contrast was fixed to 0.15 (called high target contrast group).

Each trial started by fixating within 0.8\({}^{\circ}\) away from the image center, and the target location was sampled uniformly within the region of background noise. Subjects were informed that the target could appear at anywhere within the image, even right at the initial fixation location. Subjects ended the trial by fixating on the target location and pressing a button. If the final fixation location is less than 1\({}^{\circ}\) away from the target, the trial is labeled as correct. Subjects first practiced for 100-150 trials, and performed 200-400 formal trials depending on their available time. We would re-calibrate the eye-tracker if subjects reported that they had correctly found the target, but the trial was wrong.

### Eye Tracking Data Analysis Method

We analyzed eye-tracking data by the EYE-EEG extension[86] of the EEGLAB toolbox[87]. We used a velocity-based algorithm with adaptive threshold[88] to classify eye movements into saccades,
fixations, and blinks. Fixations separated by blinks were classified as two fixations. Glissades were merged into fixations. We used the minimum instantaneous eye movement velocity of both eyes in the eye movement event detection algorithm to suppress intermittent noise in the data of a single eye.

Since each subject finished a different number of trials, when summarizing eye movement statistics, we first summarize the eye movement statistics within each subject, then average the statistics across different subjects.

## Appendix B Bayesian Optimal Visual Search Model

We use the entropy-limit-minimization (ELM) searcher[89] as the optimal visual search model. The model's input is not the actual search image, but random signals generated according to human target visibility data. According to signal detection theory[84], we assume that the visual signals obtained within one fixation from one location of the visual field obey a normal distribution. The presence of the search target at this location will shift the mean of the distribution, but will not change the variance. Visibility \(d^{\prime}\) is also defined as the difference between the mean divided by the common standard deviation.

Theoretically, the ELM model should be able to fixate at any location in the image. In practice, to decrease the computational cost, the model could only fixate at 400 locations uniformly sampled inside the search image. We set the visual signal \(W_{i,L_{F}}\) obtained at location \(i\) (\(i=1\cdots n,n=400\)) during fixation \(F\) at location \(L_{F}\) obeys the normal distribution \(N(\pm 0.5,1/d_{i,L_{F}}^{2})\), where the mean is positive only when the target is at location \(i\). \(d_{i,L_{F}}^{\prime}\) is calculated by equation S4, and the parameter \(T\) in equation S4 is fixed to 250 ms (close to the average fixation duration). Therefore, the visual signals gathered from all 400 locations is a vector \(\textbf{W}_{L_{F}}=(W_{1,L_{F}},\cdots,W_{n,L_{F}})\). The posterior probability of target location given the visual signals gathered from all fixations can be calculated as:

\[P_{i,F} =P\left(i\mid\textbf{W}_{L_{1}},\cdots\textbf{W}_{L_{F}}\right)\] \[=\frac{P\left(\textbf{W}_{L_{1}},\cdots\textbf{W}_{L_{F}}\mid i \right)p(i)}{\sum_{j=1}^{n}\left[P\left(\textbf{W}_{L_{1}},\cdots\textbf{W}_{ L_{F}}\mid j\right)p(j)\right]}\] \[=\frac{e^{\sum_{f=1}^{F}W_{i,L_{f}}d_{i,L_{f}}^{2}}\cdot p(i)}{ \sum_{j=1}^{n}\left[e^{\sum_{f=1}^{F}W_{j,L_{f}}d_{j,L_{f}}^{2}}\cdot p(j) \right]}\] \[=\frac{P_{i,F-1}\cdot e^{W_{i,L_{F}}d_{i,L_{F}}^{2}}}{\sum_{j=1}^ {n}\left(P_{j,F-1}\cdot e^{W_{j,L_{F}}d_{j,L_{F}}^{2}}\right)},\] (S5)

where \(p(i)=1/400\) is the prior probability of target location.

The ELM searcher chooses the location that maximizes the expected information gained from the next fixation as the next fixation location. This can be expressed as:

\[H_{L_{F}}=-\sum_{i=1}^{n}\left[P_{i,F}\cdot\ln P_{i,F}\right],\] \[L_{F+1}=\underset{L_{F+1}}{\text{argmax}}\left\{H_{L_{F}}-E \left[H_{L_{F+1}}\right]\right\},\] (S6)

where \(E[\cdot]\) represents expectation with respect to the visual signal obtained in the next fixation. \(E\left[H_{L_{F+1}}\right]\) is an unknown variable that should be estimated, and \(H_{L_{F}}\) is a known variable that can be calculated from the posterior probability of target location. Najemnik and Geisler[89] derive a simple way to estimate \(E\left[H_{L_{F+1}}\right]\), and found that the next fixation location can be selected as:

\[L_{F+1}=\underset{L_{F+1}}{\text{argmax}}\left\{\frac{1}{2}\sum_{i=1}^{n} \left(P_{i,F}\cdot d_{i,L_{F+1}}^{\prime 2}\right)\right\}.\] (S7)

When the posterior probability of target being at current fixation location exceeds a threshold \(\theta_{T}\), the ELM model sets the current fixation location as the responded target location. \(\theta_{T}\)is a constant value determined by matching the model's search accuracy to BVSM's search accuracy. In addition, the
maximum number of fixations per trial is 200, and if this number is reached, the model will set the last fixation location as the responded target location. If the responded target location matches the actual target location, the trial is considered to be a correct trial.

## Appendix C Bio-inspired Visual Search Model

### Artificial Retina

We modified the original Foveal Cartesian Geometry[90] to make the transition of sampling density between central and peripheral vision to be more smooth. We define the radius (half of the width in pixels) \(r\) of a series of sampling rings (indexed by \(i\) starting from 1) as:

\[r(i)=\begin{cases}i,&i\leq i_{\text{fovea}}\\ a^{i+b}+c,&i>i_{\text{fovea}}\end{cases}\] (S8)

where _a, b, c_ are parameters, and \(i_{\text{fovea}}\) is the radius (in pixels) of fovea in the output image. The three parameters in equation S8 should satisfy three constraints: 1. The two functions should intersect at \(i_{\text{fovea}}\); 2. \(r(i)\) should be smooth at \(i_{\text{fovea}}\) (first-order derivative is equal); 3. The radius of the outermost ring should be a predefined value \(r_{\max}\). This means that:

\[\begin{cases}a^{i_{\text{fovea}}+b}+c=i_{\text{fovea}}\\ a^{i_{\text{fovea}}+b}\cdot\ln a=1\\ a^{i_{\max}+b}+c=r_{\max}\end{cases}\] (S9)

In practice, the systems of non-linear equations can be solved by a modification of the Levenberg-Marquardt algorithm implemented in SciPy (_scipy.optimize.root(method='lm')_)[91]. We set the size of output image to be 224x224 with a fovea size of \(16\times 16\) pixels (\(i_{\max}=112,\ i_{\text{fovea}}=8\)). To make sure the entire search image is always within the visual field (as is the case for humans), we set \(r_{\max}\) to be twice the radius of the search image.

On each ring, we uniformly sample (\(8*i-4\)) number of pixels. This means that pixels within the fovea are directly copied to the output, and pixels outside of the fovea are sampled progressively sparser. The sampled pixels are stacked together to form a square image as the output. We implemented the above retinal transformation by using the Kornia package[92].

### Feature Extraction Network

The feature extraction network contains 7 convolution blocks. The first convolution block uses 7x7 convolution, and subsequent blocks use 3x3 convolution. The number of output channels of the \(i^{th}\) block is 16x\(i\). We use replicate padding in convolution, and down-sampling is achieved by setting stride \(=2\). The output of the last convolution block is flattened into a 448-dimensional vector and fed into the three fully connected heads. Each fully connected head contains one layer of neurons.

To convert FEN to SNN, we directly replace the QCFS function with the scaled integrate-and-fire (IF) neuron[93]:

\[\begin{cases}\widetilde{\mathbf{v}}_{l,t}=\mathbf{v}_{l,t-1}+\frac{\mathbf{W}_{l}\mathbf{ s}_{l-1,t}+\mathbf{b}_{l}}{\lambda}\\ \widetilde{\mathbf{s}}_{l,t}=H\left(\widetilde{\mathbf{v}}_{l,t}-\theta\right)\\ \mathbf{s}_{l,t}=\lambda\widetilde{\mathbf{s}}_{l,t}\\ \mathbf{v}_{l,t}=\widetilde{\mathbf{v}}_{l,t}-\widetilde{\mathbf{s}}_{l,t}\theta_{l}\end{cases}\] (S10)

where \(l\) is the layer index, \(t\) is the SNN inference time index, \(\mathbf{v}_{l,t}\) is the membrane potential of IF neurons in layer \(l\) at time \(t\) (\(\mathbf{v}_{l,0}=0.5\)), \(\mathbf{s}_{l,t}\) is the scaled output spikes from layer \(l\) at time \(t\), \(\theta=1\) is the firing threshold, \(H\left(\cdot\right)\) is the Heaviside step function, \(\mathbf{W}_{l}\) and \(\mathbf{b}_{l}\) are the connection weights and bias from layer \(l-1\) to layer \(l\), and \(\lambda\) is the parameter \(\lambda\) in QCFS function. When a neuron fires, we reset its membrane potential by subtracting \(\theta\) to prevent information loss[94].

We directly use the pixel values (repeated for \(T=4\) time steps) as the network's input. Given input images with batch size \(B\), the output of the converted SNN has dimension \((T\times B\times 5)\).
### Recurrent Spiking Network

Denote SNN inference time as \(T\) and batch size as \(B\). At each fixation \(l\), the network takes the predicted fixation location \(\mathbf{x}_{l}\) (\(T\times B\times 2\)) and the RNN hidden state \(\mathbf{h}_{l}\) (\(B\times 64\)) as input. Within each fixation, at the first SNN time step, the network linearly transforms \(\mathbf{x}_{l}[1,:,:]\) and \(\mathbf{h}_{l}\). The sum of the results is then passed into IF neurons and produces output spikes \(\mathbf{s}_{l}[1,:,:]\). At subsequent time steps \(t\), the network only linearly transforms \(\mathbf{x}_{l}[t,:,:]\) and produce the output spikes \(\mathbf{s}_{l}[t,:,:]\). The output spikes from the last time step become the RNN hidden state \(\mathbf{h}_{l+1}\) for the next fixation. This computation process is also shown in Algorithm S1. The IF neurons are reset to the initial membrane potential (0.5) at the start of each fixation. In practice, the last linear transform in FEN and the initial linear transform in RNN can be merged together, allowing spikes to straightforwardly transmit from FEN to RNN.

```
Input:\(\mathbf{x}\) with dimension \((T,B,L,2)\), initial RNN hidden state \(\mathbf{h}_{0}\) with dimension \((B,64)\) (filled with zeros). \(T\) is the SNN inference time. \(B\) is batch size. \(L\) is the number of fixations.  Initialize output spikes \(\mathbf{s}\) with dimension \((T,B,L,64)\).  Load linear transform weight matrix \(\mathbf{W}_{xr}\), \(\mathbf{W}_{rr}\) and bias \(\mathbf{b}_{x}\), \(\mathbf{b}_{r}\). for\(l=1\)to\(L\)do  Reset IF neuron membrane potential to 0.5. for\(t=1\)to\(T\)do \(\mathbf{y}=\mathbf{x}[t,:,l,:]\cdot\mathbf{W}_{xr}+\mathbf{b}_{x}\) ift==1then \(\mathbf{y}=\mathbf{y}+\mathbf{h}_{l-1}\cdot\mathbf{W}_{rr}+\mathbf{b}_{r}\) endif \(\mathbf{s}[t,:,l,:]=\text{IF}(\mathbf{y})\) according to equation S10.  Output spikes \(\mathbf{s}[t,:,l,:]\). endfor \(\mathbf{h}_{l}=\mathbf{s}[T,:,l,:]\) endfor Output: RNN hidden state \(\mathbf{h}_{L}\)
```

**Algorithm S1** Recurrent spiking network

### Actor and Critic Networks

The actor network makes saccade decisions. If the estimated target location error from FEN is lower than a threshold (0.58\({}^{\circ}\), or 25 pixels), the next fixation location will be sampled from a 2-dimensional Gaussian distribution centered at the predicted target location with a small symmetrical covariance matrix of 15 pixel\({}^{2}\); otherwise the actor will pass the RNN output into two layers of 480-dimension IF neurons (equation S10), and linearly transform the result into a 5-dimensional vector (population coding), which encodes the mean and covariance of the next fixation location's distribution (Fig. S1D). The fixation location output by the actor can be anywhere within the square region enclosing the circular search image.

The critic network measures saccade value (Fig. S1E). If the estimated target location error is low, it will concatenate the next fixation location with the predicted absolute target location and output a Q-value. Otherwise it will concatenate the next fixation location with RNN output, and calculate another Q-value. The critic network is not a spiking network, as it is not used in inference after training. Therefore, the output from the RNN needs to be averaged over the first dimension (representing SNN time steps) before concatenation.

### Detection Task by FEN

We use a 2IFC task similar to human experiment to measure the FEN's ability to correctly locate the target as a function of target eccentricity. In each trial the network is presented with two images, one contains the background alone, and the other contains the background and a target. We select the image whose estimated target location error is below the detection threshold (25 pixels), and the predicted target location is less than 1\({}^{\circ}\) away from the true target location. We use the true target location in decision rule because the target location is cued in human experiment. If the decision
rule is not matched for both images, then we will randomly select an image. A trial is correct if the selected image contains the target.

We measure the correct response rate of 6 target contrasts (0.11 to 0.135, step 0.005), 49 target eccentricities (0.0 to 0.98 of the image radius, step 0.02), and 8 target angular positions (0\({}^{\circ}\) to 315\({}^{\circ}\), step 45\({}^{\circ}\)). Each combination of conditions contains 400 trials. The relationship between proportion correct and target distance is fitted by the Weibull function (with inverted x-axis, because Weibull function is monotonically increasing) by maximum likelihood estimation implemented in the Palamedes toolbox [83].

### BVSM Training and Evaluation Method

In the first training stage, we train the FEN with QCFS function as activation function by supervised learning. The network is trained for 174,000 optimization steps. In the second training stage, we convert the trained FEN to spiking FEN, and then fine-tune the spiking network for 35,000 optimization steps by supervised learning. Fine-tuning is done by STBP implemented in spikingjelly [95]. All parameters remain trainable. We use the Atan function to approximate the gradient of a spike [96].

The training data in the first two stages are random samples of retinal transformed images generated at runtime during training. We first generate random samples of full-resolution background noise images. For each image, we randomly set a fixation location uniformly within the circular region. We then randomly set one target location per image whose distance to fixation location obeys exponential distribution with a mean of 4\({}^{\circ}\). The target location has to be within the circular noise image, otherwise we will resample another target location. This ensures that enough training samples contain the target near the foveal region and helps the network to converge more reliably.

In the third training stage, we fix the FEN, and use SAC [97] algorithm to train the RNN, actor, and critic networks to do visual search. The SAC agent contains one FEN, one RNN with one target network, one actor, and two critics with two target networks. We use a replay buffer of 50,000 trials (each trial can have a variable number of fixations). The state vector stored in replay buffer is the FEN output in each fixation. Optimization started after the initial 333 trials, and is performed once per fixation. In each optimization, we randomly sample 32 trials (with variable number of fixations) from the replay buffer, update critic and RNN by temporal difference learning, update actor alone by gradient ascent through the critic output, and finally update the SAC temperature parameter according to [97]. The gradients in RNN and actor are calculated by STBP and surrogate gradient (Atan function). The gradients of RNN and critic are clipped to a maximum norm of 1.0. The number of training trials is 50,000. A list of hyperparameters can be found in Table S3.

During training, target contrast is randomly chosen within [0.11, 0.15]. Target and initial fixation locations are randomly chosen within the circular search image. All three stages of training were performed on a computer with Intel i9 9900K CPU and NVIDIA GeForce RTX 2080Super GPU. The first and second stages required less than 5 hours, and the third stage required about 20 hours. The code is implemented in PyTorch [98].

During evaluation, target location is randomly chosen within the circular search image, but the initial fixation location is randomly chosen within 0.35\({}^{\circ}\) away from the image center. This is similar to human experiment where a visual search trial always started from fixation at the image center. The target contrast is chosen within [0.11, 0.136] (low target contrast) or fixed to 0.15 (high target contrast) during evaluation, corresponding to the target contrast range in human experiment. The model is evaluated for 10,000 visual search trials for each target contrast level.

### Firing Rate and Sparseness Analysis Method

We record the activities of every neuron in BVSM during evaluation. The firing rate of each neuron is calculated as the number of time steps that the neuron fires divided by the total number of time steps in 10,000 visual search trials. The total number of time steps is the total number of fixations multiplied by SNN latency (4 time steps) in each fixation. Therefore, the firing rate of each neuron falls in the region between 0 and 1.

BVSM has 359,504 neurons in total, and 99.72% of them are in the FEN. To calculate the change of population sparseness during training, we calculate the change of population sparseness during the first training stage of FEN. Note that in the first training stage, the FEN is not an SNN, but the
QCFS activation function [93] outputs the firing rate (scaled by \(\lambda\) in QCFS function) of each neuron. Therefore, we can calculate the population sparseness [99, 100, 101] by the unscaled QCFS output:

\[r_{i}=\text{clip}\left(\frac{1}{T}\text{floor}\left(\frac{xT}{ \lambda}+0.5\right),0,1\right),\] (S11) \[S=\frac{1}{1-\frac{1}{N}}\left[1-\frac{\left(\sum_{i=1}^{N}\frac {r_{i}}{N}\right)^{2}}{\sum_{i=1}^{N}\frac{r_{i}^{2}}{N}}\right],\] (S12)

where \(i\) represents neuron index and \(N\) is the total number of neurons. To obtain the final stable population sparseness during the FEN training process, we increase the number of training steps from 175,000 to 400,000. But when running visual search, we still use the parameters saved at 175,000 training steps.

### Energy Consumption Analysis Method

We compare the energy consumption of BVSM with two different models. One is BVSM based on conventional artificial neural networks, the other is a conventional image classification network that directly processes original full-resolution images. BVSM based on ANN can be constructed by replacing IF neurons in BVSM to leaky-ReLU activation function, and its search performance is similar to BVSM. The conventional full-resolution ANN has a similar convolution backbone to the feature extraction network in BVSM (Fig. S2), but is deeper since the input size is larger.

We use the technical data of Intel Stratix 10 TX FPGA chip [102] and ROLLS neuromorphic chip [103] to estimate the energy consumption of BVSM and the other two networks. The FPGA consumes 12.5 pJ per FLOP, while the neuromorphic chip consumes 0.077 pJ per synaptic operation (SOP) and 3.7 pJ per spike. The same data were used in prior works [93, 104] to estimate the energy consumption of ANN and SNN.

For SNN, one SOP is defined as the transmission of one spike from a single neuron to another single neuron. We count the average number of spikes that each neuron fires within each fixation from 10000 visual search trials. The number of post-synaptic neurons of each neuron can also be calculated from the network structure. We do not account for the energy consumed by the floating point computation in the first convolution layer, but the energy consumed by the spikes generated in the first layer is included. The reason for this is that we lack energy consumption data for neuromorphic chips under a constant floating point input. Additionally, some neuromorphic chips, such as Intel Loihi, quantize the floating point into low-precision fixed-point representations, and the energy spent on this process is difficult to estimate accurately. To ensure a fair comparison, we also disregarded the floating point operation in the first convolution layer of the ANN.

For ANN, one FLOP is defined as the elementary calculation between two scalars. The FLOP of a convolution layer is \(2\times C_{in}\times K\times K\times C_{out}\times H_{out}\times W_{out}\), where \(C\) is channels, \(K\) is the convolution kernel size, and \(H,W\) are the size of the output. The FLOP of a linear layer is \(2\times W_{in}\times W_{out}\), where \(W_{in}\) and \(W_{out}\) are input and output dimensions. Batch normalization can be merged into convolution and linear layers, so they do not add additional FLOPs. We also assume that the ANN uses simple activation functions (such as ReLU or Leaky-ReLU), so the FLOP of activation function is simply the number of output neurons of each layer.

## Appendix D Full-resolution Image Classification Model

To test whether conventional ANN that directly processes full-resolution images can detect the target, we build a convolution network that has a similar backbone structure to the FEN (Fig. S2). The network performs a binary classification task, which is to determine whether the input image contains a target or not. This is easier than the visual search task which needs the network to output the exact target location. The network outputs the classification probability by a single forward computation without making eye movements.

We generate random samples of the search images, and train the ANN by supervised learning. The loss function is binary cross-entropy, the optimizer is AdamW without weight decay, the learning rate is 1e-3, and the training batch size is 64. We train the network for 175,000 steps (same as the training
step in the first training stage of FEN) under 5 different random seeds, but the loss value cannot decrease (Fig. S7), and the network's correct response rate is only at chance level (50%). Therefore, conventional ANN that directly processes full-resolution images cannot properly detect the small search target.
Supplementary Tables

Table S1: Comparison of our task and model design to previous research. Target's size, background image's size, and target's relative size are only shown for visual search tasks. Variable fixation number means that the model can automatically stop the trial without hard coding it to make decision after a predefined number of fixations. Training method refers to the method to train eye movement policy. EM: expectation maximization. RL: reinforcement learning. SVHN: The Street View House Numbers dataset. IOR: inhibition of return. (1): This is the size of the smallest bounding box enclosing all targets. The size is calculated based on the authors' data preprocessing method. (2): A single bounding box may contain 2-5 targets. (3): The target size, image size, and target relative size are calculated from the dataset.

\begin{tabular}{c c c c c c c c c c} \hline \hline
**Source** & **Task** & **Target** & **Back-** & **Target relative** & **Artificial fovea** & **Memory** & **SNN** & **Number of** & **Training** \\  & & **size** & **ground size** & **size** & & & & & & \\ \hline \multirow{3}{*}{[106]} & MNIST search \& & 28x28 & 48x48 & 34.0\% & Multi-resolution patches & Mean of predictions at each fixation & No & Fixed & Similar to \\  & recognition & & & & & & & & & EM \\ \hline \multirow{3}{*}{[106]} & MNIST search \& & 28x28 & 60x60 & 21.8\% & Multi-resolution patches & RNN & No & Fixed & RL \\  & recognition & & & & & & & & & \\ \hline \multirow{3}{*}{[107]} & MNIST search \& & 28x28 & 100x100 & 7.84\% & Multi-resolution patches & RNN & No & Fixed per target & RL \\  & recognition & & & & & & & & & \\  & targets & & & & & & & & & \\ \hline \multirow{3}{*}{[108]} & SVHN recognition & 49x49\({}^{17}\) & 54x54 & 16.47\% & Multi-resolution patches & RNN & No & Fixed per target & RL \\  & & & & & & 41.179\% & 20\% & & & & \\  & & & & & & & & & & \\ \hline \multirow{3}{*}{[108]} & Noisy MNIST search \& & 28x28 & 128x128 & 4.79\% & Log-Polar transform & No & No & Variable & Supervised \\  & recognition & & & & & & & & & \\ \hline \multirow{3}{*}{[109]} & MNIST search \& & 28x28 & 60x60 & 21.78\% & Multi-resolution patches & RNN & No & Fixed & Supervised \\  & recognition & & & & & & & & & \\ \hline \multirow{3}{*}{[111]} & Natural object search & & (2.2 & (142.500.) & Median: 9.71\% & Single patch memory & Exponential decay memory & No & Fixed & Supervised \\  & 499\%(x49) & & (71:500.) & Mean: 16.67\% & Simplified Freemonstrical model & Bayesian prior, manual & No & Variable & Maximum-a-posteriori \\ \hline \multirow{3}{*}{[112]} & Image classification & N/A & N/A & N/A & Single patch each fixation, manual & Mean of predictions at each fixation, manual & No & Fixed & RL \\ \hline \multirow{3}{*}{[114]} & Image classification & N/A & N/A & N/A & Single patch & Transformer & No & Training, fused, Testing, variable & RL \\ \hline \multirow{3}{*}{[115]} & Video classification & N/A & N/A & N/A & Single patch & Transformer & No & Fixed & Supervised \\  & & & & & & & & & \\ \hline Our model & Gabor target search & 12x12 & 651x651 & 0.034\% & Foveal Cartesian Geometry & Spaking RNN & Yes & Variable & RL \\ \hline \hline \end{tabular}
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Source** & **Task** & **Image type** & **RL algorithm** & **Neuron type** & **Coding method** & **Training method** & **Lategory** \\ \hline
[116] & Object detection & Natural image & N/A & Signed IF & Rate coding & Conversion & \(>\)3500 \\ \hline
[117] & Object detection & Natural image & N/A & IF & Rate coding & Bayesian optimization & \(>\)500 \\ \hline
[118] & Object detection & Natural image & N/A & LIF & Rate coding & Conversion + STDP + STBP & 300 \\ \hline
[119] & Object detection & Neuromorphic image & N/A & LIF & Spike count & STBP & 100-300\({}^{(1)}\) \\ \hline
[120] & Object detection & Natural image & N/A & IF & Rate coding & Conversion & \(>\)1000 \\ \hline
[121] & Object detection & Aerospace image & N/A & Modified LIF & Rate coding & Conversion & 64-128 \\ \hline
[122] & RL (Breakout) & N/A & Deep & IF & Rate coding & Conversion & 500 \\ \hline
[123] & RL (HalfCheeta-v3, Hopper-v3, Walker2d-v3, Ant-v3) & N/A & DDPG, TD3, SAC, PPO & LIF & Population coding & STBP & 5 \\ \hline
[124] & RL (17 Atari games) & N/A & Deep & IF & Rate coding & Conversion & \(>\)100 \\ \hline
[125] & RL (Cartpole, Pong) & N/A & Deep & LIF+custom & Membrane potential of non-spiking neuron & Conversion + STBP + iterative retraining & 1 \\ \hline
[126] & RL (Custom navigation task) & N/A & DDPG & Modified LIF & Population coding & STBP & 5 \\ \hline
[127] & RL (17 Atari games) & N/A & Deep & LIF+custom & Membrane potential of non-spiking neuron & STBP & 8 \\ \hline Out model & Detection \& RL (visual search) & Naturalistic noise image & SAC & IF & Population coding & Conversion \(\star\) STBP & 4 \\ \hline \hline \end{tabular}
\end{table}
Table S2: Comparison of SNN configurations in our model and previous works for object detection and RL. IF: integrate-and-fire neuron. LIF: leaky integrate-and-fire neuron. STDP: spike-timing-dependent plasticity. STBP: spatio-temporal back-propagation. (1): The inference time depends on the length of event-based input.
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Group** & **Description** & **Value** \\ \hline \multirow{8}{*}{Visual search task} & Screen distance & 70 cm \\  & Screen size & 1920\(\times\)1080 pixels, 54.3744\(\times\)30.2616 cm \\  & Search image size & 15\({}^{\circ}\)\(\times\)15\({}^{\circ}\) (651\(\pm\)51 pixels), circular shape \\  & Search image contrast & 0.2 \\  & Target size & 0.3\({}^{\circ}\)\(\times\)0.3 (\({}^{\circ}\)1212 pixels) \\  & Target orientation & 45\({}^{\circ}\) counterclockwise from the vertical \\  & Target spatial frequency & 6 cycles/degree \\  & Target contrast & Training: [0.11–0.15]. Testing: [0.11–0.136] or 0.15 \\  & Target number per image & 1 \\  & Target location & Uniformly random in search image \\  & Initial fixation location & Training: uniformly random in image. \\  & & Testing: uniformly random within 0.35\({}^{\circ}\) away from image center. \\  & Maximum fixation number & 50 during training, 200 during testing \\  & Correct criterion & Final fixation within 1\({}^{\circ}\) from target location \\ \hline \multirow{2}{*}{Retinal transform} & Method & Modified Reveal Cartesian Geometry \\  & Fowa radius & 8 pixels \\  & Maximum peripheral radius & 651 pixels \\  & Output size & 224\(\times\)224 pixels \\ \hline \multirow{2}{*}{Reward function} & Radius of IOR reward & 2.5\({}^{\circ}\) \\  & Memory capacity of IOR reward & 8 previous fixations \\  & Slope of saccade amplitude reward & -1/7.5\({}^{\circ}\) \\ \hline \multirow{4}{*}{IF neuron} & Inference time & 4 steps \\  & Surrogate gradient function & Atan() \\  & Initial membrane potential & 0.5 \\  & Firing threshold & 1 \\  & Reset method after spike & Reset by subtracting firing threshold \\ \hline \multirow{4}{*}{FEN} & Convolution kernel size & 7\(\times\)7 at first layer, 3\(\times\)3 at subsequent layers \\  & Convolution stride & 2 \\  & Padding & Replicate padding, padding size=(kernel size \(-\)\(1)/2\) \\  & Output channel number at block \(i\) & 16\(\times\)i \\  & Activation function & QCFS(Initial \(\lambda\)=8, 7=4), \(\lambda\) is trainable \\  & Fully connected layer hidden dimension & 1 layer, 448 neurons \\ \hline RNN & Number of IF spiking neurons & 64 \\ \hline \multirow{4}{*}{Actor network} & Number of hidden layers & 2 \\  & Hidden layer dimension & 480 IF neurons \\  & Target detection threshold & 25 pixels \\ \hline \multirow{4}{*}{Critic network} & Number of hidden layers & 2 and 3 for the two heads \\  & Hidden layer dimension & 64 for both heads \\  & Activation function & Leaky-ReLU, negative slope = 0.1 \\ \hline \multirow{4}{*}{FEN training} & Training batch size & 64 \\  & Validation batch size & 256 \\  & Training steps & 174,000 steps for training, 35,000 steps for fine-tuning \\  & Optimizer & AdamW\({}^{[12]}\) with no weight decay \\  & Initial learning rate & 1,000-63 \\  & Distance between target and fixation locations & Exponential distribution with mean=4\({}^{0}\) \\ \hline \multirow{4}{*}{RL} & Algorithm & Soft Actor-Critic \\  & Entropy target & Hyperparameter group 1: -1; hyperparameter group 2: -2 \\  & Initial temperature \(\alpha\) & 1 \\  & Discount factor\(\gamma\) & 0.95 \\  & Replay buffer size & 50,000 trials \\  & Batch size in optimization & 32 trials \\  & Optimizer & AdamW with no weight decay \\  & Initial learning rate & 1e-4 for actor and \(\alpha\), 1e-3 for critic and RNN \\  & Gradient clipping & Maximum norm of 1.0 for critic and RNN \\  & Initial optimization time & After 333 trials \\  & Optimization frequency & Once per fixation \\  & Target network update speed & Polyak update, \(\tau\)=0.005 \\  & Training trials & 50,000 \\ \hline \multirow{4}{*}{Reward functions} & IOR reward radius & Hyperparameter group 1: 0.5\({}^{\circ}\), Hyperparameter group 2: 2.5\({}^{\circ}\) \\  & IOR reward memory capacity & 8 most recent fixations \\ \cline{1-1}  & Saccade amplitude reward (exponential) & Hyperparameter group 1: \(-0.5+0.5\times(\exp{(-\text{SacAmp}/2.5^{\circ})}-1)\) \\ \cline{1-1}  & Saccade amplitude reward (linear) & Hyperparameter group 2: \(-\text{SacAmp}/7.5^{\circ}\) \\ \hline \hline \end{tabular}
\end{table}
Table S3: Hyperparameters for the results reported in the main text. IF neuron: integrate-and-fire neuron. FEN: feature extraction network. RNN: recurrent neural network. RL: reinforcement learning. SacAmp: saccade amplitude.
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Changed parameter** & **New value** & \begin{tabular}{c} **Target** \\ **contrast** \\ \end{tabular} & \begin{tabular}{c} **\# of random** \\ **seeds** \\ \end{tabular} & \begin{tabular}{c} **Percent** \\ **correct** \\ \end{tabular} & \begin{tabular}{c} **Median** \\ **FixNum** \\ \end{tabular} & 
\begin{tabular}{c} **Mean** \\ **FixNum** \\ \end{tabular} \\ \hline No change & – & 0.11–0.136 & 5 & 98.2\% & 7.0 & 10.8 \\ Target diameter & 0.6\({}^{\circ}\) & 0.11–0.136 & 5 & 99.9\% & 2.0 & 4.2 \\ Discount \(\gamma\) & 0.99 & 0.11–0.136 & 5 & 98.0\% & 7.4 & 10.9 \\ IOR memory capacity & 1 & 0.11–0.136 & 5 & 98.0\% & 7.0 & 11.5 \\ IOR memory capacity & 2 & 0.11–0.136 & 5 & 98.1\% & 7.0 & 11.4 \\ IOR memory capacity & 4 & 0.11–0.136 & 5 & 98.1\% & 7.0 & 11.2 \\ IOR radius & 1.5\({}^{\circ}\) & 0.11–0.136 & 5 & 97.9\% & 8.2 & 12.5 \\ Slope of SacAmp & -1/4.5\({}^{\circ}\) & 0.11–0.136 & 5 & 98.0\% & 7.8 & 11.3 \\ reward (linear) & & & & & & \\ Reward function & Only IOR reward & 0.11–0.136 & 5 & 97.1\% & 9.8 & 17.4 \\ Reward function & Only SacAmp & 0.11–0.136 & 5 & 83.3\% & 21.6 & 44.2 \\ Reward function & & & & & & \\ reward (exp) & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table S4: Comparison of the estimated energy consumption between BVSM and conventional ANN models.
Supplementary Figures
Figure S2: Structure of the conventional ANN that processes full resolution image and classifies whether the image contains a target. The dashed lines show the internal components of convolution block. Only the first convolution block uses \(7\times 7\) convolution, and all subsequent blocks use \(3\times 3\) convolution. The leaky-ReLU function has slope of 0.1 in the negative part. The number under each block indicates the dimension of the output when batch size \(B=1\).
Figure S4: Up: Feature extraction network (FEN) initial training/validation loss. Low: Spiking FEN fine-tuning training/validation loss. The networks are trained using five different random seeds. The loss values are first smoothed by a Gaussian kernel with standard deviation of 100 steps, then average across the five repetitions. The shaded area represents the standard deviation of the smoothed loss.
Figure S5: RL training process averaged across five different random seeds for hyperparameter group 1. The values are first smoothed by a Gaussian kernel with standard deviation of 100 steps, then averaged across the five repetitions. The shaded area represents the standard deviation of the smoothed values. Note that the fixation number is averaged across both correct and error trials, so it is higher than the mean and median fixation number of only correct trials reported in the main text. During training, the maximum fixation is 50, so the percent correct is lower than the result reported in the main text.
Figure S6: RL training process averaged across five different random seeds for hyperparameter group 2. The networks are trained using five different random seeds. The loss values are first smoothed by a Gaussian kernel with standard deviation of 100 steps, then average across the five repetitions. The shaded area represents the standard deviation of the smoothed loss.
Figure S8: BVSM and human scanpath comparison. **A**: Randomly selected BVSM scanpath sorted by fixation number. Magenta cross is the final fixation location, red star is the target location. BVSM is trained by hyperparameter group 1 (HP 1). **B**: Example human scanpath grouped by different search strategies. Gray: circular scanpath. Blue: repeated fixation on suspected target locations. Yellow: confirmation scanpath when the target is close to the initial fixation location. Blue: scanpath with many short saccades.