Sparse-firing regularization methods for spiking neural networks with
  time-to-first spike coding
The training of multilayer spiking neural networks (SNNs) using the error
backpropagation algorithm has made significant progress in recent years. Among
the various training schemes, the error backpropagation method that directly
uses the firing time of neurons has attracted considerable attention because it
can realize ideal temporal coding. This method uses time-to-first spike (TTFS)
coding, in which each neuron fires at most once, and this restriction on the
number of firings enables information to be processed at a very low firing
frequency. This low firing frequency increases the energy efficiency of
information processing in SNNs, which is important not only because of its
similarity with information processing in the brain, but also from an
engineering point of view. However, only an upper limit has been provided for
TTFS-coded SNNs, and the information-processing capability of SNNs at lower
firing frequencies has not been fully investigated. In this paper, we propose
two spike timing-based sparse-firing (SSR) regularization methods to further
reduce the firing frequency of TTFS-coded SNNs. The first is the membrane
potential-aware SSR (M-SSR) method, which has been derived as an extreme form
of the loss function of the membrane potential value. The second is the firing
condition-aware SSR (F-SSR) method, which is a regularization function obtained
from the firing conditions. Both methods are characterized by the fact that
they only require information about the firing timing and associated weights.
The effects of these regularization methods were investigated on the MNIST,
Fashion-MNIST, and CIFAR-10 datasets using multilayer perceptron networks and
convolutional neural network structures.
Yusuke Sakemi, Kakei Yamamoto, Takeo Hosomi, Kazuyuki Aihara
# Sparse-firing regularization methods for spiking neural networks with time-to-first spike coding

###### Abstract

The training of multilayer spiking neural networks (SNNs) using the error backpropagation algorithm has made significant progress in recent years. Among the various training schemes, the error backpropagation method that directly uses the firing time of neurons has attracted considerable attention because it can realize ideal temporal coding. This method uses time-to-first spike (TTFS) coding, in which each neuron fires at most once, and this restriction on the number of firings enables information to be processed at a very low firing frequency. This low firing frequency increases the energy efficiency of information processing in SNNs, which is important not only because of its similarity with information processing in the brain, but also from an engineering point of view. However, only an upper limit has been provided for TTFS-coded SNNs, and the information-processing capability of SNNs at lower firing frequencies has not been fully investigated. In this paper, we propose two spike timing-based sparse-firing (SSR) regularization methods to further reduce the firing frequency of TTFS-coded SNNs. The first is the membrane potential-aware SSR (M-SSR) method, which has been derived as an extreme form of the loss function of the membrane potential value. The second is the firing condition-aware SSR (F-SSR) method, which is a regularization function obtained from the firing conditions. Both methods are characterized by the fact that they only require information about the firing timing and associated weights. The effects of these regularization methods were investigated on the MNIST, Fashion-MNIST, and CIFAR-10 datasets using multilayer perceptron networks and convolutional neural network structures.

## Introduction

Spiking neural networks (SNNs) can process information in the form of spikes in a manner similar to the way information is processed in the brain. SNNs are thereby expected to be able to achieve both high computational functionality and energy efficiency [1]. The spikes are represented as all-or-none binary values, and how information is represented by spikes is closely related to the information-processing mechanism in SNNs. The spike-based information representation methods are divided into two major categories, rate coding and temporal coding [2, 3]. In rate coding, information is contained in the average number of spikes generated by a neuron. In this case, the firing frequency can take approximately continuous values as a function of the input intensities; therefore, the resulting SNNs can be treated as differentiable models similar to an artificial neural network (ANN). Using rate coding, ANNs can be converted to SNNs, and the high learning ability of ANNs has been successfully transferred to SNNs [4, 5, 6]. However, when rate coding is used, information processing in the SNNs is just an approximation of that in ANNs. Furthermore, the precise approximation of an ANN requires many spikes, which reduces energy efficiency when implemented in neuromorphic hardware [7]. It has been experimentally shown that physiologically, neurons in certain brain regions or specific neuron types exhibit extremely sparse firing characteristics [8], and it is thought that temporal coding using not only the firing frequency but also the firing time is realized in at least some brain regions [9, 10, 11, 12].