Quantization in Spiking Neural Networks
In spiking neural networks (SNN), at each node, an incoming sequence of
weighted Dirac pulses is converted into an output sequence of weighted Dirac
pulses by a leaky-integrate-and-fire (LIF) neuron model based on spike
aggregation and thresholding. We show that this mapping can be understood as a
quantization operator and state a corresponding formula for the quantization
error by means of the Alexiewicz norm. This analysis has implications for
rethinking re-initialization in the LIF model, leading to the proposal of
'reset-to-mod' as a modulo-based reset variant.
Bernhard A. Moser, Michael Lunglmayr
# Quantization in Spiking Neural Networks

###### Abstract

In spiking neural networks (SNN), at each node, an incoming sequence of weighted Dirac pulses is converted into an output sequence of weighted Dirac pulses by a leaky-integrate-and-fire (LIF) neuron model based on spike aggregation and thresholding. We show that this mapping can be understood as a quantization operator and state a corresponding formula for the quantization error by means of the Alexiewicz norm. This analysis has implications for rethinking reinitialization in the LIF model, leading to the proposal of _reset-to-mod_ as a modulo-based reset variant.

Leaky-Integrate-and-Fire (LIF) Neuron Spiking Neural Networks (SNN) Re-Initialization Quantization Error Propagation Alexiewicz Norm

## 1 Introduction

Though its simplicity, the leaky integrate-and-fire (LIF) neuron model is widely spread in neuromorphic computing and computational neuroscience Gerstner et al. (2014); Nunes et al. (2022). In contrast to more biophysically realistic models, such as the Hodgkin-Huxley model, LIF is a middle ground to capture essential features of bio-inspired time-based information processing, and simple enough to be applicable from the point of view of neuromorphic engineering. In particular, in biology spikes show varying shapes and its generation and re-initialization follows a complex dynamics as revealed by the Hodgkin-Huxley differential equations. In contrast, LIF is based on the following major idealizations. (1), it neglects shape information and idealizes spikes as shapeless impulses, (2), it is based on the assumption that the triggering process is realized by a comparison with a given threshold-level, and, (3), the process of triggering and re-initialization acts instantaneously.

In this paper we take these assumptions as axioms for a mathematical operator that maps a sequence of weighted Dirac impulses to another one. We ask about properties of this mapping and find that it can be understood as a quantization operation in the space of spike trains. The standard model of quantization of a single number is integer truncation. That is, a given real number \(x=n+r\), \(n\in\mathbb{Z}\), \(r\in(-1,1)\) is mapped to the integer \(n\) and the quantization error is given by \(r\). The resulting mapping \(q:\mathbb{R}\to\mathbb{Z}\) can also be considered from a geometric point of view. Consider a grid of vertexes of a tessellation of polytopes induced by the unit ball of the maximum norm, \(\|.\|_{\infty}\). This way, a given point \(x\) is element of such a polytope \(P\) and mapped to that vertex of \(P\) that is closest to the origin.

In an abstract sense, we will show that LIF can be understood in the same way. Instead of the maximum norm, \(\|.\|_{\infty}\), the space \(\mathbb{S}\) of spike trains is tessellated by means of the unit balls of the Alexiewicz norm \(\|.\|_{A,\alpha}\), which will be outlined next. Then we get the formula \(\|\text{LIF}_{\alpha,\vartheta}(\eta)-\eta\|_{A,\alpha}<\vartheta\), where \(\text{LIF}_{\alpha,\vartheta}\) denotes the LIF mapping based on the leaky parameter \(\alpha\), the threshold \(\vartheta>0\) and the input spike train \(\eta\in\mathbb{S}\).

The paper is outlined as follows. In Section 2 we reformulate the LIF model, taking different re-initialization variants into account. In Section 3 we recall the Alexiewicz norm \(\|.\|_{A}\), generalize it to leaky variants \(\|.\|_{A,\alpha}\) and state the
main theorem. Section 4 presents evaluations considering the discussed reinitialization variants, based on Python code available at [https://github.com/LunglmayrMoser/AlexSNN](https://github.com/LunglmayrMoser/AlexSNN).

## 2 LIF Model and Re-Initialization Variants

The flow of information in a LIF model is outlined in Fig. 1. Arriving from several input channels, spike events trigger the dynamics of the membrane potential by a weighted linear superposition across the channels and an integration with a leaky term in time. The LIF model idealizes the superposition (and, also that of re-initialization) by an instantaneous aggregation. Due to this building process, the incoming spike events can equivalently be represented by a single channel of spikes with amplitudes resulting from a weighted sum of simultaneous spikes across the input channels. This way we have to take spike amplitudes of virtually arbitrary magnitude into account. In the literature, this aspect is pointed out as a limitation of the LIF model, since it deviates from biology in this respect Gerstner et al. (2014). As an idealized mathematical model, however, it has its justification, but then this effect must also be taken into account in the re-initialization step. While in the context of LIF there are discussed basically two modes of re-initialization (synonymously, _reset_), namely _reset-to-zero_ and _reset-by-subtraction_, in Eqn. (2) we will propose a third one, _reset-to-mod_, to be mathematically consequent regarding the outlined issue of instantaneous superposition. According to Eshraghian et al. (2021), _reset-to-zero_ means that the potential is reinitialized to zero after firing, while _reset-by-subtraction_ subtracts the \(\vartheta\)-potential \(u_{\vartheta}\) from the membrane's potential that triggers the firing event. As a third variant we introduce _reset-to-mod_, which can be understood as instantaneously cascaded application of _reset-by-subtraction_ according to the factor by which the membrane's potential exceeds the threshold which results in a modulo computation.

For an input spike train \(\eta_{in}(t)=\sum_{i}a_{i}\delta_{t_{i}}(t)\) the mapping \(\sum_{i}b_{i}\,\delta_{s_{i}}=\text{LIF}_{\vartheta,\alpha}(\eta_{in})\) is recursively given by \(s_{i+1}=\inf\left\{s\geq s_{i}:\,|u_{\vartheta,\alpha}(s_{i},s)|\geq\vartheta \right\},\) where

\[u_{\vartheta,\alpha}(t_{\text{reset}},t):=\int_{t_{\text{reset}}}^{t}e^{- \alpha(\tau-t_{\text{reset}})}\left(\eta_{in}(\tau)-\text{discharge}(t_{\text{reset}},\tau)\right)d\tau \tag{1}\]

models the dynamic change of the neuron membrane's potential after an input spike event at time \(t_{\text{reset}}\). The process of triggering an output spike is actually a charge-discharge event

\[\text{discharge}(t_{i},\tau):=\left\{\begin{array}{lcl}a_{i}\delta_{t_{i}}( \tau)&&\ldots&\text{for _reset-to-zero},\\ \text{sgn}(a_{i})\,\vartheta\,\delta_{t_{i}}(\tau)&&\ldots&\text{for _reset-by-subtraction},\\ q(a_{i}/\vartheta)\,\vartheta\,\delta_{t_{i}}(\tau)&&\ldots&\text{for _reset-to-mod}\end{array}\right. \tag{2}\]

Figure 1: Illustration of steps of information processing by a LIF model. The weighted superposition of incoming spike trains can result in spike amplitudes that can exceed the threshold many times.
where \(\text{sgn}(x)\in\{-1,0,1\}\) is the signum and \(q(x):=\text{sgn}(x)\max\{k\in\mathbb{Z}:k\leq|x|\}\) is the integer truncation quantization function.

The _reset-by-subtraction_ mode can be understood as compensation event so that the net voltage balance of the spiking event equals zero, i.e., in case of an output spike with amplitude \(\vartheta>0\) the membrane is actually discharged by this amount. Accordingly, though not always made clear in the literature, see for example Eshraghian et al. (2021), this assumption has the subtle consequence that an increase of the membrane potential \(u\) by multiples \(q(u/\vartheta)\) of the threshold level \(\vartheta\) results in a discharge of the membrane's potential by the same amount, that is \(q(u/\vartheta)\vartheta\). See Fig. 2 for an illustration and example.

## 3 Alexiewicz Norm and Spike Train Quantization

For \(\eta=\sum_{i}a_{i}\delta_{t_{i}}\in\mathbb{S}\) we introduce the measure \(\|\eta\|_{A,\alpha}:=\max_{n}\left|\sum_{j=1}^{n}a_{j}e^{-\alpha(t_{n}-t_{j})}\right|\), which satisfies the axioms of a norm on the vector space \(\mathbb{S}\). For \(\alpha=0\) we obtain the _Alexiewicz_ norm Alexiewicz (1948), which is topologically equivalent to the discrepancy norm Moser (2012). The Alexiewicz norm reveals the quantization character of the LIF model.

**Theorem 1** (reset-to-mod LIF as \(\|.\|_{A,\alpha}\)-Quantization): _Given a LIF neuron model with reset-to-mod, the threshold \(\vartheta>0\), the leaky parameter \(\alpha\in[0,\infty]\) and the spike train \(\eta\in\mathbb{S}\) with amplitudes \(a_{i}\in\mathbb{R}\). Then, \(\text{LIF}_{\vartheta,\alpha}(\eta)\) is a \(\vartheta\)-quantization of \(\eta\), i.e., the resulting spike amplitudes are multiples of \(\vartheta\), where the quantization error is bounded by_

\[\|\text{LIF}_{\vartheta,\alpha}(\eta)-\eta\|_{A,\alpha}<\vartheta. \tag{3}\]

**Proof.** Without loss of generality we may assume that \(\vartheta=1\). We have to show that \(\|\text{LIF}_{1,\alpha}(\eta)-\eta\|_{A,\alpha}<1\), which is equivalent to the discrete condition that \(\forall n:\max_{n}|\sum_{i=1}\hat{a}_{i}|<1\), where \(\text{LIF}_{1,\alpha}(\eta)-\eta=\sum_{i}\hat{a}_{i}\delta_{t_{i}}\). The proof is based on induction and traces the problem back to standard quantization by truncation.

Suppose that at time \(t_{i_{k-1}}\) after re-initialization by _reset-to-mod_ we get the residuum \(\Delta_{i_{k-1}}\) as membrane potential that is the starting point for the integration after \(t_{i_{k-1}}\). Then, as illustrated in Fig. 3, the residuum \(\Delta_{i_{k}}\) at the next triggering event \(t_{i_{k}}\) is obtained by the equation

\[\Delta_{i_{k}}=\Delta_{i_{k-1}}\oplus a_{i_{k-1}+1}\oplus\ldots\oplus a_{i_{ k}}-[\Delta_{i_{k-1}}\oplus\ldots\oplus a_{i_{k}}]. \tag{4}\]

On the other hand, note that for the differences \(\hat{a}_{i}\) we have

\[\hat{a}_{i_{k+1}}=\hat{a}_{i_{k}}\oplus a_{i_{k}+1}\oplus\ldots\oplus a_{i_{ k-1}-1}\oplus\left(a_{i_{k+1}}-[\Delta_{i_{k}}\oplus a_{i_{k+1}}\oplus \ldots\oplus a_{i_{k+1}}]\right). \tag{5}\]

Note that \(\Delta_{i_{0}}=0\) and, for induction, let us assume that up to index \(k\) we have \(\hat{a}_{i_{k}}=\Delta_{i_{k}}\). This, together with Eqn. (5) gives

\[\hat{a}_{i_{k+1}} = \Delta_{i_{k}}\oplus a_{i_{k}+1}\oplus\ldots\oplus a_{i_{k-1}-1} \oplus\left(a_{i_{k+1}}-[\Delta_{i_{k}}\oplus a_{i_{k+1}}\oplus\ldots\oplus a _{i_{k+1}}]\right) \tag{6}\] \[= \Delta_{i_{k}}\oplus a_{i_{k}+1}\oplus\ldots\oplus a_{i_{k-1}-1} \oplus a_{i_{k+1}}-[\Delta_{i_{k}}\oplus a_{i_{k+1}}\oplus\ldots\oplus a_{i_{k +1}}],\]

which proofs \(\hat{a}_{i_{k}}=\Delta_{i_{k}}\) to hold for all \(k\), showing that the deltas, \(\Delta_{i_{k}}\), can be expressed as differences of the standard quantization by truncation, hence proving the claim of Theorem 1 for all \(\alpha\in[0,\infty]\).

Figure 3: Illustration of Eqn. (4). The red arrows indicate reset by _reset-to-mod_.

Figure 2: Example of input spike train with different LIF outputs depending on the re-initialization variant.
## 4 Evaluations

We evaluate the distribution of the quantization error (3) by means of box whisker diagrams for different re-initialization variants depending on the number of spikes in the spike train and different distributions of amplitudes. Fig. 4 shows the result for incoming spike amplitudes that are below threshold. In this case, (3) holds for _reset-to-mod_ and _reset-by-subtraction_. As expected, both variants behave similarly. For _reset-to-zero_ the bound (3) only holds approximately for larger leaky parameter \(\alpha\), see first row of Fig. 4. For _reset-to-mod_ and _reset-by-subtraction_, With increasing number of spikes one can observe a concentration of measure effect, see, e.g., Vershynin).

However, as shown in Fig. 5, if the incoming spike amplitudes are not below threshold anymore Eqn. (3) only holds for _reset-to-mod_. In this case the measure of concentration effect become more apparent for smaller leaky parameter. For the other variants the quantization error increases in average with the number of spikes, and the theoretical bound (3), proven for _reset-to-mod_, does not hold anymore.

Figure 4: Evaluation of (3) for _reset-to-mod_, _reset-by-subtraction_ and _reset-to-zero_ (1.st/2nd/3rd column), based on spike trains with spike amplitudes in \([-\vartheta,\vartheta]\) and \(100\) runs. The 1st row refers to \(\alpha=1\) and the 2nd row to \(\alpha=0.1\).

Figure 5: The same as in Fig. 4 but with spike amplitudes in \([-3/2\vartheta,3/2\vartheta]\).
## 5 Conclusion

In this paper we provide a novel view on the leaky integrate-and-fire model as quantization operator in the Alexiewicz norm. This analysis gives rise to rethinking the re-initialization modes _reset-to-zero_ and _reset-by-subtraction_ that are commonly used in the context of spiking neural networks. These re-initialization modes only hold under restricted conditions while our proposed variant _reset-to-mod_ satisfies the derived quantization bound under general conditions. This general quantization error formula leads to new error bounds for LIF and SNNs, such as a quasi-isometry relation in analogy to threshold-based sampling Moser (2017), Moser and Lunglmayr (2019). Examples can be found in the github repository [https://github.com/LunglmayrMoser/AlexSNN](https://github.com/LunglmayrMoser/AlexSNN) and Moser and Lunglmayr (2023).
