Artificial to Spiking Neural Networks Conversion for Scientific Machine
  Learning
We introduce a method to convert Physics-Informed Neural Networks (PINNs),
commonly used in scientific machine learning, to Spiking Neural Networks
(SNNs), which are expected to have higher energy efficiency compared to
traditional Artificial Neural Networks (ANNs). We first extend the calibration
technique of SNNs to arbitrary activation functions beyond ReLU, making it more
versatile, and we prove a theorem that ensures the effectiveness of the
calibration. We successfully convert PINNs to SNNs, enabling computational
efficiency for diverse regression tasks in solving multiple differential
equations, including the unsteady Navier-Stokes equations. We demonstrate great
gains in terms of overall efficiency, including Separable PINNs (SPINNs), which
accelerate the training process. Overall, this is the first work of this kind
and the proposed method achieves relatively good accuracy with low spike rates.
Qian Zhang, Chenxi Wu, Adar Kahana, Youngeun Kim, Yuhang Li, George Em Karniadakis, Priyadarshini Panda
# Artificial to Spiking Neural Networks Conversion for Scientific Machine Learning +

###### Abstract

We introduce a method to convert Physics-Informed Neural Networks (PINNs), commonly used in scientific machine learning, to Spiking Neural Networks (SNNs), which are expected to have higher energy efficiency compared to traditional Artificial Neural Networks (ANNs). We first extend the calibration technique of SNNs to arbitrary activation functions beyond ReLU, making it more versatile, and we prove a theorem that ensures the effectiveness of the calibration. We successfully convert PINNs to SNNs, enabling computational efficiency for diverse regression tasks in solving multiple differential equations, including the unsteady Navier-Stokes equations. We demonstrate great gains in terms of overall efficiency, including Separable PINNs (SPINNs), which accelerate the training process. Overall, this is the first work of this kind and the proposed method achieves relatively good accuracy with low spike rates.

Spiking Neural Networks Conversion Nonlinear activation

## 1 Introduction

The use of machine learning techniques in the scientific community has been spreading widely, reaching many fields such as physics [1, 2, 3, 4], chemistry [5, 6], biology [7, 8, 9], geophysics [10, 11], epidemiology[12, 13] and many more. The advances in computation capabilities have enabled many researchers to reformulate diverse problems as data-driven problems, by combining prior knowledge of the problem with fitting a model for the available data. A prominent drawback of Scientific Machine Learning (SciML) techniques is that they are usually expensive in terms of computational cost. They require either knowledge of the governing equations that determine the process (approximating them is a costly procedure), or a large amount of data to fit (expensive as well). The SciML community is striving for a more efficient method for training and inferring neural networks. Neuromorphic chips are one edge computing component that SciML applications could benefit from. In this work, we explore methods for enabling this.

An important breakthrough in the field of SciML was the invention of the Physics-Informed Neural Networks (PINNs) [14, 15]. PINNs incorporate the knowledge of the physical experiment and governing equations into the network training step, making it a hybrid (physics and data) training. PINNs and its extensions [16, 17, 18, 19] have achieved great success in many fields and applications [20, 1, 21, 22, 23, 24, 7, 25, 26]. A disadvantage of PINNs is that like other deep neural networks, they are prone to long training times. In addition, when changing the problem conditions (initial conditions, boundary conditions, domain properties, etc.), the PINN has to be trained from scratch. Therefore, for real-time applications, a more efficient solution is sought after. For training a PINN, one usually uses smooth activation
functions (such as the Tanh or Sine activation functions), where in most ANNs the ReLU activation is dominant. Using smooth activation function in a SNN is a new challenge we address in this paper with theoretical justification.

Spiking Neural Networks (SNNs) have been gaining traction in the machine learning community for the past few years. The main reason is their expected efficiency [27; 28; 29; 30], in terms of energy consumption, compared to their Artificial Neural Network (ANN) counterparts that are commonly used for many applications. In addition, the advances in neuromorphic hardware (such as Intel's Loihi 2 chip [31; 32]), call for innovative algorithms and software that can utilize the chips and produce lighter and faster machine learning models. However, developing an SNN is a challenging task, especially for regression [33; 34]. Studies have been conducted for translating components from the popular ANNs into a spiking framework [35], but many components are not yet available in the spiking regime. In this paper we focus on that specific aspect.

There are three popular approaches for training SNNs. The first involves using mathematical formulations of the components of the brain, such as the membrane [36; 37; 38], the synapse [39], etc. In this case, one uses a Hebbian learning rule [40] to find the weights of the synapses (the trainable parameters) using forward propagation (without backward propagation [41; 42]). The second method involves building surrogate models for the elements in the SNN that block the back-propagation, such as the non-differentiable activation functions used in SNNs. The third method, which is discussed in this paper, addresses converting a trained ANN into a SNN.

The main contributions of this paper are as follows:

1. We propose a method to convert PINNs, a type of neural network commonly used for regression tasks, to Spiking Neural Networks (SNNs). The conversion allows for utilizing the advantages of SNNs in the inference stage, such as computational efficiency, in regression tasks.
2. We extend the calibration techniques used in previous studies to arbitrary activation functions, which significantly increases the applicability of the conversion method. Furthermore, we provide a convergence theorem to guarantee the effectiveness of the calibration.
3. We apply the conversion to separable PINNs (SPINNs), which accelerates the training process of the PINNs.

Overall, the proposed method extends the application of SNNs in regression tasks and provides a systematic and efficient approach to convert existing neural networks for diverse regression tasks to SNNs.

## 2 Related Work

Physics-informed neural networks (PINNs):An innovative framework that combines neural networks with physical laws to learn complex physical phenomena. In PINNs, the physical equations are integrated into the loss function, which allows the network to learn from both the given data and the underlying physics. This approach significantly improves the network's ability to handle incomplete or noisy data and performs well with limited training data. PINNs have been successfully applied to a range of problems in fluid dynamics, solid mechanics, and more [22; 1; 21; 23; 7; 24].

Separable PINNs (SPINNs):Cho et al. [43] proposed a novel neural network architecture called SPINNs, which aims to reduce the computational demands of PINNs and alleviate the curse of dimensionality. Unlike vanilla PINNs that use point-wise processing, SPINN works on a per-axis basis, thereby reducing the number of required network forward passes. SPINN utilizes factorized coordinates and separated sub-networks, where each sub-network takes an independent one-dimensional coordinate as input, and the final output is generated through an outer product and element-wise summation. Because SPINN eliminates the need to query every multidimensional coordinate input pair, it is less affected by the exponential growth of computational and memory costs associated with grid resolution in standard PINNs. Furthermore, SPINNs operate on a per-axis basis, which allows for parallelization with multiple GPUs.

Spiking Neural Networks (SNNs):A type of Artificial Neural Network (ANN), that differs in the implementation of the core components. The purpose is to create more biologically-plausible training and inference procedures. Unlike traditional ANNs, which process information through numerical values, SNNs process information through spikes, which occur in response to stimulation (much like the human brain). SNNs are becoming increasingly popular as they can mimic the temporal nature of biological neurons. Additionally, SNNs are computationally efficient and have the potential for efficient hardware implementation, making them well-suited for real-time applications. Combining SNN implementation with edge computing, training could be significantly faster, and inference as well [27; 28; 29]. Recent results have shown that SNNs can achieve high accuracy on image classification tasks, with up to 99.44% on the MNIST dataset [44] and up to 79.21% on ImageNet [45].

SNN conversionA technique to transform a trained ANN into a SNN. SNN conversion usually involves mapping the weights and activation functions of the ANN to the synaptic strengths and spike rates of SNNs. It is considered the most efficient way to train deep SNNs, as it avoids the challenges of direct SNN training, such as gradient estimation and spike generation [46; 47; 48]. The algorithm of SNN conversion can be divided into two steps: offline conversion and
online inference. In the offline conversion step, the trained ANN model is converted into an equivalent SNN model by adjusting the network parameters. In the online inference step, the converted SNN model is used for the inference. In the online step, the SNN is intended to be deployed on neuromorphic hardware, to unlock its full potential and energy efficiency.

## 3 Method

A complementary process to the conversion technique. The SNN calibration [49] is a method that minimizes the loss of accuracy and efficiency when converting an ANN into a SNN. SNN calibration leverages the knowledge of the pre-trained ANN and corrects the conversion error layer by layer. The algorithm consists of two steps: replacing the ReLU activation function with a spike activation function, and applying the calibration to adjust only the biases (light) or weights and biases (advanced) of each layer. The calibration method is based on the theoretical analysis of the conversion error and its propagation through the network. SNN calibration can achieve comparable or even better performance than the original ANN on various datasets and network architectures. This paper is a generalization of this work. We propose an extension to the SNN conversion, which is approriate for regression tasks.

### SNN Conversion setup

We consider a dataset \(D=(X,Y)\), and an ANN \(\mathcal{A}\) with \(n\) hidden layers, trained to fit it. Let \(\mathbf{x}^{(n)}=\mathcal{A}(X)\) be the output of \(\mathcal{A}\). The goal of SNN conversion is to find an SNN \(\mathcal{S}\), whose (averaged) output is \(\bar{\mathbf{s}}^{(n)}=\mathcal{S}(X)\), such that \(\mathbf{x}^{(n)}\) is close to \(\bar{\mathbf{s}}^{(n)}\). In other words, we want to minimize the norm of the error \(\mathbf{e}^{(n)}\stackrel{{ d}}{{=}}\mathbf{x}^{(n)}-\bar{ \mathbf{s}}^{(n)}\) for a given \(\mathcal{A}\) and \(D\). The same network structure as \(\mathcal{S}\) is used for \(\mathcal{A}\) and the activation function of \(\mathcal{A}\) is replaced with IF. Then we can analyze the factors that influence the total conversion error \(\mathbf{e}^{(n)}\).

### SNN Conversion with Calibration

In this section, we will briefly explain the SNN conversion with calibration proposed in [49]. Consider an MLP model with \(n\) layers, the first \(n-1\) layers use ReLU activation and the last layer has no activation. We denote \(\mathbf{W}^{(l)}\) as the weights and bias for layer \(l\).

The naive way of SNN conversion is simply replacing the ReLU activation layers with IF activation layers. In this case, we define \(\mathbf{x}^{(l)}\) as the output of layer \(l\) recursively, which is \(\mathbf{x}^{(l)}=\mathrm{ReLU}(\mathbf{W}^{(l)}\mathbf{x}^{(l-1)})\) and \(\mathbf{x}^{(0)}\) is the input. Similarly, we define \(\bar{\mathbf{s}}^{(l)}\), the output of the layer \(l\) of converted SNN, as \(\bar{\mathbf{s}}^{(l)}=\mathrm{IF}(\mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l-1)})\) and \(\bar{\mathbf{s}}^{(0)}=\mathbf{x}^{(0)}\) is the input. In fact, we can compute the expected output spikes as \(\bar{\mathbf{s}}^{(l)}=\mathrm{ClipFloor}(\mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l -1)})\). The \(\mathrm{ClipFloor}\) function is an approximation to \(\mathrm{ReLU}\) and is illustrated in Figure 1.

Then we can define the conversion error of layer \(l\) as \(\mathbf{e}^{(l)}=\mathbf{x}^{(l)}-\bar{\mathbf{s}}^{(l)}\) and decompose it as

\[\begin{split}\mathbf{e}^{(l)}&=\mathbf{x}^{(l)}- \bar{\mathbf{s}}^{(l)}\\ &=\mathrm{ReLU}(\mathbf{W}^{(l)}\mathbf{x}^{(l-1)})-\mathrm{ ClipFloor}(\mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l-1)})\\ &=\mathrm{ReLU}(\mathbf{W}^{(l)}\mathbf{x}^{(l-1)})-\mathrm{ ReLU}(\mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l-1)})+\mathrm{ReLU}(\mathbf{W}^{(l)} \bar{\mathbf{s}}^{(l-1)})-\mathrm{ClipFloor}(\mathbf{W}^{(l)}\bar{\mathbf{s}}^ {(l-1)})\\ &=\mathbf{e}^{(l)}_{r}+\mathbf{e}^{(l)}_{c}\end{split} \tag{1}\]

Here \(\mathbf{e}^{(l)}_{r}\) represents the error caused by approximating the continuous input by the spiking input, and \(\mathbf{e}^{(l)}_{c}\) represents the local conversion error caused by changing the smooth activation function to the spiking activation (IF). A major

Figure 1: Illustration of \(\mathrm{ClipFloor}\) function and its relationship to \(\mathrm{ReLU}\) (left) and \(\tanh\) (right).
result in [49] is that we can bound the conversion error \(\mathbf{e}^{(n)}\), with the weighted sum of the local conversion errors \(\mathbf{e}^{(l)}_{c}\). This allows us to minimize the conversion error via optimizing each \(\mathbf{e}^{(l)}_{c}\).

### Results for general activation functions

In fact, the results can be generalized to activation functions other than \(\mathrm{ReLU}\) by similar techniques in [49]. Since the generalized activation functions may have negative values, we introduce the idea of negative threshold, a concept in SNNs that allows neurons to fire both positive and negative spikes, depending on their membrane potential [50]. A positive spike occurs when the membrane potential exceeds the positive threshold, and a negative spike occurs when it is below the negative threshold. This mimics the biological behavior of neurons that do not fire a spike when the membrane potential does not reach the threshold. Negative threshold can be applied to different types of SNNs and learning functions depending on the problem domain and the data characteristics. It is very helpful when the dataset contains negative values.

To formulate the conversion with calibration for generalized activations, we consider the conversion error decomposition

\[\begin{split}\mathbf{e}^{(l)}&=\mathbf{x}^{(l)}- \bar{\mathbf{s}}^{(l)}\\ &=f(\mathbf{W}^{(l)}\mathbf{x}^{(l-1)})-\mathrm{ClipFloor}( \mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l-1)})\\ &=f(\mathbf{W}^{(l)}\mathbf{x}^{(l-1)})-f(\mathbf{W}^{(l)}\bar{ \mathbf{s}}^{(l-1)})+f(\mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l-1)})-\mathrm{ ClipFloor}(\mathbf{W}^{(l)}\bar{\mathbf{s}}^{(l-1)})\\ &=\mathbf{e}^{(l)}_{r}+\mathbf{e}^{(l)}_{c}\end{split} \tag{2}\]

where \(\mathbf{e}^{(l)}_{r}\) and \(\mathbf{e}^{(l)}_{c}\) have the same meaning as in Eq 1. Then we can also use the weighted sum of the local conversion errors \(\mathbf{e}^{(l)}_{c}\) to bound the total conversion error \(\mathbf{e}^{(n)}\) by the following theorem.

**Theorem 1**: _For any activation function whose function values and first-order derivatives can be uniformly approximated by piecewise linear functions and up to second-order derivatives are bounded, then the conversion error in the final network output space can be bounded by a weighted sum of local conversion errors, given by_

\[\mathbf{e}^{(n),\top}\mathbf{H}^{(n)}e^{(n)}\leq\sum_{l=1}^{n}2^{n-l+1} \mathbf{e}^{(l),\top}_{c}(\mathbf{H}^{(l)}+K_{L}^{(l)}\sqrt{L}\mathbf{I}) \mathbf{e}^{(l)}_{c} \tag{3}\]

_where \(L\) is the training loss._

We present the detailed proof in the Appendix. The main technique is to find piecewise linear functions that approximate the smooth activation function.

Therefore, the total conversion error is bounded from above by the local conversion errors. To achieve more accurate conversion performance, we can minimize the total conversion error by minimizing the local conversion error layerwise, which can be easily implemented as in [49]. We enlarge the last layer threshold to preserve the maximum value of the output. The details are discussed in the Appendix.

## 4 Results

### Function regression

We first present an example of function regression: using neural networks to approximate the \(\sin\) function. For the training dataset, the input is the uniform mesg points on \([-\pi.\pi]\), and the output is the values of \(\sin\) on these points. The ANN model has two intermediate layers, each of them has 40 neurons. The activation function is \(\tanh\) except for the last layer; there is no activation function for the last layer. The network is trained with the Adam optimizer until the training error is less then \(1^{-7}\). Then, we convert the ANN to SNN with advanced calibration with different numbers of time steps. The results are shown in Figure 2. To further investigate the impact of \(T\) on the conversion error, we train networks with different intermediate layer numbers \(L=2,3,4\) and neuron numbers per layer \(N=20,40,60,80,100\). All the other setups are the same. We obtain the results shown in Figure 3. We can find that the conversion error decreases with larger \(T\) (more specifically, conversion error \(\sim 1/T\) where ) when \(T<32\), but it becomes stable or larger after \(T\geq 32\). And for neural network with fixed depth, larger layer width usually leads to smaller conversion error. However, for fixed layer width, deeper neural networks will not bring significantly better conversion performance.

To validate the Theorem 1, we need to compute \(\mathbf{e}^{(l),\top}\mathbf{H}^{(l)}\mathbf{e}^{(l)}\) for \(l=1,2,\dots,n\). Since \(\mathbf{H}^{(l)}\) are intractable, we can only use the identity matrix to replace it and obtain some qualitative results. That is, compute \(\left\|\mathbf{e}^{(l)}\right\|^{2}\) instead of \(\mathbf{e}^{(l),\top}\mathbf{H}^{(l)}\mathbf{e}^{(l)}\). So the computed RHS is \(\sum_{l=1}^{n}2^{n-l+1}\mathbf{e}^{(l),\top}\mathbf{e}^{(l)}\). We train the ANN with layer numbers \(L=2,3,4\) with \(100\) neurons per layer. All the other setups are the same. The results are shown in Figure 4. Although the RHS term is not exactly the same as in 1, the trend agrees with our statement, that conversion error decreases as the RHS term does.
Figure 4: The number of layers are \(L=2,3,4\) from left to right. The error here refers to \(\mathbf{e}^{(n)}\). RHS is defined as before. We can find that the total conversion error is smaller than the computed RHS and decrease with the RHS as well, which is an emprical validation of the Theorem 1.

Figure 3: The number of layers are \(L=2,3,4\) from left to right. The error here refers to \(\mathbf{e}^{(n)}\). When \(T\leq 64\), the conversion error follows \(\mathbf{e}^{(n)}\approx 1/T\), and \(\mathbf{e}^{(n)}\) will get stable after \(T\) is already very large.

Figure 2: Results of converting an ANN, which is trained to approximate \(\sin\), to SNN. The number of time steps is \(T=8,32,128\) from left to right. The output of SNN is close to the ground truth, which is the output of ANN. With increasing T, the conversion error becomes smaller. When \(T=128\), the SNN output curve is almost smooth and similar to the ground truth.
### PINNs

To show the power of SNN conversion in regression tasks, we train PINNs, a MLP-based neural network which can solve PDEs, and convert them to SNNs. We present results for the Poisson equation, the diffusion-reaction equation, the wave equation, the Burgers equation, and the Navier-Stokes equations.

Poisson equationThe Poisson equation is often used to describe the potential field in electromagnetic theory. Here we solve the following boundary value problem of Poisson equation

\[\begin{split}-\Delta u(x)&=1,\quad x\in\Omega=[-1, 1]\times[-1,1]\\ u(x)&=0,\quad x\in\partial\Omega\end{split} \tag{4}\]

with PINN. The network has \(3\) intermediate layers, each of which has 100 neurons. The activation function is \(\tanh\) except for the last layer. The network is trained for 50,000 epochs. Then we convert it into SNN. The results are shown in Figure 5.

We observe that the SNN converted with calibration can achieve the same magnitude of error as the PINN evaluated as an ANN, while the SNN converted without using calibration can just obtain a rough shape of the solution and has much larger error.

Diffusion-reaction equationThis equation models reactive transport in physical and biologicak systems. Here we use PINN to solve the diffusion-reaction equation with the following initial condition:

\[\begin{split} u_{t}-u_{xx}&=ku^{2},\quad x\in \Omega=[-1,1]\\ u(x,0)&=\exp\!\left(-\frac{x^{2}}{2\sigma^{2}}\right) \end{split} \tag{5}\]

where \(k=1\), \(\sigma=0.25\) up to time \(T=0.01\). The network has \(3\) intermediate layers, each of which has 100 neurons. The activation function is \(\tanh\) except for the last layer. The network is trained for 100,000 epochs. Then we convert it into SNN.

The results are shown in 6.

Figure 5: Poisson equation: The results of converting a PINN solving the poisson equation (4). Figure 4(a) is the reference solution. Figure 4(b) is the PINN result. Figure 4(c) is the result of the SNN converted from the PINN without using calibration. Figure 4(c) is the result of the SNN converted from the PINN using calibration. Here the L2 error and relative error are defined as \(\left\|\mathbf{x}^{(n)}-\mathbf{s}^{(n)}\right\|_{2}\) and \(\left\|\mathbf{x}^{(n)}-\mathbf{s}^{(n)}\right\|_{2}\)/\left\|\mathbf{x}^{(n) }\right\|_{2}\), where \(\mathbf{x}^{(n)}\) is the reference solution and \(\mathbf{s}^{(n)}\) is the neural network output. \(\left\|\cdot\right\|_{2}\) is the \(l^{2}\) norm, which is the root of mean square error.

Figure 6: Reaction-diffusion equation: The results of converting a PINN solving the nonlinear heat equation (5). Figure 5(a) is the reference solution. Figure 5(b) is the PINN result. Figure 5(c) is the result of the SNN converted from the PINN without using calibration. Figure 5(c) is the result of the SNN converted from the PINN using calibration.
Wave equationHere we use PINN to solve the a wave equation with the following initial and boundary conditions:

\[\begin{split} u_{tt}-u_{xx}&=0,\quad x\in\Omega=[-1,1]\\ u(x,0)&=\begin{cases}1&x\in[-0.245,0.245]\\ 0&x\in[-1,-0.6]\cap[0.6,1]\\ \mathrm{linear}&\mathrm{otherwise}\end{cases}\\ u(-1,t)&=u(1,t)=0\end{split} \tag{6}\]

up to time \(T=0.5\). The network has \(3\) intermediate layers, each of which has 100 neurons. The activation function is \(\tanh\) except for the last layer. The network is trained for 100,000 epochs. Then we convert it into SNN. The results are shown in 7.

Viscous Burgers equationThe Burgers equation is a prototype PDE representing nonlinear advection-diffusion occurs fluid mechanics. Here we solve the following problem of viscous Burgers equation:

\[\begin{split}\frac{\partial u}{\partial t}-\frac{\partial}{ \partial x}(\frac{1}{2}u^{2})&=\nu\frac{\partial^{2}u}{\partial x ^{2}},\quad(x,t)\in[0,2\pi]\times[0,4]\\ u(x,0)&=\sin(x)\end{split} \tag{7}\]

with PINN. The network has \(6\) intermediate layers, each of which has \(40\) neurons. The activation function is \(\tanh\) except for the last layer. The network is trained for 100,000 epochs. Then we convert it into SNN. The results are shown in 8.

Here we can find that conversion without calibration does not give correct position of steep gradient as the position is moving left (see Figure 7(c)). But the conversion with calibration keeps the steep gradient position correct, which is important for physics.

Due to the discontinuous nature of SNN, the conversion results are not smooth. However, we can still apply some filters to smooth the outputs. For example, we apply FFT to the conversion results and remove the high frequencies, and the results are shown in Figure After the smoothing, conversion error becomes lower, hence can be adopted as the postprocessing procedure.

Figure 8: Burgers equation: The results of converting a PINN solving the viscous Burgers equation (7). Figure 7(a) is the reference solution. Figure 7(b) is the PINN result. Figure 7(c) is the result of the SNN converted from the PINN without using calibration. Figure 7(c) is the result of the SNN converted from the PINN using calibration.

Figure 7: Wave equation: The results of converting a PINN solving the wave equation (6). Figure 6(a) is the reference solution. Figure 6(b) is the PINN result. Figure 7(c) is the result of the SNN converted from the PINN without using calibration. Figure 7(c) is the result of the SNN converted from the PINN using calibration.
### Accelerating training with separable PINNs (SPINNs)

A current limitation of PINN-SNN conversion is the computational resources it requires, especially for training the ANN. Herein, we highlight the effectiveness of using the separable physics-informed neural networks to spiking neural networks (SPINN-SNN) conversion. The SPINN-SNN conversion pipeline enhances the speed of operations and provides great computational efficiency compared to the direct application of PINN. We implement the SPINN-SNN conversion to address two PDEs: a two-dimensional viscous Burgers equation and a three-dimensional unsteady Beltrami flow. We provide a comparative analysis of the training time taken by standard PINN and the SPINN-SNN conversion pipeline. Experimental results reveal that the application of SPINN-SNN conversion greatly enhances the speed of the training, particularly when solving high dimensional PDEs. Figure 10 displays a runtime comparison between SPINN-SNN conversion and PINN, applied to two and three-dimensional problems. When addressing the two-dimensional Burgers equation, the SPINN-SNN conversion process is approximately 1.7 times faster than PINN. The superiority of SPINN-SNN conversion becomes more apparent while solving the three-dimensional Beltrami flow problem, which is over 60 times faster than PINN. Notably, the time necessary for the SNN calibration remains relatively constant, regardless of the problem dimensionality. This indicates that the benefits of the SPINN-SNN conversion pipeline become increasingly prominent with the rise in the dimensionality of the problem.

Viscous Burgers equationIn order to conduct a fair comparison between the PINN-SNN and SPINN-SNN conversions, the setup of the viscous Burgers equation is kept consistent with that described in Equation 7, and the same set of hyperparameters is utilized. Figure 11 presents the results of converting a SPINN to solve the Burgers' equation. The SPINN has individual subnetworks for each independent variable, \(x\) and \(t\). Each of these subnetworks comprises three intermediate layers, each layer containing 40 neurons, and employs the tanh activation function, except in the last layer. As depicted in Figure 11, the conversion from SPINN provides an accuracy level comparable to that of PINN. An SNN converted with calibration achieves a significantly smaller error than one converted without calibration.

Beltrami FlowThe Navier-Stokes equations are fundamental in fluid mechanics as they mathematically represent the conservation of momentum in fluid systems, and there has been significant advancement in solving Navier-Stokes flow problems using scientific machine learning methods [51; 52; 53]. The Navier-Stokes equations can be presented in two forms: the velocity-pressure (VP) form and the vorticity-velocity (VV) form. The incompressible Navier-Stokes equations, in their VP form, are as follows:

\[\frac{\partial\mathbf{u}}{\partial t}+(\mathbf{u}\cdot\nabla) \mathbf{u} =-\nabla p+\frac{1}{Re}\nabla^{2}\mathbf{u} \tag{8}\] \[\nabla\cdot\mathbf{u} =0\]

Figure 10: The runtimes of the SPINN-SNN conversions solving the viscous Burgers equation (7) and Beltrami flow (8). Figure 10a is the runtime for the 2D viscous Burgers equation. Figure 10b is the runtime for the 3D Beltrami flow.

Figure 9: Figure 9a is the original conversion result. Figure 9b is the smoothed conversion result.
We chose the spatial domain to be \(\Omega\in[-1,1]^{2}\) and time interval to be \(\Gamma\in[0,1]\). Here, \(t\) is non-dimensional time, \(\mathbf{u}(\mathbf{x},t)=[u,v]^{T}\) the non-dimensional velocity in the \((x,y)\)-directions, \(p\) the non-dimensional pressure, and the Reynolds number \(Re=\frac{U_{ref}D_{ref}}{v}\) is defined by characteristic length (\(D_{ref}\)), reference velocity (\(U_{ref}\)), and kinematic viscosity (\(v\)). In this example, we simulate a three-dimensional unsteady laminar Beltrami flow where \(Re=1\). The analytical solution of the Beltrami flow is [54]:

\[\begin{split} u(x,y,t)&=-\cos x\sin y\ e^{-2t}\\ v(x,y,t)&=\sin x\cos y\ e^{-2t}\\ p(x,y,t)&=-\frac{1}{4}(\cos 2x+\cos 2y)\ e^{-4t} \end{split} \tag{9}\]

The boundary and initial conditions are extracted from this exact solution. The PINN network comprises 4 intermediate layers, with each one containing 128 neurons. The activation function is tanh applied taoi all layers except the final layer. The network is trained for 20,000 epochs before being converted into an SNN. The SPINN consists of separate subnetworks for each independent variable, u, v, and p. Every subnetwork has 2 intermediate layers, with each layer consisting of 50 neurons. They all utilize the tanh activation function, apart from the last layer. Figure 12 illustrates that the SPINN conversion offers similar accuracy compared to PINN. The error for pressure is slightly more than the velocity errors in both the \(x\) and \(y\) axes. Nevertheless, the SNN, once calibrated and converted using SPINN, attains good accuracy and notable speed improvements in comparison to PINN.

### Firing rates of the converted SNN

To demonstrate the potential efficiency of converting the ANN to the SNN, we computed the spiking rates of different equations. The spiking rate is defined as the ratio of non-zero values in the output of each layer. Prior works have suggested that SNNs with lower spiking rates will translate to energy efficient impelmentations on neurromorphic hardware. The results are shown in Table 1. We observe \(<0.5\) spiking rate in most cases demonstrating that SNNs only expend \(<50\%\) of their network computations.

\begin{table}
\begin{tabular}{c|c c} \hline Equations & Number of parameters & Spiking rate \\ \hline Poisson & 20601 & 0.3727 \\ Diffusion-reaction & 20601 & 0.2879 \\ Wave & 20601 & 0.5721 \\ Burgers & 8361 & 0.7253 \\ Burgers (Separable PINN) & 15500 & 0.2831 \\ N-S (Beltrami flow, Separable PINN) & 30900 & 0.1754 \\ \hline \end{tabular}
\end{table}
Table 1: The spiking rate of SNNs for different equations.

Figure 11: Burgers equation with Separable PINN (SPINN): The results of converting a Separable PINN solving the viscous Burgers equation (7). Figure 11a is the reference solution. Figure 11b is the Separable PINN result. Figure 11c is the result of the SNN converted from the Separable PINN without using calibration. Figure 11c is the result of the SNN converted from the Separable PINN using calibration.
## 5 Conclusion

We have successfully extended the SNN calibration method proposed in [49] to a more generalized class of activation functions beyond ReLU. The original proof relied on the specific property of ReLU's second-order derivative being zero, but our approach relaxed this constraint by incorporating the training loss as an additional term in the bound. We demonstrated the effectiveness of our method through various examples, including PINN and Separable PINN [43] (a variant of PINN), where the activation functions are not ReLU. The results demonstrated that our approach achieved good accuracy with low spike rates, making it a promising and energy-efficient solution for scientific machine learning. By enabling the conversion of a wider range of neural networks to SNNs, this method opens up new possibilities for

\begin{table}
\begin{tabular}{c|c c} \hline \hline Equations & \(L^{2}\) error & Relative \(L^{2}\) error \\ \hline Poisson & \(7.8508\times 10^{-3}\) & \(4.7095\times 10^{-2}\) \\ Diffusion-reaction & \(2.8766\times 10^{-2}\) & \(6.3267\times 10^{-2}\) \\ Wave & \(3.8965\times 10^{-2}\) & \(6.5308\times 10^{-2}\) \\ Burgers & \(3.9884\times 10^{-2}\) & \(6.9781\times 10^{-2}\) \\ Burgers (Separable PINN) & \(6.6495\times 10^{-2}\) & \(1.1634\times 10^{-1}\) \\ N-S (Beltrami flow, Separable PINN) & \(8.2512\times 10^{-3}\) & \(4.3273\times 10^{-2}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The \(L^{2}\) and relative \(L^{2}\) error of converted SNN for different equations.

Figure 12: Beltrami flow with Separable PINN (SPINN): The results of converting a SPINN solving the Beltrami flow problem(7). Figure 12(a,e,i) are the reference solutions for \(\mathbf{u},\mathbf{v},\mathbf{p}\) respectively. Figure 12(b,f,j) are the SPINN results. Figure 12(c,g,k) are the PINN results. Figure 12(d,h,i) are the results of the SNN converted from the SPINN with calibration.
harnessing the temporal processing capabilities and computational efficiency of SNNs in various scientific applications. The proposed approach contributes to advancing the field of spiking neural networks and their potential for practical real-world implementations in edge computing. Future research can explore further extensions of the calibration technique to other types of activation functions and investigate its performance on more complex neural network architectures.

## 6 Acknowledgements

This work was supported in part by the DOE SEA-CROGS project (DE-SC0023191), the ONR Vannevar Bush Faculty Fellowship (N00014-22-1-2795), CoCoSys- a JUMP2.0 center sponsored by DARPA and SRC, and the National Science Foundation CAREER Award.
