Non-negative isomorphic neural networks for photonic neuromorphic
  accelerators
Neuromorphic photonic accelerators are becoming increasingly popular, since
they can significantly improve computation speed and energy efficiency, leading
to femtojoule per MAC efficiency. However, deploying existing DL models on such
platforms is not trivial, since a great range of photonic neural network
architectures relies on incoherent setups and power addition operational
schemes that cannot natively represent negative quantities. This results in
additional hardware complexity that increases cost and reduces energy
efficiency. To overcome this, we can train non-negative neural networks and
potentially exploit the full range of incoherent neuromorphic photonic
capabilities. However, existing approaches cannot achieve the same level of
accuracy as their regular counterparts, due to training difficulties, as also
recent evidence suggests. To this end, we introduce a methodology to obtain the
non-negative isomorphic equivalents of regular neural networks that meet
requirements of neuromorphic hardware, overcoming the aforementioned
limitations. Furthermore, we also introduce a sign-preserving optimization
approach that enables training of such isomorphic networks in a non-negative
manner.
Manos Kirtas, Nikolaos Passalis, Nikolaos Pleros, Anastasios Tefas
# Non-negative isomorphic neural networks for photonic neuromorphic accelerators

###### Abstract

Neuromorphic photonic accelerators are becoming increasingly popular, since they can significantly improve computation speed and energy efficiency, leading to femtojoule per MAC efficiency. However, deploying existing DL models on such platforms is not trivial, since a great range of photonic neural network architectures relies on incoherent setups and power addition operational schemes that cannot natively represent negative quantities. This results in additional hardware complexity that increases cost and reduces energy efficiency. To overcome this, we can train non-negative neural networks and potentially exploit the full range of incoherent neuromorphic photonic capabilities. However, existing approaches cannot achieve the same level of accuracy as their regular counterparts, due to training difficulties, as also recent evidence suggests. To this end, we introduce a methodology to obtain the non-negative isomorphic equivalents of regular neural networks that meet requirements of neuromorphic hardware, overcoming the aforementioned limitations. Furthermore, we also introduce a sign-preserving optimization approach that enables training of such isomorphic networks in a non-negative manner.

## 1 Introduction

Neuromorphic architectures have gained increasing attention recently, as they provide novel electronic solutions focusing on memory architectures suitable for high-speed matrix-based calculations, which cover a significant fraction of the calculations involved during the inference of Deep Learning (DL) models, with low energy consumption [33; 21]. Neuromorphic photonics are among the most promising approaches, with recent layouts already paving a realistic road map towards femtojoule per MAC efficiencies [39], leveraging advantages in materials and waveguide technologies [7; 11], enabling ultra-fast analog processing and vector-matrix multiplication with almost zero power consumption [37; 15], significantly exceeding their electronic counterparts [29].

However, integrating DL models into physically implemented devices comes with additional cost if their physical properties are not taken into account during the implementation phase. For example, the vast majority of currently available photonic architectures relies on incoherent layouts and is facing challenges to support negative quantities, since optical signals get naturally converted into power signals during the nonlinear process that has to take place at the activation stage, implying that the sign information of the weighted sum is ignored. This mechanism turns the use of negative number representations within a Photonic Neural Network (PNN) into a challenging process, typically enforcing the adoption of higher complexity hardware architectures, such as balanced photodetector schemes [36], biasing configurations [26] and signal transformation blocks [24].

Our main contributions are two-fold:
* We propose a method to transform trained Artificial Neural Networks (ANNs) to their fully non-negative equivalent.
* We propose an optimization method which ensures that the model's parameters will remain non-negative during training, enabling one to either train a non-negative model from scratch or continue training in a non-negative manner.

## 2 Related Works

Neuromorphic PhotonicsPNNs deployment requires the employment of optically enabled mechanisms and photonic building blocks to realize the respective signals and parameters of the neural layer. Input signals are typically imprinted in the optical domain using optical modulators, while weighting functionality generally requires the use of variable optical attenuation schemes [38]. Several approaches have been utilized to date in on-chip weight implementations of photonic neural layers, including tunable i) optical filtering mechanisms [38; 24; 28], ii) waveguide absorption techniques [7; 11] and, iii) optical gain approaches [9; 23]. Summation is then performed in the optical domain via i) wavelength multiplexers or power combiners in the case of incoherent layouts [11; 13; 36] and ii) interferometric stages and optical couplers in the case of coherent architectures [37; 14; 15; 23]. Finally, the non-linear activation can be offered by either i) optoelectronic schemes [11; 14; 28] or ii) all-optical non-linear modules [11; 14; 28].

Furthermore, there are several works that take into account the unique nature of neuromorphic photonics and design the training and deployment of models accordingly [25; 27; 24]. Although such approaches integrate transfer function- and noise-related limitations of photonic hardware [18; 32; 28], leading to significant performance improvements during deployment, they typically ignore the sign limitation of photonic architectures.

Cost ReductionDecreasing the hardware complexity of PNNs has been extensively studied in the literature, and several approaches have been proposed, ranging from pruning [13], to quantization [17], and mixed-precision representations [14]. However, even the simplest PNN implementation would require amplitude modulators for its inputs, meaning that in this case both the input signal modulators and photonic weighting stage control just the amplitude of the optical field, resulting in non-negative networks [39]. Introducing sign information in a PNN has to incorporate a new physical dimension that can be used to correlate its state with the sign. Coherent photonic architectures offer a rather simple way of representing signed optical signals by correlating the sign to the phase of the propagating optical fields [15; 24; 28]. However, this requires additional phase modulation circuitry both at the input signal generation and weighting stage, also requiring more complex circuitry at the non-linear activation stage in order to account for the sign information of the weighted optical sum at the activation unit, such as an optical biasing scheme or a coherent receiver [34; 23; 15; 13].

Non-negative ANNsAlthough there are some works studying non-negativity on ANNs, they mostly target partial nonnegative architectures that are focused on reconstruction (e.g. autoencoders) [3; 8] applied on small datasets or non-traditionally used ANNs (such as Pyramid Neural Networks) [12], facing difficulties in scaling ability, generalization in DL architectures (such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs)) and result in significantly performance degradation that hinders their application on both DL and neuromorphic photonics. Our proposed framework can be applied on traditionally used architectures without any major change and performance degradation, since it is based on the non-negative isomorphic representation of traditional models.

Non-negative TrainingExisting training methods, which are oriented to non-negative architectures, are based on either limited memory optimization (such as Quasi-Newton) [3; 8] or in non-gradient based optimization methods [12], clipping parameters during the backpropagation, constraining in such a way the variance of parameters during the first epochs of training, which leads on convergence difficulties (typical examples are demonstrated in the Appendix). Our proposed non-negative optimization method is based on a multiplicative alternative of the Stochastic Gradient Descent (SGD) optimizer that combined with the proposed non-negative transformation ensures that the variance of parameters will not diminish, allowing the training process to proceed smoothly without reducing the variance.
Although multiplicative updates have been extensively studied during the early years of machine learning research [2; 19], to the best of our knowledge, this is the first work that investigates them in the context of non-negative training. Even works that target sign-preserving optimization are limited to studying both the excitatory and inhibitory functions of neurons, obeying in general Dale's rule [10], pointing out mostly the anatomical correlation with biological synapses [1; 4]. In such a direction is also the recent work [5] that leverages multiplicative updates on Adam to train lower bit-width synapses stored in logarithmic numbers oriented to software-hardware co-design. The authors note that the sign-pattern of initialized weights can possibly restrict the expressive ability of networks. Our work goes beyond such approaches since we claim that even with positive sign only parameters, we can acquire an expressive isomorphic representation of a network by applying the appropriate transformation and training it in a non-negative manner.

Isomorphism in ANNsIsomorphism is a general mathematical property that is especially useful in graph theory. As a result, there are several works that consider isomorphism to extract more expressive representations of the input graphs on graph classification problems [6; 22]. As far as we know, this is the first work studying the isomorphism of the networks and proposes a structured methodology to acquire their non-negative equivalent opening a new research direction with possible wider implications, e.g., design of isomorphic networks that are adjusted for conventional accelerators as well, providing potentially more explainable DL architectures [3; 8; 12].

## 3 On the Difficulty of Training Non-negative Neural Networks

Conceptually, ANNs project the input features to a latent dimensional space, aiming to represent them in a more separable way, seeking for a hyperplane decision boundary that can classify them in an optimal way. However, classifying non-negative features even when they are linear separable is often impossible when using non-negative parameters to a linear classifier. For example, in a two dimensional binary classification task, as depicted in Figure 1.a, the decision boundary of a traditional logistic regression classifier is given by:

\[x_{1}=-\frac{w_{0}}{w_{1}}x_{0}-\frac{b}{w_{1}}\in\mathbb{R}, \tag{1}\]

where \(x_{i}\in[0,1]\) is the input, \(w_{i}\in\mathbb{R}\) and \(b\in\mathbb{R}\) are the weights and biases of the classifier respectively and \(i\in\{0,1\}\). We can easily conclude that there is a positive slope decision, with the slope given by \(m=-w_{0}/w_{1}\), which requires a negative weight. In fact, training the linear classifier, using SGD, we obtain a positive slope decision boundary, as shown in Figure 1.a, achieving optimal performance with \(w_{0}<0\). On the other hand, constraining the classifier to non-negative parameters results in a negative slope decision boundary that is unable to discriminate the two classes, as presented in Figure 1.b.

Inspired by the fact that challenging computational tasks, such as calculating the motion of our solar system's planets in a geocentric manner, can be easily solved by changing the coordinate system, for example, calculating the motion of the planets on a heliocentric system, we considered isomorphism, i.e. the same behavior but with different implementation and/or parameters, to claim that an equivalent classifier with non-negative parameters exists. Such a non-negative isomorphic classifier produces the same classification outcome as the original one by applying a coordination change to the original

Figure 1: (a) The resulting positive-slope decision boundary as acquired by the traditional optimization process. (b) Regular training of the non-negative classifier results in a non-positive slope decision boundary. (c) The proposed transformation on the trained parameters of (a).
inputs and parameters. Indeed, in the example of Figure 1.a, we can easily transform the original problem and classifier, by shifting and rotating points and decision boundary, getting an equivalent non-negative classifier, as depicted in Figure 1.c, using only positive parameters. Such non-negative isomorphic classifier outcomes the decision boundary given by:

\[x_{1}=-\frac{|w_{0}|}{w_{1}}x_{0}^{\prime}-\frac{b-a\left|w_{0}\right|}{w_{1}}\in \mathbb{R} \tag{2}\]

where \(w_{0}<0\), \(w_{1}>0\), \(a=1\) and \(x_{0}^{\prime}=(a-x_{0})\). The non-negative classifier leads to a rotated decision boundary and is equal to the original classifier performance.

## 4 Non-negative Isomorphic Neural Networks

Definition 4.1 (Linear Neuron)Let \(z_{i}\in\mathbb{R}\) the response of the linear part of the \(i\)-th neuron of a fully connected layer, where \(i=1\ldots N\), given by:

\[z_{i}=u_{i}(\mathbf{x})=\sum_{j=1}^{M}w_{ij}x_{j}+b_{i}\in\mathbb{R}, \tag{3}\]

where \(\mathbf{w}_{i}\in\mathbb{R}^{M}\), \(b_{i}\in\mathbb{R}\) and \(\mathbf{x}\in\mathbb{R}^{M}\) are the weight, bias and input vectors of the \(i\)-th neuron, respectively. Assuming an activation function \(g(\cdot):\mathbb{R}\rightarrow\mathbb{R}_{+}\), where \(\mathbb{R}_{+}\) denotes the set of positive real values, the outputs of \(i\)-th neuron is given by:

\[y_{i}=g(z_{i})\in\mathbb{R}_{+}. \tag{4}\]

Theorem: For every linear neuron given by Definition 4.1 there is a non-negative isomorphic with the linear response provided as:

\[u_{i}^{\prime}(\mathbf{x}^{\prime})=z_{i}^{\prime}=\sum_{j=1}^{M}|w_{ij}|x_{j}^{ \prime}+b_{i}^{\prime}\in\mathbb{R}_{+}, \tag{5}\]

and the output as:

\[y_{i}^{\prime}=g_{c}(u_{i}^{\prime}(\mathbf{x}^{\prime}))\in\mathbb{R}_{+}, \tag{6}\]

where:

\[x_{j}^{\prime}=\begin{cases}a-x_{j}&\text{if }w_{ij}<0\\ x_{j}&\text{otherwise}\end{cases}. \tag{7}\]

Then, appropriate parameters \(b_{i}^{\prime}\in\mathbb{R}_{+}\) and \(a\in\mathbb{R}_{+}\), as well as activation function \(g_{c}(\cdot):\mathbb{R}\rightarrow\mathbb{R}_{+}\) exist so the neuron leads to the same response, i.e., \(y_{i}=y_{i}^{\prime}\). \({}_{\blacksquare}\)

Proof: Equation 3 can be written as:

\[z_{i}=-\sum_{\{j|w_{ij}<0\}}|w_{ij}|x_{j}+\sum_{\{j|w_{ij}>0\}}w_{ij}x_{j}+b_{ i}\in\mathbb{R}. \tag{8}\]

Assuming that every input of a linear layer is non-negative, which can be enforced in the input of the network by trivially performing normalization to features, then, by adding and subtracting the quantity \(\alpha\sum_{\{j|w_{ij}<0\}}|w_{ij}|\), where \(a\geq max(\mathcal{X})\) and _max_ denotes the maximum element in the feasible set \(\{\mathbf{x}:\mathbf{x}\in\mathbb{R}_{+}^{N}\}\), defined as \(\mathcal{X}\), the Equation 8 can be written as:

\[z_{i}^{\prime}=\alpha\sum_{\{j|w_{ij}<0\}}|w_{ij}|-\sum_{\{j|w_{ij}<0\}}|w_{ij} |x_{j}+\sum_{\{j|w_{ij}>0\}}w_{ij}x_{j}+\left(b_{i}-\alpha\sum_{\{j|w_{ij}<0\} }|w_{ij}|\right)\in\mathbb{R}. \tag{9}\]

This allows us to rotate the input feature space, similarly to Section 3. To this end, the first two terms can be merged, while the last term can be integrated to an updated bias term:

\[\tilde{b}_{i}=b_{i}-\alpha\sum_{\{j|w_{ij}<0\}}|w_{ij}|\in\mathbb{R}. \tag{10}\]
Therefore, Equation 9 can be written as:

\[z_{i}^{\prime}=\sum_{\{j|w_{ij}<0\}}|w_{ij}|(a-x_{j})+\sum_{\{j|w_{ij}>0\}}w_{ij} x_{j}+\tilde{b}_{i}\in\mathbb{R}. \tag{11}\]

To simplify the Equation 11, we can define the rotated input as:

\[x_{j}^{\prime}=f(x_{j})=\begin{cases}a-x_{j}&\text{if $w_{ij}<0$}\\ x_{j}&\text{otherwise},\end{cases} \tag{12}\]

where \(x_{j}^{\prime}\in\mathbb{R}_{+}\). Similarly, the updated non-negative weights of the \(i\)-th neuron can be directly calculated as:

\[\mathbf{w}_{i}^{\prime}=|\mathbf{w}_{i}|\in\mathbb{R}_{+}^{M}, \tag{13}\]

where \(|\cdot|\) denotes the element-wise absolute value, i.e., \(w_{ij}^{\prime}=\{|w_{ij}|,j=0\dots M\}\), since all weights involved in (11) are positive. Furthermore, the new non-negative biases are transformed according to the following formula:

\[b_{i}^{{}^{\prime}}=\tilde{b}_{i}+c\in\mathbb{R}_{+}, \tag{14}\]

where \(c\) is computed as:

\[c=max\{|\tilde{b}_{0}|,\dots|\tilde{b}_{N}|\}\in\mathbb{R}_{+}, \tag{15}\]

denoting _activation shifting point_. The _activation shifting point_ is applied to the original activation to slide it into the input domain, leading to the same output as the original network. To ensure that the network will work in a non-negative manner, the original activation function has to work on a non-negative output space, i.e., \(g(z):\mathbb{R}\rightarrow[g_{min},g_{max}]\), where \(0\leq g_{min}<g_{max}<\infty\), with the shifted activation is calculated as:

\[g_{c}(x)=g(x-c)\in\mathbb{R}_{+}. \tag{16}\]

Typically the \(a^{(k+1)}\) of the next layer, where the \(k\) is the number of current layer, can be set to \(g_{max}\), validating the aforementioned assumption, \(a^{(k+1)}=g_{max}=max(\mathcal{X})\).

Since Non-negative Transformation (NNT) targets neuromorphic architectures, the _activation shifting_ can be integrated during the design phase of the activation function, allowing one to integrate it in layer. This evaluates Theorem 4.1 since the original parameters are transformed to their non-negative equivalent, with the isomorphic neuron has the same response as the original one.

The computational complexity of the proposed transformation is equal to the number of trainable parameters and can be trivially acquired by taking into account its algorithmic representation, presented in Appendix. More precisely, if the \(\mathbf{\theta}^{(k)}\) defines the trainable parameter of \(k\)-th layer, consisting of weights \(\mathbf{w}^{(k)}\) and biases \(\mathbf{b}^{(k)}\), with \(k=1\dots n\), then the computational complexity of the transformation is \(O(|\bigcup\{\mathbf{\theta}^{(k)}\}_{1}^{n}|)\), where \(|\cdot|\) denotes the number of elements of the corresponding set.

Definition 4.2 (Recurrent Neuron): Let \(\mathbf{x}_{t}\in\mathbb{R}^{M}\) denote \(M\) features fed on \(i\)-th neuron at \(t\)-th time-step. Each recurrent neuron processes two signals: a) the current input signal, which is weighted by \(\mathbf{w}_{i}^{(in)}\in R^{M}\), and b) a recurrent feedback signal, denoted by \(\mathbf{y}_{t-1}^{(r)}\in R^{N}\) and weighted by a set of recurrent weights \(\mathbf{w}_{i}^{(r)}\in R^{N}\), which corresponds to the output of the \(N\) recurrent neurons of the same layer in a previous time step. The linear response of \(i\)-th neuron is given by:

\[u_{ti}^{(r)}(\mathbf{x}_{t},\mathbf{y}_{t-1})=\sum_{j=1}^{M}w_{ij}^{(in)}x_{tj}+\sum_{ j=1}^{N}w_{ij}^{(r)}y_{t-1,j}^{(r)}+b_{i}\in\mathbb{R}_{+}, \tag{17}\]

and the outputs as:

\[y_{ti}^{(r)}=g(u_{ti}^{(r)}(\mathbf{x}_{t},\mathbf{y}_{t-1}^{(r)}))\in\mathbb{R}_{+}, \tag{18}\]

**Theorem 4.2**: _For every recurrent neuron given by Definition 4.2 there is a non-negative isomorphic with the linear response provided as:_

\[z_{ti}^{(r)^{\prime}}=u_{ti}^{(r)^{\prime}}(\mathbf{x}_{t}^{\prime},\mathbf{y}_{t-1}^{ \prime})=\sum_{j=1}^{M}|w_{ij}^{(in)}|x_{tj}^{\prime}+\sum_{j=1}^{N}|w_{ij}^{ (r)}|y_{t-1,j}^{(r)^{\prime}}+b_{i}^{\prime}\in\mathbb{R}_{+}, \tag{19}\]
_and the output as:_

\[y_{ti}^{(r)^{\prime}}=g_{c}(u_{ti}^{(r)^{\prime}}(\mathbf{x}_{t}^{\prime},\mathbf{y}_{t-1 }^{(r)^{\prime}}))\in\mathbb{R}_{+}, \tag{20}\]

_where:_

\[x_{tj}^{\prime}=\begin{cases}a^{(in)}-x_{tj}&\text{if }w_{ij}^{(in)}<0\\ x_{tj}&\text{otherwise}\end{cases}, \tag{21}\]

_and:_

\[y_{t-1,j}^{(r)^{\prime}}=\begin{cases}a^{(r)}-y_{t-1,j}^{(r)}&\text{if }w_{ij}^{(r)}<0\\ y_{t-1,j}^{(r)}&\text{otherwise}\end{cases}. \tag{22}\]

_Then, appropriate parameters \(b_{i}^{\prime}\in\mathbb{R}_{+}\), \(a^{(in)}\in\mathbb{R}_{+}\) and \(a^{(r)}\in\mathbb{R}_{+}\), as well as activation function \(g_{c}(\cdot):\mathbb{R}\rightarrow\mathbb{R}_{+}\) exist so the neuron leads to the same response, i.e., \(y_{ti}^{(r)}=y_{ti}^{(r)^{\prime}}\)._

Proof 4.2:Recurrent neurons can conceptually be seen as two linear neurons in which the current input signal, \(x_{tj}\), is fed to one neuron and the recurrent feedback signal, \(y_{t-1,j}^{(r)}\), to the other. To this end, we can easily conclude that there is a non-negative isomorphic for recurrent neurons by applying Proof 4.1 on the first two terms of Equation 17.

**Definition 4.3** (2D Convolutional Neuron): Let \(\mathbf{X}\in\mathbb{R}^{C\times N\times M}\) denote the input of a 2D convolutional layer, where \(C\), \(N\), and \(M\) represent the number of channels, height and width of the input feature, with the layer consist of the kernel's weights \(\mathbf{W}\in\mathbb{R}^{D\times C\times N^{k}\times M^{k}}\), where \(D\) denotes the number of output channels, while \(N^{k}\) and \(M^{k}\) represent the height and width of the kernel. The bias is denoted as \(\mathbf{b}\in\mathbb{R}^{D}\). Convolutional neurons can be constructed by trivially extending Definition 4.1 by sliding the linear neuron over the input after flattening each input patch to a vector. Thus, starting from the \(c\)-th channel of the input and flattening the \(i\)-th \(N^{k}\times M^{k}\) sub-matrix of \(\mathbf{X}\in\mathbb{R}^{C\times N\times M}\), denoted as \(\mathbf{x}_{i}\in\mathbb{R}^{N^{k}M^{k}}\), we can define the \(z_{i}\) the linear output of the \(i\)-th element of \(d\)-th output channel given by:

\[z_{i}=u_{i}(\mathbf{x})=\sum_{j=1}^{M}w_{j}x_{ij}+b_{d}\in\mathbb{R}, \tag{23}\]

where \(\mathbf{w}\in\mathbb{R}^{N^{k}M^{k}}\) is the equivalent to the flattened kernel weight of \(\mathbf{W}_{cd}\in\mathbb{R}^{N^{k}_{i}\times M^{k}_{i}}\), where \(c\) and \(d\) are the current input and output channels, respectively. To this extent, Theorem 4.1 can also be applied to convolutionals, since they can be defined as linear neuron building blocks, allowing us to apply Proof 4.1. Without loss of generality the theorem can be applied to 1D, 3D and multidimensional convolutionals as well.

## 5 Non-negative Optimization

Transforming an already trained network to its non-negative isomorphic can be limiting in cases where continuing training is required. To this end, we propose an adjustments on SGD that constrains the trainable parameter \(\mathbf{\theta}^{(k)}\) of \(k\)-th layer, consisting of weights \(\mathbf{w}^{(k)}\) and biases \(\mathbf{b}^{(k)}\), to non-negative quantities. More precisely, we modify the additive update rule, in which sign shifting is attributed during SGD optimization, by normalizing the gradients using the non-linear function \(\tanh:\mathbb{R}\rightarrow(-1,1)\) and multiplying it by the absolute value of the parameter. The optimization algorithm picks a point \(\mathbf{\theta}_{\tau}^{(k)}\), at each time step \(\tau\), and updating it according to:

\[\mathbf{\theta}_{\tau}^{(k)}=\mathbf{\theta}_{\tau-1}^{(k)}+|\mathbf{\theta}_{\tau-1}^{(k )}|\tanh\left(-\eta_{in}\frac{\partial J}{\partial\mathbf{\theta}_{\tau-1}^{(k)}} \right)\eta_{out}\in\mathbb{R}^{+}, \tag{24}\]

where \(\eta_{in}\in\mathbb{R}^{+}\) is the inner and \(0<\eta_{out}\leq 1\) the outer learning rate. Essentially, the inner learning rate allows one to adjust the gradients regarding the working range of the used non-linearity. The outer learning rate affects the size of the step similarly to the learning rate used in traditionally applied optimization methods. The proposed optimization method is a sign-preserving alternative of SGD, named _non-negative stochastic gradient descent_ (NNSGD), and when combined with the NNT can be used to post-train or train from scratch DL models in a non-negative manner. Both proposed non-negative optimization is presented algorithmically as well in the Appendix.
## 6 Experimental Evaluation

We experimentally evaluate the proposed framework using a wide range of architectures and photonic configurations demonstrating its capabilities in three scenarios: a) transforming pretrained ANN to its non-negative isomorphic without any performance degradation, b) continuing training of a non-negative isomorphic network that is regularly pretrained, and c) non-negative training from scratch. We evaluate it in image classification (MNIST, Fashion MNIST, CIFAR10), malware classification (Malimg [30]), a large scale financial time-series forecasting (FI2010 [31]) and a simple natural language processing (Names [35]) task. Details about the experimental setup, models applied, datasets, photonic configurations and hyper parameter tuning process are provided in detail in the Appendix. Note that we applied the proposed framework to small models according to current neuromorphic photonics capabilities and limitations. Additionally, the employed hyper parameter tuning is applied to all scenarios for fairness since both proposed methods and evaluated baselines are targeting non-negative quantities with significantly different parameter distributions and magnitudes with those the SGD traditionally targets.

### Trained model transformation

In Table 1, we report the evaluation results of the proposed NNT when applied to traditionally trained DL models. More specifically, we optimize the network using the SGD optimizer and then we apply the proposed transformation to acquire its non-negative isomorphic network reporting if the performance matches exactly to the original one. In each case, after performing hyper parameter search, we evaluate the best configuration obtained (e.g. learning rate) evaluating them in 5 evaluation runs and report the average and variance of the evaluation accuracy (or F1 and \(\kappa\) scores on highly unbalanced datasets). The proposed transformation leads to the exact same accuracy irrespective of applied architectures, dataset, and/or photonic configuration. The proposed transformation enables us to exploit known techniques of optimization without constraining the model's parameters during training, which is shown to lead to performance degradation [3; 8; 12], and, in turn, transform to its non-negative isomorphic before being deployed on photonic hardware.

### Non-negative post training

We also evaluate the proposed transformation and non-negative optimization method when applied to regular pre-trained models. More specifically, we train models using SGD optimizer, and after a few epochs we transform the trained network and continue the training on its isomorphic model in a non-negative manner. We compare the proposed non-negative optimization method to a baseline non-negative training used in literature [3], given by the update rule \(\theta_{t}=max\{0,\theta_{t-1}-\eta(\partial J/\partial\theta_{t-1})\}\in \mathbb{R}^{+}\), applied on the SGD optimizer, denoted as Clipping Stochastic Gradient Descent (CSGD).

In Table 2, we report the average accuracy of the evaluation phase and the variance over 5 evaluation runs using MLPs. The proposed framework, including both transformation and NNSGD optimization, is reported in the 4th column. As baselines, we evaluate CSGD optimization used directly after pre

\begin{table}
\begin{tabular}{l|l|l c||c c} \multirow{2}{*}{Dataset} & \multirow{2}{*}{Architecture} & \multicolumn{2}{c||}{Photonic Sigmoid} & \multicolumn{2}{c}{Photonic Sinusoidal} \\ \cline{3-6}  & & Regular & Non-negative Match & Regular & Non-negative Match \\ \hline MNIST & MLP & \(97.44\pm 0.08\) & ✓ & \(97.90\pm 0.10\) & ✓ \\ MNIST & CNN & \(98.81\pm 0.14\) & ✓ & \(99.04\pm 0.07\) & ✓ \\ FMNIST & MLP & \(85.64\pm 0.22\) & ✓ & \(86.30\pm 0.52\) & ✓ \\ FMNIST & CNN & \(87.59\pm 0.47\) & ✓ & \(89.22\pm 0.18\) & ✓ \\ CIFAR10 & MLP & \(37.08\pm 0.37\) & ✓ & \(39.48\pm 1.69\) & ✓ \\ CIFAR10 & CNN & \(84.03\pm 0.69\) & ✓ & \(84.54\pm 0.28\) & ✓ \\ Mailing\({}^{*}\) & CNN & \(91.06\pm 4.56\) & ✓ & \(93.28\pm 4.30\) & ✓ \\ Names & RNN & \(56.02\pm 0.70\) & ✓ & \(65.82\pm 0.90\) & ✓ \\ FI2010\({}^{\dagger}\) & RNN & \(0.1371\pm 0.0024\) & ✓ & \(0.1612\pm 0.0033\) & ✓ \\ \end{tabular}

*F1 score, \(\dagger\)Cohen’s kappa score are reported, since the datasets a highly unbalanced

\end{table}
Table 1: Applying proposed transformation in various datasets, architectures and photonic configurations. The Non-negative Match column defines if the performance of the non-negative isomorphic matches trained model’s one.
training (column 2) and after applying the proposed transformation (column 3). As shown, directly applying clipping to trained parameters is catastrophic since it leads to a huge loss of information, while the network is unable to recover from it. When applying the proposed transformation, the transformed parameters preserve the information obtained from the training while the models are further optimized with CSGD. Applying the proposed NNSGD method, the evaluation performance is slightly improved compared to the CSGD. Compared to the regular training using the same epochs of training, the performance obtained is highly competitive, with some cases even exceeding them (such as in photonic sinusoidal configuration on FMINST and CIFAR10 datasets). Finally, we provide an experimental analysis on parameter variance in the Appendix, which shed light on the destructive effects of not applying the proposed transformation as well as the benefits of applying NNSGD.

We extend the evaluation of the proposed method in both CNNs and RNNs with the results reported in Table 3. Both optimization methods achieve competitive results with the regular training, while the proposed non-negative optimizer slightly improves the performance of the non-negative isomorphic models compared to CSGD. Interestingly, in the Maling dataset, the proposed non-negative optimizer improves the performance of the model even compared to the regular training (\(\sim 5\%\) in case of photonic sigmoid).

\begin{table}
\begin{tabular}{l|l l|l l}  & **Exp. Initialization** & **Proposed** & **Exp. Initialization** & **Proposed** \\
**Dataset** & **+ CSGD** & **+ CSGD** & **+ NNSGD** & **+ NNSGD** \\ \hline \hline \multicolumn{5}{c}{Photonic Sigmoid} \\ \hline MNIST & \(11.35\pm 0.00\) & \(\mathbf{92.52\pm 0.52}\) & \(11.35\pm 0.00\) & \(\mathbf{93.34\pm 1.11}\) \\ FMNIST & \(10.00\pm 0.00\) & \(\mathbf{82.31\pm 1.29}\) & \(10.00\pm 0.00\) & \(\mathbf{84.09\pm 0.56}\) \\ CIFAR10 & \(10.00\pm 0.00\) & \(\mathbf{37.25\pm 1.57}\) & \(10.00\pm 0.00\) & \(\mathbf{37.61\pm 1.76}\) \\ \hline \multicolumn{5}{c}{Photonic Sinusoidal} \\ \hline MNIST & \(86.36\pm 1.39\) & \(\mathbf{92.51\pm 0.78}\) & \(35.31\pm 6.11\) & \(\mathbf{94.14\pm 0.90}\) \\ FMNIST & \(68.95\pm 3.71\) & \(\mathbf{82.91\pm 0.55}\) & \(9.97\pm 0.02\) & \(\mathbf{83.57\pm 0.26}\) \\ CIFAR10 & \(10.00\pm 0.00\) & \(\mathbf{34.96\pm 2.44}\) & \(10.00\pm 0.00\) & \(\mathbf{35.41\pm 1.43}\) \\ \end{tabular}
\end{table}
Table 4: Non-negative training from scratch in MLPs

\begin{table}
\begin{tabular}{l|l l|l l}  & **Regular** & **Proposed** & **Proposed** \\
**Dataset** & **+ CSGD** & **+ CSGD** & **+ NNSGD** \\ \hline \hline \multicolumn{5}{c}{Photonic Sigmoid} \\ \hline MNIST & \(11.00\pm 0.00\) & \(\mathbf{97.31\pm 0.90}\) & \(\mathbf{97.40\pm 0.06}\) \\ FMNIST & \(10.00\pm 0.00\) & \(\mathbf{84.77\pm 0.21}\) & \(\mathbf{85.01\pm 0.12}\) \\ CIFAR10 & \(10.00\pm 0.00\) & \(\mathbf{36.34\pm 0.37}\) & \(\mathbf{36.81\pm 0.33}\) \\ \hline \multicolumn{5}{c}{Photonic Sinusoidal} \\ \hline MNIST & \(09.74\pm 0.00\) & \(\mathbf{97.85\pm 0.10}\) & \(\mathbf{97.86\pm 0.07}\) \\ FMNIST & \(10.00\pm 0.00\) & \(\mathbf{86.62\pm 0.09}\) & \(\mathbf{86.95\pm 0.17}\) \\ CIFAR10 & \(10.00\pm 0.00\) & \(\mathbf{40.96\pm 0.30}\) & \(\mathbf{41.19\pm 0.38}\) \\ \end{tabular}
\end{table}
Table 2: Non-negative post training after transformation using MLPs

\begin{table}
\begin{tabular}{l|l l l|l l l}  & **Exp. Initialization** & **Proposed** & **Exp. Initialization** & **Proposed** \\
**Dataset** & **+ CSGD** & **+ CSGD** & **+ NNSGD** & **+ NNSGD** \\ \hline \hline \multicolumn{5}{c}{Photonic Sigmoid} \\ \hline MNIST & \(11.35\pm 0.00\) & \(\mathbf{92.52\pm 0.52}\) & \(11.35\pm 0.00\) & \(\mathbf{93.34\pm 1.11}\) \\ FMNIST & \(10.00\pm 0.00\) & \(\mathbf{82.31\pm 1.29}\) & \(10.00\pm 0.00\) & \(\mathbf{84.09\pm 0.56}\) \\ CIFAR10 & \(10.00\pm 0.00\) & \(\mathbf{37.25\pm 1.57}\) & \(10.00\pm 0.00\) & \(\mathbf{37.61\pm 1.76}\) \\ \hline \multicolumn{5}{c}{Photonic Sinusoidal} \\ \hline MNIST & \(86.36\pm 1.39\) & \(\mathbf{92.51\pm 0.78}\) & \(35.31\pm 6.11\) & \(\mathbf{94.14\pm 0.90}\) \\ FMNIST & \(68.95\pm 3.71\) & \(\mathbf{82.91\pm 0.55}\) & \(9.97\pm 0.02\) & \(\mathbf{83.57\pm 0.26}\) \\ CIFAR10 & \(10.00\pm 0.00\) & \(\mathbf{34.96\pm 2.44}\) & \(10.00\pm 0.00\) & \(\mathbf{35.41\pm 1.43}\) \\ \end{tabular}
\end{table}
Table 4: Non-negative training from scratch in MLPs
### Non-negative training from scratch

In the final set of experiments, we evaluate the proposed framework in fully non-negative training. The proposed transformation is applied after randomly initializing the weights using the default PyTorch initialization [16]. In Table 4, we use as baseline the non-negative exponential initialization, used in [3]. Thus, four different non-negative approaches are evaluated: a) applying non-negative exponential initialization combined with CSGD optimizer, b) applying the proposed transformation along with the CSGD optimizer, c) applying non-negative exponential initialization combined with proposed NNSGD optimizer, and d) applying the proposed non-negative framework. The non-negative exponential initialization (columns 2 and 4) leads to unstable performance in contrast to the proposed NNT (columns 3 and 5) which leads to more robust non-negative training. Competing the proposed non-negative optimization method with the CSGD, we conclude that the proposed non-negative optimization leads to slightly better performance.

Due to the poor performance when non-negative exponential initialization is applied, in Table 5 we report the evaluation performance only in cases where the proposed transformation is applied. The results confirm that the proposed NNSGD can generalize to CNNs and RNNs architectures and improve the average evaluation accuracy compared to CSGD when a non-negative training from scratch is required. Similar to non-negative post-training the performance obtained is highly competitive to regular training with some cases even exceeding them (such as in Malimg dataset and in RNNs).

## 7 Conclusions

We introduce a transformation method that acquires the non-negative isomorphic of a regular trained or untrained neural network. Along with the proposed non-negative optimization method, one can either train from scratch a non-negative neural network or continue training when needed. The experimental results confirm that the acquired non-negative isomorphic results in exactly the same performance as the regular one, while when combined with the proposed non-negative optimization leads to competitive accuracies in contrast to regular training.

## 8 Limitations and future works

The proposed transformation cannot be directly applied in RNNs with gated mechanisms, such as GRUs and LSTMs, and further adjustments are needed. Also, the shifted activation function introduces a subtraction that can be integrated into the hardware implementation since it shifts the transfer function on the positive side of \(x\)-axis. As a result, the proposed non-negative optimization method ensures non-negativity to parameters only. In order to acquire also non-negative intermediate values the proposed transformation should be applied anew after non-negative post-training, which can slightly increase the computational complexity. Furthermore, it should be noted that non-negative ANNs leverage advantages on interpretability since they allow us to accomplish part-based learning due to the elimination of canceling neurons resulting in additive data representation [3] which is conceptually tied to human cognition [20]. As has been shown, non-negativity constraints significantly improve the human interpretation of ANNs through the visual representation [3; 8] of models, giving us further insights that can be used to further improve the performance of DL models [12]. The wide range of applications that the non-negative ANNs can be applied highlighting the significance of such a research direction.

\begin{table}
\begin{tabular}{l|c c c|c c}
**Method** & MNIST & FMNIST & CIFAR10 & Malimg\({}^{*}\) & Names & F2010\({}^{1}\) \\ \hline \multicolumn{6}{c}{Photonic Sigmoid} \\ \hline Proposed (CSGD) & \(97.69\pm 0.20\) & \(83.81\pm 0.85\) & \(71.61\pm 1.04\) & \(91.47\pm 4.37\) & \(54.00\pm 2.13\) & \(0.1338\pm 0.0045\) \\ Proposed (NNSGD) & \(\mathbf{98.14\pm 0.23}\) & \(\mathbf{83.87\pm 0.19}\) & \(\mathbf{72.16\pm 0.54}\) & \(\mathbf{91.98\pm 4.00}\) & \(\mathbf{56.68\pm 1.11}\) & \(\mathbf{0.1498\pm 0.0112}\) \\ \hline \multicolumn{6}{c}{Photonic Sinusoidal} \\ \hline Proposed (CSGD) & \(98.54\pm 0.14\) & \(87.77\pm 0.36\) & \(69.87\pm 1.73\) & \(96.53\pm 0.42\) & \(64.17\pm 0.14\) & \(0.1732\pm 0.0175\) \\ Proposed (NNSGD) & \(\mathbf{98.71\pm 0.05}\) & \(\mathbf{87.86\pm 0.36}\) & \(\mathbf{75.16\pm 1.83}\) & \(\mathbf{97.26\pm 0.31}\) & \(\mathbf{68.70\pm 0.78}\) & \(\mathbf{0.1789\pm 0.0102}\) \\ \multicolumn{6}{l}{*F1 score, \(\uparrow\)Cohen’s kappa score are reported, since the datasets a highly unbalanced} \\ \end{tabular}
\end{table}
Table 5: Non-negative training from scratch in CNNs and RNNs