Linearization of ReLU Activation Function for Neural Network-Embedded
  Optimization:Optimal Day-Ahead Energy Scheduling
Neural networks have been widely applied in the power system area. They can
be used for better predicting input information and modeling system performance
with increased accuracy. In some applications such as battery degradation
neural network-based microgrid day-ahead energy scheduling, the input features
of the trained learning model are variables to be solved in optimization models
that enforce limits on the output of the same learning model. This will create
a neural network-embedded optimization problem; the use of nonlinear activation
functions in the neural network will make such problems extremely hard to solve
if not unsolvable. To address this emerging challenge, this paper investigated
different methods for linearizing the nonlinear activation functions with a
particular focus on the widely used rectified linear unit (ReLU) function. Four
linearization methods tailored for the ReLU activation function are developed,
analyzed and compared in this paper. Each method employs a set of linear
constraints to replace the ReLU function, effectively linearizing the
optimization problem, which can overcome the computational challenges
associated with the nonlinearity of the neural network model. These proposed
linearization methods provide valuable tools for effectively solving
optimization problems that integrate neural network models with ReLU activation
functions.
Cunzhi Zhao, Xingpeng Li
Linearization of ReLU Activation Function for Neural Network-Embedded Optimization: Optimal Day-Ahead Energy Scheduling

###### Abstract

**Neural networks have been widely applied in the power system area. They can be used for better predicting input information and modeling system performance with increased accuracy. In some applications such as battery degradation neural network-based microgrid day-ahead energy scheduling, the input features of the trained learning model are variables to be solved in optimization models that enforce limits on the output of the same learning model. This will create a neural network-embedded optimization problem; the use of nonlinear activation functions in the neural network will make such problems extremely hard to solve if not unsolvable. To address this emerging challenge, this paper investigated different methods for linearizing the nonlinear activation functions with a particular focus on the widely used rectified linear unit (ReLU) function. Four linearization methods tailored for the ReLU activation function are developed, analyzed and compared in this paper. Each method employs a set of linear constraints to replace the ReLU function, effectively linearizing the optimization problem, which can overcome the computational challenges associated with the nonlinearity of the neural network model. These proposed linearization methods provide valuable tools for effectively solving optimization problems that integrate neural network models with ReLU activation functions.**

 Day-ahead scheduling, Linearization, Neural network, Optimization, Rectified linear unit.

## I Introduction

With the trend of decarbonization, a large number of renewable energy sources (RES)-based power plants are being constructed in the power system and the RES portfolio keeps increasing [1]. However, the intermittent and stochastic characteristics of the RES has raised the concern of system reliability and stability for high RES penetrated grid [2]. Deep learning (DL) is playing a crucial role in enhancing the operational efficiency and reliability of power systems, particularly in the context of high RES penetration and the associated challenges of intermittency and unpredictability [3]. DL has been widely adapted to assist the power system operation such as, optimal power flow, day ahead scheduling, restoration, and state estimation [4]-[6]. DL is an important technology for contributing to the transition towards more sustainable and resilient power grids.

The outstanding performance of deep learning technologies has ushered in innovative solutions for numerous challenging power system issues that traditional methods struggle to address. Today's power system is confronting formidable challenges, primarily due to the extensive integration of RES into the grid. Deep learning methods have emerged as indispensable tools within the power system area. Based on different attributes of DL models, they have been found to be useful in diverse applications in solving a range of power system problems. For example, a deep neural network (DNN) consisting of only fully connected dense layers is developed in [7] to predict active power flows and it outperforms the widely-used linearized DC power flow model. The utilization of graph neural networks (GNN) in [8] allows for efficient predictions of current and power injection, capitalizing on GNN's topological advantages. Convolutional neural network is adapted to predict the rate of change of frequency under large disturbances to ensure the system stability in [9]. Furthermore, fault detection including fault type and location can be predicted by the artificial neural network as presented in [10]. Meanwhile, recurrent neural networks are widely used in time sequential prediction such as load forecasting, electricity price prediction, weather forecasting, and renewable generation forecasting, offering versatile solutions to contemporary power system challenges [11]-[14].

Most DL models adopt the rectified linear unit (ReLU) as the nonlinear activation function between the hidden layers to enhance the training efficiency and robustness [15]. Nevertheless, the ReLU introduces the nonlinearity to the DL models which make them nonlinear [16]. While this nonlinearity poses no issue for deterministic problems like fault detection and power flow prediction, it can become a significant obstacle when integrating the trained DNN model into optimization problems where DNN's input features are
decision variables to be solved, rendering them suddenly unsolvable due to the introduced nonlinearity. Remarkably, none of the previously mentioned studies have addressed the crucial challenge of linearizing the ReLU function in DNNs. Thus, there exists a significant research gap concerning the development of methods to linearize ReLU-based DNNs.

To address the challenge, we have proposed four different models to linearize the ReLU activation function. The proposed four models include: (i) Big-M based piecewise linearization (BPWL), (ii) convex triangle area relaxation (CTAR), (iii) penalized CTAR (P-CTAR), and (iv) penalized convex area relaxation (PCAR). BPWL is able to fully linearize the ReLU activation function without any approximation while the rest three models make some approximations to decrease the computational complexity.

Our previous work [17]-[18] has introduced a novel neural network based battery degradation (NNBD) model aimed at accurately quantifying the battery degradation values per usage profile. The proposed NNBD model is able to predict the battery degradation value for each cycle based on the input of state of charge (SOC), state of health (SOH), depth of discharge (DOD), ambient temperature, and charge/discharge rate (C Rate) [19]. The NNBD model enables the incorporation of battery degradation into microgrid daily operational energy scheduling. However, this integration encountered unexpected computational burden due to the nonlinear nature of the NNBD model that utilizes the ReLU activation function in the hidden layers; this poses challenges when NNBD is incorporated into the optimal day-ahead generation scheduling problems. To address this issue, we will evaluate the proposed four ReLU linearization models in the testbed of NNBD-integrated microgrid day-ahead scheduling (MDS) model. It is worth mentioning that the proposed ReLU linearization methods not only fit the proposed NNBD model and optimal energy scheduling applications, but also applicable to a broader spectrum of DL-embedded optimization models that contains neural networks with ReLU activation functions.

The main contributions of this paper are as follows:

* _Linearization Approaches_: we introduce four novel formulations for linearizing the ReLU activation function, enhancing its applicability in various contexts, particularly when the input features of DL models are variables to be solved by an optimization model while the outputs represent physical system performance subject to some requirements enforced in the same optimization model.
* _Linearized Energy Scheduling Frameworks_: building upon the linearized ReLU models, we develop four distinct day-ahead scheduling models, providing practical solutions for generation scheduling.
* _Performance Assessment_: Comprehensive evaluations of the linearization ReLU models demonstrate their effectiveness in addressing nonlinearity challenges.
* _Optimal Configuration Exploration_: we conduct sensitivity tests to identify optimal configurations for the proposed linearization models, ensuring their robust performance in diverse scenarios.

The rest of the paper is organized as follows. Section II describes the proposed ReLU linearization models. Section III presents the traditional day-ahead scheduling model. Section IV presents the neural network integrated day-ahead scheduling model. Section V presents case studies and Section VI concludes the paper.

## II Proposed Linearization Models

This section presents the formulations of four models designed to linearize the ReLU activation function within the neural network model. The fully connected neural network models are characterized by a series of equations that describe the calculation and activation processes of neurons. Each neuron's pre-activated value \(x_{h}^{i}\) is computed by (1), factoring in input features from the previous layer, the corresponding weight matrix \(W\), and biases matrix \(B\). Most neural network models employ ReLU as the activation function, as shown in (2). While this activation function is prevalent for introducing nonlinearity to capture intricate relationships among variables, the nonlinearity of the ReLU function can pose challenges when it is embedded in the optimization problem. To address this challenge, the ReLU activation function can be linearized by applying the proposed linearization models described in this section. Notably, the proposed linearization models can be applied to any optimizations models that needs to efficiently integrate nonlinear ReLU activation function.

\[x_{h}^{i}=\sum a_{h-1}^{i}*W+B \tag{1}\]

\[a_{h}^{i}=ReLU\big{(}x_{h}^{i}\big{)}=max(0,x_{h}^{i}) \tag{2}\]

BPWL method is adapted to reformulate the ReLU function represented by (1) into (3)-(6), where \(\delta_{h}^{i}\) is a binary variable represents the activation status of neuron \(i\) in layer \(h\), and \(BigM\) is a pre-specified numerical value that is larger than any possible value of \(|x|\).

As shown in Fig. 1, the BPWL model offers the distinct advantage of perfectly linearizing the ReLU activation function without any reformulation losses, which is illustrated by the fact that the BPWL line in Fig. 1 completely overlaps the ReLU function curve. However, this method requires one additional binary variable for each neuron that applies the ReLU function, which may significantly increase the computational complexity.

\[a_{h}^{i}\leq x_{h}^{i}+BigM*(1-\delta_{h}^{i}) \tag{3}\] \[a_{h}^{i}\geq x_{h}^{i} \tag{4}\]

Fig. 1: Illustration of the BPWL model for ReLU linearization.
\[a_{\text{h}}^{i}\leq BigM*\delta_{\text{h}}^{i} \tag{5}\] \[a_{\text{h}}^{i}\geq 0 \tag{6}\]

The proposed CTAR approximate the ReLU function at each neuron with (4) and (6)-(7), which constrains the feasible solution set. CTAR offers a pure linear representation of ReLU, introducing minimal complexity to optimization problems while acknowledging the presence of approximation errors. As shown in Fig. 2, the blue area denotes the feasible region, delimited by the lower bound (LB) and upper bound (UB), both of which are determinable from the neural network model. Specifically, LB should be less than the minimum neuron preactivated value, while UB should exceed the maximum neuron preactivated value. In most cases with normalized training data, each neuron's value falls within the range between -1 to 1. It's noteworthy that different choices for LB and UB can impact the performance of the CTAR linearization method.

\[a_{\text{h}}^{i}\leq\frac{UB}{UB-LB}x_{\text{h}}^{i}-\frac{UB\cdot LB}{UB-LB} \tag{7}\]

The proposed P-CTAR model is introduced to reduce the approximation error associated with CTAR model with (4) and (6)-(8). To achieve this, a mitigation strategy involves incorporating a penalty term \(c_{h}\) into the objective function of the optimization model. This penalty encourages the nonnegative variable \(a_{h}^{i}\) to be positioned closer to the lower two sides, corresponding to the actual ReLU activated values, within the triangular representation as shown in Fig. 3.

\[f^{c}=\sum a_{\text{h}}^{i}c_{h} \tag{8}\]

The proposed PCAR method is designed to add a penalty term \(c_{h}\) on nonnegative \(a_{h}^{i}\) without LB and UB, represented by (4), (6), and (8), which force the \(a_{\text{h}}^{i}\) to be set at the ReLU activated values as shown in Fig. 4. As compared to CTAR, the PCAR method offers enhanced accuracy, especially when equipped with sufficiently large penalty terms, and notable efficiency gains due to the absence of constraints (7). Thus, this method stands out for its ability to significantly reduce computational complexity when compared to other available linearization approaches.

## III Traditional Microgrid Day-Ahead Energy Scheduling Model

This section presents the traditional microgrid day-ahead scheduling model. This model consists of (9)-(26) as described below and it does not consider battery degradation.

The objective of this traditional MDS model is to minimize the total cost of the microgrid operations as illustrated in (9). The MDS model includes a power balance equation, as detailed in (10), encompassing controllable generators, RES, power exchange with the main grid, battery energy storage system (BESS) output, and the load. Constraint (11) enforces the power limits of the controllable units such as diesel generators, while (12) and (13) enforce the ramping up and down limits. Equations (14)-(16) are employed to establish the relationship between a controllable unit's start-up status and its on/off status. Equation (17) restricts the BESS to be either in charging mode or in discharging mode or stay idle. Constraints (18)-(19) limit the charging/discharging power of BESS. Equation (20) governs the power exchange status between the microgrid and the main grid, indicating whether it involves purchasing, selling, or remaining idle. Constraints (21)-(22) define the thermal limits of the tie-line. Equation (23) computes the energy stored in the BESS for each time interval. Constraint (24) ensures that the final energy of the BESS equals the initial energy value while (25) respects the BESS capacity limit. Constraint (26) guarantees the microgrid maintains sufficient backup power to handle outage events.

Objective function:

\[\begin{split} f^{MG}=\sum\sum\left(P_{\text{G}}^{t}c_{\text{G}} \text{\emph{i}}+U_{\text{G}}\text{\emph{i}}c_{\text{G}}^{\text{NL}}+\text{ \emph{V}}_{\text{G}}\text{\emph{i}}c_{\text{G}}^{\text{SU}}\right)\\ +P_{\text{Buy}}^{t}c_{\text{Buy}}^{B}-P_{\text{S}}^{t}\text{ \emph{i}}c_{\text{S}}^{\text{EL}},\forall i,t\end{split} \tag{9}\]

Constraints are as follows:

\[\begin{split}& P_{t}^{Buy}+\sum\limits_{g\in S_{\text{G}}}\text{ \emph{g}},t+\sum\limits_{w\in S_{\text{W}}T}\text{\emph{p}}_{w,t}+\sum\limits _{\text{p}w\in S_{\text{P}}P_{\text{p}y,t}}\\ +\sum\nolimits_{s\in S_{\text{S}}S_{\text{S}}^{\text{DL}}}=P_{ \text{S}}^{\text{S}\text{S}\text{S}}+\sum\limits_{l\in S_{\text{L}}}\text{ \emph{P}}_{l,t}+\sum\limits_{s\in S_{\text{S}}}\text{\emph{P}}_{l,t}^{\text{ Char}}\\ & P_{g}^{\text{Min}}\leq P_{g,t}\leq P_{g}^{\text{Max}},\forall g,t\end{split} \tag{11}\]

\[P_{g,t+1}-P_{g,t}\leq\Delta T\cdot P_{g}^{\text{ramp}},\forall g,t \tag{12}\]

\[P_{g,t}-P_{g,t+1}\leq\Delta T\cdot p_{g}^{\text{ramp}},\forall g,t \tag{13}\]

\[V_{g,t}\geq U_{g,t}-U_{g,t-1},\forall g,t \tag{14}\]

\[V_{g,t+1}\leq 1-U_{g,t},\forall g,t \tag{15}\]

\[V_{g,t}\leq U_{g,t},\forall g,t \tag{16}\]

\[U_{g,t}^{\text{Disc}}+U_{g,t}^{\text{Charg}}\leq 1,\forall g,t \tag{17}\]

Fig. 4: Illustration of the PCAR model for ReLU linearization.

Fig. 3: Illustration of the P-CTAR model for ReLU linearization.

Fig. 2: Illustration of the CTAR model for ReLU linearization.
\[U_{s,t}^{Char}\cdot P_{s}^{Min}\leq p_{s,t}^{Char}\leq U_{s,t}^{ Char}\cdot P_{s}^{Max},\forall s,t \tag{18}\] \[U_{s,t}^{Disc}\cdot P_{s}^{Min}\leq p_{s,t}^{Disc}\leq U_{s,t}^{ Disc}\cdot P_{s}^{Max},\forall s,t\] (19) \[U_{t}^{Buy}+U_{t}^{Self}\leq 1,\forall t\] (20) \[0\leq P_{t}^{Buy}\leq U_{t}^{Buy}\cdot P_{Grid}^{Max},\forall t\] (21) \[0\leq P_{t}^{Bell}\leq U_{t}^{Self}\cdot P_{Grid}^{Max},\forall t\] (22) \[E_{s,t}-E_{s,t-1}+\Delta T\big{(}P_{s,t-1}^{Disc}\eta_{s}^{Disc}- P_{s,t-1}^{char}\eta_{s}^{char}\big{)}\] (23) \[=0,\forall s,t\] (24) \[E_{s,t=24}=E_{s}^{Initial},\forall s\] (25) \[0\leq E_{s,t}\leq E_{s,t}^{max}\] (26) \[P_{Grid}^{Max}-P_{t}^{Buy}+P_{t}^{Sell}+\sum_{g\in S_{G}}\big{(} P_{g}^{Max}-P_{g,t}\big{)}\] \[\geq R_{percent}\left(\sum_{t\in S_{L}}P_{t,t}\right),\forall t\]

## IV Neural Network Integrated Optimal Day-Ahead Scheduling Model

### _Neural Network Based Battery Degradation Model_

We've constructed a fully connected neural network model as mentioned in Section I to predict battery degradation, with five key aging factors (ambient temperature, C Rate, SOC, DOD, and SOH) comprising a five-element input vector for the network. Each input vector corresponds to a single output value, representing the battery degradation as a percentage relative to the SOH for the corresponding cycle. The structure of the trained neural network is shown in Fig. 5[17] plotted by NN-SVG. It has an input layer with 5 neurons corresponding to the 5 input features, first hidden layer with 20 neurons, second hidden layer with 10 neurons and an output layer with 1 neuron indicating the percentage battery degradation. The activation function is ReLU for the hidden layers and "linear" for the output layer.

### _NNBD based Day-ahead Scheduling Model_

Based on the BESS operation profile, the SOC level is required as the input of the proposed NNBD model. DOD is calculated by taking the absolute difference in SOC levels between time intervals \(t\) and \(t-1\), as shown in (27). C rate is calculated by (28). The input vector can be formed as shown in (29) and then fed into the trained NNBD model to obtain the total battery degradation over the MDS time horizon in (30). The sum of the battery degradation is used to calculate the equivalent battery degradation cost by (31). The updated objective function (32) is required for two of the proposed models: BPWL and CTAR. The NNBD based day-ahead scheduling model will consists of (9)-(32). However, due to the nonlinearity of the NNBD model, especially due to the ReLU activation function, the integration of the proposed NNBD model into the day-ahead scheduling framework cannot be directly solved. Therefore, we employ the proposed linearization methods within the intricate optimization model to facilitate its efficient solution.

\[DOD_{t}=|SOC_{t}-SOC_{t-1}| \tag{27}\] \[C_{t}^{Rate}=DOD_{t}/\Delta T\] (28) \[\overline{x}_{t}=(T,C,SOC,DOD,SOH)\] (29) \[BD\ =\sum_{t\in S_{T}}f^{NN}(\overline{x}_{t})SOH\] (30) \[f^{BESS}=\frac{c_{BESS}^{Capital}-c_{BESS}^{SY}}{1-SOH_{EOL}}BD\] (31) \[f=f^{MG}+f^{BESS} \tag{32}\]

### _Linearization for NNBD-based MDS Model_

Since the nonlinear ReLU activation function make the optimization model extremely hard to solve if not unsolvable, we have reformulated the MDS models to make it linear and solvable with the proposed four ReLU linearization models respectively. The updated objective function (33) is required for two of the proposed models: P-CTAR and PCAR. The linearized NNBD-integrated day-ahead energy scheduling models are defined in Table I.

\[f=f^{MG}+f^{BESS}+f^{c} \tag{33}\]

## V Case Studies

To evaluate the effectiveness of the linearized day-ahead scheduling model, we utilized a representative grid-connected microgrid [19] as the test case in this work, featuring several distributed energy resources including renewables and controllable units. The microgrid setup encompasses various key components, notably a conventional diesel generator, wind turbines, residential houses with solar panels, and a lithium-ion BESS with a charging and discharging roundtrip efficiency of 90%. The parameters for these main components are provided in Table II. To simulate real-world scenarios accurately, the load data for the microgrid is obtained from the electricity consumption patterns of 1000 residential households from the Pecan Street Dataport [17]. Additionally, for a comprehensive representation of environmental conditions, the ambient temperature and available solar power data are sourced over a 24-hour period from the same Pecan Street Dataport source.

Fig. 5: Structure of the NNBD model [17].
Furthermore, the wholesale electricity price data used in the microgrid model is obtained from the Electric Reliability Council of Texas (ERCOT) that manages majority of the Texas grid [18].

The MDS optimization problem in this paper was solved on a computer equipped with the following hardware: an AMD Ryzen 7 3800X processor, 32 GB of RAM, and an Nvidia Quadro RTX 2700 Super GPU with 8 GB of memory. The Pyomo [19] package, a powerful power system optimization modeling framework, was utilized to formulate and solve the day-ahead optimization problem. To expedite our quest for optimal solutions, we employed the high-performance mathematical programming solver Gurobi [20].

Table III presents the performance results obtained from the microgrid testbed based on the four proposed linearization models. In Table II, the term, _Degradation_, represents the degradation value calculated by the proposed linearization methods, while the term, _Real Degradation_, represents the degradation value computed separately by the NNBD model with the BESS operation profile to gauge the accuracy of the linearization models. The term, _Error_, is derived from the comparison between _Degradation_ and _Real Degradation_. The term, _Total Cost_, represents the combined MDS operation cost and the equivalent degradation cost. It's important to note that this equivalent degradation cost is derived from _Degradation_ which may introduce the linearization approximation error. The term, _Real Total Cost_ represent the combined MDS operation cost and the real equivalent degradation cost based on the _Real Degradation_. Also, the _Total Cost_ in Table III has already excluded the penalty cost for P-CTAR and PCAR models. The outcomes indicate that BPWL yields the lowest linearization error, attributed to its complete linearization of the ReLU activation function. The minimal degradation prediction error is a result of rounding the weights in the NNBD model during the battery degradation value verification process. Ideally, there should be zero linearization error.

However, it's worth mentioning that the BPWL model does require a relatively longer solving time due to the exponential addition of new binary variables to the optimization problem. Although the current solving time of 28 seconds remains acceptable for a microgrid case, it will increase significantly for larger systems. Overall, the BPWL model accurately represents the ReLU function with no reformulation losses but introduces binary variables, complicating the optimization model and leading to longer solving times. In contrast, the computation times for the other linearization methods are significantly less. Notably, PCAR yields the shortest solving time, as expected, given its fewer constraints compared to CTAR and P-CTAR.

The three MDS models with non-exact ReLU linearization achieve different accuracies. The proposed P-CTAR model exhibits the second-lowest error among the four proposed methods, signifying its ability to accurately linearize the ReLU activation function. A comparative analysis between CTAR and P-CTAR reveals that CTAR introduces substantial linearization errors, suggesting an ineffective linearization approach in this particular case. The significant approximation error in CTAR results from the absence of a penalty term in the model. This may change if the MDS model changes. In contrast, the P-CTAR model incorporates a penalty term in addition to the CTAR model, leading to a considerable enhancement in performance relative to CTAR. The P-CTAR model outperforms the PCAR model which implies that the chosen lower bound and upper bound values for CTAR are appropriate and effective in this testbed. The PCAR model, however, exhibits a linearization error of 37%, notably higher than P-CTAR. It's important to note that both PCAR and P-CTAR models' linearization errors are influenced by the penalty constant in the objective function, which requires additional sensitivity test to determine the optimal setup. While BPWL outperforms the other three models in terms of linearization accuracy, it lags behind in solving efficiency, especially when considering larger systems where it could potentially lead to significantly extended solving times. In contrast, P-CTAR stands out as the best performance model in terms of linearization error among the rest of the models. The solving time is decreased significantly compared to the BPWL model while maintaining the linearization performance.

Fig. 6: Microgrid load profile.
Fig. 7 graphically illustrates the linearization performance of the P-CTAR model when compared to the reference NNBD degradation value, which is based on the BESS operation profile per usage cycle. The linearized degradation, produced directly from the optimization model using the P-CTAR model to linearize the internal ReLU function of the NNBD model, exhibits a trend that closely aligns with the benchmark model. While there are some deviations in specific time periods, the overall degradation value over the 24-hour look-ahead scheduling horizon demonstrates minimal prediction error, affirming the efficacy of the P-CTAR linearization model.

It's noteworthy that during the design of the proposed linearization models, the expectation was that P-CTAR might introduce additional complexity to the optimization model, potentially leading to longer solving times. The results indeed align with this expectation, as the P-CTAR model's solving time surpasses that of the CTAR model. This trade-off between model efficiency and accuracy is duly considered in our evaluation.

Fig. 8 presents the results of a sensitivity test for the P-CTAR model using different penalty cost constants denoted as \(c_{h}\). This penalty cost serves the purpose of mitigating the approximation error associated with the CTAR model. However, it's important to note that different values of \(c_{h}\) may lead to different outcomes, as the penalty cost is directly integrated into the objective function of the optimization problem. The objective cost reflects the expenses incurred by the objective function (33), while the penalty cost accounts for the costs introduced by the P-CTAR model. The real cost represents the MDS operating cost and the battery degradation cost, calculated as the difference between the objective cost and the penalty cost. However, the battery degradation cost here may include an error due to the linearization approximation of the P-CTAR model. Upon examination of the figure, it becomes evident that there is no significant variation in the real cost as \(c_{h}\) increases. However, the lowest linearization error is observed when \(c_{h}\) is set to 10. Beyond this threshold, increasing \(c_{h}\) leads to an escalation in battery degradation error which is directly caused by the error of linearization model. In summary, the selection of \(c_{h}\) demands careful consideration and thorough preliminary testing to ensure the optimal performance of the P-CTAR model. It's essential to emphasize that the optimal \(c_{h}\) value provided here pertains specifically to the NNBD-based MDS optimization problem. If the optimization model undergoes any modifications, it would require recalibration to determine the ideal \(c_{h}\) value for the optimal setup.

We also conducted a sensitivity test on the PCAR model with varying \(c_{h}\) values, and the results are depicted in Figure 9. It's noticeable that as we increase the \(c_{h}\) value, the linearization error does not exhibit a substantial decrease. In essence, there doesn't appear to be an optimal \(c_{h}\) value for the PCAR model that significantly enhances linearization accuracy. While the PCAR model demonstrates effectiveness, it doesn't achieve the same level of performance as the P-CTAR model. Consequently, for the NNBD-integrated MDS optimization problem, the PCAR model may not represent the most ideal solution. However, it's worth noting that the PCAR model could potentially serve as the optimal solution for other neural network-based optimization models with distinct characteristics and requirements.

## VI Conclusion

This paper investigated four innovative linearization models designed to tackle the challenges caused by the ReLU activation function in neural networks when integrated into

Fig. 8: P-CTAR sensitivity tests.

Fig. 7: P-CTAR model degradation comparison.

Fig. 9: PCAR sensitivity tests.
optimization models where a subset of decision variables serve as the input features of the learning model. The inherent non-linearity of the ReLU function often makes such models complex and difficult to solve directly. However, by harnessing the proposed linearization models BPWL, CTAR, P-CTAR, and PCAR, it becomes feasible to effectively address the intricacies of neural network-integrated optimization problems. Our findings demonstrate that the BPWL model achieves the best performance in terms of accuracy by fully reformulating the ReLU activation function with auxiliary binary variables and it can achieve correct problem solutions. Furthermore, the proposed P-CTAR model can maintain impressive linearization accuracy without significant compromise, while significantly reducing the computing time.

Crucially, our results highlight that the choice of penalty terms is pivotal in obtaining optimal solutions for P-CTAR and PCAR models. While CTAR and PCAR may not deliver perfect performance in the NNBD-integrated microgrid testbed, they remain viable options for modeling and solving this class of problems. It's important to emphasize that when changes are made to the optimization model, the performance of the proposed linearization models should be re-evaluated. In summary, the linearization models introduced in this paper open up fresh avenues for efficiently addressing the challenges posed by non-linear neural network-integrated optimization problems. In future work, we intend to delve deeper into how these linearization models influence optimization models and further develop linearization models tailored to other activation functions such as the softmax function.
