Active Linearized Sparse Neural Network-based Frequency-Constrained Unit
  Commitment
Conventional synchronous generators are gradually being re-placed by
low-inertia inverter-based resources. Such transition introduces more
complicated operation conditions, frequency deviation stability and
rate-of-change-of-frequency (RoCoF) security are becoming great challenges.
This paper presents an active linearized sparse neural network (ALSNN) based
frequency-constrained unit commitment (ALSNN-FCUC) model to guarantee frequency
stability following the worst generator outage case while ensuring operational
efficiency. A generic data-driven predictor is first trained to predict maximal
frequency deviation and the highest locational RoCoF simultaneously based on a
high-fidelity simulation dataset, and then incorporated into ALSNN-FCUC model.
Sparse computation is introduced to avoid dense matrix multiplications. An
active data sampling method is proposed to maintain the bindingness of the
frequency related constraints. Besides, an active ReLU linearization method is
implemented to further improve the algorithm efficiency while retaining
solution quality. The effectiveness of proposed ALSNN-FCUC model is
demonstrated on the IEEE 24-bus system by conducting time domain simulations
using PSS/E.
Mingjian Tuo, Xingpeng Li
# Active Linearized Sparse Neural Network-based Frequency-Constrained Unit Commitment

###### Abstract

Conventional synchronous generators are gradually being replaced by low-inertia inverter-based resources. Such transition introduces more complicated operation conditions, frequency deviation stability and rate-of-change-of-frequency (RoCoF) security are becoming great challenges. This paper presents an active linearized sparse neural network (ALSNN) based frequency-constrained unit commitment (ALSNN-FCUC) model to guarantee frequency stability following the worst generator outage case while ensuring operational efficiency. A generic data-driven predictor is first trained to predict maximal frequency deviation and the highest locational RoCoF simultaneously based on a high-fidelity simulation dataset, and then incorporated into ALSNN-FCUC model. Sparse computation is introduced to avoid dense matrix multiplications. An active data sampling method is proposed to maintain the blindness of the frequency related constraints. Besides, an active ReLU linearization method is implemented to further improve the algorithm efficiency while retaining solution quality. The effectiveness of proposed ALSNN-FCUC model is demonstrated on the IEEE 24-bus system by conducting time domain simulations using PSS/E.

 Deep learning, Frequency deviation, Frequency stability, Low-inertia power systems, Sparse neural network, ReLU linearization, Rate of change of frequency, Unit commitment.

## Nomenclature

\begin{tabular}{l l} _Sets_ & \\ \(G\) & Set of generators. \\ \(K\) & Set of lines. \\ \(K^{*}(n)\) & Set of lines with bus \(n\) as receiving bus. \\ \(K^{-}(n)\) & Set of lines with bus \(n\) as sending bus. \\ \(T\) & Set of time periods. \\ \(N\) & Set of buses. \\ _NS_ & Set of samples. \\ _NL_ & Set of neural network layers. \\ \(\mathcal{R}\) & Set of active selected neurons. \\ \(\mathcal{R}\) & Set of not selected neurons. \\ \end{tabular} \begin{tabular}{l l} _Indices_ & \\ \(g\) & Generator \(g\). \\ \(k\) & Line \(k\). \\ \(t\) & Time \(t\). \\ \(n\) & Bus \(n\). \\ \(q\) & Neural network layer \(q\). \\ \(l\) & _l_-th neuron of a neural network layer. \\ \(s\) & Sample \(s\). \\ \end{tabular} 
\begin{tabular}{l l} _Parameters_ & \\ _c\({}_{g}\)_ & Linear operation cost for generator \(g\). \\ _r\({}_{g}^{min}\)_ & Minimum output limit of generator \(g\). \\ \end{tabular}

## I Introduction

The decarbonization of the electricity generation relies on the integration of converter-based resources. With the
increased penetration of renewable energy sources (RES), maintaining power system frequency stability has become a great challenge for reliable system operations [1]. Traditional-ly, synchronous generators play an important role in regulating frequency excursion and rate of change of frequency (RoCoF) after a disturbance as it ensures slower frequency dynamics [2]. Due to the retirement and replacement of conventional generation, more generation is coming from converter-based resources such as wind and solar power. Consequently, the system kinetic energy decreases significantly, leaving the system more vulnerable to large variations in load or generation [3].

Transmission system operators (TSOs) are concerned with the system stability issues with high penetration level of converter-based resources. Some have also suggested to impose extra RoCoF related constraints in the conventional unit commitment (UC) model to keep the minimum amount of synchronous inertia online [4]. SirGrid has introduced a synchronous inertial response constraint to ensure that the available inertia is above a minimum limit of 23 GWs in Ireland [5]. The Swedish TSO once ordered one of its nuclear power plants to reduce output by 100 MW to mitigate the risk of loss of that power plant [6].

Several papers have included frequency related constraints into traditional security-constrained unit commitment (SCUC) formulations. Reference [7] implements a system frequency stability constrained multiperiod SCUC model. In [8]-[9], uniform frequency response model was extended by including converter-based control, and constraints on RoCoF are then derived and incorporated into SCUC formulations. The work in [10] studied a mixed analytical-numerical approach based on multi-regions and investigated a model combining evolution of the center of inertia and certain inter-area oscillations. However, these approaches oversimplify the problem as they neglect nodal frequency dynamics, and the actual need for frequency ancillary services would be underestimated. Ref. [11] considers the geographical discrepancies and connectivity impacts on nodal frequency dynamics. However, results show that model-based approaches may fail to handle higher order characteristics and nonlinearities in system frequency response, approximation in model may also introduce extra errors into the derived constraints, resulting in conservative solutions. Recently, a pioneering data-driven approach has been proposed in [12], which incorporates neural network-based frequency nadir constraints against the worst-case contingency into frequency constrained unit commitment (FCUC). Ref. [13] proposes a deep neural network (DNN)-FCUC (DNN-FCUC) which incorporates DNN-based frequency predictor into formulations by introducing a set of mixed-integer linear constraints. However, computational efficiency of data-driven based approaches has not been investigated thoroughly yet. Such data-driven approaches may increase computational burden due to dense matrix multiplications [14]. On the other hand, reformulation of DNN would also introduce extra group of binary variables and further degrade the computational efficiency [13].

To bridge aforementioned gaps, we propose an active linearized sparse neural network (ALSNN) based frequency-constrained unit commitment (ALSNN-FCUC) model in this paper to secure system frequency stability following the worst contingency event. DNN based frequency metrics predictor is trained using system operation data, which can reflect geographical discrepancies and locational frequency dynamics due to non-uniform distribution of inertia. To incorporate the non-linear predictor into unit commitment model, a set of mixed-integer linear constraints are introduced. Besides, sparse computations are introduced to prune redundant parameters in the predictors to improve the efficiency of the proposed approach. In addition, an active rectified linear unit (ReLU) linearization algorithm is implemented to further improve the FCUC framework efficiency. The major contributions of this work are concluded as follows:

* First, we improve the data driven FCUC model by including the constraints on both locational RoCoF and frequency nadir security. In contrast to the existing data-driven approaches, where only frequency nadir security is considered, we propose a DNN-based frequency metrics predictor with the concept of sharing parameters. The predictor can simultaneously track the locational RoCoF and maximal frequency deviation for post-contingency conditions where oscillations in frequency cannot be ignored.
* Secondly, we analyze the dynamic model of the power system comprising practical number of generators at the same buses; heterogeneous responses of each node are then derived. Unlike [12] where random data injections may lead to post-contingency stability issues, model-based approaches that enforcing system locational frequency security are proposed to efficiently generate realistic data for predictor training. Results show that transient stability, small signal stability, and voltage stability of generated conditions are well handed.
* Thirdly, to the best of our knowledge, no prior work investigates the computational efficiency of DNN-FCUC. In this paper, we propose an ALSNN-FCUC model that incorporates sparse computations to perform parameter selection and increase neural network sparsity. This proposed sparse DNN subsequently reduces the computational burden of the framework. In addition, an active ReLU linearization method is performed over selected neurons to further improve the model efficiency.
* Last, we propose an active sampling method to improve the robustness of the trained predictor. This method allows us to increase the number of frequency constrained time intervals for an FCUC problem with acceptable solving time while maintaining the bindingness of these frequency related constraints.

The remainder of this paper is organized as follows. In section II, the model-based approaches and data-driven approaches are compared. Section III details the formulation of deep learning-based frequency constraints. Section IV describes the active ReLU linearization of sparse DNN. Section V describes the formulation of proposed ALSNN-FCUC model. The results and analysis are presented in Section VI. Section VII presents the concluding remarks and future work.
## II System Frequency Dynamics and Overview of Solution

### _Original Problem_

The frequency of the power system is one of the most important metrics that indicate the system stability. Traditionally, the frequency of the system is treated as a single-bus representation or center of inertia (COI) representation; the total power system inertia is considered as the summation of the kinetic energy stored in all dispatched generators synchronized with the power system [15].

\[E_{sys}=\sum_{t=1}^{N}2H_{i}S_{B_{i}} \tag{1}\]

where \(S_{B_{i}}\) is the generator rated power in MVA and \(H_{i}\) denotes the inertia constant of the generator.

Assuming a disturbance in the electrical power, the dynamics between power and frequency can be modeled by the swing equation described in (2) with \(M=2H\) denoting the normalized inertia constant and \(D\) denoting damping constant respectively [16].

\[\Delta P_{m}-\Delta P_{e}=M\frac{d\Delta\omega}{dt}+D\Delta\omega \tag{2}\]

where \(\Delta P_{m}\) is the total change in mechanical power and \(\Delta P_{e}\) is the total change in electric power of the power system. \(d\Delta\omega/dt\) is commonly known as RoCoF. During a short period of time following a disturbance, we can derive the initial RoCoF constraints used for system uniform model,

\[f_{rcf}=\frac{\Delta P_{m}-\Delta P_{e}}{2HS_{B}}\,\omega_{n}\leq-RoCoF_{tim} \tag{3}\]

However only considering system uniform metrics neglects the geographical discrepancies in locational frequency dynamics on each bus, which has imposed risks on power system stability [10]. Dynamic model is preferred in modern power system for frequency oscillation analysis. The topological information and system parameters can be embedded into the model by using swing equation on each individual bus to describe the oscillatory behavior within the system,

\[m_{i}\ddot{\theta}_{i}+d_{i}\dot{\theta}_{i}\,=\,P_{in,n}-\,\sum_{j=1}^{n}b_{ ij}\,sin\,(\theta_{i}\,-\,\theta_{j}) \tag{4}\]

where \(m_{i}\) and \(d_{i}\) denote the inertia coefficient and damping ratio for node \(i\) respectively, while \(P_{in,i}\) denotes the power input. A network-reduced model with \(N\) generator buses can be obtained by eliminating passive load buses via Kron reduction [17]. By focusing on the network connectivity's impact on the power system nodal dynamics, the phase angle \(\theta\) of generator buses can be expressed by the following dynamic equation [18],

\[\mathbf{M}\ddot{\mathbf{\theta}}\,+\,\mathbf{D}\,\dot{\mathbf{\theta}}\,=\,\mathbf{P}-\mathbf{L}\,\bm {\theta} \tag{5}\]

where \(\mathbf{M}=\text{diag}\left(\{m_{i}\}\right)\), \(\mathbf{D}=\text{diag}\left(\{d_{i}\}\right)\); for the Laplacian matrix \(\mathbf{L}\), its off-diagonal elements are \(l_{ij}=-b_{ij}V_{i}^{(0)}V_{j}^{(0)}\), and diagonals are \(l_{ij}=\sum_{j=1,j\neq i}^{n}b_{ij}V_{i}^{(0)}V_{j}^{(0)}\). The RoCoF value at bus \(i\) can then be derived,

\[f_{rcf,i}(t_{0})=\]

\[\frac{\Delta Pe^{-\frac{\gamma t}{2}}}{2\pi m}\sum_{a=1}^{N_{B}}\frac{\beta_{ ai}\beta_{ab}}{\frac{\beta_{ai}\,\gamma^{2}}{4}\pi t}\left[\begin{array}{c}e^{- \frac{\gamma t\pi}{2}}\,sin\left(\sqrt{\frac{\beta_{a}}{m}-\frac{\gamma^{2}}{ 4}}\,(t_{0}+\Delta t)\right)\end{array}\right]\]

where \(\lambda_{a}\) is the eigenvalue of matrix \(L\), \(\beta_{ai}\) is the eigen vector value; and \(m\) denotes average inertia distribution on generator buses; and bus \(b\) is where disturbance occurs; \(\Delta t\) is the frequency monitoring window, and \(t_{0}\) is the measuring time. \(N_{g}\) denotes the set of generator buses in the reduced model. The ratio of damping coefficient to inertia coefficient \(\gamma=d_{i}/m_{i}\) is assumed as a constant [3]. We can then derive the dynamic RoCoF related constraints as,

\[f_{rcf,i}(t_{0})\leq-\text{RoCoF}_{lim},\forall\,i\in N_{G} \tag{7}\]

### _Overview of Data Driven Approach_

Given a system with \(N\) generators, the objective goal of the ordinary UC model is to minimize the total operating cost subject to various system operational constraints.

\[\begin{split}\text{min.}&\mathcal{C}(\mathbf{s}_{t},\mathbf{u}_{t}) \\ s.\,t.&\mathcal{F}(\mathbf{s}_{t},\mathbf{u}_{t},\mathbf{d}_{t},\mathbf{r}_{t})=0, \mathcal{G}(\mathbf{s}_{t},\mathbf{u}_{t},\mathbf{d}_{t},\mathbf{r}_{t})\leq 0,\forall t\\ \end{split} \tag{8}\]

where \(\mathcal{F}\) and \(\mathcal{G}\) are the equality and inequality constraints respectively; \(\mathbf{s}_{t}\) denotes the system states, and \(\mathbf{u}_{t}\)is the generation dispatch at period \(t\). \(\mathbf{d}_{t}\) and \(\mathbf{r}_{t}\) denote the load profile and renewable forecast, respectively. Assume a potential disturbance \(\mathbf{\varpi}_{t}\) occurs in period \(t\). With system nominal frequency \(f_{n}=60\) Hz being the base frequency, the model based RoCoF constraints (6) and (7) could be derived and then added into the ordinary UC formulation to secure the system frequency stability.

Assumptions made during the system model analysis may introduce approximation error, subsequently lead to unsecure results or conservative results. The idea of data driven approach is to replace model-based constraints by DNN formulations,

\[\hat{\mathbf{h}}^{f}(\mathbf{s}_{t},\mathbf{u}_{t},\mathbf{r}_{t},\mathbf{\varpi}_{t})\leq\mathbf{\varepsilon} \tag{9}\]

where \(\hat{\mathbf{h}}^{f}\) is the nonlinear DNN-based frequency metrics predictor, including system wide maximal frequency deviation and maximal locational RoCoF; \(\mathbf{\varepsilon}\) is the vector of predefined threshold.

Fig. 1: Overview of the proposed approach.
The loss of generation not only results in the largest power outage level, but also leads to the reduction in system inertia, which further leads to the highest RoCoF value and frequency deviation; thus, G-1 event is considered as the worst contingency in this study. It should be noted that the output vector of the DNN-based predictor includes multiple elements which could be generalized for other constraints without including more parameters. The overview of the working pipeline is shown in Fig. 1, in which \(\Delta f_{max}\) is the maximal frequency deviation, and \(f_{max}\) is the highest RoCoF value.

### _Model based Data Generation_

Wide-range space of all power injections is utilized in [12] to ensure reliability under vast ranges of operating conditions for a small group of generators. However, considering a power system consists of a large group of generators, the dimension of \(\mathbf{x}_{t}\) would increase accordingly, which would further cause exponentially increase in the dimension of injections state space for practical system condition [19]. Such wide-range injections may lead to divergence during the simulation initialization process and the system would also be subject to post-contingency stability issues.

Unlike randomly data generation considering wide-range space of dispatching, a model-based systematic data generation approach is proposed to generate reasonable and representative data that will be used to train RoCoF Predictors with much less computational burden and compromise of efficiency. Traditional SCUC (T-SCUC) models and RoCoF constrained SCUC models are implemented in this process to generate training samples over various load and RES scenarios. Objective function of three models is to minimize the total system cost consisting of variable fuel costs, no-load costs, start-up costs, and reserve costs. RoCoF related constraints based on equivalent model and dynamic model are added into the formulation of system equivalent model based RoCoF constrained SCUC (ERC-SCUC) (ERC-RCUC) and location based RoCoF constrained SCUC (LRC-SCUC) respectively. The constraints on maximal frequency deviation RoCoF for locational frequency dynamics are nonlinear. In order to incorporate the nonlinear RoCoF-related constraints into the MILP model, a piecewise linear programming method is introduced, and the detail of all models is presented in [11].

## III Deep Learning-Based Frequency Constraints

### _Power System Feature Definition_

The highest locational frequency deviation and highest system locational RoCoF are considered as functions with respect to the contingency level, contingency location, system states, unit dispatch. Since both the magnitude and location of the contingency will have impact on the locational frequency deviation and locational inertial response, the generator status and dispatching values are encoded in to feature vectors [20]. For a case of period \(t\), the generator status vector is defined as follow,

\[\mathbf{u}_{t}=\begin{bmatrix}u_{1,t},u_{2,t},\cdots,u_{N_{G},t}\end{bmatrix} \tag{10}\]

The disturbance feature vector is defined against the loss of the largest generation, the magnitude of the contingency could be expressed as,

\[\mathbf{P}_{t}^{con}=\underset{g\in\mathcal{G}}{max}\left(P_{1,t},P_{2,t},\cdots,P _{N_{G},t}\right) \tag{11}\]

The location of the disturbance is represented by the index of the generator producing maximum power,

\[g_{t}^{con}=\underset{g\in\mathcal{G}}{arg}\underset{g\in\mathcal{G}}{max} \left(P_{1,t},P_{2,t},\cdots,P_{N_{G},t}\right) \tag{12}\]

The information of magnitude and location into the disturbance feature vector as [12],

\[\mathbf{\varpi}_{t}^{G}=[0,\cdots,0],\underset{g_{t}^{con}\in\mathcal{G}}{p_{t}^ {con}},0,\cdots,0] \tag{13}\]

Laplacian matrix \(\mathcal{L}\) of the grid and Fiedler mode value depend on the power-angle characteristics, which are determined by the active power injection [11]. Thus, the active power injection of all synchronous generator will be encoded into the feature vector.

\[\mathbf{P}_{t}=\begin{bmatrix}p_{1,t},\cdots,P_{2,t},\cdots,P_{N_{G},t}\end{bmatrix} \tag{14}\]

The overall feature vector of a case \(\mathbf{x}_{t}\) can be then defined as follows,

\[\mathbf{x}_{t}=[u_{1,t},\cdots,u_{N_{G},t},\ \epsilon_{1,t},\cdots,\ \epsilon_{N_{G},t},P_{1,t},\cdots,P_{N_{G},t}] \tag{15}\]

### _DNN-based Frequency Metrics Predictor_

Regarding frequency related constraints, both model-based approach and data-driven approach share the same variables which are determined by the generator status and disturbance information. Thus, A DNN-based frequency metrics predictor is proposed, which combines the functions of locational frequency deviation and highest locational RoCoF in frequency-related constraints together into a nonlinear DNN with multiple outputs. The combination of these functions could significantly decrease the number of DNN parameters thus reducing additional variables introduced into the FCUC formulation.

In the aftermath of a large \(G-1\) contingency, the frequency metrics predictor for \(h\) can be expressed as,

\[\begin{bmatrix}\hat{f}_{dev}\\ \hat{f}_{rcf}\end{bmatrix}=\widehat{h}(\mathbf{x}_{t},\mathbf{W}^{\prime},\mathbf{b}^{ \prime}) \tag{16}\]

where \(\mathbf{W}^{\prime}\) and \(\mathbf{b}^{\prime}\) denote the well-trained neural network parameters for frequency metrics predictor. Let \(\hat{f}_{dev}\) and \(\hat{f}_{rcf}\) denote the predicted frequency deviation and highest RoCoF values respectively for case \(\mathbf{x}_{t}\). Considering a DNN with \(N_{L}\)hidden layers, where ReLU is used as activation function for each hidden layer and the output layer is linear activation function. The forward propagation of each layer is then expressed as follows,

\[\mathbf{z}_{1}=\mathbf{x}_{t}W_{1}+b_{1} \tag{17}\] \[\hat{\mathbf{z}}_{q}=\mathbf{z}_{q-1}\mathbf{W}_{q}+\mathbf{b}_{q}\] (18) \[\mathbf{z}_{q}=\max\left(\hat{\mathbf{z}}_{q},0\right)\] (19) \[\hat{f}_{dev}=\mathbf{z}_{N_{L}}\mathbf{W}_{N_{L+1}^{c}}^{dev}+\mathbf{b}_{N _{L+1}^{dev}}^{dev}\] (20) \[\hat{f}_{rcf}=\mathbf{z}_{N_{L}}\mathbf{W}_{N_{L+1}^{c}}^{rcf}+\mathbf{b}_{N _{L+1}^{c}}^{rcf} \tag{21}\]

where \(W_{q}\) and vector \(b_{q}\) for \(q\in\mathbb{{}N}_{L}\) represent the set of weight and bias across all hidden layers; \(W_{N_{L}}^{dev}\) and \(b_{N_{L}}^{dev}\) represent the set of weight and bias of the output layer for maximal frequency deviation prediction, and \(W_{N_{L}}^{rcf}\) and \(b_{N_{L}}^{rcf}\) represent the output layer for system highest locational RoCoF prediction. The training process is to minimize the total mean squared error between the predicted output and the labeled outputs of all training samples as follows
\[\min_{\Phi}\frac{1}{N_{S}}\sum_{s=1}^{N_{S}}\left(\Delta f_{max}-\hat{f}_{dev} \right)^{2}+\left(\hat{f}_{max}\ -\hat{f}_{rcf}\right)^{2} \tag{22}\]

where \(\Phi=\{W_{q_{q}},b_{q},W_{N_{L}+1}^{dev},W_{N_{L}+1}^{ref},b_{N_{L}+1}^{dev},b_ {N_{L}+1}^{ref}\}\) represents the set of optimization variables.

### _Complete DNN Linearization_

Since ReLU activation functions are nonlinear, to include the DNN into the MILP, binary variables \(a_{q[l]}\) are introduced which represent the activation status of the ReLU unit at \(l\)th neuron of \(q\)th layer. For a given sample \(s\), consider \(A\) is a big number that is larger than the absolute value of all \(\hat{2}_{q[l],s}\). When preactivated value \(\hat{2}_{q[l],s}\) is larger than zero, constraints (23a) and (23b) will force binary variable \(a_{q[l],s}\) to one, and the activated value will be equal to \(\hat{2}_{q[l],s}\). When \(\hat{2}_{q[l],s}\) is less than or equal to zero, constraints (23c) and (23d) will force binary variable \(a_{q[l],s}\) to zero. Subsequently, the activated value \(a_{q[l],s}\) will be set zero.

\[z_{q[l],s} \leq\hat{2}_{q[l],s}+A(1-a_{q[l],s}),\forall q,\forall l,\forall s, \tag{23a}\] \[z_{q[l],s} \geq\hat{2}_{q[l],s},\forall q,\forall l,\forall s,\] (23b) \[z_{q[l],s} \leq Aa_{q[l],s},\forall q,\forall l,\forall s,\] (23c) \[z_{q[l],s} \geq 0,\forall q,\forall l,\forall s,\] (23d) \[a_{q[l],s} \in\{0,1\},\forall q,\forall l,\forall s, \tag{23e}\]

## IV Active Linearized Sparse Computation

Although the aforementioned reformulation of DNN introduces no approximation error, such linearization progress would introduce dense parameter matrix and additional binary variables into MILP model. Thus, the computational burden for the DNN-MILP would increase especially for the condition where multiple scheduling periods are limited by DNN constraints. To handle that problem, sparse computation and active ReLU linearization are proposed to obtain optimal efficiency.

### _Sparse Computation_

Conversion of DNN network into MILP has introduced group parameters. However, not all of these parameters are required to achieve high performance of the predictor. Instead of connecting every pair of neurons in each layer, we prune the connections and redundant parameters from networks to decrease the density of weights matrix without affecting performance of the frequency metrics predictor [21]. The example of training algorithm for sparse computation is shown in Fig. 2.

In this work, the network's connections are pruned during the training process by applying a mask [22]. First, we initialize the network and pre-trained the parameters of the frequency metrics predictor. For each layer chosen to be pruned, a binary mask variable is added which is of the same size and shape as the layer's weight matrix and determines which of the weights participate in the forward execution of the graph as shown in Fig. 3. The weights in each layer are sorted by their absolute values and mask the smallest magnitude weights to zero. The sparsity of the parameter matrix is increased from an initial sparsity value \(s_{0}\) to a final sparsity value \(s_{final}\) over a span of \(\mu\) pruning steps with pruning frequency \(\Delta e\):

\[s_{e}=s_{final}+\left(s_{0}-s_{final}\right)\left(1-\frac{e-e_{0}}{\mu\Delta e }\right)^{3} \tag{24}\]

for \(e\in\{e_{0},e_{0}+\Delta e,...,e_{0}+\mu\Delta e\}\)

Gradually increasing the sparsity of the network allows the network training steps to recover from any pruning-induced loss in accuracy. Similar binary masks are applied to the back-propagated gradients, and the weights that were masked in the forward execution are not updated in the back-propagation step, the overall strategy is introduced as follows,

```
0: Training Dataset \(\Theta=\{(x_{1},y_{1}),...,(x_{n},y_{n})\}\), \(\theta=\{W,b\}\), Mask generator \(Sp(\cdot)\), Final Sparsity \(s_{final}\), Initial epoch \(e_{0}\), Pruning frequency \(\Delta e\), Pruning times \(Q\), Total Training Epochs \(E\), Batch size \(B\)
0: optimal sparse \(W,b\)
1:\(W\gets W_{0}\)\(\triangleright\) Initialize W with Pretrained \(W_{0}\)
2:\(b\gets b_{0}\)\(\triangleright\) Initialize b with Pretrained \(b_{0}\)
3:for\(e=1,2,...,E\)do
4:if\(e\neq e_{0}+\mu\Delta e(\mu\in Q)\)then
5:\(h_{1},...,h_{B}\gets SNN(\Theta_{B},\theta)\)
6:\(\Delta_{\theta}\gets BP(\Theta_{B},h_{1},...,h_{B},\theta)\)
7:\(\theta\leftarrow LearningRule(\Delta_{\theta},\theta)\)
8:else
9:\(W\gets W\odot Sp(s_{0},s_{final},\mu,e)\)
10:\(h_{1},...,h_{B}\gets SNN(\Theta_{B},\theta)\)
11:\(\Delta_{\theta}\gets BP(\Theta_{B},h_{1},...,h_{B},\theta)\)
12:\(\theta\gets LearningRule(\Delta_{\theta},\theta,Sp(s_{0},s_{final},\mu,e))\)
13:endif
14:endfor
15:return\(\theta\)
```

**Algorithm 1** Sparse Neural Network Training

### _Active Sampling_

Even though the sparse computation and ReLU linearization have been proved to be able to improve the computational efficiency of the model, there are other important aspects that we should consider. However, with sparse computation introduced, the compressed network of predictor tuned for the data

Fig. 3: Example of sparse fully connected neural network.

Fig. 2: Example of sparse fully connected neural network.
near the critical threshold using region-of-interest active sampling method may fail to approximate the output of the original network. Subsequently, the related frequency constraints may not bind during the unit commitment process. Thus, it is reasonable to consider the tradeoff between the sparsity rate and the prediction error.

Model-based data generation is utilized to generate a group of samples which are labeled with maximal frequency deviation and the highest RoCoF following the worst contingency. Labeling the RoCoF value of adequate samples to achieve desired distributions requires significant computation resources and time, which is known as the labeling bottleneck. In other words, how to sample the cases to be labeled is essential. In order to generate and label desired samples, an active sampling (AS) method is proposed to actively select unlabeled samples.

A frequency security discriminator is first trained based on the small group of samples \(\mathcal{F}\) with highest RoCoF and frequency deviation labeled, the average value in 0.1s measuring interval is used. Security label \(f_{sec}\) is then applied to the training dataset: if the \(\Delta f_{max}\) and \(f_{max}\) of a sample are all within the limit, \(f_{sec}\) is labeled as 1. Otherwise, if one of the metrics violate the threshold, \(f_{sec}\) is labeled as 0. And then we perform the active sampling strategy on unlabeled dataset \(\mathcal{U}\) using the discriminator. The unlabeled samples whose posterior probability provided by the discriminator are then selected as training dataset for RoCoF and frequency deviation predictor. The general sampling strategy uses entropy as measure [23]:

\[x_{sec}=\operatorname*{argmax}_{u}\left(-\sum_{u}^{1}p(y_{i}|\mathcal{U})\log p (y_{i}|\mathcal{U})\right) \tag{25}\]

\[x_{insec}=\operatorname*{argmin}_{u}\left(-\sum_{u}^{1}p(y_{i}|\mathcal{U}) \log p(y_{i}|\mathcal{U})\right) \tag{26}\]

where \(y_{i}\) ranges over all possible labelings, and \(p(y_{i}|\mathcal{U})\) is the predicted posterior probability of class membership \(y_{i}\). This sorting process finds the largest and lowest from the finite set of numerical values. The selected \(x_{sec}\) is considered as sample distribute close the threshold which can improve the accuracy of predictor, while \(x_{insec}\) denotes samples which help the output of the tuned frequency metrics predictor approximate to the original. In the other word, they help keep the constraints binding after the sparse computation. The selected sample \(x\) is then added into \(\Theta\) and labeled with highest RoCoF and frequency deviation for frequency metrics predictor training.

### _Active ReLU Linearization_

Activation unit has been set on the neurons of hidden layer to realize the nonlinearization of a neural network; activation function, such as ReLU, is applied to the result of linear combination of values from neuron nodes [24]. The incorporation of ReLU function in a neural network into MILP without no approximation would introduce multiple extra binary variables. Subsequently, increasing the computational burden of the MILP model and leading to poor efficiency. Therefore, an active linearization of ReLU function has been introduced to reduce the DNN size without too much degradation of prediction accuracy [25].

Approximation of ReLU function is shown in Fig. 1. The weighted sum of input signals to the node is denoted as variable \(\hat{z}\), and the output of the node is denoted using the variable \(z\). Given the upper and lower bounds \([LB,UB]\) of \(\hat{z}\), the relationship of \(z\) and \(\hat{z}\) can then be approximated by a set of constraints \(z\geq 0\), \(z\geq\hat{z}\), and \(z\leq\frac{UB(2-LB)}{UB+LB}\). These constraints are all linear equations with constant \(UB\) and \(LB\).

Although ReLU linearization can be applied to all the neurons that would convert a DNN-FCUC model into ReLU linearized neural network based FCUC (RLNN-FCUC), this may lead to large approximation error and low prediction accuracy; and the associated frequency constraints may no longer be binding. In this work, we introduce an active selecting method to improve the computational efficiency of the DNN-FCUC model while maintaining the bindingness of derived constraints. The actively sampled dataset is first fed into the well-trained frequency metrics predictor, a nodal positivity index \(\varepsilon_{q[l]}\) is proposed to estimate the percentage of positive pre-activated values of neuron node \(l\) in \(q\)-th layer.

\[\varepsilon_{q[l]}=\frac{1}{N_{S}}\left(\sum_{\overline{N_{S}}}2_{q[l]s}-\sum _{\overline{N_{S}}}\left|2_{q[l]s}-\frac{1}{N_{S}}\sum_{\overline{N_{S}}}2_{q[ l]s}\right|\right)\geq\gamma \tag{27}\]

where \(\gamma\) is the threshold set to select nodes suitable for ReLU linearization with less approximation error. For the prediction of a given sample \(s\), equations (23a)-(23e) for ReLU function in selected neurons can be replaced by (28b)-(28d) as follows,

\[z_{q[l]s}\geq\hat{z}_{q[l]s},\forall q,\forall l,\forall s, \tag{28b}\]

\[z_{q[l]s}\leq\frac{UB_{q[l]}\cdot(\hat{z}_{q[l]s}-LB_{q[l]})}{UB_{q[l]}-LB_{q[ l]}},\forall q,\forall l,\forall s, \tag{28c}\]

\[z_{q[l]s}\geq 0,\forall q,\forall l,\forall s, \tag{28d}\]

## V Mixed-Integer Formulation of ALSNN-FCUC

In this section, the proposed ALSNN-FCUC considering frequency related constraints is formulated. The objective of the modified ALSNN-FCUC model is to minimize total operating cost subject to various system operational constraints and guarantee system frequency stability. The formulation is shown below:

\[\min_{\phi}\sum_{g\in\mathcal{G}}\sum_{t\in\mathcal{T}}(c_{g}p_{g,t}+c_{g}^{ NL}u_{g,t}+c_{g}^{SU}v_{g,t}+c_{g}^{RE}r_{g,t}) \tag{29a}\] \[\sum_{g\in\mathcal{G}}P_{g,t}+\sum_{k\in\mathcal{K}(n-)}p_{g,t}- \sum_{k\in\mathcal{K}(n+)}p_{g,t}-D_{n,t}\] (29b) \[+E_{n,t}\;=\;0,\;\;\;\forall n,t\] (29c) \[P_{k,t}-b_{k}(\theta_{n,t}-\theta_{m,t})\;=\;0,\;\;\forall k,t\]

Fig. 4: The linear approximation of ReLU activation function.
\[-p_{k}^{max}\leq P_{k,t}\leq P_{k}^{max},\ \forall k,t \tag{29d}\] \[p_{g}^{min}u_{g,t}\leq P_{g,t},\ \ \ \ \ \forall g,t\] (29e) \[P_{g,t}+r_{g,t}\leq u_{g,t}p_{g}^{max},\ \ \ \ \ \forall g,t\] (29f) \[0\leq r_{g,t}\leq R_{g}^{re}u_{g,t},\ \ \ \ \ \forall g,t\] (29g) \[\sum_{j\in G}r_{j,t}\geq P_{g,t}+r_{g,t},\ \ \ \ \ \ \forall g,t\] (29h) \[P_{g,t}-P_{g,t-1}\leq R_{g}^{hr},\ \ \ \ \ \ \forall g,t\] (29i) \[P_{g,t-1}-P_{g,t}\leq R_{g}^{hr},\ \ \ \ \ \forall g,t\] (29j) \[v_{g,t}\geq u_{g,t}-u_{g,t-1},\ \forall g,t\] (29k) \[v_{g,t+1}\leq 1-u_{g,t},\ \forall g,t\leq nT-1\] (29l) \[v_{g,t}\leq u_{g,t},\ \ \ \ \ \forall g,t\] (29m) \[\sum\nolimits^{t}_{s=t-UT_{g}}v_{g,s}\leq u_{g,t},\forall g,t \geq UT_{g}\] (29n) \[\sum\nolimits^{t+DT_{g}}v_{g,s}\leq 1-u_{g,t},\forall g,t\geq nT-DT_{g}\] (29o) \[u_{g,t},v_{g,t}\in\{0,1\},\ \ \ \ \ \forall g,t \tag{29p}\]

Equation (29a) is the objective function, and the basic constraints include (29b)-(29o). Equation (29b) enforces the nodal power balance. Network power flows are calculated in (29c) and are restricted by the transmission capacity as shown in (29d). The scheduled energy production and generation reserves are bounded by unit generation capacity and ramping rate (29e)-(29j). As defined in (29h), the reserve requirements ensure the reserve is sufficient to cover any loss of a single generator. The start-up status and on/off status of conventional units are defined as binary variables (29k)-(29m). Minimum-down time before a generator can be started-up and the minimum-up time before a generator can be shutdown are depicted in (29n) and (29o), respectively. Indicating variables for generator start-up and commitment status are binary and are represented by (29p).

Since \(\boldsymbol{x_{t}}\) contains the max operator, and disturbance vector cannot be directly used in the encoding formulation. Supplement variables \(\mu_{g,t}\) and \(\rho_{g,t}\) are used to indicate the largest generator output for period \(t\). The reformulations are expressed as follows,

\[p_{g,t}^{con}-P_{g,t}\leq A(1-\mu_{g,t}),\ \ \ \ \ \ \forall g,t \tag{30a}\] \[\sum_{g\in G}\mu_{g,t}=1,\ \ \ \ \ \ \forall t\] (30b) \[\mu_{g,t}\in\{0,1\},\ \ \ \ \ \ \ \forall g,t\] (30c) \[\rho_{g,t}-P_{g,t}\geq-A(1-\mu_{g,t}),\ \ \ \ \ \ \forall g,t\] (30d) \[\rho_{g,t}-P_{g,t}\leq A(1-\mu_{g,t}),\ \ \ \ \ \ \forall g,t\] (30e) \[0\leq\rho_{g,t}\leq A\mu_{g,t},\ \ \ \ \ \ \forall g,t \tag{30f}\]

where \(A\) is a big number. The input feature vector is then reformulated as follows.

\[x_{t}=[u_{1,t},\cdots,u_{N_{G},t},\ \rho_{g,t},\cdots,\ \rho_{N_{G},t},P_{1,t}, \cdots,P_{N_{G},s}] \tag{31}\]

Then we introduce the formulation of proposed frequency metrics predictor. As discussed before, the neuron of each layer with positivity index \(\epsilon_{g[t]}\) larger than \(\gamma\) is selected out and added into set \(\mathcal{H}\). The reformulations of ReLU activation on the selected neurons are expressed as follows.

\[z_{1}=\boldsymbol{x_{t}}W_{1}+b_{1},\forall t, \tag{32a}\] \[z_{q[1],t}\geq\hat{z}_{q[1],t},\forall q,\forall l\in\mathcal{ H},\forall t,\] (32b) \[z_{q[1],t}\leq\frac{UB_{q[t]}\cdot(\hat{z}_{q[1],t}-LB_{q[t]})}{ UB_{q[1]}-LB_{q[1]}},\forall q,\forall l\in\mathcal{H},\forall t, \tag{32c}\]

The ReLU function on the rest of neuron is reformulated with no approximation as follows. (33f) and (33g) constrain the maximal frequency deviation and maximal RoCoF.

\[z_{q[1],t}\leq\hat{z}_{q[1],t}-A(1-a_{q[1],t}),\forall q,l\in \mathcal{\bar{H}},t, \tag{33a}\] \[z_{q[1],t}\geq\hat{z}_{q[1],t},\forall q,\forall l\in\mathcal{ \bar{H}},\forall t,\] (33b) \[z_{q[1],t}\leq A\alpha_{q[1],t},\forall q,\forall l\in\mathcal{ \bar{H}},\forall t,\] (33c) \[z_{q[1],t}\geq 0,\forall q,\forall l,\forall t,\] (33d) \[z_{q[1],t}\in\{0,1\},\forall q,\forall l,\forall t,\] (33e) \[z_{N_{L}t}W_{N_{L}+1}^{dev}+b_{N_{L}+1}^{dev}\leq f_{nom}-f_{lim}\] (33f) \[z_{N_{L}t}W_{N_{L}+1}^{ref}+b_{N_{L}+1}^{ref}\leq-RoCoF_{lim} \tag{33g}\]

## VI Result Analysis

A case study on IEEE 24-bus system [26] is provided to demonstrate the effectiveness of the proposed methods. This test system contains 24 buses, 33 generators and 38 lines, which also has wind power as renewable resources. The mathematical model-based data generation is operated in Python using Pyomo [27]. The PSS/E software is used for time domain simulation and labeling process [28]. We use full-scale models for the dynamic simulation during the labeling process: GENROU and GENTPJ for the synchronous machine; IEEEX1 for the excitation system; IESGO for the turbine-governor; PSS2A for the power system stabilizer. Standard WTG and corresponding control modules are employed. The FCUC is performed using Pyomo and Gurobi on a window laptop with Intel(R) Core(TM) i7 2.60GHz CPU and 16 GB RAM.

### _Predictor Training_

We first generate 3000 samples as \(\mathcal{L}\) for predictor training. Each case is labeled with security status, 0 for insecure and 1 for secure based on post contingency conditions. To ensure the practicality of the dataset and the generality of the trained model, load profile and RES profile are sampled based on Gaussian distribution while the deviation of means value ranges from [-20%, 20%] of the based value. The optimality gap of the solver is set to 0.1%. We assume SGs have adequate reactive power capacity, and WTGs are controlled with a unity power factor.
For each strategy, we selected 500 samples from the generated training dataset. The distributions of RoCoF values of the sampled cases are depicted in Fig. 5. The interval size of the distribution bin is set 0.1, and regions closest to the threshold are [0.4, 0.5] and [0.5, 0.6]. With randomly selected method, most of the cases distribute within [0.6, 0.7] and [0.7, 0.8], while only 15% of the cases are within the range [0.4, 0.6] closest to the threshold. With uncertainty sampling applied, nearly 40% of the cases the selected cases centered around the threshold while cases with RoCoF values larger than 1.0 Hz/s are filtered out. When we use the proposed active learning method, the samples closest to the threshold increase; the number of samples whose RoCoF vales far away from the threshold increases as desired.

The proposed frequency metrics predictor with the same DNN structure is then tested on different dataset. As results shown in TABLE I and TABLE II, RoCoF prediction accuracies of all cases are above 93.27% with an error tolerance of 5%, while the frequency deviation validation accuracies are relatively lower. Predictor trained by uncertainty sampling dataset has the highest prediction accuracy on both RoCoF and frequency deviation values as expected, which gives more accurate frequency related constraints. While the validation accuracy of the proposed active learning dataset is only slightly lower than the uncertainty sampling dataset-based predictor.

Full-range out-of-sample dataset are used for sparse neural network validation. Sparsity of the neural network is increased from 0% to 90% with interval at 10%. The results of RoCoF prediction accuracy with the tolerance of 5% are depicted in Fig. 6. As we can see in both cases, validation accuracy is relatively high at the beginning of the test. When sparsity reaches 60%, the RoCoF prediction accuracy of predictor trained by uncertainty sampling dataset drops significantly, implying low robustness. For predictor based on randomly selected dataset, the accuracy drops to 80.04% at the sparsity of 50% and gets totally lost at the sparsity of 0.7. Meanwhile, the predictor trained by active sampling dataset significantly outperforms the others at the sparsity of 80%, implying accurate prediction with much less computational burden.

Results in Fig. 7 show that the predictor trained by active sampling dataset has higher robustness against sparsity than two other cases. However, comparing to the RoCoF prediction accuracy, frequency deviation prediction accuracy is more sensitive to the change in network sparsity. Although having superiority over two other predictors, the prediction accuracy of predictor trained by active sampling dataset is lost at the sparsity of 0.7.

### _Simulation Results_

The total scheduling horizon is 24 hours, and hours 9-12 are selected to be the time instance where frequency related constraints are applied to secure system stability against generator contingency considering high penetration level of intermittent wind generation and peak hour impact. The test case has a demand ranging from 1,348 MW to a peak of 1,853MW. Regarding post-contingency frequency limits, the maximal frequency deviation should not be larger than 0.5 Hz, and the

Fig. 5: RoCoF value distributions for three different datasets.

Fig. 6: RoCoF prediction accuracy with different NN sparsity.

Fig. 7: Frequency deviation prediction accuracy with different NN sparsity.
RoCoF must be higher than -0.5Hz/s to avoid the tripping of RoCoF-sensitive protection relays. The optimality gap is set to 0.1%.

We first investigate the impact of frequency metrics predictor sparsity on efficiency of sparse neural network based FCUC (SNN-FCUC) without ReLU linearization. The heatmap of computational time is shown in Fig. 8, and 100% sparsity indicates no frequency related constraints. Computational time less than 5 seconds indicating non-binding of the constraints. As we can see, the computational times of RS and US based predictors drops significantly when sparsity reaches 70%, since frequency related constraints under such sparsity are not binding, implying frequency stability is not enforced as demonstrated with time-domain simulations. For the case where predictor is trained by AS dataset, the solution is no longer valid in terms of frequency stability when sparsity reaches 90%. It should be noted that although the RS based predictor has an RoCoF prediction accuracy of 77.69% at the sparsity of 70%, frequency requirements are not respected. At the same time, the AS based predictor shows much higher robustness when being incorporated into MILP problem.

Frequency metrics predictor with sparsity of 0.8 is selected based on the proposed method. The trained well predictor is then incorporated into ALSNN-FCUC models. RoCoF prediction accuracy of the sparse predictor is 87.47%, and the frequency deviation prediction accuracy is 80.80%. \(\gamma\) for active neuron selection is set 0.25. \(\hat{f}_{max}\) and \(\Delta f_{max}\) are obtained by conducting time domain simulations under worst contingency on PSS/E at hour 10.

The simulation results of the proposed ALSNN-FCUC model and benchmark models are listed in TABLE III. Comparing to T-SCUC, all frequency constrained models have relatively higher operational cost, the extra cost comes from the efforts in securing the frequency nadir and RoCoF stability. Solution of LRC-SCUC model is relatively conservative due to approximation error. \(\hat{f}_{max}\) of DNN-FCUC model is 0.50 Hz/s, indicating that the solution perfectly satisfies the threshold and there is no conservativeness. It should be also noted that \(\Delta f_{max}\) of models with RoCoF related constraints applied are all within the safe range, implying RoCoF related constraints are more likely to bind comparing to the frequency deviation constraints. And inertia related protections would be a main factor that limits the transition toward RES dominant system. A noticeable increase in computational time can also be observed when there is no approximation in the DNN-FCUC. For the proposed ALSNN-FCUC model, the computational time is reduced by 62% from 22.56 s to 8.56 s when sparse computation as well as active ReLU linearization is applied. Results show that such algorithm significantly improves the computational efficiency while maintaining the highest post-contingency RoCoF values within an acceptable range.

We then compare the time domain simulation results of the proposed ALSNN-FCUC with ERC-SCUC where frequency related constraints are derived from system equivalent model. Worst G-1 contingency is conducted under dispatching during hour 10. RoCoF evolutions are shown in Fig. 9 and Fig. 10. As we can see, widely used uniform model-based approach cannot ensure locational RoCoF security due to approximation error of model simplification. With frequency metrics predictor-based constraints added, the proposed ALSNN-FCUC frameworks can secure the system highest locational RoCoF within a safe range.

Additionally, impact of total number of constrained hours on computational time of FCUC models over 24 hours scheduling horizon is investigated. Results in TABLE IV show that for DNN-FCUC without sparse computation and active linearization process, computational time for solving FCUC frame

Fig. 8: Computational time of SNN-FCUC with different sparsity.

Fig. 10: RoCoF evolution of ALSNN-FCUC model under worst contingency at hour 10.

Fig. 9: RoCoF evolution of ERC-SCUC model under worst contingency at hour 10.
work increases exponentially as the number of hours that enforce frequency requirements increases. DNN-FCUC reaches the time limit at 3600 seconds when the number of total constrained periods is larger than 16 periods. While the proposed ALSNN-FCUC model has much higher efficiency.

The voltage dynamics of generators are plotted in Fig. 11. As we can see, the proposed ALSNN-FCUC dispatch also satisfies the voltage limits for given conditions. It should be noted that voltage and related metrics could be potentially incorporated into the frequency metrics predictor-based constraints with nearly negligible increase in the computational burden.

## VII Conclusions

Reduced system inertial due to high renewable penetration level will lead to frequency insecurity under worst G-1 contingencies. Incorporation of frequency related constraints into SCUC has been used to secure system post contingency frequency stability. However, model-based approaches either fail to secure locational RoCoF stability due to approximation error or provides too conservative solutions with extra costs. This paper proposes an ALSNN-FCUC model which incorporates frequency related constraints derived from deep neural networks. A DNN based frequency metrics predictor is constructed to represent maximal frequency and the highest RoCoF value. With concepts of sharing parameters, more potential system metrics can be tracked without increasing much computational burden. The proposed active sampling method can improve the robustness of the predictor when applying sparse computation during training process. In addition, active ReLU linearization has been implemented to further improve the FCUC computational efficiency. Verifications on PSS/E show that the proposed ALSNN-FCUC model can secure system frequency stability without conservativeness.
