Introducing a Deep Neural Network-based Model Predictive Control
  Framework for Rapid Controller Implementation
Model Predictive Control (MPC) provides an optimal control solution based on
a cost function while allowing for the implementation of process constraints.
As a model-based optimal control technique, the performance of MPC strongly
depends on the model used where a trade-off between model computation time and
prediction performance exists. One solution is the integration of MPC with a
machine learning (ML) based process model which are quick to evaluate online.
This work presents the experimental implementation of a deep neural network
(DNN) based nonlinear MPC for Homogeneous Charge Compression Ignition (HCCI)
combustion control. The DNN model consists of a Long Short-Term Memory (LSTM)
network surrounded by fully connected layers which was trained using
experimental engine data and showed acceptable prediction performance with
under 5% error for all outputs. Using this model, the MPC is designed to track
the Indicated Mean Effective Pressure (IMEP) and combustion phasing
trajectories, while minimizing several parameters. Using the acados software
package to enable the real-time implementation of the MPC on an ARM Cortex A72,
the optimization calculations are completed within 1.4 ms. The external A72
processor is integrated with the prototyping engine controller using a UDP
connection allowing for rapid experimental deployment of the NMPC. The IMEP
trajectory following of the developed controller was excellent, with a
root-mean-square error of 0.133 bar, in addition to observing process
constraints.
David C. Gordon, Alexander Winkler, Julian Bedei, Patrick Schaber, Jakob Andert, Charles R. Koch
Introducing a Deep Neural Network-based Model Predictive Control Framework for Rapid Controller Implementation

###### Abstract

Model Predictive Control (MPC) provides an optimal control solution based on a cost function while allowing for the implementation of process constraints. As a model-based optimal control technique, the performance of MPC strongly depends on the model used where a trade-off between model computation time and prediction performance exists. One solution is the integration of MPC with a machine learning (ML) based process model which are quick to evaluate online. This work presents the experimental implementation of a deep neural network (DNN) based nonlinear MPC for Homogeneous Charge Compression Ignition (HCCI) combustion control. The DNN model consists of a Long Short-Term Memory (LSTM) network surrounded by fully connected layers which was trained using experimental engine data and showed acceptable prediction performance with under 5% error for all outputs. Using this model, the MPC is designed to track the Indicated Mean Effective Pressure (IMEP) and combustion phasing trajectories, while minimizing several parameters. Using the acados software package to enable the real-time implementation of the MPC on an ARM Cortex A72, the optimization calculations are completed within 1.4 ms. The external A72 processor is integrated with the prototyping engine controller using a UDP connection allowing for rapid experimental deployment of the NMPC. The IMEP trajectory following of the developed controller was excellent, with a root-mean-square error of 0.133 bar, in addition to observing process constraints.

## I Introduction

Model-based optimal control techniques leverage the significant advances in system modeling and the computational performance increases seen over the last two decades [1]. A wide range of model-based control methods have been investigated including: linear quadratic regulator [2], sliding mode controller [3], adaptive control [4], and Model Predictive Control (MPC) [5]. Of these model-based control strategies MPC is the most widely used in a range of applications [6]. Which takes advantage of the ability of MPC to provide an optimal control solution while allowing for the implementation of constraints on system states and controller outputs [7]. For automotive applications, MPC sounds like an ideal solution as internal combustion engines (ICEs) have nonlinear process dynamics and have many constraints that must be obeyed, however, the high computational demand of the online MPC has meant that it has only recently been applied to vehicles [8, 9]. One such solution is the integration of black-box machine learning (ML) process models which are quick to evaluate during MPC optimization.

One specific application of interest for the application of MPC is the combustion control of Homogeneous Charge Compression Ignition (HCCI) which has shown promise in reducing engine-out emissions (99% reduction in NO\({}_{\text{x}}\) vs modern gasoline combustion [10]) and increasing efficiency (up to 30% compared to current gasoline engines [11]). The challenge is that HCCI is susceptible to large cyclic variations and correspondingly poor combustion stability as the combustion process lacks a direct mechanism to control combustion timing [12]. Specifically, HCCI operation suffers from significant cycle-to-cycle coupling, resulting from high exhaust gas re-circulation, leading to bounded operation due to misfire at low loads and high-pressure rise rates and peak pressure at high loads which complicates modeling and control implementation.

Several ML techniques have been widely used for addressing engine performance, emission modeling and control [13, 14, 15]. As HCCI combustion experiences significant cyclic coupling a Recurrent Neural Network (RNN) is of interest for process modeling due to the inclusion of backward connections to handle sequential inputs [16]. However, the contribution of earlier time steps become increasingly small (the "vanishing gradient") and thus RNN cannot accurately capture long-term dependencies in the process. Therefore, memory cells can be introduced of which the Long Short-Term Memory (LSTM) cell is the most common [17]. Each LSTM cell has two recurrent loops, one for long-term information and one for short-term information. However, traditionally the integration of an LSTM model into a nonlinear Model Predictive Controller (NMPC) has been focused on slow-response applications such as temperature set-point planning for buildings [18]. Recent progress in the experimental application of LSTM-NMPC to high-speed systems including diesel combustion control has shown great potential of the strategy [19]. Unlike the existing literature, this work demonstrates the implementation of a real-time controller on an external ARM Cortex A72 processor using the acados optimal control framework to allow for
real-time execution of the NMPC for control of HCCI combustion [20]. The controller will be designed to track engine load and combustion timing trajectories while minimizing fuel and water consumption, pressure rise rates, and NO\({}_{\text{x}}\) emissions.

## II Experimental Setup

A single cylinder research engine (SCRE) outfitted with a fully variable electromechanical valve train (EMVT) is used. The EMVT system allows for engine operation with a variety of valve strategies, however, in this work only negative valve overlap (NVO) will be used to provide the required thermal energy for HCCI combustion. Fuel, conventional European RON 95 gasoline with 10 % ethanol, and distilled water is directly injected into the combustion chamber. Full engine details can be found in [21]. The SCRE is controlled using a dSPACE MicroAutoBox II (MABX) rapid control prototyping (RCP) ECU containing an Xilinx Kintex-7 FPGA which is used to calculate combustion metrics in real-time [22].

## III Deep Neural Network-based Engine Model

To model the HCCI engine performance and emissions, a deep neural network (DNN) with seven hidden layers (six Fully Connected (FC) layers and one LSTM layer) was developed as shown in Fig 1.

To train this DNN, which has 2260 learnable parameters, a data set of 65,000 consecutive cycles was collected from the SCRE. During the engine operation, the process inputs of duration of injection (DOI) of fuel, DOI of water, and negative valve overlap (NVO) duration were varied in both amplitude and frequency. Then, using a similar training process to previous work, the combustion outputs of engine load represented by indicated mean effective pressure (IMEP), combustion phasing angle (CA50), maximum pressure rise rate (MPRR) and engine out nitrogen oxide emissions NO\({}_{\text{x}}\) were predicted [19]. The overall accuracy of this model for each output is summarized in Table I which lists the mean squared error in percent for training and validation data. MPRR was the most difficult parameter to predict, with a 4.7% error on the validation dataset, while other outputs were predicted with less than 4% error.

For implementation into an MPC, this model is formulated using a nonlinear state-space representation to allow for integration into acados as described in [23]. The nonlinear state-space model is given by

\[x(k+1) =f\left(x(k),u(k)\right), \tag{1a}\] \[y(k) =f_{\text{FC,out}}\left(f\left(x(k),u(k)\right)\right),\] (1b) \[=g\left(x(k),u(k)\right),\]

with \(f\) combining the input and LSTM layers, \(f_{FC,out}\) representing the output layers. Here \(x(k)\) are the internal model states (corresponding to the LSTM: four cell states, \(c(k)\), and four hidden states, \(h(k)\)), \(y(k)\) the model outputs, and \(u(k)\) the model inputs. These are:

\[x(k) =\begin{bmatrix}c(k-1)\\ h(k-1)\end{bmatrix}\in\mathbb{R}^{8},\quad y(k)=\begin{bmatrix}y_{\text{MEP}}(k) \\ y_{\text{CA50}}(k)\\ y_{\text{NO}_{x}}(k)\\ y_{\text{MPRR}}(k)\end{bmatrix}\in\mathbb{R}^{4},\] \[u(k) =\begin{bmatrix}y_{\text{IMEP}}(k-1)\\ y_{\text{CA50}}(k-1)\\ u_{\text{DOI,fuel}}(k)\\ u_{\text{DOI,water}}(k)\\ u_{\text{NVO}}(k)\end{bmatrix}\in\mathbb{R}^{5}. \tag{2}\]

To reduce the oscillation of the outputs, the gradient of manipulated variables is added as new inputs [24]. This ensures that the positive definite weighting matrix forces the change to zero thus allowing the controller to achieve the desired output setpoints:

\[\underbrace{\begin{bmatrix}x(k+1)\\ u(k)\end{bmatrix}}_{\tilde{x}(k+1)} =\underbrace{\begin{bmatrix}f(x(k),u(k-1)+\Delta u(k))\\ u(k-1)+\Delta u(k)\end{bmatrix}}_{\tilde{f}(x(k),\Delta u(k))}, \tag{3a}\] \[\underbrace{\begin{bmatrix}y(k)\\ u(k-1)\end{bmatrix}}_{\tilde{y}(k)} =\underbrace{\begin{bmatrix}g(x(k))\\ u(k-1)\end{bmatrix}}_{\tilde{y}(\tilde{x}(k))}. \tag{3b}\]

This formulation allows for both the absolute inputs and their gradient to be penalized within the cost function.

Thus, the discrete Optimal Control Problem (OCP) is defined as follows

\[\min_{\begin{subarray}{c}\Delta u_{0},\ldots,\Delta u_{N-1}\\ \tilde{x}_{0},\ldots,\tilde{x}_{N}\\ \tilde{y}_{0},\ldots,\tilde{y}_{N}\end{subarray}} \sum_{i=0}^{N}\left\|r_{i}-\tilde{y}_{i}\right\|_{Q}^{2}+\left\| \Delta u_{j}\right\|_{R}^{2}\] (4) s.t. \[\tilde{x}_{0} =\begin{bmatrix}x(k),\ u(k-1)\end{bmatrix}^{\top},\] \[\tilde{x}_{i+1} =\tilde{f}(\tilde{x}_{i},\Delta u_{i}) \forall i\in\mathbb{H}\setminus N,\] \[\tilde{y}_{i} =\tilde{g}(\tilde{x}_{i},\Delta u_{i}) \forall i\in\mathbb{H},\] \[u_{\min} \leq F_{u}\cdot\tilde{u}_{k}\leq u_{\max} \forall i\in\mathbb{H},\] \[y_{\min} \leq F_{y}\cdot\tilde{y}_{k}\leq y_{\max} \forall i\in\mathbb{H}\]

where \(\mathbb{H}=\{0,1,\ldots,N\}\). The reference \(\tilde{r}_{i}\) and the weighting matrix \(Q\) are selected such that deviations from the requested IMEP and CA50 are penalized while minimizing NO\({}_{\text{x}}\) emissions, duration of injected fuel DOI,fuel\((k)\) and water DOI,water\((k)\) and change in control input \(\Delta u\). Therefore, the specific cost function \(J\) is

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Unit** & **Training** & **Validation** \\ \hline \multirow{2}{*}{\(y_{\text{IMEP}}\)} & [bar] & 0.074 & 0.077 \\  & [\%] & 2.7 & 2.8 \\ \hline \multirow{2}{*}{\(y_{\text{NO}_{x}}\)} & [ppn] & 18 & 16 \\  & [\%] & 4.2 & 3.8 \\ \hline \multirow{2}{*}{\(y_{\text{MPRR}}\)} & [bar/CAD] & 1.6 & 1.7 \\  & [\%] & 4.4 & 4.7 \\ \hline \multirow{2}{*}{\(y_{\text{CA50}}\)} & [CAD] & 1.5 & 1.6 \\  & [\%] & 3.4 & 3.6 \\ \hline \hline \end{tabular}
\end{table} TABLE I: RMSE and normalized RMSE of DNN model vs. experimental data
specified as

\[J =\sum_{i=0}^{N}\underbrace{\left\|r_{\text{IMEP},i}-y_{\text{IMEP},i} \right\|_{q_{\text{IMEP}}^{2}}^{2}}_{\text{Reference Tracking}}\] \[+\underbrace{\left\|r_{\text{CA50},i}-y_{\text{CA50},i}\right\|_{ \text{CA50}}^{2}}_{\text{Reference Tracking}}\] \[+\underbrace{\left\|u_{\text{DOI},\text{fuel},i}\right\|_{\text{ TOD},\text{fuel}}^{2}+\left\|u_{\text{DOI},\text{water},i}\right\|_{\text{TOD}, \text{water}}^{2}}_{\text{Fuel / Water consumption reduction}}\] \[+\underbrace{\left\|y_{\text{NO}_{x},i}\right\|_{q_{\text{NO}_{x} }}^{2}}_{\text{Emission Reduction}}+\underbrace{\left\|\Delta u_{i}\right\|_{R}^{2}}_{ \text{Oscillation Reduction}} \tag{5}\]

One significant advantage of using MPC for combustion control is the ability to impose constraints on inputs and outputs to ensure safe engine operation. \(F_{u}\) and \(F_{y}\) in Eq. 4 are diagonal matrices which map the bounded outputs and inputs. The control outputs are limited to match the hardware used (\(u_{\text{min},\text{max}}\)) while constraints imposed on the outputs (\(y_{\text{min},\text{max}}\)) are used to guarantee safe engine operation as summarized in Table II. In this work the constraints are not overly restrictive, however, in future testing, these limits could be modified to meet specific legislation or design constraints.

## IV Experimental deployment of LSTM-NMPC

One of the main challenges of deploying an NMPC for engine control is the limited calculation time available. For this work, the engine is operated at 1500 rpm resulting in one complete engine cycle taking 80 ms. The combustion metrics, calculated on the FPGA, are completed at 60 crank angle degrees (CAD) after top dead center (aTDC) [22] and the valve timing setpoint must be known by 260 CAD, so only 200 CAD or 22 ms are available for the NMPC calculation. To meet the real-time requirements, the computationally efficient open source package acados is used [25]. When compared to previous LSTM-based NMPC integration [19], this work offloads the NMPC calculation to an ARM Cortex A72 processor located in a Raspberry Pi 400 (RPI) overclocked to 2.2 Ghz clock frequency. The RPI communicates with the main engine controller running on the dSPACE MABX II over UDP as shown in Figure 2.

The NMPC calculation on the RPI is standalone from the main engine controller (dSpace MABX II) other than receiving the current measured states (MPRR, IMEP, CA50 and NO\({}_{\text{x}}\) emissions) and reference from the MABX as shown in figure 3. Both the optimizer and LSTM model are calculat

\begin{table}
\begin{tabular}{c c c} \hline Lower bound & Variable & Upper bound \\ \hline
1 bar & \(y_{\text{IMEP}}\) & 6 bar \\
0 CALQTDC & yCA50 & 17 CADATDC \\
0 ppm & \(y_{\text{NO}_{x}}\) & 500 ppm \\
0 bar/CAD & \(y_{\text{MPRR}}\) & 15 bar/CAD \\
0 ms & \(u_{\text{DOI},\text{fuel}}\) & 1.50 ms \\
0 ms & \(u_{\text{DOI},\text{water}}\) & 1.00 ms \\
150 CAD & \(u_{\text{NVO}}\) & 360 CAD \\ \hline \end{tabular}
\end{table} TABLE II: NMPC Constraint Values

Fig. 1: Structure of proposed deep neural network model for engine performance and emission modeling. LSTM: Long short-term memory, DOI: duration of injection, IMEP: indicated mean effective pressure, MPRR: maximum pressure rise rate, CA50: combustion phasing for 50% heat release

Fig. 2: Block diagram of the split controller structure running on a Raspberry Pi 400 and Rapid Control Prototyping (RCP) unit dSPACE MABX II
calculated actuations for the next cycle are sent to the MABX II. One significant benefit of this formulation is that the NMPC model can be updated or replaced without rebuilding the main engine control software, thus significantly reducing the development time of the controller. Additionally, this modular controller design allows for the NMPC to be executed on any external processor that can interface with the MABX II over the user datagram protocol (UDP).

When compared to other MPC implementations, in simulation acados outperforms both MATLAB's MPC toolbox using fmincon as well as the FORCES PRO backends [23]. This difference in solver performance is attributed to the higher dimension of the state than the control input vector in addition to the short prediction horizon of three cycles needed to model HCCI ICE dynamics. This allows the OCP solver to take full advantage of the condensation benefits [24, 26].

As no discretization is required for the model used, the plant model can be directly implemented using the discrete dynamics interface of acados. The Gauss-Newton approximation is used for the computation of the Hessian in the underlying Sequential Quadratic Programming (SQP) algorithm. The Optimal Control Problem in Eq. 4 leads to a band diagonal structure in the matrices of the Quadratic Problems (QPs) which are solved using the Interior Point (IP) based QP solver hpipm[27], that is provided by the acados package [25].

Fig. 3: Experimental cycle-to-cycle NMPC implementation on ARM Cortex A72: Showing IMEP reference tracking performance. Experimental data shown in blue, DNN model data in black and reference shown in dashed red.
Using a model-in-the-loop (MiL) simulation run on the targeted hardware; the weights, number of SQP iterations and prediction horizon are defined for the NMPC. These MiL simulations showed that three SQP iterations can be completed in the available calculation window. Additionally, as previous work has shown that the cycle-to-cycle dependency of HCCI lasts approximately two cycles so a prediction horizon of three cycles is sufficient [28].

For experimental implementation, a reference load profile is provided to the controller using a target IMEP. Additionally, a reference CA50 is provided to keep the combustion phasing at an efficient operation point of 6 CAD aTDC. Using the constraints specified in Table II the NMPC is implemented and the experimental LSTM-NMPC performance can be seen in figure 3. As the NMPC allows for multi-objective tracking both the IMEP and CA50 tracking performance will be evaluated. A simulated load profile of varying IMEP steps is provided resulting in a tracking performance of 0.133 bar RSME while being able to keep the CA50 at the setpoint of 6 CAD aTDC with an RSME of 1.83 CAD. As expected, the duration of fuel injection corresponds with the requested IMEP. This trend is also seen in the NVO duration where with increased fuelling a reduced NVO is requested to allow more air into the cylinder.

The constraints applied are obeyed for all but one of the cycles (cycle 552) where the 300 ppm constraint on \(\text{NO}_{\text{x}}\) emissions is slightly exceeded to 305 ppm. This is likely due to plant-model mismatch, resulting from the model predicting a slightly lower \(\text{NO}_{\text{x}}\) output than experimental data. With reference to Figure 3, the model prediction for both CA50 and IMEP parameters appear damped compared to the experimental values. However, for both engine performance metrics, the model is able to capture the trends quite accurately with the DNN model predicting IMEP with an RSME of 0.09 bar and for CA50 the RSME is 1.27 CAD.

The experimental testing of the acados NMPC executed on the ARM Cortex A72 external processor resulted in an average execution time of 1.18 ms with all calculations taking below 1.4 ms for each of the 650 cycles tested. Even with the 1 ms UDP communication time, the NMPC is below the 22 ms available and shows that if needed the model complexity could be increased with the computational power available.

The LSTM model used in the NMPC implementation has the ability for the cell and hidden states to vary depending on the current engine output. This allows the LSTM model to adapt as the engine changes. The change in the LSTM states can be seen in Figure 4. These are not physical but rather internal states resulting from the structure of the DNN model used.

When compared to the time and resource-intensive process traditionally used for developing a look-up table-based control strategy the LSTM-based NMPC provides a strategy to allow for rapid controller development. For the model used in this work, the experimental data collected (65,000 engine cycles) took 1.5 hours of testbench time to collect. Often the most significant drawback of blackbox-based models is the required training time for the model itself, however, using modern computing hardware (Intel I7-12700K based PC with a NVIDIA RTX 3090Ti) the model training took just under 3 hours. Taking advantage of the flexibility of acados allows for efficient NMPC design and integration to the external processor. This control design process is significantly quicker than traditional engine control development which can save time and reduce calibration effort and costs while providing improved controller performance by allowing for the integration of an optimal control strategy on experimental hardware.

## V Conclusions

Overall, the development and experimental implementation of an LSTM-based NMPC executed on an external processor using the acados framework was shown. To train the DNN process model, which has 2260 learnable parameters, a data set of 65,000 consecutive cycles was used. The DNN network consists of an LSTM layer, used to capture long-term dependencies and cyclic coupling, surrounded by fully connected layers providing a computationally efficient model of the HCCI combustion outputs (IMEP, MPRR, CA50 and NOx). This model resulted in an error below 5% for all outputs on validation data.

Using this DNN model of the HCCI process, the open-source software acados provided the required embedded programming for real-time implementation of the LSTM-NMPC. Experimental testing showed HCCI cycle-to-cycle combustion control provided good load and combustion phasing tracking while simultaneously obeying process constraints. With an average turnaround time of 1.18 milliseconds, this work showed the potential of the ML-based NMPC on a real-time system where only 22 ms was available for NMPC calculation and actuation.

Fig. 4: LSTM cell and hidden states during experimental implementation. Cell states in black and hidden states in blue.
The advantage of the proposed LSTM-NMPC implementation allows for a shorter controller development time and efficient MPC integration.

The proof of concept shown in this work can then be further improved by tightening process constraints to match emission legislation and to improve the longevity of the engine. Additionally, the developed toolchain for ML-based NMPC could be applied to other nonlinear constrained systems including ICEs utilizing alternative fuels or hydrogen fuel cells.

## Acknowledgements

The research was performed under the Natural Sciences Research Council of Canada Grant 2022-03411 and as part of the Research Unit (Forschungsgruppe) FOR 2401 "Optimization based Multiscale Control for Low Temperature Combustion Engines" which is funded by the German Research Association (Deutsche Forschungsgemeinschaft, DFG).
