DON-LSTM: Multi-Resolution Learning with DeepONets and Long Short-Term
  Memory Neural Networks
Deep operator networks (DeepONets, DONs) offer a distinct advantage over
traditional neural networks in their ability to be trained on multi-resolution
data. This property becomes especially relevant in real-world scenarios where
high-resolution measurements are difficult to obtain, while low-resolution data
is more readily available. Nevertheless, DeepONets alone often struggle to
capture and maintain dependencies over long sequences compared to other
state-of-the-art algorithms. We propose a novel architecture, named DON-LSTM,
which extends the DeepONet with a long short-term memory network (LSTM).
Combining these two architectures, we equip the network with explicit
mechanisms to leverage multi-resolution data, as well as capture temporal
dependencies in long sequences. We test our method on long-time-evolution
modeling of multiple non-linear systems and show that the proposed
multi-resolution DON-LSTM achieves significantly lower generalization error and
requires fewer high-resolution samples compared to its vanilla counterparts.
Katarzyna Michałowska, Somdatta Goswami, George Em Karniadakis, Signe Riemer-Sørensen
# DON-LSTM: Multi-Resolution Learning with DeepONets and Long Short-Term Memory Neural Networks

###### Abstract

Deep operator networks (DeepONets, DONs) offer a distinct advantage over traditional neural networks in their ability to be trained on multi-resolution data. This property becomes especially relevant in real-world scenarios where high-resolution measurements are difficult to obtain, while low-resolution data is more readily available. Nevertheless, DeepONets alone often struggle to capture and maintain dependencies over long sequences compared to other state-of-the-art algorithms. We propose a novel architecture, named DON-LSTM, which extends the DeepONet with a long short-term memory network (LSTM). Combining these two architectures, we equip the network with explicit mechanisms to leverage multi-resolution data, as well as capture temporal dependencies in long sequences. We test our method on long-time-evolution modeling of multiple non-linear systems and show that the proposed multi-resolution DON-LSTM achieves significantly lower generalization error and requires fewer high-resolution samples compared to its vanilla counterparts.

## Introduction

Modeling the temporal evolution of dynamical systems is of paramount importance across various scientific and engineering domains, including physics, biology, climate science, and industrial predictive maintenance. Creating such models enables future predictions and process optimization, and provides insights into the underlying mechanisms governing these systems. Nevertheless, obtaining precise and comprehensive data for modeling poses a challenge in practical contexts. Real-world scenarios bring along the issue of data limitations, where data is typically more readily available at lower resolutions: historical measurements are stored with reduced granularity, vast quantities of legacy data originate from an outdated technology, or the costs associated with obtaining precise measurements on a large scale are prohibitive.

Multi-resolution learning algorithms serve as a solution for effectively integrating information from data collected over diverse temporal and spatial scales. Beyond addressing data availability constraints, multi-resolution methods can offer a reduction of computational costs during training, initially leveraging only lower-resolution data to capture general dynamics, and subsequently fine-tuning the models on high-resolution data.

To this end, a wide array of approaches has been proposed to tackle multi-resolution, multi-scale, or discretization-invariant learning. Among these, a natural framework that has gained prominence is neural operators, which are neural networks that learn mappings between function spaces, e.g., DeepONet, Fourier neural operator and others (Lu et al., 2021; Li et al., 2020; Cao et al., 2023; Wang and Golland, 2022; Ronneberger et al., 2015; Seidman et al., 2022). Alternative methods for multi-resolution learning include encoder-decoder-based architectures (Ong et al., 2022; Aboutalebi et al., 2022), graph and message-passing networks (Equer et al., 2023; Liu et al., 2021), and com
binations of the above (Yang et al., 2022). In a related vein, another body of research deals with conceptually similar multi-fidelity learning in DeepONets, which tackles the problem of combining data of varying quality, where high-quality samples are sparse (Lu et al., 2022; Howard et al., 2022; De et al., 2023).

In this work, we approach multi-resolution learning through the innate discretization-invariance property of DeepONets. We further propose a new architecture, DON-LSTM, which extends the architecture of the DeepONet with a long short-term memory network (LSTM), in order to capture temporal patterns in sequential outputs of the DeepONet. The main purpose of combining these two architectures is to leverage data of different resolutions in training, effectively increasing the feasible training set, as well as to assist the modeling of time-dependent evolution through explicit mechanisms of the LSTM.

The remainder of this paper is structured as follows. First, we formulate the learning problem at hand. Next, we describe our proposed architecture and the training procedure. Finally, we present and discuss our experimental results on four non-linear partial differential equations (PDEs) and low- and high-resolution training sets of various sizes. Our main findings show that our proposed multi-resolution DON-LSTM achieves lower generalization error than its single-resolution counterparts, which require a much larger high-resolution sample size in order to achieve similar precision.

## 1 Problem statement

In this study, we learn the operator \(\mathcal{N}\) that defines the evolution of a system over time starting from any given initial condition, i.e.:

\[\mathcal{N}:u(x,t=0)\to G(u(y)), \tag{1}\]

where \(u(x,t=0)\) is the function defining the initial condition and \(G(u(y))\) is the operator describing the evolution over time for any \(y=(x,t)\), where \(x\) and \(t\) are the spatial and temporal coordinates of the system's trajectory.

In practice, \(G(u(y))\) is observed at discretized fixed locations \(\{(x_{1},t_{1}),...,(x_{m},t_{n})\}\), resulting in a vector \([u(x_{1},t_{1}),...,u(x_{m},t_{n})]\in\mathbb{R}^{m\times n}\) where \(m\) is the total number of spatial discretization points and \(n\) is the total number of temporal discretization points that define the full trajectory. For the purpose of this study, we assume that for each system we have two datasets:

* High-resolution set \(D_{H}\) with \(N_{H}\) samples of time resolution \(\Delta t_{H}\),
* Low-resolution set \(D_{L}\) of size \(N_{L}\) samples of time resolution \(\Delta t_{L}\).

For the four considered PDE-based examples in this work, we set \(N_{H}=4\times N_{L}\) and \(\Delta t_{L}=5\times\Delta t_{H}\), while the spatial discretization is the same in both datasets. The multi-resolution network is trained on both datasets. The sizes of the training data are dependent on the complexity of the problem.

## 2 Proposed Multi-resolution DON-LSTM (ours)

The schematic representation of the proposed architecture is shown in Figure 1. The architecture consists of a DeepONet followed by a reshaping layer, an LSTM layer, and a final feed-forward layer which brings the output back to the predefined spatial dimension of the solution. The architecture is set up such that the DeepONet outputs are fed as inputs to the LSTM. The DeepONet approximates the solution at locations that are determined by its inputs, which enables the initial discretization-invariant training. During the training of the LSTM, we impose the sequential nature of the outputs through these inputs and use the LSTM to process them as temporal sequences.

### Deep operator network (DeepONet)

The DeepONet is an architecture based on the universal approximation theorem for operators (Chen & Chen, 1995) which combines the outputs of two deep neural networks (the "branch" and "trunk"
networks). It is formulated as (Lu et al., 2021):

\[G(u)(y)\approx\sum_{k=1}^{p}\underbrace{b_{k}(u(x_{1}),u(x_{2}),\ldots,u(x_{m}))} _{\text{branch}}\odot\underbrace{t_{k}(y)}_{\text{trunk}}\, \tag{2}\]

where \(b_{k}\) and \(t_{k}\) denote the output embeddings of the branch and the trunk network, respectively, and \(\odot\) denotes the element-wise multiplication of the outputs of the two networks. The inputs to the branch network are the initial conditions, \(u(t=0)\), discretized at \(m\) spatial sensor locations \(\{x_{1},x_{2},\ldots,x_{m}\}\), and for the trunk network they are the locations, \(y=(x,t)\), at which the operator \(G(u)\) is evaluated. Additionally, the trunk network can incorporate periodic boundary conditions through simple feature expansion that consists of applying Fourier basis on the provided spatial locations, i.e., \(x\rightarrow\cos(\frac{2\pi x}{P})\), and \(x\rightarrow\sin(\frac{2\pi x}{P})\), where \(P\) is the period (Lu et al., 2022a).

### Long short-term memory network (LSTM)

LSTM is a type of recurrent neural network (RNN), which is a class of specialized neural networks for processing long sequences. In contrast to traditional feed-forward networks, RNNs employ a hidden state, which persists as the network iterates through data sequences and propagates the information from the previous states forward. LSTMs were developed as an extension to RNNs, aimed at mitigating the problem of vanishing gradients encountered in RNNs. Vanishing gradients occur in long sequences, as the hidden state is updated at every timestep, which leads to a large number of multiplications causing the gradients to approach zero. To address this, LSTM is equipped with a set of gates: the _input gate_ regulates the inflow of new information to the maintained hidden state, the _forget gate_ determines what proportion of the information should be discarded, and the _output gate_ controls the amount of information carried forward.

### Self-adaptive loss function

In solving time-dependent PDEs, the main challenge lies in preserving the precision of the modeled solution over a long-time horizon. Introducing appropriately structured non-uniform penalization parameters can address this aspect. In all the experiments, we equip the networks with self-adaptive weights in the loss function, first introduced by McClenny & Braga-Neto (2020), which adapt during training and force the network to focus on the most challenging regions of the solution. We take the inspiration from Kontolati et al. (2022), where in turn self-adaptive weights considerably improve the accuracy prediction of discontinuities or non-smooth features in the solution. The self-adaptive weights are defined for every spatial and temporal location, and they are updated along with the

Figure 1: The proposed architecture. In the first phase, the DeepONet maps the solution operator from the initial condition \(u_{t=0}\) to solutions at later timesteps. During the DeepONet training, the output \(u\) can be defined at arbitrary temporal locations. In the next two training stages, the outputs must have a fixed time interval \(\Delta t\) to be compatible with the LSTM. The DeepONet output solutions are reshaped to be represented in a temporally sequential manner and processed by an LSTM. The LSTM lifts the dimension to the specified number of neurons and returns all hidden states. The last layer is a fully-connected dense layer bringing the embedding to the original size of the solution.
network parameters during optimization. Specifically, the training loss is defined as:

\[\mathcal{L}(\mathbf{\theta},\mathbf{\lambda})=\frac{1}{N}\sum_{i=1}^{N}g(\mathbf{\lambda})|u( x_{i},t_{i})-\hat{u}(x_{i},t_{i})|^{2}, \tag{3}\]

where \((u(x_{i},t_{i})-\hat{u}(x_{i},t_{i}))^{2}\) is the squared error between the reference \(u\) and predicted value \(\hat{u}\), and \(g(\lambda)\) is a non-negative, strictly increasing self-adaptive mask function, and \(\mathbf{\lambda}=\{\lambda_{1},\lambda_{2},\cdots\lambda_{j}\}\) are \(j\) self-adaptive parameters, and \(j\) is the total number of evaluation points. Typically, in a neural network, we minimize the loss function with respect to the network parameters, \(\mathbf{\theta}\). However, in this approach, we additionally maximize the loss function with respect to the trainable hyper-parameters using a gradient descent/ascent procedure. The modified objective function is defined as:

\[\min_{\mathbf{\theta}}\max_{\mathbf{\lambda}}\mathcal{L}(\mathbf{\theta},\mathbf{\lambda}). \tag{4}\]

The self-adaptive weights are updated using the gradient descent method, such that

\[\mathbf{\lambda}^{k+1}=\lambda^{k}+\eta_{\lambda}\nabla_{\mathbf{\lambda}}\mathcal{L }(\mathbf{\theta},\mathbf{\lambda}), \tag{5}\]

where \(\eta_{\mathbf{\lambda}}\) is the learning rate of the self-adaptive weights and

\[\nabla_{\lambda_{i}}\mathcal{L}=\left[g^{\prime}(\lambda_{i})(u_{i}(\xi)- \mathcal{G}_{\theta}(\mathbf{v}_{i})(\xi))^{2}\right]^{T}. \tag{6}\]

Therefore, if \(g(\lambda_{i})>0\), \(\nabla_{\lambda_{i}}\mathcal{L}\) would be zero only if the term \((u_{i}(\xi)-\mathcal{G}_{\theta}(\mathbf{v}_{i})(\xi))\) is zero. In this work, the self-adaptive weights are normalized to sum up to one after each epoch. The weights \(\lambda\) are updated alongside the weights of the network during training.

### Training procedure

Training is performed iteratively over a predefined number of epochs. The training is performed in minibatches, i.e., in each epoch the training data is divided into multiple smaller subsets. After calculating the loss on each batch, the gradients are calculated w.r.t. the loss function and the trainable network parameters (weights) are updated. The optimization is performed using the Adam optimizer and a predefined learning rate. The training of the multi-resolution DON-LSTM is performed in three stages:

**Step 1:**: DeepONet training:

1. Initialize the DeepONet with a set of weights \(\mathbf{\theta}_{DON}\).
2. Iteratively update \(\mathbf{\theta}_{DON}\) on low-resolution data \(D_{L}\) with a learning rate \(lr_{1}\), saving weights \(\mathbf{\theta}_{DON_{i}}\) at every \(n_{freq}\) epochs.
3. Compute the MSE loss on the predictions on validation data and choose the set of weights that correspond to the lowest loss.
4. Iteratively update \(\mathbf{\theta}_{LSTM}\) on high-resolution data \(D_{H}\), with a learning rate \(lr_{1}\), saving models at every \(n_{freq}\) epochs.
5. Compute the MSE loss on the predictions on validation data and choose the set of weights that correspond to the lowest loss.

**Step 3:**: DON-LSTM training:

1. Unfreeze \(\mathbf{\theta}_{DON}\) (set as trainable parameters).
2. Iteratively update \(\mathbf{\theta}_{LSTM}\) on high-resolution data \(D_{H}\) with a learning rate \(lr_{2}\), where \(lr_{2}<lr_{1}\), saving models at every \(n_{freq}\) epochs.
3. Compute the MSE loss on the predictions on validation data and choose the set of weights that correspond to the lowest loss.
Problems considered and data generation

The performance of the proposed multi-resolution models is showcased on four infinite-dimensional non-linear dynamical systems. The numerical solutions to the Korteweg-de Vries, Benjamin-Bona-Mahony and Cahn-Hilliard equations were obtained through schemes that are second order in both space and time; the equations are spatially discretized using central finite differences and integrated in time using the implicit midpoint method. The solutions of the Burgers' equation were generated using PDEBench (Takamoto et al., 2022). All PDEs are evaluated on the one-dimensional spatial domain \(\Omega=[0,P]\), for different \(P\), with periodic boundary conditions \(u(P,t)=u(0,t)\) for all \(t\geq 0\). The training set for each PDE consists of data obtained from \(N=N_{H}+N_{L}\) different initial conditions integrated over time \(t=T\) with step size \(\Delta t\). Example data are visualized in Figure 3 and details of the time and space domains are given in Table 2, both in Appendix A.

### Korteweg-de Vries equation

The Korteweg-de Vries (KdV) equation (Korteweg & De Vries, 1895) is a non-linear dispersive PDE that describes the evolution of small-amplitude, long-wavelength systems in a variety of physical settings, such as shallow water waves, ion-acoustic waves in plasmas, and certain types of nonlinear optical waves. We consider a one-dimensional unforced case, which is given by:

\[\frac{\partial u}{\partial t}+\gamma\frac{\partial^{3}u}{\partial x^{3}}- \eta u\frac{\partial u}{\partial x}=0, \tag{7}\]

where \(u:=u(x,t)\) is the height of the wave at position \(x\) at time \(t\), and \(\eta=6\) and \(\gamma=1\) are chosen real-valued scalar parameters.

The initial conditions are each a sum of two solitons (solitary waves), _i.e._:

\[u(x,0)=\sum_{i=1}^{2}2k_{i}^{2}\text{sech}^{2}\bigg{(}k_{i}((x+\frac{P}{2}-Pd_{ i})\%P-\frac{P}{2})\bigg{)}, \tag{8}\]

where \(\text{sech}\) stands for hyperbolic secant (\(\frac{1}{\cosh}\)), \(P\) is the period in space, \(\%\) is the modulo operator, \(i=\{1,2\}\), \(k_{i}\in(0.5,1.0)\) and \(d_{i}\in(0,1)\) are coefficients that determine the height and location of the peak of a soliton, respectively. For each initial state \(t=0\) in the training data, these coefficients are drawn randomly from their distribution.

### Benjamin-Bona-Mahony equation

The Benjamin-Bona-Mahony (BBM) equation was derived as a higher-order improvement on the KdV equation and includes both non-linear and dispersive effects (Peregrine, 1966; Benjamin et al., 1972). It is used for studying a broader range of wave behaviors, including wave breaking and dispersion, and is given by:

\[\frac{\partial u}{\partial t}-\frac{\partial^{3}u}{\partial x^{2}\partial t}+ \frac{\partial}{\partial x}\left(u+\frac{1}{2}u^{2}\right)=0. \tag{9}\]

The chosen initial conditions are superpositions of two soliton waves of the following shape:

\[u_{i}(x,0)=\sum_{i=1}^{2}3(c_{i}-1)\text{sech}^{2}\bigg{(}\frac{1}{2}\sqrt{1- \frac{1}{c_{i}}}(x+\frac{P}{2}-Pd_{i})\%P-\frac{P}{2}\bigg{)}, \tag{10}\]

where \(c_{i}\in(1,3)\) and \(d_{i}\in(0,1)\) are coefficients that determine the height and location of the peak of a soliton, respectively.

### Cahn-Hilliard equation

The Cahn-Hilliard equation is used to describe phase separation with applications to materials science and physics (Cahn & Hilliard, 1958). It is expressed as:

\[\frac{\partial u}{\partial t}-\frac{\partial^{2}}{\partial x^{2}}(\nu u+ \alpha u^{3}+\mu\frac{\partial^{2}u}{\partial x^{2}})=0, \tag{11}\]
where we set \(\nu=-0.01\), \(\alpha=0.01\) and \(\mu=-0.00001\).

The initial conditions are superpositions of sine and cosine waves, i.e. \(u(x,0)=u_{1}+u_{2}\) with:

\[u_{i}(x,0)=a_{i}\text{sin}(k_{i}\frac{2\pi}{P}x)+b_{i}\text{cos}(j_{i}\frac{2 \pi}{P}x), \tag{12}\]

where \(a_{i},b_{i}\in(0,0.2)\) and \(k_{i},j_{i}\) are integers and \(j_{i},k_{i}\in[1,6]\).

### Viscous Burgers' equation

The viscous Burgers' equation (Bateman, 1915; Burgers, 1948) describes the behavior of waves and shocks in a viscous fluid or gas. It is given by:

\[\frac{\partial u}{\partial t}+\frac{\partial}{\partial x}\left(\frac{u^{2}}{2 }\right)=\frac{\nu}{\pi}\frac{\partial u^{2}}{\partial^{2}x}, \tag{13}\]

where \(x\in(0,1)\), \(t\in(0,2]\), and \(\nu=0.001\) is the diffusion coefficient.

The initial conditions are given by the superposition of sinusoidal waves:

\[u(x,0)=\sum_{k_{i}=k_{1},\ldots,k_{N}}A_{i}\sin(k_{i}x+\phi_{i}), \tag{14}\]

where \(k_{i}=2\pi n_{i}/P\) are coefficients where \(n_{i}\) are arbitrarily selected integers in \([1,n_{max}]\). \(N\) is the integer determining how many waves are added, \(A_{i}\) is a random float number uniformly chosen in \((0,1)\), and \(\phi_{i}\) is the randomly chosen phase in \((0,2\pi)\).

## 4 Experimental results

We compare the performance of our multi-resolution DON-LSTM against five benchmark networks discussed in section 3 using the relative squared error (RSE), the mean average error (MAE) and the root mean squared error (RMSE), described in Appendix E. The average values of these error metrics are presented in Table 1, and the log values of RSE against the increasing number of high-resolution training samples are shown in Figure 2. Due to space considerations, detailed tables with the evaluation of the models trained on distinct sample sizes are available in Appendix B. All evaluations are performed on high-resolution test samples of size \(N_{test}=1000\).

### Benchmark models

The benchmark models used to evaluate the performance of the multi-resolution DON-LSTM (DON-LSTM (\(D_{H}\), \(D_{L}\))) are the following:

* **DON** (\(D_{L}\)): The vanilla DeepONet trained on \(N_{L}\) low-resolution data,
* **DON** (\(D_{H}\)): The vanilla DeepONet trained on \(N_{H}\) high-resolution data,
* **DON** (\(D_{H},D_{L}\)): The vanilla DeepONet trained on \(N_{L}+N_{H}\) multi-resolution data (both datasets),
* **LSTM** (\(D_{H}\)): An architecture trained on \(N_{H}\) high-resolution data, consisting of one dense layer lifting the input to a dimension [-1, \(mn\)], a reshaping layer (into [-1, \(m\), \(n\)]) followed by an LSTM layer, where \(m\) is the spatial and \(t\) is the temporal dimension,
* **DON-LSTM** (\(D_{H}\)): The proposed architecture trained only on \(N_{H}\) high-resolution data.

The models vary by the amount and granularity of the training data. We take advantage of the discretization-invariance of deep neural operators and train the vanilla DeepONet and DON-LSTM on multi-resolution data (in the case of DON-LSTM only the DeepONet layers are trained with multi-resolution). In this problem formulation, the LSTM implicitly learns \(\Delta t\) and therefore has to be trained at the resolution used in testing (here, high-resolution). In the following, we refer to specific models by the data resolution used in their training and the models' name, e.g., _high-resolution DON_.
### Generalization performance

We evaluate the performance of our models grouping them by the number of samples used in their training (Figure 2). As expected, increasing the sample size leads to a reduction in the generalization error for all models. In nearly all cases, we also see that the multi-resolution DON-LSTM achieves the lowest error, followed by the multi-resolution DON for the KdV, BBM and Cahn-Hilliard equations, and the high-resolution LSTM for the Burgers' equation.

Our main findings can be summarized into the following:

1. The multi-resolution DON-LSTM generally achieves the lowest generalization error out of the five benchmarks.
2. In order to achieve similar accuracy with single-resolution methods (such as the vanilla LSTM) we need significantly more high-resolution training samples than for multi-resolution DON-LSTM.
3. In multiple cases the DON trained on larger amount of lower-resolution data obtains better results than the DON trained on fewer samples of high-resolution data.
4. While the DeepONet itself achieves reasonable performance, the time-dependent architecture is crucial for capturing long-time dynamics, as is evident by the superior performance of DON-LSTM and the vanilla LSTM.

\begin{table}
\begin{tabular}{l l l l l} \hline
**Model** & **Resolution** & \multicolumn{2}{c}{**MAE**} & **RMSE** & **RSE** \\ \hline \multicolumn{5}{c}{**Korteweg-de Vries equation**} \\ \hline

[MISSING_PAGE_POST]

STM** & \(\Delta t=0.02\) & (high) & 0.016\(\pm\)0.006 & 0.036\(\pm\)0.013 & 0.005\(\pm\)0.004 \\ \hline \end{tabular}
\end{table}
Table 1: The mean and the standard deviation of the prediction errors of all models. Each reported value aggregates the mean error values across all models trained on different sample sizes, where each model has been trained five times (i.e., the mean and standard deviation of the values reported in Appendix B).
## 5 Discussion

In all our experiments, the multi-resolution DON-LSTM achieved the lowest generalization error, while requiring fewer high-resolution training samples than its benchmarks. This advantage stems from the combination of two factors: the utilization of a larger training dataset, which encompasses both high- and low-resolution samples, and the integration of LSTM mechanisms that facilitate capturing the temporal evolution of the systems. We specifically observe that the inclusion of LSTM mechanisms is beneficial, as evidenced by the multi-resolution DON-LSTM outperforming the vanilla DON trained on multi-resolution data. Additionally, the inclusion of low-resolution data in early training contributes to the improvement of the prediction, as seen in the superior performance of multi-resolution DON-LSTM as compared to the vanilla LSTM and single-resolution DON-LSTM, as well as the superior performance of the multi-resolution DON in comparison to both single-resolution DONs.

When low-resolution data was not used in training, the comparison between the architectures was inconclusive, i.e., the high-resolution DON-LSTM outperformed its vanilla counterparts only in two experiments. We attribute it to the fact that DON-LSTM is comprised of a larger number of parameters, and a small training sample is not sufficient to effectively train the network, leading to under/over-fitting. We can also see that the vanilla DON struggles with adjusting all its parameters on a small sample, which becomes apparent through the fact that the model trained on a low-resolution data achieves better performance than when trained on the high-resolution data (regardless of being tested on high-resolution). This means that the inclusion of low-resolution data in early training is essential for the good performance of the proposed architecture.

## 6 Limitations and Future work

While DeepONets possess the discretization-invariance property in the output function, they require the input data to be defined at fixed locations. This problem is addressed in the literature through the employment of an encoder-decoder architecture integrated with the DeepONet (Ong et al., 2022; Zhang et al., 2022). We also note that the framework that is limited to discretization-invariant output

Figure 2: Model performance vs. amount of training samples. The \(y\)-axis shows the relative squared error (log values) for predictions on high-resolution test data, while the \(x\)-axis indicates the number of high- and low-resolution training samples. The figure compares the performance of three models: DON (red) and DON-LSTM (blue) and LSTM (green), with variations of training data denoted by \(D_{\square}\) and the line pattern (\(D_{H}\) and solid line for high-resolution, \(D_{L}\) and dotted line for low-resolution, and \(D_{H}\), \(D_{L}\) and dash-dotted line for multi-resolution). The error bars indicate the \(95\%\) confidence interval calculated over the five trained models (two standard deviations of the error).
is sufficient for applications where the system behaviour is modeled from a single-resolution input, e.g., an initial condition. For cases when multi-resolution input is desired, we highlight the existence of other neural operator architectures such as the Fourier neural operator (Li et al., 2020), or the Laplace neural operator (Cao et al., 2023).

In addition, we note that LSTM is specifically suited and limited to sequential data. We see this as an opportunity for future studies, in which the DeepONet can be extended with architectures appropriate for different types of data, for example convolutional neural networks in case of image data, or transformers for data governed by less-regular temporal patterns.

## 7 Conclusions

Our proposed architecture, DON-LSTM, seamlessly integrates the strengths of its two composites: the discretization invariance of the DeepONet and the improved sequential learning with the memory-preserving mechanisms of the LSTM. We have demonstrated that these properties can be leveraged to incorporate multi-resolution data in training, as well as to capture the intricate dynamics of time-dependent systems, leading to significantly improved predictive accuracy in modeling of the systems' evolution over long-time horizons. Our experiments clearly demonstrate the efficacy of our approach in creating accurate, high-resolution models even with limited training data available at fine-grained resolution. Moreover, the synergistic effect of our proposed architecture makes it an apt choice for real-world scenarios, promising substantial enhancements in prediction quality. This work not only advances the understanding and utilization of multi-resolution data in sequential analysis but also provides valuable insights for future research and applications.

## 8 Acknowledgement

For KM and SRS, this work is based upon the support from the Research Council of Norway under project SFI NorwAI 309834. Furthermore, KM is also supported by the PhD project grant 323338 Stipendiatstilling 17 SINTEF (2021-2023). For SG and GEK, this work was supported by the U.S. Department of Energy, Advanced Scientific Computing Research program, under the Scalable, Efficient and Accelerated Causal Reasoning Operators, Graphs and Spikes for Earth and Embedded Systems (SEA-CROGS) project, DE- SC0023191. Furthermore, the authors would like to acknowledge the computing support provided by the computational resources and services at the Center for Computation and Visualization (CCV), Brown University where all the experiments were carried out, as well as the invaluable contribution in generating the training data by Solve Eidnes.

## 9 Reproducibility

The code for reproducing the results is available in GitHub repository (Michalowska, 2023). The code includes the default parameters to generate the models and the data processing pipeline used in this paper. The details of the used architectures, training setup and data processing steps are also specified in Appendix C and Appendix D.
