Combining domain knowledge models with neural models has been challenging.   End-to-end trained neural models often perform better (lower Mean Square Error) than domain knowledge models or domain/neural combinations, and the combination is inefficient to train.   In this paper, we demonstrate that by composing domain models with machine learning models, by using extrapolative testing sets, and invoking decorrelation objective functions, we create models which can predict more complex systems. The models are interpretable, extrapolative, data-efficient, and capture predictable but complex non-stochastic behavior such as unmodeled degrees of freedom and systemic measurement noise.   We apply this improved modeling paradigm to several simulated systems and an actual physical system in the context of system identification.    Several ways of composing domain models with neural models are examined for time series, boosting, bagging, and auto-encoding on various systems of varying complexity and non-linearity.   Although this work is preliminary, we show that the ability to combine models is a very promising direction for neural modeling. Modeling has been used for many years to explain, predict, and control the real world. Traditional models include science/math equations, algorithms, simulations, parametric models which capture domain knowledge, and interpolative models such as cubic splines or polynomial least squares among others which do not have explanatory value but can interpolate between known values well. The nonpredictable part of the signal is captured by a stochastic noise model. The domain/physical models predict n l-dimensional output vectors, Y ∈ R n×l given n k-dimensional input vectors, X ∈ R n×k with adjustable parameters, θ used to obtain the best fit (first term in Eq. 1). The unmodeled non-deterministic part of the data is often attributed to random noise fit to various stochastic models N (φ) with parameters φ (2nd term, Eq. (1)). This traditional approach has been very successful. The advantages of a good model include high data efficiency, interpretable, the ability to extrapolate to predict outputs from inputs beyond the range of the training input data, and composable (multiple models can be combined to solve more complex problems). However, this traditional approach has limitations. Complex systems often have degrees of freedom which are not modeled by the traditional models. These unmodeled degrees of freedom or systematic errors of the measurement are not modeled adequately by the noise model. In addition, the parameters of the physical models, θ can be in error or be time dependent. In these cases the behavior of unmodeled part of the system is not random and thus the usual combined deterministic-stochastic model is inadequate. Neural models NN(X; W ), e.g. neural network models, are fundamentally just another form of parametric models where X are the inputs and W are the weight parameters. However, neural modes have unique properties to exploit. First, neural models can handle high dimensional inputoutput relations with complex patterns. Second, like interpolative models such as cubic splines or polynomials, NNs are sufficiently expressive to fit many possible relations but are often not good at extrapolation, (see below). Trask et al. (2018) Third, neural models do not require handcrafting basis functions. Hence, neural models have the potential for describing unmodeled degrees of freedom, systematic errors, and nonstationary behavior. In this paper, neural modeling is combined with traditional modeling to achieve the advantages of both traditional and neural models and compensate for the problems mentioned above using following steps. (1) Composing Hybrid Models. We examine several ways of creating hybrid models: boosting, ensemble, and cyclical autoencoder (Fig. 1) . Combining domain models and neural models requires assumptions about relationship between various system and noise. For example, Eq. (1) makes an implicit assumption that the models are composed by addition but there are many other possible assumptions. Unlike most boosting approaches, we use different model classes and loss functions for the various stages. (2) Extrapolation Testing. An extension to the traditional machine learning approach of dividing the data set into test and training portions is extended to include both interpolative and extrapolative testing sets as a stringent test of modeling power. (3) Stochastic Loss. Unlike previous approaches, the quality of the hybrid models to produce truly stochastic residuals is enforced using novel loss functions that enforce appropriate correlation of residuals. In this work, this paradigm is applied to system-identification (SysID) for simulated and real systems. The results demonstrate that these models decompose into deterministic, predictable, and stochastic components and can handle more complex systems Combining neural models with physics(domain) and stochastic models greatly expand the ability to model complex phenomena particularly for control. The expanded hybrid models incorporate the domain knowledge, interpretability, data efficiency and extrapolability of domain models with neural models which can model complex, high dimension, but predictable uncontrolled, unmodeled degrees of freedom of the system and of the measurement system (systematic noise). Using boosting, novel whitening objective functions, and extrapolative/interpolative testing sets, these hybrid models capture the behavior of more complex models in a meaningful decomposition. These models help solve the problems of unmodeled non-stochastic components of system behavior. For future work, measures such as signal-to-noise, error rates etc can be generalized to the case of non-stochastic neural modeled behavior. The combined modeling solves problems with structure noise. In the field of control, stability bounds for control systems can be implemented by using these models in the context of robust control.
Improved modeling of complex systems uses hybrid neural/domain model composition, new decorrelation loss functions and extrapolative test sets  This paper conducts experiments to compare the extrapolative predictions of various hybrid models which compose physical models, neural networks and stochastic models, and tackles the challenge of unmodeled dynamics being a bottleneck. This paper presents approaches for combining neural network with non-NN models to predict behavior of complex physical systems.