MISNN: Multiple Imputation via Semi-parametric Neural Networks
Multiple imputation (MI) has been widely applied to missing value problems in
biomedical, social and econometric research, in order to avoid improper
inference in the downstream data analysis. In the presence of high-dimensional
data, imputation models that include feature selection, especially $\ell_1$
regularized regression (such as Lasso, adaptive Lasso, and Elastic Net), are
common choices to prevent the model from underdetermination. However,
conducting MI with feature selection is difficult: existing methods are often
computationally inefficient and poor in performance. We propose MISNN, a novel
and efficient algorithm that incorporates feature selection for MI. Leveraging
the approximation power of neural networks, MISNN is a general and flexible
framework, compatible with any feature selection method, any neural network
architecture, high/low-dimensional data and general missing patterns. Through
empirical experiments, MISNN has demonstrated great advantages over
state-of-the-art imputation methods (e.g. Bayesian Lasso and matrix
completion), in terms of imputation accuracy, statistical consistency and
computation speed.
Zhiqi Bu, Zongyu Dai, Yiliang Zhang, Qi Long
# MISNN: Multiple Imputation via Semi-parametric Neural Networks

###### Abstract

Multiple imputation (MI) has been widely applied to missing value problems in biomedical, social and econometric research, in order to avoid improper inference in the downstream data analysis. In the presence of high-dimensional data, imputation models that include feature selection, especially \(\ell_{1}\) regularized regression (such as Lasso, adaptive Lasso, and Elastic Net), are common choices to prevent the model from underdetermination. However, conducting MI with feature selection is difficult: existing methods are often computationally inefficient and poor in performance. We propose MISNN, a novel and efficient algorithm that incorporates feature selection for MI. Leveraging the approximation power of neural networks, MISNN is a general and flexible framework, compatible with any feature selection method, any neural network architecture, high/low-dimensional data and general missing patterns. Through empirical experiments, MISNN has demonstrated great advantages over state-of-the-art imputation methods (e.g. Bayesian Lasso and matrix completion), in terms of imputation accuracy, statistical consistency and computation speed.

Keywords:Missing value, Imputation, Semi-supervised Learning

## 1 Introduction

### Missing Value Mechanisms and Imputation

Missing data are commonly encountered in data analyses. It is well-known that inadequate handling of missing data can lead to biased findings, improper statistical inference [11, 37] and poor prediction performance. One of the effective remedies is missing data imputation. Existing imputation methods can be mainly classified as single imputation (SI) and multiple imputation (MI) [26]. The former imputes missing values only once while the latter generates imputation values multiple times from some distribution. In fields such as finance and medical research, linear models are often preferred as it is important to not only predict accurately but also explain the uncertainty of the prediction and the effect of features. In the interest of statistical inference, MI methods, including MISNN proposed in this paper, are more suitable as they adequately account for imputation uncertainty and provide proper inference.

In general, performances of imputation are highly related to the mechanisms that generate missing values, which can be categorized into three types: missing
completely at random (MCAR), missing at random (MAR) and missing not at random (MNAR). Missing data are said to be MCAR if the probability of being missing is the same for all entries; MAR means that the missing probability only depends on the observed values; MNAR means that the missing probability depends on the unobserved missing values. Intuitively, imputation is easier under MCAR mechanisms as the missing probability is only a (unknown) constant, and therefore most methods are designed to work under MCAR. However, MAR and MNAR are usually more difficult and fewer methods perform well on these problems.

### Feature Selection in Imputation Models

In many applications including gene expression and financial time series research, we need to analyze high dimensional data with number of features being much larger than number of samples. In such cases, multiple imputation, which estimates the (conditional) distribution of missing data, can be inaccurate due to the overwhelming amount of features. Existing works [11, 37] propose to use regularized linear model for feature selection, before building the imputation model. Some representative models include Lasso [31], SLOPE [4], Elastic Net [39], Adaptive Lasso [38], Sparse Group Lasso [13, 27], etc.

While the regularized linear models successfully reduces the number of features, they often fail to capture the true distribution of missing data due to the linear dependence on the selected features and information loss in the unselected features when building the imputation model. Hence, the corresponding inference can be significantly biased. MISNN proposed in this paper overcomes the shortcome via semi-parametric neural networks. At a high level, MISNN is a semi-parametric model based on neural networks, which divides predictors into two sets: the first set are used to build a linear model and the other is used to build neural networks, which are often regarded as non-parametric models. We highlight that the outperformance of MISNN is contributed both by its neural network and linear parts. The neural networks effectively capture the non-linear relationship in the imputation model, and the linear model, in addition to capturing the linear relationships, allows efficient MI, through maximum likelihood estimation for the regression parameters.

### Our Contribution

This paper makes two contributions. Firstly, we propose MISNN, a novel imputation method that outperforms state-of-the-art imputation methods in terms of imputation accuracy, statistical consistency, and computation speed. MISNN is easy to tune, interpretable, and robust to high missing rates and high-dimensional features. Secondly, MISNN is a flexible imputation framework that can be used with any appropriate feature selection method, such as Lasso and forward-selection. Additionally, MISNN is compatible with any neural network, including under or over-parameterized networks, CNN, ResNet, dropout, and more.

## 2 Related Work

Regarding missing data imputation, SI methods have long history before the concept of MI [26], of which one representative approach is the mean imputation.
Recent work in SI include matrix completion approaches that translate the imputation into an optimization problem. Existing methods such as SoftImpute [23] and MMMF (Maximum-Margin Matrix Factorization) [28] provably work under MCAR mechanisms. Meanwhile, an increasing number of MI methods are studied: MICE [32, 5] imputes missing values through the chained equations; MissForest [30] imputes missing values via bootstrap aggregation of multiple trees. Deep generative models [34, 14, 22, 19, 9], including Generative Adversarial Impu-tation Nets (GAIN), are also proposed for imputation. We remark that most of the existing methods only provably work under MCAR (though some methods empirically work well under MAR).

Regularized linear models have been proposed for MI in high-dimensional data. Bayesian Lasso [24, 16] estimates the posterior distribution of coefficients, while alternative approaches [37, 11] de-bias the estimator from the regularized linear regression. Namely, the direct use of regularized regression (DURR) and the indirect use of regularized regression (IURR). However, linear imputation models fail to capture the potential non-linear relations in the conditional distribution of missing data. MISNN falls into this line of research, is computationally more efficient than Bayesian Lasso, and captures non-linear relations during imputation.

Figure 1: MISNN framework.
Recent work has highlighted the importance of trustworthiness in missing data imputation, with privacy-preserving [18, 10, 8] and fairness-aware [21, 36, 6, 35] imputation models drawing attention. MISNN has strong interpretability, allowing for better understanding of the imputation process and greater trust in the results.

## 3 Data Setup

Denote the data matrix by \(\mathbf{D}\in\mathbb{R}^{n\times p}\), where \(n\) is the number of samples/cases and \(p\) is the number of features/variables. We define the \(j\)-th feature by \(\mathbf{D}_{j}\) and its complement features by \(\mathbf{D}_{-j}:=\mathbf{D}_{2:p}\) for \(j\in[p]\). In the presence of missing data, \(\mathbf{D}\) can be separated into two submatrices \(\mathbf{D}_{\mathrm{cc}}\) and \(\mathbf{D}_{\mathrm{ic}}\), where \(\mathbf{D}_{\mathrm{cc}}\) denotes all complete cases (i.e. all features are observed) and \(\mathbf{D}_{\mathrm{ic}}\) denotes all incomplete cases. We let \(\mathbf{D}_{\mathrm{cc},j}\) and \(\mathbf{D}_{\mathrm{ic},j}\) denote the \(j\)-th feature of complete cases and incomplete cases, respectively. We also define \(\mathbf{D}_{\mathrm{miss}}\), the set of missing features in \(\mathbf{D}\), and \(\mathbf{D}_{\mathrm{obs}}\), the set of observed features for samples in \(\mathbf{D}\). Briefly speaking, to impute the missing values, we fit an imputation model \(g\) using \(\mathbf{D}_{\mathrm{obs}}\), and use \(\mathbf{D}_{\mathrm{ic},\mathrm{obs}}\) as input to give imputation result \(\hat{\mathbf{D}}_{\mathrm{miss}}\). For the ease of presentation, we start with a single feature missing, in which only the first column in \(\mathbf{D}\) (i.e., \(\mathbf{D}_{1}\)) contains missing values. We then move on to the general missing pattern with multiple features missing in Section 5.

### A Framework for Multiple Imputation

Here we provide a brief discussion about a general framework for multiple imputation, which is also adopted in MISNN. Under the above data setting, MI methods estimate the conditional distribution \(\rho(\mathbf{D}_{\mathrm{miss}}|\mathbf{D}_{\mathrm{obs}})\) and sample imputed values from it multiple times. Assuming the distribution of \(\mathbf{D}\) is characterized by unknown parameters \(\boldsymbol{\xi}\), then

\[\rho(\mathbf{D}_{\mathrm{miss}}|\mathbf{D}_{\mathrm{obs}})=\int\rho_{\mathrm{ miss}}(\mathbf{D}_{\mathrm{miss}}|\mathbf{D}_{\mathrm{obs}},\boldsymbol{\xi}) \rho_{2}(\boldsymbol{\xi}|\mathbf{D}_{\mathrm{obs}})d\boldsymbol{\xi}\]

in which \(\rho,\rho_{1},\rho_{2}\) are three conditional distributions. For the \(m\)-th imputation, we randomly sample \(\boldsymbol{\xi}^{(m)}\) from the posterior distribution of \(\boldsymbol{\xi}\), i.e. \(\rho_{2}(\boldsymbol{\xi}|\mathbf{D}_{\mathrm{obs}})\); we then generate the \(m\)-th imputed data \(\mathbf{D}_{\mathrm{miss}}^{(m)}\) from the predictive distribution \(\rho_{1}(\mathbf{D}_{\mathrm{miss}}|\mathbf{D}_{\mathrm{obs}},\boldsymbol{\xi})\). With multiple imputed datasets, further analysis and inference can be conducted with the help of Rubin's rule [20, 26]. A detailed introduction of Rubin's rule is provided in Appendix 0.A.

## 4 Multiple Imputation with Semi-parametric Neural Network (MISNN)

At the high level, MISNN imputes the missing data in each column through a partial linear model (PLM), which takes the form

\[\hat{\mathbf{D}}_{1}=\mathbf{X}\hat{\beta}+\hat{f}(\mathbf{T})\]

where \((\mathbf{X},\mathbf{T})\), determined through feature selection, is a partition of the rest \(p-1\) columns. While the choice of \(\hat{\beta}\) and \(\hat{f}(\mathbf{T})\) can be determined in an arbitrary
manner, we adopt a partialling out approach [25] (also known as the orthogonalization in [7]) that can provide consistent parameter estimation if the true model takes the form \(\mathbf{D}_{1}=\mathbf{X}\beta+f(\mathbf{T})+\boldsymbol{\epsilon}\). To do so, we take the conditional expectation on \(\mathbf{T}\), assuming \(\mathbb{E}(\boldsymbol{\epsilon}|\mathbf{T})=0\):

\[\begin{split}\mathbf{D}_{1}&=\mathbf{X}\boldsymbol{ \beta}+f(\mathbf{T})+\boldsymbol{\epsilon}\\ \mathbb{E}(\mathbf{D}_{1}|\mathbf{T})&=\mathbb{E}( \mathbf{X}|\mathbf{T})\boldsymbol{\beta}+f(\mathbf{T})\\ \mathbf{D}_{1}-\mathbb{E}(\mathbf{D}_{1}|\mathbf{T})& =\left(\mathbf{X}-\mathbb{E}(\mathbf{X}|\mathbf{T})\right) \boldsymbol{\beta}+\boldsymbol{\epsilon}\end{split} \tag{1}\]

Let \(\mathcal{S}\) denote the set of features selected. Notice that \(\mathbf{T}:=\mathbf{D}_{-1}\setminus\mathbf{D}_{\mathcal{S}}\) is explicitly removed in the last equation. Therefore, if the number of selected features can be controlled (i.e., \(|\mathcal{S}|\) is small), we are left with a low-dimensional linear model (as \(\mathbf{X}-\mathbb{E}(\mathbf{X}|\mathbf{T})\in\mathbb{R}^{n\times|\mathcal{S }|}\)), as long as we can estimate the mapping \(\mathbb{E}(\mathbf{D}_{1}|\mathbf{T})\) and \(\mathbb{E}(\mathbf{X}|\mathbf{T})\) properly. To realize the above approach, MISNN algorithm takes three key steps:

* **Feature Selection:** During imputation of each missing feature, MISNN conducts feature selection to select at most \(n\) features. The selected features \(\mathbf{X}\) are expected to have significant linear correlation with the missing feature, which later will be fitted in a linear model (e.g., least squares).
* **Fitting Partially Linear Model:** Suppose the remaining features after the selection are denoted by \(\mathbf{T}\), MISNN fits two neural networks to learn \(\mathbb{E}(\mathbf{D}_{\text{miss}}|\mathbf{T})\) and \(\mathbb{E}(\mathbf{X}|\mathbf{T})\), so as to derive a low-dimensional ordinary linear model (1);
* **Multiple Imputation:** MISNN uses maximum likelihood to estimate parameters in (1), then draw \(M\) times from the posterior distribution of \(\hat{\boldsymbol{\beta}}\) and further draw \(\widehat{\mathbf{D}}_{\text{miss}}\) from the predictive distribution.

Note that the first two steps in combination is closely related to DebiNet [33], though we do not refine ourselves to over-parameterized neural network, and we utilize two neural networks to learn \((\mathbb{E}(\mathbf{D}_{\text{miss}}|\mathbf{T}),\mathbb{E}(\mathbf{X}|\mathbf{ T}))\). In the following, we introduce MISNN in Algorithm 1 and validate the procedure of MISNN rigorously. Here we assume the missing feature is continuous. For non-continuous features, some modifications to the algorithm should be made. See details in Appendix 0.B.

**Remark 1**: _If one only focuses on the prediction, not the inference, single imputation can be conducted in Algorithm 1. In particular, OLS can solve the linear model in step (4) and we impute by_

\[\widehat{\mathbf{D}}_{ic,1}=\left(\mathbf{X}_{cc}-\mathbb{E}(\mathbf{X}_{cc}| \mathbf{T}_{cc})\right)\hat{\boldsymbol{\beta}}+\mathbb{E}(\mathbf{D}_{cc,1}| \mathbf{T}_{cc})\]

_We name the imputation algorithm as SISNN (see Algorithm 4 in Appendix 0.B)._

### Sampling from Posterior and Predictive Distributions

To conduct multiple imputation in MISNN, we need to sample the parameters from the posterior distribution \(\rho_{2}\left(\boldsymbol{\beta},\sigma^{2}\Big{|}\mathbf{D}_{\text{obs},1}, \mathbf{X}_{\text{obs}},\mathbf{T}_{\text{obs}}\right)\) and the predictive
distribution \(\rho_{1}\left(\mathbf{D}_{\mathrm{miss},1}|\mathbf{X}_{\mathrm{miss}},\mathbf{T}_{ \mathrm{miss}},\hat{\mathbf{\beta}}^{(m)},\hat{\mathbf{\sigma}}^{(m)^{2}}\right)\) in MISNN (c.f. Algorithm 1). With the partialling out, we fit a linear regression at step (4),

\[\mathbf{D}_{\mathrm{obs},1}-\mathbb{E}(\mathbf{D}_{\mathrm{obs},1}|\mathbf{T}_ {\mathrm{obs}})=\left(\mathbf{X}_{\mathrm{obs}}-\mathbb{E}(\mathbf{X}_{ \mathrm{obs}}|\mathbf{T}_{\mathrm{obs}})\right)\mathbf{\beta}+\mathbf{\epsilon}\]

We approximate the posterior distribution of \(\mathbf{\beta},\sigma\) using

\[\rho_{2}\left(\mathbf{\beta},\sigma^{2}\Big{|}\mathbf{D}_{\mathrm{obs},1}, \mathbf{X}_{\mathrm{obs}},\mathbf{T}_{\mathrm{obs}}\right)=f_{1}\left(\mathbf{ \beta}\Big{|}\mathbf{D}_{\mathrm{obs},1},\mathbf{X}_{\mathrm{obs}},\mathbf{T}_ {\mathrm{obs}}\right)\times f_{2}\left(\sigma^{2}\Big{|}\mathbf{D}_{\mathrm{ obs},1},\mathbf{X}_{\mathrm{obs}},\mathbf{T}_{\mathrm{obs}}\right)\]

Suppose the OLS estimate for \(\mathbf{\beta}\) and its variance are \(\bar{\mathbf{\beta}}\) and \(\Sigma_{\mathbf{\beta}}\), respectively. We can approximate the distribution of \(\mathbf{\beta}\) by a normal distribution:

\[f_{1}\left(\mathbf{\beta}\Big{|}\mathbf{D}_{\mathrm{obs},1},\mathbf{X}_{\mathrm{ obs}},\mathbf{T}_{\mathrm{obs}}\right)\sim\mathcal{N}\left(\bar{\mathbf{\beta}},\Sigma_{\mathbf{ \beta}}\right)\]

where the parameters are defined as:

\[\bar{\mathbf{\beta}}=\mathrm{argmin}_{b}\|\mathbf{D}_{\mathrm{obs},1}-\eta_{D}( \mathbf{T}_{\mathrm{obs}})-[\mathbf{X}_{\mathrm{obs}}-\eta_{X}(\mathbf{T}_{ \mathrm{obs}})]\mathbf{b}\|^{2}\]

\[\Sigma_{\mathbf{\beta}}=\bar{\sigma}^{2}\left((\mathbf{X}_{\mathrm{obs}}-\eta_{X} (\mathbf{T}_{\mathrm{obs}})^{\top}(\mathbf{X}_{\mathrm{obs}}-\eta_{X}(\mathbf{ T}_{\mathrm{obs}}))\right)^{-1}\]

Here \(\bar{\sigma}^{2}\) can be estimated as the mean of squared residuals:

\[f_{2}\left(\sigma^{2}\Big{|}\mathbf{D}_{\mathrm{obs},1},\mathbf{X}_{\mathrm{ obs}},\mathbf{T}_{\mathrm{obs}}\right)=\left\|\mathbf{D}_{\mathrm{obs},1}-\eta_{D}( \mathbf{T}_{\mathrm{obs}})-(\mathbf{X}_{\mathrm{obs}}-\eta_{X}(\mathbf{T}_{ \mathrm{obs}}))\bar{\mathbf{\beta}}\right\|^{2}/n_{\mathrm{obs}}\]
As for drawing from the predictive distribution, we calculate \(\hat{\sigma}^{(m)}\) from \(f_{2}\) (with \(\bar{\beta}\) substituted by \(\hat{\mathbf{\beta}}^{(m)}\)). At last, we can draw \(\widehat{\mathbf{D}}_{\mathrm{miss},1}^{(m)}\) from

\[\rho_{1}\left(\mathbf{D}_{\mathrm{miss},1}|\mathbf{X}_{\mathrm{miss}},\mathbf{T }_{\mathrm{miss}},\hat{\mathbf{\beta}}^{(m)},\hat{\mathbf{\sigma}}^{(m)^{2}}\right)= \eta_{D}(\mathbf{T}_{\mathrm{miss}})+(\mathbf{X}_{\mathrm{miss}}-\eta_{X}( \mathbf{T}_{\mathrm{miss}}))\hat{\mathbf{\beta}}^{(m)}+\mathcal{N}(0,\hat{\mathbf{ \sigma}}^{(m)^{2}})\]

### Flexibility of MISNN Framework

Again, we highlight that the framework of MISNN is flexible in two folds: It can incorporate arbitrary feature selection method and arbitrary neural network models during imputation.

MISNN can incorporate an arbitrary feature selection method. Here, we adopt Lasso to select features \(\mathbf{X}=\mathbf{D}_{\mathcal{S}}\) and \(\mathbf{T}=\mathbf{D}_{-1}\setminus\mathbf{D}_{\mathcal{S}}\), where \(\mathcal{S}=\{i>0:\hat{\alpha}_{i}\neq 0\}\) comes from the non-zero part of lasso estimate

\[(\hat{\mathbf{\alpha}},\hat{\alpha}_{0})=\mathrm{argmin}_{\mathbf{a},a_{0}}\frac{1}{ 2}\|\mathbf{D}_{\mathrm{cc},1}-\mathbf{D}_{\mathrm{cc},-1}\mathbf{a}-a_{0}\|_{2}^ {2}+\lambda\|\mathbf{a}\|_{1}\]

MISNN works compatibly with all types of networks. Especially, when equipped with over-parameterized neural networks, MISNN can borrow the results from DebiNet [33, Theorem 1&2] to claim \(\sqrt{n}\)-consistency and exponentially fast convergence.

In practice, MISNN can work with a much richer class of neural networks than those theoretically supported in the neural tangent kernel regime [12, 1]. This includes the under-parameterized, moderately wide and deep neural networks. Empirical experiments shows that PLM learned by such neural networks exhibit strong prediction accuracy as well as post-selection inference (see Table 2).

### Other Properties of MISNN

Here we discuss some properties that MISNN enjoys, besides the flexibility of the framework, the consistent estimation of \(\mathbf{\beta}\) and the fast training of PLM aforementioned. Numerical evidence can be found in Section 5.

**Trainability:** MISNN can be trained by existing optimizers in an efficient manner, in comparison to Bayesian Lasso (which may require expensive burn-in period, see Table 3), boostrap methods (e.g. DURR, which needs many bootstrapped subsamples to be accurate) or MICE (which fits each feature iteratively and may be slow in high dimension).

**Robustness:** Empirically, MISNN is robust to hyper-parameter tuning (e.g. the width of hidden layers does not affect the performance much). From the data perspective, in high feature dimension and high missing rate (e.g. when compared to DURR, IURR and GAIN), MISNN still works reasonably well.

### MISNN for General Missing Patterns

The imputation procedure can be naturally extended to the case of general missing patterns, in which the pseudo code is provided in Algorithm 3 in the Appendix D. Suppose the first \(K\) columns are missing in \(\mathbf{D}\), denoted as \(\mathbf{D}_{\mathrm{full},[K]}\) and the \(k\)-th column is denoted by \(\mathbf{D}_{\mathrm{full},k}\). The set \(-[K]\) represents all other columns except those in \([K]\). Similar to the case of single column missingness, to construct a partial linear model, we need to partition the data into \(\mathbf{X}\) and \(\mathbf{T}\)
We fit regularized linear regression for each of the \(K\) columns that have missing values and obtained \(K\) active sets. Then we propose to use either intersection or union to combine the sets into a single one, which will be treated as \(\mathbf{X}\). To estimate the parameters \(\beta\), during each imputation, for the \(k\)-th column, we consider an OLS model that uses \(\mathbf{D}_{\mathrm{full},[K]}\) as regressors and the \(k\)-th column as response. Maximum likelihood techniques are adopted to generate regression coefficients \(\boldsymbol{\beta}_{k}\).

We remark that other proper feature selection methods and set-merging rules can be adopted to replace what we use. It's also possible that we use an iterative approach, following the idea of MICE, to conduct column-wise imputation. Generalization to the case of discrete missing values can be realized with the help of GPLM, which is similar to the discussion in Appendix 0.B.

## 5 Numerical Results

We compared MISNN with other state-of-the-art methods on various synthetic and real-world datasets. To establish baselines, we included complete data analysis, complete case analysis, and column mean imputation. We also evaluated two MI methods that incorporate regularized linear models for feature selection in high-dimensional settings: MICE-DURR and MICE-IURR. Additionally, we included MissForest, a MICE approach that uses random forest as the regression model, as well as GAIN, a deep-learning-based imputation method, and two matrix completion methods: SoftImpute and MMMF. More details about our experimental setup and results can be found in Appendix 0.C.

In addition to imputation accuracy, we evaluate the performance of imputation models in statistical inference that are based on im

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|c|} \hline Method & Style & Bias & Imp MSE & Coverage & Seconds & SE & SD \\ \hline Complete Data & - & **0.0027** & - & **0.954** & - & 0.1126 & 0.1150 \\ Complete Case & - & 0.1333 & - & 0.854 & - & 0.1556 & 0.1605 \\ Mean-Impute & SI & 0.1508 & 12.6215 & **0.994** & **0.005** & 0.3268 & 0.1933 \\ \hline MISNN-wide (Lasso) & MI & **-0.0184** & **4.2382** & **0.902** & **0.324** & 0.1438 & 0.1713 \\ MISNN-wide (ElasticNet) & MI & **-0.0134** & **4.2191** & **0.924** & **0.286** & 0.1431 & 0.1641 \\ MISNN-narrow (Lasso) & MI & **-0.0251** & 6.2666 & **0.944** & **0.370** & 0.1816 & 0.1755 \\ MISNN-narrow (ElasticNet) & MI & **-0.0246** & 6.2550 & **0.956** & **0.344** & 0.1818 & 0.1647 \\ MICE-DURR (Lasso) & MI & 0.1815 & 12.6704 & **0.978** & 1.266 & 0.2275 & 0.1196 \\ MICE-DURR (ElasticNet) & MI & 0.1314 & 10.8060 & **0.990** & 0.633 & 0.2241 & 0.1219 \\ MICE-IURR (Lasso) & MI & 0.2527 & 15.7803 & 0.886 & 1.483 & 0.2136 & 0.1150 \\ MICE-IURR (ElasticNet) & MI & 0.2445 & 15.3266 & 0.892 & 0.566 & 0.2153 & 0.1399 \\ MissForest & MI & 0.0579 & 9.6174 & **0.962** & 69.948 & 0.2851 & 0.2609 \\ GAIN & SI & 0.7578 & 27.3505 & 0.289 & 14.812 & 0.2869 & 0.4314 \\ SoftImpute & SI & -0.1432 & **4.6206** & 0.842 & **0.019** & 0.1804 & 0.2005 \\ MMMF & SI & -0.1239 & **4.0956** & 0.782 & 3.385 & 0.1491 & 0.1869 \\ \hline \end{tabular}
\end{table}
Table 1: Multi-feature missing pattern in synthetic data over 500 Monte Carlo datasets. Bias: mean bias \(\hat{\beta}_{1}-\beta_{1}\); Imp MSE: \(\|\widehat{\mathbf{D}}_{\mathrm{miss},1:3}-\mathbf{D}_{\mathrm{miss},1:3}\|^{2 }/n_{\mathrm{miss}}\); Coverage: coverage probability of the 95% confidence interval for \(\beta_{1}\); Seconds: wall-clock imputation time; SE: mean standard error of \(\hat{\beta}_{1}\); SD: Monte Carlo standard deviation of \(\hat{\beta}_{1}\). Model settings are in Section 5.1 and data generation is left in Appendix 0.C.2.
experiments, we specify a set of predictors and a response in the data matrix \(\mathbf{D}=(\mathbf{Z},y)\). A linear regression \(\hat{y}=\mathbf{Z}\hat{\boldsymbol{\theta}}\) is fitted using imputed dataset to predict \(y\) and we record the regression parameters \(\hat{\boldsymbol{\theta}}\). In synthetic datasets, we have access to the ground truth \(\boldsymbol{\theta}\), so we focus on inference performance. In real data analysis, we lose access to the true \(\boldsymbol{\theta}\) and focus on the prediction error instead.

### Viewpoint of Statistical Inference

In terms of the statistical inference, we consider four statistical quantities: bias of \(\hat{\boldsymbol{\theta}}\), coverage rate of the 95% confidence interval (CR) for \(\boldsymbol{\theta}\), mean standard error (SE) for \(\hat{\boldsymbol{\theta}}\) and Monte Carlo standard deviation (SD) of \(\hat{\boldsymbol{\theta}}\). Imputation mean squared error (MSE) is also compared. We study the performance of MISNN under general missing patterns, in which multiple columns (features) in the dataset can contain missing values. We adopt a similar experiment setting to that in [11] and evaluate performance over 500 Monte Carlo datasets. A detailed experiment description can be found in Appendix C.

Potentially, one can combine MICE with MISNN for single-column missingness as well. Nevertheless, we avoid doing so by proposing Algorithm 3 in Appendix D, which deals with the general missing patterns differently, in a parallel computing fashion. During the experiments, we use different network structures at step (3) of Algorithm 3: MISNN-wide uses two hidden layers with width 500, each followed by ReLU activation, a Batch Normalization layer [17] and a Dropout layer [29] at rate 0.1. The neural networks in MISNN-narrow are the same as in MISNN-wide, except the hidden layers have width 50 instead.

The results are summarized in Table 1. We highlight that all MISNN give the smallest estimation bias compared with the rest of imputation methods. MISNN also achieves satisfying imputation MSE, statistical coverage and compu

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline Method & Style & Estimatol & Imp & MSE & Seconds & SE & Pred MSE \\ \hline Complete Data & - & **0.0532** & - & - & 0.0676 & **0.8695** \\ Complete Case & - & 0.1278 & - & - & 0.1392 & 1.3376 \\ Mean-Impute & SI & -0.0374 & 1.3464 & **0.006** & 0.0686 & 0.8938 \\ \hline MISNN (Lasso) & MI & **0.0545** & 0.6620 & **1.501** & 0.0681 & **0.8780** \\ MISNN (ElasticNet) & MI & **0.0521** & **0.5140** & **0.861** & 0.0716 & **0.8789** \\ MICE-DURR (Lasso) & MI & **0.0504** & 1.8256 & 3.946 & 0.0508 & **0.8755** \\ MICE-DURR (ElasticNet) & MI & 0.0426 & 1.6998 & 2.709 & 0.0552 & 0.8817 \\ MICE-IURR (Lasso) & MI & 0.0474 & 2.0404 & 4.093 & 0.0476 & **0.8747** \\ MICE-IURR (ElasticNet) & MI & 0.0318 & 2.0219 & 2.620 & 0.0484 & 0.8803 \\ GAIN & SI & 0.0304 & 0.9902 & 67.432 & 0.0504 & **0.8749** \\ SoftImpute & SI & **0.0533** & 0.6667 & **0.0344** & 0.0763 & 0.8808 \\ MMMF & SI & 0.0833 & **0.3051** & 5.0261 & 0.0838 & **0.8755** \\ \hline \end{tabular}
\end{table}
Table 2: Multi-feature missing pattern in ADNI dataset over 100 repeats. Estimator: estimated \(\hat{\beta}_{1}\) through OLS using first 5 features as regressors; Imp MSE: imputation mean squared error \(\|\widehat{\mathbf{D}}_{\text{miss},1:3}-\mathbf{D}_{\text{miss},1:3}\|^{2}/n_{ \text{miss}}\); Seconds: wall-clock imputation time; SE: mean standard error of \(\hat{\beta}_{1}\); Pred MSE: mean squared error between \(\mathbf{A}\hat{\boldsymbol{\theta}}\) and \(\mathbf{y}\). Model settings are in Section 5.2 and data generation is left in Appendix C.3. MissForest is too slow (more than 5 min per dataset) to be considered.
tation speed. In comparison, two matrix completion methods achieve comparable imputation MSE, but their coverage is much worse than MI methods.

It is interesting to note that MISNN-wide tends to have smaller imputation MSE and estimation bias than MISNN-narrow. However, the coverage of the former is not as good as the latter, mainly due to the small SE. We suggest that in practice, if the accuracy of imputation or the parameter estimation is of main interest, MISNN with wide hidden layers should be adopted. If the statistical inference on parameters of interest is emphasized, then MISNN should be equipped with narrow hidden layers.

### Viewpoint of Prediction

We applied MISNN to the Alzheimer's Disease Neuroimaging Initiative (ADNI) gene dataset *, which includes over 19k genomic features for 649 patients and a response, VBM right hippocampal volume, ranging between [0.4,0.6]. We selected the top 1000 features with the largest correlations with the response, and focused on the linear analysis model between the response and the top 5 features. Since we did not have access to the true coefficients in the linear model, we studied the difference between the estimated coefficients from complete data analysis and the ones from imputed datasets. We artificially generated missing values under MAR in the top 3 features that had the largest correlations with the response, with a missing rate of approximately 65%. We used MISNN, containing a single hidden layer with width 500 and a Batch Normalization layer, and fit a linear regression between the response \(\mathbf{y}\) and the top five features \(\mathbf{D}_{1}\sim\mathbf{D}_{5}\) for downstream prediction.

Footnote *: The complete ADNI Acknowledgement is available at [http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf](http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf).

Our results, summarized in Table 2, show that MISNN achieved small imputation and prediction MSEs in a computationally efficient manner, particularly when compared to other MI methods. Additionally, the estimators by MISNN (as well as SoftImpute) were closest to the gold criterion from complete data analysis. Further experiment details can be found in Appendix C.

## 6 Discussion

In this work, we propose MISNN, a novel deep-learning based method for multiple imputation of missing values in tabular / matrix data. We demonstrate that MISNN can flexibly work with any feature selection and any neural network architecture. MISNN can be trained with off-the-shelf optimizers at high computation speed, providing interpretability for the imputation model, as well as being robust against data dimension and missing rate. Various experiments with synthetic and real-world datasets illustrate that MISNN significantly outperforms state-of-the-art imputation models.

While MISNN works for a wide range of analysis models, we have only discussed the case for continuous missing values using the partialling out. We can easily extend MISNN to discrete missing value problems by considering the generalized partially linear models (GPLM, see Section 4.1 for details). However,
the partialling out technique generally renders invalid for GPLM. Therefore, iterative methods including the backfitting, which can be slow, may be required to learn MISNN.

## 7 Acknowledgement

This work was supported in part by National Institutes of Health grant, R01GM124111. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
