# Combining datasets to improve model fitting

Thu Nguyen  Dept. of Holistic Systems
Simula Metropolitan
Oslo, Norway
<EMAIL_ADDRESS>Rabindra Khadka⋆ Dept. of Computer Science
Oslo Metropolitan University
Oslo, Norway
<EMAIL_ADDRESS>Nhan Phan ⋆ Dept. of Mathematics and Computer Science
University of Science
Vietnam National University in Ho Chi Minh City
Ho Chi Minh city, Vietnam
<EMAIL_ADDRESS>Anis Yazidi  Dept. of Computer Science
Oslo Metropolitan University
Oslo, Norway
<EMAIL_ADDRESS>Pål Halvorsen  Dept. of Holistic Systems
Simula Metropolitan
Oslo, Norway
<EMAIL_ADDRESS>Michael A. Riegler  Dept. of Holistic Systems
Simula Metropolitan
Oslo, Norway
<EMAIL_ADDRESS>
###### Abstract

For many use cases, combining information from different datasets can be of
interest to improve a machine learning model’s performance, especially when
the number of samples from at least one of the datasets is small. An
additional challenge in such cases is that the features from these datasets
are not identical, even though there are some commonly shared features among
the datasets. To tackle this, we propose a novel framework called Combine
datasets based on Imputation (ComImp). In addition, we propose PCA-ComImp, a
variant of ComImp that utilizes Principle Component Analysis (PCA), where
dimension reduction is conducted before combining datasets. This is useful
when the datasets have a large number of features that are not shared across
them. Furthermore, our framework can also be utilized for data preprocessing
by imputing missing data, i.e., filling in the missing entries while combining
different datasets. To illustrate the performance and practicability of the
proposed methods and their potential usages, we conduct experiments for
various tasks (regression, classification) and for different data types
(tabular data, time series data) when the datasets to be combined have missing
data. We also investigate how the devised methods can be used with transfer
learning to provide even further model training improvement. Our results
indicate that can provide extra improvement when being used in combination
with transfer learning.

###### Index Terms:

combining datasets, transfer learning, missing data, imputation

$\star$$\star$footnotetext: denotes equal contribution

##  I Introduction and Motivation

Machine learning algorithms usually improve as we collect more data. However,
there are times when the data is scarce, such as in medicine. It is possible
to combine different data sets in order to obtain more information. In
databases, a multi-table query is often used to combine data from multiple
tables. However, it requires the tables need to have the same columns. In
practice, however, datasets may have overlapping features but may not have
identical columns. In this work, we present two novel methods that efficiently
combine datasets that have overlapping features to solve similar or new tasks.
In addition, we are also investigating the impact of our method on transfer
learning.

As a motivating example, suppose that we have two datasets

$\displaystyle\small\mathcal{D}_{1}$
$\displaystyle=\begin{pmatrix}\text{person}&\text{height (cm)}&\text{weight
(kg)}&\text{BSL (mg/dl)}\\\ 1&120&80&80\\\ 2&150&70&90\\\ 3&140&80&85\\\
4&135&85&95\end{pmatrix},$ $\displaystyle\mathcal{D}_{2}$
$\displaystyle=\begin{pmatrix}\text{person}&\text{weight
(kg)}&\text{calo/meal}&\text{BSL (mg/dl)}\\\ 5&90&100&100\\\ 6&85&150&95\\\
7&92&170&82\end{pmatrix}.$

Here, the first two columns in $\mathcal{D}_{1}$ (height (cm), weight (kg))
form the input $\mathcal{X}_{1}$, and the last column (BSL(mg/dl)) is the
label $\mathbf{y}_{1}$. Similarly, the first two columns in $\mathcal{D}_{2}$
(weight (kg), calo/meal) form the input $\mathcal{X}_{2}$, and the last column
(BSL (mg/dl)) is the label $\mathbf{y}_{2}$. It is worth noticing that the
number of samples from $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ are different
and are collected from different individuals. Specifically, $\mathcal{D}_{1}$
is collected from individuals 1, 2, 3, 4, while $\mathcal{D}_{2}$ is collected
from individuals 5, 6, 7. Then, combining the information from these datasets
together can help increase the sample sizes and may allow more efficient model
training, prediction, and inferences. However, $\mathcal{X}_{1}$ and
$\mathcal{X}_{2}$ do not have the features height and calo/meal in common,
which creates a challenge in combining the two datasets together.

This motivates us to propose ComImp (Combine datasets based on Imputation), a
framework that allows vertically combine datasets based on missing data
imputation.The algorithm works by vertically stacking the datasets by
overlapping features (weight) and inserting columns of missing entries to a
component dataset (the datasets that are supposed to be merged) if the
corresponding features do not exist in the dataset but exist in some other
component dataset. Finally, the algorithm conducts imputation on the stacked
data.

Moreover, note that $\mathcal{D}_{1},\mathcal{D}_{2}$ have two features that
are not in common. However, in many scenarios, the number of features not in
common can be up to thousands. Then, simply using ComImp to combine datasets
can result in a lot of noise and bias. Therefore, in addition to ComImp, we
also propose PCA-ComImp, which is a variant of ComImp that incorporate
dimension reduction as a prior step to reduce the number of entries to be
imputed.

Further, it is common for data to have missing values due to instrument
malfunctioning, skipped questionnaires by participants, or data being lost at
different stages of the experiment. Since our methods use imputation to
combine datasets, it can also act as a preprocessing step by filling in the
missing entries and creating a complete dataset. In addition, our method can
also be used with data augmentation techniques to increase the sample size and
improve learning even further.

In summary, our contributions are as follows: (i) We propose a novel yet
simple framework, named ComImp, for combining datasets to improve model
fitting; (ii) We propose PCA-ComImp, a variant of ComImp, for combining
datasets with large numbers of features not in common; (iii) We analyze the
effects of imputation methods on the proposed frameworks; (iv) We illustrate
the advantage of the given methods when acting as preprocessing steps
(imputation) when the datasets to be combined have missing values; (v) We
demonstrate how ComImp can be used with transfer learning to provide even
better classification accuracy; and (vi) We provide various experiments to
show the power of the ComImp and PCA-ComImp.

The rest of this paper is organized as follows: in section II, we review some
related works. Next, in sections III and IV, we present the ComImp and PCA-
ComImp algorithms for combining datasets, respectively. Then, in section V, we
analyze some aspects of choosing an imputer for ComImp and PCA-ComImp.
Moreover, we illustrate the effectiveness and potential use of the proposed
methods in section VI. Lastly, we conclude the paper and provide potential
future works in section VII.

##  II Related Work

A lot of research has been done on utilizing information from related datasets
to improve model fitting. These approaches fall under the fields of Multi-task
learning (MTL), transfer learning, or dataset merging within a single data
modality.

MTL seeks to learn multiple tasks together by leveraging shared information
among tasks. The tasks generally have a high degree of similarities in MTL.
This helps address the issue of the limited availability of labeled data. The
scarcely available labeled datasets, even if collected for different tasks,
can be aggregated for multi-task learning purposes. One of the reasons that
MTL performs better than the single-task learning paradigm is the effective
learning of more general representation from the datasets.A MTL model learns
useful representations so that it is able to perform well for a new task that
is closely related to the previously learned task. The MTL models are broadly
categorized as feature-based approaches and parameter-based approaches. The
feature-based approach is based on the idea of sharing information. It seeks
to learn common representation features among all the tasks [14]. The
parameter-based approach is motivated by the notion of linking among tasks by
placing priors on model parameters. Various methods have been proposed while
sharing the parameters, such as task clustering method [10], and decomposition
method [35].

Another way to make use of information from related datasets is to perform
transfer learning. The idea of transfer learning is to transfer the weights
learned from a pre-trained model and solve another task without training a
model from scratch. Transfer learning is particularly useful in scenarios with
a small training dataset [22, 2]. Transfer learning has achieved state-of-the-
art performance in domains like natural language processing [34, 26] and
natural images [8]. For example, the study by [4] adopts transfer learning to
use a large number of structurally incomplete data samples to improve the
accuracy of fault diagnosis. Next, the work by [5] applies transfer learning
for emotion recognition from speech where features are transferred from
multiple labeled speech sources. Another work by [33] proposes a novel method
to impute missing gene expression data from DNA methylation data through a
transfer learning–based neural network. On the other hand, for skin analysis,
TATL [16] is a segmentation framework that detects lesion skin regions and
then transfers this knowledge to a set of attribute-specific classifiers to
detect each particular attribute, which shows better results compared to
initialization using weights pre-trained on ImageNet. Even though transfer
learning is widely used due to its efficiency, it cannot be used when deep
learning is not applicable, such as when the number of features is bigger than
the number of samples. Therefore, in such cases, combining datasets is more
appropriate.

For the third direction, some works in combining datasets exist. For example,
in business intelligence, the work reported in [12] tries to customize an
analytic dataset at hand with additional features from other datasets by a
model that utilizes semantic relationships to deal with the problem. Another
example is [25], which presents a transfer learning method for merging brain-
computer interface datasets with different dimensionalities for business
analytic data. Next, [31] proposes a pipeline for imputing and combining
genomic datasets, where the merge is conducted based on the set of overlapping
SNPs. Note that our work should not be confused with this work since our
frameworks use imputation for combining the datasets, while [31] uses
imputation as a preprocessing step. Next, [9] proposes a method based on
multiple imputations for combining two confidential datasets for sharing
without revealing confidential information.

Even though multi-task learning methods may help improve each task’s learning
performance, the sizes between the datasets can be too significant to apply
the same type of model. For example, if one dataset has 20 features and 100
samples while the other has 25 features and 1000 samples. Then, linear
regression may be suitable for the first one, and a neural network may be more
efficient for the second one. This creates challenges in using multi-task
learning for these two datasets.

These factors make it even more necessary to combine datasets vertically, and
ComImp is suitable for that purpose.

##  III ComImp algorithm for combining datasets

Algorithm 1 ComImp algorithm

Input:

  1. 1.

datasets $\mathcal{D}_{i}=\\{\mathcal{X}_{i},\mathbf{y}_{i}\\},i=1,...,r$

  2. 2.

$\mathcal{F}_{i}$: set of features in $\mathcal{X}_{i}$

  3. 3.

$g(\mathcal{X},\mathcal{H})$: a transformation that rearranges features in
$\mathcal{X}$ to follow the order of the features in the set of features
$\mathcal{H}$ and insert empty columns if a feature in $\mathcal{H}$ does not
exist in $\mathcal{X}$

  4. 4.

Imputer $I$.

Procedure:

1:$\mathcal{F}=\bigcup_{i=1}^{r}\mathcal{F}_{i}$

2:$\mathcal{X}_{i}^{*}\leftarrow g(\mathcal{X}_{i},\mathcal{F})$

3:$\mathcal{X}^{*}=\begin{pmatrix}\mathcal{X}_{1}^{*}\\\
\mathcal{X}_{2}^{*}\\\ \vdots\\\
\mathcal{X}_{r}^{*}\end{pmatrix},\mathbf{y}=\begin{pmatrix}\mathbf{y}_{1}\\\
\mathbf{y}_{2}\\\ \vdots\\\ \mathbf{y}_{r}\end{pmatrix}$

4:$\mathcal{X}\leftarrow$ imputed version of $\mathcal{X}^{*}$ using imputer
I.

Return: $\mathcal{D}=\\{\mathcal{X},\mathbf{y}\\}$, the combined dataset of
$\mathcal{D}_{1},...,\mathcal{D}_{r}$.

This section details our ComImp framework for combining datasets. The general
ComImp framework is shown in Algorithm 1. Suppose that we have datasets
$\mathcal{D}_{1}=\\{\mathcal{X}_{1},\mathbf{y}_{1}\\}$,…,
$\mathcal{D}_{r}=\\{\mathcal{X}_{r},\mathbf{y}_{r}\\}$. In addition, assume
that the $(p_{1}+i)^{th}$ column of $\mathcal{D}_{1}$ and $i^{th}$ column of
$\mathcal{D}_{2}$ refer to the same feature. Let $\mathcal{F}$ be the union of
the sets of features in $\mathcal{X}_{1},\mathcal{X}_{2},...,\mathcal{X}_{r}$
in an ordered way (step 1 of the algorithm). Then, in step 2 of the algorithm,
we rearrange features in each $\mathcal{X}_{i}$ to follow the order of the
features in $\mathcal{F}$ and insert an empty column if a feature in
$\mathcal{F}$ does not exist in $\mathcal{X}_{i}$. This results in
$\mathcal{X}_{i}^{*}$ for each $i=1,...,r$. Next, in step 3, we stack
$\mathcal{X}_{i},i=1,..,r$ vertically and do similarly for the corresponding
labels to get $\mathcal{X}^{*}$ and $\mathbf{y}$. Finally, we impute the data
$\mathcal{X}^{*}$ to fill in these entries and create a united, complete
dataset.

As an illustration, we use the datasets $\mathcal{D}_{1},\mathcal{D}_{2}$ in
Section I. A newly merged dataset $\mathcal{D}$ can be formed by the following
process:

  1. 1.

Step 1: The union of set of features $\mathcal{F}$ consists of height, weight,
and calo/meal,

  2. 2.

Step 2:

$\displaystyle\small\mathcal{X}^{*}_{1}$
$\displaystyle=\begin{pmatrix}\text{person}&\text{height}&\text{weight}&\text{(calo/meal)}\\\
1&120&80&*\\\ 2&150&70&*\\\ 3&140&80&*\\\ 4&135&85&*\end{pmatrix},$
$\displaystyle\mathcal{X}^{*}_{2}$
$\displaystyle=\begin{pmatrix}\text{person}&\text{height}&\text{weight}&\text{calo/meal}\\\
5&*&90&100\\\ 6&*&85&150\\\ 7&*&92&170\end{pmatrix},$

  3. 3.

Stack $\mathcal{X}_{1}^{*},\mathcal{X}_{2}^{*}$ vertically, and stack
$\mathbf{y}_{1},\mathbf{y}_{2}$ vertically

$\small\mathcal{X}^{*}=\begin{pmatrix}\text{person}&\text{height}&\text{weight}&\text{calo/meal}\\\
1&120&80&*\\\ 2&150&70&*\\\ 3&140&80&*\\\ 4&135&85&*\\\ 5&*&90&100\\\
6&*&85&150\\\ 7&*&92&170\end{pmatrix},$

and

$\mathbf{y}=\begin{pmatrix}80\\\ 90\\\ 85\\\ 95\\\ 100\\\ 95\\\
82\end{pmatrix}$

  4. 4.

$\mathcal{X}=$ imputed version of $\mathcal{X}^{*}$. Suppose that mean
imputation is being used, then

$\small\mathcal{X}=\begin{pmatrix}\text{(person)}&\text{(height)}&\text{(weight)}&\text{(calo/meal)}\\\
1&120&80&140\\\ 2&150&70&140\\\ 3&140&80&140\\\ 4&135&85&140\\\
5&136.25&90&100\\\ 6&136.25&85&150\\\ 7&136.25&92&170\end{pmatrix},$

  5. 5.

$\mathcal{D}=(\mathcal{X},\mathbf{y}).$

##  IV Combining datasets with dimension reduction (PCA-ComImp)

In many scenarios, the number of features that is available in one dataset but
not available in other datasets may be very large. Therefore, if ComImp is
applied directly to those datasets, the imputation process may introduce too
many noises and biases into the combined dataset that it may suppress the
advantages of combining datasets. Then, it would be preferable to conduct
dimension reduction on the non-overlapping features of the datasets to reduce
the number of values that must be imputed. This is not only to reduce the bias
introduced by imputation to the model but also to speed up the imputation
process. To start, we adapt similar notations to [18], let $pca(A)$ be a
function of a data matrix $A$. The function returns $(\mathcal{R}_{A},V)$
where $\mathcal{R}_{A}$ is the PCA-reduced version of $A$, and $V$ is the
projection matrix where the $i^{th}$ column of $V$ is the eigenvector
corresponding to the $i^{th}$ largest eigenvalue. To simplify notations, we
also sometimes use the notation $\mathcal{F}$ that refers to a set of features
to indicate a subset of a dataset that contains only these features. Then, we
have Algorithm 2 for combining two datasets with dimension reduction.

The algorithm starts by finding the union, intersection, and complements of
feature sets as in step 1 of the procedure. Note that
$\mathcal{R}_{1}=\mathcal{F}_{1}\setminus\mathcal{F}_{2}$ means that
$\mathcal{R}_{1}$ is the set of features that belong to $\mathcal{D}_{1}$ but
not $\mathcal{D}_{2}$. Similar for $\mathcal{R}_{2}$. Then, in the second
step, the procedure uses PCA to find the reduced version and the projection
matrix $V$ of $\mathcal{S}_{i}^{(tr)}$ (the subset of the training portion of
$\mathcal{D}_{i}$ that consists of only the features $\mathcal{S}_{i}$). In
step 3, we find the set $\mathcal{H}$ of the union of features between the
common features of $\mathcal{F}_{1},\mathcal{F}_{2}$, the set of reduced
features of the features that are available in $\mathcal{F}_{1}$ but not in
$\mathcal{F}_{2}$, and the set of reduced features of the features that are
available in $\mathcal{F}_{2}$ but not in $\mathcal{F}_{1}$. Next, similar to
ComImp, we create $\mathcal{X}_{i}^{*}$ in step 4, stack them together in step
5, and impute them in step 6.

To combine multiple datasets that have many features not in common, one can
operate sequentially. For example, if we have
$\mathcal{D}_{1},...,\mathcal{D}_{r}$ then we can first combine
$\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ to get $\mathcal{D}_{12}$, and then
merge $\mathcal{D}_{12}$ with $\mathcal{D}_{3}$ to get $\mathcal{D}_{123}$ and
so on. However, attention should be paid to the number of entries to be
imputed. Otherwise, the noise and biases introduced by imputing too many
missing values may surpass the benefits of combining information.

Algorithm 2 PCA-ComImp

Input:

  1. 1.

Datasets $\mathcal{D}_{i}=\\{\mathcal{X}_{i},\mathbf{y}_{i}\\},i=1,2$ consists
of
$\mathcal{D}_{i}^{(tr)}=\\{\mathcal{X}_{i}^{(tr)},\mathbf{y}_{i}^{(tr)}\\},i=1,2$
as the training sets and
$\mathcal{D}_{i}^{(ts)}=\\{\mathcal{X}_{i}^{(ts)},\mathbf{y}_{i}^{(ts)}\\}$ as
the test sets,

  2. 2.

$\mathcal{F}_{i}$: set of features in $\mathcal{X}_{i}$,

  3. 3.

$g(\mathcal{X},\mathcal{H})$: a transformation that rearranges features in
$\mathcal{X}$ to follow the order of the features in the set of features
$\mathcal{H}$ and insert empty columns if a feature in $\mathcal{H}$ does not
exist in $\mathcal{X}$

  4. 4.

Imputer $I$, classifier $\mathcal{C}$.

Procedure:

1:$\mathcal{F}=\mathcal{F}_{1}\cup\mathcal{F}_{2},\;\mathcal{S}=\mathcal{F}_{1}\cap\mathcal{F}_{2},\;\mathcal{Q}_{1}=\mathcal{F}_{1}\setminus\mathcal{F}_{2},\;\mathcal{Q}_{2}=\mathcal{F}_{2}\setminus\mathcal{F}_{1}$

2:$(\mathcal{R}_{i}^{(tr)},V)\leftarrow pca(\mathcal{Q}_{i}^{(tr)})$ and
$\mathcal{R}_{i}^{(ts)}\leftarrow\mathcal{Q}_{i}^{(ts)}V$

3:$\mathcal{H}=\mathcal{S}\cup\mathcal{R}_{1}\cup\mathcal{R}_{2}$

4:$\mathcal{X}_{i}^{*}\leftarrow
g(\mathcal{S}\cup\mathcal{R}_{i},\mathcal{H}),i=1,2$

5:$\mathcal{X}^{*}=\begin{pmatrix}\mathcal{X}_{1}^{*}\\\
\mathcal{X}_{2}^{*}\end{pmatrix},\mathbf{y}=\begin{pmatrix}\mathbf{y}_{1}\\\
\mathbf{y}_{2}\end{pmatrix}$

6:$\mathcal{X}\leftarrow$ imputed version of $\mathcal{X}^{*}$ using imputer
I.

Return: $\mathcal{D}=\\{\mathcal{X},\mathbf{y}\\}$, the combined dataset of
$\mathcal{D}_{1},\mathcal{D}_{2}$.

##  V Choices of missing data imputation methods

Even though there are many imputation methods that one can use, several
factors should be considered when choosing the imputer for ComImp, such as
speed, imputation quality, and the ability to predict sample by sample. When
the datasets to be combined are small, methods that are slow but can give
promising imputation quality, such as kNN, MICE [1], missForest [27], can be
considered. However, these methods can be too slow for moderate large-size
[19], and therefore cannot be used for high dimensional data. Next, matrix
decomposition methods may not be suitable in many scenarios, such as when the
data is not of low rank. In addition, since the imputation is done via matrix
factorization, those methods are not suitable for imputation in online
learning, which requires the handling of each sample as it comes [20].
Moreover, when the uncertainty of imputed values is of interest, then Bayesian
techniques can be used. However, Bayesian imputation techniques can be slow.
Recently, DIMV [32] was introduced as a scalable imputation method that
estimates the conditional distribution of the missing values based on a subset
of observed entries. As a result, the method provides an explainable
imputation, along with the confident region, in a simple and straight forward
manner.

Note that the choice of imputation methods can affect the performance of the
model trained on the merged dataset as well. An example can be seen via the
following theorem

###### Theorem 1

Assume that we have two datasets
$\mathcal{D}_{1}=\\{\mathbf{U},\mathbf{y}\\},\mathcal{D}_{2}=\\{\mathbf{V},\bm{z}\\}$
where $\mathbf{U}$ , $\mathbf{V}$ are inputs, and $\mathbf{y},\bm{z}$ are the
labels, such that

$\displaystyle\mathbf{U}$
$\displaystyle=(\mathbf{1}_{n}\;\;\mathbf{u}_{1})=\begin{pmatrix}1&u_{11}\\\
1&u_{21}\\\ \vdots&\vdots\\\ 1&u_{n1}\end{pmatrix},$ (1)
$\displaystyle\mathbf{V}$
$\displaystyle=(\mathbf{1}_{m}\;\;\mathbf{v}_{1}\;\;\mathbf{v}_{2})=\begin{pmatrix}1&v_{11}&v_{12}\\\
1&v_{21}&v_{22}\\\ \vdots&\vdots&\vdots\\\ 1&v_{m1}&v_{m2}\end{pmatrix},$ (2)

where $u_{ij},v_{ij}\in\mathbb{R}$, and

$\mathbf{y}=\begin{pmatrix}y_{1}\\\ y_{2}\\\ \vdots\\\
y_{n}\end{pmatrix},\bm{z}=\begin{pmatrix}z_{1}\\\ z_{2}\\\ \vdots\\\
z_{m}\end{pmatrix},\mathbf{Y}=\begin{pmatrix}\mathbf{y}\\\
\bm{z}\end{pmatrix}.$ (3)

Next, let $\bar{\mathbf{v}}_{2}$ be the mean of $\mathbf{v}_{2}$. If mean
imputation is being used for ComImp then the resulting combined input is

$\mathbf{X}=\begin{pmatrix}1_{n}&\mathbf{u}_{1}&\bar{\mathbf{v}}_{2}\\\
1_{m}&\mathbf{v}_{1}&\mathbf{v}_{2}\end{pmatrix}.$ (4)

Then, we have the following relation between the sum of squared errors ($SSE$)
of the model fitted on $\mathcal{D}=\\{\mathbf{X},\mathbf{Y}\\}$,

$SSE_{\mathcal{D}}\geq SSE_{\mathcal{D}_{1}}+SSE_{\mathcal{D}_{2}}.$ (5)

###### Proof 1

Note that equation (5) is equivalent to

$\mathbf{Y}^{\prime}(I-\mathbf{H}_{x})\mathbf{Y}\leq\mathbf{y}^{\prime}(I-\mathbf{H}_{u})\mathbf{y}+\mathbf{z}^{\prime}(I-\mathbf{H}_{v})\mathbf{z},$
(6)

where

$\displaystyle\mathbf{H}_{u}$
$\displaystyle=\mathbf{U}(\mathbf{U}^{\prime}\mathbf{U})^{-1}\mathbf{U}^{\prime},$
(7) $\displaystyle\mathbf{H}_{v}$
$\displaystyle=\mathbf{V}(\mathbf{V}^{\prime}\mathbf{V})^{-1}\mathbf{V}^{\prime},$
(8) $\displaystyle\mathbf{H}_{x}$
$\displaystyle=\mathbf{X}(\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}.$
(9)

Hence, we want to prove that

$\mathbf{y}^{\prime}\mathbf{H}_{u}\mathbf{y}+\bm{z}^{\prime}\mathbf{H}_{v}\bm{z}\geq\mathbf{Y}^{\prime}\mathbf{H}_{x}\mathbf{Y},$
(10)

Without loss of generality, assume that the data is centered so that
$\sum\limits_{i=1}^{n}u_{i1}=0$, $\sum\limits_{i=1}^{n}v_{i1}=0$,
$\sum\limits_{i=1}^{n}v_{i2}=0$, so $\bar{\mathbf{v}}_{2}=0$. Next, let
$\alpha=\sum\limits_{i=1}^{n}y_{i}$,
$\alpha_{1}=\sum\limits_{i=1}^{n}y_{i}u_{i1}$,
$a_{1}=\sum\limits_{i=1}^{n}u_{i1}^{2}$, we have

$\displaystyle\mathbf{y}^{\prime}\mathbf{U}$
$\displaystyle=\begin{pmatrix}\sum\limits_{i=1}^{n}y_{i}&\sum\limits_{i=1}^{n}y_{i}u_{i1}\end{pmatrix}=\begin{pmatrix}\alpha&\alpha_{1}\end{pmatrix},$
$\displaystyle\mathbf{U}^{\prime}\mathbf{U}$
$\displaystyle=\begin{pmatrix}1&\sum\limits_{i=1}^{n}u_{i1}\\\
\sum\limits_{i=1}^{n}u_{i1}&\sum\limits_{i=1}^{n}u_{i1}^{2}\end{pmatrix}=\begin{pmatrix}n&0\\\
0&a_{1}\end{pmatrix},$ $\displaystyle\mathbf{U}^{\prime}\mathbf{y}$
$\displaystyle=\begin{pmatrix}\sum\limits_{i=1}^{n}y_{i}\\\
\sum\limits_{i=1}^{n}y_{i}u_{i1}\end{pmatrix}=\begin{pmatrix}\alpha\\\
\alpha_{1}\end{pmatrix}.$

This implies

$\displaystyle\mathbf{y}^{\prime}\mathbf{H}_{u}\mathbf{y}$
$\displaystyle=\mathbf{y}^{\prime}\left(\mathbf{U}(\mathbf{U}^{\prime}\mathbf{U})^{-1}\mathbf{U}^{\prime}\right)\mathbf{y}=\left(\mathbf{y}^{\prime}\mathbf{U}\right)\left(\mathbf{U}^{\prime}\mathbf{U}\right)^{-1}\left(\mathbf{U}^{\prime}\mathbf{y}\right)$
$\displaystyle=\frac{\alpha^{2}}{n}+\frac{\alpha_{1}^{2}}{a_{1}}.$ (11)

Similarly, let $\beta=\sum\limits_{i=1}^{m}z_{i}$,
$\beta_{1}=\sum\limits_{i=1}^{m}z_{i}v_{i1}$,
$\beta_{2}=\sum\limits_{i=1}^{m}z_{i}v_{i2}$,
$b_{1}=\sum\limits_{i=1}^{m}v_{i1}^{2}$,
$b_{1}=\sum\limits_{i=1}^{m}v_{i2}^{2}$,
$c=\sum\limits_{i=1}^{m}v_{i1}v_{i2}$. Then,

$\displaystyle\mathbf{z}^{\prime}\mathbf{V}$
$\displaystyle=\begin{pmatrix}\sum\limits_{i=1}^{m}z_{i}&\sum\limits_{i=1}^{m}z_{i}v_{i1}&\sum\limits_{i=1}^{m}z_{i}v_{i12}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}\beta&\beta_{1}&\beta_{2}\end{pmatrix},$
$\displaystyle\mathbf{V}^{\prime}\mathbf{V}$
$\displaystyle=\begin{pmatrix}m&\sum\limits_{i=1}^{m}v_{i1}&\sum\limits_{i=1}^{m}v_{i2}\\\
\sum\limits_{i=1}^{m}v_{i1}&\sum\limits_{i=1}^{m}v_{i1}^{2}&\sum\limits_{i=1}^{m}v_{i1}v_{i2}\\\
\sum\limits_{i=1}^{m}v_{i2}&\sum\limits_{i=1}^{m}v_{i1}v_{i2}&\sum\limits_{i=1}^{m}v_{i2}^{2}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}m&0&0\\\ 0&b_{1}&c\\\ 0&c&b_{2}\end{pmatrix},$
$\displaystyle\mathbf{V}^{\prime}\mathbf{z}$
$\displaystyle=\begin{pmatrix}\sum\limits_{i=1}^{m}z_{i}\\\
\sum\limits_{i=1}^{m}z_{i}v_{i1}\\\
z_{i}v_{i12}\end{pmatrix}=\begin{pmatrix}\beta\\\ \beta_{1}\\\
\beta_{2}\end{pmatrix}.$

Therefore,

$\displaystyle\mathbf{z}^{\prime}\mathbf{H}_{z}\mathbf{z}$
$\displaystyle=\mathbf{z}^{\prime}\left(\mathbf{V}(\mathbf{V}^{\prime}\mathbf{V})^{-1}\mathbf{V}^{\prime}\right)\mathbf{z}=\left(\mathbf{z}^{\prime}\mathbf{V}\right)\left(\mathbf{V}^{\prime}\mathbf{V}\right)^{-1}\left(\mathbf{V}^{\prime}\mathbf{z}\right)$
$\displaystyle=\frac{\beta^{2}}{m}+\frac{\beta_{1}^{2}b_{2}+\beta_{2}^{2}b_{1}-2\beta_{1}\beta_{2}c}{b_{1}b_{2}-c^{2}}$
$\displaystyle=\frac{\beta^{2}}{m}+\frac{\left(\beta_{1}b_{2}-\beta_{2}c\right)^{2}}{b_{2}\left(b_{1}b_{2}-c^{2}\right)}+\frac{\beta_{2}^{2}}{b_{2}}.$
(12)

Besides,

$\displaystyle\mathbf{Y}^{\prime}\mathbf{X}$
$\displaystyle=\begin{pmatrix}\sum\limits_{i=1}^{n}y_{i}+\sum\limits_{i=1}^{m}z_{i}&\sum\limits_{i=1}^{n}y_{i}u_{i1}+\sum\limits_{i=1}^{m}z_{i}v_{i1}&\sum\limits_{i=1}^{m}z_{i}v_{i12}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}\alpha+\beta&\alpha_{1}+\beta_{1}&\beta_{2}\end{pmatrix},$
$\displaystyle\mathbf{X}^{\prime}\mathbf{X}$
$\displaystyle=\begin{pmatrix}m&\sum\limits_{i=1}^{n}u_{i1}+\sum\limits_{i=1}^{m}v_{i1}&\sum\limits_{i=1}^{m}v_{i2}\\\
\sum\limits_{i=1}^{n}u_{i1}+\sum\limits_{i=1}^{m}v_{i1}&\sum\limits_{i=1}^{n}u_{i1}^{2}+\sum\limits_{i=1}^{m}v_{i1}^{2}&\sum\limits_{i=1}^{m}v_{i1}v_{i2}\\\
\sum\limits_{i=1}^{m}v_{i2}&\sum\limits_{i=1}^{m}v_{i1}v_{i2}&\sum\limits_{i=1}^{m}v_{i2}^{2}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}m&0&0\\\ 0&a_{1}+b_{1}&c\\\
0&c&b_{2}\end{pmatrix},$ $\displaystyle\mathbf{X}^{\prime}\mathbf{Y}$
$\displaystyle=\begin{pmatrix}\sum\limits_{i=1}^{n}y_{i}+\sum\limits_{i=1}^{m}z_{i}\\\
\sum\limits_{i=1}^{n}y_{i}u_{i1}+\sum\limits_{i=1}^{m}z_{i}v_{i1}\\\
\sum\limits_{i=1}^{m}z_{i}v_{i12}\end{pmatrix}=\begin{pmatrix}\alpha+\beta\\\
\alpha_{1}+\beta_{1}\\\ \beta_{2}\end{pmatrix}.$

Hence,

$\displaystyle\mathbf{Y}^{\prime}\mathbf{H}_{x}\mathbf{Y}$
$\displaystyle=\mathbf{Y}^{\prime}\left(\mathbf{X}(\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}\right)\mathbf{Y}=\left(\mathbf{Y}^{\prime}\mathbf{X}\right)\left(\mathbf{X}^{\prime}\mathbf{X}\right)^{-1}\left(\mathbf{X}^{\prime}\mathbf{Y}\right)$
$\displaystyle=\frac{\left(\alpha+\beta\right)^{2}}{m}+\frac{\left[\alpha_{1}b_{2}+\beta_{1}b_{2}-\beta_{2}c\right]^{2}}{b_{2}\left[a_{1}b_{2}+b_{1}b_{2}-c^{2}\right]}+\frac{\beta_{2}^{2}}{b_{2}}.$
(13)

Now, applying Cauchy-Schwarz inequality, we have

$\frac{\alpha^{2}}{n}+\frac{\beta^{2}}{m}\geq\frac{\left(\alpha+\beta\right)^{2}}{m+n},$
(14)
$\displaystyle\frac{\alpha_{1}^{2}}{a_{1}}+\frac{\left(\beta_{1}b_{2}-\beta_{2}c\right)^{2}}{\beta_{2}\left(b_{1}b_{2}-c^{2}\right)}$
$\displaystyle=\frac{1}{b_{2}}\left[\frac{\alpha_{1}^{2}b_{2}^{2}}{a_{1}b_{2}}+\frac{\left(\beta_{1}b_{2}-\beta_{2}c\right)^{2}}{b_{1}b_{2}-c^{2}}\right]$
$\displaystyle\geq\frac{\left(\alpha_{1}b_{2}+\beta_{1}b_{2}-\beta_{2}c\right)^{2}}{b_{2}\left(a_{1}b_{2}+b_{1}b_{2}-c^{2}\right)}.$
(15)

From (1), (1), (1), (14) and (1), we have
$\mathbf{y}^{\prime}\mathbf{H}_{u}\mathbf{y}+\bm{z}^{\prime}\mathbf{H}_{v}\bm{z}\geq\mathbf{Y}^{\prime}\mathbf{H}_{x}\mathbf{Y},$
as desired.

From the theorem, we can see that in the given setting stated in the theorem,
the SSE of the regression model fitted on the merged data is bigger than the
sum of the SSE of the models trained on each component dataset. Hence, using
mean imputation in this case actually leads to worse performance.

##  VI Experiments

In this section, we present various experiments on a variety of scenarios to
explore the potential of the proposed techniques. Throughout all the following
experiments, if not further commented, we use soft-impute [13] as the
imputation method, each experiment is repeated 10000 times before reporting
the mean and variance of the results, and the train and test set are split
with ratio 1:1. The descriptions of the datasets used are as in Table I. The
codes for the experiments are available at github.com/thunguyen177/ComImp.

TABLE I: Descriptions of datasets used in the experiments Dataset | # Classes | # Features | # Samples
---|---|---|---
Seed [3] | 3 | 7 | 210
Wine | 3 | 13 | 178
Epileptic Seizure ([23]) | 2 | 178 | 11,500
Gene | $5$ | $20531$ | $801$

###  VI-A Regression simulation

TABLE II: mean $\pm$ variance of MSE on test sets of regression models fitted on $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$, respectively. model | $f_{1}$ | $f_{2}$ | $f$
---|---|---|---
MSE | 3.327 $\pm$ 0.093 | 3.044 $\pm$ 0.131 | 0.734 $\pm$ 0.002

#### Data generation

We run a Monte Carlo experiment for regression with 10000 repeats. For each
loop, we generate a dataset $\mathcal{D}_{1}$ of 300 samples, and a dataset
$\mathcal{D}_{2}$ of 200 samples based on the following relation

$Y=1+X_{1}+0.5X_{2}+X_{3}+\epsilon$ (16)

where $\mathbf{X}=(X_{1},X_{2},X_{3})^{T}$ follows a multivariate Gaussian
distribution with the following mean and covariance matrix

$\bm{\mu}=\begin{pmatrix}1\\\ 2\\\
0.5\end{pmatrix},\mathbf{\Sigma}=\begin{pmatrix}1&0.5&0.3\\\ 0.5&1&0.4\\\
0.3&0.4&1\end{pmatrix}$ (17)

and $\epsilon\sim\mathcal{N}(0,0.2\,I)$, where $I$ is the identity matrix.

Then, we delete the first feature of $\mathcal{D}_{1}$ and the second feature
of $\mathcal{D}_{2}$. To make it more similar to practical settings, where
there the data collecting machines/sensors can also be the source of
variation, we added Gaussian noise with variance $0.05$ to the second feature
and Gaussian noise with variance $0.1$ to the third feature of
$\mathcal{D}_{2}$. Then, $\mathcal{D}_{1},\mathcal{D}_{2}$ is split again into
training and testing sets ratio 7:3, respectively.

Experimental setup and results.  For each simulation, after generating the
data, we use ComImp to merge the training sets of $\mathcal{D}_{1}$ and
$\mathcal{D}_{2}$ into the corresponding training set of $\mathcal{D}$. We do
similarly for the test sets. Next, we fit a regression model on each training
set of $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$, which gives fitted
models $f_{1},f_{2},f$, respectively. We record the MSE of $f_{1},f_{2},f_{3}$
on the corresponding test set of
$\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$, respectively. Note that the
test set of $\mathcal{D}$ is the merge between the test sets of
$\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. Then, we report the mean and variance
of the mean square error (MSE) on test sets of 10000 runs in Table II.

Analysis.  From Table II, we see that the MSE of $f$ (the model fitted on
merged training parts of $\mathcal{D}_{1},\mathcal{D}_{2}$) on the test of
$\mathcal{D}$ is significantly lower than $f_{1}$ and $f_{2}$. Also, recall
that the test set of $\mathcal{D}$ is the merge between the test sets of
$\mathcal{D}_{1}$ and $\mathcal{D}_{2}$, and the labels are stacked
accordingly. This shows the power of merging datasets via ComImp for
regression.

###  VI-B Experiments of merging two datasets

#### Data

We conduct experiments on the Seed dataset. We delete the first two columns of
the input in $\mathcal{D}_{1}$ and split it into training and testing sets of
equal sizes. In addition, we delete the last columns of the input data in
$\mathcal{D}_{2}$ and split it into the training and testing sets of equal
sizes. We fit models $f_{1},f_{2}$ on the training set of
$\mathcal{D}_{1},\mathcal{D}_{2}$, respectively. In addition, we use ComImp to
combine the training sets of $\mathcal{D}_{1},\mathcal{D}_{2}$ to the training
sets of $\mathcal{D}$, and do similarly for the testing portions. Then, we fit
a model $f$ on the training set of $\mathcal{D}$. We repeat the experiment
1000 times and report the mean and variance of the accuracy in Table III.

#### Analysis

Note that in Table III, for $f$, the third (fifth) column contains the results
on the portion of the test set of $\mathcal{D}$ that corresponds to the test
set of $\mathcal{D}_{1}(\mathcal{D}_{2})$. From Table III, we can see that the
performance of the model fitted on the merged data improves significantly for
$\mathcal{D}_{2}$, the smaller dataset, and slightly for $\mathcal{D}_{1}$.
Specifically, when using Logistic Regression, the accuracy of $f$ on
$\mathcal{D}_{2}$ is 0.896, which is 3.8% higher than $f_{2}$, whose accuracy
is 0.858. Therefore, ComImp benefits the small sample dataset much better than
the large sample one.

TABLE III: mean $\pm$ variance of the accuracy on test sets of models fitted on $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$ of the Seed dataset, respectively. | test set of $\mathcal{D}_{1}$ | test set of $\mathcal{D}_{2}$
---|---|---
Classifier | ${f}_{1}$ | ${f}$ | ${f}_{2}$ | ${f}$
Logistic Regression | 0.909 $\pm$ 0.001 | 0.913 $\pm$ 0.001 | 0.858 $\pm$ 0.006 | 0.896 $\pm$ 0.003
SVM | 0.917 $\pm$ 0.001 | 0.925 $\pm$ 0.001 | 0.882 $\pm$ 0.003 | 0.890 $\pm$ 0.003

###  VI-C Experiments of merging three datasets

#### Data

We conduct experiments on the Wine dataset. We first randomly split the data
into three datasets $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{3}$. Then,
we delete the first column of the input of $\mathcal{D}_{1}$, the last 8
columns of $\mathcal{D}_{2}$, and the $5^{th}$ and $6^{th}$ column of the
input of $\mathcal{D}_{3}$.

#### Analysis

Note that in Table IV of the results for this experiment, for $f$, the
third/fifth/seventh column contains the results on the portion of the test set
of $\mathcal{D}$ that corresponding to the test set of
$\mathcal{D}_{1}/\mathcal{D}_{2}/\mathcal{D}_{3}$. From Table IV, we can see
that the performance of the model fitted on the merged data improves the
accuracy on the test sets of all the component datasets
$\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{3}$. Note that the sizes of
$\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{3}$ are all relatively small.
So, we can see that ComImp improves the model fitting process when combining
small datasets together.

TABLE IV: mean $\pm$ variance of the accuracy on test sets of models fitted on $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}_{3},\mathcal{D}$ of the Wine dataset, respectively. | test set of $\mathcal{D}_{1}$ | test set of $\mathcal{D}_{2}$ | test set of $\mathcal{D}_{3}$
---|---|---|---
Classifier | ${f}_{1}$ | ${f}$ | ${f}_{2}$ | ${f}$ | ${f}_{3}$ | ${f}$
Logistic Regression |  0.912 $\pm$ 0.004 |  0.941 $\pm$ 0.002 |  0.735 $\pm$ 0.010 |  0.781 $\pm$ 0.007 |  0.946 $\pm$ 0.003 |  0.971 $\pm$ 0.001
SVM |  0.934 $\pm$ 0.003 |  0.955 $\pm$ 0.001 |  0.793 $\pm$ 0.007 |  0.826 $\pm$ 0.005 |  0.953 $\pm$ 0.002 |  0.968 $\pm$ 0.001

###  VI-D ComImp with transfer learning

#### Data

We conduct experiments on multi-variate time series datasets related to EEG
signals. The Electroencephalogram (EEG) dataset is collected for epileptic
seizure recognition [23]. The dataset used for the experiment consists of 178
time points equivalent to one second of EEG recording. The recordings of
seizure activity and non-seizure activity are labeled in the dataset. We first
randomly split the data into two datasets $\mathcal{D}_{1},\mathcal{D}_{2}$
with 6:4 ratio. Then, to simulate the bad EEG recordings with missing values,
we delete the first 16 columns of $\mathcal{D}_{1}$ and the last 17 columns of
$\mathcal{D}_{2}$. Next, we split $\mathcal{D}_{1},\mathcal{D}_{2}$ into
training and testing sets with a 7:3 ratio. To compare our methods with
transfer learning, we conduct the following two procedures:

  * •

We train a fully connected two-layered neural network ($f_{1}$) on the
training portion of $\mathcal{D}_{1}$. Next, we transfer the weights of
$f_{1}$ to train a fully connected network $f_{2}$ on the training portion of
$\mathcal{D}_{2}$ and fine-tune $f_{2}$.

  * •

For our method, we use ComImp to combine $\mathcal{D}_{1}$ and
$\mathcal{D}_{2}$, which gives us $\mathcal{D}$ with $\mathcal{D}_{train}$ is
the merge between the training portion of $\mathcal{D}_{1}$ and
$\mathcal{D}_{2}$. In addition, $\mathcal{D}_{test_{1}}$ corresponding to the
testing portion of $\mathcal{D}_{1}$, and $\mathcal{D}_{test_{2}}$
corresponding to the testing portion of $\mathcal{D}_{2}$. We train a model
$f$ on $\mathcal{D}_{train}$. Then, we transfer the weights of $f$ onto the
training portion of $\mathcal{D}_{1}$ and fine-tune the model, which gives us
model $f^{ComImp}_{1}$. We do similarly for $\mathcal{D}_{2}$, which gives us
$f^{ComImp}_{2}$.

We run the model for 10,000 epochs and report the classification accuracy of
$f_{1}$ on the testing portion of $\mathcal{D}_{1}$, and the classification
accuracy of $f_{2}$ on the testing portion of $\mathcal{D}_{2}$ in Table VII.
In addition, we report the classification accuracy of
$f^{ComImp}_{1},f^{ComImp}_{2}$ on $\mathcal{D}_{test_{1}}$,
$\mathcal{D}_{test_{2}}$, which corresponding to the testing portion of
$\mathcal{D}_{1},\mathcal{D}_{2}$, in column 3 and 5 of Table VII,
respectively.

Analysis. We observe in Table VII that using ComImp to combine the training
parts of $\mathcal{D}_{1},\mathcal{D}_{2}$, train a model $f$ on the merged
and fine-tuning on the corresponding training parts of $\mathcal{D}_{1}$ and
$\mathcal{D}_{2}$ yields a better classification accuracy result.
Specifically, the performance gain is 2.2 $\%$ for the test of
$\mathcal{D}_{1}$ and 1.0 $\%$ over the test set of $\mathcal{D}_{2}$. The
gain in accuracy result demonstrates the positive impact of merging datasets
by ComImp before fine-tuning.

###  VI-E Data imputation performance for missing datasets

#### Data

We conduct experiments on the Wine dataset. We delete the first two columns of
the input in $\mathcal{D}_{1}$ and split it into training and testing sets of
equal sizes. In addition, we delete the last two columns of the input data in
$\mathcal{D}_{2}$. Then, we generate missing data at missing rates of
$20\%,40\%,60\%$ on each training/testing set. In addition, we use ComImp to
combine the corresponding training and testing sets of
$\mathcal{D}_{1},\mathcal{D}_{2}$ to the training and testing sets of
$\mathcal{D}$.

For the ComImp approach, the missing values are automatically filled after
merging the datasets. For the non-ComImp approach, we use softImpute to impute
missing values. Then, we fit a models $f_{1},f_{2},f$ on the training set of
$\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$, respectively.

#### Analysis.

Note that in the table of results (Table V), for $f$, the fourth (sixth)
column contains the results on the portion of the test set of $\mathcal{D}$
that corresponding to the test set of $\mathcal{D}_{1}(\mathcal{D}_{2})$. From
Table V, we can see that the performance of $f$ on the test set of
$\mathcal{D}_{1}$ is the same as $f_{1}$ in most cases. However, the
performance of $f$ on the test set of $\mathcal{D}_{2}$ is significantly
better than $f_{2}$. For example, at 80% missing rate, when using SVM as the
classifier, the accuracy of $f$ on the test set of $\mathcal{D}_{2}$ is
$40.692$, which is 8.9% higher than $f_{2}$, whose accuracy is 0.603. Note
that $\mathcal{D}_{1}$ has 125 samples and $\mathcal{D}_{2}$ has 53 samples.
Therefore, combining datasets aid the dataset with a smaller sample size
significantly.

TABLE V: mean $\pm$ variance of the accuracy on test sets of models fitted on $\mathcal{D}_{1},\mathcal{D}_{2}$ of the Wine dataset, respectively. | test set of $\mathcal{D}_{1}$ | test set of $\mathcal{D}_{2}$
---|---|---
missing rate | Classifier | ${f}_{1}$ | ${f}$ | ${f}_{2}$ | ${f}$
20% | Logistic Regression |  0.917 $\pm$ 0.001 |  0.908 $\pm$ 0.001 |  0.859 $\pm$ 0.007 |  0.874 $\pm$ 0.004
SVM |  0.939 $\pm$ 0.001 |  0.939 $\pm$ 0.001 |  0.882 $\pm$ 0.006 |  0.893 $\pm$ 0.004
40% | Logistic Regression |  0.883 $\pm$ 0.002 |  0.877 $\pm$ 0.002 |  0.804 $\pm$ 0.009 |  0.837 $\pm$ 0.005
SVM |  0.901 $\pm$ 0.001 |  0.901 $\pm$ 0.001 |  0.814 $\pm$ 0.009 |  0.846 $\pm$ 0.005
60% | Logistic Regression |  0.834 $\pm$ 0.002 |  0.831 $\pm$ 0.002 |  0.733 $\pm$ 0.010 |  0.783 $\pm$ 0.007
SVM |  0.843 $\pm$ 0.002 |  0.846 $\pm$ 0.002 |  0.716 $\pm$ 0.012 |  0.779 $\pm$ 0.008
80% | Logistic Regression |  0.762 $\pm$ 0.004 |  0.762 $\pm$ 0.003 |  0.645 $\pm$ 0.012 |  0.712 $\pm$ 0.008
SVM |  0.762 $\pm$ 0.004 |  0.766 $\pm$ 0.003 |  0.603 $\pm$ 0.014 |  0.692 $\pm$ 0.010

###  VI-F Combining datasets using dimension reduction

#### Data

We conduct experiments on the Gene dataset [6]. First, we split it into
$\mathcal{D}_{1},\mathcal{D}_{2}$ with a ratio $7:3$, and then split each of
them into halves for training and testing. Then, we delete the first 10,000
columns of the input in $\mathcal{D}_{1}$ and the last 10,000 columns of the
input in $\mathcal{D}_{2}$. We apply ComImp with PCA and train a neural
network $f$ on the merged data. Also, we train a separate model $f_{1}$ on
$\mathcal{D}_{1}$, and $f_{2}$ on $\mathcal{D}_{2}$. We repeat the experiment
100 times and report the mean and variance of the accuracy in Table VI.

#### Analysis

Similar to the previous experiments, in Table VI, for $f$, the third (fifth)
column contains the results on the portion of the test set of $\mathcal{D}$
that corresponding to the test set of $\mathcal{D}_{1}(\mathcal{D}_{2})$. From
Table VI, we can see that the PCA-ComImp improves the performance for SVM on
the test sets of both $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. For example, on
the test set of $\mathcal{D}_{2}$, $f$ has an accuracy of $0.971$, which is
higher than $f_{2}$ (0.955).

TABLE VI: mean $\pm$ variance of the accuracy on test sets of regression models fitted on $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$ of the Gene dataset, respectively. | test set of $\mathcal{D}_{1}$ | test set of $\mathcal{D}_{2}$
---|---|---
Classifier | ${f}_{1}$ | ${f}$ | ${f}_{2}$ | ${f}$
Logistic Regression | 0.998 $\pm$ 0.002 | 0.998 $\pm$ 0.002 | 0.996 $\pm$ 0.006 | 0.996 $\pm$ 0.006
SVM | 0.995 $\pm$ 0.007 | 0.997 $\pm$ 0.003 | 0.955 $\pm$ 0.021 | 0.971 $\pm$ 0.012
TABLE VII: Comparison of transfer learning and transfer learning with ComImp on the EEG dataset. | test set of $\mathcal{D}_{1}$ | test set of $\mathcal{D}_{2}$
---|---|---
Classifier | ${f}_{1}$ | $f^{ComImp}_{1}$ | ${f}_{2}$ | $f^{ComImp}_{2}$
Transfer Learning | 0.774 | 0.796 | 0.839 | 0.849

###  VI-G A case analysis of when OsImp may fail

This experiment is conducted on the Seed dataset with the same setup as in
section VI-B except that we delete the first three features of
$\mathcal{D}_{1}$ and the last four features of $\mathcal{D}_{2}$. The results
are reported in Table VIII.

From Table VIII, we see that the performance of the model trained only on the
training set of $\mathcal{D}_{1}$ and the model trained only on the training
set of $\mathcal{D}_{2}$ are better than the model trained on the merge of
$\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. Next, note that the Seed dataset has
only seven features, and therefore, $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$
have only one overlapping feature. This is an example of when the imputation
of too many features while having only a few overlapping features may
introduce too much noise and bias to compensate for the benefits of having
more samples.

TABLE VIII: mean $\pm$ variance of the accuracy on test sets of models fitted on $\mathcal{D}_{1},\mathcal{D}_{2},\mathcal{D}$ of the Seed dataset, respectively. | test set of $\mathcal{D}_{1}$ | test set of $\mathcal{D}_{2}$
---|---|---
Classifier | ${f}_{1}$ | ${f}$ | ${f}_{2}$ | ${f}$
Logistic Regression | 0.912 $\pm$ 0.001 | 0.911 $\pm$ 0.001 | 0.730 $\pm$ 0.017 | 0.675 $\pm$ 0.020
SVM | 0.926 $\pm$ 0.001 | 0.923 $\pm$ 0.001 | 0.827 $\pm$ 0.008 | 0.826 $\pm$ 0.010

##  VII Conclusions

In this paper, we introduced a novel method, ComImp, to combine datasets. The
proposed method increases the sample size and makes use of information from
multiple datasets, and is highly beneficial for small sample datasets, as
illustrated in the experiments. We also illustrated that ComImp could improve
model fitting even more when being used with transfer learning. In addition,
we introduced PCA-ComImp, a variant of ComImp, for combining datasets with a
large number of features not in common.

It is worth noting that imputation may also introduce noise and bias. As
illustrated in section VI-G, if the number of overlapping features is too
small compared to the number of non-overlapping features, the advantage of
having more samples may not be able to compensate for that. In addition, the
choice of imputation methods has crucial effects on classification results.
Therefore, it is also worth investigating more imputation methods for ComImp
and PCA-ComImp in the future. Moreover, when the datasets under combination
are of high dimension, it is possible to increase the imputation speed of an
imputation algorithm by using the PCAI framework [18].

In addition, it is also interesting to see if the idea in this paper can be
applied to images or other types of data. Further, the method may also be
applicable for combining the datasets for MTL approaches. For example, ComImp
can be used in the multi-exit architecture proposed for MTL in [11], where the
different datasets need to be combined.

Next, note that about the proposed PCA-ComImp technique uses PCA, which can
lead to a loss of interpretability of the features that are not shared among
datasets. Therefore, in the future, it is worth studying how feature selection
techniques [17] can be used instead of dimension reduction for this problem or
parallel features selection techniques [21] for large datasets.

Lastly, it is worth investigating further the use of ComImp with domain
adaptation for further boosting of performance in medical tasks such as in
diabetic retinopathy grading [15], disease detection [7], spermatozoa tracking
[29, 28], life-logging [24], etc. Another interesting problem is how the
proposed techniques can be used with data augmentation or synthetic data
generation [30] to ameliorate the deficiency of data problems in medicine and
various other fields. These questions are worth investigating and will be the
topics for our future research.

## References

  * [1] S van Buuren and Karin Groothuis-Oudshoorn.  mice: Multivariate imputation by chained equations in r.  Journal of statistical software, pages 1–68, 2010.
  * [2] Bin Cao, Nathan Nan Liu, and Qiang Yang.  Transfer learning for collective link prediction in multiple heterogenous domains.  In ICML, 2010.
  * [3] Małgorzata Charytanowicz, Jerzy Niewczas, Piotr Kulczycki, Piotr A Kowalski, Szymon Łukasik, and Sławomir Żak.  Complete gradient clustering algorithm for features analysis of x-ray images.  In Information technologies in biomedicine, pages 15–24. Springer, 2010.
  * [4] Danmin Chen, Shuai Yang, and Funa Zhou.  Transfer learning based fault diagnosis with missing data due to multi-rate sampling.  Sensors, 19(8):1826, 2019.
  * [5] Jun Deng, Zixing Zhang, Erik Marchi, and Björn Schuller.  Sparse autoencoder-based feature transfer learning for speech emotion recognition.  In 2013 humaine association conference on affective computing and intelligent interaction, pages 511–516. IEEE, 2013.
  * [6] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.
  * [7] Steven Alexander Hicks, Pia H Smedsrud, Pål Halvorsen, and Michael Riegler.  Deep learning based disease detection using domain specific transfer learning.  MediaEval, 18:29–31, 2018.
  * [8] Mahbub Hussain, Jordan J Bird, and Diego R Faria.  A study on cnn transfer learning for image classification.  In UK Workshop on computational Intelligence, pages 191–202. Springer, 2018.
  * [9] Christine N Kohnen and Jerome P Reiter.  Multiple imputation for combining confidential data owned by two agencies.  Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(2):511–528, 2009.
  * [10] Abhishek Kumar and Hal Daume III.  Learning task grouping and overlap in multi-task learning.  arXiv preprint arXiv:1206.6417, 2012.
  * [11] Sang-woo Lee, Ryong Lee, Min-seok Seo, Jong-chan Park, Hyeon-cheol Noh, Jin-gi Ju, Rae-young Jang, Gun-woo Lee, Myung-seok Choi, and Dong-geol Choi.  Multi-task learning with task-specific feature filtering in low-data condition.  Electronics, 10(21):2691, 2021.
  * [12] Rutian Liu, Eric Simon, Bernd Amann, and Stéphane Gançarski.  Discovering and merging related analytic datasets.  Information Systems, 91:101495, 2020.
  * [13] Rahul Mazumder, Trevor Hastie, and Robert Tibshirani.  Spectral regularization algorithms for learning large incomplete matrices.  Journal of machine learning research, 11(Aug):2287–2322, 2010.
  * [14] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3994–4003, 2016.
  * [15] Duy MH Nguyen, Truong TN Mai, Ngoc TT Than, Alexander Prange, and Daniel Sonntag.  Self-supervised domain adaptation for diabetic retinopathy grading using vessel image reconstruction.  In KI 2021: Advances in Artificial Intelligence: 44th German Conference on AI, Virtual Event, September 27–October 1, 2021, Proceedings 44, pages 349–361. Springer, 2021.
  * [16] Duy MH Nguyen, Thu T Nguyen, Huong Vu, Quang Pham, Manh-Duy Nguyen, Binh T Nguyen, and Daniel Sonntag.  Tatl: task agnostic transfer learning for skin attributes detection.  Medical Image Analysis, 78:102359, 2022.
  * [17] Thu Nguyen.  Faster feature selection with a dropping forward-backward algorithm.  arXiv:1910.08007, 2019.
  * [18] Thu Nguyen, Hoang Thien Ly, Michael Alexander Riegler, and Pål Halvorsen.  Principle components analysis based frameworks for efficient missing data imputation algorithms.  arXiv preprint arXiv:2205.15150, 2022.
  * [19] Thu Nguyen, Khoi Minh Nguyen-Duy, Duy Ho Minh Nguyen, Binh T. Nguyen, and Bruce Alan Wade.  Dper: Direct parameter estimation for randomly missing data.  Knowledge-Based Systems, 240:108082, 2022.
  * [20] Thu Nguyen, Nhan Thanh Phan, Ha Van Hoang, Pål Halvorsen, Michael A Riegler, and Binh T Nguyen.  Pmf: Efficient parameter estimation for data sets with missing data in some features.  Available at SSRN 4260235, 2022.
  * [21] Thu Nguyen, Thanh Nhan Phan, Van Nhuong Nguyen, Thanh Binh Nguyen, Pål Halvorsen, and Michael Riegler.  Parallel feature selection based on the trace ratio criterion.  In International Joint Conference on Neural Networks (IJCNN), 2022\.
  * [22] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2010.
  * [23] Ernest Fokoue Qiuyi Wu.  Epileptic seizure recognition data set, 2017.
  * [24] Michael A Riegler, Vajira Thambawita, Ayan Chatterjee, Thu Nguyen, Steven A Hicks, Vibeke Telle-Hansen, Svein Arne Pettersen, Dag Johansen, Ramesh Jain, and Pål Halvorsen.  Scopesense: An 8.5-month sport, nutrition, and lifestyle lifelogging dataset.  In MultiMedia Modeling: 29th International Conference, MMM 2023, Bergen, Norway, January 9–12, 2023, Proceedings, Part I, pages 502–514. Springer International Publishing Cham, 2023.
  * [25] Pedro LC Rodrigues, Marco Congedo, and Christian Jutten.  Dimensionality transcending: a method for merging bci datasets with different dimensionalities.  IEEE Transactions on Biomedical Engineering, 68(2):673–684, 2020\.
  * [26] Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf.  Transfer learning in natural language processing.  In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Tutorials, pages 15–18, 2019.
  * [27] Daniel J Stekhoven and Peter Bühlmann.  Missforest-non-parametric missing value imputation for mixed-type data.  Bioinformatics, 28(1):112–118, 2012.
  * [28] Vajira Thambawita, Steven Hicks, Andrea M Storås, Jorunn M Andersen, Oliwia Witczak, Trine B Haugen, Hugo Hammer, Thu Nguyen, Pål Halvorsen, and Michael A Riegler.  Medico multimedia task at mediaeval 2022: Transparent tracking of spermatozoa.  In Proceedings of MediaEval 2022 CEUR Workshop, 2022.
  * [29] Vajira Thambawita, Steven A Hicks, Andrea M Storås, Thu Nguyen, Jorunn M Andersen, Oliwia Witczak, Trine B Haugen, Hugo L Hammer, Pål Halvorsen, and Michael A Riegler.  Visem-tracking: Human spermatozoa tracking dataset.  arXiv preprint arXiv:2212.02842, 2022.
  * [30] Vajira Thambawita, Pegah Salehi, Sajad Amouei Sheshkal, Steven A Hicks, Hugo L Hammer, Sravanthi Parasa, Thomas de Lange, Pål Halvorsen, and Michael A Riegler.  Singan-seg: Synthetic training data generation for medical image segmentation.  PloS one, 17(5):e0267976, 2022.
  * [31] Shefali S Verma, Mariza de Andrade, Gerard Tromp, Helena Kuivaniemi, Elizabeth Pugh, Bahram Namjou-Khales, Shubhabrata Mukherjee, Gail P Jarvik, Leah C Kottyan, Amber Burt, et al.  Imputation and quality control steps for combining multiple genome-wide datasets.  Frontiers in genetics, 5:370, 2014.
  * [32] Mai Anh Vu, Thu Nguyen, Tu T Do, Nhan Phan, Pål Halvorsen, Michael A Riegler, and Binh T Nguyen.  Conditional expectation for missing data imputation.  arXiv preprint arXiv:2302.00911, 2023.
  * [33] Xiang Zhou, Hua Chai, Huiying Zhao, Ching-Hsing Luo, and Yuedong Yang.  Imputing missing rna-sequencing data from dna methylation by using a transfer learning–based neural network.  GigaScience, 9(7):giaa076, 2020.
  * [34] Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight.  Transfer learning for low-resource neural machine translation.  arXiv preprint arXiv:1604.02201, 2016.
  * [35] Alon Zweig and Daphna Weinshall.  Hierarchical regularization cascade for joint learning.  In International conference on machine learning, pages 37–45. PMLR, 2013.

  *[MTL]: Multi-task learning
  *[EEG]: Electroencephalogram