# Improving Prediction of Cognitive Performance using Deep Neural Networks in
Sparse Data

Sharath Koorathota  Arunesh Mittal  Department of Electrical Engineering,
Columbia University, New York, USA.  Richard Sloan  Department of Psychiatry,
Columbia University Medical Center, New York, USA.  Paul Sajda  Department of
Biomedical Engineering, Columbia University, New York, USA.  Department of
Electrical Engineering, Columbia University, New York, USA.  Data Science
Institute, Columbia University, New York, USA.

###### Abstract

Cognition in midlife is an important predictor of age-related mental decline
and statistical models that predict cognitive performance can be useful for
predicting decline. However, existing models struggle to capture complex
relationships between physical, sociodemographic, psychological and mental
health factors that effect cognition. Using data from an observational, cohort
study, Midlife in the United States (MIDUS), we modeled a large number of
variables to predict executive function and episodic memory measures. We used
cross-sectional and longitudinal outcomes with varying sparsity, or amount of
missing data. Deep neural network (DNN) models consistently ranked highest in
all of the cognitive performance prediction tasks, as assessed with root mean
squared error (RMSE) on out-of-sample data. RMSE differences between DNN and
other model types were statistically significant (T(8) = -3.70; p $<$ 0.05).
The interaction effect between model type and sparsity was significant
($F_{9}$=59.20; p $<$ 0.01), indicating the success of DNNs can partly be
attributed to their robustness and ability to model hierarchical relationships
between health-related factors. Our findings underscore the potential of
neural networks to model clinical datasets and allow better understanding of
factors that lead to cognitive decline.

##  1 Introduction

Cognitive performance is a predictor of a variety of age-related clinical
conditions. For example, substantial research has linked measures of cognitive
performance and its decline to conditions such as attention-
deficit/hyperactivity disorder (ADHD), mild cognitive impairment (MCI), and
Alzheimer’s disease (AD) [1, 2, 3, 4, 5]. Furthermore, interventions enhancing
cognitive performance have had measurable effects on progression of MCI [6,
7]. Improving prediction of cognitive performance has the potential to impact
the diagnosis and treatment of mental illness.

Cognitive performance is affected by a host of factors. A complex system of
physical, psychosocial and emotional factors affect performance in patient and
healthy populations [8]. For example, better executive function and episodic
memory have been associated with demographic variables such as sex [9],
recovery of vagal tone from cognitive challenge [10] and higher educational
attainment [9]. The majority of past approaches to understanding factors that
are associated with cognitive performance have used linear models, such as
regression, where the relationship between variables or factors can be
reported through beta weights [11, 12, 13, 9], with more recent work using
interpretable, non-linear models such as random forests (RF) [14].

A challenge in predicting cognitive performance in clinical populations is the
trade-off in the size of a dataset and the quality. For large, national,
studies with a broad aim, data may be collected at multiple sites, in-lab
(e.g., biological specimen data) or remotely (e.g., phone questionnaires).
Statistical models in this case may suffer from overfitting the data, driven
by large number of missing data [15] or sparse variables, where only a small
proportion of the participant pool has measured values [16, 17, 18].
Furthermore, model fitting and assessment in this area of research is often
performed on the same samples. While this approach makes sense in order to
understand associations between variables and their strength in predicting
cognitive performance, these methods are not ideal for improving the
robustness and generalizability of prediction itself.

The primary purpose of this study is to improve cross-sectional and
longitudinal prediction of cognitive performance. To this end, we use
aggregated data from the Midlife in the United States (MIDUS) study. Our
innovations are twofold. First, we introduce non-linear modeling frameworks
which yield lower prediction errors across multiple cognitive outcome measures
compared to traditional, statistical models. Specifically, we show deep neural
networks (DNNs) outperform other models through learning the complex
relationships of factors in a heterogeneous, sparse dataspace. Secondly, we
present a method for interpreting hierarchical relationships between clinical
factors using vectorized representations, or embeddings, of groups of related
variables. Together, our results highlight the advantages of DNNs in the
context of modeling large, mixed, and sparse clinical datasets.

##  2 Methods

###  2.1 Dataset

The MIDUS-AGG dataset was aggregated from the MIDUS national study [19, 20].
The overall goal of MIDUS is to investigate the factors that influence
physical and mental health as people age. The current study focused on the
participants who completed the cognitive assessment from the second and third
waves of the longitudinal study. The sample was recruited in 1995–1996 through
random digit dialing of U.S. households having at least one telephone in the
contiguous 48 states, stratified by age with an oversample of those between 40
and 60 years of age.

A second wave (MIDUS 2, M2) of data collection took place in 2004-2005, in
4955 participants from M1 (53.8% female; 90.1% white) aged 32 to 84 years
(M=55.36; SD=12.40), with a mean education level of 14.24 years (SD=2.60)
[21]. A third wave of data collection (MIDUS 3, M3) was conducted on average 9
years (SD=.53) after M2, with 76.9% retention of eligible M1 participants
(N=3294; 55.3% female; 90.4% white). Whereas in M1, only survey data were
collected, in M2 ad M3, data were collected in daily stressor, cognitive,
biomarker, and neuroimaging projects. Participants ranged in age from 42 to 92
years (M=64.30; SD=11.20) at M3. Typical of longitudinal studies, MIDUS 3
participants showed some differences on MIDUS 2 variables compared with those
who dropped out of the study, with dropouts performing significantly more
poorly on all cognitive tests at MIDUS 2 [9].

####  2.1.1 Cognitive Measures

Cognitive performance was assessed using the Brief Test of Adult Cognition by
Telephone (BTACT), designed to be sensitive to changes in cognitive function
[22, 23]. The BTACT included assessed word-list free recalls, digits backward
span tasks, verbal fluency tasks, number series pattern tasks and the Stop and
Go Switch Task. In addition, composite scores for executive function (EF) and
episodic memory (EM) were computed.

In the present study, we used machine learning approaches applied to data from
all MIDUS projects to predict EF, EM and COMP at M2 and M3 and change scores
($\Delta$) from M2 to M3 in 9 separate prediction tasks. We defined sparsity
simply as the count of missing, invalid or inapplicable data in each
prediction task. Selected participant demographics, variable counts, and
sparsity of the datasets used for prediction of each outcome are listed in
Table 2.1.1.

| n | Score (SD) | Variable Count | Sparsity
---|---|---|---|---
M2 Age | <55 | >=55 | <55 | >=55
Sex | Female | Male | Female | Male | Female | Male | Female | Male
Prediction Task |  |  |  |  |  |  |  |  |  |
COMP(M2) | 257 | 194 | 243 | 187 | 80.12 (16.84) | 82.48 (16.62) | 70.07 (15.74) | 70.02 (15.55) | 8542 | 3459284
EF(M2) | 259 | 191 | 236 | 188 | 66.16 (15.01) | 70.48 (14.35) | 57.1 (13.28) | 60.98 (14.04) | 8542 | 3445748
EM(M2) | 249 | 186 | 242 | 184 | 12.62 (3.75) | 10.97 (3.86) | 11.43 (4.41) | 8.47 (3.29) | 8542 | 3475186
COMP(M3) | 197 | 145 | 159 | 140 | 81.47 (15.41) | 82.1 (17.37) | 68.31 (15.74) | 69.41 (14.75) | 9831 | 2660931
$\Delta$ COMP(M3) | 188 | 142 | 153 | 133 | -2.18 (9.75) | -2.92 (9.91) | -5.15 (9.22) | -3.86 (8.42) | 9831 | 2560627
EF(M3) | 193 | 147 | 166 | 144 | 65.76 (13.66) | 69.74 (15.36) | 54.81 (13.83) | 59.57 (13.2) | 9831 | 2691342
$\Delta$ EF(M3) | 188 | 149 | 160 | 141 | -3.4 (8.71) | -2.7 (9.1) | -4.35 (7.79) | -3.28 (6.87) | 9831 | 2627376
EM(M3) | 194 | 147 | 162 | 143 | 13.28 (4.23) | 10.69 (3.74) | 11.18 (4.58) | 8.31 (3.31) | 9831 | 2697842
$\Delta$ EM(M3) | 173 | 146 | 158 | 138 | 0.14 (3.94) | -0.72 (3.72) | -1.09 (3.84) | -0.97 (3.25) | 9831 | 2561928

Table 1: Description of the 9 prediction tasks: their outcome, cognitive
measure, and the input data breakdown. Cognitive outcome measures included
BTACT composite score (COMP), episodic memory score (EM), executive function
score (EF). Note that COMP is comprised of EF and EM scores. Both MIDUS 2 (M2)
and MIDUS 3 (M3) timepoints were predicted, along with change from M2 to M3
($\Delta$). Only participants who complete all M2 projects and relevant M3
projects (Survey Measures, Cognitive Measures) were included in the study.
Predictor variable count varied with task, and sparsity is simply quantified
as the count of missing, inapplicable or invalid data points.

####  2.1.2 Survey Measures

Survey data included sociodemographic, physical and mental health history. For
example, a series of questions addressed alcohol use, including history of
use, regularity of use, attempts to quit, and how the use of those substances
affected respondents’ physical and mental well-being. Questions included a mix
of numerical, interval, ordinal and scaled data. Survey measures also asked
about their perceived well-being compared to peers, exercise and diet habits,
experience with menopause, and past therapies attempted. Demographic questions
included work histories, presence or absence of their parents in childhood,
religion, history of physical abuse, and the quality of their relationships to
those around them. Respondents also were asked about their significant others’
occupations, and about familial relationships.

####  2.1.3 Daily Stress Measures

Daily stress measures also were mixed in variable types, and included
sociodemographic, health status, personality characteristics, and genetic
factors related to exposure, coping and reactivity to day-to-day life
stressors. These were obtained using a daily telephone interview, with
participants, on average, completing 7.2 out of 8 days of surveys.
Participants were asked about frequency, content and severity of daily
stressors and how they compared to levels of stress in the past 10 years. They
were also asked, through open-ended questions, about personality
characteristics that may affect their response to stress.

####  2.1.4 Biomarker Measures

Biomarker measures from the MIDUS dataset related to non-disease specific,
biological data collection to understand how the functioning of various
biological systems relate to behavioral and psychosocial factors.
Specifically, biomarkers assessed functionality of the hypothalamic-pituitary-
adrenal axis, autonomic nervous system, immune system, cardiovascular system,
musculoskeletal system, antioxidants, and metabolic processes. Data related to
biomarkers were collected through specimens, clinical assessments, self-
reports and in-lab protocols. Notably, self-reported sleep assessments were
also collected as a part of the Biomarker project.

####  2.1.5 Data Aggregation

We used two main approaches to aggregating MIDUS data for the purposes of this
study. As a first step, we collected all M2 data available and all M3 data
from projects that were complete (Survey Measures, Cognitive Measures; other
projects are ongoing) from ICPSR, an online data archive [20]. We then hand-
annotated variables to include or exclude from analysis. Variables that were
unique identifiers of participants were excluded (e.g., the date of their
phone calls), are specific to the MIDUS study with potential to not be
replicable, or had zero variance.

We used hierarchical metadata from the original dataset to assign factor-level
groups (Supplementary Table 4). Missing metadata from the MIDUS dataset were
annotated and a total of 10,134 usable variables from M2 and M3 were isolated
for processing.

###  2.2 Data Processing

Only participants who completed the cognitive, survey, daily stress and
biomarker projects were included in the analyses. Numerical data were
normalized and categorical data were one-hot encoded prior to training.

###  2.3 Prediction Approach

We report two, separate approaches to predicting cognitive performance (Figure
1). The first approach used the full dataset and the second approach used
lower-dimensional representations. We expected that the sparseness and high
dimensionality of MIDUS-AGG would make it difficult to avoid common issues
such as overfitting [24] and curse-of-dimensionality [25].

Figure 1: Overview of methods in the present study. We used the original
MIDUS-AGG with over 10,134 variables and a variable number of participants,
depending on the prediction task. Separately, we used an embedding
representation of data as input to the DNN. In embeddings, each of the 55
factors were represented by a 16-dimensional vector, learned during training.
Training utilized one of 6 models (linear or non-linear methods). Model
performance was assessed using mean RMSE values from 10 runs, on random splits
of the test data.

####  2.3.1 Baseline Models

To establish baseline comparisons, we selected models that are typically used
for cognitive performance prediction, through a literature search of common
approaches. We identified three model classes that have been used most often
by researchers: linear regression methods, regression with regularization, and
tree-based machine learning frameworks. We selected the following baseline,
off-the-shelf [26, 27] models for fitting (Figure 1):

  1. 1.

linear regression with no regularization,

  2. 2.

ridge regression with L2 regularization,

  3. 3.

lasso regression with L1 regularization,

  4. 4.

tree-based, RF,

  5. 5.

gradient boosting (XGBoost)

RF and XGBoost are machine learning approaches that have yielded encouraging,
interpretable results in clinical data [28].

####  2.3.2 Deep Neural Network

In addition to assessing the performance of off-the-shelf models, we designed
a DNN (Figure 2) to provide the expected advantage of non-linear model
frameworks, while addressing problems related to overfitting that may be
present in a large, sparse dataset like MIDUS-AGG.

Figure 2: (a) DNN framework. Input data comprises of Survey Measures, Daily
Diary and Biomarker data from M2 and Survey Measures data from M3, similar to
baseline models. The core architecture comprises of 3 layers of dense
networks, with batch normalization, leaky ReLU and a high rate of dropout. The
prediction was a ReLU output in cases of positive scores, or linear output
when predicting changes, and the network was optimized using a mean squared
loss function. (b) In a separate architecture, embeddings were grouped by
factors prior to the dense layers and prediction.

We concatenated the mixed, processed data using hyperparameters optimized
through manual search and designed the core architecture that included three
layers comprised of: (1) a 256-dimensional dense, feedforward (FF) network,
(2) batch normalization, (3) leaky rectified linear unit (Leaky ReLU), and (4)
a high dropout rate of 0.5. We utilized a batch size of 32, a ceiling of
10,000 epochs, a learning rate of 0.001 and early stopping loss of 500 epochs.
Additionally, we used a random, 15% of the training data, each epoch, for
validation and only saved the DNN weights from the last epoch with an
improvement in prediction, through a drop in mean squared error (MSE) loss. We
used a cross-validation approach to model fitting as described previously,
reporting the average performance of each model.

Base cognitive measures (always $\geq$ 0) were predicted using a ReLU output
and change in cognitive scores ($\Delta$) were predicted using a linear output
to account for negative values.

Using an independent split of data, we initially experimented with variants of
the model using a standard ReLU, lower (more typical) dropout rates, lower
learning rates, and 32, 64, 128 and 512-dimensional FF networks. We
experienced common problems faced in machine learning such as vanishing
gradients [29] and overfitting, and determined hyperparameters that
contributed to a relative decrease in error rates across cognitive measures.

To assess the independent contribution of each factor, we added an embedding
layer [30] to the DNN after data processing (Figure 2b). For example, 274
”Health” factor variables (Supplementary Table 4) were first reduced to a
16-dimensional, learned, vector which was used as input to a DNN. This process
is similar to other dimensionality reduction techniques, such as PCA [31], but
has the added benefit of learning the context of groups of variables during
training.

We utilized a total of 55, 16-dimensional embeddings (one for each factor),
decreasing the number of inputs to the model to 880. An embedding captured the
learned, numerical representation of each group of variables (Table 4). We
chose 16 as the number of dimensions since most factors had at least 16
variables, and language models, where embeddings are used extensively, have
found success with 16-dimensional vector representations [32, 33, 34]. Our
expectation was that the modularity in designing embeddings will allow a more
thorough understanding about each factor’s similarity to other factors and to
further understand how important factors are to prediction. We used the
Kullback-Leibler divergence between the joint probabilities of the average,
16-dimensional embedding and a 2-dimensional coordinate for ease of plotting
using t-SNE.

###  2.4 Evaluation

Using the processed, full MIDUS-AGG data and, separately the 16-dimensional
data, we fitted the model parameters using MSE as the loss function and report
the predicted root mean squared error (RMSE) for each model. Because our goal
was prediction, we utilized cross-validation techniques to estimate out-of-
sample performance errors. We averaged the error after model fitting and
assessment of 10 different splits of the dataset into training (75%) and test
(25%) data.

Using the average RMSE for each factor embedding, we ranked predictions from
each factor from most important (low RMSE) to least important (high RMSE) and
report the average rank of each factor across all cognitive measures. To
quantify the similarity between factors, we isolated weights from each
factor’s embeddings and used t-SNE [35], a visualization technique used for
high-dimensional data. We note, however, there is a well-known trade-off
between temporal coherence and projection reliability with visualizing high-
dimensional data.

###  2.5 Significance Tests

Model performance differences, in RMSE, were assessed using paired, pairwise,
T-tests with Holm-Bonferroni correction. To test whether sparsity
significantly affected model performance, we used an ANOVA to test for
interaction effect between sparsity and model type.

##  3 Results

After data processing, the total number of participants with usable data
ranged from 615 for M3 and change measures (M2 mean age=54.67; 54.4% female)
to 881 for M2 measures (M2 mean age=55.74; 56.6% female).

Figure 3: Ranks of cognitive measures. (a) We independently assessed
performance from 7 models (colored), as described previously, and ranked the
performance using RMSE (lower is better). $embed\\_dnn$ refers to the DNN
utilizing embeddings from groups of variables, rather than the full dataset as
input. SD is displayed in parentheses. (b) Model performance vs. sparsity of
input data, averaged across all prediction tasks. Each dot presents a single,
test sample, participant, colored by model used in the prediction. A 95%
confidence interval is displayed around each fit.

###  3.1 Model Rankings

Our primary objective was to assess model performance in predicting executive
function and episodic memory, where the predictions were either of raw scores
from M2/M3 or change from M2 to M3 ($\Delta$). We ranked performance of models
by test data RMSE, averaged over 10, random, splits of the dataset (Figure
3a). This ranking includes the DNN network after reducing the dimensionality
of input data through factor-specific embeddings (i.e., using the
$embed\\_dnn$ model).

Across prediction tasks using the full dataset, the DNNs had lower RMSE than
other non-linear models (i.e., RF, GB), regularized regression models (i.e.,
Lasso, Ridge) and unregularized models (i.e., linear regression) (Figure 3a)
at M2 and M3. There were no significant differences in errors between sex or
age groups across all model types ($F_{231,20}=0.80$, p=0.71) (Table 2). In
all 9 tasks, model rankings when predicting M3 measures differed from M2 model
rankings.

We assessed significant differences in errors across model types, through
parametric t-test with Holm-Bonferroni correction (Supplementary Table 5). On
average, $embed\\_dnn$ significantly differed (T(8)$<$-5.08, p $\leq$ 0.05) in
mean RMSE from all other model types, except the DNN.

Prediction Task | Sex | M2 Age | Sparsity | RMSE (SD)
---|---|---|---|---
COMP(M2) | Female |  $<$55 | 4327.78 | 14.04 (0.41)
|  |  $\geq$55 | 4333.53 | 13.99 (0.42)
| Male |  $<$55 | 4457.71 | 14.02 (0.44)
|  |  $\geq$55 | 4489.13 | 14.04 (0.41)
EF(M2) | Female |  $<$55 | 4320.30 | 12.8 (0.52)
|  |  $\geq$55 | 4332.89 | 12.78 (0.53)
| Male |  $<$55 | 4458.73 | 12.8 (0.52)
|  |  $\geq$55 | 4482.27 | 12.74 (0.51)
EM(M2) | Female |  $<$55 | 4331.18 | 3.92 (0.1)
|  |  $\geq$55 | 4338.65 | 3.92 (0.1)
| Male |  $<$55 | 4468.83 | 3.92 (0.1)
|  |  $\geq$55 | 4492.75 | 3.93 (0.1)
COMP(M3) | Female |  $<$55 | 4104.33 | 10.34 (0.39)
|  |  $\geq$55 | 4137.47 | 10.33 (0.39)
| Male |  $<$55 | 4304.29 | 10.33 (0.39)
|  |  $\geq$55 | 4274.46 | 10.34 (0.4)
$\Delta$ COMP(M3) | Female |  $<$55 | 4110.12 | 9.12 (0.26)
|  |  $\geq$55 | 4137.43 | 9.1 (0.26)
| Male |  $<$55 | 4287.56 | 9.13 (0.28)
|  |  $\geq$55 | 4263.45 | 9.1 (0.27)
EF(M3) | Female |  $<$55 | 4101.07 | 8.87 (0.37)
|  |  $\geq$55 | 4122.14 | 8.86 (0.4)
| Male |  $<$55 | 4304.95 | 8.86 (0.38)
|  |  $\geq$55 | 4274.40 | 8.89 (0.39)
$\Delta$ EF(M3) | Female |  $<$55 | 4105.48 | 7.72 (0.28)
|  |  $\geq$55 | 4125.95 | 7.72 (0.25)
| Male |  $<$55 | 4298.70 | 7.72 (0.25)
|  |  $\geq$55 | 4270.60 | 7.7 (0.24)
EM(M3) | Female |  $<$55 | 4104.97 | 3.69 (0.08)
|  |  $\geq$55 | 4137.07 | 3.69 (0.08)
| Male |  $<$55 | 4297.32 | 3.69 (0.09)
|  |  $\geq$55 | 4290.35 | 3.69 (0.1)
$\Delta$ EM(M3) | Female |  $<$55 | 4119.27 | 3.42 (0.09)
|  |  $\geq$55 | 4118.28 | 3.42 (0.09)
| Male |  $<$55 | 4286.56 | 3.42 (0.08)
|  |  $\geq$55 | 4273.48 | 3.43 (0.09)
Table 2: Breakdown of RMSE by sex and average M2 age. Sparsity is averaged by
subject, and represents the amount of missing predictor variables. RMSE is
averaged across all models. There was no significant difference in RMSE
between sex and ages.

###  3.2 Sparsity Effect

Figure 3b shows the effect of sparsity on predicting cognitive performance
across all tasks. The slopes of the linear trends appear different for DNN
models compared to the other model types, suggesting a possible interaction
effect of sparsity on model performance. We tested the interaction effect of
sparsity and model type on RMSE with a ANOVA for two linear regression models:
without a sparsity interaction term ($F_{36239,6}=327.10$,p $<$ 0.01) and with
a sparsity interaction term ($F_{36232,13}=184.50$,p $<$ 0.01) on model type.
We found a significant interaction effect ($F_{9}=59.20$,p $<$ 0.01),
indicating that DNN models are affected less by sparsity than the other models
we assessed.

Figure 4: 2-dimensional, t-SNE, visualization representing learned embeddings
of factor similarity. Each dot represents a 16-dimensional, factor embedding
learned during training and used as DNN input for prediction of test data. We
display the index of factors, and detail the corresponding factors and
measures in Table 3. Each factor was independently used for prediction, and we
averaged across prediction tasks. The inverse rank of each factor (i.e., its
importance) is represented by the diameter of the circle. Each color
represents the hierarchical category, and distance represents the relative
relatedness of each factor to others. Note that the cognitive factors (pink)
were only used when predicting (M3) and $\Delta$ cognitive measures.

Index | Factor | Project | Average Rank | Mean Inverse Rank
---|---|---|---|---
0 | Actigraphy | Biomarkers | 45.56 | 0.02
1 | Administration | Survey | 8.11 | 0.12
2 | Affect | Daily Diary | 32.67 | 0.03
3 | Assay Data | Biomarkers | 22.11 | 0.05
4 | Assistance | Daily Diary | 39.44 | 0.03
5 | Backward Counting | Cognitive | 26.67 | 0.04
6 | Caregiving | Survey | 34.56 | 0.03
7 | Category Fluency | Cognitive | 14.17 | 0.07
8 | Children | Survey | 29.22 | 0.03
9 | Cognitive Battery Factor Scores | Cognitive | 1.50 | 0.67
10 | Community Involvement | Survey | 20.00 | 0.05
11 | Cortisol | Daily Diary | 35.44 | 0.03
12 | Daily Discrimination | Daily Diary | 24.67 | 0.04
13 | Daily Medications | Daily Diary | 31.44 | 0.03
14 | Daily Stressors | Daily Diary | 37.33 | 0.03
15 | Delayed Word List Recall | Cognitive | 13.50 | 0.07
16 | Digits Backward | Cognitive | 17.33 | 0.06
17 | Disability Assistance | Daily Diary | 33.44 | 0.03
18 | Discrimination | Survey | 35.22 | 0.03
19 | Education, Occupation, and Marital Status | Survey | 15.78 | 0.06
20 | Emotional Support | Daily Diary | 39.89 | 0.03
21 | Finances | Survey | 13.78 | 0.07
22 | Health | Survey | 24.11 | 0.04
23 | Health Behaviors | Daily Diary | 31.22 | 0.03
24 | Health Insurance | Survey | 9.78 | 0.10
25 | Health Questions for Women | Survey | 26.89 | 0.04
26 | Household Roster and Children | Survey | 25.22 | 0.04
27 | Immediate Word List Recall | Cognitive | 13.67 | 0.07
28 | Life Overall | Survey | 35.00 | 0.03
29 | Life Satisfaction | Survey | 33.00 | 0.03
30 | Living Arrangements | Survey | 35.00 | 0.03
31 | Marriage or Close Relationship | Survey | 26.67 | 0.04
32 | Medical History | Biomarkers | 21.89 | 0.05
33 | Medication Chart | Biomarkers | 27.56 | 0.04
34 | Musculoskeletal | Biomarkers | 33.33 | 0.03
35 | Number Series | Cognitive | 24.17 | 0.04
36 | Parent’s Health | Survey | 21.44 | 0.05
37 | Personal Beliefs | Survey | 4.89 | 0.20
38 | Physical Exam | Biomarkers | 16.78 | 0.06
39 | Physical Symptoms | Daily Diary | 40.22 | 0.02
40 | Pittsburgh Sleep Questionnaire (PSQ) | Biomarkers | 32.11 | 0.03
41 | Positive Events | Daily Diary | 40.78 | 0.02
42 | Psychophysiology Protocol | Biomarkers | 29.22 | 0.03
43 | Race and Ethnicity | Survey | 33.22 | 0.03
44 | Religion and Spirituality | Survey | 32.22 | 0.03
45 | Scale Variables | Daily Diary | 31.78 | 0.03
46 | Self-Administrated Questionnaire (SAQ) | Biomarkers | 17.00 | 0.06
47 | Sexuality | Survey | 28.00 | 0.04
48 | Social Networks | Survey | 20.33 | 0.05
49 | Stop and Go Switch Task - Composite Scores | Cognitive | 22.33 | 0.04
50 | Time use | Daily Diary | 41.11 | 0.02
51 | Week Summary | Daily Diary | 30.67 | 0.03
52 | Work | Survey | 27.33 | 0.04
53 | Work Behaviors | Daily Diary | 33.56 | 0.03
54 | Your Health | Survey | 14.44 | 0.07
55 | Your Neighborhood | Survey | 22.78 | 0.04

Table 3: Embedding similarity and ranking. The diameter of the points in
Figure 4, i.e., importance towards prediction, was calculated as the inverse
of the average rank across variables.

###  3.3 Embedding Similarity

During training for each prediction task, the $embed\\_dnn$ model learned the
16-dimensional embedding of related variables, i.e., factors. Using t-SNE to
visualize the factor embeddings, we show each factor on a reduced,
2-dimensional plot, colored by its hierarchical group, using labels specified
by the MIDUS project (Figure 4). We sized each factor in the plot by its
independent influence on decreasing RMSE. Cognitive measures were only used in
the prediction of raw and $\Delta$ M3 measures. In other words, we used M2
cognitive measures as variables when predicting M3 cognitive performance. No
cognitive variables were used in cross-sectional prediction of M2 cognitive
measures. Corresponding labels of indices and average rank measures are
detailed in Table 3.

We noted some clustering of cognitive measures in their embedding
representation, and their relatively large influence when used to predict
future cognitive performance. On average, the most important factors were
”Administrative” or demographic in nature (i.e., age and gender), related to
education, occupation, marital status, or self-described psychological well-
being and subjective age (”Personal Beliefs”). Questions related to menopause,
and hormonal drugs were also important in predicting cognitive performance for
women. Additionally, questions related to participants’ health insurance
status (e.g., whether they were covered by private health insurance or
Medicare) were important in prediction. While questions related to medical
history and biomarker data (e.g., ”Cortisol”) possibly improved prediction
differently than other factors containing variables describing medications,
medical history, psychophysiology, biomarker sleep quality and physical exams
generally ranked lower. Specifically, the top five factors that most improved
prediction (average rank $\leq$ 13.5), in order, were (1) Cognitive Battery
Factor Scores, (2) Personal Beliefs, (3) Administration, (4) Health Insurance,
(5) Delayed Word List Recall. The bottom five factors that least improved
prediction (average rank $\geq$ 39.5), in order, were (1) Actigraphy, (2) Time
use, (3) Positive Events, (4) Physical Symptoms, (5) Emotional Support.

Interestingly, cognitive factors such as the ”Immediate Word List Recall” and
”Stop and Go Switch Task” measures did not clearly cluster together. This may
indicate that the DNN models interpreted these factors as independently
contributing to better prediction of cognitive performance relative to other
cognitive measures.

##  4 Discussion

In this study, we developed a modular approach to improving prediction of
cognitive performance in a sparse, heterogeneous dataset, which may be used to
diagnose conditions that rely on accurate executive function and episodic
memory measurements, such as ADHD, MCI and AD. By using a large dataset
containing behavioral, biological and psychosocial data, we reported the
relative advantage of deep neural networks, compared to linear and other non-
linear models, in consistently improving prediction of cognitive scores, under
varying levels of data sparsity. Through evaluating cognitive performance
prediction using over 5000 predictor variables, across 9 prediction tasks, we
found the DNN framework produced the most accurate and consistent results. Our
cross-validation approach to measure mean performance and variance of errors
across frameworks did not produce significant differences between sex and age
groups.

The DNN performance yielded lower RMSEs in all prediction tasks compared to
other models (Figure 3a). Because prediction of change in cognitive
performance is often a more challenging problem, deep architectures may be
preferable to other frameworks, as they are more likely to model the
complicated, temporal relationships between behavioral, physiological and
other factors that affect cognitive decline.

The consistency in the rankings of DNNs over other model types we tested may
be due to their relative advantage under data sparsity. By showing a
significant interaction between model type and sparsity in prediction
cognitive performance, we show that deep networks may be less affected by
varying levels of sparsity than linear or other non-linear models. This is
visualized by the relatively flatter slopes of DNN and $embed\\_dnn$ in Figure
3b. The methods we employed to control for overfitting may have allowed better
generalization and robustness of DNNs, relative to other model types we
assessed.

While we expected factor embeddings to outperform linear methods, we were
surprised to find that, for all of the assessed prediction tasks, using
embeddings yielded more accurate predictions than the DNN with access to the
full dataset on average, although this finding not statistically significant.
However, we achieved similar accuracy despite a tenfold reduction to the
number of input variables.

By providing the DNN with relevant information in how similar variables are to
each other, we allowed meaningful context when predicting cognitive
performance, and achieved better prediction accuracy with fewer parameters and
in fewer iterations. This finding is similar to other observations for DNNs,
where hyperparameter design choice with the dataset in mind both improves
accuracy and training time [36]. Expert clinical guidance on how variables are
grouped prior to prediction in a deep network may thus improve performance in
and interpretability of similar datasets.

We believe our pattern of results is related to the sparseness and variability
present in the full MIDUS-AGG dataset, which is less of an issue when first
grouping variables before training. In fact, architectures designed with
characteristics of sparse data in mind during design tend to optimize faster
and avoid local minima [37]. We hypothesize that, given a low enough learning
rate and long enough training time, our DNN using the full dataset would
eventually outperform the DNN using embeddings as input. Our DNN performance
may also be improved by inclusion of residual connections that may yield loss
functions that are easier to minimize [38].

We recommend future study to understand how the size and sparseness of
clinical data may influence the rate of learning, which may have resulted in
this finding. Prediction using meaningful groupings of variables is of
interest to clinicians and researchers who work with large, sparse datasets
like MIDUS-AGG. Working with embeddings allows prediction models to scale
efficiently as dataset sizes grow in volume. Through improving prediction of
cognitive performance, clinicians and researchers, often constrained to
working with sparse data, may be better able to diagnose a host of clinical
conditions that rely on accurate measurement of cognitive performance.

##  5 Conclusion

In this evaluation of deep neural architectures to improve prediction of
cognitive performance, we found robust performance of DNNs under multiple,
cross-sectional and longitudinal, measures. We show that the robustness can
partly be attributed to how DNNs handle data sparsity compared with linear and
other non-linear models. Using factor-level, low-dimensional embeddings, we
find no drop in model accuracy while allowing clinical interpretability of
variables impacting executive and episodic function. Our results highlight the
potential of DNNs to improve clinical understanding of age-related cognitive
decline and in the analysis of large, mixed datasets.

## Acknowledgement

This work was supported by grants from the National Science Foundation
(IIS-1513853, IIS-1816363), the Army Research Laboratory Cooperative Agreement
W911NF-10-2-0022, a Vannevar Bush Faculty Fellowship from the US Department of
Defense (N00014-20-1-2027), and National Institute of Aging (P01-AG020166 and
U19-AG051426).

## References

  * [1] Göran Söderlund, Sverker Sikström and Andrew Smart  “Listen to the noise: noise is beneficial for cognitive performance in ADHD”  In _Journal of Child Psychology and Psychiatry_ 48.8 Wiley Online Library, 2007, pp. 840–847
  * [2] Kristiaan B Van Heijden et al.  “Effect of melatonin on sleep, behavior, and cognition in ADHD and chronic sleep-onset insomnia”  In _Journal of the American Academy of Child & Adolescent Psychiatry_ 46.2 Elsevier, 2007, pp. 233–241
  * [3] Carolyn Baum and Dorothy F Edwards  “Cognitive performance in senile dementia of the Alzheimer’s type: The Kitchen Task Assessment”  In _American Journal of Occupational Therapy_ 47.5 American Occupational Therapy Association, 1993, pp. 431–436
  * [4] Marwan N Sabbagh et al.  “Early detection of Mild Cognitive Impairment (MCI) in primary care”  In _The Journal of prevention of Alzheimer’s disease_ Springer, 2020, pp. 1–6
  * [5] PM Doraiswamy et al.  “The Alzheimer’s Disease Assessment Scale: patterns and predictors of baseline cognitive performance in multicenter Alzheimer’s disease trials”  In _Neurology_ 48.6 AAN Enterprises, 1997, pp. 1511–1517
  * [6] Maira Rozenfeld Olchik et al.  “Memory training (MT) in mild cognitive impairment (MCI) generates change in cognitive performance”  In _Archives of gerontology and geriatrics_ 56.3 Elsevier, 2013, pp. 442–447
  * [7] Kate E Devenney et al.  “The effects of an extensive exercise programme on the progression of Mild Cognitive Impairment (MCI): study protocol for a randomised controlled trial”  In _BMC geriatrics_ 17.1 Springer, 2017, pp. 75
  * [8] Connie A. Tompkins et al.  “Predicting cognitive recovery from closed head-injury in children and adolescents”  In _Brain and Cognition_ 13.1, 1990, pp. 86–97  DOI: 10.1016/0278-2626(90)90042-M
  * [9] Matthew L. Hughes et al.  “Change in Cognitive Performance from Midlife into Old Age: Findings from the Midlife in the United States (MIDUS) Study”  In _Journal of the International Neuropsychological Society_ 24.8, 2018, pp. 805–820  DOI: 10.1017/S1355617718000425
  * [10] Anita Lill Hansen, Bjørn Helge Johnsen and Julian F. Thayer  “Vagal influence on working memory and attention”  In _International Journal of Psychophysiology_ 48.3, 2003, pp. 263–274  DOI: 10.1016/S0167-8760(03)00073-4
  * [11] Thomas G. Raslear, Steven R. Hursh and Hans P.A. Van Dongen  “Predicting cognitive impairment and accident risk”  In _Progress in Brain Research_ 190 Elsevier B.V., 2011, pp. 155–167  DOI: 10.1016/B978-0-444-53817-8.00010-4
  * [12] Mike O’Sullivan et al.  “Hippocampal volume is an independent predictor of cognitive performance in CADASIL”  In _Neurobiology of Aging_ 30.6, 2009, pp. 890–897  DOI: 10.1016/j.neurobiolaging.2007.09.002
  * [13] Eileen K. Graham and Margie E. Lachman  “Personality stability is associated with better cognitive performance in adulthood: Are the stable more able?”  In _Journals of Gerontology - Series B Psychological Sciences and Social Sciences_ 67 B.5, 2012, pp. 545–554  DOI: 10.1093/geronb/gbr149
  * [14] Ramon Casanova et al.  “Investigating predictors of cognitive decline using machine learning”  In _Journals of Gerontology - Series B Psychological Sciences and Social Sciences_ 75.4, 2020, pp. 733–742  DOI: 10.1093/geronb/gby054
  * [15] Alexander Ilin and Tapani Raiko  “Practical approaches to principal component analysis in the presence of missing values”  In _Journal of Machine Learning Research_ 11, 2010, pp. 1957–2000
  * [16] Graeme Hawthorne and Peter Elliott  “Imputing cross-sectional missing data: Comparison of common techniques”  In _Australian and New Zealand Journal of Psychiatry_ 39.7, 2005, pp. 583–590  DOI: 10.1111/j.1440-1614.2005.01630.x
  * [17] Pedro J. García-Laencina, Pedro Henriques Abreu, Miguel Henriques Abreu and Noémia Afonoso  “Missing data imputation on the 5-year survival prediction of breast cancer patients with unknown discrete values”  In _Computers in Biology and Medicine_ 59, 2015, pp. 125–133  DOI: 10.1016/j.compbiomed.2015.02.006
  * [18] Shona Fielding, Peter M. Fayers and Craig R. Ramsay  “Investigating the missing data mechanism in quality of life outcomes: A comparison of approaches”  In _Health and Quality of Life Outcomes_ 7, 2009, pp. 1–10  DOI: 10.1186/1477-7525-7-57
  * [19] Orville Gilbert; Brim, Carol D; Ryff and Ronald C Kessler  “The MIDUS national survey: An overview”  In _How healthy are we_ , 2004, pp. 1–36
  * [20] Carol Ryff et al.  “Midlife in the United States (MIDUS 2), 2004-2006”  Inter-university Consortium for PoliticalSocial Research [distributor], 2017  DOI: 10.3886/ICPSR04652.v7
  * [21] Barry T. Radler and Carol D. Ryff  “Who Participates? Accounting for Longitudinal Retention in the MIDUS National Study of Health and Well-Being”  In _Journal of Aging and Health_ 22.3, 2010, pp. 307–331  DOI: 10.1177/0898264309358617
  * [22] Patricia A ; Tun and Margie E Lachman  “Telephone assessment of cognitive function in adulthood: the Brief Test of Adult Cognition by Telephone”  In _Age and Ageing_ 35.6, 2006, pp. 629–632  DOI: 10.1093/ageing/afl099
  * [23] Margie E. Lachman, Stefan Agrigoroaei, Patricia A. Tun and Suzanne L. Weaver  “Monitoring Cognitive Functioning: Psychometric Properties of the Brief Test of Adult Cognition by Telephone”  In _Assessment_ 21.4, 2014, pp. 404–417  DOI: 10.1177/1073191113508807
  * [24] Douglas M. Hawkins  “The Problem of Overfitting”  In _Journal of Chemical Information and Computer Sciences_ 44.1, 2004, pp. 1–12  DOI: 10.1021/ci0342472
  * [25] Jerome H. Friedman  “On bias, variance, 0/1-loss, and the curse-of-dimensionality”  In _Data Mining and Knowledge Discovery_ 1.1, 1997, pp. 55–77  DOI: 10.1023/A:1009778005914
  * [26] Fabian Pedregosa, Ron Weiss and Matthieu Brucher  “Scikit-learn : Machine Learning in Python”, 2011, pp. 2825–2830
  * [27] Harsha Nori, Samuel Jenkins, Paul Koch and Rich Caruana  “InterpretML: A Unified Framework for Machine Learning Interpretability”, 2019, pp. 1–8  URL: http://arxiv.org/abs/1909.09223
  * [28] Rich Caruana et al.  “Intelligible Models for HealthCare”, 2015, pp. 1721–1730  DOI: 10.1145/2783258.2788613
  * [29] Boris Hanin  “Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients ?”, 2018, pp. 1–10
  * [30] Xin Rong  “word2vec Parameter Learning Explained”, 2014, pp. 1–21  URL: http://arxiv.org/abs/1411.2738
  * [31] Yoshua Bengio et al.  “Learning Eigenfunctions Links Spectral Embedding and Kernel PCA”  In _Neural Computation_ 16.10, 2004, pp. 2197–2219  DOI: 10.1162/0899766041732396
  * [32] Hervé Bredin  “Tristounet: triplet loss for speaker turn embedding”  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2017, pp. 5430–5434  IEEE
  * [33] AS Barnard and G Opletal  “Predicting structure/property relationships in multi-dimensional nanoparticle data using t-distributed stochastic neighbour embedding and machine learning”  In _Nanoscale_ 11.48 Royal Society of Chemistry, 2019, pp. 23165–23172
  * [34] Raviteja Vemulapalli and Aseem Agarwala  “A compact embedding for facial expression similarity”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 5683–5692
  * [35] Laurens; Maaten and Geoffrey Hinton  “Visualizing Data using t-SNE”  In _Journal of Machine Learning Research_ , 2008, pp. 2579–2605  DOI: 10.1007/s10479-011-0841-3
  * [36] Leslie N Smith  “Cyclical learning rates for training neural networks”  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2017, pp. 464–472  IEEE
  * [37] John C. Duchi, Michael I. Jordan and H. McMahan  “Estimation, optimization, and parallelism when data is sparse”  In _Advances in Neural Information Processing Systems_ , 2013, pp. 1–9
  * [38] Hao Li et al.  “Visualizing the loss landscape of neural nets”  In _Advances in Neural Information Processing Systems_ 2018-Decem.Nips 2018, 2018, pp. 6389–6399

## Supplementary material

Project | Factors | Categorical Variables | Numerical Variables | Total Variables
---|---|---|---|---
| Administration | 3 | 3 | 6
| Caregiving | 20 | 5 | 25
| Children | 16 | 8 | 24
| Community Involvement | 30 | 59 | 89
| Discrimination | 21 | 13 | 34
| Education, Occupation, and Marital Status | 71 | 29 | 100
| Finances | 34 | 35 | 69
| Health | 231 | 43 | 274
| Health Insurance | 42 |  | 42
| Health Questions for Women | 53 | 4 | 57
Survey | Household Roster and Children | 256 | 68 | 324
| Life Overall |  | 6 | 6
| Life Satisfaction | 10 |  | 10
| Living Arrangements | 11 | 2 | 13
| Marriage or Close Relationship | 36 | 21 | 57
| Parent’s Health | 6 | 4 | 10
| Personal Beliefs | 310 | 95 | 405
| Race and Ethnicity | 71 |  | 71
| Religion and Spirituality | 48 | 8 | 56
| Sexuality | 6 | 6 | 12
| Social Networks | 52 | 7 | 59
| Work | 108 | 31 | 139
| Your Health (Self Evaluation) | 294 | 55 | 349
| Your Neighborhood (Self Evaluation) | 16 | 4 | 20
| Group Total | 1745 | 506 | 2251
Cognitive |  |  |  |
BTACT |  | 10 | 10
Group Total |  | 10 | 10
Daily Diary |  |  |  |
Affect | 27 |  | 27
Assistance | 52 | 4 | 56
Cortisol | 4 | 4 | 8
Daily Discrimination | 21 |  | 21
Daily Medications | 10 |  | 10
Daily Stressors | 129 | 14 | 143
Disability Assistance | 26 | 2 | 28
Emotional Support | 51 | 4 | 55
Health Behaviors | 2 |  | 2
Physical Symptoms | 56 |  | 56
Positive Events | 20 | 10 | 30
Scale Variables | 13 | 9 | 22
Time use | 13 | 16 | 29
Week Summary | 35 |  | 35
Work Behaviors | 8 |  | 8
| Group Total | 467 | 63 | 530
Biomarkers |  |  |  |
Actigraphy | 128 | 217 | 345
Assay Data | 1 | 62 | 63
Medical History | 493 | 94 | 587
Medication Chart | 204 | 81 | 285
Musculoskeletal | 6 | 17 | 23
Physical Exam | 263 | 42 | 305
Pittsburgh Sleep Questionnaire (PSQ) | 22 | 10 | 32
Psychophysiology Protocol | 32 | 153 | 185
Self-Administrated Questionnaire (SAQ) | 405 | 43 | 448
| Group Total | 1554 | 719 | 2273
Total Counts | 3766 | 1298 | 5064

Table 4: Variables used in the present study, by groups. Groups were utilized for factor analysis and embeddings in the DNN. Model A | Model B | T | dof | p-corrected
---|---|---|---|---
embed_dnn | dnn | 2.83 | 8 | 0.11
embed_dnn | lasso_regression | -5.08 | 8 | 0.02*
embed_dnn | xgboost | -5.17 | 8 | 0.02*
embed_dnn | linear_regression | -5.75 | 8 | 0.01**
embed_dnn | random_forest | -5.74 | 8 | 0.01**
embed_dnn | ridge_regression | -5.77 | 8 | 0.01**
dnn | lasso_regression | -3.70 | 8 | 0.04*
dnn | random_forest | -4.23 | 8 | 0.03*
dnn | ridge_regression | -4.86 | 8 | 0.02*
dnn | linear_regression | -4.88 | 8 | 0.02*
dnn | xgboost | -5.04 | 8 | 0.02*
ridge_regression | lasso_regression | 5.14 | 8 | 0.02*
ridge_regression | random_forest | 4.45 | 8 | 0.02*
ridge_regression | xgboost | -1.40 | 8 | 0.45
ridge_regression | linear_regression | -2.43 | 8 | 0.17
linear_regression | lasso_regression | 5.12 | 8 | 0.02*
linear_regression | random_forest | 4.45 | 8 | 0.02*
linear_regression | xgboost | -1.36 | 8 | 0.45
lasso_regression | random_forest | 1.59 | 8 | 0.45
lasso_regression | xgboost | -3.37 | 8 | 0.06
random_forest | xgboost | -3.90 | 8 | 0.04*
Table 5: Paired T-Tests: Pairwise, parametric, T-tests between model types.
The Holm-Bonferroni correction was used to judge significance. * $\leq$ 0.05,
** $\leq$ 0.01


#  Ego-Motion Estimation and Dynamic Motion Separation from 3D Point Clouds
for Accumulating Data and Improving 3D Object Detection

Patrick Palmer  TU Dortmund  Institute of Control Theory and Systems
Engineering  44227 Dortmund
<EMAIL_ADDRESS>
Martin Krüger  TU Dortmund  Institute of Control Theory and Systems
Engineering  44227 Dortmund
<EMAIL_ADDRESS>
Dr. Richard Altendorfer  ZF Group  56070 Koblenz<EMAIL_ADDRESS>
Univ.-Prof. Dr.-Ing. Prof. h.c. Dr. h.c. Torsten Bertram  TU Dortmund
Institute of Control Theory and Systems Engineering  44227 Dortmund
<EMAIL_ADDRESS>
###### Abstract

New 3+1D high-resolution radar sensors are gaining importance for 3D object
detection in the automotive domain due to their relative affordability and
improved detection compared to classic low-resolution radar sensors. One
limitation of high-resolution radar sensors, compared to lidar sensors, is the
sparsity of the generated point cloud. This sparsity could be partially
overcome by accumulating radar point clouds of subsequent time steps. This
contribution analyzes limitations of accumulating radar point clouds on the
View-of-Delft dataset [1]. By employing different ego-motion estimation
approaches, the dataset’s inherent constraints, and possible solutions are
analyzed. Additionally, a learning-based instance motion estimation approach
is deployed to investigate the influence of dynamic motion on the accumulated
point cloud for object detection. Experiments document an improved object
detection performance by applying an ego-motion estimation and dynamic motion
correction approach.

##  1 Introduction

One of the critical challenges of automating vehicles and the driving process
is the perception of the environment. Precise knowledge of the traffic scene
is necessary to make well-informed decisions on the automated vehicle’s path
planning and react adequately to sudden actions of other traffic participants.
False, missing or imprecise detections entail errors in the environment model.
This increases the risk of accidents and limits the time horizon for safe and
comfortable vehicle path planning. Different sensor modalities are utilized in
research and production vehicles for environment perception. Research vehicles
mainly use high-resolution lidars due to their high information density and
accuracy, whereas series production vehicles primarily use low-cost sensors
like radars and cameras. An emerging sensor technology that is supposed to
bridge the gap between lidar and traditional low-resolution radar sensors are
3+1D high-resolution radar sensors. Compared to traditional radars, these
high-resolution radar sensors also measure the elevation angle and generate a
denser point cloud while preserving the advantages of radar sensors like the
direct estimation of the relative radial velocity $v_{rr}$, robustness against
adverse weather conditions, and relatively low cost. Despite improvements in
radar technology, currently available 3+1D radar sensors still suffer from
noisy measurements and relatively sparse point clouds (compared to lidar).
These limitations constrain the perception performance and thus affect the
following modules (and their performance) in an automated driving stack. A
common strategy to overcome the sparsity of low-resolution lidar and radar
point clouds is the aggregation of information over concurrent time steps by
accumulation. This yields a denser point cloud that can be used for perception
tasks like 3D object detection. For static objects, an accumulation can be
done by transforming point positions from the previous to the current
coordinate frame using the ego-motion alone. Nowadays, ego-motion is mostly
estimated by combining GPS, wheel odometry, and inertial measurements (angular
rates and accelerations) and is available for most public datasets. Targets
from dynamic objects, such as reflections from cars, pedestrians, and
cyclists, must be considered separately. A naive accumulation of these points
based on the ego-motion alone results in an error, represented by trailing
points behind the object. An example can be seen in Fig. 1, the bicycle on the
bottom left (ID: VIII) has a tail of points outside the bounding box. Due to
the radar sensor’s direct measurement of radar radial velocity $v_{rr}$, an
accurate distinction between static and dynamic points is possible [2]. The
$v_{rr}$ can additionally be used for estimating the velocity over ground of
objects. Considering the motion of dynamic objects when accumulating point
clouds from subsequent frames leads to improved consistency of the accumulated
point cloud. Hence, the accuracy of object detection approaches applied to the
point cloud increases [3].

Figure 1: Bird’s eye view perspective of a traffic scene visualizing an
aggregated point cloud, accumulated over five frames. The ego-vehicle is
located on the left, represented by the colored coordinate system indicating
the sensor mounting position (viewing towards the right). Orange boxes
represent annotated ground truth bounding boxes and the purple rectangle
indicates a manually added ground truth bounding box. The roman numerals mark
the objects identifier (ID). The arrow represents the direction of motion.

Related Work: For lidar [4, 5] and 3+1D high-resolution radar data [6] flow-
based methods, that only utilize the point coordinates, can accurately
estimate the scene flow for a point cloud. The point cloud can be separated
into static and dynamic areas using the scene flow. Static points can then be
used to estimate the ego-motion, while the instance’s motion can be derived
from the dynamic points. By extending the considered time horizon and
employing motion segmentation [7] (PCAc) has shown that accurate accumulation
of point clouds is possible given only the naively accumulated point clouds.

As summarized by [8], multiple 3+1D radar datasets have been proposed, each
suffering from unique limitations, making most of them unfeasible for the
ultimate goal of 3D object detection. The View-of-Delft (VoD) dataset [1]
meets the requirements. It provides time-continuous labeled objects, non-
accumulated radar data, and additional sensor modalities like high-resolution
lidar and camera sensor data.

The main contributions of the work at hand are:

  * •

Evaluating ego-motion estimation approaches to accumulate high-resolution
radar point clouds on subsequent frames on the VoD dataset.

  * •

Adaptation of PCAc [7] approach to the VoD dataset and evaluation of the
transferability to radar data.

  * •

Investigating the influence of different dynamic and static motion correction
approaches on the performance of 3D object detection.

##  2 Limitations of the VoD dataset

The VoD dataset is one of very few datasets that include 3+1D high-resolution
radar measurements. In addition to the radar sensor, mono camera and high-
resolution 64-layer lidar data are provided. Due to the dataset’s focus on
urban scenarios where different types of traffic participants share the same
space, only a limited view range of up to 51.2m is considered, and the amount
of observed pedestrians and cyclist is high compared to other datasets like
the Astyx dataset [9]. One major limitation of the dataset is the accuracy of
labeled objects. Multiple objects are either not labeled or mislabeled. An
example is visualized in Fig. 1. In the lower part of the image are three
bicycles. The foremost one (ID: VIII) is labeled correctly. The middle one
(ID: VII) is not labeled at all, and the label for the sternmost one (ID: VI)
is not well aligned to the cluster of close targets. The questionable quality
of such annotations negatively influences the training because misaligned or
missing annotations confuse the training procedure. Due to the time and effort
necessary for correcting annotations, only minor label adaptations have been
applied to mitigate the problem of object reflection points that lay outside
the bounding box. The box is extended by $0.2\text{\,}\mathrm{m}$ in each
direction to capture most of the points belonging to an object, similar to
[10].

The second observed limitation is the unavailability of ground truth ego-
motion information. The authors provide an accumulated set of radar scans but
do not provide the utilized ego-motion for generating them. This limits the
ability to investigate dynamic object motion. Additionally, the reproduction
of the accumulated scans is hindered by different sampling frequencies of the
radar sensor and the provided single radar scans. The utilized sensor captures
the scene with a frequency of $13\text{\,}\mathrm{Hz}$, but the single-frame
data is only provided with a frequency of $10\text{\,}\mathrm{Hz}$, to
synchronize the measurement of the radar to the other sensor modalities. The
provided accumulated point cloud is collected from the base
$13\text{\,}\mathrm{Hz}$ data. Certain radar scans are therefore not provided
and the time interval between the provided frames varies. Different strategies
of estimating ego-motion are investigated in the following chapter to mitigate
the missing ego-motion information.

### Ego-Motion Estimation

The ego-vehicle motion is described as a transformation matrix that consists
of the translation and rotation of the ego-vehicle between two consecutive
time steps. It can be derived either from an inertial measurement-based system
or directly from environmental measurement sensors.

Ego-Motion-Estimation from Pose (EM-P): The VoD dataset does not directly
provide information about the ego-motion. However, transformation matrices
from the camera coordinate system to a map and a world coordinate system are
provided. Both transformations only differ in the definition of the point of
origin. Hence, the ego-motion is calculated from the transformation between
consecutive frames.

Ego-Motion-Estimation from Point Cloud with GICP (EM-G): Due to limitations
observed in the ego-motion from the VoD pose estimation, a method for
estimating the motion directly from the recorded point cloud is investigated.
Generalized Iterative Closest Point (GICP)[11] is chosen due to its simple
implementation and the assumed sufficient accuracy. It is applied to the lidar
and high-resolution radar point cloud separately.

Averaging of estimated motion over multiple time steps (EM-mG): Due to the
high oscillation of the EM-G output, as can be seen in Fig. 2, an averaging of
the transformation matrix is employed to reduce implausible fluctuations. The
average of the translation and rotation matrices are calculated separately.
Due to an observed pattern that repeats every 6 time steps, the translation is
averaged by calculating the moving average over the translation at 6
consecutive time steps. An averaging of the rotation matrix is not trivial.
One way of averaging utilized in this work is to transform the rotation matrix
into a quaternion representation. Then, the average quaternion is calculated
using singular value decomposition (SVD) [12], over the same 6 consecutive
time steps. Afterward, the quaternion is re-transformed into the rotation
matrix.

Estimation from measured radar radial velocity (EM-$v_{rr}$): One unique
feature of the radar point cloud that can be utilized to estimate ego-motion
is the direct measurement of $v_{rr}$. The ego-motion can be estimated by
applying a least squares-based approach to the static points in the scene, as
described in [13]. The main difference to alternative approaches is, that only
measurements from a single frame are needed. The differentiation of static and
dynamic points is done by estimating the static points using a random sample
consensus algorithm [14]. Due to the additional information about the
elevation angle $\varepsilon$ measured by the 3+1D radar sensor, the
observation angle $\varphi$ is utilized, instead of the azimuth angle
$\vartheta$. Calculated using the spherical Pythagorean theorem:

$\varphi=\text{cos}^{-1}(\text{cos}(\vartheta)\text{cos}(\varepsilon))$ (1)

Estimation from static objects (EM-SO): The VoD dataset includes labels and
track identifiers for some static objects like bicycle racks and unused
bicycles. The track identifiers enable the re-identification of objects in
consecutive frames. Assuming noise-free data, the rotation $R$ and translation
$t$ can be derived by solving:

$RC_{k-1}+t=C_{k}$ (2)

Where $C$ represents the center points of all objects at time step $k-1$ and
$k$. Due to noise in the data, a least squares-based approach is used to
derive the motion. Unfortunately, static objects do not occur in every frame.
That limits the usefulness and accuracy of this method significantly and also
makes it dependent on the accuracy of the annotation of static objects. Hence,
this approach is only considered as a baseline for comparing the previous
methods against.

##  3 Treatment of Dynamic Objects

Due to the motion of certain objects an accumulation of the point cloud by
just considering ego-motion results in an error for moving objects. For the
foremost bicycle on the bottom left in Fig. 1, this is represented by the
smearing pattern of points behind the object. This negatively influences the
training as objects get represented differently depending on their motion.

Estimation from the provided labels (Dyn-GT): For a baseline approach the
motion of objects can be derived from the labels by estimating the
transformation between the center points of objects. This is the best-case
baseline to which the other approaches will be compared.

Relative Radial Velocity (Dyn-$v_{rr}$): If an object only moves in a radial
direction to the radar sensor, the full motion of the object can be measured
accurately. Unfortunately, objects rarely just move in radial direction, which
limits this approach. Nevertheless, a correction by $v_{rr}$ will result in a
good motion correction for all vehicles that move in the same or opposite
direction to the ego-vehicle.

Learning-Based Estimation of Dynamic Motion (Dyn-PCAc): To directly estimate
the dynamic object motion from the point cloud, PCAc [7] is adapted to the VoD
dataset. PCAc [7] is a supervised end-to-end learning-based algorithm that
first estimates the ego-motion (EM-PCAc) of the vehicle by separating the
scene into static and dynamic areas and employing robust point matching on the
static points. Afterward, motion segmentation and instance association steps
are executed to derive the instance motion and correct the point’s position in
the accumulated point cloud. When using the estimated ego-motion and the
labels of dynamic objects, a ground truth motion set can be derived. This is
used during training.

$0$$0.5$$1$EM-PEM-G-LidarEM-mG-LidarEM-G-Radar012345678$0$$0.5$$1$Time [s]EM-
PEM-SOEM-$v_{rr}$Longitudinal translation between consecutive frames [m]
Figure 2: Translation between frames in driving direction estimated from
various sensor modalities for an exemplary scene with an approximately
constant velocity of $15\text{\,}\mathrm{km}\text{\,}{\mathrm{h}}^{-1}$ in
driving direction.

##  4 Experimental Evaluation

For the overall goal of improving the object detection performance the
evaluation of the accuracy of the ego-motion estimation as well as the
instance motion assignment are analyzed by investigating their influence on
the object detection performance. PointPillars [15] is used as the 3D object
detection method as implemented in the OpenPCDet framework [16]. Minor
adaptations to the model configuration were made, as described by [1].
PointPillars is chosen over other object detectors due to its popularity for
radar-based perception methods, see e.g. [1]. In previous investigations [17]
it has shown to provide at least comparable, if not better detection
performance than other detectors while also being computationally efficient.

Training Details: All experiments are carried out with three different
reproducible model initializations to reduce performance dependencies on the
initialization. The model is trained for at most 125 epochs utilizing a one-
cycle learning rate scheduler [18]. Due to observations that the bulk of
training progress happens at low learning rates, the learning rate scheduler
is adapted to spend more time at lower learning rates while preserving the
one-cycle character. For the VoD test set no labels are provided as the
authors intend to provide an evaluation server later, similar to the KITTI
dataset [19]. Hence, we used the validation set for evaluation.

Evaluation Metrics: Similar to KITTI [19] and [1], we use Average Precision
(AP) as the primary performance metric but require a higher minimum
intersection over union (IoU) of [70%, 50%, 50%] in [x, y, z] direction for
cars and [50%, 50%, 50%] for pedestrians and cyclists. This stricter
requirement emphasizes the improved accuracy of the detection position by
dynamic motion handling as well as the performance in object height detection
by a more significant difference between 3D and 2D Bird’s eye view (BEV)
evaluation. The mean Average Precision (mAP) is the mean of the three class-
wise APs. The results are specified for two different range regions, short-
range (SR): evaluating all objects at distance of
$0\text{\,}$-$30\text{\,}\mathrm{m}$ from the ego-vehicle and long-range (LR):
evaluating all objects at distances $>30\text{\,}\mathrm{m}$. The symmetric
Chamfer Distance (sCD) quantifies the distance between (static) points after
ego-motion correction for two consecutive time steps and is used as a metric
to represent the error of estimated ego-motion. Due to the noisy radar point
cloud, this metric is calculated utilizing the lidar point cloud.

###  4.1 Ego-Motion Estimation Results

Table 1: 3D object detection results on the VoD dataset quantified as mAP and sCD between the current point cloud and the ego-motion corrected point cloud from the previous time step. All results, irrespective of the ego-motion (EM) correction approach, are generated by applying PointPillars on radar data accumulated over 5 frames, ignoring the motion of dynamic objects. The best results are marked in bold. The arrow marks direction of better score. |  sCD [m] $\downarrow$ | mAP $\uparrow$
---|---|---
EM estimation approach |  | 3D | BEV
|  SR |  LR |  SR |  LR
No correction |  0.285 |  32.0 |  10.1 |  42.9 |  22.6
EM-P |  0.191 |  35.1 |  12.9 |  48.6 |  28.8
EM-G-Lidar |  0.180 |  38.4 |  14.2 |  50.5 |  29.6
EM-mG-Lidar |  0.235 |  33.5 |  9.4 |  46.3 |  24.8
EM-G-Radar |  0.296 |  32.3 |  12.2 |  44.1 |  25.6
EM-$v_{rr}$ |  0.235 |  32.9 |  11.9 |  45.6 |  27.7
EM-PCAc |  0.216 |  38.7 |  11.4 |  48.5 |  25.4

From Fig. 2, two categories of ego-motion patterns can be differentiated.
Approaches resulting in a smooth motion estimation, EM-mG-Lidar and
EM-$v_{rr}$, and algorithms with large deviations between frames. All
deviating motion estimations show a similar pattern, most visible for the EM-
G-Lidar estimated motion. This pattern repeats every 6 time steps. An
explanation for this might be a time synchronization issue in data recording.
This could also explain the apparent noise in the motion estimated from EM-P.
Unfortunately, further investigations of this effect are impossible given the
provided data. The amount of noise deviates between the different estimation
strategies. Visual analysis shows that GICP-radar-based motion is most noise-
prone. This can also be observed regarding the sCD between time steps in Tab.
1. Employing EM-G-Radar for correction results in larger and therefore worse
sCD compared to the uncorrected data. The EM-G-Radar reflects the time
synchronization issue, and the inherent noisy measurements of the radar point
cloud, which results in an insufficient correction in regard to the lidar
point cloud, while still improving the object detection on radar data over no
correction. The robust point matching utilized in PCAc yields a better motion
estimation from the radar point cloud. It has a significantly lower sCD and
higher mAP for the SR 3D-mAP but cannot reach the EM-P and EM-G-Lidar
estimation performance. Comparing the sCD in Tab. 1 to the mAP shows that a
low sCD corresponds to a better object detection performance. All approaches
improve on the uncorrected baseline. For EM-mG-Lidar and EM-$v_{rr}$, it is
observable that the performance is low compared to the other approaches. This
strengthens the assumption that there is an inherent synchronization
limitation.

###  4.2 Influence of the Accumulation Horizon

Different time horizons for point cloud accumulation are investigated to
validate the aggregation with respect to the provided single frame data and to
investigate the effects of longer time horizons for dynamic object handling.
From Tab. 2, it is apparent that a large part of the improvement is already
reached when accumulating just 3 frames. Accumulating more frames can result
in better performance for some of the considered metrics but is diminished by
worse performance in others. Accumulation over 5 frames balances good
performance in SR and LR. This is especially apparent when the dynamic motion
of objects is considered. A longer time horizon of 10 frames only improves the
SR BEV performance, while in other cases, a shorter accumulation horizon
results in better performance.

Table 2: 3D object detection results on the VoD dataset quantified as mAP. All models, irrespective of the accumulation horizon are generated by applying PointPillars on radar data corrected by the EM-P approach and if applicable, dynamic objects were corrected by the baseline method Dyn-GT. The best results, of only ego-motion and the best considering dynamic motion correction are marked in bold. | 3D | BEV
---|---|---
Accumulation strategy |  SR |  LR |  SR |  LR
1 scan |  30.0 |  9.5 |  42.7 |  22.1
3 scans |  35.9 |  12.8 |  47.2 |  26.9
5 scans |  35.1 |  12.9 |  48.6 |  28.8
8 scans |  35.7 |  12.0 |  49.1 |  26.4
10 scans |  35.3 |  10.2 |  49.3 |  26.1
5 scans; Dyn-GT |  36.3 |  13.5 |  47.6 |  27.1
10 scans; Dyn-GT |  35.4 |  15.4 |  47.0 |  26.2

(a) Ground truth

(b) Prediction

Figure 3: BEV perspective of the instance association on an accumulated point
cloud. The ego-vehicle is located at the origin of the red arrow (outside of
image). Blue: static points; points in other colors than blue belong to object
instances and the same color indicates the same instance ID.

###  4.3 Learning-Based Estimation of Dynamic Motion and Consideration for the
Accumulation

The evaluation in Sec. 4.1 has shown that the PCAc approach can estimate the
ego-motion comparatively well, concluding that the PCAc approach works well on
radar data too. Similar effects can be observed considering the instance
association. Fig. 3 shows that the model can distinguish between different
instances, but not all objects are detected correctly. For example, the model
could not estimate the purple-colored object in Fig. 3a. The dark green
colored object in Fig. 3a has been estimated correctly, but not all of the
points that belong to the object are correctly predicted as part of an object.
Above the light green object in Fig. 3a is another object (observable in
camera data) that is not labeled but predicted in Fig. 3b. The red points in
3b are predicted as dynamic while not belonging to any object. These effects
could be a result of imperfect labels. Similar effects can be observed for the
VoD lidar data.

Utilizing the object motion for the separate treatment of dynamic objects in
accumulated point cloud for object detection shows a clear advantage for all
approaches in Tab. 3 over just considering the ego-motion in Tab. 1. The naive
approach of correcting the motion by just using $v_{rr}$ performs better than
the label-based baseline. This could be due to errors in the motion estimation
from the labels. PCAc yields the best results. This is partly due to the
inherently improved ego-motion estimation, while the estimation of dynamic
motion improves on the results from Tab. 1 in all metrics but the SR 3D-mAP
measurement.

Table 3:  3D object detection results on the VoD dataset quantified as mAP. All models, irrespective of the motion correction approach, are generated by applying PointPillars on radar data accumulated over 5 frames. The best results are marked in bold. Dynamic motion estimation approach | 3D | BEV
---|---|---
SR |  LR |  SR |  LR
Dyn-GT |  36.3 |  13.5 |  47.6 |  27.1
Dyn-PCAc |  38.7 |  16.0 |  51.4 |  31.1
Dyn-$v_{rr}$ |  37.6 |  13.8 |  50.1 |  29.0

##  5 Conclusion

This work investigated the limitations of the VoD dataset for radar point
cloud accumulation and approaches to mitigate missing ground truth ego-motion.
It is shown that an improved ego-motion estimation improves the object
detection performance. In addition to a better ego-motion estimation, a
learning-based approach to mitigate the accumulation error for dynamic objects
is applied to the dataset. Finally, it is shown that the motion of dynamic
objects can be derived purely from the radar point cloud and that this
improves object detection performance.

For future work, assuming a highly accurate inertial measurement-based ego-
motion estimation will be available, the PCAc should be modified to utilize
this motion instead of the estimated motion. This simplifies the architecture
and might further improve the instance motion estimation. Furthermore,
additional radar features like $v_{rr}$ and the radar cross-section could be
integrated into PCAc for better instance motion estimation. This could also
lead to improvements of the ultimate task of 3D object detection. Eventually,
the PCAc model and the object detector could be trained as a single end-to-end
model to boost the detection performance further.

## References

  * [1] A. Palffy, E. Pool, S. Baratam, J. F. P. Kooij, and D. M. Gavrila, “Multi-class road user detection with 3+1d radar in the view-of-delft dataset,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4961–4968, 2022.
  * [2] O. Schumann, M. Hahn, J. Dickmann, and C. Wöhler, “Semantic segmentation on radar point clouds,” in 2018 21st International Conference on Information Fusion (FUSION), pp. 2179–2186, 2018.
  * [3] Y. Long, D. Morris, X. Liu, M. Castro, P. Chakravarty, and P. Narayanan, “Full-velocity radar returns by radar-camera fusion,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16178–16187, 2021.
  * [4] A. Dewan, T. Caselitz, G. D. Tipaldi, and W. Burgard, “Rigid scene flow for 3d lidar scans,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1765–1770, 2016.
  * [5] X. Liu, C. R. Qi, and L. J. Guibas, “Flownet3d: Learning scene flow in 3d point clouds,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 529–537, 2019.
  * [6] F. Ding, Z. Pan, Y. Deng, J. Deng, and C. X. Lu, “Self-supervised scene flow estimation with 4-d automotive radar,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 8233–8240, 2022.
  * [7] S. Huang, Z. Gojcic, J. Huang, A. Wieser, and K. Schindler, “Dynamic 3d scene analysis by point cloud accumulation,” in Computer Vision – ECCV 2022 (S. Avidan, G. Brostow, M. Cisse, G. M. Farinella, and T. Hassner, eds.), (Cham), pp. 674–690, Springer Nature Switzerland, 2022.
  * [8] Y. Zhou, L. Liu, H. Zhao, M. López-Benítez, L. Yu, and Y. Yue, “Towards deep radar perception for autonomous driving: Datasets, methods, and challenges,” Sensors, vol. 22, no. 11, 2022.
  * [9] M. Meyer and G. Kuschk, “Automotive radar dataset for deep learning based 3d object detection,” in 2019 16th European Radar Conference (EuRAD), pp. 129–132, 2019.
  * [10] S. Baratam, “Radar-guided monocular depth estimation and point cloud fusion for 3d object detection,” 2022.
  * [11] A. Segal, D. Hähnel, and S. Thrun, “Generalized-icp.,” in Robotics: Science and Systems (J. Trinkle, Y. Matsuoka, and J. A. Castellanos, eds.), The MIT Press, 2009.
  * [12] F. L. Markley, Y. Cheng, J. L. Crassidis, and Y. Oshman, “Averaging quaternions,” Journal of Guidance, Control, and Dynamics, vol. 30, no. 4, pp. 1193–1197, 2007.
  * [13] D. Kellner, M. Barjenbruch, J. Klappstein, J. Dickmann, and K. Dietmayer, “Instantaneous ego-motion estimation using doppler radar,” in 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), pp. 869–874, 2013.
  * [14] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, vol. 24, p. 381–395, jun 1981.
  * [15] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12689–12697, 2019.
  * [16] O. D. Team, “Openpcdet: An open-source toolbox for 3d object detection from point clouds,” 2020.
  * [17] P. Palmer, M. Krueger, R. Altendorfer, G. Adam, and T. Bertram, “Reviewing 3d object detectors in the context of high-resolution 3+1d radar,” in Accepted for publication on 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop on 3D Vision and Robotics (CVPRW), pp. 1–10, 2023\.
  * [18] L. N. Smith and N. Topin, “Super-convergence: very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications (T. Pham, ed.), vol. 11006, p. 1100612, International Society for Optics and Photonics, SPIE, 2019\.
  * [19] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1231–1237, 2013.