# Benchmarking with MIMIC-IV, an irregular, sparse clinical time series
dataset

Hung Bui
HCLTech
Hanoi, Vietnam
<EMAIL_ADDRESS>
Harikrishna Warrier
HCLTech
Bangalore, India
<EMAIL_ADDRESS>
Yogesh Gupta
HCLTech
Noida, India
<EMAIL_ADDRESS>

###### Abstract

Electronic health record (EHR) is more and more popular, and it comes with
applying machine learning solutions to resolve various problems in the domain.
This growing research area also raises the need for EHRs accessibility.
Medical Information Mart for Intensive Care (MIMIC) dataset is a popular,
public, and free EHR dataset in a raw format that has been used in numerous
studies. However, despite of its popularity, it is lacking benchmarking work,
especially with recent state-of-the-art works in the field of deep learning
with time-series tabular data. The aim of this work is to fill this lack by
providing a benchmark for latest version of MIMIC dataset, MIMIC-IV. We also
give a detailed literature survey about studies that has been already done for
MIIMIC-III.

_K_ eywords Electronic Health Record $\cdot$ MIMIC-IV $\cdot$ Time Series
$\cdot$ Mortality $\cdot$ Length of Stay

##  1 Introduction

Irregularly sampled time series data occur in multiple scientific and
industrial domains including finance, climate science and healthcare. In
healthcare, electronic health records (EHR) have been widely adopted with the
hope they would save time and improve the quality of patient care. The role of
Artificial Intelligence (AI) in EHR is rapidly transforming the healthcare
landscape, offering new opportunities to improve patient care, enhance
decision-making, and optimize healthcare operations Shukla and Marlin (2022).
Time-series data is routinely collected in various healthcare settings where
different measurements are recorded for patients throughout their course of
stay. Predicting clinical outcomes like mortality, decompensation, length of
stay, and disease risk from such complex multivariate time-series data can
facilitate both effective management of critical care units and automatic
personalized treatment recommendations for patients Tipirneni et al. (2022).
However, modeling time series data subject to irregular sampling poses a
significant challenge to machine learning models that assume fully observed,
fixed-size feature representations Shukla and Marlin (2021). MIMIC is
mentioned as a pioneer and prominent large-scale openly available EHR dataset
that addresses the privacy issues of EHR data by carefully de-identifying
patient information. MIMIC-III is a freely available database with 58,976
admissions from 48,520 patients who stayed in critical care units of the Beth
Israel Deaconess Medical Center between 2001 and 2012. The database records
information such as demographics, vital sign measurements, laboratory test
results, medications, imaging reports, etc. MIMIC-III has played an important
role in inspiring large amounts of scientific works in clinical informatics,
artificial intelligence, and epidemiology. The latest version of the dataset,
MIMIC-IV is an update to MIMIC-III, which incorporates contemporary data and
improves on numerous aspects of MIMIC-III. It adopts a modular approach to
data organization, highlighting data provenance and facilitating both
individual and combined use of disparate data sources. MIMIC-IV is aimed to
continue the success of MIMIC-III and provide a broad set of healthcare
applications. The MIMIC-IV comprises EHRs from the patients admitted at Beth
Israel Deaconess Medical Center in Boston, MA, USA, from the years 2008 to
2019. It contains data of 383,220 patients admitted to an intensive care unit
(ICU) or the emergency, which is considerably larger than previous MIMIC-III.
In this paper, we present a study that verifies different algorithms for
predicting irregular, sparse clinical data with MIMIC-IV, and provide a
detailed benchmark for it. The remainder of this paper is arranged as follows:

  * •

Related work: Brief review some of deep learning models that have been trained
with MIMIC dataset and then describe some similar work that present benchmark
for MIMIC-III, and different data pipeline for MIMIC III and IV dataset.

  * •

Experiment: Describe training pipeline, models and their hyperparameters that
will be included in the experiment. Then, give conclusion about the training
result.

##  2 Related work

The EHR data are known to be messy, often consisting of high-dimensional,
irregularly sampled time series with multiple data types and missing values,
which requires large effort to build systematic, producible extracting and
preprocessing pipeline. Many works have tried to reduce the time and effort
spent on labor-intensive data extracting/preprocessing by providing a standard
pipeline for extracting and processing the MIMIC data. MIMIC-EXTRACT, Wang et
al. (2020) is a popular pipeline for MIMIC-III that uses a larger set of
variables and includes ventilation, vasopressors, and fluid bolus therapies,
expanding the tractability for downstream task. Tang et al. (2020) introduce
FIDDLE (Flexible Data-Driven Pipeline), that developed generalizable tools for
EHR feature extraction, that can be applied for both MIMIC and eICU dataset
Pollard et al. (2018). However, both MIMIC-EXTRACT and FIDDLE are applicable
for MIMIC-III dataset only. Jarrett et al. (2021), proposes a unified, end-to-
end, autoML-friendly pipeline that serves as a software toolkit, empirical
standard, and interface for optimization. However, Clairvoyance does not
assist in cohort extraction steps but provides different preprocessing and
imputation techniques for time-series medical data, and it also works with
MIMIC-III only. A recently proposed pipeline for MIMICIV is COP-E-CAT Mandyam
et al. (2021) that can configure preprocessing steps, providing a user some
flexibility to define cohorts as needed for analysis tasks. The limitation of
this work is that it does not support non-ICU data, and it leaves out
readmission, and LOS prediction tasks. One of the recent works is Gupta et al.
(2022), that provides a pipeline for MIMIC-IV. This work has different
modules, forming end-to-end pipeline: data extraction, data preprocessing,
predictive modelling, and evaluation. It also provides add-on features such as
model calibration and fairness evaluation. However, the number of supported
predicting models in this pipeline is limited. Another recent work is
presented by Liao and Voldman (2023) which supports both MIMIC-IV and ICU
datasets, the two most popular public clinical datasets. However, this work
only provides predictions for in-hospital mortality, acute respiratory failure
and shock which excludes length of stay predictions.

The success of deep learning in image and text domains realized by
convolutional and recurrent networks, and transformer models have inspired the
application of these architectures to develop better prediction models for
irregularly time-series data as well. Many researchers have adopted these
models for clinical prediction using MIMIC dataset. A straight-forward
approach to deal with irregular time intervals and missingness is to aggregate
measurements into discrete time intervals and add missingness indicators,
respectively, before feeding them to a classifier. Existing sequence models
address this issue by including a learnable imputation or interpolation
component. Shukla and Marlin (2021) proposed a VAE-based (Variational Auto
Encoder) architecture with a Multi-Time Attention Networks (mTAN) module.
Their primary innovations are the inclusion of a learned continuous-time
embedding mechanism coupled with a multi-head temporal cross attention encoder
and decoder module (the mTAND module) to provide the interface to produce a
fixed-length representation of a time series containing a variable number of
observations. The limitation of mTAN is that it uses a homoscedastic output
distribution that assumes constant uncertainty and the mTAN model’s cross
attention operation normalizes away information about input sparsity. It
becomes problematic in cases where there is variable input density through
time resulting in the need for encoding, propagating, and reflecting that
uncertainty in the output distribution. The Heteroscedastic Temporal
Variational Autoencoder (HeTVAE) Shukla and Marlin (2022) addresses mTAN’s
issues by introducing an uncertainty-aware multi-time attention network
(UnTAN). The UnTAN layer utilizes an attention mechanism to generate a
distributed latent representation of irregularly sampled time series at a set
of reference time points, which can more directly encode information about
input uncertainty. However, both works only consider a very small number of
time series data features in MIMIC dataset for training, and also exclude its
demographics and static features. On the other hand, Self-supervised
Transformer for Time-Series (STraTS) Tipirneni et al. (2022) model, does not
use imputation/interpolation scheme, overcomes these pitfalls by treating
time-series as a set of observation triplets, in which a novel Continuous
Value Embedding technique is utilized to encode continuous time and variable
values without the need for discretization. The model also includes a
transformer component with multi-head attention layers, helping it to learn
contextual triplet embeddings. Another model is worth to mention is Temporal
Convolutional Networks (TCNs) Lea et al. (2017), that combines best practices
such as dilations and residual connections with the causal convolutions, which
can take a sequence of any length and map it to an output sequence of the same
length, just as with an RNN. Bednarski et al. (2022) claimed TCN models
substantially outperform generic recurrent architectures such as Long Short-
Term Memory (LSTMs) and Gate Recurrent Unit (GRUs) on MIMIC-III dataset. Other
methods modify traditional recurrent current network architecture for clinical
time-series to handle missing values and/or irregular time intervals. For
example, Baytas et al. (2017) developed a time-aware long short-term memory
(T-LSTM) which is a modification of the LSTM cell to adjust the hidden state
according to the irregular time gaps. ODE-RNN (Ordinary Differential Equation
– Recurrent Neural Network) Rubanova et al. (2019) uses ODEs to model the
continuous-time dynamics of the hidden state while also updating the hidden
state at each observed time point by utilizing a standard GRU layer. The GRU-D
model Che et al. (2018) is a modification of the GRU cell which decays inputs
(to global means) and hidden states through unobserved time intervals. DATA-
GRU Tan et al. (2020), in addition to decaying the GRU hidden state according
to elapsed time, also employs a dual attention mechanism to preprocess the
training data before feeding them to a GRU cell.

Regarding benchmarks for MIMIC dataset, there are couples of works that
benchmarks on MIMIC-III dataset. Purushotham et al. (2018) provided three
clinical prediction tasks including mortality prediction (in-hospital
prediction, short-term mortality prediction, long-term mortality prediction),
ICD-9 code group prediction (ICD stands for International Statistical
Classification of Diseases and Related Health Problems), and length of stay
prediction. In this work, they include different prediction algorithms for
benchmarking tasks such as SOFA (Sequential Organ Failure Assessment) and SAPS
II (Simplified Acute Physiology Score). Other supervised learning algorithms
are also provided, including logistic regression, random forest, and deep
learning such as Feedforward Neural Network and Recurrent Neural Network. They
also provide an ensemble of FFN and GRU deep learning models to learn shared
representations from multiple modalities. A more recent work is from
Harutyunyan et al. (2019) in which they focused on Logistic Regression and
LSTM-based algorithms with 4 clinical prediction tasks: in-hospital mortality,
physiologic decompensation, length of stay (LOS), and phenotype. They also
formulate a heterogeneous multitask learning problem that involves jointly
learning all four prediction tasks simultaneously. Beside the numbers of
supported prediction tasks between Purushotham et al. (2018) and Harutyunyan
et al. (2019), there is one critical different is that in Purushotham et al.
(2018) the data input is either the first 24 or 48 hours for all prediction
tasks, while in Harutyunyan et al. (2019), they do LOS and decompensation
prediction at each hour of the stay, and do phenotyping based on the data of
the entire stay. In addition, in Harutyunyan et al. (2019), they frame the LOS
prediction as a classification problem and use Cohen’s kappa score, whereas in
Purushotham et al. (2018), they frame it as a regression problem and use mean
squared error as its metric. The limitation of these two works is that the
scoring methods (SAPS II, SOFA) was not efficient and failed predict the
mortality, and these machine learning and deep learning algorithms are also
outdated and does not compete with current state of the art models in term of
time series prediction for sparse and irregular clinical data. Another thing
is that these works do benchmark for MIMIC-III dataset only. All mentioned
works were implemented with different data extraction, data preprocessing,
evaluation metrics. As a result, it is needed to have standardized
benchmarking with same data pipeline for verifying those algorithms. In our
work, we present detailed benchmarking results of deep learning models on
MIMIC-IV dataset for two clinical prediction tasks including mortality
prediction and length of stay prediction. The in-ICU mortality prediction is
predicting the risk of in-ICU death in patient, whereas the patient ICU length
of stay can be defined as the number of days that an in-patient will remain in
ICU during a single admission by subtracting day of admission from day of
discharge. We formulate the length of stay problem as a binary classification
whether the patient’s length of stay is smaller than 3 days or not. We do not
consider readmission prediction since it is required to process the notes
which plays an important role for model to learn Hsu et al. (2020).

##  3 Experiments

###  3.1 MIMIC-IV dataset

MIMIC-IV is the result of a collaboration between Beth Israel Deaconess
Medical Center (BIDMC) and Massachusetts Institute of Technology (MIT), which
was first released with version 0.3 in August 2020. This work is using MIMIC-
IV 2.2, which was released in January 2023. The data were sourced from two in-
hospital database systems: a custom hospital wide HER and an ICU specific
clinical information system, which are included in two separate modules hosp
and icu respectively, to highlight their provenance. Previously, in MIMIC-III,
the whole data was given as one large set, with no obvious differentiation
between them. This modular organization allows linking of the database to
external departments and distinct modalities of data. All modules can be
linked by identifiers such as subject_id, hadm_id, and de-identified data and
time. Those two modules have been included in MIMIC-III. MIMIC-IV has also
improved their data coverage by renaming icustay_id with stay_id, which is
used to index the stays across different areas of the hospital, i.e a stay in
the emergency department, ICU, and operating room will all be distinct and
referred to by the same identifier. As a result, if a table data were
collected from the ICU clinical information system, it will only provide data
for patients while they are in an ICU. In addition, MIMIC-IV is collected from
2008 to 2019, rather than from 2001 to 2012 like MIMIC-III, which is updated
with more recent biomarkers. MIMIC-IV also has table-wise improvements over
MIMIC-III, that can be seen in Johnson et al. (2023). MIMIC-IV encompasses
both ICD-9 and ICD-10 codes, unlike MIMIC-III, which exclusively contains
ICD-9 codes. The dataset includes 209,359 hospital admissions with ICD-9 codes
and 122,317 hospital admissions with ICD-10 codes, compared to MIMIC-III’s
52,726 documents Nguyen et al. (2023).

###  3.2 Data preparation

Figure 1: Pipeline Overview from Gupta et al. (2022)

In this work, the MIMIC-IV dataset is extracted and preprocessed by using the
work from Gupta et al. (2022). They provide the complete pipeline from
extracting data, preprocessing, training, and evaluation which is described in
Figure 1. This pipeline supports MIMIC-IV 2.0, however it is also able to be
used with the latest, MIMIC-IV 2.2. Regarding evaluation, we utilize area
under the ROC curve (OC-AUC score) and area under precision-recall curve (PR-
AUC score). One of the reasons for selecting this pipeline for our work is its
extensibility which helps us to customize the pipeline based on our
preference. Another reason is that they provide rich features for extracting
and preprocessing data, especially for time series features. We used the data
which were collected from ICU only, and the predict tasks include mortality
prediction, length of stay prediction. We ignore the admission prediction,
since it requires additional text feature for training with large language
model, which is out of scope for this work.

Table 1: Data extraction and preprocessing configuration Data Extraction | Data Preprocessing
---|---
1\. MIMIC-IV 2.2 | 1\. Group Diagnoses codes for with ICD-10
2\. ICU | 2\. Produce data summary for features
3\. Mortality first 48h, LOS >=3 | 3\. Feature selection for Labs/Vitals
4\. Admitted for Chronic Kidney Disease | 4\. Outlier removal with threshold 0.98
5\. Diagnosis and Labs/Vitals | 5\. 2-hour time series resolution
| 6\. Forward fill and mean imputation for labs/vitals
| 7\. First 48 hours of admission data

Table 1 illustrates the detailed data extraction and preprocessing steps for
the pipeline, which is proposed, and experimented by Gupta et al. (2022), and
fully provided by the pipeline. During data extraction steps, we used only the
data collected from ICU, and admitted for Chronic Kidney Disease only. We also
select the features from Diagnosis and Labs/Vitals tables, which contains 407
labs/vitals features, 1034 diagnosis features for mortality prediction task,
and 408 labs/vitals features, 1117 diagnosis features for length of stay
prediction task. In the data preprocessing step, the pipeline supports
clinical grouping for diagnosis and medications features, which helps reduce
data dimensionality while preserving important clinical information. The
International Classification of Diseases (ICD) is a medical classification
list by World Health Organization, containing code for diseases, signs, and
symptom, etc. The MIMIC-IV dataset supports both ICD-9 and ICD-10, in which
ICD-10 is its 10th revision, which is the latest. Prior to any grouping of
diagnosis data, the ICD-9 codes are converted to ICD-10, so that all codes can
be grouped by the same standard. After grouping, the pipeline provides
summaries of the features extracted. These summaries are provided in CSV
files, which contain information about the mean frequency and the percentage
of missing values for different lab and/or medication codes. The number of
admissions is utilized for mortality and length of stay training, are 7910 and
12604 respectively.

###  3.3 Prediction Algorithms

Table 2: Hyperparameters/Parameters of predictive models Model |  Hyper-parameters/Parameters
---|---
XGBoost |  objective=binary:logistic
LSTM |  Input size = 152, hidden size = 256, layers = 2, d/o = 0.8, learning rate = 0.001, batch size = 128, epoch = 20
TCN |  Layers=4, filters = 128, kernel size = 10, d/o = 0.8, learning rate = 0.001, batch size = 128, epoch = 20

We compare several algorithms including a traditional machine learning model,
i.e xgboost and two complex deep learning models which are convolutional
neural network as well as recurrent neural network. Here is the list of
models:

  * •

XGBoost (Extreme Gradient Boosting): Use gradient boosting to combine
predictions of multiple weak models, which operates on decision trees,
constructing a graph of if conditions to make prediction. Chen and Guestrin
(2016)

  * •

LSTM (Long Short-Term Memory): Has two mechanisms, i.e hidden, and internal
cell. The hidden state corresponds to the Short-Term Memory (STM) component,
and the cell state corresponds to the Long-Term Memory (LTM) Hochreiter and
Schmidhuber (1997).

  * •

TCN (Temporal Convolution Network): The convolutions are causal, and model can
take a sequence of any length and map it to an output sequence of the same
length with an RNN Lea et al. (2017).

Table 2 summarizes the hyperparameters of the two deep learning models for our
experiments.

###  3.4 Result

We use 5-folded cross validation to verify the model performance, in which the
dataset is spit 80% for training, and 20% for testing. For deep learning model
training, we split the training data further by taking 10% for validation at
the end of each training epoch.

As can be seen from Table 3, XGBoost achieve the best scores in both mortality
and length of stay prediction tasks. On the other hand, LTSM and TCN
performance are quite similar in which LSTM outperforms TCN at mortality task
but falls behind TCN at length of stay task.

Table 3: Performance of different models with MIMIC-IV on 2 tasks: in-icu mortality, length of stay. Task | Mortality | Length of Stay (LOS>3)
---|---|---
Metrics |  AUC-ROC |  PR-ROC |  AUC-ROC |  PR-ROC
XGBoost |  0.87 |  0.52 |  0.76 |  0.73
LSTM |  0.83 |  0.45 |  0.72 |  0.69
TCN |  0.82 |  0.43 |  0.73 |  0.73

##  4 Conclusion

This work is aimed to provide a standardized benchmark for MIMIC-IV, one of
the most popular EHRs datasets. Although much research are focusing on
applying complex deep learning algorithms to MIMIC dataset, traditional
machine learning model like XGBoost still able to outperform, and achieve good
results in mortality and length of stay prediction tasks. In the future, we
will keep updating this work by adding more algorithms, especially state of
the art ones into the benchmark, and also extend it with more prediction tasks
such as phenotyping and admission.

## References

  * Shukla and Marlin (2022) Satya Narayan Shukla and Benjamin M. Marlin.  Heteroscedastic temporal variational autoencoder for irregularly sampled time series.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=Az7opqbQE-3.
  * Tipirneni et al. (2022) Tipirneni, Sindhu, Reddy, and Chandan K.  Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series.  _ACM Trans. Knowl. Discov. Data_ , 16(6), jul 2022.  ISSN 1556-4681.  doi: 10.1145/3516367.  URL https://doi.org/10.1145/3516367.
  * Shukla and Marlin (2021) Satya Narayan Shukla and Benjamin Marlin.  Multi-time attention networks for irregularly sampled time series.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=4c0J6lwQ4_.
  * Wang et al. (2020) Shirly Wang, Matthew B. A. McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C. Hughes, and Tristan Naumann.  Mimic-extract: a data extraction, preprocessing, and representation pipeline for mimic-iii.  In _Proceedings of the ACM Conference on Health, Inference, and Learning_ , CHIL ’20, page 222–235, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450370462.  doi: 10.1145/3368555.3384469.  URL https://doi.org/10.1145/3368555.3384469.
  * Tang et al. (2020) Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens.  Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data.  _Journal of the American Medical Informatics Association_ , 27(12):1921–1934, 10 2020.  ISSN 1527-974X.  doi: 10.1093/jamia/ocaa139.  URL https://doi.org/10.1093/jamia/ocaa139.
  * Pollard et al. (2018) Tom J. Pollard, Alistair E. W. Johnson, Jesse Daniel Raffa, Leo Anthony Celi, Roger G. Mark, and Omar Badawi.  The eicu collaborative research database, a freely available multi-center database for critical care research.  _Scientific Data_ , 5, 2018.  URL https://api.semanticscholar.org/CorpusID:52182706.
  * Jarrett et al. (2021) Daniel Jarrett, Jinsung Yoon, Ioana Bica, Zhaozhi Qian, Ari Ercole, and Mihaela van der Schaar.  Clairvoyance: A pipeline toolkit for medical time series.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=xnC8YwKUE3k.
  * Mandyam et al. (2021) Aishwarya Mandyam, Elizabeth C. Yoo, Jeff Soules, Krzysztof Laudanski, and Barbara E. Engelhardt.  Cop-e-cat: cleaning and organization pipeline for ehr computational and analytic tasks.  In _Proceedings of the 12th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics_ , BCB ’21, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450384506.  doi: 10.1145/3459930.3469536.  URL https://doi.org/10.1145/3459930.3469536.
  * Gupta et al. (2022) Mehak Gupta, Brennan Gallamoza, Nicolas Cutrona, Pranjal Dhakal, Raphael Poulain, and Rahmatollah Beheshti.  An Extensive Data Processing Pipeline for MIMIC-IV.  In _Proceedings of the 2nd Machine Learning for Health symposium_ , volume 193 of _Proceedings of Machine Learning Research_ , pages 311–325. PMLR, 28 Nov 2022.  URL https://proceedings.mlr.press/v193/gupta22a.html.
  * Liao and Voldman (2023) Wei Liao and Joel Voldman.  A multidatabase extraction pipeline (metre) for facile cross validation in critical care research.  _Journal of Biomedical Informatics_ , 141:104356, 2023.  ISSN 1532-0464.  doi: https://doi.org/10.1016/j.jbi.2023.104356.  URL https://www.sciencedirect.com/science/article/pii/S1532046423000771.
  * Lea et al. (2017) C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager.  Temporal convolutional networks for action segmentation and detection.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1003–1012, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.  doi: 10.1109/CVPR.2017.113.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2017.113.
  * Bednarski et al. (2022) Bryan P. Bednarski, Akash Deep Singh, Wenhao Zhang, William M. Jones, Arash Naeim, and Ramin Ramezani.  Temporal convolutional networks and data rebalancing for clinical length of stay and mortality prediction.  In _Scientific Report 12_. Nature Publishing Group UK, 2022.  doi: https://doi.org/10.1038/s41598-022-25472-z.
  * Baytas et al. (2017) Inci M. Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K. Jain, and Jiayu Zhou.  Patient subtyping via time-aware lstm networks.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’17, page 65–74, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450348874.  doi: 10.1145/3097983.3097997.  URL https://doi.org/10.1145/3097983.3097997.
  * Rubanova et al. (2019) Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud.  Latent ordinary differential equations for irregularly-sampled time series.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/42a6845a557bef704ad8ac9cb4461d43-Paper.pdf.
  * Che et al. (2018) Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu.  Recurrent neural networks for multivariate time series with missing values.  In _Scientific Report 8_. Scientific Report, 2018.  doi: https://doi.org/10.1038/s41598-018-24271-9.
  * Tan et al. (2020) Qingxiong Tan, Mang Ye, Baoyao Yang, Siqi Liu, Andy Jinhua Ma, Terry Cheuk-Fung Yip, Grace Lai-Hung Wong, and PongChi Yuen.  Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(01):930–937, Apr. 2020.  doi: 10.1609/aaai.v34i01.5440.  URL https://ojs.aaai.org/index.php/AAAI/article/view/5440.
  * Purushotham et al. (2018) Sanjay Purushotham, Chuizheng Meng, Zhongping Che, and Yan Liu.  Benchmark of deep learning models on large healthcare mimic datasets.  In _Journal of Biomedical Informatics_ , pages 112–134, 2018.
  * Harutyunyan et al. (2019) Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan.  Multitask learning and benchmarking with clinical time series data.  In _Scientific Data 6_. Scientific Data, 2019.  doi: https://doi.org/10.1038/s41597-019-0103-9.
  * Hsu et al. (2020) Chao-Chun Hsu, Shantanu Karnwal, Sendhil Mullainathan, Ziad Obermeyer, and Chenhao Tan.  Characterizing the value of information in medical notes.  In Trevor Cohn, Yulan He, and Yang Liu, editors, _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2062–2072, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.187.  URL https://aclanthology.org/2020.findings-emnlp.187.
  * Johnson et al. (2023) Alistair Johnson, Lucas BulgarelliLucas, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark.  Mimic-iv (version 2.2)., 2023.  URL https://doi.org/10.13026/6mm1-ek67.
  * Nguyen et al. (2023) Thanh-Tung Nguyen, Viktor Schlegel, Abhinav Kashyap, Stefan Winkler, Shao-Syuan Huang, Jie-Jyun Liu, and Chih-Jen Lin.  Mimic-iv-icd: A new benchmark for extreme multilabel classification, 2023.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 785–794, New York, NY, USA, 2016. Association for Computing Machinery.  ISBN 9781450342322.  doi: 10.1145/2939672.2939785.  URL https://doi.org/10.1145/2939672.2939785.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780, 11 1997.  ISSN 0899-7667.  doi: 10.1162/neco.1997.9.8.1735.  URL https://doi.org/10.1162/neco.1997.9.8.1735.


# COROID: A Crowdsourcing-based Companion Drones to Tackle Current and Future
Pandemics

Ashish Rauniyar1, Desta Haileselassie Hagos2, Debesh Jha3, Jan Erik Håkegård1
<EMAIL_ADDRESS><EMAIL_ADDRESS>1Sustainable Communication
Technologies Group, SINTEF Digital, Norway
2Division of Software and Computer Systems, KTH Royal Institute of Technology,
Sweden
3Department of Radiology, Northwestern University, USA
Emails: {ashish.rauniyar<EMAIL_ADDRESS>
###### Abstract

Due to the current COVID-19 virus, which has already been declared a pandemic
by the World Health Organization (WHO), we are witnessing the greatest
pandemic of the decade. Millions of people are being infected, resulting in
thousands of deaths every day across the globe. Even the world’s best
healthcare-providing countries could not handle the pandemic because of the
strain of treating thousands of patients at a time. The count of infections
and deaths is increasing at an alarming rate because of the spread of the
virus. We believe that innovative technologies could help reduce pandemics to
a certain extent until we find a definite solution from the medical field to
handle and treat such pandemic situations. Technology innovation has the
potential to introduce new technologies that could support people and society
during these difficult times. Therefore, this paper proposes the idea of using
drones as a companion to tackle current and future pandemics. Our COROID drone
is based on the principle of crowdsourcing sensors data of the public’s smart
devices, which can correlate the reading of the infrared cameras equipped on
COROID drones. To the best of our knowledge, this concept has yet to be
investigated either as a concept or as a product. Therefore, we believe that
the COROID drone is innovative and has a huge potential to tackle COVID-19 and
future pandemics.

###### Index Terms:

Internet of Things, Sensors, Crowdsourcing, Drones, COVID-19, Machine
Learning, Monitoring.

##  I Introduction

The current COVID-19 virus has already been declared a pandemic by the World
Health Organization (WHO) [1]. We are already witnessing the greatest pandemic
of the decade. Millions of people are being infected, resulting in thousands
of deaths every day across the globe [2, 3]. The count of infections due to
the spread of the virus is increasing at an alarming rate, at least in China,
although COVID-19 vaccines have been successfully developed [4]. We believe
that innovative technologies could help reduce the pandemics to a certain
extent until we find a definite solution from the medical field to handle and
treat such pandemic situations [5]. We must embrace ourselves by leveraging
innovative technology to prepare ourselves to tackle future pandemics.
Application of such innovative technologies should lead to progress, making
our lives better and easier in both a physical and psychological way.
Technology innovation has the potential to introduce new technologies that
could support people and society during these difficult times [6].

Figure 1: Thermal imaging produced by a DJI Mavic 2 enterprise drone equipped
with a thermal sensor for checking people’s temperature deployed in Treviolo,
Italy [7].

Nowadays, drones are commonly used for many purposes such as delivering goods,
surveying, medical goods transportation, etc. These are just a few examples of
using drones and solving real-life practical problems. A comprehensive review
on the use of different technologies to combat COVID-19 are outlined in [8,
9].

Because of the current COVID-19 situation, countries like Italy and China are
already using heat-sensing drones to measure a person’s body temperature [7,
10]. As shown in Figure 1, the thermal imaging produced by drones such as DJI
Mavic 2 Enterprise equipped with thermal sensors for checking people’s
temperature in Treviolo, Italy was already in place during the country’s
lockdown aimed at stopping the spread of the COVID-19 pandemic. These drones
generate an alert, and a police officer acts immediately and goes to check the
person’s temperature. We believe that such authorities (e.g., police officers)
are at risk of being infected by the reported patients. In addition to this,
there will be a lot of burden on the authorities to go and check the
reportedly infected persons if their sensing drones generate many false
alarms. Moreover, we conjecture that the patient’s temperature can not be the
only parameter used to identify such a pandemic virus.

Figure 2: Crowdsourcing-based system model for companion COROID drones.

Recently it has been possible to collect datasets, anonymize them and monitor
a large population according to the General Data Protection Regulation (GDPR)
[11]. This could be one of the promising solutions for identifying an
individual with viral symptoms and monitoring large populations to address the
challenges related to COVID-19 and other related future pandemics.
Crowdsourcing models are believed to help correlate and do analysis with the
crowdsourced real-time data to develop an action plan or decision [12].
Drone technology and its application are being used in many research domains
for different purposes. In the context of combating the COVID-19 pandemic,
many previous studies have addressed how drone technologies can be efficiently
utilized to fight the COVID-19 pandemic [13, 14, 15]. However, most of these
studies have not proposed any architecture that could be effectively utilized
to tackle pandemic situations. There is a potential research gap to address to
propose an efficient architecture that could potentially be realized in a
real-world scenario. Leveraging the idea of using drones for crowdsourcing
health-related sensors data to monitor a large population, we propose an
innovative solution called COROID. COROID is a crowdsourcing-based companion
drone that can tackle current and future pandemics. Our idea is novel as our
companion COROID drone uses crowdsourced health-related sensors data of the
person along with the readings of the cameras such as infrared camera abiding
the privacy laws.
To the best of our knowledge, this idea hasn’t been introduced either as a
concept or product. Therefore, we believe our idea is novel and has huge
potential to tackle current COVID-19 pandemics, and it can also be used for
future pandemics. Moreover, we can use our system to monitor the patients who
are supposed to be self quarantined or in isolation to prevent further the
spread of the disease. In addition, we also believe that the proposed solution
will be efficient and low-cost, and it could potentially be deployed in all
countries.
The rest of the paper is organized as follows. In Section II, we explain our
proposed crowdsourcing-based companion COROID drones. The System model and
added features of COROID drone are outlined in Section III. Users of COROID
drones are presented in Section IV. Finally, conclusions and future works are
drawn in Section V.

##  II Proposed Crowdsourcing-based Companion COROID Drones

We have named our companion drone as “COROID” as it is designed to handle the
current COVID-19 situations, but it can also be extended to tackling future
pandemic situations. COROID is equipped with a camera such as an infrared
camera that can be used to monitor the temperature of people in public places
from a distance. With digitally collecting and using the lifelogging dataset
by utilizing sensors in smart devices and gadgets, GPS location tracking has
been common to monitor a person’s activity to improve their health. Our COROID
drone is based on the principle of using crowdsourcing data from the public,
which can be used to correlate along with the reading of the infrared cameras
equipped on COROID drones. Crowdsourcing enables a mobile ecosystem to collect
a huge quantity of data and share different services among IoT devices [16,
17].

Nowadays, most people use smartphones, smartwatches, and electronic gadgets to
track their health conditions. Such smart devices come in varieties of
flavors. Take, for example, a person wearing a smartwatch that could monitor
temperature, blood pressure, heart rate, breathing rate, blood glucose level,
etc. A person’s smartwatch or electronic gadgets to track personal health are
always connected to their smartphone. Whenever a person visits the public
places where our COROID drones are already deployed in place will read the
temperature of a person using the equipped infrared camera from a distance.
However, it should be noted that the readings through the thermal camera are
not always accurate. Our COROID drone act as an access point as it is capable
of crowdsourcing people’s smart wearables gadget data. COROID will correlate
its infrared camera readings with the crowdsourced data from the smart devices
in the connected secured cloud. In case a person’s temperature is above the
normal readings, and the crowdsourced data suggests abnormalities in a person,
it can send an alert to the healthcare authorities regarding the suspicion of
the infected person. At the same time, it will also generate an alert to the
nearby public and police authorities who are in the transmission range of
COROID. Thus, COROID is believed to work as a companion for the general public
and healthcare authorities to reduce the transmission of the pandemic virus.
After receiving the necessary information of the susceptible users from the
COROID drones, the healthcare authorities can plan out actions and act
accordingly.

##  III System Model

Our crowdsourcing-based system model for the companion COROID drone is shown
in Figure 2. COROID drone is equipped with thermal imaging or infrared
cameras. Also, these COROID drones can act as access points and sense the
smartwatch/health gadgets data through a person’s mobile phone connected to
the COROID drone in the vicinity. The drone receives information from mobile
phones using both WiFi and Bluetooth interfaces in our system model. We
collect information from persons with the COROID application installed on
their mobile phones under their consent to consider privacy concerns complying
with the GDPR regulations.GPS information is also accessible to COROID drones
to estimate a person’s location information, which then consults a mobile
phone registration database of a city to have exact detail and profile of the
person such as phone number, and address details. Mobile phone data has
already been used for emergency management scenarios. A detailed survey on the
use of mobile phone data for emergency management is outlined in [18]. The
COROID drones also have the artificial intelligence capability to detect face
masks and coughing sounds. Face mask detection and coughing sound detection
are useful features to prevent the spread of the virus. More details on the
face mask and coughing sound detection are provided in the following
subsection.

The software infrastructure that integrates sensing and data processing to
categorize a person’s infections has four major components: the Synchronizer,
the Infection Data Hub, the COROID Dialog System, and the Analytics.

  * •

Synchronizer. The COROID drones are installed at various locations around the
city to monitor large groups of population. The raw measurement data collected
by the COROID drones through cameras such as thermal imaging/infrared cameras,
smartwatch/heath gadget, GPS location are synchronized by Synchronizer at each
location. After synchronizing the raw data, the Synchronizer then consults a
mobile registration database to obtain accurate profile information of the
person. This information, along with sensors data, is subsequently sent to a
centralized cloud platform from the sensing site.

  * •

Infection Data Hub. The Infection Data Hub in the cloud records the health
information of a person, resulting in the creation of a complete registration
for each person around the COROID drones. The health-related sensors data and
thermal images are fused together for more robust AI-enabled classification.
It is now possible to use AI on the sensors data and images together for more
robust classification results [19]. The health classification results are then
compared to a pre-defined infection profile. The classifications made are
saved in the Infection Data Hub in the cloud.

  * •

COROID Dialog System. Following classification, the Infection Data Hub
notifies the COROID Dialog System, which transmits the infection probability
result of the selected person (probably virus-infected) on his/her mobile
phone and also to the city hospitals.

  * •

Healthcare Professional Analytics. This module acts as a detailed data
analysis platform for the healthcare professional to investigate health
reports of the person in-depth, enriching the reasoning used to create
restrictions for the probably infected person. The healthcare professional
analytics platform can send the data to hospitals and key performance
indicators (KPI) such as the number of probably infected persons to the local
government for detailed analysis, which can be useful during the pandemic.

The following are the added features and functionalities that could be
deployed on our COROID drones to monitor the crowd during the pandemics
effectively.

###  III-A Public Mobility Monitoring

Human mobility has demonstrated that aggregate and pseudo-anonymized mobile
phone data can help to model the geographic spread of epidemics [20]. The
COROID drones can continuously detect and calculate human density and vehicle
density as an indicator of community mobility based on pseudo-anonymized
mobile phone data and camera data in public areas. It is believed that the
crowded and high-density areas are at high risk of transmitting the virus
during such pandemics. Suppose a designated monitoring area where our COROID
drones are deployed and detect a high density of humans and vehicles. In that
case, it can relay the information to the concerned authority, who can plan
out the action accordingly.

Figure 3: Surveillance camera on a bus in Cannes, France [21]. Figure 4:
Application of AI for face mask monitoring [22].

###  III-B Face Mask Monitoring

With COVID-19, WHO has recommended to wear face mask continuously to avoid the
transmission of the virus. Countries such as France have already taken the
initiative to use artificial intelligence (AI) equipped cameras to check
whether people are wearing masks on public transport [23, 21]. An AI-equipped
surveillance camera being deployed on a bus in Cannes, France, is shown in
Figure 3. This innovative solution could send an alert if a person is not
obeying the rules to wear masks in public places. With the same motive, the
AI-equipped cameras on our COROID drones can automatically monitor and send a
warning signal when it detects a person who is not wearing masks and violating
the rules. A successful use case of monitoring construction workers for face
masks is shown in Figure 4, which is carried out in Ref [22].

###  III-C Social Distance Monitoring

WHO has recommended maintaining at least 1 meter (3 feet) distance with others
to avoid the transmission of COVID-19 [24]. It is highly possible that one may
breathe in the droplets of the infected person when social distancing protocol
is not followed. Therefore, many countries worldwide have made it mandatory to
follow social distancing protocol [25]. In this regard, our COROID drones can
also monitor and warn automatically when detecting distances between people
less than 1 meter.

###  III-D Coughing Sound Monitoring

Coughing sound monitoring entails collecting cough episodes from audio
recordings of various real-world loud situations and then screening for
COVID-19 infection based on cough features. It is now possible to use AI for
cough sound monitoring to detect COVID-19 patients [26, 27]. To have this
feature, our COROID drones can utilize its onboard microphone to capture
people’s coughing sound and then screen it for COVID-19 or other known
infection profiles.

With all these added features, we believe the capability of companion COROID
drones would be significantly higher to tackle pandemic situations in the
future.

##  IV Users of COROID Drones

The users of our COROID drones could be the healthcare authorities, medical
personnel, police officers, government, and the general public who are
concerned about being infected by any pandemic virus. Our COROID drones are a
proactive solution in identifying suspected or infected people beforehand so
that they can be quickly treated without transmitting the infection to others
and the general public. Currently, if a person develops symptoms of any virus
then they have to go to hospital or healthcare centers to get it tested for
the virus infection. In the meantime, the infected person can transmit the
virus to many people even without knowing they already have the virus.
Therefore, our proposed solution can minimize the rates of spreading the virus
and mortality rates by proactively notifying the healthcare authorities or the
individuals so that the infected person can be tested at an early stage. We do
not have any proactive solution to tackle such serious situations until now.
Therefore, we believe that our proposed solution is practical and it can be
beneficial in reducing the transmission of the infection during a pandemic.

##  V Conclusions and Future Works

Monitoring a large crowd for viral infections during a pandemic time could be
quite challenging, as observed in the COVID-19 pandemic. We discussed how
innovative technologies could support people and society during these
difficult times. We introduced the COROID drone, a crowdsourcing-based
companion drone designed to tackle current and future pandemics. We presented
the component details of the COROID drones with added features such as face
mask monitoring, social distance monitoring, cough sound monitoring, etc.
Hospitals can potentially utilize the infection data recorded by our COROID
drones. The local government could utilize KPI values from the infection data
hub for detailed analysis, such as the number of probably infected persons
that can be pretty useful during the pandemic. With all these features, we
believe that the proposed solution will be efficient and low-cost, and it
could be potentially deployed in all the countries to tackle future pandemics.
Field testing of the companion COROID drones to integrate all innovative
features is our interest for future works. In addition to this, we will also
investigate the use of AI for hierarchical multi-modal sensors and image
fusion to devise enhanced classification results that could detect viral
infections in real-time.

## References

  * [1] D. Cucinotta and M. Vanelli, “Who declares covid-19 a pandemic,” _Acta Bio Medica: Atenei Parmensis_ , vol. 91, no. 1, p. 157, 2020.
  * [2] M. Pradhan, A. Chettri, and S. Maheshwari, “Fear of death in the shadow of covid-19: The mediating role of perceived stress in the relationship between neuroticism and death anxiety,” _Death Studies_ , pp. 1–5, 2020.
  * [3] C. Bulut and Y. Kato, “Epidemiology of covid-19,” _Turkish journal of medical sciences_ , vol. 50, no. SI-1, pp. 563–570, 2020.
  * [4] C. Jee. (2022, March) China just reported its biggest one-day covid case increase yet. [Online]. Available: https://www.technologyreview.com/2022/03/15/1047273/the-download-china-just-reported-its-biggest-one-day-covid-case-increase-yet/
  * [5] J. O. Woolliscroft, “Innovation in response to the covid-19 pandemic crisis,” _Academic medicine_ , 2020.
  * [6] Z. Zeng, P.-J. Chen, and A. A. Lew, “From high-touch to high-tech: Covid-19 drives robotics adoption,” _Tourism geographies_ , vol. 22, no. 3, pp. 724–734, 2020.
  * [7] N. France. (2020, April) Drones take italians’ temperature and issue fines. [Online]. Available: https://www.france24.com/en/20200410-drones-take-italians-temperature-and-issue-fines
  * [8] V. Chamola, V. Hassija, V. Gupta, and M. Guizani, “A comprehensive review of the covid-19 pandemic and the role of iot, drones, ai, blockchain, and 5g in managing its impact,” _Ieee access_ , vol. 8, pp. 90 225–90 265, 2020\.
  * [9] D. Vargo, L. Zhu, B. Benwell, and Z. Yan, “Digital technology use during covid-19 pandemic: A rapid review,” _Human Behavior and Emerging Technologies_ , vol. 3, no. 1, pp. 13–24, 2021.
  * [10] P. Jakhar. (2020, March) Coronavirus: China’s tech fights back. [Online]. Available: https://www.bbc.com/news/technology-51717164
  * [11] V. Thambawita, S. A. Hicks, H. Borgli, H. K. Stensland, D. Jha, M. K. Svensen, S.-A. Pettersen, D. Johansen, H. D. Johansen, S. D. Pettersen _et al._ , “Pmdata: a sports logging dataset,” in _Proceedings of the 11th ACM Multimedia Systems Conference_ , 2020, pp. 231–236.
  * [12] Q. Pan, Z. Jin, and T. Tao, “Improving public transportation safety in covid-19 era through crowdsourcing technique,” in _Mapping COVID-19 in Space and Time_. Springer, 2021, pp. 325–346.
  * [13] Á. Restás, “Drone Applications Fighting COVID-19 Pandemic—Towards Good Practices,” _Drones_ , vol. 6, no. 1, p. 15, 2022.
  * [14] S. UNICEF _et al._ , “How Drones Can Be Used to Combat COVID-19,” 2020.
  * [15] A. Kumar, K. Sharma, H. Singh, S. G. Naugriya, S. S. Gill, and R. Buyya, “A drone-based networked system and methods for combating coronavirus disease (COVID-19) pandemic,” _Future Generation Computer Systems_ , vol. 115, pp. 1–19, 2021.
  * [16] A. Lakhdari, A. Bouguettaya, S. Mistry, and A. G. Neiat, “Composing energy services in a crowdsourced iot environment,” _IEEE Transactions on Services Computing_ , 2020.
  * [17] A. Rauniyar, P. Engelstad, B. Feng _et al._ , “Crowdsourcing-based disaster management using fog computing in internet of things paradigm,” in _2016 IEEE 2nd international conference on collaboration and internet computing (CIC)_. IEEE, 2016, pp. 490–494.
  * [18] Y. Wang, J. Li, X. Zhao, G. Feng, and X. R. Luo, “Using mobile phone data for emergency management: a systematic literature review,” _Information Systems Frontiers_ , vol. 22, no. 6, pp. 1539–1559, 2020.
  * [19] Y. Fang, R. Rong, and J. Huang, “Hierarchical fusion of visual and physiological signals for emotion recognition,” _Multidimensional Systems and Signal Processing_ , vol. 32, no. 4, pp. 1103–1121, 2021.
  * [20] N. Oliver, B. Lepri, H. Sterly, R. Lambiotte, S. Delataille, M. De Nadai, E. Letouzé, A. A. Salah, R. Benjamins, C. Cattuto _et al._ , “Mobile phone data for informing public health actions across the covid-19 pandemic life cycle,” 2020.
  * [21] BBC, “Coronavirus france: Cameras to monitor masks and social distancing,” _https://www.bbc.com/news/world-europe-52529981_ , 2020.
  * [22] M. Razavi, H. Alikhani, V. Janfaza, B. Sadeghi, and E. Alikhani, “An automatic system to monitor the physical distance and face mask wearing of construction workers in covid-19 pandemic,” _SN computer science_ , vol. 3, no. 1, pp. 1–8, 2022.
  * [23] J. Vincent, “France is using ai to check whether people are wearing masks on public transport,” _https://www.theverge.com/2020/5/7/21250357/france-masks-public-transport-mandatory-ai-surveillance-camera-software_ , 2020\.
  * [24] WHO, “Coronavirus disease (covid-19) advice for the public,” _https://www.who.int/emergencies/diseases/novel-coronavirus-2019/advice-for-public_ , 2020\.
  * [25] M. Qian and J. Jiang, “Covid-19 and social distancing,” _Journal of Public Health_ , pp. 1–3, 2020.
  * [26] X. Zhang, M. Pettinati, A. Jalali, K. S. Rajput, and N. Selvaraj, “Novel covid-19 screening using cough recordings of a mobile patient monitoring system,” in _2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2021, pp. 2353–2357.
  * [27] L. Orlandic, T. Teijeiro, and D. Atienza, “The coughvid crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms,” _Scientific Data_ , vol. 8, no. 1, pp. 1–10, 2021.